Nov 2021

Regulatory Challenges to Catastrophic AI Risk

Regulation is necessary, overdue, and not sufficient. The hard part is what comes next.

Perfect is the enemy of better

Many factors influence whether regulation helps or hinders catastrophic AI safety, with tradeoffs in every direction. Below are the major factors as I perceive them.

Risk Reduction Factors:

Standard Setting: Regulations set a bar for responsibility and accountability. Standards can become soft law once incorporated into government tenders or embedded in professional credentials, and improved professionalism tends to improve governance and record-keeping.

Public Safety and Liability: Insurance, security red teams, and crisis management facilities limit less-catastrophic risks and may provide survivable early warnings of greater disaster.

Compounding Iterations: Each advance in AI safety builds the knowledge infrastructure needed to mitigate catastrophic risk. The more basic research is funded, and the more career paths a formal safety discipline offers, the greater the likelihood of discovering advances that pave the way to reduced catastrophic risks.

Commercial Opportunities: A marketable safety improvement is a competitive advantage, even if a modest one. Benchmarks that appear in comparison and promotional materials create incentives for innovation and improved standards.

Risk Increase Factors:

Obfuscation: Regulation may drive research underground, to ‘flag of convenience’ jurisdictions with lax restrictions, or into apparently benign cover operations. Dangerous work can hide inside multipurpose technologies, and externalised effects can be obfuscated -- as in the vehicle emissions scandal (Wikipedia).

Arms race: Advances in multimodal foundation models (Transformers, LLMs) such as GPT-3 and DALL-E show that pouring compute into ever-larger models keeps paying off. With no apparent diminishing return on scale, state and non-state actors are scrambling to build the largest models feasible. The arms race can produce rapid, unexpected leaps in capability, and the rush blindsides people to risks -- especially when losing the race looks like an existential threat to a nation or organisation.

Perverse incentives: Incentives are powerful forces inside organisations, and financialisation, moral panic, or fear of political danger can drive irrational or incorrigible behaviour.

Postmodern Warfare: Inexpensive drones and other AI-enabled technologies carry enormous disruptive potential in warfare, especially given their asymmetric nature. Controlling drone swarms requires AI, and this may push the entire theatre of war toward AI delegation -- including interpretation of rules of engagement and grand strategy. (Lsusr, 2021)

Cyber Warfare: Hacking is increasingly augmented with machine intelligence (CISO MAG, 2019), through GAN-enabled password crackers (Griffin, 2019) and advanced social engineering tools (Newman, 2021). Defence is equally affected: only machine intelligence may be fast enough to counter machine-speed attacks. The lack of international cyber war regulations and poor policing of organised cyber crime compound the catastrophic risk to societal systems.

Zersetzung: The human mind is becoming a theatre of war in its own right, through personalised generative propaganda that can extend to gaslighting attacks on targeted individuals, destabilising societies (Williams, 2021). Such technologies are also plausibly deniable -- proving responsibility is difficult.

Inflexibility: The German military after WW1, forbidden from developing artillery, developed powerful rocket technologies instead -- because rockets were not subject to the same rules. Inflexible regulations create exploitable loopholes, and may fail to accommodate new technologies or even improved industry standards.

Another example is how the Titanic was permitted to sail with not enough lifeboats for everyone due to a primitive Board of Trade algorithm that calculated lifeboat requirements based upon tonnage and cubic feet of accommodations, which became outdated due to scaling factors as ship sizes increased, as well as due to a limited lookup table in the regulations that stopped at 10,000 tons and was not updated.

The inverse could also occur. A rule that ‘any model with a parameter size greater than n must…’ could become meaningless if models become much more efficient, or if parameters cease to be an applicable measure of model power.

Inflexibility also manifests when a solution becomes accepted best practice and anchors against better solutions being innovated or adopted.

Limitation of problem spaces: It may become taboo to let machine intelligence work on sensitive issues or train on controversial (if potentially accurate) datasets. This can limit AI’s ability to make sense of complex problems, frustrating solutions to genuine crises.

Conclusions:

Greater transparency and accountability should be major factors in reducing catastrophic risk: all else being equal, it becomes easier to know a system’s risks and who is culpable for its externalised effects.

On balance I would expect regulation to be generally a beneficial aspect for AI ethics, as long as it is not too inflexible or restrictive, or overly politicised.

Four risk reduction factors against eight risk increase factors, and the verdict that runs against the tally The essay names four factors by which regulation reduces catastrophic AI risk (standard setting, public safety and liability, compounding iterations, commercial opportunities) and eight by which it increases risk (obfuscation, arms race, perverse incentives, postmodern warfare, cyber warfare, zersetzung, inflexibility, limitation of problem spaces). A beam below tips towards the four, because the essay’s verdict is that regulation is on balance generally beneficial. The essay gives no weights, so the tilt reflects the author’s judgement rather than any arithmetic, and it rests on a stated condition: as long as regulation is not too inflexible or restrictive, or overly politicised. Risk reduction factors (4) Risk increase factors (8) Standard Setting Sets the bar; standards can become soft law via tenders. Public Safety and Liability Insurance, red teams, crisis facilities: early warnings. Compounding Iterations More safety research builds the knowledge infrastructure. Commercial Opportunities A marketable safety improvement is a competitive edge. Obfuscation Drives research underground or to lax jurisdictions. Arms race The rush to the largest models blindsides people to risks. Perverse incentives Financialisation, moral panic, fear of political danger. Postmodern Warfare Drone swarms push the whole theatre toward AI delegation. Cyber Warfare Machine-augmented hacking; no cyber war rules. Zersetzung Personalised generative propaganda; plausibly deniable. Inflexibility Rigid rules leave loopholes, and anchor against better ideas. Limitation of problem spaces Taboo datasets frustrate AI sensemaking on crises. ON BALANCE: WEIGHT, NOT COUNT 4 factors 8 factors The fulcrum is a condition, not a sum “as long as it is not too inflexible or restrictive, or overly politicized”
The verdict runs against the tally: eight named risk increase factors, four reduction factors, and regulation still comes out ahead. That is because the conclusion is about weight, and the essay assigns no weights: these are “the major factors as I perceive them”, so the tilt of the beam is the author’s judgement rather than arithmetic. What holds the beam up is a condition, quoted at the foot of the figure; remove it and the figure says nothing about which way the beam falls.

It is critical that technology regulation never becomes a partisan issue. Without broad, bipartisan support it will fail: one faction of the population ignores it, while another wields it as a cudgel, wilfully taking behaviour out of context to label people antisocial.

Correspondence

← All writing