Yoshua Bengio and the International Case for AI Safety Science

Yoshua Bengio was a co-recipient of the Turing Award for work that made deep learning practical. He has since devoted much of his time to building institutions rather than models. Most men in his position go on to found a company and name a chip after themselves. He has gone into the committee room, which is a slower road and a considerably less photographed one.

His argument is that safety cannot remain an internal discipline of the companies building the systems. It needs independent researchers, shared benchmarks, published methods and international coordination — the apparatus that every other technology with broad consequences eventually acquires.

Institution building

• International scientific reports synthesising what is and is not known about capability and risk.
• Independent research funding that does not depend on the labs being studied.
• Shared evaluation frameworks that allow comparisons across companies and countries.
• Formal channels between researchers and the governments writing rules.

The credibility problem

The difficulty is structural. Independent safety research needs access to frontier systems to be useful, and that access is controlled by the same companies the research would evaluate. Compute for academic labs is scarce. Publishing enough detail to be replicable risks revealing capabilities.

Bengio has been unusually direct that this is a problem without a clean solution, and that the interim answer is transparency requirements written into regulation rather than volunteered by industry. I have heard a great many men volunteer to be inspected on a voluntary basis, and I have noticed that the volunteers tend to be the ones with nothing in the cellar.

The international dimension

Much of his recent effort has gone into cross-border coordination, on the theory that a safety regime that exists in only one jurisdiction simply relocates the risk. That work is slow, diplomatic and easy to dismiss as gesture. It is also the only mechanism available for a technology that does not respect borders.

The compute asymmetry

Academic safety research operates on budgets that are rounding errors compared with industrial labs. This is not only a funding problem: without access to frontier systems, a researcher cannot study the behaviours that matter most, and cannot validate methods at the scale where they would be used.

Proposals to allocate public compute to safety research have been made in several jurisdictions and implemented in a few. The amounts involved are small relative to the industry, and the practical effect is to give a handful of groups enough capacity to do credible work. A bucket against a flood is still a bucket, and it is still worth having when the measurement is what is in dispute.

The publication dilemma

Safety research has an unusual publication problem. A method for detecting a dangerous capability is also a description of the capability, and a defence against a misuse technique can function as an instruction. Researchers increasingly publish methods while withholding the specifics needed to reproduce them.

That is a defensible compromise and clearly an imperfect one. It also means the safety literature is less replicable than the rest of machine learning, which slows progress in exactly the field where progress is most urgent. It is the sort of arrangement a man agrees to and then thinks about at night.