Jeff Dean and Google’s Race to Rebuild the Stack

I have never met a man whose fingerprints were on so many of the machines I depend on without knowing it. Jeff Dean’s are on a remarkable share of the systems that modern computing runs on: distributed storage, large-scale computation frameworks, the design of specialised accelerators, and several machine learning architectures that became standard practice. That is not a career. That is a municipal water supply. Nobody thanks the pipes, and everything stops the moment they fail.

His current focus reflects a shift in the field’s centre of gravity. The interesting questions are no longer only about model architecture; they are about the systems that train and serve models. The questions have moved out of the parlour and into the engine room.

The stack is being rebuilt

• Training runs that span tens of thousands of accelerators and must tolerate constant failure.
• Custom chips designed for specific inference patterns rather than general matrix multiplication.
• Compilers and runtime systems that map enormous models onto limited memory.
• Serving infrastructure that must meet latency budgets at global scale.

Efficiency as the new frontier

For several years the field’s dominant assumption was that capability followed compute — pour in more machines, and more intelligence comes out the other end. That assumption is now complicated by physics and economics: power availability, memory bandwidth, interconnect topology and cooling have all become binding constraints. The wall is not made of mathematics. It is made of electricity, heat, and the bill for both.

The practical consequence is that efficiency research — sparsity, quantisation, mixture-of-experts routing, better parallelism strategies — determines who can afford to run a model as much as architecture research does.

The institutional advantage

Google’s position in this area is unusual because it designs its own chips, builds its own data centres, writes its own compilers and owns its own research organisation. Very few companies can make a change at any layer of that stack without negotiating with a supplier, and a negotiation is a delay with paperwork.

Dean has argued that this vertical integration is the company’s real advantage in the current period, more so than any single model. It is a claim that will be tested as competitors either match the integration or find ways to rent equivalent capability.

Why failure tolerance is the hard part

A training run spanning tens of thousands of accelerators runs for weeks. At that scale, hardware failures are not exceptional events but a continuous background condition, and a system that stops on error will never finish.

The engineering response is checkpointing, automatic rescheduling and deterministic replay, all of which cost throughput. Getting the balance right is a research problem in systems rather than in machine learning, and it is one of the reasons frontier training remains limited to organisations with deep infrastructure expertise.

The specialisation trade-off

Designing chips for specific workloads produces large efficiency gains and a strategic dependency: a chip optimised for today’s model architecture may be poorly suited to tomorrow’s. This is the tension in every custom silicon programme, and it explains why the largest buyers hedge — buying their own and somebody else’s, and keeping both doors oiled.

The organisations that can afford to design for flexibility rather than peak efficiency are the ones with enough volume to amortise several designs. That is a small group, and the gap between them and everyone else is widening.