But let’s recall: this is the summer. So here comes the Special Summer Book Edition of Machine Readable. I’m recommending something fun.

—

How To Rule the World: An Education in Power at Stanford University

By Theo Baker

At the grand old age of 21, Theo Baker graduated from Stanford and hit The New York Times bestseller list. At the core of his book is a David v. Goliath thriller: a series of freshman articles published in The Stanford Daily, exposing scientific misconduct by the university president. The campaign culminates in the resignation of that president, Mark Tessier-Lavigne, and it is hard to recall something equivalent since Watergate. For Nixon and two Washington Post investigative reporters, substitute Tessier-Lavigne and one scrappy undergraduate.

The series in the Stanford Daily won a George Polk Award for investigative journalism. Never had the winner been so young. Now, in his coming-of-age memoir, Baker attempts to parlay the Tessier-Lavigne story into an indictment of Silicon Valley writ large. He is 100 percent entertaining and 90 percent right.

Following the model of This Town, a classic portrait of Washington D.C.’s incestuous culture by the journalist Mark Liebovich, Baker describes the venture capitalists circling Stanford. They are desperate to throw money at gifted students who might found the next billion-dollar startup. To identify them, to get their claws into them, they moonlight as quasi-professors, retain agents on campus, and pay for lavish parties for the most precocious coders. Faced with these incentives, students reciprocate in not-so-healthy fashion: they exaggerate technical accomplishments, they fake it ’til they make it. The story of Theranos, a Potemkin biotech founded by a Stanford undergraduate, hovers in the background. So does the crypto fraudster, Sam Bankman-Fried, who grew up on the Stanford campus. (Bad Blood, the journalist John Carreyrou’s expose of Theranos’s faked data, ranks as another triumph of investigative reporting.)

I met Baker recently and asked if he might have been unfair in picking on Stanford. All elite campuses attract hungry recruiters—perhaps the banks and management consulting firms circling Columbia or Harvard are equally pernicious? Baker responded that tech money is crazier than Wall Street money, and that the cult of youth is more extreme—in his experience, Silicon Valley regards nineteen-year-olds as more bankable than twenty-one-year-olds. I suspect Baker is roughly right on this. Power corrupts, and absolute power corrupts absolutely. Money corrupts, and extreme quantities of money corrupt extremely.

I’ll register two minor quibbles. First, it’s worth recalling an essay by Paul Graham, founder of the Silicon Valley tech incubator Y Combinator. “A Student’s Guide to Startups,” goes the title. Writing in 2006, Graham argued that the two main options for elite graduates–get a job or do a post-graduate degree–were being supplemented by a third: found a company. Hitherto, being a founder had been almost impossibly difficult. Where would you find money? Whom would you go to for advice? But thanks precisely to Y Combinator, these challenges had been relaxed. If you gained admission to a YC batch, you’d get cash and counsel, not to mention the camaraderie of batch-mates.

What does this tell us about Baker’s thesis? It shows that the venture-capital behavior he describes is not evil in principle. To the contrary, you can see it, in some cases, as a leveling force. I’ve often heard people gripe that so-and-so might be talented, but that he started with unfair advantages: he had money and connections. Well, a venture capitalist is someone who bonds with a talented person and provides exactly those advantages—money and connections. This is good for social mobility as well as for entrepreneurship and innovation.

Here is the second quibble. Baker has fun with the anthropology of venture capital. There are networks within networks—a secret Stanford within Stanford. Students who ought to be enjoying a fantastic university are instead wracked with FOMO. But it’s important to recognize that networks are not just status markers that make the excluded feel unwanted. They are the engines that power entrepreneurial ecosystems.

You can think of Silicon Valley as a vast network of networks. Its genius is to circulate money, talent and ideas, these being the three ingredients that kindle startups, combining them in iterative experiments until magic happens. The more networked the Valley, the faster the three inputs combine and recombine, boosting the odds of a hit company.

The role of venture capitalists is to connect up the nodes in this network, maximizing its creativity. A VC is someone who breakfasts with a founder, then has coffees with coders who might be hired by that founder, then meets enterprise-sales people, go-to-market experts, product managers and other useful sorts, all the while drinking more coffee, hopefully decaffeinated, until the day is exhausted. If you want to understand why Silicon Valley overtook other tech ecosystems such as the Route 128 corridor around MIT and Harvard, the answer is that the venture capitalists out West were more numerous and aggressive.

The moral test for this venture-capital aggression is almost the same as the business one. If the money-spraying and the networking generate only failed or fraudulent startups, this is neither humanly uplifting nor commercially useful. It’s a waste to divert youthful talent into frothy startups if all of them go bankrupt. But if the VCs find the next Steve Jobs or Larry Page or Patrick Collison, the result is Apple or Google or Stripe–companies that put talent to good use, creating jobs and human fulfillment, not to mention cool products.

But, as I say, these are just quibbles. So long as it’s agreed that entrepreneurial ecosystems–including the money-spraying and the networking–are generally a force for good, I’m happy to concede that they will experience phases of unseemly decadence. Baker arrived at Stanford in 2022, a few weeks before ChatGPT’s launch triggered a historic gold rush. Today, the amount of cash sloshing around tech is the most extreme ever, and Baker is completely justified in calling this out. With luck, this phase will pass. With luck, the culture will swing back toward a healthier balance.

—

Four points stand out.

1. Frontier labs, however well intentioned, cannot be trusted to control their own technology. OpenAI deserves credit for being transparent after the fact. But clearly it would have been better if a third-party had stress-tested its models’ potential misbehavior before things went wrong. AI risks are no longer theoretical. The time for theorizing about the potential need for oversight is past.

2. The nature of this oversight can be debated. Ideas range from a traditional government regulatory agency (modeled on the Food and Drug Administration) to a loose cabal of AI tycoons keeping an eye on one another (Elon Musk proposes this in his new interview with the editor-in-chief of The Economist, aka my wife). For reasons explained in last week’s Machine Readable, there is much to be said for the middle ground: a self-regulatory agency, as proposed by Demis Hassabis.

3. I haven’t seen other commentators say this, but the OpenAI exploit underscores the case for cracking down on open-weight AI systems. The two escapee models were being tested without guardrails; their alarming shenanigans demonstrate that guardrails are needed. Open-weight AI systems are freely modifiable by users, which is to say that bad guys can remove their guardrails. In the wake of this week’s sorcerer’s apprentice spectacle, surely people see that allowing removable guardrails is nuts?

4. Relatedly, it’s worth checking out Xi Jinping’s speech from last week, in which he recommits China to open-weight models. (Link here to the text, annotated by @MattSheehan.) Before Xi’s declaration, there had been speculation that, given the power of new Chinese systems such as Kimi K3, China might shift its policy in favor of restrictions. But there’s no sign of that so far. Perhaps in hopes of shifting Beijing’s thinking, Treasury Secretary Scott Bessent is said to be planning AI talks with Chinese counterparts, to be held in September. The OpenAI scare makes those talks more urgent. It also boosts the chances that China will get the point.

On the subject of open weights and the need to speak with China, check out the podcast I recorded with Daniel Kurtz-Phelan, the editor of Foreign Affairs. A friend remarked that in this episode I discard my polite English veneer and unleash my inner Australian. Of course I understand that people love open-weight models: they are cheap and customizable. But aviators once loved zipping through the heavens untrammeled by air-traffic control. We have to trade convenience for safety.

In other news, I published a New York Times guest essay pushing back against Sam Altman’s proposal that the government take 5% of his company. Even if the government took 5% of the top dozen tech companies engaged in AI, the dividends from this pot would be insufficient to achieve the declared goal: to protect American citizens from an AI jobs apocalypse. AI-driven profits will accrue to users of AI as well as the producers, so the government should claim a share of this bonanza through a broad company tax, not through a narrow equity appropriation. Unfortunately, the Trump administration is fond of collecting corporate equity stakes, so Altman’s idea stands a fair chance of adoption.

These are wild times. Buckle up.

This week Demis Hassabis, the central figure in my book, The Infinity Machine, published a prescription for AI safety. Hence a slightly longer post than usual.

This was a major moment for Hassabis. Relative to other AI lab leaders—Dario Amodei, Sam Altman—Hassabis is sparing with his written interventions. He is comfortable in the loose format of interviews; he is wary of committing to precise ideas in essays. He loves to riff about mind-stretching ideas; he is cautious about attaching his name to detailed policy recommendations. If Hassabis is going to have an argument, he prefers to do so behind closed doors. As the founder of the London-based DeepMind, he spent three years battling his corporate overlords at Google. He never spoke about it publicly.

The core of Hassabis’s AI manifesto is unsurprising, but the juice is in the details. He wants a national regulatory body to assess models before release, testing whether they could be used for nefarious purposes—cyberhacking, bioweapons research, and so forth. The United States and other countries have already experimented with national AI safety institutes: so far, so obvious. But Hassabis adds three creative elements.

1. Rather than building up its existing, threadbare public regulatory apparatus, the United States should create a federally overseen but industry-funded self-regulatory body. This would short-circuit the need for a drawn-out fight over Congressional appropriations and make it easier to hire top AI experts and amass computing resources. The standards body would lay down best practices for the top AI labs, including strong internal cybersecurity, strict vetting of key personnel, and transparency about new models. After an initial period of voluntary compliance, the overseer would have the power to ban deployment of unsafe models. It might also be able to enforce a coordinated slowdown in AI roll out, should this be needed. Whatever the details, Hassabis’s main point is that the self-regulatory route stands the best chance of generating a well-funded, technically proficient safety team without the world having to wait years for it.

2. Rather than following the usual practice of prioritizing AI models that have been trained on the largest arsenals of computing resources, Hassabis proposes that the new self-regulatory body should choose which models to vet based on capabilities. This shift in selection criteria would have a big practical effect. Vetting only compute-intensive models means focusing exclusively on frontier systems—Anthropic’s Mythos, for example. But the equally serious safety threat comes from follower systems: copy-cat versions of Mythos that will arrive in the next months. Because these followers can reverse engineer Mythos’s accomplishments, they will cost relatively little to train, and so might escape scrutiny in a vetting system that filters based on computational resources. And because these followers, notably the Chinese ones, tend to release models on a freely downloadable, “open weight” basis, there is nothing anyone can do to stop bad guys from abusing them. Hassabis’s capability-based vetting might be a first step toward closing the follower-lab loophole. Indeed, his essay specifies that open-weight models should be subject to it.

3. Rather than aiming for an international regulatory effort, Hassabis proposes a narrowly American one. This goes against his core instincts: He is an idealistic internationalist who views AI as a technology that should benefit the whole planet. But as a practical matter, Hassabis is making the right call, even it if irritates his fellow Europeans. If the United States goes it alone in setting up a new self-regulatory body, AI labs in other countries will abide by its decisions because they want to distribute their models to American customers. Better to move quickly toward unilateral self-regulation with multilateral spillovers than to waste endless time seeking international consensus.

As interesting as Hassabis’s prescriptions are, more interesting is the fact that he chose to publish anything. He has an acute sense of “ripeness”—he is careful to choose the moment when he is pushing on an open door, not bruising his fist on a locked one. He has no interest in wasting energy on public debates that don’t actually change outcomes.

Hassabis’s decision to publish is therefore a signal that this is regulation’s moment. Policy makers are scared: since the arrival of Mythos in April, they have been forced to reckon with the fact that dangerous AI is no longer hypothetical. In June a White House Executive Order promised that the administration would develop a new benchmarking strategy for AI models, with a deadline of the end of July. Having talked extensively with administration officials in recent weeks, Hassabis’s bet is that he can influence what they come up with.

If he succeeds, he will have achieved something remarkable. The June Executive Order indicates a set of instincts that diverge sharply from his. The Trump team wants to keep regulatory authority within the government rather than empowering an industry-led group. The Trump team wants its AI benchmarks to be classified; Hassabis argues for transparency. The Executive Order is focused on cyber-risks; Hassabis expands that to all risks, notably bio. Perhaps most fundamentally, the EO rules out the vetoing of dangerous models, and nor does it contemplate a coordinated slowdown. That is a long way from the Hassabis manifesto.

If the White House fails to shift in Hassabis’s direction, the post-Mythos regulatory moment will be squandered. Overseeing AI is going to require a well-funded expert body that operates in the open, not a secretive administration committee. The good news is that Hassabis’s manifesto has been endorsed by fellow AI leaders, including Sam Altman, Elon Musk, and Satya Nadella. But the news that really matters will soon emerge from the administration.

~

One more note on this week’s AI news, which I can’t resist adding. Thinking Machines, the AI lab founded last year by Mira Murati, formerly a top scientist at OpenAI, released a new open-weight model called Inkling. Open-weight models from follower labs are popular with businesses right now, because they are cheaper to run than the massive frontier ones. On the other hand, as I noted above, open-weight models are dangerous. In releasing Inkling, Thinking Machines offered an imaginative justification for its use of open weight. It is hilariously ridiculous.

The Wall Street Journal reports:

On Friday, Thinking Machines released its first manifesto, outlining its vision for a future in which AI was decentralized and built on local knowledge. The company, whose CEO, Murati, witnessed the collapse of communism in her native Albania as a child, compared the current dominant AI paradigm of close-source frontier labs to “central planning”—great for bounded tasks like chess and math, but not for the real work humans do every day.

“Central planning fails not because of insufficient intelligence, but because of the nature of productive knowledge: tacit, local, fleeting, and held privately by those who acquired it through their work,” the company wrote, citing Friedrich Hayek. “Attempting to aggregate knowledge for the use of a centralized intelligence faces the same challenge.”

I mean, come on. Centralizing vast amounts of data in one training set works great! This is the essence of the scaling laws! The whole point about large language models is that they have escaped “bounded tasks like chess and math.” You can say, totally fairly, that once you have trained an LLM on a vast trove of centralized data, you will want to apply it to specific data sets that are proprietary. But the grand Hayekian thesis falls flat. I wish Thinking Machines well, but they need to do some Thinking about Messaging.

~

Finally, in other news this week, Rebecca Patterson and I released two episodes of our CFR podcast, The Spillover.

We examined the Trump administration’s enthusiasm for taking equity stakes in private companies, which may go into overdrive if the OpenAI trial balloon (giving the government 5 percent of its stock) is implemented. If the Trump administration wants to turn its current ad hoc purchases into a full-blown sovereign wealth fund, it will have to consider a raft of trade offs that have barely been discussed. Kudos to Jonathan E. Hillman, our CFR colleague, who did the research showing the extent of the administration’s equity purchases.

We also did an interview with Democratic presidential hopeful Rahm Emanuel, who put on a fine display of combative authenticity. He riffed on the math of the midterms, on Israel/Gaza, on fixing U.S. education, and much more. We tried to ding him on his silence about the U.S. budget deficit (embarrassing for a self-professed truth teller), but he gave better than he got. I asked him, a bit nervously, whether a Jewish candidate stood a chance of getting elected, and he hit back with persuasive passion. I gotta say, I hope the answer to that question of mine is yes. Emanuel is formidably intelligent, authentic, and unafraid to be honest. I wish there were more candidates like him.

Thanks for reading. And please give The Spillover a listen.

This week on The Spillover podcast, Rebecca Patterson and I debate the relative merits of gold and Bitcoin as a hedge against dollar debasement. You can watch or listen wherever you get your podcasts; the YouTube version is here. Self-punishingly, I argue the merits of Bitcoin while Rebecca plays the role of gold bug. Our producers scored the outcome as a 3-3 tie, but the truth is that Rebecca had the better of it, because, let’s face it, Bitcoin is a bad idea whose time will probably never come. I had a fun period of crypto curiosity back in 2022, and went about interviewing kids who had made fortunes on imaginative speculations: the ambition to reinvent finance and payments were intellectually thrilling. But when you come right down to it, the dollar-based financial system is already fairly innovative. The world has been changed more by a fintech such as Stripe than by the phalanx of funky pseudo-assets.

But we did have fun debating! Making the best of a bad position, I scored points by citing Bitcoin’s more limited supply, superior portability, and upside potential, though of course I conceded that the upside reflects the prospect of boosterism by corrupt Trumpy insiders plus a bunch of Mad Hatter, long-shot pro-Bitcoin bills in Congress. Rebecca scored points by citing actual data on gold’s usefulness as a hedge against dollar devaluation, and by noting that gold is less likely to be stolen and also marginally useful (though she missed a trick by not wearing a gold necklace on the show). Anyway, we had fun arguing, but the truth, as I say toward the end of the recording, is that the best dollar hedges are probably equities and real estate. Plus, the modern Fed has an excellent record of safeguarding the currency: the need for a hedge is overdone. Sure, the big Covid inflation was an exception. But no financial forecaster, official or private, had a crystal ball on what was happening to output and prices during that murky period.

I’ll sign off with a book recommendation: The God Test, by Robert Wright. An author who operates on the frontier between science, philosophy and religion, Wright is ideally placed to reckon with the tingling mysteries of AI. In an approachable way, his book explains both the functioning of AI systems and the challenges that they pose to humans.

Thanks for reading. All feedback welcome.

Sebastian Mallaby