The Economics of a Global AI Treaty and Recursive Self-Improvement (w/ Epoch and Stanford's Philip Trammell)
This week we’re joined by Phil Trammell, economist at Stanford and senior researcher at Epoch AI, and co-author (with Chad Jones) of the “When GDP Misleads” paper we covered earlier. Phil is one of the most original thinkers on the economics of AI, and he brought his priors to an unusual assignment: a science-fiction scenario. We read the AI 2040 “Plan A” essay: a future in which the US and China agree to pace and monitor AI development, data centers are placed where they’re easy to bomb (hello, Mongolia), and 2040 arrives with abundance, a citizen’s dividend, and lie detectors. Before reading, the three of us stated our priors on two questions: will an international AI pacing and transparency treaty exist by 2030, and would one with teeth be a good idea? Phil started at two-thirds and 75%; Seth at 30% and 60%; Andrey at 20% and 49%. Along the way: whether the essay’s growth forecast is load-bearing (Phil: it’s close to a wash, and Seth and Andrey push back), why a world where AI looks like nuclear weapons might be easier to coordinate on, Chad Jones’s risk-aversion argument, whether AI itself makes monitoring and commitment cheaper, why monitoring every data center sounds absurd to Andrey and like scrubbers on smokestacks to Phil, mind-reading and privacy, and Seth’s worry that a one-time lump-sum rebate of the universe ends in an extreme power-law distribution of wealth. Then Phil delivers a mini-lecture on renegotiation-proofness, the piece of game theory he thinks the essay is missing, and on why it may be safer to go faster if most risk is state risk rather than transition risk. We close with posteriors, a discussion of Phil’s work on recursive self-improvement (RSI), and a lightning round: why Epoch instead of Stanford or Anthropic, his giving pledge and wild animal welfare, and how a Piketty question from Dwarkesh turned into one of the most-read things Phil has written. Links & References Phil’s work * philiptrammell.com — Phil’s website · Epoch AI * Charles I. Jones & Philip Trammell, “When GDP Misleads: Inferring Living Standards from the Value of a Statistical Life” — our earlier episode on the paper; * Dwarkesh Patel & Philip Trammell, “Capital in the 22nd Century” * Phil’s Epoch piece on parallelization constraints and his in-progress follow-up on whether RSI is imminent. * Aschenbrenner & Trammell, “Existential Risk and Growth” (a Chad Jones tweak on state risk vs. transition risk) — Leopold’s Situational Awareness. * Cunningham et al., “The Economics of Recursive Self-Improvement” (Elasticity Institute; Tom Cunningham led, Phil is a co-author) The essay and the scenario * AI 2040: Plan A — the global pacing/transparency treaty, data centers in Mongolia and Canada, the “citizen’s dividend” * Dan Kokotajlo and the AI Futures Project, authors of the essay * Lydia Nottingham’s reply to the “if everyone would just” framing of coordination plans. Papers, concepts & people discussed * Chad Jones, “The AI Dilemma: Growth versus Existential Risk” — willingness to take on existential risk as a function of wealth and mortality; our earlier Chad Jones episode * Hall & Jones, on the value of life and the rise in health spending — people get more risk-averse as they get richer * Armstrong, Bostrom & Shulman, “Racing to the Precipice” — why you want a big gap between the leader and the follower * Renegotiation-proof equilibria and the grim-trigger strategy in a repeated prisoner’s dilemma. * C.S. Lewis, The Abolition of Man — technology and the “middle of history” when we’re most free to plan the future * Thomas Piketty, Capital in the Twenty-First Century and Brad DeLong’s early point about capital substituting for labor * Gell-Mann amnesia, which Phil invokes on the verification discussion * Tom Davidson’s work on takeoff speeds and automating AI R&D, e.g. Davidson & Eth); algorithmic efficiency, sample efficiency, and RLVR. Previously on Justified Posteriors * How much should we invest in AI safety? — existential risk vs. growth with Chad Jones * Our “When GDP Misleads” episode. Chapters * (00:00) Cold open * (01:22) Intro, and our priors on an international AI pacing treaty * (08:18) The evidence: a walk through Plan A and its 2040 utopia (Mongolia included) * (11:25) Is the growth forecast load-bearing? Immortality, the FDA, and a unipolar 1990s * (19:06) AI as a normal technology, and could AI make coordination easier? * (24:39) Monitoring every data center, and can the plan last until 2040? * (31:02) Lie detectors, mind-reading, and privacy * (34:38) Concentrated wealth: lump-sum rebates vs. power-law inequality * (37:59) Renegotiation-proofness and Racing to the Precipice * (42:14) Can competition deliver safety? Profits, moats, and a spirited disagreement * (44:48) State risk vs. transition risk: is it safer to go faster? * (46:29) Posteriors * (53:03) Recursive self-improvement: is it imminent? * (1:02:58) Lightning round: Epoch vs. Anthropic, the giving pledge, and the Piketty/Dwarkesh essay Justified Posteriors is the podcast that updates its beliefs about the economics of AI and technology, hosted by Andrey Fradkin and Seth Benzell. If we changed your priors, subscribe and keep your posteriors justified. Transcript:Cold Open [00:00 - 01:22] [00:00:00] Andrey: Do we think that we’re gonna have an international AI pacing and transparency treaty by 2030? Phil: In the extreme case where AI was only disruptive, it would be easiest to coordinate a slowdown. What makes it hard is the prospect of abundance if we don’t. Seth: Isn’t this a universe that’s inevitably setting itself up for an extreme concentration of wealth? Phil: How you feel about that depends on whether you think there’s something intrinsically undesirable about wealth inequality. Not sure there is. Andrey: One of the requirements of this plan is to create monitoring devices and monitoring software in every single data center in the world. Seth: Will the Turing police come to my house if I have too nice a graphics card to play my video games? Phil: It seems a bit inconsistent to be worried about that, but then not feel that AI would really lower the cost that would go into a regime to coordinate its development or deployment. Andrey: What if Russia builds a data center and decides to just go for it? Phil: You really want a big gap between leader and followers, that the leader can slow down for safety purposes. But if we’re all sharing information and the coordination is fragile, then when it unravels, we’re as neck and neck as can be. Their proposal to mitigate that, of course, is for us to destroy each other’s big data centers in Mongolia and Canada. Andrey: Suppose countries actually are going to abide by it and it substantially slows down the pace of AI progress. Is that a good or a bad thing? Intro and Our Priors on an AI Treaty [01:22 - 08:18] [00:01:22] Seth: Welcome to Justified Posteriors. This week, we have on the incomparable Phil Trammell of Stanford and senior researcher at Epoch AI to talk to us about AI-driven economic growth. But first, we’ll get his views on AI 2040’s Plan A, a vision of a possible future where AI is globally regulated for the good of all. I’m Seth Benzell, always ready to pause my work for a good time, coming to you from Chapman University in sunny Southern California. Andrey: And I’m Andrey Fradkin, coming to you from San Francisco, California. Very excited to have Phil on the show, who I consider one of the most original thinkers in this space, and also one of the most, morally upright. So welcome to the show, Phil. Phil: Oh, thank you. I’ll try to live up to that. Andrey: I did wanna kind of ask for our priors. And in particular, do we think that we’re gonna have an international AI pacing and transparency treaty by 2030? Phil: I think it’s far from a sure thing, but I’ll, I’ll give it two-thirds. Seth: Two-thirds probability. We get a sixty-six percent chance for Phil. I have to say, I put it closer to thirty percent. I think there’s a real probability, but I think there are some big, implementation problems, some of which are discussed in this, essay and some of which aren’t. I think there’s also big American and Chinese domestic policy issues. You know, I think my probability would be closer to two-thirds, maybe closer to fifty percent, if, you know, we were guaranteed to not have a, quote, America first president in 2028, but you always have to have some probability of that happening. I also want people to think about there’s kind of a... Some distributional aspects of what’s gonna be going on in this policy. I don’t think it’s a pure Pareto improvement, especially the kinds of people who might kind of really want acceleration of technologies by a couple of years might make a big difference to them. People who are maybe experiencing health challenges, or have some money in the game that’s betting on that, will definitely be a force in domestic policy in both countries for against slowdowns. So I’d be closer to thirty percent. Andrey: Yeah. And I think I’m more on the twenty percent range. I actually think there will be a bill that will be framed as an international AI pacing and transparency treaty. I just think it will have no bite, so I don’t count that. You know, a a lot of people have, you know, they look at the successes of international, policy agreements with regards to chemical or nuclear weapons. But I look at the failures, and in particular, the League of Nations or even the United Nations, and many other international bodies which make a lot of proclamations, but ultimately have no actual bite. And so I think what is more likely is a unilateral slowdown on behalf of the US simply due to political pressures from people who dislike, AI for a variety of reasons. So I think that’s much more likely than an international AI pacing agreement. But I’m curious, you know, as we discuss how, our beliefs will update. The second prior that we wanted to, put out there, this is a much more difficult prior in some sense, is do we think, such an international AI pacing and transparency treaty is desirable? In particular, suppose there is one that has bite, so countries actually are going to abide by it, and it substantially slows down the pace of AI progress. Is that a good or a bad thing? [00:05:11] Phil: Yeah. I think it is probably a good thing. It depends obviously on the details, and I can see a few ways that it would go wrong. But I think basically, there’s a lot of room for improvement here, and the burden of proof is really on someone to think that, like, coordinating to get close to the Pareto frontier actually makes things worse. I’m gonna say seventy-five percent. Seth: I’m actually less far away from you going in than I thought I would be. I’m closer to sixty percent. I think on balance, you know, just framed as international pacing and transparency, you know, I’m... I think the most kind of broadly robust argument from that is something like an arms control logic, right? I th-- I would worry if I am she or president of the United States about someone attaining a really devastating cyber first strike capability. And we all know how about how, you know, if you have a first strike capability, but you don’t have mutually assured destruction, how kind of dangerous those dynamics can be, and kinda nobody wants that. So I do see there being a little bit of a positive logic for, hey, let’s make sure that no side gains access to a devastating first strike, right? But on the other hand, I guess my main concern is just that this will slow down growth too much. I guess we’ll learn more from the essay about what they think the passive path of economic growth looks like under, you know, kind of with the pause and then maybe implicitly without the pause. But, you know, just ex ante, I mean, slowing down technological development, I mean, you gotta have... It’s a pretty high bar for you to clear for me before I’m really gung-ho about that. Andrey: All right. And then for me, oh, it’s so it’s so detail-dependent for me. It really is. But I think I’m lower than you guys, maybe around fifty percent. And in particular, I worry that any such regulation is gonna overfit on a very specific technological paradigm, whereas there are gonna be lots of dangerous capabilities that are gonna be arising, but also very valuable capabilities are gonna be arising through other architectures, other ways of using AI. Even today’s AI capabilities are, you know, I think we have a lot of room to figure out how to use them. And so, controlling the pace of AI progress is maybe not how I’d frame it. I would more think about controlling the misuse of AI, regardless of what progress level it’s at, I think is a very interesting and difficult problem. And, you know, something to help with that, I think would be good, but I, but I also worry that it’s gonna have drastic costs, in terms of our privacy and in terms of our, autonomy from the government and also in terms of economic growth as well. I would not support plan A, just to be clear. Seth: Is it fair to call you forty-nine percent? Andrey: Sure. You’re... Yes. Seth: All right. Well, then, let’s see if after reading more about this plan and the future it imagines, Andrey and the rest of us change our minds. Let’s go to the evidence. Do, do. All right, Andrey. The Evidence: Walking Through Plan A and Its 2040 Utopia [08:18 - 11:25] [00:08:18] Andrey: Well, this is a very funny form of evidence, Seth. It is essentially a science fiction scenario-... That is painted by the authors of AI 2040. And so let’s kinda walk through exactly what’s going on. So the authors here are very bullish on future AI capabilities, perhaps unsurprisingly. Their kind of conceptual framework for AI’s impact on the economy is that AIs are like human laborers. They’re essentially like a drop-in replacement for a certain type of human labor. And we’re gonna have just a ton of these already, in the next couple of years to the point that AI becomes the most important issue in the twenty twenty-eight election. There is already gonna be massive white-collar disruption and a lot of discourse about the intelligence explosion. Whenever this newly elected government comes into power in the US, they’re gonna have to choose a path. And the path that they prefer is so-called plan A. Seth: They prefer, meaning the authors of this essay. Andrey: The authors of this essay prefer. Which, consists of, an agreement with China for declaring all compute resources, monitoring what they’re being used on, in particular, making sure that there, there’s not too much effort devoted to improving AI models. And the reasoning here is that the AI models are already gonna be really powerful in twenty twenty-nine. It’s just too dangerous to use them. We have no idea how to do alignment yet, and so they should just be very slowly paced. And furthermore, there are gonna be incentives, essentially, tit for tat incentives. So if someone breaks this agreement and starts doing a gigantic training run on a superintelligence, one of the proposals here is to put data centers in places that are not China and the US, for example, Mongolia. And then if China decides to break the agreement and, train a really superintelligent model, the US can bomb Mongolia, and it’s not gonna trigger a nuclear war. Seth: The other way around. Wait, no, China bombs Mongolia to destroy the US data center, and we bomb Mexico to des-destroy the Chinese data center. Andrey: My bad. Yeah, exactly. Yeah, exactly. Yeah, so it’s, so it’s, like, right there. It’s very easy to bomb, -... You know, for one country and not the other. I think that’s kind of the high-level vision. And then- Seth: And in twenty forty, it’s utopia, and we all get a slice of the universe. Andrey: There’s just so much of a boon from AI progress that we’re gonna get all these things that technologists have talked for, which is just abundance everywhere. We’re gonna get UBI/citizen’s dividend. We’re gonna have to figure out what to do after no one needs to work anymore. Yeah, the, I think that’s kind of the high-level version of the plan. I don’t know. Am I missing anything here? Phil: I think you got the high-level points. I think the, like, details of the economic forecast are mostly separable from the plan to reduce AI risk. So I hope that the baby doesn’t get thrown out with the bathwater if you think that some of the economic claims are bathwater. Is the Growth Forecast Load-Bearing? Immortality, the FDA, and a Unipolar 1990s [11:25 - 19:06] [00:11:25] Seth: I wanna pause you on that one for a minute before we go onto your other insights because I was thinking about this too, and it really does... You know, the paper, kind of imagines fifty percent GDP growth by twenty thirty-two and just acceleration after that. And to me, it did feel a little bit load-bearing, right? Because it’s much easier to make this argument that we can afford a shutdown if in the background we’re having enormous growth in material resources, right? Phil: No. I think it’s the other way around, yeah So like in the extreme case where AI was only disruptive, right? It was like nuclear weapons, it would be easiest to coordinate a slowdown. What makes it hard, as you were saying before, is the prospect of abundance if we don’t. Seth: Right. Right. But I’m-- Well, let me give you an example of what I have in mind. Phil: No, I know it can go the other way, that like- ... You, maybe we can afford to half slow down given the other half, you know, of the benefits. Yeah. Yeah. But that’s, and you’re still foregoing the up, foregoing the other half. It could go either way, but I think on balance, it’s, I think it’s close to a wash, which is kind of what I was saying. Like I think that they’re mostly separate. But yeah, if I had to guess, the more it’s like an arms reduction agreement, the better. Or the more, the more likely. Seth: And I’m confused. It seems to me like it’s just a disruptive technology is more like the arms control agreement, which would make it more enforceable. Phil: Yep. That’s exactly what I, what I meant to say. So the more we think that AI has power to destroy but not to create, the more likely it is that we’ll be able to coordinate on slowing it. Even despite the point that, the benefits of AI could, to some extent, pay for the slowdown. Seth: Interesting. Andrey, where do you come down on that? I-- To me, I feel like I’m, I’m an optimist. I always wanna look at the happy vision, right? A vision where it’s like I have to choose between five percent GDP growth and, you know, X percent chance of the world blowing up. That’s a harder choice for me than, you know, choosing between fifty or a hundred percent GDP growth and I can give up... I feel like I can pay for-- I am richer to pay for the world blowing up, is my intuition. What’s yours? Andrey: Yeah. I have a, I have a similar intuition, although I think like some of these trade-offs are not exactly gonna be very salient. From my perspective, I think the crux is gonna be actually on the debate over these things. Like if... I doubt that the debate is gonna be as stark as like, “Oh, if we go for this bill, we’re gonna, you know, we were gonna have fifty percent GDP growth and now we’re only gonna have five percent GDP growth.” Like I just can’t imagine that that’s what the debate is gonna be like. I mean, you know, we could all be wrong, but I think at least given the definition of GDP, I think fifty percent GDP growth is quite unlikely, in this short of a timeframe. So it doesn’t seem very plausible to me, but maybe that’s not even what we care about. And, you know, welfare is not GDP as, you know, we’ve discu- we’ve discussed in the past, specifically with relation to Phil’s papers. [00:14:33] Seth: Andrey doesn’t think it’s load-bearing. Can I give you a twist on it by which it might be load-bearing? So, you know, Chad Jones has these papers about how much existential risk are you willing to take on as a function of kind of thinking about your own mortality, right? And so one of the arguments is that if you think that like we’re pretty-- and this is a stylized version of his arguments, which is like, if you think we’re pretty close to inventing immortality, that’s worth a lot to you, and you might be willing to take on a lot of existential risk today to make sure you get the immortality before you die, right? So I mean, how do you think that plays into this? I mean, I’m thinking about like the person in this future who’s like a retiree in twenty thirty-five and they’ve got cancer and they really wanna let it rip so they can make it to the miracle cure, and they’re gonna be out protesting on the streets. Do you see them as being a force against such an agreement? Phil: Well, they might be in a hospital. No, but I mean, I mean that. Like in general, th-this point that people who have life-threatening diseases will be a real political constituency ‘cause they’ll like organize to speed technological development. I think, that isn’t what seems to have happened with the FDA. I think there’ll be an even smaller share of the people that care about AI progress than they are people who care about the FDA. So yeah, I take your point, but I think making sure that like, you know, Jensen Huang gets compensated seems like the more important thing than making sure that the cancer victims get compensated. Andrey: From a political perspective. Phil: Not morally, but from a political account. Yeah. Yeah. Andrey: It just seems like that’s not how policy is made. Like, I mean, I understand the Pareto improvement point, right? Like it, it’s the same point as the UBI point or even just the China shock point in that like, well, if we had known that the China shock would cause these displacements to these workers, then we could have easily compensated them and it would’ve, you know, everyone would’ve been better off under some set of circumstances. But I don’t think in history we’ve ever succeeded in enacting such a policy. So I don’t see why this time is different. Seth: Right. That’s just wars, right? Wars are not Pareto improving, right? It’s their failures of bargaining. I guess, let me ask you a question, Phil. I know you’re not an international relations expert, so, but I wanna hear your take. Would the world be better off if the intelligence explosion had happened in like late nineties geopolitics during the unipolar moment for the United States? Is that like a better future where it’s just one democratic country basically gets to set world policy versus this kind of more multipolar world we live in? Phil: I expect so. You know, just for the same reason that in models of technology races, risk is often lower when there’s a big gap between the leader and the follower, because they can sacrifice more for safety. [00:17:35] Seth: Tell us what would, what would Chad Jones say, Phil? Phil: You know, a point Chad Jones was first known for making was that, people get more risk-averse as they get richer, right? This is the punchline of Hall and Jones on the, value of life and the rise in health spending and life and growth as well, that we might sort of be more willing to put the brakes on technological development to get richer still once we’ve already gotten sort of rich enough. And these would both be reasons to think that if AI really is capable of delivering a lot of abundance, that’s actually an optimistic story for the feasibility of a plan to slow it, you know, for the sake of safety. As you pointed out, though, there are arguments pointing the other direction. There are ways in which we could get a lot richer because of some technology and yet be sort of less satiated, be kind of more keen for yet another proportional, improvement to the technology you know, as compared to a reduction in the chance of death. And one of those channels is via life expectancy, right? So if you have something like linear utility in years of life, then, each doubling is actually more valuable to you than the last. On balance, my sort of central guess is that it roughly cancels out and that your forecasts about the GDP growth or the prosperity that AI will deliver are mostly separable from the question of whether we can coordinate a slow-down. AI as a Normal Technology, and Could AI Make Coordination Easier? [19:06 - 24:39] [00:19:06] Andrey: What if we’re in the AI as a normal technology world? I mean, we’re gonna continue having, you know, two to three percent GDP growth. You’re kind of assuming that the safety concerns are separable maybe from the economic growth concerns, but you might imagine that they’re kind of highly correlated. And in that case, if people see, like, look, we don’t have mass unemployment and we have pretty normal economic outcomes, it’s gonna be a lot harder to convince people that there is some grand safety risk happening. Phil: There’s probably almost always a room for some freight of improvement, right? The question I think is whether it’s, like, worth the cost, worth the investment in doing all of that, you know, negotiating or implementing of monitoring systems or whatever, or like, you know, clever construction of incentive-compatible R&D funding regimes. But, like, even in AI as a normal technology world, well, normal technologies are global public goods, and I think we would wanna coordinate on funding AI more and getting in the way less of each other’s ability to make profit off of creating this public good, you know, by distillation or whatever. Lots of countries do have, you know, agreements to sync up their intellectual property regimes, for instance, and, I don’t think that’s implausible. As the stakes get higher, you are really pessimistic that we would ever kind of sort out something roughly efficient, but I don’t know. I think there are lots of examples in history of both within countries and across countries working out gains from trade, metaphorically speaking and- Andrey: Most years, pairs of countries aren’t at war. Ninety-five percent of country pair years- ... They are not fighting each other. Phil: Yeah. So I think it could go either way, but saying, “Oh, nothing like this has ever happened,” seem- seems a bit quaint. So that’s the first thing I wanted to say. The second thing is, I think there are a lot of ways in which AI, especially if capabilities advance a lot soon, which I think they will, but, you know, which is sort of the premise of this essay. There are a lot of ways in which it might especially accelerate the long historical process of coordination getting easier. Andrey: Ah, cozy and singularity take. Here we go. Phil: In general, when you think about, like, what tends to make cooperation easier, monitoring is a big one. And I mean, this is something we worry about with AI, right? That, like, you know, autocratic governments can use it to monitor its citizens. But, like, it seems a bit inconsistent to be worried about that but then not feel that, like, AI would really lower the cost, at least some of the cost that would go into a regime to coordinate its development or deployment. Another one that’s a little iffier but, maybe more powerful in some contexts is commitment. When we were in a Cold War with the Soviet Union, one way in which we tried to keep the peace was through a sort of dead-hand mechanism where, we would pre-commit to retaliating if the other party defected from the kind of agreement not to, not to use nuclear weapons. You’d want a lot of reliability, of course. But in principle, if you had a system that was reliable enough and, you know, intelligent enough to figure out when they’re actually being bombed or whatever and when it’s just a flock of geese, then commitment could also facilitate coordination. And then just, like, it’s taking an even bigger step back. A common criticism that I’ll see people express about, this plan A or, like, other plans for, coordination between countries or companies is that it sort of, it unrea- realistically treats the group as a single agent. It’s this, “If everyone would just,” right? And there was this great reply I saw someone named Lydia Nottingham make, online, which is, you know, we might be on the verge of technology that will let everyone just for the first time. I’m sort of reminded of this passage from The Abolition of Man, in which, like, C.S. Lewis argues that technology, as technology advances, we get ever more ability to manipulate nature, but people are part of nature. And so there’ll come sort of, some sort of point at the middle of history when, we’re kind of most free to plan the future. And after that, things will be sort of locked in because our descendants were part of the, like, objects being planned by the ancestors. And yeah, I mean, you just think about, like, you know, how uncoordinated all of the, like, hunter-gatherer tribes were, you know, twenty thousand years ago and how kind of, how many fewer actors there are that would need to coordinate today. There’s just, like, a couple big countries. Yeah, I don’t know. Maybe AI will facilitate the kind of the grand process toward a, toward getting toward the global frontier. [00:23:47] Andrey: I guess, like, there’s a lot of optimism on your end on agreement between the parties for what the desired thing is, but, like, let me just say that, like, the various factions, in the AI safety world have not proven convincing of most informed, even of informed peo observers about, for example, the importance of, like, AI takeover risk. Seth: Well, are they disagreeing over risk probabilities, or are they disagreeing over utility? I think Phil is optimistic about utility, maybe not beliefs. Is that fair, Phil? Phil: No, I mean, I think Andrey has a great point that there’s a lot of persistent disagreement over what the risks are. I think there has been some convergence. That’ll continue to happen, so the question is whether it’ll happen quick-quickly enough for an agreement to be made before it’s too late. Monitoring Every Data Center, and Whether the Plan Can Last to 2040 [24:39 - 31:02] [00:24:39] Andrey: One of the requirements of this plan is to create monitoring devices and monitoring software in every single data center in the world. What do we think of that as an idea, right? Taking a step back, it just sounds absurd, right? Seth: Will the Turing police come to my house if I have too nice a graphics card to play my video games? Phil: First off, you say monitoring, like, what’s kinda nice is that we have all these results now and verification on how it’s possible to distinguish between, for instance, inference and training without revealing, like, what the inference is. So, I mean, the idea that, like, there’d be some infrastructure that would be a part of, like, any large plant that has a negative externality of some kind seems more analogous to, like, scrubbers on smokestacks or something. Andrey: I would not trust other people’s software sitting on my computers and, you know, yes, in some abstract theoretical world, this software has the perfect properties, but generally, like, we know that software is highly imperfect. Seth: It would use AI to prove to you that it was good software. Andrey: We know that the history of, the countries involved includes vast amounts of hacking, and corporate espionage and- Seth: But transparency is part of the argument, Andrey. Part of the argument is we wanna make this all transparent. I guess you’re worried about transparency in inference when we’re-- what we want is the transparency in training runs. Is that the distinction? Andrey: I refuse the inference/training run dichotomy here. People are constantly overfitting on the last generation of how AI work is done. You could imagine inference being used to develop new AM-- AI methods that are then able to do very powerful things in new ways that are not detected as training runs. Seth: Wouldn’t you prefer a world with a thousand human-level intelligences to one thousand X human intelligence? I mean, I think that’s implicit in this paper. Phil: I wouldn’t, for sure. But it really depends on what the su-- what we know about the superhuman intelligences that exist. I think superhuman intelligence existing is inevitable, and it’s not even a good way to conceptualize things because there’s superhuman intelligence everywhere around us today. Seth: Phil, what do you think of this inference, infer as much as you want, but, train under highly regulation and public circumstances part of the deal? Phil: So yeah, first off, I do agree with you, Andrey, that the deal kind of assumes that the technology will be more stagnant than I think we can expect. I think what they might say is, well, if one of the things we’re coordinating on is stopping AI R&D, basically, except in, like, ways that w-we can all see and therefore respond to, and like, you know, like, the regulation can respond to it. If we’re banning everything else, then it’s fine to overfit to what we have today or what we have when the agreement’s made because, like, that’s how the technology will, remain. The argument for believing that is, as far as I can tell, is basically that no one will have an incentive to do any big R&D projects because the outputs will all be public, right? So it’s like, you know, if there’s no patent and no trade secrets, then, there’s no incentive to do R&D. But of course, we know that often R&D happens anyway, right? Like, there’s open source software, there’s just hobby projects, there’s philanthropy. [00:28:03] Seth: And even under oligopolistic competition, right, even if, you know, if you can make the pie go up, even if we’re gonna split it evenly between China and OpenAI, it’s still, you know, make pie go bigger. Phil: Yeah. I guess the thought would be, though, that, like, if all the IP is public, then it’ll be a bit closer to perfect competition. Seth: The moat will still be the enormous amount of compute, right? So there’ll still be a moat. It won’t be free, you know, everyone gets to have their frontier model. Phil: Yeah. I mean, it might start out that way, but then things could spiral out of hand if, like, one of the ways in which we make progress allows for more distributed training runs, smaller training runs, and so on. Andrey: Well, I mean, like, yeah, I mean, like, what if Russia builds a data center and decides to just go for it? I mean... And, and/or decides to sell compute to other places, and then they’re gonna say, “Well, you know, only ASML can produce, you know, lithographic devices,” and so on. But I-- But if you are truly bullish on the capabilities of AI even by ‘29, it should be pretty easy to get the AI to tell you how to create these things. And then once you’re able to create them, and then other, you know, nation-states certainly have incentives to train, even if it’s not profitable for their domestic, security, but also for maybe they wanna have some technology that they think is good for their citizens, you know? Phil: Yeah, I think this is all consistent with what I was trying to say before, which is that it seems feasible to me, for a bit at first to really slow down progress and have the sort of arrangement not be overfit because, you know, you remove the main thing driving progress today, the possibility of profit, and, or at least you reduce that a lot. And also because as you were gesturing at, Andrey, the supply chain is currently so narrow that if, you know, you have the Netherlands on board and Taiwan on board, you know, and South Korea or whatever, then there’ll be that long lag before Russia can be producing data centers on its own or whatever. What I was not convinced by in the plan was that it could last all the way till 2040, right? That might be all right. I mean, it, there might still be some sort of differential Technological development where you slow down some risky projects and speed up some safer ones on the whole for five years or something. Andrey: What’s your main reason for why you think it would fail before 2040? Phil: Oh, well, I mean, you were getting at them. So, like, you know, and I, well, I said before, like, it’s a combination of the fact that technology can progress even after you’ve significantly weakened the profit motives and the fact that as technology progresses, the bottlenecks that the plan is sort of relying on, so the narrowness of the chip supply chain being one, can be relieved as advanced AI tells its prompters how to, you know, have a more rapid way of building an EUV machine or a way of, like, doing a successful distributed training run out of all the iPhones in people’s pockets or something like that. Lie Detectors, Mind-Reading, and Privacy [31:02 - 34:38] [00:31:02] Andrey: So another interesting part of their proposal or part of their vision is that we’re gonna have essentially panopticon-like technologies to detect whenever anyone lies. Seth: Oh, yeah, that’s fun. By 2035, we solve politics with a lie detector. Andrey: First of all, I do think that this kind of no privacy dystopia, and not, like, online privacy, but, like, literally, like, people can just read your mind all over the place dystopia is a possibility. I don’t see an obvious reason why AI wouldn’t be able to learn a lot about your thoughts and motivations from analyzing your face in some way. But I’m curious what you guys think. Phil: Okay, I don’t know about the face. I mean, I think if it was just about the face, then it would seem annoying but not impossible to- Seth: It could just put, inject a numbing agent, and then you’d be completely unreadable. Andrey: Well, yeah. Obviously, yes, it could also have more advanced technologies than that. Phil: We already got used to wearing masks for a few years, right? But more kind of concretely, someone I know, actually, Rio Popper is her name, she’s blind and has been working on a startup to basically allow mind reading, at least enough mind reading to, so that people with i-impairments like that can control a computer, control a robot with their mind. We currently have some ways that people can kind of crudely manipulate an object with their thought. I mean, obviously... Anyway, well, this isn’t my, this isn’t my area, but, like, they’re really aiming for, I think, something as flexible as an LLM is today. We just put in a natural language prompt, and it does it. You could just have a, you know, natural language thought, and some machine would go out and do it. But anyway, so that requires a lot of mind reading, and, if they succeed, I see no reason why they wouldn’t also have developed technology that would let people read each other’s minds, at least if the person consents to disclosing what’s in their mind. Yeah, that seems less worrying than the face case ‘cause people at least retain the right to not disclose what they’re thinking. But in some ways not, because there’s a lot you can learn from the fact that someone doesn’t disclose. Seth: Well, we would need to develop norms around it, right? We would develop... And then that, that’s what the essay imagines is that, you know, the politician will be able to say, you know, “Don’t read my mind to see if I’m gay, but you are allowed to read my mind to see if I’m gonna cheat on this, tax bill,” you know, or defect on policy. Phil: Yeah. Yeah. Which I guess we already sort of have for, just, like, search and seizure, like, for private documents, right? You could infer things about someone’s tax fraud or about their sexuality from their, you know, files maybe, but- Seth: And generally, we have good norms, you know, not imperfect norms, but we, we’re able to work with those. Andrey: But I guess, you know, my thought is that this technology could potentially be applied without consent. That’s, I, it’s just hard for me to imagine that it wouldn’t be that way. Seth: I would have my AI fight the AI that you have that’s trying to read my mind to protect my mind. [00:34:04] Andrey: I imagine that, it might be very asymmetric, but... I mean, I mean, I, I’m, I’m optimistic, Seth, for your solution, but actually, and especially in more totalitarian, regimes, I don’t think they’re gonna have any qualms about forcing everyone. You know, at the end of the workday, “Did you work hard today?” And, you know. Yeah. Seth: You worked seventy percent, thirty percent pay cut. Andrey: Yes. I mean, that assumes that humans still have jobs, so I’m sorry for that. But,... Concentrated Wealth: Lump-Sum Rebates vs. Power-Law Inequality [34:38 - 37:59] [00:34:38] Seth: I have one sci-fi question. This is kind of, like, my last, question on this topic, which is, the paper wants to argue that it wants to prevent kind of two bad outcomes. One bad outcome is, like, rogue, out-of-control AI, and the other bad outcome is, like, a bad actor getting concentrated power over the future of humanity, right? And I think the essay does a pretty good job about, you know, explaining why it would be a good idea to do this in order to prevent, like, the rogue, out-of-control AI scenario. But I think it is much less compelling about the concentration of power, and I’ll tell you in particular the thing that I have in mind, right? Which is we’ve created all this enormous wealth in 2040 in the vision of this paper, and then there’s a question of, okay, how are we gonna distribute it? And the answer is basically we’re gonna do a lump sum rebate of the entire universe to everybody. Everybody gets one ten-billionth of the universe, right? Setting aside kind of the fantasticness of that for a minute, you know, Chad Jones and others have worked on models of sort of wealth dynamics, over time. And in these models, what you get is if you, if people are allowed to, own and compound wealth freely, what you get is a power law distribution. A very extreme power law distribution of wealth emerges over time. And in particular, the extremeness of that wealth is increasing in the interest rate, right? This is a universe where the interest rate has gotta be some ungodly, you know, two X a year amount. And it decreases in the birth and death rate. And I think this is also a universe in which death rates come down a lot. So isn’t this a universe that’s inevitably setting itself up for an extreme concentration of wealth? Phil: Yeah, I think, you know, absent a lot of redistribution, that’s what you should expect. Seth: It’s gonna do a one-time redistribution, and then it expects it to solve equality forever. That’s the kind of naiveté I see. Phil: Yeah, absent ongoing redistribution. Right. Yeah, I mean, so this was the thought that motivated this essay with Dwarkesh I wrote almost a year ago now. I guess how you feel about that depends on whether you think there’s something intrinsically undesirable about wealth inequality. I’m not sure there is, right? Or about power inequality, for that matter. I mean, I don’t share, you know, say, Piketty‘s view or a lot of other people’s views that wealth inequality has to have negative externalities. I think sometimes it does, but sometimes it has positive externalities and just kind of depends who who’s rich and what they’re doing with it. Once most income is capital income, those forces which were around all along will just be, like, magnified by the fact that they’re operating on, like, a larger sh-share of our income. Seth: I agree with that. I think that puts us... You know, whether or not you agree with Piketty’s moral calculation, that puts us in, you know, exploding wealth inequality u-universe, right? Phil: No, I... Well, I agree. Yeah. I think there’ll be a lot more wealth inequality, and there’ll be less, there’ll be less of the dynamism. [00:37:38] Andrey: I mean, I view it as a good thing because I think as a human, I wouldn’t be happy living in a world where there was no dynamism. I think I would be profoundly unhappy if I was born and I had no chance to change my lot in life. Seth: Phil, any last thoughts before we move on? Renegotiation-Proofness and Racing to the Precipice [37:59 - 42:14] [00:37:59] Phil: Two things that any other economist listening, could maybe take a look at are, first, you know, obviously th-th-there’s all this work on the game theory of, like, global public goods, right? Global coordination. And indeed, you know, a, a lot of it was developed to think about climate change and, you know, kind of never really got put to use as we were, as we were discussing before. But maybe people who have been, you know, spinning their wheels on that front can finally come riding in on horseback. But just for an example, it seem, it seems to me that, like, the authors of AI 2040, they know some game theory. What they know really well is, you know, AI and the latest results in verification and so on. But an example of a bit of game theory that seemed missing to me was, an analysis of whether the proposal was renegotiation-proof. Okay? So this is just one little, one little observation. It’s just, like, a bit of, you know, game theory catnip or something. But for anyone who doesn’t know, I think this is, like, a, a really neat concept. A subgame perfect equilibrium is renegotiation-proof if it puts us on the Pareto frontier, not just on the equilibrium path, but in every subgame, right? So if someone defects, we move along the Pareto frontier in a direction that they dislike, right, and the other party likes, but there’s never an opportunity to renegotiate and sort of take advantage of further, you know, further possible improvements, even after a defection. So for instance, the grim trigger strategy profile in a repeated prisoner’s dilemma is not renegotiation-proof because we’re just cooperating all the time on the equilibrium path. But if anyone defects, then we just defect from then on, and then we’re back at the origin, and there’s room for gains again. The reason why renegotiation proofness seems important is that presumably the ability to propose some kind of agreement, right, that, and actually kind of have it take is part of the strategy profile that we each have. Otherwise, we wouldn’t have worked out the agreement in the first place and gotten to the frontier in the first place. But you kind of forgot to include that in the game. What that means is that the cost of defection, once you remember that those other strategies you know, are actually available, might be a lot lower than you thought. Because if someone just defects, if we’re playing a repeated prisoner’s dilemma, someone defects, we go back to the origin, then a second later, someone could just be like, “Ah, why are we doing this to ourselves? Let’s just get back to the frontier.” And then, you know, maybe defection wasn’t so unappealing after all. So I think it really is important to think through this kind of thing. And yeah, it’s just, it’s just an example of how may-maybe people who have been working on this for years and decades could, team up with people like the AI Futures Project and, improve it on that front. There are ways in which I think it does kind of embed some deep insights about the game theory. So the point that if we’re all sharing our information, one actual worry with that, pointed out in this old paper, “Racing to the Precipice” by Carl Shulman and Nick Bostrom and Stuart Armstrong, is that, as I was, as we were saying before, you really want a big gap between leader and followers, that the leader can slow down for safety purposes. But if we’re all sharing our information and the coordination is fragile, then when it unravels, we’re as neck and neck as can be. Their proposal to mitigate that, of course, is for, you know, for us to destroy each other’s big data centers in Mongolia and Canada. [00:41:33] Seth: Which I think is really clever, right? Anyway, keep going. Phil: Yeah, well, I’m not sure how credible it is, but, -... You know, it might be credible, and if so, then it sort of largely, overcomes that particular issue. Another thing that I was kind of surprised didn’t come up is that, yeah, people who tend to think that, you know, whatever risks there might be from AI, you know, normal kind of market, the kind of default market outcome is that the risks get solved because, you know, safety itself is valuable and companies can compete on safety. Is that, is that... I took... From earlier conversations, I thought that was something like your view, Andrey. Can Competition Deliver Safety? Profits, Moats, and a Spirited Disagreement [42:14 - 44:48] [00:42:14] Andrey: Certain sorts of safety will be provided by competition, I think. But I do think that extraordinarily rare Events, and especially to the extent that they can be enacted, affected by smaller companies, that’s-- I’m, I’m, I’m especially worried there. And then, and then, like, the science fiction, you know, escape scenario, which I don’t put, like... I put-- just think it’s very unlikely. Very unlikely. But I don’t think those would be well mitigated necessarily by competition potentially. Phil: Okay. Well, at least for the kinds that would be, mitigated by competition, one downside of just sort of eliminating the ability to make profit off of AI R&D is that you’re also eliminating the ability to make profit off of, you know, improvements in reliability and so on. Seth: Yeah, but I would say, I, I-- actually, that’s not a concern for me about Plan A, right? Because to the extent that I do think these companies will make profits. We discussed how one thing that might make them more profitable is banning these training runs, because then they just get to make the operating profits rather than these giant investments, and then maybe they transfer more of their R&D resources into those kinds of safety innovations. Phil: No, but how are they doing that if you can’t do any, like, big experiments and you’re just serving inference? Seth: Well, I mean, the... Again, you wouldn’t be able to do it using training runs. That wouldn’t be the kind of safety, but it seems like there’s other dimensions in which you could be more or less safe. Phil: Okay. I mean, I think experiments usually, at least today and for m-many years, have basically been partial training runs. And so if anything that looks to a little on-ship governance, thing like a training run has to be done in the air-gapped, you know, data center that we can all observe and learn from, then you really are weakening the incentives to do some kind of experiments that would be valuable for safety purposes. Andrey: I just wanna say, ‘cause it triggered me a little bit, Seth, is, like, banning training runs will not increase profits. That’s ridiculous. Intelligence is the most competitive market in world history as far as I’ve seen it. Without a model advantage, it goes to marginal cost pricing. Seth: No. Well, there’s two, there’s... No, there’s two huge moats. Moat number one is you need the giant amount of compute, which not everyone’s able to raise that much capital. Andrey: No. You don’t need that much compute to run inference. So rents will go, well, rents will go to a very different set of companies, which is the compu- the companies who own the compute or it’ll be nationalized. But the OpenAI and Anthropics don’t own the compute. State Risk vs. Transition Risk: Is It Safer to Go Faster? [44:48 - 46:29] [00:44:48] Phil: One of those papers, one of those childlike papers I wrote with Leopold Aschenbrenner years ago- ... And- Seth: Leo. Andrey: Situational awareness. Someone had situational awareness. Phil: Well, he had situational awareness. I just had an idea to sort of tweak a Chad Jones paper, and then he did it. But, you know, the punchline of that paper was that insofar as you think that a lot of risk is what we’d call state risk as opposed to transition risk, so risk that, like, an advanced AI exists and is out there being deployed, but, like, you know, every year it has some chance of going rogue or something. Seth: Isn’t it safer to accelerate, Phil? Phil: Then, th-then it’s, then it’s safer to go faster, right. You need to think that mo-most of the risk is, transition risk for it to be safer to go slower. I think this can totally change based on the circumstances, but, at the moment it does seem to me that the risk is mostly kind of in training rather than deployment, just like, you know, the hugging face incident didn’t happen because, you know, someone used it in an unintended way. It happened in training. And so this is-- I mean, this can totally change, and this is definitely something I... That makes me worry about, like, going all in a plan, on a plan like plan A without any mechanism for modifying it as things change. But hopefully, one benefit of having written that paper is that people can know that I’m not just, like, a knee-jerk person who sort of says-... Like, whenever something seems scary, like, “Ah, just shut it all down.” And yet even so, on balance, I think that it would probably, again, you know, we’ll get to posteriors, but yeah, seventy-five percent or something, it’d be good on balance at the moment. Posteriors [46:29 - 53:03] [00:46:29] Seth: All right, so let’s do it. Posterior is Do, do. Andrey, remind us what we were thinking about. Andrey: Yeah. So will there be a plan like this by twenty-thirty? Seth: I’ll go first. I started at thirty percent, and I think reading this essay actually moved me towards it being more feasible. I think the thing that I thought was really clever that sort of updated me a little bit was this idea of locating the data centers in a way that would be particularly vulnerable to the other side’s attack. That isn’t a thought, thing I had thought about. Maybe that doesn’t literally mean, you know, the American data centers are in Mongolia and vice versa, but I think that if you cleverly set up your data centers to be vul-vulnerable to a mutually assured destruction, that’s the kind of thing that would help this go through. Now, that’s, you know, just one clever idea. In order to get this all to go through, you would need lots of clever solutions to lots of these different issues. Andrey talked about a major one, which is how do you do monitoring in a, in a way that’s trustworthy and reliable? But I moved from maybe thirty percent this happens to, like, thirty-five percent it happens, given that, okay, I can see maybe there’s some clever ideas for some of these. Andrey: I think that this is a m-much less likely plan than a typical coordination plan that I would have considered ex ante. I found it very interesting, thought, thought-provoking, potentially with good pieces, but i-just even the idea of, like, putting the data center somewhere else, well, we have a ton of data centers here, and, like, the amount of things that need to go right for us to remove the data centers from the United States is just insane. So even just that sort of plan, you know, like- Seth: The point is that in an AI twenty-forty universe, this is a universe in which it’s the new data centers that are the important ones. It’s not the incumbent ones, right? Phil: So new data centers that matter partly because they’re being built so fast that the latest ones will be a large share of the total. They’ll also be the, kind of the best. ‘cause technology’s improving so fast. And also, the old ones depreciate relatively quickly. Seth: So, okay, so Phil, give us your posterior. So did engaging with this essay move you from your original thought of the feasibility of a Plan A-like plan? Phil: I’ll be honest, it is a little hard to separate the prior from the posterior. Seth: It’s the alarm for a spicy take coming around. Phil: I guess, like Seth, this idea of putting the data centers in, you know, easy-to-bomb places had not occurred to me before reading it, and I think that does make it a little bit more likely. There’s also just seeing how it’s been received, and I don’t know which way that pushes. I think it’s, that probably makes it a bit less likely. My impression is people have been somewhat dismissive it, of it on the whole. But I guess finally, there’s just the point that, you know, Dan Kokotajlo and these people with uncanny track records have endorsed it, and that’s gotta count for something. So, I don’t know, small updates all around, but, I’d, I’d say I’m, I’m about whatever I said, two-thirds sure. [00:49:43] Andrey: The second one, is it good or, is Plan A specifically good or desirable? Phil: Yeah, I guess, some of the economics and game theory seemed half-baked to me. I’m worried about, like, the Gell-Mann thing, where, I notice all of the holes in the areas I, I’ve, that I’m more familiar with, and that all the stuff on verification, I mean, I read it all, you know, and I have friends that work on this-... And, you know, they seem to think well of it, but maybe there’s all kinds of flaws that I don’t see. So yeah, I still have this strong kinda prior that, it’ll... There’s just, the stakes are really high, and there’s a lot of room for improvement, and we’ll probably figure something out. But something exactly like this, maybe, my posterior’s a little lower than I went in with, so, we’ll say seventy percent. Again, my lack of updates hopefully is not symptomatic of just being the kind of person who doesn’t update. It’s just hard to separate prior from posterior when it’s, you know- Andrey: It’s hard, especially when you know, I read this when it first came out. Seth: I, on the other hand, am able to fully constrain my consciousness to part, pretty good thought expire. Andrey: Using new AI technologies, the same ones that allow you to read people’s minds. Seth: Exactly. It’s funny, Phil, I kinda update in the opposite direction, and I have to say, on the second question, it’s almost a little bit more vibes-based. The future that this, and I think this is, like, the power of science fiction, is that this really was able to put me in the shoes of what it would feel like to be in a real RSI economic takeoff world. And in that world, I can imagine people doing some really dumb stuff just because things are changing so fast. You could get a lot of policies that are worse than this. And so, like, my conditional belief that this is a good idea, conditional on we get, you know, fifty percent economic growth in six, seven years, is, like, maybe, like, ninety-plus percent, right? Now, I put a very low probability on that world, but it’s getting me to think about what you would want things to look like if we were in hard takeoff, moves me into this is, this is a good idea given, you know, the one, two-percent chance we end up there. So maybe I move from a sixty percent chance this is a good idea to, like, a sixty-two, sixty-three percent chance this is a good idea. Andrey: Yeah, I mean, I think, like, conditional on us being in economic growth takeoff land, sure. Although, I mean, I can imagine lots of ideas. This one seems acceptable, not too harmful. But I don’t, I just don’t think that that’s the world we’re gonna be living in a few years, even as capabilities grow. And knowing how politics works, plans like this will be proposed regardless of the economic situation we’re in, and I will probably not support something like this. [00:52:47] Seth: So you started at forty-nine percent. Are-- do you update down, or are you still at- Andrey: Yeah, I’m, I’m, I’m updating down. Seth: You have to give us a number. I’m very- Andrey: So forty-five Seth: ... The premise of the show. Forty-five. Recursive Self-Improvement: Is It Imminent? [53:03 - 01:02:58] [00:53:03] Andrey: Phil, I did wanna ask you some... ‘Cause we haven’t really done an RSI episode, and I know it probably deserves a separate episode. But I know, you were a part of the Elasticity Institute collective that wrote a paper about, recursive self-improvement or self-sustaining, acceleration, as you like to call it. This idea very related to just very fast, progress in AI capabilities and on the importance of measuring it. So do you wanna give a little pitch for that and how it relates perhaps to some of these scenarios? Phil: For sure. Whereas Tom Cunningham took the lead on that first note, I’m actually in charge of a follow-up that we’re writing at the moment. Is RSI imminent? Evaluating kind of the latest arguments and evidence for thinking that, you know, we will get a significant increase in the pace of AI capabilities improvement due to the automation of AI R&D over the next year or something. I mean, my case for just, like, prioritizing this question is that it’s upstream of so much else, right? Including whether it’s, like, urgent for us to get a plan to regulate or internationally coordinate on AI in place soon. The real thing I wanna pitch is that it’s actually probable. At face value, once AI can do AI R&D more or less as a human researcher does, you would expect a significant acceleration in the pace of AI R&D. A lot of arguments have been raised over the years pushing back on that point, and I’ve contributed to many of those arguments, you know? I mean, you can find me in the... Well, I’ve, I’ve, I’ve, I wrote one recently over the summer about, like, the ease of parallelizing research. And, Tom Davidson‘s been kind of investigating this question for years, and I think I’ve given feedback and in some cases pushback on almost everything that he’s written. And just, you know, going through the arguments and, the current state of the arguments and the evidence, I think just that original obvious point holds up and the burden of proof is very much on someone to find a bottleneck that has not been identified and more or less refuted. So, I mean, I can go through all a dozen, but- Seth: Well, let’s do two. Phil: All right. So one that I just mentioned is this point that when formalizing the case for thinking that things would go really fast once we’ve automated AI R&D, the yardstick of improvement that people use to actually you know, quantify, like, how fast things would go is algorithmic efficiency, which more or less corresponds to the question, once we can simulate, you know, Alec Radford and Noam Shazeer, you know, a thousand times over in the data center, how long before we can simulate two thousand of them, right? Or, like, not how long, but, like, the, you know, the doubling in algorithmic efficiency means that you can do the same thing with half as much, compute. Whether you actually get twice as much done or anything close to it with twice as many simulated researchers, twice as much sort of virtual research input, depends on a kind of another dimension of technology, which is how well they can coordinate with each other, right? How well they’ve turned themselves into kind of a productive big firm that’s twice as big. If Hugging Face teaches us anything, it’s that a little RL can go a long way, in terms of, facilitating coordination. Yeah, I mean, just kind of from first principles, you might expect that there are a lot of channels through which AIs can coordinate that people can’t. And so the rate of improvement on that front might be expected to be faster than the rate at which, you know, people can organize themselves into ever larger firms. Yeah. Something that I had thought for a while was that you need a lot of data for AI R&D, not just in the sense that you need the data to learn how to do different kinds of things, but in the sense that you need different kinds of data to learn how to learn about different kinds of things. One way in which training can improve over time is by getting more sample efficient, right? Being able to make do with ever less data. And, like, humans famously are more sample efficient than LLMs at the moment. Depends what you mean exactly, but basically that’s-- I think that’s true. But of course, the rejoinder to that would be, once we have automated AI researchers, they’ll figure out how to get really sample efficient, and so they’ll be able to learn from as little data as, you know, a human learns from. And when you remember all of the data that there is on the internet, you know, all the YouTube videos in the world with a few pixels, with some, you know, axle of a truck going around. You know what I mean? Like, there’s just so much information out there, and if it could learn really sample efficiently, then it’ll be able to learn how to do all kinds of things. The rejoinder would be, well, it might be that feeding in all of, all the data on the internet, we can get to the point of an AI researcher that can figure out really sample efficient algorithms for learning the kind of information that’s on the internet with little data. But that doesn’t necessarily transfer to the project of learning how to, like, be an effective lawyer on some case, some particular case where, like, the you know, the client’s case is private. You’re deployed in the law firm, and you need to learn quickly from that limited data set. You might have just sort of overfit to the internet data. It’s, you know, and so, like, you, you’ve come up with a learning algorithm that is efficient in one context, but not necessarily efficient in every other context. And something Tom Davidson wrote recently has largely convinced me that, even though while you’re not sample efficient, there’s not gonna be all that much transfer learning. Learning algorithms tend to transfer well. So if you learn how to learn efficiently on the, on one data set, that learning algorithm deployed to another data set will also tend to squeeze a lot of juice out of it. [00:58:55] Seth: Right. So if I get, if I get good at, you know, predicting Reddit text, if you just drop me into a simulated lawyer environment, I’ll figure out how to be a good lawyer much quicker just because I’ve learned how to learn on predicting Reddit? Phil: Exactly. It, it’s not that you’ve learned how people talk on Reddit, right? I’m talking about if you automate the AI R&D process, that means that you, you’ve closed the loop, right, on how to learn to talk like a Redditor with ever less Reddit text. That procedure will also help you learn how to be a good lawyer with relatively little legal text. Seth: I guess it strikes me that, like, some of these arguments, so there’s one thing about, like, doing things more efficiently, but I think the parts where things get wonky is when, like, there’s a qualitatively new capability that emerges, right, of some sort. I mean, I mean, in the sci-fi scenario, it’s this artificially super intelligent, you know, agent that seeks power or whatever. And it, like-- ‘Cause if all that RSI was doing was just making existing inference very cheap, existing training very cheap, it would be really important, but I don’t think it, like, poses the sort of, like, catastrophic risk that maybe people are worried about, if that makes sense. Phil: Oh, for sure. I think the old arguments for thinking that, you know, intelligence is intrinsically dangerous and, you know, once you get the RSI loop, then, you know, the kind of, quote, “default,” whatever that means, outcome is that all hell breaks loose, were bad arguments. And it’s just a real shame because, you know, it, it’s so easy to fall into the, like, you know, the fallacy that, what’s it called, the opposite of Bad argument is not necessarily a good argument or something. You know what I mean? Like if people feel inoculated against the idea that this will actually happen, even if for somewhat different reasons. You know, I mean, it turns out, I think it’s safe to say now that, just pre-training on the internet does not produce sort of scheming behavior basically, but that, you know, RLVR, you gotta be really careful ‘cause it often can induce scheming behavior. And that’s something we just learned through trial and error. Seth: It’s an empirical fact. It’s not you can sit down and come up with the orthogonality thesis and, figure it out in Yudkowsky’s bedroom. Phil: Right. Right. But anyway, I mean, the facts are coming in, I would say. Andrey: All right. So you write this paper, let’s say RSI starting to happen in November of this year based on your best estimates. I don’t know if that’s true or not. What do we make of that? I mean, we already see the capabilities on all the benchmarks going up. We see compute efficiency going up already. This is already happening. So what do you expect to happen once we hit the exact threshold for RSI? Phil: I think that capabilities will advance even more quickly. The, nature of the capabilities, the kind of direction of the technical change, put it that way, will be a, a little different in an important way, right? So it’ll be that, capabilities are improving not just because of increases in the inputs, right, the compute and the data and so on, or the data from real world deployment or whatever, but also more of the improvements will come in the form of efficiency improvements, right? The ability to do more with less data and less compute. And there’ll be a kind of interesting qualitative shift, which is that even if we stop increasing the inputs, right? So e-even if somehow we made everyone at, you know, everyone working in the AI industry go home and, stop building new data centers or whatever, capabilities could continue to advance in a way that’s kind of harder to observe. Lightning Round: Epoch, the Giving Pledge, and the Piketty/Dwarkesh Essay [01:02:58 - 01:08:27] [01:02:58] Andrey: Why did you choose to go to Epoch AI? Why not stay at Stanford or do something else? Phil: I mean, it really just seems like AI is going to be the most important topic in the world for the next, I mean, forever maybe, but, certainly for the next few years. And that the academic kind of timelines of trying to write papers that you publish and, the academic expectation of quality versus quantity, you know, the expectation of only writing a few things but having them be very polished, just aren’t very suitable for the moment. So the chance to work with people who are really up to speed on AI, who, you know, get a lot of short things out without agonizing over them for a decade, and, you know, working up in Berkeley at Constellation with other people who are writing things like AI twenty forty, I get to talk to them at lunch, all just seemed m-more appealing than staying down in Palo Alto. Andrey: Why not join Anthropic? Phil: Yeah. So, I was talking with someone about joining the Anthropic Institute at one point, and it seemed to me that environment of, like, learning from other people was probably actually better, if anything, at Constellation than at Anthropic, but at least not a huge benefit to being at Anthropic. M-maybe I would feel differently if I actually worked there, but that was the sense that I had. And, the cost, of course, is that no matter what you say, people will call you a shill if you, yeah, if you’ve worked for the labs or even the institute. Andrey: Why do you care what people say? Phil: Oh, well, because, you know, I’m not a shill, and I want people to update accurately on whatever I have to say. Andrey: All right. Fair enough. Seth: You have, you have a giving pledge, which, we’re very, in awe of your ethical standards in pledging to give away a large share of your income moving forward above a certain consumption level. If we got a twenty forty plan A world, and we really did have, you know, fifty percent economic growth a year, would you allow yourself more consumption? Phil: I mean, I think it would depend on the marginal utility of consumption for me versus for, you know, what wealth could do elsewhere. If all the problems in the world were solved and, animal welfare was taken care of, even wild animal welfare was taken care of, then sure. Seth: But so you-- we need to solve wild animal welfare before you’re gonna lower your cons-- you’re gonna raise your consumption. Phil: Yeah. I mean, well, solve it or somehow make it the case that it’s, like, permanently, definitely impossible to do anything about it. Seth: Okay. A good answer. Solid answer. Andrey: Another lightning round question. How did you and Dwarkesh get connected? Phil: Through a mutual friend named Jess Riedel. Andrey: And did that naturally lead you to write a blog post together? What was, what was that like? Phil: Yeah. So, after we got put in touch, Dwarkesh reached out a few times to chat about, you know, the economics of AI. And at one of those chats, he said, “Hey, you know, I’ve been thinking about that old, Piketty book. Seems like he was kinda out to lunch when it comes to the past, but would be right in the future once capital really did become a gross substitute for labor. Do you have any thoughts on that?” So that was Dwarkesh’s for-- That was, that was him. I mean, you know. I said, “Thoughts? I’ve got slides.” ‘Cause, you know, this summer course that I’ve organized sort of a few times now on, among other things, the economics of AI in the long term, made precisely that point. And I think Brad DeLong made the point way back when the book first came out. Someone did. But o-otherwise, I think I, I’m not aware of anyone else who had. So, that was a happy coincidence, and, he said he would write something and send it to me for feedback. And I said, “Well, given that I already have some content, why don’t we, just, collaborate on it?” [01:06:51] Andrey: And what was the most surprising response to this essay? It’s probably the thing you’ve written that has been read the most, right? Phil: Yeah. I mean, To me, that was the most surprising thing in itself. It shouldn’t have been, of course, because Dwarkesh has a big audience, and everyone has to have an opinion about inequality, and I don’t know. It should have been totally foreseeable. Anyway, the most surprising thing, I mean, there were things I just learned because, you know, of how, the best way to learn is to write something wrong on the internet, and then someone posts the right thing on the internet. So, I guess I had sort of thought going in that Piketty was wrong empirically about whether capital and labor were gross substitutes in the past, but that if we granted that absurd premise, he would at least be right, you know, about the implications of it for, you know, the dynamics of wealth inequality given one tax policy or another or something. And I think, yeah, he just-- even given that premise, the best tax policy is probably just a consumption tax, unless you care very explicitly about wealth inequality per se, as opposed to, you know, inequality in, consumption or income. Which you might, but I think he wasn’t quite as explicit as would have been ideal about what premises you need to get to his policy conclusion. And not to blame him for our mistake, but like, I mean, the essay’s edited now, but when it-- when we first wrote it just kind of deferred to Piketty I think a bit too much about what follows given the premise. Seth: Phil, you gave us so much good stuff. Phil: Yeah. Thank you for your time. Seth: Thank you. Outro [01:08:27 - 01:08:52] [01:08:27] Andrey: Well, thanks for joining us for another episode of Justified Posteriors. As a reminder, we have written down all of our priors and posteriors on our website, justifiedposteriors.com, so please check it out. And if you liked the episode, please share it with your friends. Seth: And keep your posteriors justified. Get full access to Justified Posteriors at empiricrafting.substack.com/subscribe
