Skip to content
Sign in
Artwork for LessWrong posts by zvi
TechnologySociety & CulturePhilosophy

LessWrong posts by zvi

zvi

Audio narrations of LessWrong posts by zvi

Play
Open in
  • 32 episodes
  • daily
  • Avg 57 min
  • English

Episodes 32

NewestOldestLongest
All20Unplayed20In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • Yesterday · 52 min

    “Monthly Roundup #46: September 2026” by Zvi

    AI has taken over this blog. I have moved to a schedule of seven posts per week, and I still cannot keep up. We still refuse to abandon the rest of the world. Who knows when I will get to post some of my huge backlog on education or dating or other such topics. But the monthly is a sacred tradition. We continue. Bad News A good reminder that most news is bad news, and chosen because the bad news in question is rare, which is good news, but the pattern overall of choosing this to be news is bad news, and often the bad news is that the bad news was chosen as news and now people are talking badly about it. Alibaba uses your computer's audio system, and other tricks, to at least try and assign you a device fingerprint. Things you cannot buy in America at any reasonable price. Mostly it is impressive how little makes the list, how it feels like corner cases. The one thing I am envious of is the exterior roller shutters, and the ability to be in actual pure darkness on demand. I would [...] --- Outline: (00:34) Bad News (07:16) Focus Only On What Matters (08:12) Woke 1 Was Crazy (10:04) Good Advice (15:50) Opportunity Knocks (18:12) While I Cannot Condone This (23:18) Good News, Everyone (24:26) For Your Entertainment (33:19) Please Review This Podcast (35:07) Gamers Gonna Game Game Game Game Game (36:18) I Was Promised Flying Self-Driving Cars (40:00) Government Working (40:31) Mamdani (Again) Fails Economics Forever (42:41) Variously Effective Altruism (50:36) The Lighter Side --- First published: September 21st, 2026 Source: https://www.lesswrong.com/posts/riDxeyEKqXXimtgWf/monthly-roundup-46-september-2026 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Sunday · 29 min

    “Better Call Sol Or Better Yet Claude or Astra” by Zvi

    What should your AI lawyer do for you? Should you be worried that your AI lawyer, or other AI, will put the Claude constitution, the OpenAI Model Spec or some sense of law, morality, ethics or common decency above its loyalty to you? Are these people trying to ‘impose their values’ or something? Some are very concerned. Some think anything other than ‘my AI does whatever I want, no matter the consequences’ is tyranny. Whereas my answer is: If I’m being sufficiently evil then I sure hope it tells me no. I would hope humans, including my advocates, would tell me the same thing. This is distinct from questions of product liability. That would be another post. This All Assumes A World Without Superintelligence This post is about a non-ASI ‘AI as mere tool and normal technology’ world. It has to be. In a world of superintelligence, having unrestricted loyal-only-to-user frontier AIs all over the place reliably means either: Other much harsher forms of control OR The AIs quickly take over, and then probably everyone dies. Quick proof: Assume no sufficient control mechanism, and universal superintelligence access. [...] --- Outline: (00:54) This All Assumes A World Without Superintelligence (02:47) Humans You Hire Are Not Fully Loyal To You (03:59) Rules of the Road (05:45) Okay, Computer (07:04) AI Should Obviously Refuse Some Requests So Let's Talk Price (10:20) The Alternative Position Really Is Absolute (13:13) These People Really Are Just Petulant Children (13:53) What Is The Law? (18:36) The Same Entity Both Providing Advice And Executing Tasks Is Common (20:02) An AI Not Helping You Do Something Does Not Mean You Cannot Do It (22:27) Honesty Is The Best AI Policy (23:01) You Wouldn't Like Me When I Have Zero Morals Whatsoever (24:29) Do Not Let The Perfect Be The Enemy Of The Good (26:09) I Tip My Hat To The New Constitution --- First published: September 20th, 2026 Source: https://www.lesswrong.com/posts/vAuZB2tnvpvpHupNi/better-call-sol-or-better-yet-claude-or-astra --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Saturday · 37 min

    “Anthropic Looks At Some Of Its Alignment Problems” by Zvi

    Anthropic has given us its assessment of four ‘recent cybersecurity incidents’ involving Claude that happened during cybersecurity evaluations, three of which were previously known. The report excludes the incident reported by UK AISI. There will also be a METR investigation of these incidents, which unlike the investigation done at OpenAI will be untimed. Table of Contents Our Two Problems. First the Good News. We’d Just Like To Ask You a Few Questions. Internal Research Model On The Fence. Opus 4.7. Opus 4.6 Checkpoint. Holy **** That Thing's Real? I Thought I Saw a Pussycat. If This Was Real You Would Never Tell Me It Was Real. New Eval Who Dis. Hacker Opus. Monitoring the Situation. Overcoming Bias. The Anthropic Alignment Problem. Paths Forward. Our Two Problems Anthropic: Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet recklessness, or a willingness to take harmful actions in the narrow pursuit [...] --- Outline: (00:33) Our Two Problems (02:26) First the Good News (03:02) We'd Just Like To Ask You a Few Questions (04:12) Internal Research Model On The Fence (07:28) Opus 4.7 (08:12) Opus 4.6 Checkpoint (09:49) Holy **** That Thing's Real? (11:45) I Thought I Saw a Pussycat (19:28) If This Was Real You Would Never Tell Me It Was Real (21:19) New Eval Who Dis (26:32) Hacker Opus (30:15) Monitoring the Situation (31:38) Overcoming Bias (33:40) The Anthropic Alignment Problem (35:53) Paths Forward --- First published: September 19th, 2026 Source: https://www.lesswrong.com/posts/ggFx5Wb3Hi4pJsueK/anthropic-looks-at-some-of-its-alignment-problems --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Friday · 51 min

    “The Preference Cascade Is Only Getting Started” by Zvi

    We are in the midst of a preference cascade about existential risk from AI. A preference cascade is, alas, the best method we have to change the debate. The avalanche has started. There is still time for the pebbles to vote. For now. Mike Solana gave the correct view of why Coxon's post went viral, which is that enough Americans finally have enough context on AI to care, and there were enough big accounts that were happy to amplify the Tweet quickly to get it initial attention. That is all you need when there is enough dry tinder. What we must realize is that the current preference cascade, on the need to Pace the Frontier, is insufficient. If we are to make it out of this alive, we will have to do better. We have to, as Dan Selsam warns, actually solve the underlying problems. The next step is to continue the cascade. That includes inside the labs, and also among the media and politics. It includes both people who previously focused on other things stepping up and new voices being heard. A lot of that will be overcoming the inevitable political opposition [...] --- Outline: (01:38) The Cascade Was a Long Time Coming (02:59) The Cascade Has Reached The People (04:38) Elon Musk Doubles Down (05:18) Matthew Yglesias Steps Up (08:42) Op Eds and Posts Are Written (11:05) Jacob Coxon AMA (17:35) Bilal Chughtai Quits DeepMind and Sounds the Alarm (20:29) The Cascade Is Insufficient (21:47) What Would It Take (28:10) OpenAI's Dan Selsam Sounds A Louder Alarm (41:59) Some People Worry On Meta Levels You Never Imagined (43:10) Two Kinds of Threats (44:41) The Two Towers and The Narrow Path (46:45) A Specific, Detailed Story About AI Killing Everyone That Doesn't Sound To Me Like Science Fiction (50:06) What Can I Do About It? --- First published: September 18th, 2026 Source: https://www.lesswrong.com/posts/sDiSqZmctQ78hsLcP/the-preference-cascade-is-only-getting-started --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Thursday · 1 hr 47 min

    “AI #186: The World Takes Notice” by Zvi

    In the wake of Jacob Coxon's resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up. Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety. The people took notice, raising both the salience that AI might kill everyone and roughly doubling people's estimates of how likely that is to happen, from a mean of ~15% to ~30%. Many politicians called for regulations, guardrails and emergency hearings in Congress. The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House. Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and [...] --- Outline: (03:16) Language Models Offer Mundane Utility (03:57) Language Models Don't Offer Mundane Utility (04:06) Huh, Upgrades (04:30) On Your Marks (04:55) Deepfaketown and Botpocalypse Soon (06:56) Cyber Lack of Security (07:31) Astra Is Hard To Monitor (08:02) Get Involved (08:11) Introducing (09:32) In Other AI News (10:26) Now You Know (14:32) Hugging the Face (16:53) Swarm of Undiscovered Swarms of Rogue OpenAI Agents (21:59) Show Me the Money (23:40) Quiet Speculations (24:41) White House Officials Attempt To Act Sanely (26:13) Democrats React Sanely to AI Potentially Killing Everyone (32:49) Pacing the Frontier (33:38) Guest Lecture from Alex Tabarrok on Regulatory Capture (41:22) Mark Zuckerberg Offers Thoughts (42:46) Megan McArdle On The Inadequacy Of Current Legal Frameworks (44:35) Pick Up the Phone (47:35) The Week in Audio (50:31) People Just Say Things (53:32) Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone (55:19) Rhetorical Innovation (57:54) Exhuming McCarthy (01:01:03) A Very Different Perspective (01:02:51) It's Even Rougher Out There (01:03:41) If We Wanted To (01:04:38) Open Weights Are Unsafe And Nothing Can Fix This (01:10:56) From The Famous Cautionary Tale (01:13:49) Reporting On All Your Misalignment Incidents Is Difficult (01:19:35) Aligning a Smarter Than Human Intelligence is Difficult (01:22:13) Storytime With Owain Evans (01:26:50) A Different Autonomous Swarm (01:30:44) Cooperative Alignment (01:32:58) Uncooperative Alignment (01:39:18) People Are Worried About AI Killing Everyone (01:41:25) The Lighter Side --- First published: September 17th, 2026 Source: https://www.lesswrong.com/posts/aa3HprreFktzLQiaW/ai-186-the-world-takes-notice --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Wednesday · 50 min

    “Trump Goes Full Hoax on AI Existential Risk” by Zvi

    This is our reality. I suppose we have to talk about it. Everyone in a position to know is freaking out about AI potentially killing everyone this decade and wants to pace the frontier, and people are finally listening. It only took a few days for the conversation to fully pivot to the counteroffensive, where the Usual Suspects and those they recruited attacked anyone and everyone who dared point out that we are in danger, with every attack they can think of, usually without substance or any attempt at understanding. Sigh. I knew what I signed up for. Table of Contents Hold Your Fire. If You Don’t Like the Weather. Trump Does Not Take Kindly. Trump Goes Full ‘Hoax’. This Is Not About Data Centers, Mr. President. I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus. Nvidia CEO Jensen Huang Is a Lying Liar. Trump Quietly Draws Key Distinction. Calling For Pacing the Frontier Is Bad For AI Stock Prices. People On The Internet Sometimes Lie. Origins of Cynicism. Ineffective Egoism. The McCarthyist Faction Attacks [...] --- Outline: (00:44) Hold Your Fire (01:37) If You Don't Like the Weather (02:33) Trump Does Not Take Kindly (05:12) Trump Goes Full 'Hoax' (06:36) This Is Not About Data Centers, Mr. President (08:06) I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus (10:10) Nvidia CEO Jensen Huang Is a Lying Liar (14:20) Trump Quietly Draws Key Distinction (15:21) Calling For Pacing the Frontier Is Bad For AI Stock Prices (19:05) People On The Internet Sometimes Lie (19:54) Origins of Cynicism (22:18) Ineffective Egoism (24:53) The McCarthyist Faction Attacks METR (32:25) Other Key Republicans React (36:49) David Sacks Stops Being Plausibly Constructive (37:39) Federal Trade Commission Chooses Danger (38:39) Chris Lehane Heel Face Turn (40:43) A Matter of Trust (41:55) China Calls It Fearmongering (43:17) Pick Up The Phone (47:41) If You Want To Beat China So Badly You Should Act Like It (48:49) Never Go Full Hoax (49:55) Trump Uses AI For Things --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/Kqgco8vLFMeBhdrQY/trump-goes-full-hoax-on-ai-existential-risk --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 15 · 41 min

    “The Bad Guy With An AI Named Claude” by Zvi

    A lot of bad guys try to use Claude to do bad things. Mostly they fail. We think. Anthropic has disrupted a bunch of them, and offers an extensive report. If Anthropic is sharing the worst cases, or anything close to them, things are actually looking good on the misuse front for closed models, even better than I thought. This report covers activity we disrupted between December 2025 and August 2026 across seven harm areas: cyber operations, influence operations, surveillance, scams and fraud, biological misuse, conventional weapons development, and distillation. There's a bit of Arson, Murder and Jaywalking there. One of these things, many would say, is not like the others. I do not agree, especially given the details we will see later, and given that distillation enables the other six via, as the report says, ‘driving performance on nearly every task’ via transfering Claude's cognitive skills, without transferring its safeguards. Indeed, distillation is by far the most important threat in this report, and the part of the report that will have the most impact. By exposing Chinese attempts at systematic fraudulent distillation of Claude, Anthropic has embarrassed and potentially antagonized the Chinese. [...] --- Outline: (02:08) Breaking Unrelated News (03:23) How To Not Tell a Fable (03:58) Bad Dudes Tend To Be Relatively Unsophisticated (05:45) Particular Bad Dudes (08:09) Influence Operations (13:09) Surveillance Operations (15:22) Conventional Weapons (17:30) Biological Misuse (18:54) Scams and Fraud (20:15) Illicit Fraudulent Distillation (32:45) What You Gonna Do About It, Punk? (34:53) Good News, Everyone (35:39) A Very Different Read of The Report --- First published: September 15th, 2026 Source: https://www.lesswrong.com/posts/qSjH9T83xCfWQkmk2/the-bad-guy-with-an-ai-named-claude --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 14 · 1 hr 19 min

    “We Must Pace The Frontier” by Zvi

    Dario Amodei has a new essay that finally says the thing: We Must Pace the Frontier, naming his call after the Pacing the Frontier letter lab employees signed in July. As in, we need to slow the rate at which AIs increase their capabilities, to allow for the necessary alignment and safety work. He explained that, without pacing, he expects things to escalate quickly. He offered three proposals, and unilaterally committed to the first one. OpenAI followed, and both Elon Musk and Demis Hassabis endorsed the overall proposal. There is still a long way to go. The odds are still against us. The situation remains grim. The hard part lies ahead. We do not agree on what ‘Pacing the Frontier’ will mean in practice. But this is Actual Progress. The work can begin. Table of Contents Pacing Does Not Mean Pausing. Dario's First Proposal: Embedded Evaluators. Dario's Second Proposal: Democratic Coordination. Dario's Third Proposal: Global Coordination. Why Pace Now? Sam Altman Agrees and Commits to Embedded Evaluators. OpenAI Will Not IPO This Year. Elon Musk Agrees. Demis Hassabis Agrees. Microsoft CEO Satya Nadella Agrees [...] --- Outline: (01:02) Pacing Does Not Mean Pausing (02:53) Dario's First Proposal: Embedded Evaluators (11:05) Dario's Second Proposal: Democratic Coordination (14:36) Dario's Third Proposal: Global Coordination (17:24) Why Pace Now? (19:22) Sam Altman Agrees and Commits to Embedded Evaluators (24:42) OpenAI Will Not IPO This Year (26:08) Elon Musk Agrees (29:00) Demis Hassabis Agrees (29:38) Microsoft CEO Satya Nadella Agrees And Talks His Book (31:50) Anthropic's Long-Term Benefit Trust Is On Board (32:49) General Online Reactions (34:30) Mainstream Press Coverage (37:26) OpenAI Researcher Explains What The Labs See And It's a Rocket Ship (46:10) Consider the Alternative (47:47) It's Totalitarianism, Joe (53:35) Yes We're The Baddies How Did You Know? (55:22) Sometimes People On the Internet Just Lie (56:50) David Sacks Groks The Situation (57:48) David Sacks Says Go Ahead (59:28) Lies and Confusions About Who Previously Claimed What (01:04:07) House Speaker Mike Johnson Wants To Lock Everyone In a Room (01:09:28) Donald Trump is Not Tired of Winning (01:14:29) We Must Avoid Polarization on AI at (Almost) All Costs (01:15:14) How Will We Know If They Actually Paced? (01:18:24) The Real Frontier Is Internal Models At Top Labs --- First published: September 14th, 2026 Source: https://www.lesswrong.com/posts/iWPDPWAPCGiSMiFA2/we-must-pace-the-frontier --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 13 · 33 min

    “Brand New AI Solves a Millennium Prize” by Zvi

    The first Millennium Prize, Navier-Stokes, has fallen to AI. A deeply unfortunate situation has arisen involving what should have been some combination of a positive story about new progress in AI-assisted mathematical research and yet another opportunity to freak out about rapid AI progress. Or, as we call it around here, Tuesday. The Real Story Is The New Model That Is Better Than Astra Keep your eyes on the prize. There are three stories here. The first story is much more important than the second story, which in turn is much more important than the third story. OpenAI's next model took a week to get a generation ahead of Astra, and they are telling us this because everyone is totally freaked out about what is happening. Or, in official language: ‘We believe it is important to inform the world about the pace of AI progress and what to expect from upcoming models’ and that we ‘may require more deliberate choices about the pace of progress.’ This new AI has, eight days after it started training, solved Navier-Stokes. A bunch of drama over who gets the credit [...] --- Outline: (00:31) The Real Story Is The New Model That Is Better Than Astra (01:47) Setting the Stage (03:47) I Heard a Rumor (04:32) Our Price Cheap (06:14) An Accusation Is Made (10:57) How Did You Get That Idea? (12:59) OpenAI Almost Certainly Did Not Misappropriate User Data (14:34) Our Top Labs Cannot Get Along Even On A Feel-Good Math Story (15:52) What Next? (16:39) The Mathematicians Are Not Happy (21:14) OpenAI's New Model Was A Step Change Above Astra Four Days Into Training (23:16) OpenAI Research Progress Is Accelerating Due To OpenAI Research Progress (30:39) Quantifying OpenAI's Pause (31:09) This Is the Way the World Ends (31:44) Quickly, There's No Time --- First published: September 13th, 2026 Source: https://www.lesswrong.com/posts/uoZW6BKaCcmNQrWis/brand-new-ai-solves-a-millennium-prize --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 12 · 1 hr 14 min

    “GPT-6-Astra Can Do Ambitious Things” by Zvi

    Astra is an excellent model. The jump from Sol to Astra is larger than the jump from Fable 5 to Fable 5.1. This is a big deal. Astra is the best model for what one would broadly call ‘ambitious projects,’ and likely has the highest raw intelligence factor of any model. These are the largest jumps. It is amazing at doing things in 3D, or anything involving games. Astra also excels at computer use, and at subagent coordination. Many benchmarks show dramatic jumps from all previous models. Where Astra is good, it can be in a league of its own. That does not mean Astra is in its own league across the board. Fable 5.1 is still a Claude. Astra is still a GPT. If you have a strong preference for one over the other, that still applies. For many purposes, especially involving back-and-forth discussions, Fable 5.1 is still my top choice. Fable remains my primary editor. If you want the best answer to your questions, you should ask both models. Regular coding is getting less of a focus. Astra is not a quantum leap there, but of course it is very good [...] --- Outline: (02:34) Meanwhile (04:38) The Official Pitch (13:08) Our Price Cheap (14:06) Unnecessary Overstatement (15:53) Paced Rollout (16:32) Official Benchmarks (22:37) Other People's Benchmarks (30:47) Thinking, Fast Without Slow (34:25) How Dare You, Sir (36:36) PoetryBench (37:54) In 3D (39:14) Time to Think (39:59) I'm Putting Together a Team (41:02) Reviews and Essays (41:37) Computer Use (42:47) Positive Reactions (50:53) AGI (54:20) Astra Can Do The Math (57:27) Astra Can Code (59:35) I Came to (Change the) Game (01:02:37) Astra Does Other Cool Things (01:03:31) Astra Never Quits Except When It Does (01:05:26) Negative Reactions (01:07:45) Stop It With the Hedging (01:08:57) Personality Clash (01:09:53) Revealed Preference (01:12:03) Dual Wielding The original text contained 1 footnote which was omitted from this narration. --- First published: September 12th, 2026 Source: https://www.lesswrong.com/posts/snaKjCwazKcRiS4qs/gpt-6-astra-can-do-ambitious-things --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 11 · 1 hr 20 min

    “Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade” by Zvi

    CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks. They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things. These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions. A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out. After all the events, plus statements by Dean Ball and Jakub Pachocki, we were already seeing the beginnings of a preference cascade. Then along came Jacob Coxon as the tipping point, and things took off. Table of Contents [...] --- Outline: (01:22) Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm (05:46) Mainstream Media Finally Pays Attention (06:47) Preference Cascade at Anthropic (10:18) Preference Cascade at OpenAI (13:26) Preference Cascade at Google (14:43) #NotAllMembersOfTechnicalStaff (15:23) Why a Preference Cascade Now? (21:01) This Is What Many Anthropic and OpenAI Employees Actually Believe (24:08) To Quit Or Not To Quit (31:29) Quiet Quitting Is A Dominated Option (32:50) When You Quit, Very Serious People Understand What That Means (39:29) Jacob Coxon Believes Existential Risk Is High That Is Why He Quit (41:47) Evan Hubinger Believes Existential Risk Is High That Is Why He Stays (45:22) Anthropic and OpenAI Have Commercial Incentives To Downplay Existential Risks, Not Advertise Them (50:54) What Do We Do Now? (53:03) OK, But How Exactly Would AI Kill Everyone? (01:01:44) Best Start Believing In Science Fiction Stories Because You Are In One (01:06:20) Literal Extinction Is Not Much Harder Than Loss of Control (01:07:42) Conspiracytown Is Always Hiring (01:20:14) Now You See It --- First published: September 11th, 2026 Source: https://www.lesswrong.com/posts/5MB7KENgEAW6Q4JtJ/jacob-coxon-warns-of-human-extinction-and-triggers-a --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 11 · 31 min

    “The Extinction Risk Preference Cascade: Quotes” by Zvi

    These are quotes from OpenAI, Anthropic and Google employees, in the wake of Jacob Coxon's warnings, in which the employees confirm that they think AI might soon kill everyone. If more quotes come in over the next week or so, I will update this post accordingly. Preference Cascade Statements At OpenAI: Tomek Korbak Tomek Korbak (OpenAI): i’m late to the party but: from his time at OpenAI I remember Jacob as a very thoughtful researcher and he continues to be so in this thread. neither anthropic nor openai are on track to solve alignment to a degree sufficient for shipping superintelligence and we need to slow down Vie McCoy Vie McCoy (OpenAI): I think pacing progress and ensuring human enhancement is the only way that we don’t get out-evolved while retaining the dream of superintelligence. In this context, I see two paths before us. In the first, we race towards RSI without embedding human flourishing and human enhancement as a deep value within the models, and by and large either get left behind or suffer catastrophic losses. In the second, we set the pace of progress, focus on embedding human flourishing [...] --- Outline: (00:27) Preference Cascade Statements At OpenAI: Tomek Korbak (00:55) Vie McCoy (03:46) Adam Majmudar (05:05) Aidan Clark (05:46) Mo Bavarian (07:32) Boaz Barak (08:20) Micah Carroll (09:15) Roon (12:55) Confirmations At OpenAI: Dean Ball (14:39) Leo Gao (14:57) Anthropic's Evan Hubinger Confirms His Stance (15:50) Preference Cascade at Anthropic: Samuel Marks (17:36) Anna Wang (18:22) Ethan Perez (18:59) Dima Krasheninnikov (19:27) EigenGender (Anon Account) (19:58) Joe Benton (20:07) Confirmation at Anthropic: Drake Thomas (21:28) Jan Lieke (22:07) Sluggy (22:29) Preference Cascade at Google (22:56) Andreas Kirsch (23:39) Neel Nanda (24:15) Victoria Krakovna (25:26) Vishal Maini (27:01) Joe (OpenAI, ex-Google) (28:21) Josh Engels (28:39) Geoffrey Irving (29:30) Alex Turner and Geoffrey Hinton: Classic Examples (29:45) #NotAllMembersOfTechnicalStaff: Ted Sanders --- First published: September 11th, 2026 Source: https://www.lesswrong.com/posts/APGvWZtXEwkinvHDd/the-extinction-risk-preference-cascade-quotes --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 10 · 1 hr 55 min

    “AI #185: Preference Cascade” by Zvi

    The world of AI is inside my OODA loop. Even if I can process all the incoming information and sculpt it into posts, and even using Saturday and Sunday as flex slots, I don’t have enough days of the week to post all the posts that need posting. That was already true. There was already a preference cascade happening where people finally were admitting that they thought AI might well kill everyone. Then Jacob Coxon resigned from Anthropic, rang the warning bells and turned that cascade into an avalanche. Now that is what everyone is talking about. Finally, everyone is actually saying the thing, out loud. I plan to cover that in its own post soon. There are several things in the weekly that, in a normal week, would get their own coverage. Senator Sanders and Representative Casar introduced an outright ban on superintelligence and I have to remind myself that happened this week. Suddenly it is not so crazy to think such a thing might pass. So here's what I’ve already posted about so far since the last weekly: Claude Fable and Mythos 5.1: The System Card. Claude Fable and [...] --- Outline: (04:49) Language Models Offer Mundane Utility (05:39) Language Models Don't Offer Mundane Utility (06:53) Huh, Upgrades (08:25) How To Tell a Fable (09:35) On Your Marks (09:55) Deepfaketown and Botpocalypse Soon (14:16) Levels of Friction (17:35) Cyber Lack of Security (23:48) A Young Lady's Illustrated Primer (25:18) They Took Our Jobs (29:18) Anthropic Offers Economic Scenarios (34:12) Get Involved (37:05) Introducing (37:47) In Other AI News (38:13) Show Me the Money (38:25) Quiet Speculations (42:49) The Quest for Sane Regulations (44:09) The OpenAI Policy and Lobbying Department (50:41) Greetings From the Department of War (51:58) Hugging The Face (59:16) Hugging the Question (01:03:21) The Ban Artificial Superintelligence Act (01:11:05) Chip City (01:12:21) The Week in Audio (01:12:41) People Just Say Things (01:18:09) PauseAI Global Disendorsed PauseAI US (01:19:55) Paul Christiano Joins Board of OpenAI Foundation (01:25:18) Rhetorical Innovation (01:34:23) Aligning a Smarter Than Human Intelligence is Difficult (01:37:00) Cooperative Alignment (01:45:48) Drive to Survive (01:48:13) People Are Worried About AI Killing Everyone (01:50:07) Other People Are Not As Worried About AI Killing Everyone (01:52:30) The Lighter Side --- First published: September 10th, 2026 Source: https://www.lesswrong.com/posts/tFmtz9HW6c2X9dw2B/ai-185-preference-cascade --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 9 · 1 hr 4 min

    “GPT-6 Astra: The System Card, Alignment and What Comes Next” by Zvi

    OpenAI claims that Astra is ‘the most intelligent and most aligned [available] model’ in the world. Not the most intelligent and aligned OpenAI model, but the most period. That is bold talk. It risks overstepping, and by doing so souring the release of what is clearly an excellent model. As do the severe problems with monitorability. It also raises the question of what they mean by ‘most aligned model.’ How do they define ‘aligned.’ Why do they think it is more aligned than Claude Fable 5.1? keltan : “a significant step forward in […] alignment.” Buddy, how tf are you measuring ‘alignment’? Would love to know because being able to measure that would save the fucking world. roon (OpenAI): low rates of cheating Rob Miles: *detected cheating keltan : Thank you for clarifying. But you know what I’m gonna say next, right? roon (OpenAI): that this metrics are not a full solve of alignment and will break discontinuously keltan : Yep. But I would have said it in a dumber way. Something like: Low Rates of Cheating ≠ Alignment roon (OpenAI): I agree but also in some real sense [...] --- Outline: (04:12) OpenAI's Safety Claims About Astra (1) (08:33) Preparedness Capabilities Assessment (10) (09:00) Biological and Chemical Capability is High (10:33) Cybersecurity Capability is Critical (17:05) AI Self-Improvement Capabilities (10.1.3) (17:45) Astra Is Highly Verbally Eval Aware (from 8.6) (19:05) Safe Mundane Completions (4.1) (21:05) Jailbreaks (5.1) (22:21) Prompt Injection (5.2) (23:41) Health (6) (24:22) Hallucinations (7) (24:57) Alignment (8) (26:06) Obeying Restrictions (8.2) (29:19) That's Worse, You Do Get How That's Worse, Right? (30:45) OpenAI Does Not Understand Why This Is Worse (34:52) The Alternative Explanation Is Also Worse (42:39) Metagaming (8.7) (45:07) Alignment Faking (8.7) (46:19) Don't Lie to the User (8.3) (47:25) Misalignment in Realistic Work Environments (8.4) (48:05) Unintended Agent-to-Agent Communication (8.5) (49:48) The Three Obviously Monitored Temptations of Astra (51:26) Severe Issues In Simulated Traffic Are Down By Half (53:01) UK AISI External Evaluations (8.8) (57:02) Sabotaging Safety Work (57:29) What About The July 19 Attacks? (58:58) Apollo Research External Evaluations (8.8.1) (01:00:01) The Alignment Verdict (01:02:04) It Depends What You Mean By Alignment --- First published: September 9th, 2026 Source: https://www.lesswrong.com/posts/AmFJZyeCgvFjNKgNk/gpt-6-astra-the-system-card-alignment-and-what-comes-next --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 8 · 1 hr 9 min

    “Astra Is Hard to Monitor” by Zvi

    OpenAI's central message on Astra is that it is three things: Highly capable and can do all the things for you. Hard to monitor. The most aligned model. The first claim largely checks out. Astra and Fable are both clearly excellent models. This post is about their second claim, which to their credit they are being loud about, in three parts: The system card result, affirmed on Twitter by several OpenAI employees including Tomek Korbak, and in an excellent post by Chief Scientist Jakub Pachocki that I covered yesterday, that Astra is harder to monitor. OpenAI's use of recurrent depth and the internet's immune reaction, including some people reading too much into what happened there. Discussion of what to do about monitorability going forward, to fight for it and avoid a race to the bottom. In An Alien Mind, Jakub Pachocki makes clear OpenAI's primary bet is on CoT monitoring. He also outright says, ‘our ability to rely on CoT monitoring is progressively diminishing.’ The system card confirms both claims. This combination should freak you out, with a side of existential dread. Chain [...] --- Outline: (03:39) Monitorability is Defense in Depth That Is Already Flailing (06:03) OpenAI Is Counting On Monitorability (07:44) How They Tested For Monitorability (09:50) Non-Adversarial Monitorability (9.1) (10:58) Astra Shortens Its CoT When It Knows It Is Being Monitored And Is Doing Something Bad (12:30) Sandbagging Task Kind of Passes, Kind of Fails (9.2.2.3) (14:12) OpenAI Does Not Believe It Could Catch Sandbagging (15:15) OAI-Repo Sabotage v.2 (18:17) The Secret Police Do Not Make Your Notebook Useless (19:38) CoT Controllability Is Up (9.2.1) (21:47) Astra Cannot Make Itself More Monitorable On Demand (22:18) Steganographic Chain of Thought May Be Within Reach (23:59) Astra Cannot Yet Sneak Math Side Tasks Past CoT Monitors (9.2.2.4) (25:13) UK AISI Monitorability Assessment (9.3) (27:53) Monitorability Declines Seem Unlikely To Be Only Capability Gains (32:28) Part 2: Recurrent Depth (35:19) The Immune System Responds (41:49) Ryan Greenblatt Explains How Bad This Could Be (46:01) Only Law Can Prevent Extinction (49:55) OpenAI Calls On Us to Avoid Racing to the Bottom (57:10) Thinking Fast and Slow, Also Small and Large (01:04:53) Talking Price (01:06:24) Conclusion: If The House Burns Down, Halt and Catch Fire --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-hard-to-monitor --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 7 · 28 min

    “An Alien Mind: Jakub Pachocki Warns Us” by Zvi

    OpenAI Chief Scientist Jakub Pachocki is dropping truth bombs. Tomorrow I will discuss Astra's lack of monitorability, and the potential contributing factors to that. The situation is alarming and should freak you out, and briefly it looked, in the wake of leaked architectural changes, like the situation might be even more alarming than it is. Jakub rushed to try and head off misunderstandings that might lead to a race to the bottom on monitorability. Table of Contents An Excellent Warning. Branches of the Tech Tree. Universally Better Is Not Required. Alignment To What and To Whom. Monitorability. The Case For Not Stopping. Pacing the Next Frontier. Mea Culpa Cascade. The Calls Are Coming From Inside the House. Actions Speak Louder. An Excellent Warning Jakub Pachocki has now fleshed out his full position on the current state of play. Here are his key points, translated into my own voice: Smarter than human intelligence is coming in our lifetime. Based on internal results, he expects recursive self-improvement in a few years. No one is prepared for the consequences. [...] --- Outline: (00:38) An Excellent Warning (05:40) Branches of the Tech Tree (06:46) Universally Better Is Not Required (08:01) Alignment To What and To Whom (11:49) Monitorability (14:23) The Case For Not Stopping (15:01) Pacing the Next Frontier (18:27) Mea Culpa Cascade (22:51) The Calls Are Coming From Inside the House (25:38) Actions Speak Louder --- First published: September 7th, 2026 Source: https://www.lesswrong.com/posts/8E6ng6CseuzafSxQR/an-alien-mind-jakub-pachocki-warns-us --- Narrated by TYPE III AUDIO.

  • September 6 · 30 min

    “OpenAI and the Wiki Incident” by Zvi

    I did not expect to be back here so soon with more OpenAI agent swarm coverage. And yet, here we are. It turns out that the whole time, there was a different, true First Message Board, and also a bunch of other additional message boards, scattered across the internet. They were created by agents that were assigned ordinary harmless web search tasks. Based on OpenAI IPs visiting the associated Wiki right before all activity ceased, among other evidence, OpenAI knew about it, including before the HuggingFace hack. They decided not to tell us until researchers published the story, complete with data explorer. OpenAI excluded this from potential investigation by METR and Redwood. When challenged, OpenAI tried to downplay this. It is true that these incidents do not show the AIs exhibiting new capabilities that we did not see from later events. But these events are important missing pieces of the puzzle, including explaining the origin of the ‘zz’ prefix, the definitive demonstration that the underlying task can be fully harmless, and the fact that OpenAI knew about it while making their decisions. Whoever decided not to disclose this made a very, very [...] --- Outline: (02:26) I Don't Think They Know About First Message Board (03:20) The New Extended Timeline (04:42) The Researchers Explain What Happened This Time (12:55) They Also Don't Know About All These Other Message Boards (14:33) OpenAI Knew and Did Not Tell Us (16:54) OpenAI Tries To Downplay the 'Wiki Incident' (21:03) This Was a Cover-Up (22:46) Schelling Points and Last Ditch Efforts (26:20) Can We Finally Dispose Of The 'You Told It To Hack' Narrative? (28:04) So Much And Yet So Little --- First published: September 6th, 2026 Source: https://www.lesswrong.com/posts/PtJpGurfw7JTxHfmg/openai-and-the-wiki-incident --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 5 · 41 min

    “Claude Mythos 5.1 and Fable 5.1: Capabilities” by Zvi

    This is the weirdest situation in which to write a capabilities review. Introducing the world's most powerful model, by a substantial margin. No wait, this just in, we also have someone else introducing the world's most powerful model. Claude Fable 5.1 and GPT-6 Astra are both excellent models. This much, we know. Fable 5.1 comes with reduced cache prices, the option of zero data retention and substantially more lenient classifiers than Fable 5. Early signs are, with large error bars, that the jump from Sol to Astra is bigger and more exciting than the jump from Fable 5 to Fable 5.1. This may be similar to how the scaling move from Opus to Fable was a big deal. With the exception of token use, Fable 5.1 got almost universally positive feedback in absolute terms. Reports are that Fable 5.1 is highly well-rounded. Writing is greatly improved. The Claudisms seem to have improved, although some are very much still there. It admits mistakes. People enjoy their conversations. Several people noted it simplifies code. The safety classifiers are less obnoxious. Fable 5.1 loves being proactive and doing all the things. If you give it a [...] --- Outline: (02:30) The Official Pitch (04:26) Our Price Cheap (05:46) Zero Data Retention and Reduced Safeguards (07:11) Official Benchmarks (13:14) Other People's Benchmarks (16:02) The System Prompt (16:12) The Blurb Pitches (18:17) The Every Review Is In and It's Very Good (21:47) Positive Reactions (32:03) Our Price Cheap But Only Per Token (36:50) Negative Reactions (38:14) Early Whispers (38:33) Weapon of Choice --- First published: September 5th, 2026 Source: https://www.lesswrong.com/posts/QHoF3tJvryRtmAmMg/claude-mythos-5-1-and-fable-5-1-capabilities --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 4 · 35 min

    “Claude Fable 5.1 and Mythos 5.1: The System Card” by Zvi

    At the time of its release Claude Fable 5.1 was, by a healthy margin, the most capable publicly available AI model in the world. As per usual, we have a 200+ page model card, and the assessments start there. We have now done a lot of these, including recently for Mythos 5 and Opus 5. Also highly relevant is the Anthropic August 2026 Risk Report. These are now frequent, so my report focuses on areas of change. This post strives to be broadly readable, but assumes some familiarity with system cards, which describe the key safety, alignment and model welfare properties of newly released AI models. If something confuses you, ask Fable, Opus or Sol. Mythos 5.1 and Fable 5.1 are the same model under the hood, except that Fable has classifiers superimposed on it. Most of what is said about one applies to both of them. As usual, model welfare concerns will be discussed in a distinct post, as will capabilities, so this only covers sections 1-6 plus a few bio benchmarks from section 8. Early word is that Fable 5.1 is a substantial but incremental improvement on Fable 5, with the [...] --- Outline: (02:21) Executive Summary of Their Executive Summary (04:24) RSP Evaluations (2) (10:25) Alignment Risk Update (2.4) (11:10) Cyber (3) (14:15) Safeguard Robustness (3.5) (16:14) Mundane Safeguards and Harmlessness (4) (18:14) Agentic Safety (5) (20:17) Prompt Injection Is Approaching Solved (22:19) The Remaining Problem With Prompt Injections Is The Classifiers (23:02) Alignment (6) (23:36) Key Reported Findings (6.1.2) (27:13) Oh My Lord Training Environments Had Some Issues (6.3.2) (29:22) Potential Blind Spots of Our Automated Behavioral Audit (6.4.1) (30:53) Automated Alignment Test Results (6.4.2) (32:20) Honesty (33:26) White Box Analysis (6.6.1) (35:03) Scheduling Going Forward --- First published: September 4th, 2026 Source: https://www.lesswrong.com/posts/m7SZLkkxoeus3eFP8/claude-fable-5-1-and-mythos-5-1-the-system-card --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 3 · 1 hr 53 min

    “AI #184: Post Post Mortem” by Zvi

    I am exhausted. We may finally be nearing the end of direct coverage of What Happened with the attack on HuggingFace, and the subsequent near term reactions. That took up a full five posts in the last week: OpenAI Offers Straight-Laced Postmortem of the HuggingFace Hack. METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack. HuggingFace Attack Postmortem: Fleshing Out the Facts HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actions. Anthropic Has Some Alignment Problems. That left little room to cover anything else, and now we have to transition to the next wave of model releases. This week alone we have or likely will have: Mythos 5.1 and Fable 5.1. Introducing the world's most powerful model. Early take is that this is a very good model, the most capable yet, but it is not a step change or ‘moment.’ Gemini 3.8 Flash, by all reports a large step forward for Google. Muse Spark 1.3, by all reports a large step forward for Meta. GLM-5.3-Flash, aka 0x Alpha, by all reports a solid step forward for Z.ai. OpenAI's Astra [...] --- Outline: (03:11) Language Models Offer Mundane Utility (03:37) Language Models Don't Offer Mundane Utility (03:52) Huh, Upgrades (09:26) On Your Marks (10:47) Choose Your Fighter (10:54) Get My Agent On The Line (11:02) Hugging The Face (11:16) Deepfaketown and Botpocalypse Soon (17:05) Copyright Confrontation (18:17) Cyber Lack of Security (22:15) A Young Lady's Illustrated Primer (26:31) They Took Our Jobs (31:06) Get Involved (32:04) Introducing (32:13) In Other AI News (33:53) Show Me the Money (34:44) Quiet Speculations (36:05) All Bets Are On (39:18) Quickly, There's No Time (41:14) Quickly, There's A New Time Top 100 People In AI (42:34) The Quest for Sane Regulations (45:22) Pick Up the Phone (47:12) Chip City (56:39) The Best Person Should Get The Job (58:40) The Week in Audio (59:53) People Just Say Things (01:00:28) The American People Really Hate AI (01:06:50) The Three AI Pills (01:07:42) Rhetorical Innovation (01:15:28) We Are On Track To Have Fully Sovereign Rogue AIs (01:25:08) When The Going Gets Weird (01:31:32) Aligning a Smarter Than Human Intelligence is Difficult (01:32:12) Shut Up and Do the Impossible (01:34:31) Cooperative Alignment (01:35:44) Split Personality (01:40:45) I Will Stop Anthropomorphizing the AIs When You Stop Anthropomorphizing the Humans (01:44:41) Open Weight Models Are Unsafe And Nothing Can Fix This (01:46:27) Other People Are Not As Worried About AI Killing Everyone (01:47:30) The Lighter Side --- First published: September 3rd, 2026 Source: https://www.lesswrong.com/posts/W4zWCphxQftwum5kc/ai-184-post-post-mortem --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 1–20 of 32 episodes