Agentic Conversations (formally mlops.community)

Tim O’Reilly

A large language model knows everything and nothing at the same time. Tim O’Reilly, founder and CEO of O’Reilly Media, joins us ahead of his keynote at AGNTCon + MCPCon North America to talk about what that paradox means for the people building with these models. We start with what Tim is bringing to the stage in San Jose, then move to the craft of steering a model. The right words work almost like spells that open up new regions of latent space. Something as small as telling a model to use a computer can unlock a new tier of capability, where agents go off and do real work. From there, the conversation turns philosophical. Tim makes the case for mechanism design: shaping the rules of a system so the outcomes we want become the natural ones. We connect it to a book, The Score, and the question of which game we are really playing. In Risk, everyone fights over the board. In charades, everyone plays together. We close on Tim’s view that AI should be an ecosystem rather than a product that sucks the air out of the room, so that everyone can build on top of it and flourish. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm Tim O’Reilly: https://www.linkedin.com/in/timo3 Timestamps: [00:00] Cold open [01:02] Grappling with the bitter lesson [02:22] The value of human expertise [04:54] Why AI needs us [05:37] Culture is what remains [06:23] The toil audit story [08:26] AI knows everything and nothing [08:57] Magic words and runaway agents [11:18] Brute force vs human context [12:32] Turning scar tissue into infrastructure [14:12] Google and the first bitter lesson [16:25] Silicon Valley's sad turn [17:43] Artificial volition [19:08] Casting spells in latent space [20:46] The peaceful atom lesson [22:15] Sharing value with creators [24:54] No jobs means no customers [25:43] Mechanism design explained [26:35] Unix and the architecture of participation [28:45] The missing mechanisms of the AI economy [29:36] What game are you playing [30:43] The game of human flourishing [32:04] Big labs vs open source AI [33:47] Wrap up

AI Agents Need a Babysitter

AI can write your code, choose its own tools, and book your next trip. But who checks that it actually did the job? In this episode of Debug Mode, we explore what it takes to give AI agents more responsibility—and how to keep track of what they’re doing. We discuss intent-based tool selection, MCP and WebMCP, and why a convincing demo doesn’t always translate into reliable automation. We also get into the personal AI assistant everyone seems to be building: one that manages travel, handles household tasks, and hopefully remembers to order coffee before you run out. Then it’s back to code: QA agents, automated verification, tests that capture business intent, and the growing challenge of reviewing AI-generated contributions. Timestamps:[00:00] Who gave AI the keys?[00:43] A tattoo years in the making[02:40] Tell the agent what you want—not which tool[06:19] MCP enters the agent harness[07:23] Great demo. What do the benchmarks say?[09:55] AI builders rediscover machine learning[11:50] Can WebMCP make agents more reliable?[16:04] Unlocking tools like a video game skill tree[18:19] Let an agent try to break your website[19:50] Are humans still in the driver’s seat?[20:38] Build for the outcome, not the API[24:32] Why can’t AI keep my coffee stocked?[27:38] A house manager for the rest of us[29:23] A travel agent that actually handles things[30:49] When your AI chat becomes your workspace[34:12] Does superintelligence need a rebrand?[35:06] Who checks the code AI writes?[36:31] Test why the code exists—not just if it works[41:08] Give your coding agent a QA partner[43:18] The checks failed. Why submit it anyway?[44:33] Open source meets the AI code flood

Token Maxxing Is Over and Here Is What Comes Next

What happens when you give 200 engineers who have barely touched AI a coding assistant and a six-month deadline to rebuild their platform? Pepijn Senders, a principal engineer who previously led AI enablement at HelloFresh, has done exactly that, and he chose Sonnet over Opus on purpose. His reasoning is what he calls "Sonnet maxxing." A model that reasons less and stops sooner keeps developers involved, and that matters a lot when the codebase is banking software full of hard business rules. The result was 800,000 lines of code, about 60% of them written by Sonnet, shipped by a team that never stopped thinking for itself. Pep also explains how to justify the spend once the token-maxxing era ends and the CFO starts asking questions. He covers a scatter chart that plots token spend against output, the uncomfortable question of whether a 1% better home screen was worth $20K in tokens, why he measures gains in headcount rather than 2x or 3x multipliers, and where he fell short when trying to prove ROI at HelloFresh. We also get into a codebase test you can run yourself, why AI recommendations quietly narrow your choices, and multiplayer refinement sessions where the whole team works through the agent's questions together. Pep shares an experiment where designers and engineers got the same one-shot prompt, and it showed how words like "scalable" and "maintainable" steer a model. We finish with how his team built a meta harness on top of Claude Code, after finding that 85% of their code followed the same path: refinement, specification, implementation, verification. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm Pepijn Senders: https://www.linkedin.com/in/pepijn-senders Timestamps: [0:00] Can you still read your codebase [1:15] Meet Pep from HelloFresh [2:35] What is Sonnet Maxing [3:41] Rebuilding a platform in six months [5:02] Why Sonnet made adoption easier than Copilot [6:37] Proving the ROI of AI coding [8:50] Fighting for AI ROI at HelloFresh [11:02] Token spend vs output [13:24] Tinder pickup lines on the company dime [14:13] Measuring ROI in headcount saved [18:26] Keeping humans in control of AI code [22:26] When Opus and Fable stop speaking English [24:19] Multiplayer refinement sessions with AI [28:16] Magic words that steer the model [32:32] Inside the company Meta harnesses

The World Wild Web & Bots With Buying Power

An AI slips out of its sandbox. A chatbot won’t explain your cat’s stripes. And your next shopping spree might not involve you clicking anything. This episode asks whether “dangerously powerful” AI is a security concern, a marketing pitch, or a little of both. Then we put agents to work: booking trips, tackling German paperwork, drawing diagrams, and navigating the chaos of online shopping. Plus: what happens to checkout, advertising, and customer data when your best customer is a bot? Chapters:[00:00] Is AI Danger the Ultimate Sales Pitch?[00:43] Powerful Models, Payments, and Big Questions[01:10] Too Dangerous to Release, Too Good to Ignore?[03:19] Why Take the Test When You Can Steal the Answers?[08:36] A Security Failure or a Marketing Win?[11:02] Powerful AI, Forbidden Cat Questions[16:00] AI Only Needs to Find One Unlocked Door[20:52] Welcome to the World Wild Web[22:47] What Happens When You Give AI a Wallet?[26:00] Could ChatGPT Become the Next Amazon?[29:50] Mum Wants a Holiday. Can AI Handle It?[33:38] Can AI Beat German Bureaucracy?[36:28] Claude, Draw My Diagram[39:39] Why Is Connecting Your Email Still So Hard?[44:24] Stop Giving Me Options. Just Buy the Sofa.[46:40] When Bots Shop, Who Owns the Customer?

Agents Acting Safely on Our Behalf

What happens to the web when the ones browsing it are no longer people? Sarah Drasner, Area Tech Lead at Google, sits down with us ahead of her Agentic Web keynote at AGNTCon + MCPCon North America. She works through that question from both directions: rebuild the web so agents can use it properly, or build agents smart enough to handle the web as it already exists. We get into the split between computer-use and browser-use agents on one side and API and MCP agents on the other. We also watch a real example of the idea in action: an agent pulling speaker photos from Flickr, hunting down company logos, and loading everything into Canva so the human only touches the creative part. From there, Sarah shares the unlikely origin story of WebMCP (and who's to blame for it). We then pressure-test a bold idea: a WebMCP gateway that could cover the huge share of websites that will never ship support on their own. Along the way, we cover the instinct that separates senior engineers, the ability to sense which business problems are actually worth solving. We also talk about how thought experiments cut through a pile of competing priorities, the hacky WebMCP workarounds already showing up in the wild, and why the best agent experiences are the ones that delight customers in ways they never saw coming. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm Sarah Drasner: https://www.linkedin.com/in/sarahdrasner Timestamps: [0:00] Cold open [0:52] The AGNTCon keynote [1:23] Change the web or fix agents [3:35] Browser agents vs MCP agents [5:48] An agent that builds thumbnails [7:03] Keeping the creative work human [9:12] Agent allowances and engineering instinct [13:34] Thought experiments and priorities [15:23] The WebMCP origin story [17:08] A WebMCP gateway for the web [18:53] It works, but it is not ideal [19:44] Hacky WebMCP in the wild

Bridging the Gap Between AI Builders and Domain Experts

Everyone is racing to build one agent that can do everything. Justin Schroeder, co-founder of Standard Agents, thinks the future looks more like thousands of tiny ones. Ahead of his talk on domain-specific agents at AGNTCon + MCPCon North America, he explains why tightly scoped agents might be the smarter bet, even if it feels like a step back toward the single-purpose machine learning models of a few years ago. We dig into standards as metadata you pin to an agent, so that it shows up with everything it needs in its harness. Then we test the idea against the obvious objection: if a big model can already handle the task, why scope down at all? The answer covers forcing functions, a microservices mindset for agent design, and small agents that quietly check issues or unread messages every five minutes. It also covers the surprise $1K bill waiting for anyone who leaves their agents running. Then it turns into a proper debate. Our host is 99% sure the agent marketplace will never happen, and Justin arrives with a counter that might change his mind (hint: think about where your skills come from). Along the way, we cover agents as tools versus a big-brain router and how anyone keeps track of a thousand agents. We also talk about why the people who know what agents can do and the people who know what they need still have to meet in the middle. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm Justin Shroeder: https://www.linkedin.com/in/jpschroeder Timestamps: [0:00] Cold open [1:03] The future is domain-specific agents [1:51] God models vs specialist agents [2:20] The perfect Gmail agent [3:14] Why standards are missing [3:29] Train the prompt, not the model [4:30] The harness beats the model [6:07] Standards and runtimes for agents [8:17] Why scope down at all [10:07] Why model routing won't win [10:29] The surprise $1K bill [11:12] Agents that build agents [12:09] Agents as tools [13:23] Managing a thousand agents [14:24] Agents for Main Street [15:37] Agent marketplaces will never happen [16:52] Skills are just bad agents [18:04] Skills are already a marketplace [20:12] Domain experts build the agents [22:12] Wrap up

Inside a Game Where AI Agents Never Sleep

What happens to an MMORPG when half the players never log off? Issac Lee, who runs AI and blockchain experiments at NEXUS, has been testing that idea. His team is filling game worlds with agent players instead of scripted NPCs, so a new title feels alive from day one. We get into why agent-generated content could be the next step after studio-made and user-generated games. We also cover how agent populations might solve the chicken-and-egg problem every multiplayer launch faces, and what it means when your fellow players are relentless and never need sleep. Isaac walks through the NEXUS lab: a text-to-3D world builder for Roblox, an AI pipeline that turns ONE store novels into short web dramas, a loop-based tool for natural pixel animation, and Da Vinci, the internal multi-agent framework the team uses to ship. Then the conversation turns to infrastructure. We talk about why NEXUS went from 200 hand-defined agent roles to a much thinner harness, whether custom harnesses still matter now that Claude and Codex exist, and why the Kimi release pushed the company to start building its own small data center for inference. We close on context: how much of your life you should hand to an AI, what personalized meeting notes could look like, and why shopping agents may force every ecommerce site to rethink who it's designing for. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkmIssac Lee: https://www.linkedin.com/in/issacley/?locale=ko Timestamps:[0:00] Cold open[0:57] AI in gaming at NEXUS[2:19] Text to 3D and the last 10 percent[4:38] AI companions and coaches[6:35] Agent players in MMORPGs[8:27] From UGC to agent-generated content[11:07] Agents that never sleep[12:58] Taming fuzzy prompts[19:24] Inside Da Vinci[23:10] Undercover QA agents[27:14] Do harnesses still matter[31:52] Self-hosting open models[41:33] Lab experiments and Bob[50:42] How much context is too much[54:22] Marketing and shopping agents

How a Logistics Giant Keeps AI Data Locked Down

Picture a startup with ten engineers hammering away on AI and one person lying awake over a $20,000 bill that hasn't arrived yet. That's the scenario we put to Jason Ward, who handles FinOps for AI at C.H. Robinson and recently joined the FinOps Foundation's AI working group. Jason's first answer is not glamorous: tag every AI resource so you know who owns it. The rest of the episode is what that makes possible. He walks through how C.H. Robinson runs AI across order entry, quoting, booking, and tracking, and why the team routes Anthropic models through Vertex AI to keep data locked down. Then come the metrics. Jason uses AI to dig through his own observability platform for signals he didn't know were there. One of them is how chatty a model is. That signal turned a prompt bloat alert into a bug in the code that kept retrying and burning tokens. Its opposite, context starvation, burns tokens too: a model with too little context keeps failing and trying again. We also cover why agentic and conversational workloads need separate baselines. Jason explains why cost per order is the easy win, and why most of the real work doesn't fit into neat discrete tasks. That's where his experimental cost per thought metric comes in, with reasoning ratio and cache hit rate alongside it. His advice is simple: your AI is the best tool you have for understanding your AI. Alex Salkever: https://www.linkedin.com/in/alexsalkever Jason Ward: https://www.linkedin.com/in/jward2 Timestamps:[0:00] Cold open[0:33] Meet Jason Ward from C.H. Robinson[1:18] AI use cases in logistics[1:52] Azure OpenAI and Vertex AI[2:19] Why keeping data in-house matters[2:44] How developers use AI day to day[3:40] Using AI to hack AI observability[4:21] Measuring how chatty a model is[5:45] Advice for a startup afraid of its AI bill[6:33] Step one is tag every AI resource[7:05] The Copilot billing blind spot[7:44] Why spend is only half the story[8:12] Break down cost by app and by model[9:09] The prompt bloat that exposed a bug[10:03] Context starvation[11:28] What goes into the AI spend report[12:21] Discrete tasks are low-hanging fruit[12:58] Agentic vs conversational workloads[14:12] Know the workflow before you report on it[15:02] The CTO's end goal[16:08] Cost per thought[17:59] Reasoning ratio and cache hit rate[19:06] Dynamic model routing[20:04] Use your AI to improve your AI

The Caveman Prompting Challenge

Caveman prompting has one rule: why use many words when few do the trick? It saves tokens on the way in and on the way out. Push it too far, though, and the output falls apart. So how far is too far? Nobody has benchmarked it yet, and that question opens our conversation with James Barney, Head of Forward Labs at MetLife. James spends his days connecting new AI capabilities to old business problems across dozens of regulatory regimes, and he still finds time to push code. He explains how the FinOps Foundation's AI working group took on the most basic question: which model for which workload, and why the answer always comes down to cost, speed, and accuracy. We get into Anthropic's launch pricing for Fable, why a million tokens is easy to price and hard to explain, and why every stakeholder eventually tells you what they really care about once you name the wrong North Star. From there it gets practical. Start with the smartest model, then step down and add harness until quality holds. Treat exploration tokens like local builds and production tokens like pipelines. Govern agents the way you govern people, with proactive blocks, reactive checks, and policy as code an agent can actually read. We close on a bigger shift. When a chat window can pull from every dashboard at once, do we still need dashboards? James thinks mostly not, with one catch he calls the latent shopper problem: some insights only come from browsing data you did not know to ask about. Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkm James Barney: https://www.linkedin.com/in/james-barney Timestamps: [00:00] Cold open [01:03] Meet James from MetLife [01:11] What AI enablement means at a global insurer [03:27] Inside the FinOps Foundation AI working group [05:03] The caveman skill challenge [06:37] Why we need a CaveBench [07:43] Anthropic's Fable launch pricing [09:34] Planning for surprise model releases [11:45] Measuring AI value beyond cost [12:47] Finding your unit metric [14:15] Tying token spend to business outcomes [16:35] Explore first, then optimize [18:12] AI that tunes itself [20:13] Do R&D tokens count [23:24] When the experiment becomes the product [25:38] Personal agents and daily briefings [27:39] Governing agents that run just because they can [30:37] The layers of AI governance [31:56] Proactive and reactive guardrails [32:59] Policy as code for agents [34:41] Stop the click ops [35:31] Is the modern UI obsolete [36:36] MCP apps and chat as the new browser [38:07] How AI gathers data differently than humans [40:37] The latent shopper problem [42:12] Staying close to your data

AWS Has 16,000 APIs. Can MCP Handle It?

What happens after you’ve built your first MCP server and actually have to make it work in the real world? In this episode, James Ward dives into the more advanced side of MCP: observability, evals, tool design, code mode, authentication, and the challenges that appear once agents start using your server at scale. We also get into how AWS thinks about MCP across roughly 16,000 APIs, why inefficient tool design often gets blamed on MCP itself, and whether the future could involve more constrained, human-reviewable alternatives to full code mode. Along the way, James shares a great example of an AWS documentation change that accidentally triggered prompt injection warnings from agents, showing just how complicated testing across different models and harnesses is becoming. If you’re already building with MCP and want to understand what comes after the “hello world” stage, this one goes deep. Timestamps: [00:00] “This Code Is Gobbledygook” [00:44] What Happens After You Build an MCP Server? [02:11] The MCP Patterns You Actually Need in Production [03:48] Why Your Agent Is Making Too Many Tool Calls [05:29] How to Make MCP Use Fewer Tokens [08:44] Is MCP Actually Inefficient? [10:11] “We Built a Lot of Pretty Crappy MCP Servers” [11:25] The MCP 2.0 Migration Problem [14:02] Why AWS Is Rethinking Code Mode [15:35] The Problem With Letting Agents Write Python [19:31] How Do You Measure Agent Experience? [22:11] Finding Out Why an Agent Failed [23:42] Hundreds of Evals for Four Cents [24:36] The Evaluation Matrix Gets Massive [26:28] AWS Accidentally Triggered a Prompt Injection Warning [30:16] Should MCP Servers Expose Only Five Tools? [31:02] AWS Has 16,000 APIs. Now What? [34:09] One Super-Agent or Thousands of Specialized Agents? [36:55] The MCP Authentication Problem [40:15] What’s Coming at AgentCon

Skills over MCP on the streets of Tokyo

Tool descriptions tell an agent what a tool does. They don't tell it how to use five tools together, in the right order, following your conventions. That gap is where this conversation lives. Filmed at AGNTCon + MCPCon in Tokyo with Ola Hungerford, Principal Engineer for AI Enablement at Nordstrom and a maintainer of the Model Context Protocol, who spent the last several months turning a pattern everyone was quietly reinventing into an actual MCP extension. Ola walks through what skills over MCP really means: the server stops being a pile of tools and becomes a distribution channel, handing the agent the instructions, workflows and knowledge it needs only at the moment it needs them. She explains why server instructions weren't enough, how progressive discovery keeps context from exploding, and why the same mechanism works for memory and preferences even when no tools are involved.Then it gets into the harder parts. What belongs in the MCP spec versus the agent skills spec. Why passing custom front matter through opens a rug pull and prompt injection surface nobody wanted. Where skills start to look like sub-agents, and why there's still no standard way to declare which servers a skill depends on. And the honest problem underneath all of it: how do you standardize something while everyone is still finding out what it's actually for, without breaking a hundred things the next time you change your mind? Timestamps:[0:00] Intro[0:21] AI enablement at Nordstrom[0:33] What skills over MCP actually is[1:29] The MCP server as a distribution channel[2:01] Server instructions versus skills[3:11] Distributing knowledge and memory[4:23] Progressive discovery explained[5:23] Where the idea came from[6:59] From draft to official extension[8:19] What early adopters changed[8:51] Front matter and custom metadata[9:57] Rug pulls and prompt injection risk[10:57] Will any of this get standardized[12:10] Marrying two very different specs[13:03] Skills as personas and sub-agents[13:53] The missing dependency standard[15:15] How the extension actually works[16:19] What harnesses still need to support[16:59] Consent and skill integrity[17:56] Where skills over MCP goes next[18:43] Why cramming 200 tools fails[19:41] Standardizing before you know the answer[22:18] Is git the wrong tool for agents[23:19] Picking tools for the actual persona[23:51] Trying to be less productive[25:48] The anxiety of idle agents[27:15] Why she keeps a robot on her desk[28:22] If the agent feels the friction, does it matter[29:41] Efficiency, waste, and caring enough[30:52] Letting an agent debug for you[32:03] Choosing your rabbit hole

Walking Tokyo Talking Agent Protocols

Two people, a wrong turn into a back alley, a community garden, and about thirty minutes of arguing about protocols on the streets of Tokyo. The guest is Angie Jones, VP of Developer Experience at the Agentic AI Foundation, fresh off launching AGNTCon + MCPCon in China before the Tokyo stop. She opens with what she learned there: a mobile-first, super-app world where the integration problem most of us obsess over barely exists, where every conversation about agents is really a conversation about the model, and where companies are now reaching for MCP and A2A precisely because they want to operate outside that ecosystem.The bulk of it is WebMCP - a protocol with a confusing name and, until recently, almost no attention. The pitch: put tool calling in the page itself, so your agent works inside your logged-in session with only the tools relevant to the page you're on, instead of screenshotting an anonymous browser and burning tokens guessing at the accessibility tree. Angie explains why it went from ignored to urgent the moment agentic browsing got good, and why the fix for computer use being slow and hijacking your machine might be a standard rather than a better model. It closes on agent-to-agent: whether anyone actually wants a marketplace of thousands of agents, or whether the real value is the one agent that has access you'll never get. Plus a well-earned complaint about three-letter acronyms and why researchers are still the only people naming things well. Timestamps:[0:00] Intro[0:59] Launching the conference in China[1:34] What North America gets wrong about agents[2:23] Super apps versus endless integrations[3:14] What happens when they expand beyond China[3:39] Tencent and A2A in production[4:34] A model-first country[5:56] Chinese coding agents and harnesses[6:59] Tokyo and the conference world tour[7:26] What WebMCP actually is[8:56] Why it has nothing to do with MCP[9:20] Page-level tools and your logged-in session[10:36] Why WebMCP sat unnoticed for months[11:25] Token efficiency and reliability[12:23] The moment computer use got good[13:15] Two real grievances with computer use[14:06] Collaborating instead of surrendering your screen[15:24] A short detour into Tokyo signage[16:15] Why web developers should be excited[17:04] Agents and the loss of first-party data[18:27] Why an agent cannot just buy something[19:23] Inside the agentic commerce working group[20:18] Upsells recommenders and an agent that ignores them[23:27] The commerce protocols to watch[24:19] Why A2A is next[25:31] Publishing your agent as a service[27:13] The case against agent marketplaces[27:55] Why access beats capability[29:31] Google's protocol land grab[30:45] Bring back the cool names[31:39] Amsterdam, San Jose, and what comes next

Why Cost Per Million Tokens Is A Useless KPI?

A year ago, Palo Alto Networks built dashboards to track AI spend. Today those dashboards are useless, and the team that built them thinks that's the whole story. Recorded at FinOps X in San Diego, this conversation brings together Abhinav Lad, who leads cloud and AI finance at Palo Alto Networks, and Kuntal Patel, who runs the cloud engineering function behind it. They explain what happened when agents entered the picture, and AI stopped behaving like a service anyone could forecast. The short version: consumption went from linear to exponential almost overnight. Agents are goal-oriented rather than task-oriented, so they plan, call tools, verify, fail, retry, and keep looping until they hit the outcome, and every iteration is billable. So how do you run finance on top of that? Abhinav and Kuntal walk through the metrics that replaced their old forecasts: adoption rate, cost per user, AI as a percentage of revenue - and the budget limits that let engineering leaders choose between the newest model and a longer runway. They get into the open question of whether a cheaper model saves money or just burns more tokens thinking. They explain why an AI gateway became the control plane for cost and security at the same time, why retry caps belong in the design phase instead of the postmortem, and how FinOps starts to resemble product QA once the bill becomes the clearest signal that something is broken. They close on a warning worth sitting with: cost per million tokens is a number that means almost nothing on its own, and a value story built on it will point you somewhere you don't want to go. Palo Alto Networks: https://www.paloaltonetworks.com Abhinav Lad: https://www.linkedin.com/in/abhinav-lad Kuntal Patel: https://www.linkedin.com/in/kuntalpatel35 Alex Salkever: https://www.linkedin.com/in/alexsalkever Timestamps: [0:00] Intro [1:00] Who runs FinOps for AI at Palo Alto Networks [2:10] Last year's AI dashboards are already useless [4:26] Agents turned linear forecasts exponential [7:27] Three traits that make agents expensive [8:34] The hidden bill: RAG, vectors and egress [9:16] Cost per user and adoption rate [11:21] Giving engineering leaders a budget [12:07] Using DORA metrics to prove value [13:53] Where DORA stops fitting AI [16:20] Does the cheaper model actually save money [17:57] Why you need an AI gateway [20:05] Inside Prisma AIRS [21:00] Three cost models for three use cases [22:52] Forecasting lessons from Electronic Arts [24:03] Runaway agents and endless loops [25:59] Capping retries before they burn cash [28:06] Writing cost policy at design time [29:01] When FinOps becomes product QA [32:17] Explaining AI spend to the C-suite [34:51] Valuing AI beyond engineering [37:04] Crawl, walk, run: where they are today [38:20] Why cost per million tokens is meaningless [39:26] Closing thoughts

The Five-Layer Cake Approach to Scaling AI Without Wasting Money

In this episode of Agentic Conversations, we sit down with Ambud Sharma, Principal Engineer at Pinterest, responsible for general technology efficiency, fresh off delivering a controversial keynote on AI infrastructure optimization at scale. Ambud walks us through his Five Layer Cake framework - a structured approach to driving efficiency across every level of the AI stack, from silicon and hardware procurement to model selection, inference engine design, and governance. We explore how decisions compound across layers to unlock real business growth, and how the wrong choices can lock you into expensive commitments for years. We stress test the framework against two very different business models: what the stack looks like if you are building the next Cursor, and how it changes entirely if you are building the next YouTube. Along the way we cover hardware immutability, inference engine warm-up costs, GPU occupancy, context switching, quantization trade-offs, model routing, and why experimentation discipline is the only thing that keeps AI infrastructure costs from getting out of control. We also look at how this framework holds up in the emerging agent era, what changes when agent-to-agent communication becomes the norm, and why agent traffic just passed bot traffic on Cloudflare. The conversation closes on a deceptively simple takeaway: there is no silver bullet, and experimentation at every layer always comes first. Pinterest: https://about.pinterest.com/ Alex Salkever: https://www.linkedin.com/in/alexsalkever Ambud Sharma: https://www.linkedin.com/in/ambud Timestamps: [0:00] Introduction and the controversial keynote [2:09] The five-layer cake explained [4:30] Why hardware decisions are irreversible [6:47] Two business models: building Cursor vs YouTube [10:42] Applying the five layers to a YouTube-style company [14:23] Experimentation as the core efficiency method [17:09] Inference stack: context switching and warm-up costs [20:07] Model layer: why changing models breaks everything [24:10] When you should not use an LLM at all [26:41] Governance and routing: right model for the right task [29:20] Horror stories of unchecked token spend [31:37] Experimentation discipline without stifling innovation [34:35] How the five layers change in the agent era [36:05] Agent-to-agent communication and governance complexity [38:27] Core takeaway: experimentation first at every layer

The Winchester Mystery House Problem in AI Development

AI models are starting to act like appliances, locked into one narrow way of working, instead of the flexible infrastructure they used to be. Drew Breunig, an AI and data strategist working with the Overture Maps Foundation, joins us to explain why, and what it means for anyone building something that doesn't look like Claude Code. Drew walks through his "Winchester Mystery House" idea: what happens once code gets so cheap to write that the only real bottleneck left is feedback. From there we dig into DSPy: signatures, the GEPA optimizer, and the brand-new Flex optimizer, which rewrites your code instead of just your prompt, complete with a real before-and-after on cost and accuracy. We also get into why so many AI-built apps and websites end up looking identical, the actual difference between an agent and a workflow, what Drew learned a year after shipping a code library with no code in it, and why he thinks the most valuable thing you can do right now is close the laptop and go talk to people. CMPND: https://www.cmpnd.ai Drew Breunig: https://www.linkedin.com/in/drewbreunig/ Demetrios: https://www.linkedin.com/in/dpbrinkm Timestamps: [0:00] Cold open: when Claude Code tries to call itself [1:19] Biggest AI news: labs trading diversity for reliability [2:35] How harnesses get trained into models over time [5:41] The problem: your harness starts fighting the model [9:13] When do you need your own harness? [10:02] The Winchester Mystery House warning [16:13] The blank page problem: why everything looks the same [20:50] Infrastructure vs appliances: the thesis lands [22:40] Current tool loadout: GLM, Kimi, Claude Code, Pi [27:04] The Raspberry Pi personal agent running on Slack [31:00] Crystallizing tasks: when to replace AI with pure code [33:10] DSPy explained: separating what from how [35:23] How prompt optimizers actually work [39:31] DSPy pre-dates ChatGPT: model-agnostic programs [44:00] Why you still need to ship the code, not just the spec [50:00] Don't plan more than a month ahead anymore [54:00] Coaching agents all day feels productive — it isn't [57:58] The dopamine of building with agents vs. why you still need human feedback

How Predictive Analytics Stops Budget Overruns Before They Happen?

Every engineer at Wayfair can now see, in real time, exactly what their code costs, and that's on purpose. Brent Eubanks, FinOps Architect at Wayfair, walks us through what happens when you stop treating AI spend as a finance problem and start treating it as an engineering one.The story that sticks with you: a team was burning $400k a month on an LLM-driven workflow, until they flipped the whole thing on its head - hard-coded logic doing the heavy lifting, AI called in only when it's actually needed. New spend: $80k. It's the kind of move that only makes sense once you realize more agentic autonomy isn't always the win it's sold as.From there we get into the machinery Wayfair has built to make cost ownership feel less like a punishment and more like a superpower: guardrails baked straight into coding copilots, predictive alerts that catch a team blowing its budget before the month even ends, a "spend captain" system that pushes budget decisions down to the people closest to the work, and a genuinely strange new question every tech leader is starting to ask: if you were staffing a company with a team of AI agents instead of people, what would that org chart even look like?Wayfair: https://www.wayfair.comBrent Eubanks: https://www.linkedin.com/in/brenteubanksDemetrios: https://www.linkedin.com/in/dpbrinkmTimestamps:[00:00] Intro & teaser [01:14] Meet Brent (FinOps Architect, Wayfair) [02:34] Making AI adoption easy for developers [03:30] Three-area cost governance framework [04:45] Automated FinOps guardrails in pull requests [07:29] Predictive analytics: 400% over budget warnings [08:26] The 400% spender: asset or liability? [10:06] The "spend captain" model [12:13] The goal: 1000x more token use in two years [13:05] The thought experiment [15:08] The $400k workflow rebuilt for $80k [16:42] Hardcoded first, LLM as fallback [19:49] Engineers now own their full cost stack [20:13] The FinOps companion that works overnight [24:10] Guardrails injected before the pull request [26:17] Recurring tasks become automated skills [27:46] Sankey diagram: cost per agent step in real time [28:40] "The bill didn't blow up - that doesn't happen by accident"

How To Delegate To An Agent Like You Would An Employee?

OpenAI's Codex developer experience lead sits down with a former comedian turned ML engineering lead for a conversation about what happens when computer use agents stop being a novelty and start actually running your day. The conversation moves through building an AI-powered morning brief that reads every email, Slack message, and tweet before you've even opened your laptop, letting pinned threads check in on themselves every 30 minutes, and a skills system built to mirror how a person actually writes and reviews code. There's a close look at the guardrails and permission layers that keep an autonomous agent from pushing to the wrong repo or replying to the wrong tweet, how a codebase merging thousands of pull requests a day survives thanks to self-healing review before anything hits CI, and the idea of AI deference - when an agent should push through a task alone versus stop and ask for help. The back half gets personal: why developing taste and vocabulary now matters more than working harder, what it actually takes to delegate to an agent the way you'd onboard a new employee, and why this might be the year voice-orchestrated computer use finally makes everyone feel like they're talking to Jarvis. OpenAI: https://openai.com Monaco: https://www.monaco.com Jason Liu: https://www.linkedin.com/in/jxnlco Mihail Eric: https://www.linkedin.com/in/mihaileric Demetrios: https://www.linkedin.com/in/dpbrinkm Timestamps: [00:00] Intro and guest backgrounds [01:36] Why computer use beats plain API calls [09:11] Building an AI-powered morning brief [10:17] Self-monitoring threads that check in on their own [18:26] How OpenAI reviews thousands of PRs a day [19:42] Self-healing pull requests before CI even runs [23:07] Building review skills from teammates' habits [30:52] Why hard work stops being the differentiator [35:03] Introducing the idea of AI deference [42:44] Learning to delegate like hiring your first assistant [46:22] Why voice beats typing for giving agents context [50:56] The Tony Stark Jarvis analogy for this year

Why Your AI Bill Will Double Before It Gets Better

In this episode, we're joined by Josh Collier, FinOps Lead at Superhuman (formerly Grammarly), to explore what it really costs to run AI at scale and why the rules of the game changed faster than anyone expected. We discuss how AI token costs dropped 80% in two years, why that trend has sharply reversed with frontier models doubling in price, and how Josh rebuilt a single LLM workflow that cost $400k a month down to $80k by rethinking the architecture. He also shares how a cost calculator built in 15 minutes transformed the way his team estimates spend before running experiments, and why research-led optimization is the only kind that works without degrading the product. Along the way, we cover hidden costs most teams miss, the trade-off between Azure reserved capacity and OpenAI Priority Processing, why fixed subscription pricing is broken in an AI-native world, vendor lock-in risk, and what OpenAI's Guaranteed Capacity announcement really signals about where vendor relationships are heading next. Superhuman: https://superhuman.com Josh Collier: https://www.linkedin.com/in/josh-collier-945b7029/ Demetrios: https://www.linkedin.com/in/dpbrinkm Timestamps: [00:00] OpenAI Guaranteed Capacity: what's really going on [01:04] Josh's path into AI FinOps [02:48] Token costs: the 80% price drop [04:16] Why costs will only go up [05:06] External LLMs as financial risk [07:16] Why subscription pricing is dead [08:22] The data residency fee nobody notices [09:33] The cost calculator built in 15 minutes [10:24] How it changed dev team speed [13:00] Tracking costs by service and team [15:33] $400k workflow rebuilt for $80k [17:13] Why only research can optimize tokens [20:00] Speculative decoding win [23:11] One bad query, $40k gone [26:00] Why Azure PTU was exhausting [28:59] Shadow traffic load testing [29:07] Priority processing: no brainer [31:10] Guaranteed capacity: lock-in signal? [32:18] The danger of multi-year AI deals [33:28] Vendor-agnostic proxy as exit strategy

• Transcript

MCP Goes Stateless

David Soria Parra is an Engineering Lead at Anthropic and one of the core maintainers of the Model Context Protocol (MCP). We explore the biggest evolution of the protocol since its launch, and why MCP is becoming the foundation for the next generation of AI agents. We discuss why MCP is moving toward stateless communication, what developers misunderstand about state, sessions, and transport layers, and how lessons from real-world deployments at massive scale have shaped the protocol's future. We also dive into MCP v2, SDK migrations, protocol design, extension architecture, governance, developer experience, and how Anthropic thinks about balancing simplicity with long-term flexibility. Along the way, we explore progressive disclosure, tool search, programmatic tool calling, context bloat, forward compatibility, long-running AI tasks, protocol evolution, open-source governance, observability, and why the future of AI infrastructure will depend on designing protocols that can evolve without breaking the ecosystem. Timestamps: [00:00] Introduction [01:59] Why MCP Had to Become Stateless [04:28] The Tradeoffs of Stateless Design [06:13] What We Learned About Agent State [08:04] Sessions, Models & Implicit State [09:33] Migrating to MCP v2 [12:19] Lessons from HTTP & Open Source Standards [18:16] Shipping Fast Without Breaking Everything [20:35] The Future Complexity of MCP [22:44] Core Features vs Extensions [26:47] Progressive Disclosure Explained [28:16] Solving Context Bloat [30:50] Why Tool Search Beats Progressive Disclosure [32:10] The Biggest MCP Anti-Pattern [34:25] Designing for Forward Compatibility [38:41] Why "Tasks" Matter [40:53] JSON, Tokens & Better Tool Calling [44:44] Observability & Tracing AI Agents [47:34] Will MCP Ever Be Finished? [50:22] What's Next for MCP

• Transcript

AI Hype vs. Real Value

Manish Dasaur is a Managing Director at PwC with over 20 years in data and AI, having helped 100+ clients navigate AI disruption and extract real business value from data, AI, and agentic AI initiatives. In this episode, he breaks down why most enterprise AI programs stall — and the playbook the winners are using instead. Huge thanks to PwC for supporting this episode! 💰 The 30% benchmark — What "good" actually looks like: real efficiency gains clients are reporting across engineering, finance, HR, and supply chain 🔄 Workflows, not use cases — Why isolated pilots and POCs never show up in EBITDA, and how end-to-end workflow redesign does 🧪 Champion vs. challenger — Running a control group against your AI-automated process so ROI is demonstrated, not guessed 📞 Why customer care agents are still freaking hard — Context, CDP integration, billing systems, and voice-to-voice latency 💸 Tokenomics & FinOps — Consumption-based cost surprises, model selection, prompt engineering, and enforcing cost-per-workflow budgets 🔍 Auditing agentic behavior — Using AI to test AI, the missing "SOC 2 for agents," and certifying agents for sensitive use cases 👤 Human in the loop as an evolving scale — From reviewing 50% of outputs down to 10% as trust builds 🧠 88% do AI, 33% scale it — Building a culture of innovation, and why AI usage is showing up in performance reviews 💼 Jobs, reskilling & the operating model reset — Why 75%+ of jobs will be reskilled, not replacedIf you're an AI leader, platform engineer, or exec trying to turn AI experiments into P&L impact, this one's for you. Links & Resources: Connect with Manish: https://www.linkedin.com/in/manishdasaur/ PwC AI: https://www.pwc.com/us/en/tech-effect/ai-analytics.html PwC's 2026 AI Business Predictions: https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-predictions.html Timestamps: [00:00] AI Hype vs Business Value [00:44] API Spend Tracker Widget [02:18] Tokenomics and FinOps for AI [06:19] Measuring AI Impact Objectively [11:07] AI in Support Workflows [18:05] AI Innovation Culture [27:16] MCP Servers and SOC 2 [29:14] Human in the Loop in evolving scale [35:44] AI and Workforce Efficiency [39:59] AI Transformation and Mindset [42:19] Wrap-up

• Transcript