Skip to content
Artwork for The Experimentation Edge
BusinessTechnology

The Experimentation Edge

Growthbook

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.

Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.

Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Play
  • 46 episodes
  • daily
  • Avg 27 min
  • English
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • S2 · E9
    Thursday · 36 min

    What Wealthsimple decides to test and what to just ship

    Summary Delina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams. Delina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and combined offers. She also shares how to diagnose a losing test, her plan for meta-learning and uplift modeling, and how AI makes it cheaper to test ideas before investing in a full build. For product, data and growth leaders who want experiments that measure the thing they are trying to drive. Chapters 00:00 Wealthsimple, 4 million clients and a never-ending experimentation world 02:46 Two years of formalizing experimentation 03:42 What Misplay taught her about when you need a test 06:00 Rules of thumb: when to test and when to just ship 08:21 Ronny Kohavi, Bing and the value of a millisecond 09:26 How Wealthsimple's 3 data science teams run experiments 12:15 Engagement features, lagging metrics and early signals 14:49 Launching margin and finding an education gap 16:57 Why surveys need experiments 19:10 Sharing results and building systemic learning 21:56 The margin promotion that taught a messaging lesson 26:23 Experiments as a way to learn about people 27:39 Messaging, targeting or experience: diagnosing a losing test 30:35 Meta-learning and uplift modeling 33:42 Using AI to test ideas faster and cheaper Takeaways - Skip the test when you are removing a clear blocker for clients, and test when the change is optional or the outcome is uncertain - Deposits and revenue can lag 60 to 90 days in financial services, so pick early metrics that are proven to lead to them - Combining 2 incentives in one message hides what you are trying to learn; decouple offers so each test isolates one outcome - When a feature you believe in loses, check whether it is a messaging, targeting or experience problem, and design variants to find out - Pooling every past experiment into one view turns tests into a mega learning system for pricing, messaging and personalization Connect with the Guest Delina Ivanova LinkedIn: https://www.linkedin.com/in/delina-ivanova/ Company Website: https://www.wealthsimple.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E8
    Wednesday · 32 min

    How Andy Denning decides which A/B tests are worth running

    Summary Andy Denning, Product Leader, joins host Ashley Stirrup to talk about the judgment that comes before any A/B test. Drawing on his experience with a two-sided jobs marketplace serving job seekers and recruiters across the UK and US, Andy explains why underpowered, badly chosen tests can create false confidence, how experimentation brings organizational honesty, and why his teams start with the riskiest assumption instead of the most comfortable one. He breaks down a US branding test that barely moved applications, how traffic and market shape test design, and why AI makes judgment, not test volume, the real bottleneck. For product and growth leaders who want fewer wasted tests and more learning from the ones they run. Chapters 00:00 Test the riskiest assumption, not the comfortable one 00:55 A two-sided jobs marketplace in the UK and US 01:53 Andy's product role and AI-powered job matching 02:52 Deciding what's worth testing at a smaller company 04:21 Underpowered tests and false confidence 04:45 Experimentation as organizational honesty 05:49 From prototyping software to Optimizely 06:58 Cross-functional teams that own experiments end to end 08:13 Sharing learnings and popcorn sessions 10:31 Guardrail metrics in regulated industries 11:53 Experimenting across the UK and US markets 13:38 The branding test that didn't move applications 17:04 The judgment layer: is the test worth running? 19:03 Experiment design starts with the riskiest assumption 20:39 What to do when a test will take 6 weeks 21:27 B2C vs B2B: what your traffic can support 22:27 Pre-mortems, non-goals and trade-offs 23:49 AI and the flood of low-conviction tests 27:36 Where to spend human judgment 28:26 The AI apply tsunami in hiring Takeaways - Test the riskiest assumption, not the most comfortable one; it is the one the bet rests on - Underpowered or badly chosen tests can manufacture false confidence that is worse than not testing - Before you test, ask whether the outcome is uncertain and whether you would ship it anyway - Run the numbers up front; if a test needs 6 weeks, find a faster or cheaper way to learn - AI makes variants cheap, so the bottleneck is now the quality of the hypothesis Connect with the Guest Andy Denning LinkedIn: https://www.linkedin.com/in/andy-denning-547a0862/ Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E7
    Tuesday · 18 min

    How GE HealthCare tests for quality leads, not just more clicks

    Summary Tom Gores, Director, Product - Digital Experience & E-commerce at GE HealthCare, joins host Ashley Stirrup to share what changes when you take an experimentation program from B2C retail to B2B healthcare. Tom built his first A/B testing program at Kohl's 12 years ago and now runs a centralized program at the center of GE HealthCare's digital ecosystem, where buying cycles can take a year and each qualified lead carries much higher stakes. Tom explains why a Contact Us test that lifted clicks also lifted junk leads, how his team turns one-off page ideas into bigger, broader tests across templates, and why planning for failure before launch keeps experimentation framed as learning. He also covers the 3 areas where he sees experimentation growing, people and culture, data, and technology, and the metrics he watches, from CSAT and conversion rate to visibility in AI search. For product, digital and growth leaders running experiments in long, high-stakes B2B journeys. Chapters 00:00 Welcome Tom Gores of GE HealthCare 00:51 From Kohl's to GE HealthCare: 12 years of A/B testing 02:01 GE HealthCare's digital ecosystem 02:51 B2B vs B2C: longer buying cycles and the long game 03:49 A centralized program with lower traffic and higher stakes 05:52 The Contact Us test: more clicks, junkier leads 07:26 Funneling the right buyers to the right experience 08:47 Designing experiments for the most learning 10:33 Planning for failure and picking leading indicators 13:07 Where experimentation grows next: people, data and technology 15:22 CSAT, conversion rate and AI visibility 17:34 How GE HealthCare measures CSAT Takeaways - In B2B healthcare, buying cycles can run months to a year, so experiments need leading indicators and long-term tracking rather than instant conversions - Pushing a metric in front of people can lift clicks while lowering lead quality; check results with the sales teams who work the leads - Turn a single-page idea into a bigger, broader test by applying it to a whole template, so thousands of visitors are affected instead of hundreds - Map out what you will do if a test fails before launch, so results stay framed as learning and avoid wasted builds - Track CSAT, conversion rate and AI visibility together to measure an ecosystem that serves many audiences and journey stages Connect with the Guest Tom Gores LinkedIn: https://www.linkedin.com/in/tom-gores-6925a81a Company Website - GE HealthCare: https://www.gehealthcare.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E12
    November 9 · 22 min

    How Glassdoor turned a handful of A/B tests a year into dozens a quarter

    Summary Jeremy Echols, Director of Digital Analytics at Glassdoor, part of Indeed, joins host Ashley Stirrup to talk about building an experimentation program around people, not sessions. Jeremy came to analytics from an unusual path, as a former pastor with a master of divinity, and he sees math and statistics as a way to do applied psychology. Jeremy explains how Glassdoor went from a slow homegrown system and a handful of tests a year to dozens of tests a quarter, with a center of excellence, office hours, Amplitude templates and a shared data language. He breaks down an Indeed account-linking test that exposed variant jumpers across devices, why every test needs a hypothesis and a curious set of secondary metrics, and how some friction is worth keeping. For analytics, product and growth leaders who want faster iterations and deeper learning from every test. Chapters 00:00 Sessions don't buy anything, people do 00:42 Glassdoor's mission: workplace transparency 01:27 Glassdoor Recruiter: a recruiter that works for the job seeker 02:50 From pastor to analytics: statistics as applied psychology 03:46 Why synthetic users can't replace real people 04:14 Centralizing experimentation at Glassdoor 05:24 Templates, guardrails and an experimentation community 06:49 The account-linking test that exposed variant jumpers 08:10 Why users browse without logging in 09:48 Guest checkout hides the customer journey 11:01 Costco and the case for personalization 11:40 What to learn when a new feature fails 12:11 Write the hypothesis, then ask what else might happen 14:00 Tying experiments to your analytics funnels 14:51 Pre-mortems and testing the opposite 15:56 When removing friction hurts the funnel 17:04 Turning experimentation marathons into sprints 18:19 One data language for every team 19:02 Glassdoor Recruiter and SuperMatch 20:35 Hiring managers and passive candidates Takeaways - Measure people, not sessions; users move across devices, so incentivize login to see the whole journey - Bake best-practice settings and guardrail metrics into test templates so no one has to remember them - Write a hypothesis for every test, then add the secondary metrics that could teach you something unexpected - Some friction is good; more leads can mean fewer qualified leads and a weaker downstream funnel - One shared data language across marketing, email and product makes experiments faster and easier to compare Connect with the Guest Jeremy Echols LinkedIn: https://www.linkedin.com/in/jeremyechols/ Company Website: https://www.glassdoor.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E6
    November 5 · 32 min

    How RealPage finds its biggest A/B test wins, one funnel at a time

    Summary Andrew Miller, Director of Product Management, Growth and E-commerce at RealPage, joins host Ashley Stirrup to share how experimentation works inside Loft, RealPage's end-to-end renter app. Andrew has stood up experimentation programs at 3 companies, and today he runs a portfolio of moving services where a winning A/B test ties directly to revenue. Andrew explains how a small, centralized optimization team ships tests in Optimizely faster than engineering release cycles allow, and how he builds a culture of experimentation with funnels, group ideation, the Opportunity Solution Tree and simple test templates. He also breaks down 2 electricity experiments: one that cut choices from 8 to 3 and lifted conversion 12%, and a price display test that lost. For product and growth leaders who want more ideas, faster launches and tests that actually teach them something. Chapters 00:00 RealPage, Loft and a B2B2C business 01:17 Why every A/B test in Loft focuses on the renter 03:02 The moving marketplace and how it makes money 04:53 Standing up experimentation programs at 3 companies 07:10 The T-shaped PM: depth in data and experimentation 08:57 A centralized optimization team for all of Loft 11:05 Why tests run in Optimizely instead of engineering 12:42 Building a culture of experimentation with funnels 15:34 The Opportunity Solution Tree and test templates 17:06 How good is your gut at predicting winners? 19:01 Cutting electricity options from 8 to 3 for a 12% lift 21:13 The price display test that lost 24:26 Minimizing variables when traffic is limited 26:04 Why diversity of ideas matters 27:38 PMs submitting PRs, not just hypotheses 30:45 Prototyping tests with AI: from 2 weeks to 1 day Takeaways - Start with telemetry and funnels; funnel drop-offs get stakeholders to the insight before you have to say it - Run the first ideation sessions as a group and organize ideas with an Opportunity Solution Tree; one session can fill a year of backlog - Require a hypothesis, a metric and a definition of success for every test idea so learning is the baseline outcome - Fewer choices can lift conversion: cutting electricity plans from 8 to 3 raised conversion 12% - An estimate shown with too much precision can backfire; round estimated prices so they read as estimates Connect with the Guest Andrew Miller LinkedIn: https://www.linkedin.com/in/findandrew/ Company Website: https://www.realpage.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E51
    November 4 · 31 min

    Why you're already wrong 75% of the time

    Summary In this special episode of The Experimentation Edge, host Ashley Stirrup looks back at lessons from more than 50 guests with two people from GrowthBook: co-founder and CEO Graham McNicoll and Head of Customer Success & Strategy Cris Dizon, who ran experimentation programs at The Home Depot, Walmart and Peacock. The starting point is a humbling number: across companies, only about 25% of experiments win, so even your best ideas are wrong most of the time. Graham shares the untested redesign that quietly cut new-user sign-ups by about 20% for six months, and the bug that set a paywall to zero and overturned years of tribal knowledge. Cris explains how a streaming onboarding flow that failed came back and won once personalization caught up, and why heavy promo codes train customers to wait. Along the way they revisit stories from UPS, JPMorgan Chase, Fanatics, Chess.com, Groupon, Signet Jewelers, Box, Disney, Twitch, Learneo and Charlie Health, and close with five ways executives can build a culture that measures and learns from being wrong. Chapters 00:00 Welcome to a special episode with Graham McNicoll and Cris Dizon 01:24 Why only about 25% of experiments win 03:28 Building a culture where it's okay to fail 04:31 From Fyxer to UPS: guests who admit their gut is wrong 05:05 The untested redesign that cost millions 06:17 A big-bang homepage redesign that took months to fix 07:48 Spot the million-dollar change in the JPMorgan Chase funnel 08:48 Twitch pricing and the risk of anti-knowledge 09:28 The bug that set the paywall to zero 11:05 The onboarding test that failed, then won 12:59 Capturing learnings so they don't get lost 13:54 UPS, JPMorgan Chase and Fanatics: the size of the wins 14:18 Reframing wins and losses as gains and savings 15:45 Measure your learning rate, not your win rate 16:49 Chess.com: review your wins, not your losses 18:09 Groupon swaps flash sales for scarcity 18:52 How promo codes train customers to wait 20:18 Signet Jewelers and friction that helps buyers 21:11 Box and the wine effect 22:34 The checkbox with more text that lifted conversions 23:28 Test the opposite of your hypothesis 24:15 Disney's painted door test for video support 25:55 Five ways executives can build an experimentation culture 27:58 Could we have learned this faster? 28:32 Share the design and let people pick the winner 29:35 Buddy programs and pre-mortems 30:27 You're already wrong 75% of the time Takeaways - Expect to be wrong: even mature programs see only about 1 in 4 experiments win - Never ship a big-bang redesign untested; a hidden loss can run for months - Track savings from losses alongside gains from wins - Measure your learning rate, not your win rate - Revisit old losers when your product or capabilities change Connect with the Guests Graham McNicoll LinkedIn: https://www.linkedin.com/in/grahammcnicoll/ Company Website: https://www.growthbook.io Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E5
    November 3 · 32 min

    How Preply runs close to 1,000 experiments a year

    Summary Sebastien Polis, Senior Experimentation Manager at Preply, joins host Ashley Stirrup to explain how a learning marketplace with 25 to 30 product teams runs close to 1,000 experiments a year. At Preply, no change reaches customers without a test, OKRs can only be claimed through experiments with an approved target metric, and a public Slack channel puts every launch, result and kill decision in front of the exec team. Sebastien shares why roughly 80% of experiments lose and 30 to 40% are iterations of an earlier idea, how a global control group shows double-digit growth, and what Preply learned when a control group went negative and a simple wording win turned out to be Twyman's law in action. He also covers why marketplaces randomize tutors instead of learners, the clustered analysis that requires, and his plan to build a learning system that gives AI the context it needs. For product, data and growth leaders building an experimentation culture at scale. Chapters 00:00 Preply's learning marketplace and Sebastien's path into experimentation 04:52 Tying OKRs to experiments across 25 to 30 product teams 06:39 Why 80% of experiments lose and 30 to 40% are iterations 08:35 Double-digit growth from a global control group 09:46 The control group that went negative and Twyman's law 15:03 How a new PM designs an experiment at Preply 17:57 A public Slack channel where execs question every test 20:48 When clicks go up but learners don't buy 23:23 Building a learning system before feeding AI 25:20 An experiment run entirely by AI, with a human approver 28:13 Randomizing tutors, not learners, in a 2-sided marketplace Takeaways - Preply ties OKRs to experimentation: teams can only claim results through experiments that use an approved target metric - Losing is the norm; about 80% of Preply's tests fail and 30 to 40% of each year's experiments are iterations of an earlier idea - A surprising win from a simple change deserves a rerun; Preply unscaled a wording test and the lift disappeared - A public Slack channel where execs question PMs about launches and kills keeps rigor and transparency high across 1,000 tests a year - In a marketplace, randomize the tutor and measure learner outcomes with a clustered analysis, and ship bug-free code because sample size is scarce Connect with the Guest Sebastien Polis LinkedIn: https://www.linkedin.com/in/sebastien-polis-b5aa8256/ Company Website: https://preply.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E4
    October 29 · 25 min

    Building a Culture of Experimentation in Sports Analytics with New York Red Bulls

    Summary Vamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin. Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized fan experiences. For analytics, marketing and product leaders building experimentation into a business where every event is different. Chapters 00:00 Intro 00:20 New York Red Bulls: an original MLS club 01:32 Data-driven fan engagement 04:46 Single game buyers outspend season ticket holders 07:21 Testing frameworks 09:29 Holistic fan value 11:11 Segments can never be static 12:59 A losing test isn't a failure 14:51 Loser or lever: judging tests against the North Star 17:13 Building a culture with clear end goals 20:25 Ideas populate when there is more data 23:10 Identity resolution, AI and personalization 26:06 Empowering more teams to experiment 27:57 Every game is like hosting a wedding Takeaways - We have seen instances where single game buyers at the New York Red Bulls outspend season ticket holders once concessions and the Hype House zone are counted - The club moved its North Star from average ticket price to gross margin, which changed couponing, fan profiles and retargeting - A losing test is not a failure; judged against the North Star, a negative result shows what not to do and that a lever exists - Segments should be dynamic, and the club is building toward a separate segmentation for every event - Ideas populate when there is more data, and the stakeholders who know the business best should drive the questions Connect with the Guest Vamsi Krishna Kunapuli LinkedIn: https://www.linkedin.com/in/vamsikunapuli/ Company Website: https://www.newyorkredbulls.com Vamsi is a panelist at DMWF North America 2026, "Connected Data: The Foundation of Intelligent Marketing" Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E3
    October 27 · 38 min

    Motive's Shiva Manjunath on CRO as a mindset, not a role

    Summary Shiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas. The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For CROs, product managers and growth leaders building an experimentation culture. Chapters 00:00 Cold open: gifts under $50 01:18 What a true culture of experimentation looks like 04:43 CRO as a mindset across every function 11:27 Become the sales team's best advocate 15:29 Fear, uncertainty and doubt: the window shade story 21:51 Customers ask for price when they want value 24:17 GTA 6 and why button colors cost CROs trust 29:14 Making an impact without an executive sponsor 34:27 Making experimentation as exciting as AI Takeaways - CRO works best as a mindset shared by design, marketing and product, not as one team's job. - Sales and customer service teams hold free qualitative data that makes strong test ideas. - When customers ask for price, test value messaging such as benefits, ROI and time saved. - Understanding the emotional context around a purchase earns a CRO more trust than tactical button tests. - A culture of experimentation needs data-driven decision making first and a safe space to lose. Connect with the Guest Shiva Manjunath LinkedIn: https://www.linkedin.com/in/shiva-manjunath/ From A to B podcast: https://open.spotify.com/show/5xDlubE3GZwUDDen64hefz Newsletter: https://fromatob.beehiiv.com/ Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E3
    October 22 · 26 min

    How Quill turned loyalty points into a 1% conversion lift

    Summary Vipul Tyagi, Senior Product Manager at Quill, owns the online experience from add to cart through checkout, with a focus on getting customers to use Quill's loyalty program. Host Ashley Stirrup digs into why a homepage rewards widget drew under 0.5% engagement over 60 days, and how moving the value to the cart with a Lego-inspired points to dollars option lifted conversion by 1% while shrinking the points liability on Quill's books. The conversation covers Vipul's 8 winners out of 11 loyalty initiatives and the BillionToOne advice behind them, how he coaches new PMs on primary and secondary hypotheses and guardrail metrics, a live test on showing the $5.99 shipping saving at checkout, a semantic layer experiment built for AI crawlers, and how AI has cut his research time from two months to seven days. For product managers, growth leaders and e-commerce teams who want a higher win rate from their experiments. Chapters 00:00 Cold open 01:06 Quill, office supplies and three kinds of customers 04:20 Owning add to cart through checkout, and 8 winners out of 11 05:32 Know the direction before you run the test 06:54 The homepage loyalty widget and the icing on the cake 10:19 Points into dollars on the cart page: a 1% conversion lift 12:39 Coaching new PMs: ranked hypotheses and guardrail metrics 15:41 Taking credit for free shipping in the order summary 18:10 Copy testing, strategic bets and a semantic layer for AI crawlers 22:24 AI in the PM workflow: research in seven days instead of two months Takeaways - Rewards work where purchase intent already exists: Quill's homepage loyalty widget drew under 0.5% engagement, while the same value on the cart page lifted conversion. - Converting points into dollars at the cart raised conversion 1%, brought small orders back and reduced the unredeemed points liability on the books. - Know the direction of a test before you run it; the A/B test measures how much, which is how Vipul's loyalty work reached 8 winners out of 11. - Design every experiment with a primary hypothesis, ranked secondary hypotheses and guardrail metrics so a loss still explains itself. - Showing the $5.99 shipping saving at checkout raised conversions without changing order value, a sign that transparency builds trust. Connect with the Guest Vipul Tyagi LinkedIn: https://www.linkedin.com/in/vipul-tyagi Company Website: https://www.quill.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io (00:00) - Cold open (01:06) - Quill, office supplies and three kinds of customers (04:20) - Owning add to cart through checkout, and 8 winners out of 11 (05:32) - Know the direction before you run the test (06:54) - The homepage loyalty widget and the icing on the cake (10:19) - Points into dollars on the cart page: a 1% conversion lift (12:39) - Coaching new PMs: ranked hypotheses and guardrail metrics (15:41) - Taking credit for free shipping in the order summary (18:10) - Copy testing, strategic bets and a semantic layer for AI crawlers (22:24) - AI in the PM workflow: research in seven days instead of two months

    • Transcript
    • Chapters
  • S2 · E2
    October 20 · 40 min

    Engagement with safety: Headspace's guardrails for AI tests

    Mary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defi

    • Transcript
  • S2 · E1
    October 15 · 19 min

    How Walmart Connect learned to align cross-sell with shopper intent

    Summary Arv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains

    • Transcript
  • S1 · E50
    October 14 · 19 min

    Why more info lifted VidaXL's account creation nearly 40%

    Summary Sapna Kashyap, CRO Lead at VidaXL, runs the experimentation program for an e-commerce company operating in more than 30 countries, where product, development and business teams all validate their decisions with an A/B test first. She explains the

    • Transcript
  • S1 · E49
    October 13 · 40 min

    The auto researcher: How FanDuel finds a winning model in 300 attempts

    Michael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for ro

    • Transcript
  • S1 · E48
    October 8 · 21 min

    What Grubhub tests for big product bets before they ship

    Summary Grubhub does not run much classic A/B testing. Michal Lenik, Associate Director of Product Design at Grubhub, explains what the company does instead to make sure its big product bets win before they reach millions of users. She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes, and a concept test in the merchant promotions space that sent the team toward a much simpler dashboard than their research predicted. Michal also explains why she follows losing experiments with user interviews, why "I didn't even notice it" is valuable feedback, and why every company should decide deliberately whether it de-risks upstream or in a fast follow after launch. Chapters 00:00 Where design is heading in the AI era 00:53 Meet Michal Lenik of Grubhub 02:21 Design as an upstream business lever 03:48 Small cohorts instead of classic A/B tests 05:27 The merchant concept test that changed direction 08:18 Prototyping with AI and merchant round tables 10:16 Success metrics for product, design and engineering 11:56 Pairing experiment data with user interviews 14:26 AI, personalization and the designer's new role 17:45 De-risking upstream versus shipping fast Takeaways - Grubhub tests big bets by releasing to internal users or small cohorts first, so it can read business impact without disrupting core revenue and order metrics. - A concept test with merchants overturned what fifteen research interviews suggested, steering Grubhub to a simpler dashboard and away from a launch that likely would not have won. - AI lets Grubhub's design teams put clickable prototypes in front of users early, turning each iteration into a micro experiment. - Pairing quantitative results with five to ten user interviews after a test explains why it underperformed, including whether users even saw the change. - Both upstream de-risking and fast iteration work. The right choice depends on how quickly engineering can ship, and it should be made on purpose. Connect with the Guest Michal Lenik LinkedIn: https://www.linkedin.com/in/michallenik/ About the guest: https://www.growthbook.io/podcast/guests/michal-lenik Episode page: https://www.growthbook.io/podcast/episode/1-48 Company Website: https://www.grubhub.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io (00:00) - Where design is heading in the AI era (00:53) - Meet Michal Lenik of Grubhub (02:21) - Design as an upstream business lever (03:48) - Small cohorts instead of classic A/B tests (05:27) - The merchant concept test that changed direction (08:18) - Prototyping with AI and merchant round tables (10:16) - Success metrics for product, design and engineering (11:56) - Pairing experiment data with user interviews (14:26) - AI, personalization and the designer's new role (17:45) - De-risking upstream versus shipping fast

    • Transcript
    • Chapters
  • S1 · E47
    October 6 · 20 min

    How MilliporeSigma lifted add to cart with a smarter search

    Summary Dorothy Crepin runs the A/B testing program at MilliporeSigma, the life sciences business of Merck KGaA, where one global catalog has to serve three very different visitors: scientists hunting for a specific instrument, procurement buyers working off a list someone else wrote, and students doing research with no purchasing power at all. Add a regulated industry where what you can sell changes country by country, and personas stop being a marketing exercise. She walks Ashley Stirrup through the test that changed how her team thinks about discovery. MilliporeSigma rebuilt its on-site type ahead so it suggests a search term instead of pushing a matching product, and saw double digit increases in feature usage and add to cart. From there Dorothy makes the case for measuring the action a page is actually responsible for rather than revenue alone, for pairing A/B tests with a user testing ambassador panel to get at the why, and for sending product managers back to one question before anything gets built: what problem are you solving for the customer? Chapters 00:00 Revenue is an outcome, not the metric 01:26 Welcome and what MilliporeSigma does 02:22 The three audiences on a B2B science site 03:53 What an A/B testing analyst coordinates 04:20 Running experiments across product pods 05:31 Building leadership buy in for testing 08:08 Where A/B testing ends and user testing begins 08:52 The type ahead search test that lifted add to cart 11:30 How to frame a test with a product manager 13:34 Why testing the opposite pays off 15:26 Personas, personalization and what AI changes 18:57 Building trust across competing teams Takeaways - MilliporeSigma changed its on site search to suggest search terms instead of products and saw double digit gains in feature usage and add to cart. - Revenue is an outcome, so Dorothy's team measures the next action each page is responsible for, like add to cart on a product detail page. - A/B testing tells you what happened. MilliporeSigma's user testing ambassador panel supplies the why behind the click. - Every test starts with one question to the product manager: what problem are you solving for the customer, not what is your hypothesis. - Dorothy is building personas into agents so browsers, not just buyers, can be tested for a directional read before an experiment ships. Connect with the Guest Dorothy Crepin LinkedIn: https://www.linkedin.com/in/dorothycrepin/ About the guest: https://www.growthbook.io/podcast/guests/dorothy-crepin Episode page: https://www.growthbook.io/podcast/episode/1-47 Company Website: https://www.sigmaaldrich.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E46
    September 30 · 31 min

    PayPal's $180 million experimentation win

    Summary Gaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of exposure data, carrying 25% to 30% dilution. The conversation covers the exposure event his team introduced, the instrumentation and metric standards that cut readouts to 24 hours, a carousel test that a multi armed bandit resolved in 51 days instead of a projected 700, and why cost avoidance from losing experiments belongs in the ROI number. It closes on what changes when the thing you are testing is an AI agent rather than a button. Useful for product managers, engineers, data scientists and growth leaders building or defending an experimentation program at scale. Chapters 00:00 Cold open 00:54 Welcome Gaurav Sethi of PayPal 01:30 Elmo, PayPal's homegrown experimentation platform 03:53 $180 million in revenue impact and cost avoidance 04:28 The win rate that was too good to be true 06:29 Instrumentation standards and the exposure event 08:56 The carousel test: 700 days down to 51 11:29 Designing experiments so every result teaches you 13:45 Building the platform is only half the job 15:07 Education, office hours and executive support 18:28 Exposure events and joining transactional data 24:14 AI for experimentation, and experimentation for AI Takeaways - A win rate of 55% to 60% against an industry average of 11% to 14% was a tracking problem, not a performance one: assignment data carried 25% to 30% dilution and reached significance on users who never saw the test. - The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load. - Standardized instrumentation and canonical metric definitions are what let the analysis pipeline run without human cleanup, taking experiment readouts from four weeks to 24 hours. - A six variant carousel test projected at 700 days was resolved by a multi armed bandit in 51 days, with a winner worth almost five basis points. - About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments. Connect with the Guest Gaurav Sethi LinkedIn: https://www.linkedin.com/in/gauravsethi22 About the guest: https://www.growthbook.io/podcast/guests/gaurav-sethi Episode page: https://www.growthbook.io/podcast/episode/1-46 Company Website: https://www.paypal.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E45
    September 29 · 27 min

    ServiceNow's Customer Zero approach to AI experimentation

    Summary ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer. She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it. Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late. Chapters 00:00 What customers actually care about 01:14 Meet Ashraf Karim of ServiceNow 01:39 Owning the entire post-sale experience 02:08 Why ServiceNow runs as its own Customer Zero 03:02 Lessons from Google, PayPal and Verizon 07:11 The agentic skills users kept abandoning 09:29 Latency, expectations and the feedback loop fix 12:29 Users wanted the outcome, not the process 14:49 Designing experiments that can afford to lose 18:13 Using an LLM judge on nondeterministic models 21:48 Customer effort score as a guardrail metric 25:03 Where experimentation at ServiceNow goes next Takeaways - ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it. - The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like. - The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs. - Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps. - Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer. Connect with the Guest Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim Episode page: https://www.growthbook.io/podcast/episode/1-45 Company Website: ServiceNow Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E44
    September 24 · 24 min

    Why Twilio ships on signals instead of significance

    Summary Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not. The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off decision framework comes from. She also explains the parts of B2B experimentation that have no B2C equivalent: choosing between user-level and account-level randomization, the interference risk when two colleagues at the same company see different pricing, and the many-to-many problem of one developer belonging to several accounts. And because Twilio's customer counts are lower than a consumer business, her teams often ship on signals and team conviction rather than waiting for statistical significance. Chapters 00:00 Cold open 01:02 Welcome Wanli Lau of Twilio 01:23 Inside Wanli's role at Twilio 04:03 Eleven years and 1,500 tests a year at Expedia 04:58 Why B2B experimentation differs from B2C 05:33 Cross-functional collaboration and good hypotheses 06:39 Teaching teams to experiment rigorously 07:48 The Expedia sign-up takeover that won on conversion 11:09 Guardrail metrics and the trade-off framework 14:52 Designing experiments to maximize learning 17:46 Diagnosing where a feature failed 19:10 North Star metrics and account-level randomization at Twilio 21:40 Learning from signals instead of significance 22:11 Where experimentation goes next at Twilio Takeaways - A test that wins on the primary metric can still be a loss. Expedia's sign-up takeover raised conversion and damaged return rate and engagement at the same time. - Name the primary, secondary and guardrail metrics before the test runs, not at readout. Deciding afterwards turns a result into a debate. - Conversion should be a do-no-harm guardrail for teams that do not own it, even when it is not their primary metric. - In B2B the randomization unit is a design decision. User-level bucketing risks two colleagues at one account seeing different prices, and account-level avoids that but costs sample size. - When customer counts are low, significance is often out of reach. Learning from signals and shipping on team conviction beats waiting for a number that will never arrive. Connect with the Guest Wanli Lau LinkedIn: https://www.linkedin.com/in/wanlilau About the guest: https://www.growthbook.io/podcast/guests/wanli-lau Episode page: https://www.growthbook.io/podcast/episode/1-44 Company Website: https://www.twilio.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E43
    September 22 · 26 min

    Why GoPro stopped judging A/B tests by win rate

    Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test program. Will shares the Barnes & Noble order confirmation test he was sure would lose, and why the same idea never worked for any other client. He walks through a recent GoPro Mission launch test that asked whether a step-by-step configurator adds too much friction, and what a flat result revealed about high consideration buyers. He also explains why GoPro shares interim readouts across the company, why win rate makes a poor North Star for an experimentation program, and how his team plans to use AI for speed without outrunning its own learnings. Chapters 00:00 Intro 00:52 Will's role running e-commerce at GoPro 01:43 Learning A/B testing across retail at Brooks Bell 05:24 How GoPro runs one to three tests a month 06:35 Sharing learnings and interim readouts across teams 09:20 The Barnes & Noble order confirmation win 13:02 Why the win did not transfer to other clients 14:41 Testing friction on the GoPro Mission configurator 18:51 Why win rate is the wrong North Star 22:19 How AI will shape experimentation at GoPro Takeaways - A winning idea rarely travels. The Barnes & Noble recommendation module worked because of that audience's low order values and reading habits, and it failed for every other client that tried it. - Design every test so it teaches you something whether it wins, loses or ends flat. Losing tests are jet fuel when the learning is built in. - A flat result is still an answer. GoPro's configurator test showed that buyers of high consideration products accept extra steps when each choice adds value. - Share interim readouts across the company, and use them to show how volatile results are before a test reaches statistical significance. - AI can speed up building and running experiments, but a team that runs more tests than it can learn from is not getting better. Connect with the Guest Will Guyeskey LinkedIn: https://www.linkedin.com/in/willguyeskey/ About the guest: https://www.growthbook.io/podcast/guests/will-guyeskey Episode page: https://www.growthbook.io/podcast/episode/1-43 Company Website: https://gopro.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
Showing 1–20 of 46 episodes