
What Wealthsimple decides to test and what to just ship
Summary Delina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams. Delina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and combined offers. She also shares how to diagnose a losing test, her plan for meta-learning and uplift modeling, and how AI makes it cheaper to test ideas before investing in a full build. For product, data and growth leaders who want experiments that measure the thing they are trying to drive. Chapters 00:00 Wealthsimple, 4 million clients and a never-ending experimentation world 02:46 Two years of formalizing experimentation 03:42 What Misplay taught her about when you need a test 06:00 Rules of thumb: when to test and when to just ship 08:21 Ronny Kohavi, Bing and the value of a millisecond 09:26 How Wealthsimple's 3 data science teams run experiments 12:15 Engagement features, lagging metrics and early signals 14:49 Launching margin and finding an education gap 16:57 Why surveys need experiments 19:10 Sharing results and building systemic learning 21:56 The margin promotion that taught a messaging lesson 26:23 Experiments as a way to learn about people 27:39 Messaging, targeting or experience: diagnosing a losing test 30:35 Meta-learning and uplift modeling 33:42 Using AI to test ideas faster and cheaper Takeaways - Skip the test when you are removing a clear blocker for clients, and test when the change is optional or the outcome is uncertain - Deposits and revenue can lag 60 to 90 days in financial services, so pick early metrics that are proven to lead to them - Combining 2 incentives in one message hides what you are trying to learn; decouple offers so each test isolates one outcome - When a feature you believe in loses, check whether it is a messaging, targeting or experience problem, and design variants to find out - Pooling every past experiment into one view turns tests into a mega learning system for pricing, messaging and personalization Connect with the Guest Delina Ivanova LinkedIn: https://www.linkedin.com/in/delina-ivanova/ Company Website: https://www.wealthsimple.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript


















