Skip to content
Sign in
Artwork for Data Engineering Podcast
TechnologyEducation

Data Engineering Podcast

Tobias Macey

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Play
Open in
  • 23 episodes
  • Avg 54 min
  • English

Support the show

Goes straight to the publisher. podnod takes nothing.

Episodes 23

NewestOldestLongest
All3Unplayed3In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • #497
    January 18 · 1 hr 12 min

    Your Data, Your Lake: How Observe Uses Iceberg and Streaming ETL for Observability

    Summary In this episode Jacob Leverich, cofounder and CTO of Observe, talks about applying lakehouse architectures to observability workloads. Jacob discusses Observe’s decision to leverage cloud-native warehousing and open table formats for scale and cost efficiency. He digs into the core pain points teams face with fragmented tools, soaring costs, and data silos, and how a lakehouse approach - paired with streaming ingest via OpenTelemetry, Kafka-backed durability, curated/columnarized tables, and query orchestration - can deliver low-latency, interactive troubleshooting across logs, metrics, and traces at petabyte scale. He also explore the practicalities of loading and organizing telemetry by use case to reduce read amplification, the role of Iceberg (including v3’s JSON shredding) and Snowflake’s implementation, and why open table formats enable “your data in your lake” strategies. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests. You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud. Your host is Tobias Macey and today I'm interviewing Jacob Leverich about how data lakehouse technologies can be applied to observability for unlimited scale and orders of magnitude improvement on economics Interview Introduction How did you get involved in the area of data management? Can you start by giving an overview of what the major pain points have been in the observability space? (e.g. limited scale/retention, costs, integration fragmentation) What are the elements of the ecosystem and tech stacks that led to that state of the world? What are you building at Observe that circumvents those pain points? What are the major ecosystem evolutions that make this a feasible architecture? (e.g. columnar storage, distributed compute, protocol consolidation) Can you describe the architecture of the Observe platform? How have the design of the platform evolved/changed direction since you first started working on it? What was your process for determining which core technologies to build on top of? What were the missing pieces that you had to engineer around to get a cohesive and performant platform? The perennial problem with observability systems and data lakes is their tendency to succumb to entropy. What are the guardrails that you are relying on to help customers maintain a well-structured and usable repository of information? Data lakehouses are excellent for flexibility and scaling to massive data volumes, but they're not known for being fast. What are the areas of investment in the ecosystem that is changing that narrative? As organizations overcome the constraints of limited retention periods and anxiety over cost, what new use cases does that unlock for their observability data? How do AI applications/agents change the requirements around observability data? (collection, scale, complexity, applications, etc.) What are the most interesting, innovative, or unexpected ways that you have seen Observe/lakehouse technologies used for observability? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Observe? When is Observe/lakehouse technologies the wrong choice? What do you have planned for the future of Observe? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links Observe Inc. Lakehouse Architecture Splunk Observability RSyslog GlusterFS Dremel Drill BigQuery Snowflake SIGMOD Paper Prometheus Datadog NewRelic AppDynamics DynaTrace Loki Cortex Mimir Tempo Cardinality FluentBit FluentD OpenTelemetry OTLP == OpenTelemetry Line Protocol Kafka VPC Flow Logs Read Amplification Lance Iceberg Hudi PromQL The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    • Transcript
    • Chapters
  • #496
    January 12 · 56 min

    Semantic Operators Meet Dataframes: Building Context for Agents with FENIC

    Summary In this episode Kostas Pardalis talks about Fenic - an open-source, PySpark-inspired dataframe engine designed to bring LLM-powered semantics into reliable data engineering workflows. Kostas shares why today’s data infrastructure assumptions (BI-first, expert-operated, CPU-bound) fall short for AI-era tasks that are increasingly inference- and IO-bound. He explores how Fenic introduces semantic operators (e.g., semantic filter, extract, join) as first-class citizens in the logical plan so the optimizer can reason about inference, costs, and constraints. This enables developers to turn unstructured data into explicit schemas, compose transformations lazily, and offload LLM work safely and efficiently. He digs into Fenic’s architecture (lazy dataframe API, logical/physical plans, Polars execution, DuckDB/Arrow SQL path), how it exposes tools via MCP for agent integration, and where it fits in context engineering as a companion for memory/state management in agentic systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud. If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests. Your host is Tobias Macey and today I'm interviewing Kostas Pardalis about Fenic, an opinionated, PySpark-inspired DataFrame framework for building AI and agentic applications Interview Introduction How did you get involved in the area of data management? Can you describe what Fenic is and the story behind it? What are the core problems that you are trying to address with Fenic? Dataframes have become a popular interface for doing chained transformations on structured data. What are the benefits of using that paradigm for LLM use-cases? Can you describe the architecture and implementation of Fenic? How have the design and scope of the project changed since you first started working on it? You position Fenic as a means of bringing reliability to LLM-powered transformations. What are some of the anti-patterns that teams should be aware of when getting started with Fenic? What are some of the most common first steps that teams take when integrating Fenic into their pipelines or applications? What are some of the ways that teams should be thinking about using Fenic and semantic operations for data pipelines and transformations? How does Fenic help with context engineering for agentic use cases? What are some examples of toolchains/workflows that could be replaced with Fenic? How does Fenic integrate with the broader ecosystem of data and AI frameworks? (e.g. Polars, Arrow, Qdrant, LangChan/Pydantic AI) What are the most interesting, innovative, or unexpected ways that you have seen Fenic used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Fenic? When is Fenic the wrong choice? What do you have planned for the future of Fenic? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links Fenic RudderStack Podcast Episode Trino Starburst Trino Project Tardigrade Typedef AI dbt PySpark UDF == User-Defined Function LOTUS Pandas Polars Relational Algebra Arrow DuckDB Markdown Pydantic AI AI Engineering Podcast Episode LangChain Ray Dask The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    • Transcript
    • Chapters
  • #495
    January 5 · 49 min

    Beyond Dashboards: How Data Teams Earn a Seat at the Table

    Summary In this episode Goutham Budati about his Data–Perspective–Action framework and how it empowers data teams to become true business partners. Gautham traces his path from automating Excel reports to leading high‑impact data organizations, then breaks down why technical excellence alone isn’t enough: teams must pair reliable data systems with deliberate storytelling, clear problem framing, and concrete action plans. He digs into tactics for moving from reactive ticket-taking to proactive influence — weekly one‑page narratives, design-first discovery, sampling stakeholders for real pain points, and treating dashboards as living roadmaps. He also explores how to right-size technical scope, preserve trust in core metrics, organize teams as “build” and “storytelling” duos, and translate business macros and micros into resilient system designs. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud. You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build Your host is Tobias Macey and today I'm interviewing Goutham Budati about his data-perspective-action framework for empowering data teams to be more influential in the business Interview Introduction How did you get involved in the area of data management? Can you describe what the Data-Perspective-Action framework is and the story behind it? What does it look like when someone operates at each of those three levels? How does that change the day-to-day work of an individual contributor? Why does technically excellent data work sometimes fail to drive decisions? How do you identify whether a data system or pipeline is actually creating value versus just existing? What's the moment when you realized that building reliable systems wasn't the same as enabling better decisions? Better decisions still need to be powered by reliable systems. How do you manage the tension of focusing on up-time against focusing on impact? What does it mean to add "Perspective" to data? How is that different from analysis or insights? How do you know when you're overwhelming stakeholders versus giving them what they need? What changes when you start designing systems to surface signal rather than just providing comprehensive data? How do you learn what business context matters for turning data into something actionable? What does it mean to design for Action from day one? How does that change what you build? How do you get stakeholders to actually act on data instead of just consuming it? Walk us through how you structure collaboration with business partners when you're trying to drive decisions, not just inform them. What's the relationship between iteration and trust when you're building data products? What does the transition from order-taker to strategic partner actually look like? What has to change? How do you position data work as driving the business rather than supporting it? Why does storytelling matter for data professionals? What role does it play that technical communication doesn't cover? What organizational structures or team setups help data people gain influence? Tell us about a time when you built something technically sound that failed to create impact. What did you learn? What are the common patterns in dysfunctional data organizations? What causes the breakdown? How do you rebuild credibility when you inherit a data function that's lost trust with the business? What's the relationship between technical excellence and stakeholder trust? Can you have one without the other? When is this framework the wrong lens? What situations call for a different approach? How do you balance the demand for technical depth with the need to develop business and communication skills? How should data professionals position themselves as AI and ML tools become more accessible? What shifts do you see coming in how businesses think about data work? How is your thinking about data impact evolving? For someone who recognizes they're focused purely on the technical work and wants to expand their impact—where should they start? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    • Transcript
    • Chapters
Showing 21–23 of 23 episodes