Skip to content
Sign in
Artwork for Data Engineering Central Podcast
TechnologyNewsTech News

Data Engineering Central Podcast

Data Engineering in Real Life

Long Live the Data Engineer. No holds barred. Talking about Data Engineering news, topics, and general mayhem.

dataengineeringcentral.substack.com
Play
Open in
  • 24 episodes
  • weekly
  • Avg 52 min
  • English

Episodes 24

NewestOldestLongest
All4Unplayed4In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • March 24 · 46 min

    Spark, AI, and the Future of Data Engineering with Daniel Aronovich

    In this episode of Data Engineering Central, I sit down with the founder of DataFlint, Daniel Aronovich, to talk about the realities of working with Apache Spark, distributed data systems, and the future of data engineering. We start with his early journey into tech—how he first discovered large-scale data systems and the lessons he learned from working with real-world Spark workloads. * The conversation then turns toward the future of data engineering, particularly the growing role of AI in software development and data infrastructure. We discuss why generic AI coding assistants often struggle with complex distributed systems, whether AI will eventually be able to automatically optimize data pipelines, and how the role of the data engineer may evolve in the coming years. We covered a lot of career advice for new and upcoming data professionals. We also discuss the origin of DataFlint, a tool designed to help engineers better understand and optimize Spark workloads by analyzing execution plans, logs, and runtime context. If you work with Spark, large-scale data pipelines, or modern data platforms, this conversation will give you a deeper look into how the data engineering landscape is evolving. Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

  • March 18 · 1 hr

    DuckDB, AI, and the Future of Data Engineering

    In this episode, I sit down with Matt Martin, Staff Engineer, data architect, ETL practitioner, and author of a new book on DuckDB coming soon, to talk about the past, present, and future of data engineering. Matt has spent decades building and architecting data platforms across technologies such as SQL Server, Oracle, DB2, Hadoop, Redshift, and BigQuery, and now focuses on modern tools such as DuckDB and single-node analytics. We discuss how the data industry has evolved, what actually makes data platforms succeed, and where tools like DuckDB, Polars, Databricks, and Snowflake fit into the future of analytics. We also dive into the impact of AI on coding and data engineering, and whether distributed compute clusters will remain dominant — or if more workloads will move toward high-performance single-node systems. Topics Covered * Matt’s early career and journey into data engineering * The evolution of data warehousing and ETL frameworks * Traditional enterprise data systems vs modern cloud platforms * DuckDB and the rise of single-node analytics * Polars vs DuckDB: where each tool shines * Databricks vs Snowflake * AI-assisted coding and its impact on engineers * The current data engineering job market * Lessons learned from decades of building data systems * Writing a book on DuckDB This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

  • March 11 · 1 hr 6 min

    What Decades in Software Engineering Teaches You

    In this episode of Data Engineering Central, I sit down with a veteran Software Engineer John Crickett; with decades of experience in the industry to unpack what really matters in building a long and successful engineering career. We talk about how he first got into software, the early jobs and tools that shaped his thinking, and the massive technology shifts he’s witnessed across decades of engineering—from early stacks and tools to today’s AI-assisted workflows. * We also dive into the difference between coding and real-world software engineering, what separates junior, senior, and principal engineers, and why many developers misunderstand what it takes to grow in this field. * We discuss leadership vs individual contributor paths, the origin of his Coding Challenges platform, why algorithm puzzles dominate developer culture, and what actually makes engineers improve quickly. Finally, we tackle the big question everyone is asking right now: how AI is reshaping software engineering, and what skills will matter most over the next decade. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

  • March 3 · 47 min

    Data Engineering, AI, and Career Growth

    In this episode of the Data Engineering Central Podcast, I sit down with Yuki (Yuki Kakegawa) to talk about his journey into tech, the tools and platforms he’s worked with, and where he thinks data engineering and AI are headed next. We cover: • How Yuki got into tech• Early career lessons and pivots• Tools and technologies he’s worked with over the years• How data engineering has evolved• The impact of AI on software development• What engineers should focus on right now• Advice for those building their careers in data Yuki shares practical insights on navigating the industry, staying adaptable, and thinking long-term about your technical growth. If you’re a data engineer, aspiring engineer, or just interested in where AI and modern software are going, this one’s for you. Yuki writes on … LinkedIn - https://www.linkedin.com/in/yukikakegawa/ https://yukikakegawa.me/#blog Thanks for reading Data Engineering Central! This post is public so feel free to share it. 🔔 Subscribe for more interviews with leaders in data engineering, AI, and modern data platforms. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

Showing 21–24 of 24 episodes