Data Engineering Podcast


This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Support the show!

Rewind 10 seconds
1X
Skip 30 seconds ahead
0:00/0:00

Listen in your favorite app:



More options

Here are shows you might like

See show recommendations
AI Engineering Podcast
Tobias Macey
The Python Podcast.__init__
Tobias Macey

445 Episodes

Feldera: Bridging Batch and Streaming with Incremental Computation - E444

Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The discussion covers the concept of incremental computation, the origins of Feldera, and its unique ability to handle both streaming and batch…

Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their…

04 November 2024 | 00:47:36


Accelerate Migration Of Your Data Warehouse with Datafold's AI Powered Migration Agent - E443

Summary Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the challenges and innovations in data migrations. Gleb shares his experiences building and scaling data platforms at companies like Autodesk and Lyft, and how these experiences inspired the creation of DataFold to address data quality issues across teams. He…

Summary Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the…

27 October 2024 | 00:48:50


Bring Vector Search And Storage To The Data Lake With Lance - E442

Summary The rapid growth of generative AI applications has prompted a surge of investment in vector databases. While there are numerous engines available now, Lance is designed to integrate with data lake and lakehouse architectures. In this episode Weston Pace explains the inner workings of the Lance format for table definitions and file storage,…

Summary The rapid growth of generative AI applications has prompted a surge of investment in vector…

20 October 2024 | 00:58:01


The Role of Python in Shaping the Future of Data Platforms with DLT - E441

Summary In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph, co-founders of DLT Hub, delve into the principles guiding DLT's development, emphasizing its role as a library rather than a platform, and its integration with lakehouse architectures and AI application frameworks. The episode explores the impact of the…

Summary In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph,…

13 October 2024 | 00:54:08


Build Your Data Transformations Faster And Safer With SDF - E440

Summary In this episode of the Data Engineering Podcast Lukas Schulte, co-founder and CEO of SDF, explores the development and capabilities of this fast and expressive SQL transformation tool. From its origins as a solution for addressing data privacy, governance, and quality concerns in modern data management, to its unique features like static…

Summary In this episode of the Data Engineering Podcast Lukas Schulte, co-founder and CEO of SDF,…

06 October 2024 | 00:42:36


Scaling Airbyte: Challenges and Milestones on the Road to 1.0 - E439

Summary Airbyte is one of the most prominent platforms for data movement. Over the past 4 years they have invested heavily in solutions for scaling the self-hosted and cloud operations, as well as the quality and stability of their connectors. As a result of that hard work, they have declared their commitment to the future of the platform with a…

Summary Airbyte is one of the most prominent platforms for data movement. Over the past 4 years they…

23 September 2024 | 00:57:11


Enhancing Data Accessibility and Governance with Gravitino - E438

Summary As data architectures become more elaborate and the number of applications of data increases, it becomes increasingly challenging to locate and access the underlying data. Gravitino was created to provide a single interface to locate and query your data. In this episode Junping Du explains how Gravitino works, the capabilities that it…

Summary As data architectures become more elaborate and the number of applications of data…

01 September 2024 | 00:38:41


The Evolution of DataOps: Insights from DataKitchen's CEO - E437

Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to simplify the lives of data engineers. Chris explains the challenges faced by data engineers, such as constant system failures, the need for rapid changes, and high customer demands. Chris delves…

Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg,…

04 August 2024 | 00:53:30


Achieving Data Reliability: The Role of Data Contracts in Modern Data Management - E436

Summary Data contracts are both an enforcement mechanism for data quality, and a promise to downstream consumers. In this episode Tom Baeyens returns to discuss the purpose and scope of data contracts, emphasizing their importance in achieving reliable analytical data and preventing issues before they arise. He explains how data contracts can be…

Summary Data contracts are both an enforcement mechanism for data quality, and a promise to…

28 July 2024 | 00:49:26


How Generative AI Is Impacting Data Engineering Teams - E435

Summary Generative AI has rapidly gained adoption for numerous use cases. To support those applications, organizational data platforms need to add new features and data teams have increased responsibility. In this episode Lior Gavish, co-founder of Monte Carlo, discusses the various ways that data teams are evolving to support AI powered features…

Summary Generative AI has rapidly gained adoption for numerous use cases. To support those…

21 July 2024 | 00:54:45