Data Engineering Podcast

Share on social media:

Support Data Engineering Podcast

[00:00:13] Unknown:

Hello, and welcome to the Data Engineering Podcast, the show about modern data management.   Have you ever woken up to a crisis because a number on a dashboard is broken and no 1 knows why? Or sent out frustrating Slack messages trying to find the right dataset? Or tried to understand   They started building Outland as an internal tool for themselves. They started building Outland as an internal tool for themselves.   Outland is a collaborative workspace for data driven teams, like GitHub for engineering or Figma for design teams.   By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets and code, Atlan enables teams to create a single source of truth for all of their data assets   and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker, and more.  

Go to dataengineeringpodcast.com/outland   today. That's a t l a n, and sign up for a free trial.   If you're a data engineering podcast listener, you get credits worth   $3, 000 on an annual subscription.   Check out our friends   over at Linode.   With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Pacaderm, and Dagster.   With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.   Go to data engineering podcast.com/linode   today. That's   l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host is Tobias Macy. And today, I'm interviewing Kishore Gopalakrishna   and Shang Fu about Apache Pinot and its applications for powering user facing analytics. So, Kishore, can you start by introducing yourself?  



[00:02:08] Unknown:

Definitely. Thank you so much, Tobias, for inviting us here. Super excited to be on this show. My name is Kishore Gopalakrishna, and I'm the CEO and cofounder of Startree. I'm also the cocreator of Apache Pinot.   Before Strattery, I was at LinkedIn for 8 years. I built a multiple a lot of distributed systems such as Espresso, ThirdEye, Apache Helix, which is a cluster management framework,   which help build a lot of distributed systems.   Also involved in a lot of Apache projects like Goblin, s 4,  

[00:02:37] Unknown:

Dataverse, and many others. And, Sean, how about yourself? Hi. This is Sean Fu, and, super excited   to join this.   So for my background, I'm currently a cofounder at StarTree,   and I'm also a cocreator of a project.   Before StarTree,   I was a architecture at Uber's streaming data team. So build this onto a streaming solution   for data infrastructure.  

[00:03:02] Unknown:

And before Uber, I was at same thing to build Apache Pino, basically. Going back to you, Kishore, do you remember how you first got involved in the area of working on data? Yeah. It actually goes  

[00:03:12] Unknown:

a long way back, even before Hadoop became a thing. Back in Yahoo, I was working in this forecasting team. We needed data to do the forecasting as usual. But the funny thing that happened at that time was we were to fill up the form,   and then we were supposed to wait for a month before the data shows up because all this data was in Oracle,   and we never had access to this. So this was a time when data democratization was not unit so when it was more like a dream for us. That's when Hadoop came in at Yahoo and everything changed. And I was so   excited with this change, and I thought I'm more   directly diving to that ecosystem.  

That's when I started building   a lot of distributed systems. Started with Apache s 4, which is a streaming system.   I also worked in ad indexing and, lot   of data related work that got me excited.   What really changed something for me was   LinkedIn being a huge proponent of open source. So I was able to join LinkedIn very early on in their journey of going from single monolith of Oracle   to building all these distributed systems such as Espresso, Kafka,   etcetera.   The first system that we built where we built is a NoSQL   distributed systems database. And as part of that, I also got ability to build opportunity to build Apache Helix, which is a cluster management framework.  

So from then, there was no turning around for me. It was all data, data, data. And I've been super excited and super fortunate to get this opportunity at LinkedIn,   build a bunch of systems.   So overall for me, just that ability to have access to this data and get all these insights   is super exciting and also super challenging, and I love all the challenges there. So that's kind of when I started, but I haven't really gotten out of the data management yet.  

[00:05:04] Unknown:

And, Sean, do you remember how you first started working with data? Yeah. So for me, my story was pretty simple. I was just a graduate from the school, and   I joined the LinkedIn. So my first project, though, was actually the Pinot project. It was, like, in the ads team and then building, like, those ads reporting things around the p and o and for serving as customers.   This project was, like, very challenging. And later on, we got a lot more use cases   on Pino, and, all those things are, like, super, super, super challenging. And you learned a lot from the data and from working with the engineers. Those things are pretty fun, and I actually continue my career on the data   world for, like, past 10 years. And after I left the LinkedIn and joined the Uber, at Uber, we actually build the entire streaming   data infrastructure there. We first stabilized   the Kafka infrastructure at   Uber, took a lot of time to, like, get things   being successfully   produced to Kafka   for, like,   half a year, and then we build a streaming processing,   infrastructure using, like, SAMSA and, Flink.  

After that,   the story   comes to, like, the real time analytics infrastructure. So we have Pinot being adopted at Uber, and we   also build a lot of use cases on top of Pinel.   Later on, we use   Presto to provide   better   kind of full SQL support and the joins with different   data sources across, like, the entire Uber.   And, also, I've been involved to, like, the data visualization   layer, like Supersat,   and also, like, the data management   layer, like, metadata stores kind of things.  

[00:06:50] Unknown:

In terms of the Pino project, I'm wondering if you can describe a bit about what it is and some of the story behind   how it came to be and why it was necessary and why you've decided that this is something that you wanted to spend your time and focus on in the form of StarTree.  

[00:07:06] Unknown:

So just just a brief intro in in terms of what is Apache Pinot. So it's a real time OLAP data store specifically designed for   interactive user facing applications. Right? I mean, that's a lot of words right there in that statement.   The key aspect of these applications, user facing applications, is that they demand   very, very low latency, even at high throughput. Right? Another way that people internally   at LinkedIn used to describe this describe, you know, as was queryable Kafka. So you can pretty much point Pinot to any stream, and then you can start querying. That's the right way you can think of what Pinot is. And   in terms of how this happened at LinkedIn,   it was actually a very exciting story. There are 2 parts to that. The first part is, like, when   what do you call as p no 1 dot of in fact, I was not even part of that project. It was really Sean who built the first version of p no that's   which was really built on top of a Lucene. Right? And then we had an internal framework called Bobo and Sensei, which was also built on Lucene, and that's what we use to power some of our initial versions of who viewed my profile. If you're familiar with that product on LinkedIn where you go to your profile, who viewed my profile page, and you get to know everyone that viewed your profile and where did they come from, which company,   what is the skill set. And this was a very important application for LinkedIn. Because what happened in 2011, 2012   is many would come to LinkedIn only for when they're changing their job. It was not really the vibrant   social network that it is today. Right? And that was super important for LinkedIn to change that paradigm and then where the engagement had to increase drastically. So what this app, who in my profile,   allowed LinkedIn to do was   a lot of people would come to this page, know who are actually looking at their profile, and then lot of connections   and invites would actually originate from this page. So it's a very, very important thing for, for LinkedIn that changed the whole dynamics of how many people who came to LinkedIn.  

And the professional network that LinkedIn is today wouldn't be possible without this app. Yeah. So the engagement metric had to increase a lot for LinkedIn. And this is when COVID my profile was built as 1 of the first applications to change that. And we were, like, super surprised with the success of this application. At any   given time, there were, like, some thousands of people looking at this application. Right? And the first version that we built   was really not scaling. I remember when I joined the the p note team, we had almost 1, 000 notes serving just who will make profile. And it was crazy to operate this because it was not built to operate at that scale,   and we were expecting less than 100 millisecond latency. So we used to break it up into vertical charts.  

We had, like, multiple clusters to just serve 1 application.   It was a huge challenge, and it was a nightmare, to be honest, for all of us at LinkedIn. In fact, we were internally joking that we should just kill this product, go back to batch mode, and just show static graphs to the end users. But the metrics really showed a told a very different story, right, in terms of the engagements, the metrics, and the number of people actually coming back to LinkedIn was really, really high.   And that got us that got the product management super excited, and they said, hey. We need to build a new system that can actually handle the scale. And that's where   we built Pino from scratch. And we learned a lot from these issues that we had seen with the Pino 1.0.  

Once we rebuild it, the 1, 000 nodes came down to, like, 75 nodes. And by when we'd actually   along with that change, the traffic had grown to 5 x. So overall, we saw almost, like, 45 x improvement in efficiency, and   that was really the game changer for LinkedIn as well. So what happened after that was   every entity on LinkedIn, call it, a post that you put it on LinkedIn   or a jobs or ad or even a company profile. Any entity had events associated with that, and those events would flow into Kafka. And Pinot would point at those events, and there would be an app built on top of it. So there are tons of apps, like, almost 70 to 100 apps on LinkedIn that you see today, and they're all being powered by Pinot in the back end. So any number that you see, any chart, most likely coming from Pinot. And today, if I remember the numbers correctly, it's serving around 200, 000 queries per second. This is really, really great achievement for us in terms of what we were able to do with Pinot and how we change the whole way of thinking about data at LinkedIn. And pretty much it became a de facto thing. Any page, any entity that LinkedIn created,   analytics came as along with it by default. So that's kind of how we started and where we are now today with Pino at LinkedIn.  



[00:11:55] Unknown:

You mentioned that the primary focus of Pino has been around these   user facing analytics,   for instance, with LinkedIn being able to see, you know, who viewed your profile graph.   And I'm curious,   what are some of the   alternative approaches that people may have used for being able to power these user facing analytical use cases and some of the capabilities that Pino provides that   make it stand out as a method for being able to power these experiences?  

[00:12:27] Unknown:

There are multiple   technologies that were used before Pinot. Right? I think 1 of the things that really changed at LinkedIn was we were providing these analytics to   100 of millions of members and being able to also keep the latency very, very low. Because   as you know, users,   they don't have the same patience as the external users. They don't have the same patience as the internal analyst or the prep managers within the company. So there was a huge difference in terms of the expectations from an external   user facing versus internal use users.  

And   in the classic alternatives where, as I said, elastic search was something and we, in fact, use that. There are also key value stores that people use it where they precompute everything,   but that has the curse of dimensionality.   Right? As you add more dimensions,   the storage explodes and they are not able and there is also   lot of inflexibility in the query patterns when you use a p value store. So those those were some of the things. I think the key change,   even though it was user facing analytics,   what really   changed at LinkedIn for us was this notion of   building data products. Right? And if you look at, existing tools,   they always   existing technologies, the way people consume the insights from them were through SQL editors or internal dashboards. Right? And that never   will   generate the level of query that you can generate through an app. Like, if you look at   all the apps on LinkedIn or on Uber,   no 1 knows that they're actually running a query behind the scene. So it is really a simple click is running a query behind the scenes. So our motto and our philosophy at LinkedIn was really all these reports, custom reports that we were generating.  

Can we actually   convert them into data products? I'll give you an example here. For   1 of the products that LinkedIn had was talent insights. Right? Which is   simply providing insights on how many people moved from 1 company to another company. So you can think of it from Amazon.   They can think, hey. There are so many people moving out of Amazon. Where did they go to? Did they go to Google? Did did they go to Facebook?   Which area are we losing people?   Which skill sets we need to improve on? So there are a lot of these insights. And before Pinot, the way this was delivered to the users was someone internally would generate a report every month and then send it out. Right? And then there are only few questions that these reports could answer. And every time they needed some change, again, there would be an interaction between   the Amazon folks and the LinkedIn folks, and then someone had to write a Spark job or a Hadoop job to generate these reports and send it back. Now fast forward few years, now you have a Talent Insights as a product on LinkedIn, right, where the entire data is just every profile changes just simply fed into Pinot. And the customers can now come in, and then they can ask the questions. And then right there, you are able to slice and dice and then generate insights that you want. And there is no middle person in involved in it. Right? It's a huge change in the way things are seen both from LinkedIn's perspective and also the amount of power given to the consumers.  

So that's the whole idea in terms of the primary use cases that motivated us to build, you know, and we capitalized   heavily on that. And as it's really amazing to see all these different products using Pino as the back end. So as far as Pino itself, it seems to sit at kind of a cross section of technologies  

[00:15:59] Unknown:

where, in 1 sense, it seems to fit in the OLAP use case that products such   as ClickHouse or numerous other database engines   work   for as well as sitting in this embedded   end user analytics use case, such as what cube JS or good data are trying to power.   And I'm wondering if you can just talk through some of the   sort of juxtapositions   of Pinot as compared to some of these other technologies and   some of the systems that Pino is  

[00:16:30] Unknown:

maybe designed to work in conjunction with and is assuming in order to be able to get the best possible use out of it? Sure. I think I'll let Sean actually talk a little bit about Uber itself right now. Like, how, you know, evolved in Uber, especially   seeing Uber is actually a 3 sided marketplace.   So there are a lot of cool things that happen, and then we can probably get into the   different trade offs and comparing with the roll up systems as well. So I just want to talk about some, like, Uber stories.  

[00:16:58] Unknown:

If you think of Uber, it's actually a marketplace. If you take,   like, Uber Eats as an example,   they are, like, service providers.   Those are, like, the restaurants, and then they are   consumers,   which are the riders   to deliver your order, and there are also eaters who will order the phone app, whatever things they want to eat. Right? There will be a third part to which are the internal   operations.   They actually support all the business, and they can give coupons to the eters if they don't want to other. So those kind of things are, like, form, like, 1 marketplace for Uber Eats.   For Uber's usage at   Pino,   we first start with this kind of city   life dashboard   for Uber Eats internal operations.  

So this was actually a dashboard that's shown   every TV for, like, Uber Eats soft Office.   So all these operation people, they come actually look at this dashboard   to see how many live orders in my city, in my area,   and what are the long waiting orders   that they need to take actions on those things. Right? They probably need to call the riders   to say, like, are you in a traffic jam, or if you are waiting in the restaurant. So at the very beginning of the Uber is business. So this kind of dashboard has been, like, helped the business grows a lot to, like, greatly improve the customer satisfaction and improve the   cost efficiency   of the whole end to end pipeline   workflow.  

So   we actually expand   keynote's usage   to more like user facing analytics.   If you think of, like, service provider, like restaurants, they are also, like, 1 part   of Uber Eats business.   We actually want to provide   the more insights on, like, how your business is doing. Right? So   hence, we built something called restaurant manager,   which is, like, the   analytics   dashboard for   the Uber Eats restaurant.   The restaurant's owners actually look at those dashboard   and find out, oh, this is my revenue so far, and this is my projected revenue   for today.  

And here are all my upcoming orders.   Also, for existing,   like, orders, what are the reviews   for every dishes. Right? So they can get all these kind of insights   from the dashboard.   And that's actually greatly   improved,   restaurant engagement, because they can actually   find out how their business is doing, and they can actually get the real time feedback   from, like, how their business is doing. Because all those things, usually, they won't be able to get unless you   ask specifically to every customers in your restaurant. Right? So this kind of experience actually improved   in our restaurant   engagement a lot. And after that,   we started some more use cases on the consumer side on the eaters.  

So if you think of Uber Eats, right, so the initial   Uber is app only shows you the restaurants nearby with some kind of heuristics   to   suggest, like, those are the restaurants you like. You'll probably want to order something from there.   Figuring out that people may just order from those   restaurants they like, and they may not   pick those new restaurants, new joining restaurant. Right? So we actually want to improve   those kind of engagement   for new restaurants   and also   let eaters to actually exploring   more kind of restaurants.   So here, we actually   use Pinel   to build a feature called the orders   near you   to power the Uber Eats home feed.  

So   this kind of feature will actually show what are the current orders   nearby   you, and it's basically   whatever things your neighbors are actually ordering.   This kind of a social experience,   very unique for   the eaters.   This kind of thing also inspiring the food and the restaurant discovery,   and they're showing, like, what your neighbors are ordering. This feature is actually being   quite popular when you are looking at the dishes your neighbors are actually ordering. So since this feature is actually part of the Uber Eats home feed, If you think of, like, how many   users are actually being looking at their app at the lunch or dinner time, has to be, like, a super fast and personalized and scalable   system in order to support as a backbone for this kind of system. And, of course, all the geospatial   features index,   those things, so we need to build for that.  

Soon, a lot of usage has been expanded to broad areas at Uber, so Uber later   on allows   ads system   allows, like, ads to be, like, placed at Uber app, so you can't so actually in Uber, we build the ad system. You can actually do ads bidding,   and those kind of things at the Uber app. For this kind of system, we actually need to provide a super accurate performance and the business metrics.   Don't count down any kind of impression   or, like, click events, or you don't want to miss any of this kind of events. Those things will actually   make the ads   customers to be like   having bad experience about ads performance, and you also don't want to overcharge   or mischarge   your ads customers.  

So that we actually build the upsell feature in Pinot along with exact ones, Kafka and the Flink. We actually provide the exact ones guarantee for the whole n to n streaming data infrastructure at Uber. That was actually   pretty impressive and very hard to achieve across the industry. Of course, at Uber, there are, like, so many data scientists, and they actually   need kind of the full SQL   access to your data, and hence we   are building the Presto Pino connector,   which   really connect Presto to Pino. And data science team,   actually, they use full SQL to explore all the data in Pino,   along with, like, joining with their, like, hive table   and do more advanced   algorithms   and verifications,   a lot of things, build models as well. So all those things are, like, being build across.  



[00:23:30] Unknown:

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.   Their SDKs make event streaming from any app or website easy, and their state of the art reverse ETL pipelines enable you to send enriched data to any cloud tool.   Sign up for free or just get the free t shirt for being a listener of the data engineering podcast atdataengineeringpodcast.com/rudder.   1 of the interesting   complexities   of the   particular   problem that Pino is trying to solve for is   because you are exposing this to end users,   it needs to be able to support sort of high concurrency. And I'm wondering, what are some of the ways that that   influences the overall architecture of the system to be able to manage that type of workload as compared to what some of these other OLAP engines are optimized for?  



[00:24:33] Unknown:

Yes. So I think 1 of the design principles that we had very early on, we looked at all these other OLAP systems and these databases.   So the typical way most databases   thought about this solving these problems was, hey. We have to   scan all these data. We have to do x numb amount of work. How can we do this work faster? You had Cindy. You had columnar store. Those are all the classic things that people adopted.   But we took a very different approach because we knew that we could only push them as fast as the SSDs or the CPUs   that were all already available.  

So our philosophy   was to eliminate that work instead of trying to do the work faster. Right? That's where we build a ton of indexes in Pinot. So we have   wide range of, indexes like sorted index. We have the bitmap index, which is the inverted index concept as well. We have the range index.   We have JSON index. We have text index. We have geospatial.   So there are a lot of different indexing techniques that we built, which   eliminated the work that needs to be done. So we would use this index, get to the right   location. So we even added a lot of enhancement in terms of rearranging the data. So even though we ingest it while we actually write the segments, we would rearrange the data so that we get locality. So we could solve   the query with 1 seek instead of having 100 seeks. So we did all these low level optimizations and a lot of these happen automatically.  

And then last but not the least, we went beyond just the filtering and the indexing. Right? And a lot of them also took a lot of time in terms of aggregation because a lot of these user facing applications needed aggregation queries. So we built powerful indexes such as StarTree index. In fact, that's 1 of the reasons why we named the company as well as start re because it's a unique index that's not available in any of these other systems.   It's something that allows you to get even scanning billions of rows. With starter index, you can actually get that in millisecond level latency.   Lot of cool things that have happened in in the indexing area within Pino,   and that kind of what makes it   very, very different from this other OLED engines.  

And, also, another principle here is that you have all these different   features within, you know, and you can use   things when you want them. Right? You don't have to use all of these at once. But the idea is that you have the knobs that can take you all the way from   tens of QPS to thousands of QPS. Right? And that's the beauty of the system. It's 1 system that can get you started on a project   and get get can take you all the way if your project becomes very popular. So we have a lot of products at LinkedIn which started very small, but then as soon as they become popular, now you need to move that to into another system. So that's introduced a lot of operational overhead. Now you need to have multiple systems within the company.   1 for internal, 1 for external.  

That's kind of what we were able to prove at LinkedIn and Uber. And now you see that that has   spread across all these other companies like Walmart, Stripe, Target,   Cisco. All these companies have started doing the same thing. So it becomes 1 platform, and you can solve both your internal and external use cases.  

[00:27:46] Unknown:

For those types of use cases, I'm curious   how   you have seen these businesses approach the question of which database engine and query engine they should be using to power these analytics, whether it's ClickHouse or Snowflake or,   you know, Postgres and then some of the different additional systems that they're going to need to be able to complement and support and integrate with   the ultimate technology that they decide as sort of that core building block.  

[00:28:15] Unknown:

It's interesting to see the journey from the company's point of view. Right?   How do they look at the evolution of their analytics? So most companies, if you look at their journey,   they start off with the business intelligence first. They then get into the operational analytics.   So that's their evolution. And most of these tools, like a data warehouse or the whole app tools, the prior ones, we're able to address these needs. But now companies are seeing the value of providing these insights to their users and their customers,   and that's when they get stuck. They hit a wall, and they are not able to take these existing systems.  

So the way we see it is you need all these different tools. Right? It's not just 1 tool. It's data warehouse is not just going to solve all the problems, but it's about how do you actually integrate these systems easily. If you look at data warehouse, that has a very different set of needs and differ different persona. So you have this business analyst   who write, like, 100 will join and then lot of complex queries that they keep changing and they create now. So for them, data warehouse is a perfect tool where they can actually create dashboards on top of it and then share the results. But when you look at app based consumption   where the end user is not technically,   well versed in terms of writing a SQL SQL query, for them, it is through apps. So they need to simply be able to click. They need to be context aware. They need to know what tool they're using and how to use that UX. For them, it's a very different experience. But then they end up generating a lot more queries,   but the query patterns are kind of well known in advance because it's the app that's generating the query. So different purposes, different engines, and they're all made for different purpose. Just to give you an example, Snowflake,   across all their customers, is really running 1 query per second. Right? Whereas LinkedIn, you know, just at LinkedIn is running at 200, 000 queries. And the main reason for this is with the access pattern. So 1 on 1 side, it's   through SQL editors   and dashboards, which doesn't really generate a lot of QPS. And on the other side, it is apps which end up generating tons of QPS behind the So the workload characteristics is very different.  

The freshness requirement is very different. Also, the latency requirement is very different. So depending on these, you pick 1 system versus the other. But in the end, you need to have   ability to solve all the different use cases.  

[00:30:37] Unknown:

Yep. I can add something related to Uber.   So at Uber,   I can take the example of the others near you feature. Actually, when we build these applications,   the initial version   of this kind of implementation was actually a sender. Cassandra is actually a data back end, like, the permanent store for all the place, the Uber's orders, and all those, like, restaurants.   So at the first iteration,   we actually implemented by, like, first, retrieve   all the nearby   restaurants   for given user's location. Right? You have the latitude and the longitude.   And once you have all the nearby restaurants,   you actually fetch   all the active orders   from another service called the order gateway service.  

Once you find out all those orders,   then you will try to rank all those results based on the residency and, merge those things and push it to the home fix. However, for this kind of application,   it actually translate   1 users   into, like, 100   Cassandra lookups.   Right? So this kind of   translation will actually   greatly   increase   the workload on Cassandra.   So based on the estimation   for the Cassandra team in order to support this kind of use cases, it'll need to 6 times   Cassandra   store because all those kind of grid workload. They didn't need, like, more applications, the failure to store in order to start that. And once we move those things   into PNO right? So if you think of, like, all the all their status change, those kind of things going to PNO, Pinot is also optimized   for   readability.  

Right? And with   the geospatial indexing support,   we actually convert this kind of 1   user's page review   into just 1 query inside Pinot. Also, with the geospatial index, the core latency was actually   dropped from in Cassandra, it was, like, a couple of seconds. You actually finish this kind of 100 lookup round trip. The p 99 was less than 50 milliseconds for just the 1 core, which means this also greatly improves   the user experience in terms of when you open the Uber Race app, you don't need to wait the app logo to spinning multiple seconds in order to see everything you come. Immediately see what are the others nearby you. Right? So all those kind of user experience improvement are super critical for the product team as well. I just mentioned those are cost effective for Cassandra. You need so many   machines. Right? And for pinot side,   the this kind of use cases are just being supported with 10 regular machines.  

And that's also, like, very impressive achievement from the data team side to actually being able to provide this kind of support with a very small, like, hardware footprint.  

[00:33:35] Unknown:

Digging more into the architecture of P and O itself, can you talk through the different   components of the platform and some of the ways that you proved out some of those early design decisions and   some of the elements that were key to being able to support these low latency and high concurrency queries?  

[00:33:52] Unknown:

So from the Pinot side, the key design principle is we want to provide a predictable core latency and high availability and the scalability system for our users.   So for our external clients, we need the the always on system.   So you don't have any luxury for downtime by saying, okay. We have a database maintenance. Right? So we we actually need to make sure all the replications are there. There's no single point of value of the system.   So for all the layers inside the panel, we are using Apache Helix and the zookeeper for the cluster management and handle all those node value, all the node upgrade   seamlessly.  

Second part is, like, for a p node to be, like,   super operational   friendly.   So at the LinkedIn, there were only 3 SREs there. At Uber,   when we build a PNO from the very beginning and then till now, the Uber's PNO team size hasn't been grow significantly.   I think we only have, like, 3 to 5 people at Uber to support the entire PIN node cluster, and the PIN node cluster size has been grew from, initially, it's only 3 nodes to guarantee your applications.   And now in 1 data center, there are more than 500 nodes   to   serve all burst use cases. And the worst to mention that this kind of pinhole has been designed as, like, a highly configurable.  

So we actually want 1 system and 1 cluster for multiple use cases. Right? So if you think of, like, at Uber, this 1 cluster   in 1 region has been serving all the use cases along with the internal dashboarding   with the user facing analytics,   data scientist team to do full table scan on some kind of the tables.   So we're actually building the multi tenancy model PNodes so you can achieve all those kind of resource isolation  

[00:35:45] Unknown:

inside the PNodes. And for the sort of typical end to end use case of   I have this event. It's triggered by some user using the application   through to,   I have aggregated across this event and all similar events to be able to provide some useful insight to an end user.   What are all of the different   system components that are necessary   for Pino to be able to   incorporate and aggregate and deliver that information to the end user, whether that's   Kafka for managing the streaming event inputs through to some of the kind of UI and application development considerations for integrating with Pinot.  

[00:36:25] Unknown:

I would like to see this whole stack as, like, track 1st is the tracking of click clients. Right? Do you need to somehow track all the activities that is done by the user?   And, typically, this is either   a Kafka client or it's some sort of an application in this d k that is embedded in the application itself. The second part is, as you mentioned, the collection. Right? So this is where something like Kafka   or Pulsar or PubSub or even have all these things come in as this. All these different systems come in play. The third 1 is really the store plus analyze. Right? And that's where something like either the data warehouse   or something like p no comes into picture, both storing and computing. So there are different applications here. Some of them just provide the storage. They don't provide the compute. Some of them just provide the compute. Some of them provide both compute and storage. So there is a lot of different systems that are getting that are being built in just the compute and storage part. It's complex, and it's also very important piece of the tech stack. And the last but not the least is, like, really your visualization layer or any sort of apps that you build on top of it. So in terms of, you know, we try to make it as simple as possible. There is an intermediate layer that I missed, which is the processing layer before it gets to the storage. So if you look at it, it's collection,   and then there is a processing layer like Flink, Samsa,   and other ETL systems, case streams, or even Spark and Huddl in the batch. Right? I think there are these processing system that just process and then put it back into the original store, be it Kafka or a Datalink.  

So what we are doing is part of Pino is we want to simplify   the ability for any user to get started. So   when we initially started, you know, we   had this need for only having structured data. So that means you needed a processing   layer in between that would take the data from Kafka   and convert it into a structured form and then feed it into Pinot.   So off late, we are going from just having them supporting the structure   to even being able to support semi structured   and even getting all the way to the unstructured data as well. Now we can just   simply point at Kafka   in just the complex arrow format as it is in we convert internally into JSON.  

We are able to add indexes on any nested complex JSON fields, and that the and the users can actually start querying.   And as they start querying, if they want to create new fields for some of the most commonly used ones, they can add derived columns on on the fly. So we are trying to change the workflow from the user and also the experience where they can simply get started very fast.   They don't have to worry about indexes. They don't have to worry about the e ETL step in between. But as they learn, as they build more and more applications, they can create the right columns on the fly without having to ingest the data. They can add indexes on the fly. They can configure pre in the pre aggregations using start re index. So we are trying to make it more iterative process, and this every iteration is dynamic and can be done on demand   without having to reboot stuff the data. That's the way we are going about because we don't want to have a lot of initial step for the users to actually get something out of Pinot. And the second 1 on the consumption side is also the need to whether you do a pre join   or you do a post join. Right? And, again, this depends on the application itself.  

Sometimes the latency is they have a much bigger buffer for the latency requirements.   In that case, they can just ingest the data as it is, have a layer on top of it, which is Presto that can actually do the joints across data in p node and also in some other data sources.   Even trying to simplify that if the data is always in p node, then the joints can also be pushed down into p node. So we are adding joints as we speak, and we had the first version of join, PR go out last week. That's another capability that we are adding. So overall, we are trying to   reduce the amount of preprocessing   and the post processing that is needed   for someone to con build applications on top of Pinot. At the same time, we have all these other knobs   in case the workload requirement changes   and you have   a very popular product and then you need really, really low latency for the end users, you can still do it. If you want to run internal applications,   we want to use build applications such as anomaly detection and things like that. You can do that too. So overall,   we are trying to improve the coverage,   simplify the initial onboarding experience for Pino, and then but at the same time, have the ability that if they want to build, like, really, really powerful applications,   they still have the knobs to actually get there.  



[00:41:07] Unknown:

As far as the data modeling considerations,   you mentioned that in order to be able to   run joins   across data that's stored in Pinot, you end up having to use Presto   for being able to support that type of syntax. And so I'm I'm wondering if you can talk through some of the ways that   users of P and O need to be thinking about the way that they lay out their data structures and data models to be able to power these   user facing analytics use cases   and some of the edge cases that you've run into as you start to design and build out these systems?  

[00:41:41] Unknown:

Definitely.   So as I said, this was a big concern in terms of the usage of Pinot because they had to do a lot of thinking   upfront because we had only structured data support.   Right? And that means we could only support column types of int, long, all the primitives.   But now we have with the support of JSON and being able to even index fields within a JSON structure   and even unstructured,   This is reducing   drastically.   You can defer that thinking to a later time so that you can get started very quickly.   And the second, in terms of pre join versus the post join, this is a great thing that keeps coming up every now and then. It's just simply impossible to get the latency that you need for your user facing analytics when you have joins during the query time. So   you will be sort of forced to do the join up front, if you are going for user facing analytics   because the experience of the user is really, really critical.  

Lot of times we have seen   companies try to avoid the join and then they try to put the do the join during the query time and the experience is so bad because you have this cursor spinning all the time and the experience is really bad from the user point of view. So it is worth paying that additional   overhead upfront in terms of doing the joins   before and then avoiding the joins during the real time. So push it. So it's really in   the overall scheme. It's where do you put the join? Where do you put the transformation?   That is really what someone needs to think about. You can always get started with   no joins, no transformation   as it is. Now as you see the product becoming more and more successful,   then you can introduce the pre join and also the pre transformation if needed. It's all about having that flexibility   that you can change this, but you don't have to change the system itself. That's the most important thing because changing the system means that you're trying to change the whole stack. Now you need to change your clients. You need to change   a bunch of things, and that is something that is not needed that we want to avoid as much as possible. Somewhere down to, like, from an Uber side, initially, we actually  

[00:43:49] Unknown:

just try to do the very simple thing. We try to dump 1 Kafka   topic into 1 p note table. That's kind of the story you are like.   We allow user to do the self serve email onboarding. They can't just, say, okay. I have a 1 type of topic which has schema inside and can pick some columns and dump the data into Pinot. And then I can use, like, Presto for joining and for exploring,   then they will figure out, okay. So for my use cases,   I can't survey it from the Presto side for some kind of users.   And then they later on figuring out, okay. I need this kind of latency and the QPS   requirements to be supported. Then they actually move this kind of presto drawing or whatever the declaration queries or pre aggregations   into something like stream processing, framework like Flink. And they can write, like, stream SQL to actually create those kind of new tables, so flatten the tables. And then ingest that into Pinot. And in Pinot side, you can also build assorted index   or inverting index in terms of serving more, like, user facing analysis queries, like since those queries are, like, per user business thing, you can you can actually do more indexing and the query performance furthermore. So all those kind of are more, like, journey or, like, how a product or the data model has been built. People really, although they have the rough idea of their product, they don't really get the data modeling   first and make it right. It's always iterative.  

If you build the entire stack, which from the Kafka to Presto,   it actually allows the product team or the application people to figuring out what they really need, and, also,   it greatly improves   the iterative time. Right? So if you want to upgrade   your system, add 1 more new feature. The turnaround time is actually much faster in terms of using this whole stack.  

[00:45:45] Unknown:

Are you looking for a structured and battle tested approach for learning data engineering?   Would you like to know how you can build proper data infrastructures that are built to last? Would you like to have a seasoned industry expert guide you and answer all of your questions?   Join Pipeline Academy,   the world's first data engineering boot camp. Learn in small groups with like minded professionals for 9 weeks part time to level up in your career.   The course covers the most relevant and essential data and software engineering topics that enable you to start your journey as a professional data engineer or analytics engineer.   Plus, they have ask me anythings with world class guest speakers every week. The next cohort starts in April of 2022.  

Visitdataengineeringpodcast.com/academytoday   and apply now.   Another interesting element of   this application space is the question of the data life cycle,   where when you're working on internal analytics for an organization, you're likely going to want to keep all of the data available for querying for a fairly long time horizon.   But for user facing applications, those time horizons can be highly variable depending on context.   And I'm curious   how you have approached that aspect of the problem in P and O to be able to support this   sort of management of time horizon and then data tiering so that you don't necessarily   have to have an all or nothing approach where maybe you just have to accept some level of latency for queries that are going further back in time.  



[00:47:14] Unknown:

Absolutely. No. That's a great point. As you saw, we started off with the intent of user facing analytics where the retention span is much lower. Right? I mean, we see somewhere   between 30 days to 2 years. Right? That's where and do you know the value of data also decreases drastically as they age?   But 1 of the common things that we have been asked from the users is if we allow the speed of pino,   and we also want to use it for internal analytics as well, but it is not cost effective because now we need very tightly coupled architecture. So now we need to scale the compute and the storage together,   but we very infrequently   query the older   data. So that's 1 other key big change that we did in Pino. So we were able to add this concept of tiered storage.  

So that again, kudos to all the engineers in the Pino   team. We built it in a way that there was a right abstraction in terms of the   file system API. We were able to move from the local file system to the remote file system. So now, you know, can actually run directly on s 3. That   changes the whole game because we are seeing   someone like Stripe, for example, has   the transactions from day 1. It's huge. They're already at, like, 2, 000 notes with petabyte of data and it's already   overtaken LinkedIn and Uber in terms of the scale.   And something like a tiered storage is a big game changer because now you can keep the data for as long as you want, and you can tier it to the s 3 or any other blob store as the data ages. It's a really big feature. And the best part about this, as with AD, everything else is, you know, has been designed where you can almost think of about this as a slider. You can actually decide at which point in the slider that you want to be. You can have 1 extreme of all of them are tightly coupled, local both the computer and storage,   and the other extremes, everything is decoupled. And you can also be in between. You can say for the 1st 10 days, I want it to be tightly coupled. I want the data to be local. And after this data ages more than 10, it has to it can go go to the s 3 tier. Those are all, again, configurable   not only at a cluster level, but you can even configure it at every table level. You can have some tables   have   moved the data after 30 days, some of them after 10 days. Again, it's a ability of not only providing you the trade off, but also the ability to provide that control   that you can decide the trade off point. And depending on the application and depending on the requirement,   you can change that. So tier 2 is 1 of the big big things for us. We have added the support for s 3. We plan to add for GCS and Azure as well very soon. Apart from that, for data life cycle management,  

[00:49:53] Unknown:

you know, side that you can also use to   do, like, aggregations   oppose aggregations   for your real time data as well. Because typically,   when you are ingesting data from Kafka, all those things are event based. However, for your queries or your applications,   a lot of users, they they actually just to show the aggregation results. Right? So in terms of that, Pinel Millian can actually do the post aggregation   after 1 day is finished or the configurable time, actually. You can you can write in hour or in   1 day to   aggregate all those events and generate much compact data block and,   serve it in a more efficient way, which means for even for the same data,   you can trace many more years of the data, but with, like, a very low cost. You will also then come do this kind of aggregation.  



[00:50:44] Unknown:

As far as being able to customize Pino, you mentioned that it was designed to be flexible and adaptable to the sort of end users' requirements. I'm wondering if you can talk through some of the   interfaces that are available for people to be able to extend and implement their own capabilities   on top of or within Pinot?  

[00:51:04] Unknown:

So for the Pinot side, we basically make, like, every component   to be, like, extensible or pluggable. For example, at the LinkedIn time, LinkedIn has the stack of all the batch processing in Hadoop.   The LinkedIn has its building data format is is several, and the real time ingesting are coming always from Kafka.   So that's kind of, like, how the initial version of penal, which you only need to support this kind of system integrations.   And the later on, when we move to the open source, we've got many people has, like, different requirements, and they have different systems.   So we make a lot of things to be customized. The first is definitely the encoding and the file formats. So apart from the Averro side, we also support   straight for the above   and also like a or, say, all those kind of different   file format is already been built in inside the Pinot.  

Execution framework. The batch execution framework, we have the support from   Hadoop, from Spark, from Flink.   So all those kind of batch processing are now available   from the open source side that you can actually run your, you know, segmented generation job in Spark or Flink as you wish. Also, for the   streaming data ingestion,   Pinocall actually   now, you know, abstract   all those kind of streaming interface, how you get data into Pinel. So from Kafka side, it's just like version upgrade. We have 2 Kafka interface, 1 4, Kafka point 9 and the 1 for point Kafka, kava 2.0 plus. So you have 2 Kafka connectors, and then we later on build the Kinesis port. But, also, we have the Google Pubs app and the Apache browser. Recently, there some more contribution   from the community for RabbitMQ.  

So all those things are, like, all based on   the clean,   the right abstraction   for the interface. So it makes   the user easily   hook up their existing system.   And, also, they can contribute   effect to the Pino community.  

[00:53:09] Unknown:

And in terms of the applications   of Pino, I'm wondering what have been some of the most interesting or innovative or unexpected ways that you've seen it used. I probably was definitely the 1 that came to my mind is  

[00:53:23] Unknown:

the ability to just keep the raw data forever and at the same time be able to do aggregation also on top of it using start index. So now you can start at a very high level, and then you can start doing root cause analysis. You can narrow down, and you can get to all the way to a raw event. So that is a very interesting use case that we saw.   And in terms of just the ads itself, I think there is   the way people have started using the geospatial indexing   and the ad tracking and the ad accounting is also another   amazing thing that we saw where ads are shown on buses.  

And then   you track the buses on 1 side, and then you track the users on the other side. And then you are able to   do the in intersection between the 2, and then you are now able to see how many people actually saw an ad on the bus. So that's an amazing use case that we saw. It's really cool to   see how many people   get inspired from things like Uber Eats. Like, you can think of a website now selling cars. Now they can provide analytics to their dealers   who can actually   how many people viewed your car page, how many people clicked on what, and you can kind of provide all these insights back to the dealer. So there is lot of these insights that can be provided.   So just a call, for example, another cool application that we saw is the transcripts of the call that we are talking about. Right? Like, that can actually be fed into Pinot. You can think about, like, how long did Tobias talk, how long did Sean talk about in this call? And how long did I talk? What was the tone? Who said hi first? Who said bye? All those things can actually be provided. It's like lot of these things that we never thought being in a company at LinkedIn and Uber,   but now it is lot of things in terms of both b to b and also b to b to c. Trading is another application. So there is tons of things that we are actually seeing in the wild that we had definitely not thought about before. But really good to see how people draw analogies between   the ones that we have already built and how they translate it to their use case. Education is another 1. They're looking at the videos. They're like, okay. How long did the users watch this video? When did they pause?  

How many was there any section that was watched multiple times? So again, so a lot of analytics on pretty much every entity that you can see on the web, you can basically provide analytics around that. So, yeah, definitely   lot more things than   than what we expected. And, again, LinkedIn is a great example. We already have, like, 70 products on LinkedIn itself.   That itself is was a huge win for us in terms of seeing   how quickly the adoption grew and what kind of products they were able to build.  

[00:55:59] Unknown:

And in terms of the kind of growth and adoption of Pino, I know you mentioned a number of different companies that have been relying on it. But   from when it first became public to where you are now, I'm wondering if you can talk to some of the ways that you're thinking about the   sort of community engagement, community involvement,   and areas of contribution that you're looking for.  

[00:56:21] Unknown:

Yeah. Definitely. I remember when we started, we had a very small community. We had, like, a Slack channel of, like, 100 people in 2020, and then we were not even tracking their downloads. We had 0 downloads.   And fast forward 2 years, we have, like, more than 2, 000 people in the Slack community.   Very vibrant, asking a lot of different very, very exciting questions, and we get to know all the use cases that we had not thought about before.   And at the same time, we had, like, almost a 1000000 downloads more than a 1000000 downloads last year itself.   So huge huge growth in the community   across again, lot of help from LinkedIn, Uber, and Stripe, and all these other companies getting involved in the project as well. It keeps pushing the envelope further and further.  

The bunch of things that we do want help from the community is extending the features at Uber, for example, helped us add the absurd feature, which is a big differentiator and something that a lot of other systems are not able to add. Then tiered storage is something that we from start re is contributing to that towards that. So join is another 1 that we did. We are adding that as well. To JSON storage. So there are a lot of these things, and we actually even publish the road map. And we also share the road map, and people community can actually work on that. Any members from the community are listening to this. Definitely encourage them to vote on these features because   take that very seriously. That's something that we consider for the prioritization of the work that we do. Again, it's it's community is our product management for us. They tell us what the priority is. So really looking forward, not just from contribution,   but also in terms of their requirements and where they want to take the system forward.  



[00:57:59] Unknown:

In your own experience   of working on the Pino project and now your work at StarTree, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?  

[00:58:10] Unknown:

I think what's the biggest challenging here is   when we were at LinkedIn and, Uber, we actually focused a lot to the speed of PayNow. Because for all those kind of well established companies,   you have a pretty good kind of ecosystem, which all the software foundation has been placed there. So you can you can leverage those. For example,   how you deploy Pinot. At the LinkedIn, they're, like, content management. At Uber, they're also, like, have all the Docker, Mesos,   and all those things there. But when we come to start with and we figure out, oh, this industry are using Docker, using Kubernetes,   They want   to have those kind of integrations. So we actually quickly build those   recipes on how you can simply deploy Payno. You can   also do things like the management. Right? So at Uber and LinkedIn, we have been build a lot of toolings.  

Now we actually   make all those toolings to be actually built in p node APIs,   which you can easily do, let's say, able rebalance, those kind of things in p node itself.   Another thing here is we actually transformed   to make p node plug in model, and we will focus we will actually   focus the more focus the more on the right abstraction of the interfaces.   So panel itself can be easily extended   to, like,   suitable to, integrate with other systems.   So   at the very beginning, when we build, for example, the Kafka   connector, it's a chips like a couple of months since. The once we make the clean interface for that. So for the NIS   connector, the POC was down in 2 days, and it took, like, a week for us to really productionize it. But this kind of turnaround time are actually super impressive,   and you actually realize the power of you are doing the right thing and making the right app structure. And all those pluggable model   really makes the difference also for the community to contribute things into your system, and then you have the right capability for, like, unit test, integration test, all those things. Yeah. I think Sean pretty much touched on lot of different points here. Only thing that I would probably add there is  

[01:00:25] Unknown:

kind of our thought process being that LinkedIn was always about speed. If you kind of look at the evolution of how you build a feature or a project is, like, make it work first and then make it right and make it fast. But at LinkedIn, we would always shoot for make it fast because that's kind of the scale at which we were operating. So speed was, like, really, really important for us. And I think as we see more in the open source, it's about the iteration speed is also very important. So we want to get the feature out. We get it in a good enough shape so that people can use it, but then constantly iterate on top of it. So being more focused on the iteration speed and then taking it to the finish lines   in multiple steps versus trying to get to the best in the first version. I think that's something that we are being aware of and then trying to change that.  



[01:01:12] Unknown:

For people who are looking for a system that will provide low latency queries and potentially serve,   you know, multiple concurrent users for user facing analytics? What are the cases where Pinot is the wrong choice and they might be better served with a different architecture   or a different storage engine or, you know, maybe reconsidering their requirements?  

[01:01:33] Unknown:

Yeah. Definitely. I mean, the first thing is it's definitely not a replacement for data warehouse. We have seen many cases where, you know, coexist with BigQuery, Snowflakes. They have all this data in Snowflake, but it doesn't allow get them the speed that they want to serve external applications.   So they take the data and then put it into p n o. So that's perfectly fine. This is probably not a tool where you give it to your business analyst and then ask them to write, like, a super complex 100 way joint query. So definitely not a replacement for data warehouse.   It's also not a replacement for, like, your traditional key value stores. So I wouldn't,   instead of using MongoDB, Cassandra, or any of these other systems, you don't wanna use Pino for that. There's no transactionality   that is there. It is not really built for asset compliance, so it's not an old TP store in that way. Definitely not a relational database as well like old TP, not MySQL, Postgres, So that's not a replacement for that.  

And it's also not a replacement for such systems where you want   ranking relevance   like Lucene and Elasticsearch. They have the right systems.   Though we are seeing a lot of workloads   move from some of these systems into Pino. And the reason is just that something like p no was not available, so people ended up using whatever system that they had. But they hit the scale and they they hit a wall, and they're not able to get what they want from the system. So we do see a lot of use cases   move from Postgres, from Elasticsearch,   from Snowflake data warehouses, and also even from the key value stores. A lot of them do move to Pinot, but that's because they were not the right systems to use for analytics in the first place.  



[01:03:12] Unknown:

As you continue to   build and iterate on the Pino project and grow the business around StarTree? What are some of the things you have planned for the near to medium term or any projects that you're excited to dig into?  

[01:03:24] Unknown:

Yeah. Definitely. Eared storage is 1 thing that we all already released a preview version of that. So now we can we support s 3. We we will plan to continue iterate a lot on that. We see that,   our goal is to be able to provide almost the same performance as we are able to provide   with local storage. We are already seeing that it's very close with, like, literally, like, some 200 milliseconds extra latency, and we have done ton of   innovations there in terms of being able to provide this performance with the tiered storage by being able to work directly on s 3. So you are still getting sub second latency   with that. And the second 1 that we are super excited but we'll continue to invest is going from this concept of data is immutable once you put in p note 2 being mutable. So it's not only mutable in terms of batch,   but you are now able to   update every single row in p note. Right? I think that changes a lot of things that you need to join before. It doesn't need any does do not need joins anymore.  

You can get accurate metrics. A lot of things open up because of this ability. So that's another thing that's super excited.   Last but not the least, there are data models that are structured. There are data models that are semi structured, and they're unstructured.   We are taking that to an extreme where we say that you can have a table in p and o and   you can decide what each column should be. Each column can be either a structured column or a semi structured. So you can have a table with   lots of different columns, and each 1 can be its own format.   So that's another thing that we are excited about. Join is a new 1, and we plan to continue to invest heavily on the join. Again, in collaboration with   Uber and LinkedIn,   build core distributed   joins, multistage.  

So we already have the first version that's already   in progress in preview,   and then we plan to add the collocated joins and all those other fancy things in the join site.   In terms of the use cases, so 1 of the things that, you know, was really for the metrics use case, but we are seeing a lot more of logs and phrases, applications coming into Pinot. So people are using Pinot as, like, the strong back end defacto back end to build their   observability   systems on top of it. So a lot of companies are using Pinot as that back end. So we will continue to enhance   Pinot being your single store for your logs, metrics, and phrases. All of them can be served in a more very efficient   manner with all the features that we added so far. And last but not the least is, like, integration with all the other   players in the ecosystem.  

Presto, Trino,   Kafka, Pulsar, all these spark. We want to make sure that our integration is really, really strong. As you see, the stack is be is is quite complex right now, and we want to see take a lot of burden out of the users to be able to integrate this with other systems in the world.  

[01:06:16] Unknown:

Are there any other aspects of the work that you're doing at Pino or the ways that it's applied or the specifics of its implementation that we didn't discuss yet that you'd like to cover before we close out the show? 1 thing that I would like to add is  

[01:06:29] Unknown:

really the way we see   the whole tech stack layout and how it's changing.   I kind of see a very strong   analogy with how the monolith   is splitting into microservices   in the stateless system.   I see a similar thing play out in the data system as well. What used to be just 1 giant, if you look at it decades ago, was, like, Oracle doing   your collection,   the storage, analysis.   We it did everything in 1 system, and that's the thing that we are seeing that change. Now there is 1 system for just collecting   and which is Kafka, and there is 1 system just for storage. There is 1 system just for compute. Each 1 is becoming so complex and, and also for a good reason in terms of evolution.   And I think it's a good thing to happen. I mean, the same thing is happening on the data warehouse side as well where everything was solved in a data warehouse, and now that is getting split apart in terms of different systems coming into picture and doing 1 thing really, really well. So what this is definitely   a little bit of an overhead for the users because now they don't have 1 tool. They have hundreds of tools. Some might argue this is a bad thing, but I think it's a part of the evolution in my view. What we really need is the tooling system to   make sure that these systems can integrate seamlessly.  

And so more standardization   being evolved across all these applications will definitely help in building those type integrations.   For example, s 3 is definitely 1 of the things that standardized on the file systems. Right? And Kafka is getting there in terms of stand being this the fact of standard in terms of the messaging   API.   So the more these standardizations   evolve, there is a lot more  

[01:08:11] Unknown:

ability to seamlessly integrate these different components. We are super excited about that. We continue to make sure that Pinot is super easy to integrate with other systems out there. Well, for anybody who wants to get in touch with either of you, I'll have you each add your preferred contact information to the show notes. And as a final question, I'd like to get each of your perspectives on what you see as being the biggest gap in the tooling or technology that's available for data management today.  

[01:08:36] Unknown:

I kind of covered that in the previous 1. I think it's built on that. It's really that there are so many different tools out there, and it's definitely confusing for any user to know what tool should I use for what and making it really simple for them to put these pieces together. Right? I think the way I see it was previously   in in data, you would buy   a prebuilt toy. Right? And you were not really building that toy. But now requirements from within the company have changed so much that you just 1 toy is not going to serve all your purpose. So you need, like, a lot of different toys. But you go to the market and you what you get is, like, Lego pieces.   So you now need to put those things together to build the stack that you want. So it's very powerful because now you can build so many different toys from these Lego pieces, but at the same time, you need to know how to connect them. It's a different challenge from what used to be there before. The previous 1 was, like, you would have, like, 100 different systems doing 1 thing,   and you don't have very clean stack. So you have the same data sitting in, like, 100 different systems. So that was a different problem. Now you have a different problem that you can actually build an amazing stack,   but putting them together is not easy and not trivial. So I think that's the gap. Again, it's part of the evolution. I think 10 years from now, it will be a very different thing where makes it so simple and seamless, especially with the SaaS companies coming out and everything being offered as a service. You don't have to do that. It should be There will be seamless integration between all these services so that you can actually put together   completely end to end stack within within a day or 2. From my side, I think, Deepak pretty much, like,  

[01:10:16] Unknown:

cover all my thoughts here. Yeah. 1 thing here is, like, about more advanced kind of integrations with all the other   surrounding areas apart from, like, the reporting, dashboarding, those traditional   BI use cases. Here, we have, like, more things like data scientist kind of integrations.   There are a lot of data scientist   people. They are   building models. They build companies to say, okay. You can roll this fancy model, saw this machine learning platform,   hop off, like, different data stores. And here, since the trend   of moving to real time data   is,   like, becoming more and more useful, and people realize how powerful those real time data are. In the old time, we don't have a way to provide the reliable,   fast   real time data. Nowadays, this is not a dream. It's   the reality. Those kind of integrations and how people are leveraging real time data and discover this kind of gold mine   is   the next thing and the missing part that you can actually leverage this real time to do more things. And people should, like, open up their mind, find out, like, what are those things actually  

[01:11:27] Unknown:

being being achieved. Alright. Well, thank you both very much for taking the time today to join me and share the work that you've been doing at Pinot and now at StarTree. It's definitely a very   interesting product,  

[01:11:38] Unknown:

interesting project, and an interesting problem space with a lot of complex requirements. So I appreciate all of the time and energy that you and the rest of the community have put into that, and I hope you enjoy the rest of your day. Thanks again, Tobias, for having us, and kudos to all the great work that you are doing. I've been listening to your podcast for a long time. It was very, very insightful. Get to learn a lot of good things. Keep up the good work, and thanks for doing this for the community.  

[01:12:03] Unknown:

Thanks, Tobias. Thank you.   Listening. Don't forget to check out our other show, podcast.init@pythonpodcast.com   to learn about the Python language, its community, and the innovative ways it is being used.   And visit the site at dataengineeringpodcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com   with your story. And to help other people find the show, please leave review on iTunes and tell your friends and coworkers.