The Role of Product Managers in Data-Centric Organizations

Hello, and welcome to the Data Engineering podcast, the show about modern data management.

Data lakes are notoriously complex.

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end to end data lake has platform built on Trino, the query engine Apache Iceberg was designed for.

Starburst has complete support for all table formats, including Apache Iceberg, Hive, and Delta Lake.

And Starburst is trusted by teams of all sizes, including Comcast and DoorDash.

Want to see Starburst in action? Go to data engineering podcast.com/starburst

today and get $500 in credits to try Starburst Galaxy, the easiest and fastest way to get started using Trino.

Your host is Tobias Macy, and today I'm interviewing Praveen Gujjar about product management for data and analytical platforms. So, Praveen, can you start by introducing yourself?

Yeah. Thanks, first of all, for inviting me to the session. My name is Praveen Gujjar. I currently work as a director of product at LinkedIn,

but my journey in product management has been,

ticket test right now.

Especially after my MBA, I started with Amazon, in a product management capacity,

worked in Amazon Web Services,

specifically in building a lot of the early generations of, AWS or public cloud, products as well.

Then

I joined Twitter,

where I basically,

played a very critical role for a short duration, though,

on moving some of the, you know, on premise workload that we'd all had to public cloud.

Later on for the last, 7 years close to now, I've been with LinkedIn. At LinkedIn, I've paid multiple roles, specifically building data platforms and products and also

digital advertising. So in summary,

my experience spans across data products, obviously,

building data products and platforms,

digital advertising,

AI, and cloud as well. So a lot of,

different experience and in the process, basically, have built, multibillion dollar businesses for these organizations as well.

Fortunately, I have been part of big tech company. I have seen these, internal operations very closely as well. So, yeah, that's my introduction in short. And do you remember how you first got started working in the data space?

Yeah. As I said,

I started with Amazon

after my MBA, but I do have a engineering degree where I was a developer before as well. Transitioned into product management role after

my MBA.

I first started with, Amazon Web Services, as I said, where we built

a lot of the early public, cloud,

products as well.

My area of expertise was, building easy two instances, a lot more focused towards, storage optimized and IO optimized instances.

And these really were instances that were predominantly

catering to the needs of our workloads, such as,

Redshift, for example, Amazon's Redshift,

or, data warehouses, as well as in memory,

in memory workloads like SAP HANA and other things as well. So a lot of analytics product data,

warehouses, data lakes that were basically coming up during that time, we built specifically

storage optimized and IO optimized instances at that time. So that's really how I got into the data product and platform space as well.

Worked in that space for about, you know, in AWS for about 5 years.

Then, again, continued very similar trend with Twitter as well where

we negotiated contract with some of the biggest public cloud providers to move Twitter's workload to public cloud. Ended up, doing, moving their, couple of the Huddl clusters,

to,

cloud as well. And last but not the least, continued the same trend within LinkedIn as well.

Was one of the early PMs who started data products and platforms division within LinkedIn. As LinkedIn

started to become larger organization as well, we clearly saw a need to, have a specific, product management team

leading data product and platform. So there, I built a portfolio of platforms and products as well.

My current work is a little bit different than, just data products and platforms, but I still,

very closely associated with enterprise grade data products in the digital,

advertising space,

specifically catering to the needs of

large customers,

like maybe a media agency,

big brands,

as well.

Product management, as it is commonly thought of for people in the engineering space is largely focused on those end user features where a product in that case is maybe a website or

Obviously, for organizations like LinkedIn, Twitter, Amazon, data is a huge component of that functionality. And I'm wondering if you can give some comparison between the typical view of product management

and product management as it exists in this view of being very data oriented and working very closely with the data teams?

Yeah. I think that's a really good question. So I mentor,

multiple product management aspirants as well, and most of them really

carry this view that, product managers build,

UI experiences for our customers,

build apps for our customers alone. But you have product managers that have a varied

experiences in background, and they build different kind of products as well. If I can loosely talk about 3 specific,

product types of product managers.

1,

I would say is a consumer product manager who built in, apps like the DoorDash app, for example, or the Airbnb app as well.

2nd is b to b product managers who build b to b specific products. It can be a mixture of data products as well as

user facing UI experience as well for large, like, for another organization.

These are the b to b product managers. And the third one basically is we are product managers who build data products and platforms

and platforms in general, for both internal and external use cases for organizations.

So these are 3 loose categories.

But if you stick a step back, every organizations,

when they basically scale, they clearly see a need

for building in house data platforms and products as well.

Few things that really, matters the most for them, data obviously, as we say.

It's been old saying now already that data is a new oil.

So data definitely is a clear mode for a lot of these organization as well. So making sure that the data is clean, reliable,

and compliant with regulatory guidelines in a way is very key for these organizations as well.

Second one is scale, the massive amount of scale that this organization has. That means it introduces a lot of complexity into the data ecosystem.

So it's critical to basically build for longer term platforms or products.

3rd, I think, understanding data flow within the ecosystem is very, very key to build every single product and features

at the scale of these larger organization as well. Hence,

investing in data platforms and products in house becomes very important,

because

off the shelf,

our fully managed services

do not often provide the flexibility,

the customization

needed to meet the needs of these larger organization as well.

So where does product manager come in? So you can imagine now that,

you are pretty much building data products and platform for your entire organization. It may be 10,000

plus people or 15 or even 100 k plus people,

depending upon how big the organization is.

And, you need a a in a spoken hub condition, you need a hub kind of a personality or a function

that,

really enables these teams to come together, work on a common vision,

solve the complex problem in a good way because you basically are now interacting with engineers,

back end engineers or infra engineers,

as well as data scientists and other folks as well. So how can you bring everyone together under the common vision becomes important. That's where really product managers play a role. So I was speaking to someone in Stripe.

Stripe, has historically

not had product managers in this particular space, but they clearly have seen need for having product managers in the space in the last 3 to 4 years, and they are building

a product manager,

organization for data platform and products as well, where you can clearly demonstrate,

results from, having a function like this.

For cases where you are working on determining what features to provide

during the product ideation phase,

because of the fact that in very data oriented products and capabilities, the data that you have becomes a limiting factor.

Wondering if you can talk through some of the tensions that exist in the process of figuring out if this is something that you would like to do and then doing some of that feasibility analysis and deciding what is a reasonable amount of investment to put into enabling this feature

versus the proposed gain that you're going to get from the amount of engagement you're trying to drive with it, especially in the case where either you don't have that data yet or making that data available

in a very high availability,

low latency use case is going to require a significant technical investment.

Yeah. I think that's a really good question as well. Some of the, like, as an organization, you really

go back and forth in making these decisions as well. And if you are, there are both, cultural and organizational aspects that are necessary, to make this happen, and, also, there are technical,

aspects that are necessary to happen. Let's just talk about the organizational

aspects first to begin with, and then we can talk about the technical aspects as well. First and foremost, I think,

large organization like this have to structure their internal workings to,

enable teams to build for these large data products and platforms as well. That, can be as simple as,

having organization structure. For example, a city organization

that completely manages the infra, the data platforms

necessary as well is very key to the thing. Then basically comes the cultural aspects as well. Rolling out the culture, that is necessary for to these,

teams to actually,

use the data platform and products, that are built in as well. For that, we need to demonstrate value because no team basically will just adapt to our data product or platform if, unless it delivers on the similar value that you have talked about. It has basically

so that use cases,

meets the complexity requirement,

and the performance requirement, be it latency, throughput, or anything else as well. So, it's equally important to basically demonstrate the value, but, culturally,

activating these teams to use these data platform products or some kind of a mandate also becomes very clear as well. So these are definitely organization and cultural challenge that the organization has to go through to ensure

the data platform and products are first built. They are solving key use cases or problems for the, internal customers. And 3rd, they are actually used by these internal customers as well. The second piece,

is very interesting question as well. When you talk about

the feasibility of building the data product, the data availability or anything else. It's a very complex questions as well. I often see this as,

a 3 circular VIN diagrams as well where the user needs are the business needs basically be a very critical role as well. How strategically aligned some of the capabilities that we have to build for business needs is very important. The second, basically, is obviously what you talked about, the technological,

feasibility that includes availability of the data,

available technology to meet the performance requirements, for example, is very critical as well. 3rd, basically, is what is the leverage, that we are creating by building these capabilities as well. I think very initially, we talked about what is the the minimum capability that you can build that produces the highest leverage opportunity.

That is very key as well.

So,

thinking about this intersection is very important as well, and ruthlessly prioritizing

is key.

And, some of this basically may also, you may have to create requirements for other teams that may not be something that you immediately

control or, work with as well.

It can be building data pipelines.

It can be basically upgrading your software of our infrastructure to make sure that we have performance both in terms of latency and throughput, can be managed as well. That's a very key thing as well. Like, if you take an example of today, like, AI is definitely acting as a catalyst for a lot of these capabilities as well. Organizations are rushing into upgrading their infrastructure and capabilities

to ensure,

we go through the massive inflow of data that is expected because of,

AI

related products and features. And in addition to that, like, what is needed to train these models as well. So

for these,

for our organization to be successful,

you need to really think 5 years, 10 years ahead and be committed to invest in these. Because at this scale,

you cannot share a share in 6 months or 1 year. So really planning for the future and working towards it very important.

To that point of the significant investment, both in terms of engineering capacity and financial input to that engineering headcount to be able to support these features and the time horizons that you're looking towards,

the fact that you're trying to build these systems in a way that they could be used in an interactive capacity by a large number of users, you ideally don't want to have to create

individual point solutions every single time you want to create a new feature.

And so I'm wondering how you think about that congregation

or grouping of the types of features in terms of the technical platforms that are going to be required to support them so that you can try to build those technical capabilities in a way that they can be used across a wide number of potential use cases both now and into the future?

Yeah. I think that's a really good question.

So platformization

is, kind of,

both tabooed and can be both,

can be, both tabooed and also overused,

within,

organizations

as well. Everyone wants to build a platform, but people are also really,

have a negative reaction to platform because there's a couple lot of time in building, and we often tend to overdo platformization as well. So finding a really,

a good balance is very key as well. What is the right amount of platformization that provides them maximum leverage?

So that's a very critical component of, of it. The second piece of the puzzle also is,

really,

doing extensive research across multiple teams as well to understand what are the common capabilities that are necessary that can be, platformized as well to provide the maximum leverage. I think this goes back to the point of right amount of platformization as well. You want to provide

common set of capabilities packaged together through an API,

but you also want to provide ability for these, teams to customize that on top of,

the platform as well. So that modularity, that customization capability is very key,

important piece of the puzzle as well. So I think that's really where,

having that, these two criteria is very important as well. And the 3rd piece of the puzzle also is,

clearly, in all of these bigger organizations, you have, 2 different types of platforms as well. 1 is a product platform. 1's a tech platform as well.

So there is need to build,

these 2 capabilities very clearly because they serve different

use cases as well and often have overlaps

as well. So for example, our product platform can be a messaging platform that can be used by

multiple teams,

to build messaging capabilities as well. Common set of capabilities that it provides that can can be customized by other, teams as well. Whereas a tech platform can be an experimentation platform as well that can be used by both your b to b organizations as well as b to c organizations as well to run the experimentation

as well. So building these, 2 different platforms, you need to make sure that you invest,

in these 2 different platforms,

as well. And also, as I said before, like, really looking at the long term, you should have a longer term road map, like 5 to 10 years to really understand, like, how do you evolve this, platform as well. All the technology changes every few year as well. Having that vision is very important as well. And as soon as there are critical milestones in technology, you need to make sure that you need to work on these platforms, customer like, evolution of these platforms and products

very clearly as well. So, hence yeah. I think to your question,

longer term vision is very important.

Having a tops down approach is very important as well.

Making sure that you build the, right amount of platformization,

and not fall to the trap, trap of old platformization is important as well. And about all these things, going back to my earlier,

response as well,

having,

an organization structure and culture is very important. We have top down execs kind of mandate sometimes as well in investment in these and making these hard decisions are important.

I think in many organizations,

there will be a time where,

every 5 years or 7 years where you have to see that my infrastructure,

my platform, and product capabilities are now getting accelerated. So I need to deliberately invest next couple of years, for example, to overall all of them to set myself for another 5 to 7 minutes.

This is key aspect of it.

Many of the organizations that I have worked on have deliberately

made this a priority

organization wide. Otherwise, it simply cannot turn you cease to exist.

To your point of the technology moving fast, the requirements constantly

substantial amount of early

A substantial amount of early investment in data platforms and data capabilities was oriented around the data warehouse and supporting business intelligence

use cases.

We've now moved beyond that largely to these real time systems being able to do things like embedded analytics, data products for end users

That has substantially changed the types of technologies that we need to be able to support those use cases while also maintaining the analytical capacity of the warehouse and business intelligence.

I'm wondering if you can talk to some of the ways that you see technical teams breaking down the categories of functionality into consumable chunks

so that they can figure out how to build these new technologies or bring in new platforms or components to be able to support these growing use cases and the increased demands on the capabilities of those data platforms to be able to operate in a higher capacity and lower latency format?

That is an awesome question as well. I think, specifically,

as you described, right, this needs a very methodical and deliberate approach,

as well. And this is really where the product management function plays a very critical role as well.

I think the we recently went through something very similar. So I'm gonna basically

layover the framework on what this was actually, did as well.

It all really starts from,

working backwards from the customers as well. At, when I was at Amazon, we used to have this, like, customer obsession story. Right? Like, working backwards from customers,

various organizations,

especially big organizations,

follow different kind of a methodology or framework to,

eventually,

achieve this goal. Like, working back from customers, how can

I ensure

that we build the right product as well? So, one such framework that can be used is jobs to be done as well. So jobs to be done is a framework that basically enables you to first map

all the key jobs or all the key problems that your customers wants to solve, be it internal or external.

And with that as a foundation,

now you can actually build upon that to understand how do they solve this problem and how can we capability how can we build capabilities both in platforms and products to solve this problem as well.

So I think a simple exercise that, teams like this can actually do is

to begin with, map all the jobs to be done. You start with certain key capabilities or key problems that your customers want to solve. It can be basically like what you talked about, real time analytics that should be available as well, and a massive amount of data that can be,

used to try and, train the AI models,

as well, and your infrastructure that basically to support lower latency and higher performance to train these models as well. And,

specifically, if you work in an ecosystem like our ecosystem,

where real time nature is very critical as well to make bidding decisions, to make serving decisions.

How do you basically,

leverage the

upcoming technologies

to increasingly,

reduce the latency in making these decisions as well is where it becomes a very key. So you map all these capabilities

as jobs to be done, framework as well.

Then, what you start doing is writing user stories or

customer, personas and user stories basically to represent these jobs to be done as well. As simple to me, if my user wants to run-in

product, you know, like

embedded analytics, basically,

with my listing. These are the journey that we'll actually go through as well.

So you come you come up with a few user stories

to that. And then each user stories, you start mapping what are the capabilities I need to build to,

achieve these,

fulfill these use cases as well. In those capabilities, what you'll also see is a clear trend of similar capability that are needed by multiple such user stories and multiple such persona as well. That's really the cue for you to, build platforms,

for those meeting those capabilities as well. These personas may be different. These use cases may be different as well, but those commonalities in the user story start emerging. So you map those commonalities

and capabilities,

the common capabilities,

to come up with the platform as well. Now you have jobs to be done, jobs that you have. You have user stories, and now you have collated

common capabilities.

And, the remaining capabilities that we have to build can be the products that are built on top of these capabilities as well.

Now this exercise gives you a huge metrics that tells your organizations

what platforms are either necessary, are missing in our portfolio,

what platforms basically needs to be upgraded

to meet the demand of the next few years as well, and what products are should be customized or built on top of these platforms as well to make sure we serve the needs of our internal customers and also the end consumers who eventually will see the benefit of these as well. So this is really a a elaborate exercise that sometimes can take you months to really go through. And, again, I cannot emphasize the fact that how critical it is to have a culture like this and the top down buying like this to really have this kind of an approach.

And one day, then you can actually fully understand your technology,

tech portfolio,

and the missing gaps in it and what is needed to really

overhaul it for the next few years.

That's really the TLDR

version of how, organizations go about this.

In your experience of working in this space of technical product management around these very data focused features, I'm wondering if there is an example of a product either specifically or a general category of product that you can talk through the overall journey from this is the goal that we're trying to achieve through the process of figuring out how to break that down into those technical components and the custom technologies that you've had to build to help people understand all the different stages of the life cycle

and the types of organizational

involvement that you have to bring in to be able to bring that type of project to a successful conclusion?

Yeah. I think that's a great question. Let's take an example of, experimentation,

as well. So if you are a large organization,

you like, you're always experimenting on new capabilities and, features as well, whether it's in the user facing app, whether it's a b to b app that you are building for your customers as well. You are constantly experimenting on look and feel, on models that basically

optimize

either your notification

or your,

the fee that you basically see as well. These are constant,

experimentation as well.

No one

sees a a common experience. It's basically a custom experience that is determined by the experimentations as well. So that can,

so how and,

you can now think about how these experimentations

can be serving different,

platforms to sorry, different capabilities and different,

applications within your organizations as well. So this becomes a very classic use case to actually build a platform,

for to begin with as well. Now the question basically comes in, you have experimentation

capabilities. What are it should basically

encompass it as well. What is that platform definition as well? So the way we, have seen some of these, to play out is,

what are the fundamental things that you need to basically do experimentation as well? So one, you need the ability to understand and target the customers as well. Like,

you when you're on LinkedIn or when you're Amazon, you have billions of customers or billions of users that you, that make purchases or, our decisions on your platform on every single day as well. So, you need the ability to basically segment these customers, and you need the ability to basically run experiments

across these different segments as well. So you need targeting capability to begin with. Right. So then, basically,

is,

ability to basically move through this, targeting segments as well. This becomes a very key aspects,

for example. What do we mean by that? Right? So let's simplify. If you have 1,000,000 users, you want to start a feature sometimes only with 1% of the population exposed, to it as well, where your critical thing is to first evaluate what is the risk of launching this feature as well. We are trying to understand, is it breaking an existing experience?

Is it, causing a negative reaction to certain segment of the population as well? So you randomly select 1% of the population

and introduce this segment as well. But you want to you won't get typically statistical signal data at this point, but you're trying to really evaluate what are the risk of not in the feature as well. Then you basically go up to 50% to understand, like, is this feature really

delivering the value as well? Your goal is to first mitigate risk and then go to the 50% to evaluate

whether the feature is delivering on these capable the capability that we wanted to deliver and the value it is delivering as well. And then if everything looks good and there is statistical signal that this feature has the value, then you go to basically

a 100% ramp and everyone seeing those features as well. And now this is a complex process which needs to be,

managed by the experimentation platform as well. So ramping process directly. And along with that is the a b test capabilities

that are necessary as well. So the at every percentage ramp, we're trying to evaluate different things. So at some point, you are trying to evaluate the risk. At second point, you're trying to evaluate the value addition measured by statistical signals as well. So

that's the key thing. So you now have 3 requirements already within your experimentation platform, the ability to target, the ability to ramp, and the ability to experiment,

our AB test as well. Now not every functions or every capabilities can be experimented as well.

Different teams will have,

our AB tested as well. Different teams will have different needs. So how can they use this basic platform to build additional capabilities?

So our data teams team that is working with,

say, a b to b product can then come and say, I will use the same basic capabilities, but now I do I'm doing

because my products don't get, I cannot AB test on my products, I will use observational,

inference methodology

That is our longer term test, for 6 months to 1 year, and I can basically build on this platform as well. So that's really how you make decisions. 1st, understand, like, what are the common functions that every single team actually need, build those common capabilities.

And on top of it, like, you enable certain APIs and capabilities so that the other teams can take that last mile approach and build cost, solutions for their custom needs as well. And that's really how we think about platforms and,

how the platform enables common set of capabilities that can then be used by other teams to build for their specific needs.

In your work of acting in this product management role, bringing requests to the engineering teams, trying to figure out where your goals fit in their overall priorities and capacity.

I'm wondering if you could talk to some of the challenges and tensions that exist in that relationship and some of the ways that you think about how to

engage with the engineering side of the requirements and work with them to build the capacity that's required and understand how to build a kind of synergistic relationship

that they're building something that they're interested in and that they

want to support rather than just building something because they've been told to and they have to?

Yeah. I think that's a good question. So first and foremost,

gone are the days where anyone in the team basically builds whatever they are told to, unless, like, it's really clear top to mind that as well. And this is really one of the biggest thing that I had talked to upcoming product managers as well, how they have to be inspirational to the teams, how they have set the common vision and ask, inspire the team to basically play towards that common vision as well. It's not an easy job for a product manager to begin with. There are a couple of things that, play a very critical role. Completing priorities will always be there. You have always

have very sim smaller resources around,

like, our challenged resources.

Right? So competing priorities were a very critical role as well. The clarity in thinking vision and, is very important as well. And especially when you basically work in a longer term platform roles, you need to ensure that milestones are critical as well. So you deliver on certain milestones

to ensure that there is continuous motivation for the team. There is a sense of accomplishment wins for the team as well. So I think we have to admit

that we all live in the world of instant gratification as well, and building platforms and products are not easy because it takes years sometimes to really see the meaningful results. So really breaking it down into milestones is, very, very key to motivate the team to continue to work on it as well. So all these things are basic necessity that the product manager has to do, like, inspire the team with a common vision,

have a detailed road map, have a a clear milestones, that are delivered on a regular basis to keep the team motivated. These are very important as well. Without this, like, you're not even basically getting started with any kind of a platform product itself. Now comes the challenging part of how to convince basically the team on specific capabilities and other things as well.

So to begin with, the distinction of, building common platform capabilities, like, the bare minimum to this thing is very key as well. This is where a lot of the tension really lies. If a product manager is trying to basically grab an x capability within a platform,

you have to make a very compelling case to the,

to the engineering team as well and why this is going to give the leverage that, the platform is originally intended to do do it as well. It can be,

you like, simple things, like how many teams can basically use these capabilities.

What is the net impact on business basically because of these capabilities

play a very critical role. So leverage and in business impact, you need to delete the most of it as well.

Then basically tying it to the milestones as well.

If these capabilities

take multiple years to basically build, how can you actually break it down into smaller capabilities,

evaluate

the impact basically delivering, and only then build on future milestones is very critical as well. You need to have that clarity with, within yourself first to convince your

engineering team that this is the right thing to do as well. And this is really where having a, a good strong relationship with the engineering team really plays a critical role. I often say to people that I actually mentor that,

there is no better relationship than an engineering and a product manager relationship. The stronger,

your bond is, the easier do it is for you to do your job and also easier for you to basically

have these kinds of an alignment in a more seamless manner as well. And, again, I'm not saying that you can get away with anything as well. You that build that that trust and that relationship with your engineering partner. The primary partner that you have is built by years of experience by doing the right things for our customers, for our product, for our team,

and everyone around me as well. Right? So this is really where I think, finally, you need to have that strong relationship to make sure that lot of the tough decisions are made. If nothing else work, there should be an escalation channel that should be put in place as well that you should be agreed upon priorly and if we are willing to not align on these things to be done. And it happens more much more often than we think about. But the escalation channel basically helps you to make faster decision so you're not really going back and forth for weeks or months, sometimes as well, so that you can actually make a decision. And last but not the least, you need to make sure that in this escalation, there is

very good clarity to the decision makers as well if this is a one way street or a two way street. Right? For example, what do we mean by that? Right? Certain decisions are one way decisions. That means it can significantly

change, and there is no undoing it,

as well.

For example, the experimentation that I talked about. If you are thinking about building a platform that supports targeting, ramping, and experimentation,

it's a significant decision. And if you, build those capabilities for the next 1 plus 2 years and you cannot go back from those decisions, you you can technically go back, but there's a waste of last 2, years. Our 2 way decision is something like,

that can be

undone with minimal damage as well. And for those things, you need to make faster decisions as well. And, really, in this escalation, making sure that that clarity is provided. That, again, really anchors on the product manager doing his or her job to ensure that

I actually you can actually deliver, this escalation message very clearly and get out of a decision as well. So very yeah. I think the order dynamics is very complicated, but really end of the day, the product manager has to step up and deliver on these things to make sure these conflicts are resolved more amicably.

In that case of building that relationship with the engineering team and engineering management,

you're also working on the other side with the business and sales side. And one of the oldest tropes in engineering is that salespeople will sell something that you haven't built yet. And so I'm wondering some of the ways that you have in your experience been an ally to the engineering team to help advocate for their needs with the business and sales side of the house

and some of the ways that you act as that kind of peacekeeper between the two sides of the organization?

I think if I get a penny for every time I do this, I would probably be super rich by now. It's really the the right way to begin this.

I think it's a very good observation, and I think it's pretty much industry wide

challenge that we actually face as well.

The sales first,

persons have a very difficult challenge of demanding customers.

They are they do one of the toughest job. I I consider dealing with people. That's one of the toughest job and building technology products in this thing. And, again, an engineer will probably completely disagree with me, but, like, people come in all flavor,

and they have own challenges. But so sales persons basically have to make on the fly decisions sometimes to promise something that may or maybe difficult to get it done as well. And that often comes back as a a heavy business push,

as well. That's really where,

product manager basically,

comes into,

a critical,

plays a very critical role as well. 1, I think having a open communication channel within your sales counterpart and your engineering counterpart is key to the needs as well. But trying to understand each other is very important. So that's number 1.

Second one,

often, like, what comes as a build me a Taj Mahal kind of a request can be broken down into

smaller entities that can actually be delivered in a shorter time frame. And that basically can keep both the salespeople and the customers happy as well. That's just a very common, thing that you can actually more look as a product manager if you don't really dig into the details of what is being asked as well. You have to remember that sometimes

customers may, not know what they need. Sometimes salespeople misinterpret what as well. So really digging into the data, there is no substitute to talk to the customers directly as well. So that's the 3rd piece of the puzzle as well. Understand,

the customer problem really, really deeply,

and then basically whatever is as well. And then last bit of of present really is, like, having a backbone to say no is important to sales cycle or to the business as well.

Despite doing all these efforts, certain promises are something that you may not be able to deliver on.

Maybe you you're technologically not ready at this point, or this may not be in the best interest of your business, for example.

So product manager plays a very critical role in having a backbone and saying no and going through the necessary escalations to basically,

resolve this,

is very important as well. So now we are playing a 2 critical role. 1, you're really trying to understand

the salespeople's

intention behind it and the customer intention behind it as well. And, also, you are trying to basically translate that into,

what isn't feasible from an engineering side. And when things you see that you this is not in the best interest of either the team or the product or the business, then having a backbone to push back is equally important. So you're shielding the engineering team from an unnecessary churn

as well. So I think, yeah, this is where really product manager plays a role. And this is one of the reasons why a lot of the organizations are seeing that there's a need for a product management function

in,

data products and platforms as well because these are complex decisions that you have to make on a day to day basis sometimes, and also make sure that both the different go to market teams and the r and d teams

are in sync, and you freely form a bridge between them.

In your experience

of working in this space of technical product management

and working in the data space and with very data heavy products, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?

Yeah. I think,

the leverage of platform

is unimaginable

sometimes, to be honest as well,

because you actually

irrespective of how much of a research you actually do,

to collect what are the common capabilities we need to build, when you really build a platform and

expose it to your customers, both internal or externals, the amount of creativity they can actually do to build on top of the platform

is amazing as well. So I'll give you a very specific example without diving too much into the details,

at this point. But think about,

compliance and regulations at this point. It's really a common

theme everywhere.

Bigger organizations are putting in privacy regulations.

Bigger government organizations are introducing compliance rules and regulations as well to name a few, GDPR,

CCPA,

and they're upcoming on AI models as well, c 2 p a and everything. They're very critical for organizations to meet those requirements

as well. So in, and often, like, sometimes, like, when these kind of variations come, that's when the platforms that you are actually built can be really, really

useful as well. So I talked about,

our, I worked on this platform called Data Hub, which is now an open source product as well.

The crux of basically what this is is a metadata management,

platform, which understand enables you to understand

effectively the data flow within your ecosystem as well by

building lineages between datasets and,

exposing APIs that can actually be used to understand these relationships over the

data as well.

So we built this platform, and that's when GDPR came into feature as well.

The first of all, understanding of our data ecosystem really helped us to meet the compliant needs of GDPR as well. But,

it's not a one time done activity. Right? Like, you have to constantly monitor if there are any compliance violation that actually happens as well. That's why one of the team basically built, an application called compliance monitoring solution on top of it. It the crux of it is very simple. Built on this,

data management,

platform,

Every time there used to be new regulations,

we used to introduce,

a script to basically

ensure that all the regulations needs are actually met by leveraging or building on this type of platform. So if there is any potential violation, for example,

an individual x was working in team y and had access to a dataset. And now that individual moved from team y to team z and technically,

having still continue to have access to the dataset can be a compliance violation as well. In that case, basically, how do you identify and revoke access or ask the individuals to give up access is a critical thing as well. And this is how the team basically used the data,

by the data metadata management platform to build solutions on top of it. Because now you're bringing in all the data together in your ecosystem, understanding it. So now you're able better able to build these kinds of,

applications.

So,

that's why I say, like, platforms surprise you quite a bit with what kind of applications can be built on top of it. And there are plenty of such examples in my,

own experience as well with the teams even surprised,

or even customers even surprised what you're basically, leveraging, what you have basically built.

Obviously, there are major benefits to be had with having a product management capacity

in this data ecosystem.

I'm wondering if there are any examples of cases where either the organization is too small or the focus is not sufficient

where it doesn't matter having a specific product management role for a given organization or a given team.

Yeah. I think definitely. So if you think about

product management,

functions,

but even keep it aside, the product management function. Right? Why do you build in house data platform and products rather than basically,

using,

off the shelf solutions? It's because

you have unique needs. It can be the complexity

of the data ecosystem that you actually have. It can be the scale of the data ecosystem you actually have as well or the customization that you need or ever changing business needs that are important for you to basically have a better control about it. And last but not the least, the compliance as well. Some of these larger organizations,

like,

can be in

severe trouble if there is a compliance violation as well. For example, GTA PR famously

mandated 6% of overall global revenue as well. That's a huge chunk for larger organizations as well. So it's worth every penny

to,

build,

their only now solutions as well to ensure that,

you have better understanding about how you're being compliant and have better control over it as well.

So

there are way too many reasons why large organizations should basically build as well. But if you're a small start up, at this point, I if you're a lot smaller organizations as well, you don't have these complexity, these scales, these requirements as a whole as well. This is really where you have to make more judicious decision about

integrating with the fully managed services

and leveraging open source

platforms too for your needs is a critical thing as well, other than building in house a lot of these capabilities are distinct. And but as you grow,

really making very deliberate choices

about investing,

in some of these,

products and capabilities as well.

Earlier, I alluded every organization

looks back in 5 to 7 years timeline

to see,

do I have to significantly

overhaul some of these capabilities as well? And if you are a start up, you have gone through some of these decision makings, you know that at some point you run into a junction. Like, we are very fair very aware of, the blue whale story of Twitter as well, how they want from a monolithic to,

microservice architecture. Similarly,

every, such junction you need to evaluate if this is the time for you to basically build capabilities in house and search yourself for the next 5 to 10 years of growth. Are you basically

cease to exist in certain case if you don't do that as well. There are multiple instances where organizations basically

stitch together with different capabilities that they actually,

had but did not invest properly,

and couldn't really,

these systems and capabilities couldn't manage the scale.

Lot of this has changed with public cloud right now, with fully managed services as well. So there are organizations that are very effective in not having such large data product

and listing as well, but still manage the business. But when it comes to the cost of lots and flexibility,

expenses sometimes as well. So these are really the trade off that you have to do.

These kinds of organizations, maybe Netflix is probably case in point here, tend to be run very, very lean.

So they manage

their, you know, in that infrastructure for fully managed services and argument it with certain capabilities.

So it's really there is no one size for, fits all kind of a story here. But for certain scale, there is clearly no substitute

building some of these capabilities.

But that's not for everyone for sure.

As you continue to work on investing in product capabilities,

leveling up the engineering teams to be more engaged with the organization

and motivated to support the functionalities,

I'm wondering if there are any aspects of the data ecosystem

and the way that you think about the engineering work as being product focused

or the ways that you think about being a product manager in this ecosystem,

if there's any specific advice for people who are looking to get more involved there or things that you're you're thinking about trying to help promote and encouraging people to do?

Yeah.

Again, I think it's a great question. A lot of what basically, people that I mentor who want to be in product management role understand as well. So I don't really understand and ask this question as well.

I think, to be honest, like,

product management is a more, like, really sought out role these especially these days.

Especially, like, I get a lot of requests from MBA students who used to, earlier before management consultant and our investment banking. They now really want to,

be a product manager in, big tech companies, specifically the Silicon Valley.

And, a lot of them don't really understand these nuances as well. So specifically for a product manager who is really want to break into the space, I think I would say certain things are very important.

Number 1 would be,

that, you have to have a typical,

background to some sort.

Can you be successful in this space without technical background? You can be, but that basically means you need to,

ramp up those skills. Because these are technically heavy product products. And without technical capabilities,

you won't first,

have your respect of the engineering team. 2nd, you won't build meaningful products. 3rd, you won't be successful on a product management. So technical skills basically is very important. That's number 1. Number 2 basically is

some of these, products

don't deliver the instant gratification as well. Hard to measure the impact,

sometimes as well. And these are challenges that you should basically be fully aware of that actually comes with a whole set of other challenges as well. So, people are motivated by incentives. So how do you keep your team motivated

when, you have a product that has been working on for 3 years, for example? Right? A platform that you are building for 3 plus years. So really having,

those milestone based approaches shown that in the milestones and molding the team is very critical as well. And for yourself as a as well as a product manager to make sure that you don't change after instant gratification, but you're really trying to build for the next 5 years, is very critical as well. Then I will say it's very to basically come to terms with, to begin with, because we are now,

often seeking instant gratification as well.

And last but not the least,

I think there should be a passion to solve some of these complex problem as well.

I would say this basically trumps everything, to be honest. Some of these space are increasingly difficult space

to lead as a product manager.

So you need to have a passion to solve these complex problems.

Get up every day to basically solve this problem for your customers, for your business, for your team.

And that's very important. So these are 3 skills that are really these 3 things I would say are very important for the upcoming product management and data platform and product space to be pretty aware of.

Are there any other aspects of product management in the data ecosystem,

your specific experience in that role, or the technical capabilities required to support these end user facing features that we didn't discuss yet that you'd like to cover before we close out the show? I think,

if you have if you are a product manager without technical background, there are plenty of,

capable

courses that are no are already available as well. And even if you are basically a product manager with a lot of background, as I said, the technology world is very dynamic. You have to basically continue to refresh yourself. I, myself, am going through

much more detailed

AI courses as well to a level, basically, I can code and build my own models as well. Right? Like, really understanding,

going to the depth is very important,

as a product manager, we want to basically,

understand the challenges of your engineering team.

Number 2, basically, is to be in sync with the technology trends as well. So I would say it's very important for you to basically do that. So I think I will, my well, probably kind of an advice would be to

really feel really understand the technology goal or the depth of, understanding it, live a life of an engineer, for example, to be successful product manager,

in this, in this capacity, particularly.

Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today. Yeah. I think, like, this shouldn't be a surprise to anyone, but specifically the advent of GAI the last few years,

we are looking at,

astronomical base of innovation that is, powered by AI in the next few futures as well. Every organization is basically investing in some other capacity as well. That basically introduces a whole set of challenge related to data, related to tool,

to make sure that these AI foundational models, number 1, is available

to everyone. They can be basically used in the right manner as well. And any any rack based approach that you actually use to augment these models, you have the right data, you know, that is readily available, can train these models, and go live as well. You have the necessary infrastructure

to experiment

with these models as well. These are all the keys that basically

there are,

there are certain of these areas still have the capabilities, but they are not really upgraded to a level that,

a piece of innovation needs at this point. And I would say, like, a lot of the organizations are really thinking about this, how to basically

upgrade their infrastructure, their own tooling, their own capacity to really cater the deeds of,

AI and the innovation that is now following.

Well, thank you very much for taking the time today to join me and share your experience working in this role of product management for data systems. It's definitely a very interesting position.

It's one that is continuing to evolve and change, particularly as the expectations around capabilities for these different end user products are evolving

along with the technology landscape. So it's great to have your perspective and your advice for people

who are working in the space or working with somebody who's in that space.

So thank you again for taking the time, and I hope you enjoy the rest of your day.

Yeah.

Yeah. And

and thanks for having,

me, in the session as well. This was,

it was great to basically share some of my experience as well. And,

yeah. Looking forward to other new things as well.

Thank you for listening, and don't forget to check out our other shows. Podcast.netcoversthepythonlanguage,

its community, and the innovative ways it is being used, and the AI Engineering Podcast is your guide to the fast moving world of building AI systems.

Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at data engineering podcast.com

with your story.

Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.

Data Engineering Podcast