Stretching The Elastic Stack with Philipp Krenn

Hello, and welcome to the Data Engineering podcast, the show about modern data management.

When you're ready to build your next pipeline, you'll need somewhere to deploy it, so you should check out Linode. With private networking, shared block storage, node balancers, and a 40 gigabit network all controlled by a brand new API, you've got everything you need to run a bulletproof data platform. Go to data engineering podcast.com/linode

to get a $20 credit and launch a new server in under a minute. And for complete visibility into the health of your pipeline, including deployment tracking and powerful alerting driven by machine learning, Datadog has got you covered.

With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you'll have everything you need to find and fix performance in no time. Go to data engineering podcast.com/datadog

today to start your free 14 day trial and get a sweet new t shirt.

And go to data engineering podcast.com

to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. Your host is Tobias Macy. And today, I'm interviewing Philip Cren about the Elastic Stack and the ways that you can use it in your systems. So, Philip, could you start by introducing yourself? Hi, Tobias.

So I'm Philip. I'm based in Vienna, Austria, and I work for Elastic. I started off for I'm still spending some of my time in the infrastructure team, but I'm also a developer advocate now. So I am pretty much hopping between conferences

and meetups around just to kind of spread the word of the good stuff that we're doing. And how have you found the transition from being on the infrastructure team to being more forward facing and doing the developer advocacy?

So it was always planned that I would spend a good amount of my time at conferences and that's kind of like how I got too elastic because at my previous job I started to realize that I actually do a lot of conferences and that I enjoyed it. So that was also always the plan, but now I've kind of like pretty much full time transitioned to that. Though I I kind of like doing conferences. On the other hand, I like that I have my engineering background and mostly my ops background and I still

use that a lot in all my demos. So instead of kind of producing something for my colleagues to

use internally now, I cut just got to play or just got to create some demos that everybody can play around with, which is good fun, but it's kind of similar still, so no regrets definitely.

And how did you first get involved in the area of data management?

So at the company where I was previous, I was kind of the ops and database guy and

it started way back at university when NoSQL was kind of the hype and I wanted to do something in NoSQL

area.

And then I started to work in a startup

and well, we picked MongoDB back then because it was older age and I started to do trainings and lots of stuff around,

no sequel systems like I was using Redis. I was using Elasticsearch back when it was

just released. I remember

2010 was the first time I used it.

I played around with a lot of other systems like Cassandra and Neo4 j and Hazelcast

and whatever. So that's kind of like where I got started

and at some point when I reached that point that my previous job where I saw well, it's kind of time for the next step,

I kind of accidentally saw that Elastic was looking for somebody

in my area who would speak German, do conferences, but also, work in infrastructure stuff and I thought, well, that might be a great opportunity. So I kind of jumped on that on that ship and that worked out pretty well. And so the elastic search project has been around for a long time and has become fairly widely known for a variety of use cases.

So I'm wondering if you can give a brief overview of the other components that have been built around that to create the Elastic Stack and how they all work together.

Well, let me take a step back first. So Elasticsearch,

is the oldest product, but that's already kind of the 3rd iteration

because, Shai, who started Elasticsearch, he had 2 other search products before he even got started with Elasticsearch, which were called Compass 1 and Compass 2. So Elasticsearch is kind of the 3rd iteration already or a full reimplementation

and I guess 3 is the lucky number. So that's the 1 that

stuck around and really got started. So that's kind of Elasticsearch is at the core of everything what we do. So if people don't know, Elasticsearch is based on Apache Lucene,

the search library

and Elasticsearch is then kind of the thing around it that makes it work in a nice way. So we have a nice rest API. We provide a query DSL. We do all the distribution and replication of your data for you, and that is at the core of what we do. And all the other fields we have kind of added over time, we always see that as an addition to search.

The search is always kind of the core of what we do and what we will keep doing, but there are so many other use cases that you can kind of add on top of this search use case. So the first thing that people wanted to do is at some point they want to visualize their data and see what is going on to build dashboards

that are easy to build and nice to visualize.

So that is what

Kibana is for, which started off as another open source product on its own and then we kind of joined forces with Kibana.

And then there was this other project Logstash to get logs, but more generally any kind of data into Elasticsearch.

Elasticsearch and those 3 together formed the famous or infamous

ELK Stack, so Elasticsearch Logs Desk,

which already was very widely used for logging,

but it's not really stopping there. A bit later on we added beats, which are like lightweight agents or forwarders.

Does the written go

So

lockstash

started off as a Ruby project, is now JRuby mainly for performance reasons, but you always have this JVM dependency then. And if I talk to people like in the PHP or Ruby ecosystem and tell them, well, just install the JVM on all your servers to collect your logs,

people are very much less than thrilled. So that's where the beats come in. Their written in goes, you have your native binaries to, like,

ship your data, which could be other log files, metrics, network data. We do security data. We can ping services and check if they're available. We can even do windows event logs even though I try to stay away from windows,

but we covered those as well.

So this is the open source part of what we do and that is what we call the elastic stack now. We'll probably never fully get rid of elk just because it's

a very cute. You have the the elk. We even had like a plushie elk as a toy to give away

and 1 of my colleagues is still always taking pictures with the elk. So we're not getting rid of the elk, but we are trying to make it a bit broader and just call it elastic stack by now. And also since our products are always about scaling,

even for marketing, scaling is kind of an issue and they figured out, well, we take Elk and then we could try to find another acronym, like, since we have Elasticsearch Loxus Skibana for Elk and then we added Beats, we tried to experiment with the name Belk or Elkbee

and we even had a logo for that. So it was like,

bee body and with elk horns,

but then since marketing is also concerned about kind of like how can we scale the product, what happens if we add another product? Like, what if we have add another open source product and then we get another letter? Then we need to redo the entire,

branding and also kind of make up another animal.

And the more letters we get, the harder that will probably be. So we kind of stuck with

the optimized or scalable version of just calling it elastic stack now. So whatever we do and whatever open source products we add, we can just stick them into the elastic stack and it's still called the elastic stack and this does all the things,

that we're doing.

Yeah. I think that that was probably a smart move particularly given the fact that you now have the Xpack as part of that overall

set of products. And so I can imagine how you would try to incorporate an x into it

into the broader acronym. So

Right. So, Xpack is not open source. So Elastic Stack, that is really the open source parts that you're using and then we have these,

partly free, partly commercial,

extensions in xspec, which, yeah, which is mainly

kind of keeping the company alive since making busy, money out of open source software is kind of a very delicate issue, which we are always trying to work out. But, yeah, it's not always easy.

The original

usage pattern for Elasticsearch

was generally for utilizing it as a search back end for doing full text,

for doing queries against full text indices

usually coupled with some form of an application.

And as the various other parts of the Elastic Stack have been built up around it, some of the usage patterns and use cases for the overall

environment have expanded beyond that. So I'm wondering if you can enumerate some of the other target use cases that Elastic

is focusing on for added functionality

beyond just that core, application search functionality.

Right. So Elasticsearch is widely used for full text search, so I guess most listeners are kind of familiar. If you search on GitHub, Stack Overflow, Wikipedia, like, all of these behind the search box there is Elasticsearch doing the search for you. But then, yeah, with the Alex Stack, we very much branched out into that log management

problem, especially since applications are getting more and more distributed,

finding and centralizing logs is becoming a bigger issue.

Previously, you could just SSH into the 1 box and then do a grab or search with less or tail or whatever,

but that's not going to work out if your services are spread out over dozens of services. So centralized log management is a big thing

and that's where the yelksac

came in and solved a lot of problems but we have kind of branched out from that. So with the addition of beats, we have made a metrics a first class citizen as well. So it can be metrics like CPU and memory usage. It can be some data that you collect from your application.

We can hook into various things like we can get metrics from MySQL Nginx, Apache.

We can get JMX data to hook into your Java applications

and loads of other things. So metrics are now a first class citizen also kind of as for the storage layer because sometimes people are concerned like, okay, full text search engine doesn't sound great for, metrics,

but that is highly optimized by now as well. And we keep continuing to branch out into other areas, which we all think makes sense to have in that stack. So we've recently added an APM company for application performance monitoring,

which also we always say, they join the family. So that APM

company joined the family, and we're trying to branch out in that APM world with tracing now.

And they're also trying to build more and more solutions around what we have because sometimes, especially in the past, the or the products we have were a bit barebone. Like, you always had to do some

assembly to do. I always compared it to kind of Lego. So in Lego, you have all these building blocks and then you need to assemble them. But sometimes people don't want to spend too much time on assembling stuff. They just want the solution. And that is always a bit of what we are working right now is to get more into this solution space. Where we just have something that will work out of the box very easily for you and you do have to do less of this plumbing work.

On the other hand, you will always be able to do that plumbing and kind of build whatever you think makes sense for your use case, and you have a lot of freedom. We just want to make it easier for beginners to start out of the box with somebody who just wants to find a quick solution and do a bit less of that plumbing. But you will always be able to still do that and build whatever you think makes most sense.

Going back to the point that you were making about the metric storage

and the fact that the storage engine was originally created and optimized for managing textual data,

Was there any re engineering

required for being able to handle more numerical data and being able to search and aggregate across that?

Right. So,

numeric data,

and also aggregations are totally first class citizens in Lucene now. So we have a couple of Lucene committers

at Elastic as well, just because we need all of these features and they keep improving a lot of stuff that we just need there. We've also, worked a lot on the geo stuff. So 1 of my colleagues

called Nick, we all always call him GeoNick because he's doing all the geo stuff.

He has reworked a lot on that side, but also,

any general numeric data has been optimized. Like we added or there were different data types were being added. So instead of just using floats, we do now have half floats, which basically give you half the precision for half the space or half the storage space,

or scaled floats,

which are stored as like an integer number but have a scaling factor. So if you, for example, have a scaling factor of a 100, you basically you store a value to get to the original. You would divide it by a 100. So that is the scaling factor. So with these data types, for example, we use less

storage on this, to make that more efficient. And there have been a lot of, improvements around numeric data overall,

in Lucene and Elasticsearch.

And

another thing that you mentioned earlier is that Elastic

as a company and as a product is very focused on being scalable.

And 1 of the primitives that Elasticsearch

adds on top of Lucene is the ability to distribute the index across multiple instances.

But

as with

as with anything that you're trying to scale, there are certain complications that come up. So I'm wondering what you have seen as being the most common bottlenecks that

users come across when trying to scale their systems using Elasticsearch

and some of the edge cases that they should be aware of when they're trying to deal with large volumes of data?

I think we had a lot of pain points like in the beginning of the product and then we had some customers who were like having a lot of trouble.

Recently,

we have done a lot of refactorings and improvements.

So stuff should work out a lot better now and be more stable.

So depending on your size,

for like the

the medium sized, projects,

if you're somewhere, I don't know,

In a terabyte range of,

of data, I wouldn't expect,

you to have any major issues because you have a lot of people who are doing that. Yes, you will need some hardware to accommodate that, but there shouldn't be like any major issues or any any major things that you will need to

to optimize for that much. Like it used to be worse in the past. It has improved a lot and we're also working a lot on the internal protocols. So there is a subgroup within Elasticsearch just working on the distributed

parts to make,

the sharding, which kind of splitting up of data and the replication much better like the protocols internally

and make failures less likely, recovery quicker. So there are a lot of improvements,

going on on that.

The 2

main things I always try to point out, these are pretty basic and a lot of people might have heard of them but this is kind of like the the main stumbling blocks that people do is,

the first thing is that they have too many shards because every shard has some overhead in terms of file handle and memory

and if you're searching across a lot of charts, you will need to combine the results from a lot of different charts as well. So this might add a lot of overhead and we've seen,

especially in the beginning a lot of projects which were prone to this, over sharding because people thought like well how many shards do I need? The default is 5, but I don't really know what I will need in the future. So we'll just use, I don't know, a 100 shards per index.

And that will probably add up to a lot of charts which are kind of, like, too small.

So depending on who you talk to, 1 of these charts should be somewhere between

10 50 gigs of data.

Simon, our tech lead for Elasticsearch always says, like, start with 1 chart

and then see when it blows up and when it blows up then you can think about having more shards.

Not the other way around of thinking like, oh, I'll just add a gazillion shards to see what will happen because that might be much more scalable in the long run. So that's 1 thing to be aware of

and the other thing is not specifically about sharding, but something that is kind of like 1 of the main problems or main misconfigurations of Elasticsearch.

So we have masternodes and these masternodes keep,

track of the metadata of the cluster state. So kind of like which data is located where, what does your mapping, which is like the schema and relational data base, look like,

which nodes are even part of the cluster. And of these so called master nodes, for a production environment, we always,

require

or recommend to have 3

dedicated masternodes.

And then there's this 1 setting where you define minimum,

masternodes,

which basically defines, like, how many of these need to work together so that you can kind of form 1 or elect 1 master

to make changes to that cluster state. And that always needs to be set to the majority

of all the eligible masternodes in your system. So if you have 3 eligible masternodes in your system, the minimum masternodes need to be set to 2 to have that majority.

And if you set that wrong or don't set that value at all,

then you have the chance of having a split brain situation so that kind of, like, the network force apart somewhere in the middle and then 1 node thinks, oh, I'm the master node. I don't see the others, but I would just keep working on that And then everybody connecting

or every part of the system being connected to that master node might go into a different direction than the remaining 2 nodes.

So So you will always need to make sure that this minimum master node is being set. That is 1 of the main causes for trouble in a cluster. So if you keep track of or keep kind of a reasonable bound to the number of shards and set the right minimum

the right number of minimum masternodes,

that is normally the most or those are the most important things that you should do. You can tweak a lot of other settings. Of course, you should set the heap, which is kind of like half of the available memory because half should go to the heap, half should go to, file system level caching of your data. So these are also important settings,

but they are normally not that catastrophic and sometimes

or quite a few times actually we see that people make very advanced configuration settings and sometimes they're kind of making it worse.

So you're adding some very fancy,

JVM flags, which we have never tested and people don't test them that well. And in some case, they might just break in an unexpected fashion or have, like, some unexpected side effects.

So we're would normally be very conservative with adding, like, lots of fancy settings. And

as always with benchmarking,

change 1 setting and then benchmark it the proper way and only then change something else. But yeah, as with every product that's had some complexity,

the answer will always be for your use case, what you need to do, it depends a bit

because there are a lot of ways to use something and a lot of ins and outs that you need to be aware of. So,

yeah. Like, the global rule, like, this is what you always need to do is very hard to do.

Just be careful and benchmark the right way. And in terms of the

total scalability of the system, what kind of support does Elasticsearch

have for being able to do geographically distributed

deployments

where you might have instances across multiple data centers and what are some issues that people should watch out for in that kind of context?

That is something what we are actually,

working on,

right now.

So at the moment,

1 cluster, so all the nodes working together on 1 problem. Basically, 1 cluster

should always be in 1 geographic regions. People sometimes try to stretch that

as long as the latency is like, single digit milliseconds,

it might be okay, but generally,

1

cluster should always be in 1 geographic region. If you're hosted on AWS, different availability zones are definitely fine, but different regions are kind of problematic, for example. What the colleagues are now working on is,

cross cross

cross cluster replication

so that you can replicate the data to a remote cluster. Company, like you're a big gaming company and you

want to store data in lots of different regions,

we can do a cross cluster search. So 1 search query can actually spend multiple clusters right now, which is very common. For example, we have some very big gaming companies. They have different

geographic regions and have different clusters there, but they still want to get the overall picture of them. So,

they use cross cluster search to search across all of these clusters. The cross cluster replication in Elasticsearch

itself

is currently work in progress. No guarantees when it will be done. It's quite an engineering effort, but it's well on the way and we'll see when we will get there.

Other forms to get your data into different data center might be taking a snapshot and copying that over or if you're using any queue, you could just write that data into a queue and multiple Elasticsearch clusters in different regions could consume out of that queue, for example. But, yeah, we're still working on the geographic distribution of data within Elasticsearch capabilities. That is something that is work in progress at the moment.

And your point about being able to ingest from a queue reminds me of 1 of the new capabilities

in some of the recent releases of Elasticsearch

in that you can actually,

in some cases, bypass

Logstash

for doing some of the transformations

in your,

source to destination pipeline, and you can actually perform some transformations

as part of the

record ingestion within Elasticsearch itself. So I don't know if you can speak to that a little bit and where it makes sense to use the capabilities directly in Elasticsearch search versus what Logstash or other queuing mechanisms might provide.

Right. So in version 5.0,

which came out quite a bit over a year ago now, we have added this new node type called an ingest node. So in Elasticsearch, node type ingest node. And that can can run a subset of the features of Logstash. For example, if you want to use, grok patterns, which are like regular expression or a named regular expression to parse fields in the different pit,

so 1 bit longer line into different parts that can be done, in the ingest node. The ingest node can also do stuff like a geo IP lookup, for example, or it can filter out or

extract the meaning meaningful information from a user agent like the browser user user agent that you send so that it can find out, okay, that many people were using, I don't know, Internet Explorer or Firefox or that specific version, that operating system. So these are things that you can do,

in Elasticsearch in just notes. It doesn't have all the bells and whistles that LogSash has. So people are sometimes confused like will LogSash go away since you have so many features now in ingest node. I don't really see that happen.

So

Logstash will still remain,

maybe ETL,

the extract transform load is kind of the wrong term, but it's kind of something like that since Logstash has a lot of plugins to speak to different servers or get data from different servers, then filter or transform the data and then write it out either to elasticsearch but also to other systems. So we could write out to a flat file for example, or we could forward the data to a queue again. So Logstash definitely has its kind of like part to take over and it's not going away. It's just the use cases might be slightly different. For example, if you're using the beats, these can be configured by default to just talk to Elasticsearch directly and do the transformations that they need within Elasticsearch and then you don't have this additional component of running Logstash. If your use case gets more complicated

at some point or for for some features, you will probably still use Logstash.

Though it's not always required to have Logstash right from the start. Though that doesn't mean that Logstash is going away. Don't don't be afraid.

We are still hiring Logstash developers,

so it's not going away anytime soon. And another use case where it makes sense to have Logstash as the intermediary

between your source data and the Elasticsearch destination is from a security perspective because

the Elasticsearch product itself doesn't natively have a lot in the way of access control capabilities, whereas Logstash is a little bit more featureful for that. So you don't necessarily want to expose Elasticsearch directly to the Internet, whereas you might be able to have an endpoint on an Elasticsearch aggregator for feeding the the data into and then have that as your transport to the Elasticsearch

cluster that's behind the DMZ.

I'm a bit hesitant to agreeing that we we don't have,

security natively because well, as we talked before, we have Xpac,

which provide the security features

and that even though it is a plugin, it's kind of developed by the same team and

that is pretty much an native solution. However, it's not open source and it's not really available.

So, yes. In that regard, you don't want to expose your Elasticsearch nodes to the Internet. By default, we've changed it, I think, in 2.0.

We don't or we own or Elasticsearch only binds to local host anymore. So, yes. Some people still got ransom like

it started a year ago so,

that people try to take your data hostage and then ask for bitcoins or whatever other cryptocurrency.

Don't do that. Either just bind to local host and just use a reverse proxy or get xpack, which are the proper security features to properly secure your cluster,

to secure that. I don't think that Logsys is kind of like a good security solution. It might be kind of like an intermediary. Yes, you cannot talk to Elasticsearch directly,

but I wouldn't try to sell lock just as an explicit security solution. Absolutely. And on the topic of plug ins, I know that Elasticsearch

actually has a fairly rich ecosystem of plug ins for being able to add enhanced functionality, whether that's for making it easier to form clusters in different cloud environments

or even adding new query types or data types within Elastic search. So I don't know if you can speak to that a little bit as far as what's possible within those plugins.

Right. So

there are there are various plugins either provided by us or people from the community,

to extend functionalities.

These can be for example, specific languages.

So if you take a complex language, I think like Polish, most people don't need all the rules,

for Polish and it doesn't make sense to to ship that.

But some people actually do and then you can load a plugin to add proper full text search features or more advanced full text search features for Polish, for example. The same is true for a lot of Asian languages.

Since that would add a lot of both download size

to since you would need to include a lot of additional code there, as well as memory constraints since you probably would need to load, a lot of stuff into memory that you don't actually need. Not all of that functionality is bundled into the Elasticsearch core download, but you can extend those,

with the plugins. So whatever you need language wise, or feature wise, we have a good selection on the website which links I think to all the ones that are being provided by the community,

to add features like more

more or specific languages

or specific use cases. I think there is an emoji plugin and, yeah. There are various things what you can do, with your data to extend that functionality and

there is, an API and you can just write something to do that. And some of us are provided by us officially and some of them are some of these plugins are being done by the community

or by Elastic employees on the side. So for example, 1 of my German colleagues,

he has a plugin to detect the language of a string because by default you will need to tell Elasticsearch, for example, that

this string is English or German or French or whatever and it doesn't have any way to detect it. But 1 of my colleagues wrote the plugin for the ingest node and so for ingest node that can

analyze a string and then say, okay, I assume this

is English and then run the right,

search

features on that for English.

So there are a lot of features that you can add when you need them.

And moving to talk about the Kibana project,

I know that that is largely used for the Elk Stack for being able to

search and analyze and create dashboards for log data, and it can also be used for other cases of dashboarding and presenting information. So I don't know if you can speak a bit to how that can be used for

promoting data sharing and self-service data access within a company, particularly

in the context of using Elasticsearch

for storing log and metrics data?

Kibana kind of started off, to visualize,

and we always say it's there to democratize your data. So whoever needs to find something in your data can just use Kibana. We try to make it as simple as possible. So even if you're not really trained on that or you're not a data scientist, you can just click together, some visualizations and combine them in a dashboard

to get a good picture of what is going on in your data. Because just with the raw data,

either you don't see the information or you need to write queries, which might be hard. So Kibana should make that easier.

However, Kibana also evolved a bit now.

We see this kind of like the management part for the entire stack as well. So we are getting more and more

UI elements to see, for example, like which components are running in your stack to monitor them,

to

create alerts in a visual way. And Kibana is kind of like more of this centralized way,

both to see what is going on in your data but also a bit to manage your entire stack

and aid you in that. You can, for example, run tools to analyze queries and see,

how are they performing. It's a bit like an explaining the SQL world and Kibana provides a visual way to see, what is going on there.

We provide a nice

UI.

We call it console to actually interact with Elasticsearch queries, which adds syntax highlighting and auto completion,

and it's just a very long text box where you can

add all the queries you want and you can keep them. It's much more convenient than using plain curl for example. So that is the role that we have for Kibana and Kibana definitely keeps expanding and adding more and more features as well as visualization. So we have a kick ass visualization team

and they are bringing new shiny toys, on a quite frequent basis. And I know that 1 of the things that I have

seen that was attractive for my use case as an operations engineer

is with the Packet Beat data collection

and being able to feed that into Elasticsearch and then use the network map plugin for Kibana to be able to actually view the overall

architecture of your network and see where the,

see what the latencies are between different points and see which services are actually communicating with each other and via what proxies. So there are definitely a lot of interesting capabilities

being added into Kibana

for various other use cases beyond just

seeing the textual data within the system.

Right. So once you have the data, you can build a lot of cool stuff and stepping back again, in case anybody is not familiar with Packet Beat. Packet Beat is a lot like Wireshark.

Just,

there to centralize all of that information that you're collecting. Since Wireshark is cool, as long as you're just running it on a single node,

We're using the same base library, libpickup,

to collect the data and basically,

what packet beat is then doing is extracting

the relevant information from the headers. For example, it knows, oh, this is an HTTP request. This is the response. The whole thing took a 100 milliseconds.

You requested the URL foobar

and the return code or response code was a 200.

So that is what packet beat is doing. And once you have all of that information, you can, of course, start visualizing and see who is communicating with who, where is the latency, where are the errors,

what is going on in my network. And we have the same, well, for log files, for metrics.

We're pretty good with monitoring Docker or Kubernetes now as well.

So we're trying to get more and more data out and then put everything into 1 central place where you can see what is going on and build whatever visualization makes sense for you. And given the breadth

of usage cases for Elasticsearch

and the various products that have been built around it, I'm wondering

what products or services

you see as being the biggest competition to the overall Elastic Stack as you

continue to expand the capabilities.

That's an interesting question and I must admit

I had a bit of a mind shift when I was at Elastic about that because while we keep, of course, track of the competition,

we don't really want to be defined by them. Especially,

since Elasticsearch started off with just search and then the competition was very different to what we have today because we're in so many areas that we have probably a lot of competitors in every single 1 of them, but we don't really try to focus on, oh, this is the main competitor and we need to try to beat them.

But it's more about, okay, we have this awesome stack and then we kind of like try to push it further and make it broader and add more features and capabilities to that. So we're not very,

I don't know, focused on saying okay, this is the main competitor or this is the main area where we are focused on.

We're just trying to build something useful and are not that concerned like who is doing what and how do we catch up with them. I think we're more focused on ourselves and building our own thing

and by that, I hope to have we just hope to have a better product

than the others and then we don't need to think so much. Oh, what is everybody else doing? Yeah. It definitely seems that that that would be difficult to define any single competitor because of the fact that there are so many different areas where the Elastic Stack can be used, whether it's in monitoring and metrics or log aggregation

or as a search engine, whereas most companies are going to be focusing on 1 of those verticals and not necessarily working across them.

Exactly. But we see a lot of value in actually covering

a lot of use cases together so you don't have 1 tool per kind of problem, but you have like 1 tool that handles a lot of that and I think that is 1 of the main advantages that we have.

And because we do so many different areas we also don't like stuff like magic quadrant or something like that. We're being put into 1 box and you're kind of confined to that box.

We try to get out of these little boxes and build something a bit bigger and broader.

And looking forward,

what do you see as being the biggest challenges

that you're tackling in the elastic stack, whether it's technical or otherwise?

So my opinion that

that is very personal probably,

depending on who you ask they might have see different challenges.

I think, we are growing at an

pretty

crazy pace right now like in terms of people and also products.

And I think

a kind of keeping the right company culture

is an issue.

The other thing is also like building a compelling package out of that because you have so many different products and when every time you add another company it's kind of like

slightly different product and you then need to work hard to actually integrate that in a nice way with the rest of the stack. So I think that integrating that and

building 1 compelling package, that is an ongoing struggle that we have. We are very much aware of that,

but that is something,

that I see as a trend and I think that is 1 of the things,

that is kind of the main challenge. How well we can execute to integrate everything into this compelling package,

to make it easier for everybody.

And

talking a bit more about the business now, I know that the main focus has been around open source

and how you can build these open source products

and then have some sort of monetization capability around that rather than crippling the product offering in in favor of licensing. So I don't know if you can speak a bit about the

ways that Elastic is able to sustain

the continued development

of the overall technical stack while still having a supportable business model.

Right. So

stop me if I'm going too far. This is kind of like a very interesting topic and it's pretty deep I think,

because, yeah,

finding or forming a working open source business is kind of main challenge.

So Elasticsearch

started off as an open source product being Apache 2 licensed and all the open source stuff that we do is still Apache 2 licenced. So we want to be very permissive and just give you this stuff without, making a GPL or even a GPL and kind of like constricting what you can do with that.

On the other hand, you need to be able to kind of build

a working company that makes some money because we're by now more than 800

employees

and well,

I I want to be kind of paid at the end of the month or I I need to,

like all the other colleagues. So yeah,

there needs to be a working company. So generally, I think there are like 3 things you can do if you're an open source company what you do. Like the,

probably the oldest model is to have, like, some

support or training,

capabilities around it. That is pretty much what Red Hat is doing. They they have their products and if you want to get like a version that is supported,

and you can actually ask somebody questions and get a response in a specified time,

that is what you pay for them. We do have that. So support is important for us and we totally do that.

Part of the problem of support is generally

that the renewal rates are often challenging because after 1 or 2 years, people check or when you have to renew your support contract, people check and say, oh, we didn't have any major incidents

and kind of we know the product by now. So why should we pay you so much money?

That leads you kind of in a bad position because you don't really want to make your product worse or harder to use. On the other hand, if your product is too good and too easy to use, you're kind of starving your own business and that's not a place where you want to go.

Second area is normally what you have is like open core where you have the open source foundation and to have some additional features around that.

That is also what we do with xpack. So we have the Apache 2 licensed code that we

give away for free and everybody can use, change, share, modify.

But we also have some features, which we think add value for businesses

and well,

we need to take some money into keep working on awesome products. So that is what we're doing,

with with expect, which is, for example, alerting features, security features.

We have now a machine learning component, which is a nominee detection of time series data, which is very relevant

for, operational data.

We have a graph functionality

to explore your data,

and we're adding more and more features that are paid. We have

and the third thing that you can do as an open source company is having a cloud service,

which we have as well, which is called Elastic Cloud. Kind of makes sense from all the elastic branding,

where we provide elasticsearch and Kibana as a service. We provide different cloud providers or we're available on different cloud providers and you can just say, I don't know, I want 1 instance with a certain amount of memory and this space spin me up a cluster, give me a nice interface,

to

to upgrade the version or to resize that.

We also add the commercial features in there,

so you get all of the security features, for example,

to make your life easy. And that is another way where we are making money.

Just the problem is,

some of the big cloud providers are kind of competing with our services,

which is totally legal. It's just kind of a problem for a company if somebody else is kind of taking away the money or making money out of your products without

really contributing back.

And kind of to complete

conclude,

we're making 1 big change now,

where xspec, which used to be closed source,

will go open. It's not going to be open source in the kind of strict sense of open source because I think in the there's an an OC definition of open source, which means it's freely available. It's modifiable,

and you can freely and you can distribute it.

It's not going to be in that sense. It will be open code that you can

see our commercial,

or intellectual property.

You will still need a commercial license to use that in production,

but you can actually see the code and you can probably

find or check the code to see how stuff is working. You're totally able to contribute back,

bug fixes,

all the features if you want to,

though that is not the main reason.

We want to just be a bit more open and share a what we have and show you, okay, this is good code and this is stuff you want to use,

but it's also there to make our life a bit easier because previously we had kind of this weird split where some of the stuff was open source and some was closed source. And for the closed source stuff we also had always had private repositories and then we had just private GitHub issues. And if somebody said, oh, there is a bug, we didn't have a good way to communicate with them and also we had this weird split that you have different products you need to build them, which made our,

CI pipeline pretty complicated.

It's just much easier since now all the code will be open and on GitHub.

It's just what we had under Apache 2 license will stay under that license, but we'll open up the commercial or the commercial intellectual property that we have. The code for that will also be opened. So it will be viewable, but you will still require a commercial license to use it in production.

Unfortunately, that is a not very common model

and it seems to cause a lot of confusion. So in the upcoming version 6.3,

we will make that change and we're still working on that license since there are not that many people actually having such a license.

We're kind of like working that out with the lawyers and there are a lot of interesting edge cases and yeah. It's kind of interesting that the company very much depends on that intellectual property, so we shouldn't screw that 1 up.

Yeah. It's interesting discussions.

I'm normally trying to stay away from legal issues, but this 1 is actually very interesting.

How you can make your code more open

while still making sure you can succeed as a company.

Yeah. It's definitely an interesting challenge

and an interesting move on the part of Elastic to make the code visible because it does potentially open you up to people copying the

functionality

without,

actually using the license, but that's where it gets into the legal areas of intellectual property law and the complicated areas thereof, which I don't think we need to dig into here.

Yeah. So to be honest,

we know that people have done that in the past already. So since Elasticsearch is Java, decompiling Java is is not a big thing. You can do that.

But we don't think that people who would generally be willing to or be able to pay us, they would not do that. And if somebody else hacks the license, they are probably not or they would not pay us in any way

anyways. So I don't think there's any business to lose on that.

And we're not trying to screw anybody over,

with kind of like, oh, making everything,

with a paid license now. That is not the plan. We're just really trying to open up the code and making our own life and also

visibility for customers easier.

But, yeah, we'll see how that works out and I'm I'm looking forward to that. And I assume I will get a lot of, confused questions and I've already seen a lot of tweets where people said like, oh everything will be free and then you always have to tell them no. Sorry. You kind of misread that statement,

But it I guess it's the case that people always read what they want to read. So a lot of people kind of like hope to get everything for free, but why we would love to do that, that's not sustainable as a company. Absolutely.

Yeah. And when I was reading that post, the, with about the announcement of opening XPAC, 1 of the other motivations that was mentioned there is that

by incorporating

the source code in the package distribution when you're installing Elasticsearch, it makes it easier for people to

be able to

incorporate that into their deployments who are paying for a license. And it also might potentially make it more likely for people to actually obtain a license because of the

easier distribution of the additional functionality.

Right. And 1 of the change that we're making is that by default, so there will still be the open source only version in the packages you can get. But the default version you will get is have this code in x pack that is free. You don't need a commercial license. It's really available, but was closed source so far.

That will be bundled in the default distribution and that's that contains,

some very neat features as well. So we're looking forward to kind of share that and get that out to more and more people. You won't even need to register like you will get, an unlimited license

both in times and number or size of deployment.

You will just get that for free,

to share that.

But again, that is kind of like or sometimes people are asking like, why are you even doing that with a free license? Why is it not just open source code?

That is mainly again,

competitors providing our services as a service.

They're legally not allowed to offer these services.

So we're kind of trying or letting people, like, use that

on premise,

in any way they want.

But if you want to have that as a service, you will need to come to us and use us for yours as your service provider

just to funnel back some money to the development of all the tools.

Otherwise, the whole system won't work. And yeah. Sometimes people are very much like everything should be free.

But if you think about it, nobody really wants or can work for free,

over a prolonged time. And I guess everybody wants to get a paycheck at the end of the month. So, yeah, it's the only way to get there.

And

are there any new features or functionality

in upcoming releases

of Elasticsearch

and the products in the Elastic Stack that you, would like to mention that people can look forward to? I mean, what we've already talked about was the cross cluster,

replication that you can replicate data to a remote data center.

That is, an interesting feature.

Other than that, we just had Elastic on our annual conference.

We demoed some very cool stuff. So 1 of the things that is coming up for Kibana, for example, is Canvas.

Canvas is or canvas can kind of get data in real time,

but you get like a big canvas where you can freely create whatever you want. It looks like a presentation tool. So, if you view it in full screen mode, you don't even know that this is Kibana or not like some presentation tool like keynote or PowerPoint.

The big advantage is that it gets the data in real time. So either you need to do something very much on the spot and you need up to date data

or if you have, like, some monthly reports where you're also always taking screenshots in Kibana, copying that into your presentation and then showing that off. That is something that, Canvas can do, and we're really betting big now on Canvas and trying to push that out so that people can kind of, like, have a nice way to visualize whatever data they have,

and to make that easy to to share,

wherever they need to to do that.

There's also like on the operational side, there is a lot of stuff coming like we will have a new infra UI to make visualizations

of your entire infrastructure,

better to integrate logging

in an even nicer way there. Then we have bought another company recently, which was called SwiftType,

which we will provide,

search as a service basically.

So you don't need to configure all the ins and outs of Elasticsearch itself, but you get more of a solution.

They can, for example, just crawl your website and then provide a search for you. You don't need to configure anything or don't know you don't need to know how stuff works under the hood, but just getting more in that solution space where people can just use something as a service without

needing to ramp up and having a steep learning curve.

So we're trying to get more and more into these things. I don't think I can share anything other than what we have announced at Elasticon just, like, bit over a week ago.

But, yeah, internally we have a lot of stuff cooking and since, well, we're a lot of people now and we're aggressively hiring,

a lot more cool stuff will come out. I can definitely say that. Alright. Are there any other topics that we didn't cover yet that you think we should discuss before we close out the show?

Not really that I could think of. I think we covered a fair bit of ground, so, like, good overview.

Then I I always like to talk about the business side as well a bit so that people understand what we're doing. Now with that, I think we're pretty good. Okay. So for anybody who wants to get in touch with you or follow the work that you're up to, I'll have you add your preferred contact information to the show notes.

And as a final question,

from your perspective,

what do you see as being the biggest gap in the tooling or technology that's available for data management today? Technology that's available for data management today? That's an interesting 1. I'm not sure if there's really that that much of a gap. I think it's just like picking the right trade offs,

which is sometimes not easy because you don't know the trade offs from the start, but you need to kind of like just find the right toolset.

I mean it's tools are always

in in in movement

and more stuff is being built, but I I couldn't say that, oh, there is this big gap that I see. Otherwise, I probably would have started my own company around that already. So I think we're so either I cannot see it or we're a bit in a kind of refinement stage.

So NoSQL kind of, like, tried to push the entire world over.

Now

both NoSQL and the relational world are kind of, like, converging a bit together both in performance and feature sets and stuff they do.

It's like all the no sequel stores add an SQL interface to their data

stores as well.

By the way, we're be building a read only SQL interface,

for Elasticsearch as well to cover some of that ground. Still work in progress, but we hope to release that soon as well. Alright. Well, thank you very much for taking the time out of your day to join me and talk about the work that you're doing at elastic.

It's a set of products that I use day to day in my work and I know a lot of other people do as well. So thank you for that and I hope you enjoy the rest of your day. Thanks for the invitation and have a good day as well.

Data Engineering Podcast

Summary

Preamble

Interview

Contact Info

Parting Question

Links