Elasticsearch from the Bottom Up (2013)

#1

Post by bobjordan » Thu, Jan 02, 2020, 9:54 AM UTC

Elasticsearch from the Bottom Up (2013)
elastic.co

Re: Elasticsearch from the Bottom Up (2013)

#2

Post by MuffinFlavored » Fri, Jan 03, 2020, 8:15 PM UTC

Does Elasticsearch need to be as complicated as it is?

I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler".

I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

Re: Elasticsearch from the Bottom Up (2013)

#3

Post by mountaineer » Fri, Jan 03, 2020, 8:24 PM UTC

MuffinFlavored wrote: post #2

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

> JSON logs with a timestamp + a bunch of fields then search them

There is S3+Athena for this with AWS and Google can store/query JSON with BigQuery. The nice little UI doesn't come with it, but at least you don't have to spin up an Elastic cluster.

Re: Elasticsearch from the Bottom Up (2013)

#4

Post by fizx » Fri, Jan 03, 2020, 8:26 PM UTC

MuffinFlavored wrote: post #2

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

I mean, that product is called LogDNA.

But Elasticsearch has evolved into a whole bunch of things to meet everyone's needs. There's a way to do what you want and simply, but you have to find the simple path in the middle of the big product.

Re: Elasticsearch from the Bottom Up (2013)

#5

Post by pixelmonkey » Fri, Jan 03, 2020, 8:28 PM UTC

There's also a YouTube recording of a talk with similar content by the same author from EuroPython 2014. Helped me out when I was adopting ES at scale in that time period. (And the principles are pretty timeless to modern ES, too.)

https://youtu.be/PpX7J-G2PEo

If you like this, you might also enjoy my deep dive on Lucene (the indexing technology underneath Elasticsearch) in "Lucene: The Good Parts":

https://blog.parse.ly/post/1691/lucene/?utm_source=hn

Re: Elasticsearch from the Bottom Up (2013)

#6

Post by bratao » Fri, Jan 03, 2020, 8:33 PM UTC

Shameless plug from someone who want this project to flourish. Check https://vespa.ai as an alternative to Elasticsearch. Migrating from a ES to it, I got a faster search, never had to face a unhealthy node and native tensor support (And Native ANN is coming soon https://github.com/vespa-engine/vespa/issues/9747).

Very mature, and still progressing at a neck-break rate (https://blog.vespa.ai)

Re: Elasticsearch from the Bottom Up (2013)

#7

Post by inertiatic » Fri, Jan 03, 2020, 8:54 PM UTC

MuffinFlavored wrote: post #2

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

I think the question you should be asking is, does my use case require a tool as powerful as ES?

And only you can answer that fully.

Re: Elasticsearch from the Bottom Up (2013)

#8

Post by softwaredoug » Fri, Jan 03, 2020, 8:58 PM UTC

MuffinFlavored wrote: post #2

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

I mean, the same reason that SQL isn't any simpler? It takes arbitrary data, and you can filter and aggregate it in arbitrary ways? People aren't just doing log data... it's just been heavily adopted for that purpose

I think the "simpler" version of ES+Kibana is probably a spreadsheet.

Re: Elasticsearch from the Bottom Up (2013)

#9

Post by inertiatic » Fri, Jan 03, 2020, 9:02 PM UTC

One thing that really bothers me about ES is that compared to Solr, some terms that have a specific meaning in Lucene are either not used for the corresponding concept or even worse re-used for a different one.

It sometimes makes explaining the underlying implementation a bit harder to people who are Lucene-agnostic but are ES users, with no good reason apart from, I would guess, brand differentiation?

Re: Elasticsearch from the Bottom Up (2013)

#10

Post by liveoneggs » Fri, Jan 03, 2020, 9:17 PM UTC

mountaineer wrote: post #3

MuffinFlavored wrote: post #2

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

> JSON logs with a timestamp + a bunch of fields then search them There is S3+Athena for this with AWS and Google can store/query JSON with BigQuery. The nice little UI doesn't come with it, but at least you don't have to spin up an Elastic cluster.

aws cloudwatch log insights can do a lot natively without the s3/athena step now