Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dejavu – Web UI for Elasticsearch

github.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Dejavu – Web UI for Elasticsearch

#21

Post by zimbatm »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

I spent so much time resurrecting ElasticSearch and then developers wouldn't even use it because the query language is based on ngrams and they want grep-like capabilities instead.

Unless there are hundreds of GB/day to index it's much simpler to forward the logs using syslog or journald and then use grep on the collector.

Re: Dejavu – Web UI for Elasticsearch

#22

Post by ddorian43 »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

By default elasticsearch stores data in 3 ways: rowstore (original doc/row), inverted-index (for fast queries), doc-values (columnar store for aggregations). So you need to configure it for what you want, for each field.

While you can do all of that with plain pg and having no indexes and doing table-scans, it will be slow when you search (depends on how often you need to search and type of queries).

So es is very good at some stuff compared to every other type of db. For logging, yeah, there are some companies who just do full-scan on each query and are fine.

Re: Dejavu – Web UI for Elasticsearch

#23

Post by riku_iki »

ddorian43 wrote: post #22
giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

By default elasticsearch stores data in 3 ways: rowstore (original doc/row), inverted-index (for fast queries), doc-values (columnar store for aggregations). So you need to configure it for what you want, for each field. While you can do all of that with plain pg and having no indexes and doing table-scans, it will be slow when you search (depends on how often you need to search and type of queries). So es is very go…

> plain pg and having no indexes

but you can have indexes in pg..

Re: Dejavu – Web UI for Elasticsearch

#24

Post by FridgeSeal »

burner8309 wrote: post #10

> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server side page rendering techniques (I am looking at you, Kibana). When did server-side rendering become a bad thing? It tends to be much faster. Kibana feels slow to me, and that's because of the JavaScript it does have.

I recently finished a small project at work for another business unit-a straightforward tableau dashboard-only for them to turn around and say "our laptops aren't powerful enough to run tableau, thanks but we're going back to excel".

They were the ones who insisted I use tableau, I had originally planned and suggested a nice server-side rendered dashboard that they could have used on any grade of laptop, but no they wanted something _right now_ and any kind of proper solution was simply "too hard".

The real irony being that they'll waste more time doing and redoing these excel reports every time they want one of these reports than it would have taken me and my teammate to build a solution for them.

Re: Dejavu – Web UI for Elasticsearch

#25

Post by ddorian43 »

riku_iki wrote: post #23
ddorian43 wrote: post #22

Earlier quoted context omitted.

By default elasticsearch stores data in 3 ways: rowstore (original doc/row), inverted-index (for fast queries), doc-values (columnar store for aggregations). So you need to configure it for what you want, for each field. While you can do all of that with plain pg and having no indexes and doing table-scans, it will be slow when you search (depends on how often you need to search and type of queries). So es is very go…

> plain pg and having no indexes but you can have indexes in pg..

Yes, but I was saying about overhead.

Re: Dejavu – Web UI for Elasticsearch

#26

Post by jstarfish »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

> tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog.

Part of the problem is what it's promoted for. It's a great drop-in, horizontally-scalable, full-text search engine, that's inexplicably become popular for log ingestion and analysis.

To those ends, I hate it, I hate every bit of it, from the atrocious JSON-based query DSL (seriously thought it was a joke at first) to its unpredictable timeouts, shard storms, mapping conflicts and other problems at scale. Elementary SQL concepts aren't possible ('select someone_elses_poorly_named_key as first_name', nope, you gotta reindex). High-cardinality aggregations fail in spectacular ways. High-anything aggregations fail in spectacular ways. The scroll API returns results unordered. There's no way to properly spec your cluster; the docs explicitly take a trial-and-error approach to design.

It's not just you. Elasticsearch does me no favors with the task of log analysis. I'd sooner normalize and grep a pile of gzipped log files than keep dealing with this mess, but this is the second job I've been at that's built their logging infrastructure on top of it.

> I do not understand why they built Elasticsearch.

"You know, for search."

It's great for searching proper text. Documents, comments, blog posts, etc.

> I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly.

You don't necessarily need 32GB to do anything. The required heap size scales with your intended workload, but it's not like you can do a back-of-napkin calculation to figure out the relation. I run a 1GB instance for development.

It uses the memory to do a lot of caching so the queries you throw it are lightning fast. Mongo does something similar (but crappier IMO) with its concept of working sets.

Re: Dejavu – Web UI for Elasticsearch

#27

Post by fareesh »

jstarfish wrote: post #26
giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

> tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. Part of the problem is what it's promoted for. It's a great drop-in, horizontally-scalable, full-text search engine, that's inexplicably become popular for log ingestion and analysis. To those ends, I hate it, I hate every bit of it, from the atrocious JSON-based query DSL (seriously…

I've deployed it a few times. In some cases it randomly spikes to 100% cpu usage until you restart it.

Is there a way to know how much RAM you are going to need for your dataset? I think I was using it for looking up restaurant names from a database of 100,000 and I wanted to factor in misspellings and partial matches.

Re: Dejavu – Web UI for Elasticsearch

#28

Post by rpedela »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

You don't actually need 64GB or 32GB or whatever RAM. The docs should be more clear. What they mean to say is if you have a large enough dataset in production, 64GB RAM per node is the ideal maximum size. That is because 32GB is the max Java heap size that uses 32-bit pointers. So 32GB for Java heap and 32GB for everything else. Although more RAM is generally better because the OS cache will still be utilized making queries faster overall.

I think ES's Java heap default is 1 or 2 GB, and that is more than enough for many use cases. Heap-heavy operations like sorting and aggregations may need more RAM depending on the index size. As far as I know, search isn't heap-heavy so you only need more RAM as query volume increases or index size increases.

For crashes, what version? Version 6.x has never crashed on me, but previous versions did have a tendency to crash for me.

Re: Dejavu – Web UI for Elasticsearch

#29

Post by manigandham »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

Elasticsearch is a search engine built on Apache Lucene and basically makes it much more usable. It's similar to Apache Solr but easier to get up and running.

It's good at text but can be used as a similarity search system, so you can index and find similar images, audio waveforms, binary data, etc.

Over the years, search queries have become useful for log analytics and so the ELK stack has been developed to become a single solution to do both text search as well as ingest logs and run all kinds of queries, aggregations, and even machine learning.

I think that's what fundamentally hurts it as they are very separate use-cases trying to be served by the same system. We still use ES for search but wish it just focused on that with much better performance, reliable clustering and proper transactions instead.

Post reply on HN