Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dejavu – Web UI for Elasticsearch

github.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Dejavu – Web UI for Elasticsearch

#31

Post by forgotcreds »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

According to the cloud.elastic.co recommendations for 'production' grade clusters suggests from 4GB upwards.

Re: Dejavu – Web UI for Elasticsearch

#33

Post by reactivesearch »

tenken wrote: post #32

Is there anything similar for Solr??

Not that I know of, but given the similarities in the underlying APIs - we would be open to supporting this and help guide the implementation if someone is interested in sending a PR. Feel free to file an issue - https://github.com/appbaseio/dejavu/issues/.

Re: Dejavu – Web UI for Elasticsearch

#34

Post by reactivesearch »

forgotcreds wrote: post #30

>built with server side page rendering techniques (I am looking at you, Kibana). I'm not sure if this is actually accurate: 1. https://github.com/elastic/kibana/search?utf8=&q=React 2. https://elastic.github.io/eui

I will have to relook at this - the comment is from a year ago when they weren't using React.

On a cursory look, it still seems to use a Node.JS based server [1]. One of the key side effects of needing a server is that there is a more involved distribution + installation process. You just can't run it on a static server like github pages or as a browser extension.

[1] https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=ha...

Re: Dejavu – Web UI for Elasticsearch

#35

Post by rhacker »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

I definitely used it on 8GB or RAM but only about 10M documents or so. It's pretty kick-ass for ad-hoc queries about data that you would typically have to set up a star schema. I tell you probably the best thing you can do is set up a Kafka queue, Apache Spark and ElasticSearch (do the research around these 3) but you'll love the ability to find out things like how many M(ale) patients that are above the age of 30 that have diabetes have died shortly after a surgery. Trying to set all that up with complicated star-schemas etc.. really sucks compared to just building a JSON format that you pipe through Kafka or Spark.

Edit: And yes originally used it for log processing, but really people should definitely try it out for replacing very expensive BA stacks. For log analysis something called Graylog that actually uses Elk internally or just go with Splunk which gives you out of the box primitives for session length calculations. In ELK if you want to do something that sounds as simple as session lenght - you'll ending having to reprocess documents using Kafka or Spark with background jobs that reprocess documents greater than 1 hour old (or something to that effect).

Re: Dejavu – Web UI for Elasticsearch

#36

Post by artellectual »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

I'm not sure why elasticsearch documentation recommends such memory usage, however for small apps I have elasticsearch running for years (version 1.x) and it's been running on a shared 2GB virtual machine for the past 4 years, I've had to restart it a few times, but seriously you don't need 32GB or 64GB. It depends on your use case.

Re: Dejavu – Web UI for Elasticsearch

#37

Post by catmanjan »

fareesh wrote: post #27
jstarfish wrote: post #26

Earlier quoted context omitted.

> tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. Part of the problem is what it's promoted for. It's a great drop-in, horizontally-scalable, full-text search engine, that's inexplicably become popular for log ingestion and analysis. To those ends, I hate it, I hate every bit of it, from the atrocious JSON-based query DSL (seriously…

I've deployed it a few times. In some cases it randomly spikes to 100% cpu usage until you restart it. Is there a way to know how much RAM you are going to need for your dataset? I think I was using it for looking up restaurant names from a database of 100,000 and I wanted to factor in misspellings and partial matches.

Not really, depends on the size of the dataset and the complexity of the operations - when did you use it? Sounds like you were on a dodgy build or set up

I think Elastisearches policy on sizing is pay us or a partner to have a look and give you a guesstimate, which is pretty standard

Re: Dejavu – Web UI for Elasticsearch

#38

Post by artellectual »

One thing to note for anyone using elasticsearch in production, if you setup your own cluster. It might be wise to put nginx as a reverse proxy. By just doing that we reduced the amount of crashes when the server got hammered. Since nginx will help with request queueing

Re: Dejavu – Web UI for Elasticsearch

#39

Post by INTPenis »

giancarlostoro wrote: post #15

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in M…

We are so many that have been through this just like you.

My realization was also what others have mentioned, that I'm trying to use a search engine backend for log storage.

Specifically 3 months, which is the law here. That meant that ES had to keep 3 months of logs readily searchable. It's just not feasible when you're generating 25-30G logs each day.

I still use ES for search engines but I've stopped using ELK for logs unless it's for small environments.

Re: Dejavu – Web UI for Elasticsearch

#40

Post by INTPenis »

burner8309 wrote: post #10

> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server side page rendering techniques (I am looking at you, Kibana). When did server-side rendering become a bad thing? It tends to be much faster. Kibana feels slow to me, and that's because of the JavaScript it does have.

It's pretty obvious that server side rendering has more potential bottlenecks and IO overhead. But with that said, nothing says it must be slower. It just has more potential to be slower.
Post reply on HN