Reading this reminds me of the pains of running an ElasticSearch cluster. We just moved to Elassandra. No more red status.
Storing 50M events per second in Elasticsearch
11–20 of 50 posts
Re: Storing 50M events per second in Elasticsearch
#12Re: Storing 50M events per second in Elasticsearch
#13I've seen elasticsearch clusters like this have consistency problems. Turns out it's a problem in a security setting to have an off-by-one error.
Re: Storing 50M events per second in Elasticsearch
#14Re: Storing 50M events per second in Elasticsearch
#15Re: Storing 50M events per second in Elasticsearch
#163 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database.
I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine.
Pain points:
1) You have to setup and recreate part of your database in elastic search. So you essentially end up with two databases. Which now you have to keep in sync.
2) I was getting unpredictable query results from Elasticsearch, which after a few days, and much head scratching turned out to be that I was running out of memory.
3) When a user added a new event, it was not being added to elastcsearch index automatically. I could not figure out how to do this reliably. I could make it work reliably only after a sync of the entire Elasticsearch index. But this meant that it was next to useless, to use for the Upcoming Events List. Since I only wanted to sync the index once a day. Confusing the users, as to why their event was not showing up. And I gave up, and just ended up implementing the Upcoming Events List directly from my database in python.
4) Elasticsearch came without some security settings not set by default, and after a few months it was hacked. I had to download a new version and wasted more time.
I still use Elasticsearch, but only for search, and not the upcoming event list. And I don't think it was worth the complexity that it added to my project.
Re: Storing 50M events per second in Elasticsearch
#173 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…
I have been put off by Elasticsearch's complexity a number of times. Can I ask why with searching a limited about of text you didn't juse use Postgres' full-text search?
Re: Storing 50M events per second in Elasticsearch
#183 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…
Kafka streams can solve this use case fairly well - though setting up & managing infra may be a bit more than what you'd want to deal with for a hobby project
Re: Storing 50M events per second in Elasticsearch
#193 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…
I have been put off by Elasticsearch's complexity a number of times. Can I ask why with searching a limited about of text you didn't juse use Postgres' full-text search?
I was new to Python, Django and Postgres. I was looking up how to do search and stumbled on articles how to use Elasticsearch in Django. So I went with it.
Re: Storing 50M events per second in Elasticsearch
#203 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…
I have been put off by Elasticsearch's complexity a number of times. Can I ask why with searching a limited about of text you didn't juse use Postgres' full-text search?
I'm not the person you're replying to, but does Postgres nowadays have a straightforward way to do tf-idf or BM25-style information retrieval?