Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Storing 50M events per second in Elasticsearch

datadome.co

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Storing 50M events per second in Elasticsearch

#31

Post by atomashpolskiy »

porker wrote: post #28
atomashpolskiy wrote: post #23

Earlier quoted context omitted.

+1, or use any other log-based replication mechanism (e.g. Logstash). The point is that instead of having two independent systems that can easily go out of sync (if not using distributed transactions) and become permanently incostistent with each other, you'll now have the database as the primary source of data (commonly referred to as the system of record) and Elasticseach as a secondary, eventually consistent searc…

If the database is the primary source of data, how do you get the data from there into the log-based replication method? I assumed the OP meant you'd write to Kafka, and the messages would be processed twice: once to write to the DB, and once to ElasticSearch. Not wanting to do that for a small project, but wanting a better architecture than I've got, I'm curious about your proposed approach.

Two possibilities: either the app writes both to database and Kafka (ideally using an atomic commit) or CDC is setup in Kafka to read database's transaction log (this is faster)

> you'd write to Kafka, and the messages would be processed twice: once to write to the DB, and once to ElasticSearch

This would be equivalent to using a message queue, which (in contrast to log-ordered replication) does not ensure same consistency guarantees (in this case (1) RYW for database writes and (2) database being always at least as up-to-date as the search index)

Re: Storing 50M events per second in Elasticsearch

#32

Post by virgilp »

MuffinFlavored wrote: post #12
uncoder0 wrote: post #9

Earlier quoted context omitted.

It's misleading for sure but they're writing 250 'events' per document.

200k documents per second is a lot less impressive, no?

I don't know. Have you tried it? I worked on a Kafka streams service written in Java that processed "changelog" messages (it involved one query to CosmosDB per message, and logging the result to Kafka for downstream processing by other systems). Now, we had a rather limited number of workers (4 or 8? I don't remmember), but getting to 100k messages per second was rather challenging.

Re: Storing 50M events per second in Elasticsearch

#33

Post by anon1253 »

jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

we use zombodb https://www.zombodb.com/ to keep ES and Postgresql in sync, it has its flaws but when it works it works perfectly. That at least helps with 1 and 3, which are indeed a major annoyances.

Re: Storing 50M events per second in Elasticsearch

#34

Post by tty6 »

outworlder wrote: post #2

This part left me scratching my head: > We have set “replica 0” in our indexes settings > Now let’s assume that node 3 goes down: > As expected, all shards from node 3 are moved to node 1 and node 2 No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here. > In order to resolve this issue, we introduced a job which runs e…

author here: > replica 0 It's an example for the article and the intention was to remove the complexity of primary/replica shards. Let's say "shard" is an unit and no matter about primary or replica. In fact with replica to 1, the behaviour would be the same but in the diagram it will have twice more shards. What we wanted to show is IF one node goes down and up after few times AND a rollover occurs just after then this node will handle all those new shards and so handle all the write. here we have a spread write issue

> introduction of a job which runs each day this job has many purposes: 1) because of rollover, our indexes are now suffixed by -000001 then -000002 etc...our applications no matter of the rollover post and get doc by the alias in front of these indexes suffixed by -000001... So if you don't create the index for the next day in a daily basis index design, your application will push at 00:00:01 a new index with the alias name and it won't be in rollover "mode". 2) because we are using ILM feature, we need to define the ILM rollover alias in the template and it changes each day because our indexes name are "index_$date" 3) performance issues: we have a lot of traffic and if we do not create index before for the next day, we will except a lot of unassigned shards, cluster yellow etc...

in fact, yes, it's a common use-case (daily based index) but maybe not with rollover + ilm

Re: Storing 50M events per second in Elasticsearch

#35

Post by qeternity »

jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

10k? Hell, any vanilla RDBMS can handle this (including SQLite).

Re: Storing 50M events per second in Elasticsearch

#36

Post by FBISurveillance »

I don't think writing a clickbaity title like this is fair. You just write 200k large documents per second, period. Good for you but to be blatantly honest it's actually not a lot.

I'm not saying you shouldn't have written this post, but rather suggest you be fair to your readers (and yourself). Otherwise you could just make up random titles like "Writing 1 trillion log lines per second" (by storing 1,000,000 1-byte, newline-separated log lines per document).

Re: Storing 50M events per second in Elasticsearch

#37

Post by frumiousirc »

mistrial9 wrote: post #4

it appears in this document: * DataDome is a security company, and gets web traffic in near real-time for clients; a lot of traffic in some cases with very specific numbers given, like daily peak loads. * DataDome only retains records for 30 days, and the most attention is given to the most recent traffic, to detect attacks * an ElasticSearch deployment records all of the traffic records downstream from Apache Flink;…

I was curious about their numbers

> Storing 50 million of events per second

> A few numbers: our cluster stores [...], 15 trillion of events

> We provide up to 30 days of data retention to our customers. The first seven days of data were stored in the hot layer, and the rest in the warm layer.

15e12 / 50 MHz is 3.5 days.

I guess 50 MHz is the peak ingest rate.

Re: Storing 50M events per second in Elasticsearch

#38

Post by [deleted] »

rexer wrote: post #24
outworlder wrote: post #2

This part left me scratching my head: > We have set “replica 0” in our indexes settings > Now let’s assume that node 3 goes down: > As expected, all shards from node 3 are moved to node 1 and node 2 No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here. > In order to resolve this issue, we introduced a job which runs e…

> This is a very common use-case(eg. logging), but it's surprising that Elastic has nothing to automate this. You can set an index template to be used on new indices that match a pattern, which is a very common thing to do. It sounds like what they did was modify the template daily, which is less common IME. It's not clear why they had to manually create the index, though. That should happen automatically.

[deleted]

Re: Storing 50M events per second in Elasticsearch

#39

Post by linuxdude314 »

jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

This is a mistake many people make. Elasticsearch is probably overkill for your particular use case.

It’s similar to bringing a F1 car to a go-cart race and then being surprised you aren’t able to finish the race because you don’t have a pit crew able to maintain your vehicle.

I’ve built and owned large Elasticseach clusters at Fortune 50 companies for providing log search as well as document search. Like anything, administering an ES cluster requires planning, engineering, and process/documentation.

I wouldn’t consider using ES if I didn’t have a dedicated ops team to help in its administration unless possibly using a managed service like the one AWS provides.

It’s a very powerful tool; it was a mistake to think you can just casually throw it in your stack without fully understanding its complexity.

Post reply on HN