Storing 50M events per second in Elasticsearch
1–10 of 50 posts
Re: Storing 50M events per second in Elasticsearch
#2This part left me scratching my head:
> We have set “replica 0” in our indexes settings
> Now let’s assume that node 3 goes down:
> As expected, all shards from node 3 are moved to node 1 and node 2
No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here.
> In order to resolve this issue, we introduced a job which runs each day in order to update the mapping template and create the index for the day of tomorrow, with the right number of shards according to the number of hits our customer received the previous day.
This is a very common use-case(eg. logging), but it's surprising that Elastic has nothing to automate this.
Re: Storing 50M events per second in Elasticsearch
#3>> Each day, during peak charge, our Elasticsearch cluster writes more than 200 000 documents per second
What is this 50M in the title?
Re: Storing 50M events per second in Elasticsearch
#4it appears in this document:
* DataDome is a security company, and gets web traffic in near real-time for clients; a lot of traffic in some cases with very specific numbers given, like daily peak loads.
* DataDome only retains records for 30 days, and the most attention is given to the most recent traffic, to detect attacks
* an ElasticSearch deployment records all of the traffic records downstream from Apache Flink; a new feature added to ES this year, improves the management of ES indexing, and that solved problems that DataDome was having.. things are better! write an engineering blog post !
* re-indexing is done nightly, and implemented in a cloud environment that can handle the (heavy) work to rebuild the indexing.
These numbers are impressive. Earlier criticisms of ES are being addressed, and ES is stable and a cornerstone of the architecture. A company called DataDome is providing real services in near real-time. Congratulations to the team and an interesting read.
Re: Storing 50M events per second in Elasticsearch
#5This part left me scratching my head: > We have set “replica 0” in our indexes settings > Now let’s assume that node 3 goes down: > As expected, all shards from node 3 are moved to node 1 and node 2 No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here. > In order to resolve this issue, we introduced a job which runs e…
Indices are composed of one or more primary shards. Each primary shard can have one replica. Three nodes, each with one primary shard as a part of that sjngle index, no replicas in play at all.
Re: Storing 50M events per second in Elasticsearch
#6Does anyone know what log shipper they are using?
Re: Storing 50M events per second in Elasticsearch
#7>> Each day, during peak charge, our Elasticsearch cluster writes more than 200 000 documents per second What is this 50M in the title?
They state each document has 250 events, 200,000 document/sec x 250 events/document = 50m events/sec.
Re: Storing 50M events per second in Elasticsearch
#8This part left me scratching my head: > We have set “replica 0” in our indexes settings > Now let’s assume that node 3 goes down: > As expected, all shards from node 3 are moved to node 1 and node 2 No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here. > In order to resolve this issue, we introduced a job which runs e…
Indices are composed of one or more primary shards. Each primary shard can have one replica. Three nodes, each with one primary shard as a part of that sjngle index, no replicas in play at all.
> Indices are composed of one or more primary shards. Each primary shard can have one replica. Three nodes, each with one primary shard as a part of that sjngle index, no replicas in play at all.
Ok, 3 nodes, each with one primary shard. No replicas. 1 node goes down, one shard is no longer found in the cluster, because it was in the missing node. That particular index, and in fact the whole cluster, are now RED.
Unless you discard that shard (force reroute, with accept_data_loss), nothing is going to be recovered and the missing shards will not be allocated anywhere.
Re: Storing 50M events per second in Elasticsearch
#9>> Each day, during peak charge, our Elasticsearch cluster writes more than 200 000 documents per second What is this 50M in the title?
It's misleading for sure but they're writing 250 'events' per document.
Re: Storing 50M events per second in Elasticsearch
#10>> Each day, during peak charge, our Elasticsearch cluster writes more than 200 000 documents per second What is this 50M in the title?
They state each document has 250 events, 200,000 document/sec x 250 events/document = 50m events/sec.
"Each document stores 250 events in a seperate field."
Curiouser and curiouser.