Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Storing 50M events per second in Elasticsearch

datadome.co

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Storing 50M events per second in Elasticsearch

#41

Post by rsrx »

jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

It requires a lot of pampering, but I quite enjoy using it and discovering it's possibilities. I am using it for the startup project which I work on, that has search feature very similar to Instagram one. Do you have any other suggestions for search engine that is flexible enough? I don't want to couple things too tightly by using zombodb and similar stuff.

Re: Storing 50M events per second in Elasticsearch

#42

Post by linuxdude314 »

rsrx wrote: post #41
jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

It requires a lot of pampering, but I quite enjoy using it and discovering it's possibilities. I am using it for the startup project which I work on, that has search feature very similar to Instagram one. Do you have any other suggestions for search engine that is flexible enough? I don't want to couple things too tightly by using zombodb and similar stuff.

If you really need search, I think its the clear winner still. I don't think its terribly hard to manage/operate, just that you do need to do some initial planning otherwise it will balloon out of control.

Dynamic mappings can mess things up really easily, so its best to disable them in favor of using a pre-defined static map for the type of documents you will be ingesting. What I've encountered in the past that usually causes things to break, is when 90% of your documents contain a field called "Date" that contains a ANSI date field, but the other 10% contain "Null" (string instead of an ANSI date). Since the documents don't match the dynamically generated mapping, they fail to be indexed.

Shard management is also critical and this largely depends on the type of data you are indexing. If the data in ES is unique (not just a copy of a database you already have), you will want to have some sort of cross-region/DC replication strategy as well as a backup strategy.

Fortunately both of these are pretty easy. ES has a mechanism of using tags that allows you to define things like regions, data centers, really whatever you want, and shards can be routed based on rules defined over these tags.

A setup I've used in the past is to have 5 nodes in LAX DC 5 nodes in LAS DC, any data that is ingested into LAX is replicated into shards in LAS and vice versa.

Backup to S3 is rather trivial now thanks to the built in export options in the newer versions of ES.

With a little bit of planning ES can be a great addition to your stack, just be sure you do the initial engineering so you can avoid a big headache in the future.

Re: Storing 50M events per second in Elasticsearch

#43

Post by jungletime »

jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

Here it is in glorious action, if anyone is curious: https://www.pincalendar.com/search/events/?q=

Re: Storing 50M events per second in Elasticsearch

#44

Post by [deleted] »

linuxdude314 wrote: post #39
jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

This is a mistake many people make. Elasticsearch is probably overkill for your particular use case. It’s similar to bringing a F1 car to a go-cart race and then being surprised you aren’t able to finish the race because you don’t have a pit crew able to maintain your vehicle. I’ve built and owned large Elasticseach clusters at Fortune 50 companies for providing log search as well as document search. Like anything, a…

[deleted]

Re: Storing 50M events per second in Elasticsearch

#45

Post by hmschreck »

MuffinFlavored wrote: post #12
uncoder0 wrote: post #9

Earlier quoted context omitted.

It's misleading for sure but they're writing 250 'events' per document.

200k documents per second is a lot less impressive, no?

200k documents per second really isn't that much for Elasticsearch. The single-instance setup we have in my small company (around 25 people total) has been sent in excess of 40k/s at times, and even then it doesn't slow down noticeably.

Re: Storing 50M events per second in Elasticsearch

#46

Post by outworlder »

rexer wrote: post #24
outworlder wrote: post #2

This part left me scratching my head: > We have set “replica 0” in our indexes settings > Now let’s assume that node 3 goes down: > As expected, all shards from node 3 are moved to node 1 and node 2 No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here. > In order to resolve this issue, we introduced a job which runs e…

> This is a very common use-case(eg. logging), but it's surprising that Elastic has nothing to automate this. You can set an index template to be used on new indices that match a pattern, which is a very common thing to do. It sounds like what they did was modify the template daily, which is less common IME. It's not clear why they had to manually create the index, though. That should happen automatically.

> You can set an index template to be used on new indices that match a pattern, which is a very common thing to do

It is, but how can you tell in your template you want to keep shard sizes under 50GB? You can't.

The best thing you can do (as they did) is, based on historical data, update the template, so that the new index will have shards that (hopefully) are under 50GB.

Re: Storing 50M events per second in Elasticsearch

#47

Post by outworlder »

tty6 wrote: post #34
outworlder wrote: post #2

This part left me scratching my head: > We have set “replica 0” in our indexes settings > Now let’s assume that node 3 goes down: > As expected, all shards from node 3 are moved to node 1 and node 2 No, as there are no shards that can be moved, as number of replicas was set to zero and one node went down. Not sure what they are trying to explain here. > In order to resolve this issue, we introduced a job which runs e…

author here: > replica 0 It's an example for the article and the intention was to remove the complexity of primary/replica shards. Let's say "shard" is an unit and no matter about primary or replica. In fact with replica to 1, the behaviour would be the same but in the diagram it will have twice more shards. What we wanted to show is IF one node goes down and up after few times AND a rollover occurs just after then t…

> replica 0 It's an example for the article and the intention was to remove the complexity of primary/replica shards.

Ah, got it. So maybe it would be best said as "for the following example, ignore any replicas".

> in fact, yes, it's a common use-case (daily based index)

But it is not automated by the Elastic folks. Do you have any intentions of open-sourcing a portion of this job?

Re: Storing 50M events per second in Elasticsearch

#48

Post by IanGabes »

outworlder wrote: post #8
IanGabes wrote: post #5

Earlier quoted context omitted.

Indices are composed of one or more primary shards. Each primary shard can have one replica. Three nodes, each with one primary shard as a part of that sjngle index, no replicas in play at all.

> Indices are composed of one or more primary shards. Each primary shard can have one replica. Three nodes, each with one primary shard as a part of that sjngle index, no replicas in play at all. Ok, 3 nodes, each with one primary shard. No replicas. 1 node goes down, one shard is no longer found in the cluster, because it was in the missing node. That particular index, and in fact the whole cluster, are now RED. Unl…

Oh, yes, sorry i see your perspective now. You will get data loss in this example. My understanding of the example was that it is showing how one node can end up with all the write operations, i wasnt under the impression that it was a "real" cluster.

Re: Storing 50M events per second in Elasticsearch

#49

Post by rsrx »

linuxdude314 wrote: post #42
rsrx wrote: post #41

Earlier quoted context omitted.

It requires a lot of pampering, but I quite enjoy using it and discovering it's possibilities. I am using it for the startup project which I work on, that has search feature very similar to Instagram one. Do you have any other suggestions for search engine that is flexible enough? I don't want to couple things too tightly by using zombodb and similar stuff.

If you really need search, I think its the clear winner still. I don't think its terribly hard to manage/operate, just that you do need to do some initial planning otherwise it will balloon out of control. Dynamic mappings can mess things up really easily, so its best to disable them in favor of using a pre-defined static map for the type of documents you will be ingesting. What I've encountered in the past that usua…

Thanks, very insightful.

Re: Storing 50M events per second in Elasticsearch

#50

Post by zihotki »

jungletime wrote: post #16

3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There's only about 10,000 events in the database. I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine. Pain points: 1) You have to setup and recreate part of your database in elastic search. So you essentially…

That's why whenever I need to have a good full text search I pick https://ravendb.net/ . Literally close to 0 maintenance required and it's very feature rich.
Post reply on HN