This is a great overview. Some numbers from step 4 - size of data set, time to load index and space used on disk by that index+ES would be interesting to see. I have a feeling for small datasets postgres/mongo even sqlites FTS would perform better.
Building a Full-Text Search App Using Docker and Elasticsearch
11–15 of 15 posts
Re: Building a Full-Text Search App Using Docker and Elasticsearch
#12The sync issues between an RDBMS and ElasticSearch are definitely the most annoying aspect of getting something like this working. In the case of my current project, I've got an 'indexer' service whose sole job is to await messages from RabbitMQ to create/update/delete records to/from the index. But sometimes an update may get missed for any number of reasons. So, I've got a systemd timer that fires off a little scri…
What if your server crashes after you commit to the database, but before you submit to elastic-search? Even if submitting is flawless, this still presents a problem.
A server crash should be a rare event but if you reindex data regularly, it should not be a huge worry.
Re: Building a Full-Text Search App Using Docker and Elasticsearch
#13When I run elastic search out of the box, I've noticed that search is quite slow, on the order of 5 seconds per search, over say 10000 PDF documents (papers), on a single node. Is this normal? And is there an obvious place to look or a (quick) fix which doesn't require me to delve into the ES details too much? (as I'm not a Java person)
Sounds slow, but it depends on the total size of the index. I could see 10K PDFs being big enough to slow things down. The easiest thing to try is setting the Java heap to half the RAM up to 32GB. The more RAM the better in general. I have also found setting the number of shards to the number of CPU cores to be optimal in many cases, however if a shard is too big (>50GB) then it can start causing other problems. Again the more CPU cores the better in general.
Here are the relevant docs, and don't worry you don't need to know Java to get good performance.
indexing: https://www.elastic.co/guide/en/elasticsearch/reference/curr...
searching: https://www.elastic.co/guide/en/elasticsearch/reference/curr...
disk usage: https://www.elastic.co/guide/en/elasticsearch/reference/curr...
Re: Building a Full-Text Search App Using Docker and Elasticsearch
#14The sync issues between an RDBMS and ElasticSearch are definitely the most annoying aspect of getting something like this working. In the case of my current project, I've got an 'indexer' service whose sole job is to await messages from RabbitMQ to create/update/delete records to/from the index. But sometimes an update may get missed for any number of reasons. So, I've got a systemd timer that fires off a little scri…
In Rails, the searchkick gem seems to do a good job of solving this pain point
Re: Building a Full-Text Search App Using Docker and Elasticsearch
#15When I run elastic search out of the box, I've noticed that search is quite slow, on the order of 5 seconds per search, over say 10000 PDF documents (papers), on a single node. Is this normal? And is there an obvious place to look or a (quick) fix which doesn't require me to delve into the ES details too much? (as I'm not a Java person)
Sounds slow, but it depends on the total size of the index. I could see 10K PDFs being big enough to slow things down. The easiest thing to try is setting the Java heap to half the RAM up to 32GB. The more RAM the better in general. I have also found setting the number of shards to the number of CPU cores to be optimal in many cases, however if a shard is too big (>50GB) then it can start causing other problems. Agai…
Thanks! I'll have a look at that!