Live data from Hacker News

Elasticsearch was never a database

paradedb.com

31–40 of 109 posts

Re: Elasticsearch was never a database

#31

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

> Accenture

They messed up a $30 million dollar project big time at a previous company. My cto swore to never recommend them

Re: Elasticsearch was never a database

#32

> Elastic has been working on this gap. The more recent ES|QL introduces a similar feature called lookup joins, and Elastic SQL provides a more familiar syntax (with no joins). But these are still bound by Lucene’s underlying index model. On top of that, developers now face a confusing sprawl of overlapping query syntaxes (currently: Query DSL, ES|QL, SQL, EQL, KQL), each suited to different use cases, and with diffe…

ICYMI https://en.wikipedia.org/wiki/Greenspun's_tenth_rule

Re: Elasticsearch was never a database

#33

> Elastic has been working on this gap. The more recent ES|QL introduces a similar feature called lookup joins, and Elastic SQL provides a more familiar syntax (with no joins). But these are still bound by Lucene’s underlying index model. On top of that, developers now face a confusing sprawl of overlapping query syntaxes (currently: Query DSL, ES|QL, SQL, EQL, KQL), each suited to different use cases, and with diffe…

Funny argument on the query languages in hindsight, since the latest release (https://www.paradedb.com/blog/paradedb-0-20-0 but that was after this blog) just completely changed the API. To be seen how many different API versions you get if you make it to 15 years ;)

PS: I've worked at Elastic for a long time, so it is fun to see the arguments for a young product.

Re: Elasticsearch was never a database

#34
post #17

Earlier quoted context omitted.

Even most toy databases "built in a weekend" can be very stable for years if: - No edge-case is thrown at them - No part of the system is stressed ( software modules, OS,firmware, hardware ) - No plug is pulled Crank the requests to 11 or import a billion rows of data with another billion relations and watch what happens. The main problem isn't the system refusing to serve a request or throwing "No soup for you!" err…

I'm talking about production loads, but thanks.

Production loads mean a lot of different things to a lot of different people.

Re: Elasticsearch was never a database

#35

I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…

I've managed a 100+ node cluster for years without seeing any corruption. Where are you getting this from?

Re: Elasticsearch was never a database

#36

I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…

We only used it on top of the primary databases, just like many other components for scaling or auxiliary functionalities. Not sure how others use it

Re: Elasticsearch was never a database

#37
post #29
post #9

I think elastic always clearly documented to expect "eventual consistency", they never claimed to be a "database" in the sense that tfa defines.

First step of a marketing campaign: Claim something never said and then tell everyone why it's wrong ;)

It's not so much that Elastic is saying it as a lot of people doing the supposed wrong the advert-article describes.

I've seen some examples of people using ES as a database, which I'd advise against for pretty much the reasons TFA brings up, unless I can get by on just a YAGNI reasoning.

Re: Elasticsearch was never a database

#39

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

This is made possible because Elastic gained a write-ahead log that actually syncs to disk after each write, like Postgres.

Re: Elasticsearch was never a database

#40

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

I worked in a company that used elastic search as main db. It worked, company made alot of money from that project. It was a wrong decision but helped us complete the project very fast. We needed search capability and a db. ES did it both.

Problems that we faced by using elastic search: High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce company)

To Write and read after, you need to refresh the index or wait a refresh. More inserts, more index refreshes. Which ES is not designed for, inserts become slow. You need to find a way to insert in bulk.

Api starts, cannot find es alias because of connection issue, creates a new alias(our code did that when it cant find alias, bad idea). Oops whole data on alias is gone.

Most important thing to use ES as main db is to use "keyword" type for every field that you don't text search.

No transaction: if second insert fails you need to delete first insert by hand. Makes code look ugly.

Advantages: you can search, every field is indexed, super fast reads. Fast development. Easy to learn. We never faced data loss, even if db crashed.

Post reply on HN