Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch was never a database

paradedb.com

Post reply on HN
Print view View original

41–50 of 109 posts

Re: Elasticsearch was never a database

#41

Post by thisisananth »

kubi07 wrote: post #40
speedgoose wrote: post #11

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

I worked in a company that used elastic search as main db. It worked, company made alot of money from that project. It was a wrong decision but helped us complete the project very fast. We needed search capability and a db. ES did it both. Problems that we faced by using elastic search: High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce compan…

agree with comment. We use ES quite extensively as a database with huge documents and touchwood we haven't had any data loss. We take hourly backups and it is simple to restore. You have to get used to eventual consistency. If you want to read after writing even by id, you have to wait for the indexing to be complete (around 1 second). You have to design the documents in such a way that you shouldn't need to join the data with anything else. So make sure you have all the data you need for the document inside it. In an SQL db you would normalize the data and then join. Here assume you have only one table and put all the data inside the doc. But as we evolved and added more and more fields into the document, the document sizes have grown a lot (Megabytes) and hitting limits like (max searchable fields :1000 can be increased but not recommended) search buffer limits 100MB).

My take is that ES is good for exploration and faster development but should switch to SQL as soon the product is successful if you're using it as the main db.

Re: Elasticsearch was never a database

#43

Post by xeraa »

cess11 wrote: post #37
xeraa wrote: post #29

Earlier quoted context omitted.

First step of a marketing campaign: Claim something never said and then tell everyone why it's wrong ;)

It's not so much that Elastic is saying it as a lot of people doing the supposed wrong the advert-article describes. I've seen some examples of people using ES as a database, which I'd advise against for pretty much the reasons TFA brings up, unless I can get by on just a YAGNI reasoning.

It will also depend a lot on the type of data: Logs are an easy yes. Something that required multi-document transactions (unless you're able to structure it differently) is a harder tradeoff. Though loss of ACKed documents shouldn't really be a thing any more.

Re: Elasticsearch was never a database

#44

Post by nkmnz »

cluckindan wrote: post #16
nkmnz wrote: post #15

Earlier quoted context omitted.

It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.

Well, most people working on a car don’t have a car lift: it only makes sense when you need to safely work on a large volume of cars. If you only work on one or two, a jack and a pile of wood works just fine.

Please don't move the goal post. Writing `no database ”just works” without (...)` is gatekeeping behavior, creating an image of complexity that for most use cases - especially for those starting out - just doesn't exist.

Re: Elasticsearch was never a database

#45

Post by wdfx »

WASDx wrote: post #35
PedroBatista wrote: post #6

I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…

I've managed a 100+ node cluster for years without seeing any corruption. Where are you getting this from?

I'm actually struggling to imagine exactly what warrants a 100+ node cluster of ES?

Re: Elasticsearch was never a database

#46

Post by rectang »

kubi07 wrote: post #40
speedgoose wrote: post #11

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

I worked in a company that used elastic search as main db. It worked, company made alot of money from that project. It was a wrong decision but helped us complete the project very fast. We needed search capability and a db. ES did it both. Problems that we faced by using elastic search: High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce compan…

Databases and search engines have different engineering priorities, and data integrity is not a top tier priority for search engine developers because a search engine is assumed not to be the primary data store. Search engines are designed to build an index which augments a data store and which can be regenerated when needed.

Anyone in engineering who recommends using a search engine as a primary data store is taking on risk of data loss for their organization that most non-engineering people do not understand.

In one org I worked for, we put the search engine in front of the database for retrieval, but we also made sure that the data was going to Postgres.

Re: Elasticsearch was never a database

#47

Post by 9rx »

victor106 wrote: post #31
speedgoose wrote: post #11

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

> Accenture They messed up a $30 million dollar project big time at a previous company. My cto swore to never recommend them

I've seen some mess-ups in my life, but they started sticking out like a sore thumb long, long, long, long before anywhere close to $30 million was spent on it.

What does a $30 million dollar mess-up look like?

Re: Elasticsearch was never a database

#48

Post by jfengel »

No, of course not. But the question is, do you need a database?

A database is a big proposition: transactions, indexes, query processing, replication, distribution, etc. A fair number of use cases are just "Take this data and give it back to me when I ask for it".

ES (or any other not-a-database) might not be a full-bore DBMS. But it might be what you need.

Re: Elasticsearch was never a database

#49

Post by 9rx »

rectang wrote: post #46
kubi07 wrote: post #40

Earlier quoted context omitted.

I worked in a company that used elastic search as main db. It worked, company made alot of money from that project. It was a wrong decision but helped us complete the project very fast. We needed search capability and a db. ES did it both. Problems that we faced by using elastic search: High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce compan…

Databases and search engines have different engineering priorities, and data integrity is not a top tier priority for search engine developers because a search engine is assumed not to be the primary data store. Search engines are designed to build an index which augments a data store and which can be regenerated when needed. Anyone in engineering who recommends using a search engine as a primary data store is taking…

> Anyone in engineering who recommends using a search engine as a primary data store is taking on risk of data loss for their organization.

It is true that Elasticsearch was not designed for it, but there is no reason why another "search engine" designed for that purpose couldn't fit that role.

Re: Elasticsearch was never a database

#50

Post by wasting_time »

esafak wrote: post #32
roywiggins wrote: post #10

> Elastic has been working on this gap. The more recent ES|QL introduces a similar feature called lookup joins, and Elastic SQL provides a more familiar syntax (with no joins). But these are still bound by Lucene’s underlying index model. On top of that, developers now face a confusing sprawl of overlapping query syntaxes (currently: Query DSL, ES|QL, SQL, EQL, KQL), each suited to different use cases, and with diffe…

ICYMI https://en.wikipedia.org/wiki/Greenspun's_tenth_rule

ICYMI expands to "in case you missed it", ICYMI.
Post reply on HN