Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch was never a database

paradedb.com

Post reply on HN
Print view View original

81–90 of 109 posts

Re: Elasticsearch was never a database

#81

Post by Joker_vD »

roywiggins wrote: post #10

> Elastic has been working on this gap. The more recent ES|QL introduces a similar feature called lookup joins, and Elastic SQL provides a more familiar syntax (with no joins). But these are still bound by Lucene’s underlying index model. On top of that, developers now face a confusing sprawl of overlapping query syntaxes (currently: Query DSL, ES|QL, SQL, EQL, KQL), each suited to different use cases, and with diffe…

Just as any "plain blob storage" eventually evolves a hierarchical filesystem (but with silly quirks!) on top of it.

Re: Elasticsearch was never a database

#82

Post by 9rx »

rawgabbit wrote: post #78
9rx wrote: post #47

Earlier quoted context omitted.

I've seen some mess-ups in my life, but they started sticking out like a sore thumb long, long, long, long before anywhere close to $30 million was spent on it. What does a $30 million dollar mess-up look like?

Teams of consultants on site, some remote, and many offshore. Tons of documents are created and many environments and DevOps pipelines are stood up. First code release is when the people who push buttons touch the system for the first time. It is crap. Several more code releases attempt to make the system usable. Eventually another consultant or two are brought to evaluate the project and they say the project violate…

So much the same as what I've seen before, except instead of abandoning ship when the mess was clear and present, doubling down to see how far of a hole one can dig? Sunk cost must be one hell of a drug for the aforementioned CTO.

Re: Elasticsearch was never a database

#83

Post by ananthakumaran »

kubi07 wrote: post #40
speedgoose wrote: post #11

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

I worked in a company that used elastic search as main db. It worked, company made alot of money from that project. It was a wrong decision but helped us complete the project very fast. We needed search capability and a db. ES did it both. Problems that we faced by using elastic search: High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce compan…

ES should be thought of as a json key value store and search engine. The json key value store is fully consistent and supports read after write semantics, refresh is needed for search api. In some cases it does make sense to treat it as a database provided the key value store semantics is enough.

I used it about 7 years ago. Text search was not that heavily used, but we utilized the keyword filter heavily. It's like having a database where you can throw any query at it and it would return a response in reasonable time, because you are just creating an index on all fields.

Re: Elasticsearch was never a database

#85

Post by stackskipton »

bigfatkitten wrote: post #51
victor106 wrote: post #31

Earlier quoted context omitted.

> Accenture They messed up a $30 million dollar project big time at a previous company. My cto swore to never recommend them

How are they still in business? I’ve either been involved with or adjacent to dozens of Accenture projects at 5 companies over the last 20 years, and not a single one had a satisfactory outcome. I’ve never heard a single story of “Accenture came in, and we got what we wanted, on time and on budget.” Cases of “we got a minimum viable solution for $100m instead of $30m, and it was four years late” seem more typical.

Just like IBM, they are big enough that no one ever got fired for buying them.

I've also found they do a good job of getting cadre of executives that float between companies hiring them when they move between companies while they get wined and dined.

Re: Elasticsearch was never a database

#86

Post by wpaladin »

largbae wrote: post #84

Are the folks still using ES simply unaware of the performance advantages of ClickHouse, or is there some use case that ES covers that CH is still missing?

Full-text search. It was only added to Clickhouse relatively recently, and is still in Beta. It's a core feature of ES from the beginning.

https://clickhouse.com/docs/engines/table-engines/mergetree-...

Re: Elasticsearch was never a database

#87

Post by zacksiri »

I never thought of Elasticsearch as a database and always designed systems around what elasticsearch is supposed to be an index based document store for used with search.

I think their API is great and have had amazing results with it. Their recent innovations around quantization (bbq) has been amazing for my use case building an agentic movie database for discovering movies and personalized movie recommendations.

There are benefits to not using your database for everything, even if it adds a bit of complexity by introducing another dependency. If the benefits out weigh the cost of complexity reaching for elastic has almost always been worth it for me.

Re: Elasticsearch was never a database

#88

Post by deepsun »

Joker_vD wrote: post #81
roywiggins wrote: post #10

> Elastic has been working on this gap. The more recent ES|QL introduces a similar feature called lookup joins, and Elastic SQL provides a more familiar syntax (with no joins). But these are still bound by Lucene’s underlying index model. On top of that, developers now face a confusing sprawl of overlapping query syntaxes (currently: Query DSL, ES|QL, SQL, EQL, KQL), each suited to different use cases, and with diffe…

Just as any "plain blob storage" eventually evolves a hierarchical filesystem (but with silly quirks!) on top of it.

AFAIK, in Google it was the other way around -- their main blob storage (BigTable) is built on top of GFS (distributed filesystem).

Re: Elasticsearch was never a database

#89

Post by YetAnotherNick »

PedroBatista wrote: post #6

I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…

Neither the blogpost(beside consistency which most people don't care much about) nor your post describe any issue.

> things got corrupted and indexes needed to be rebuilt.

How is postgres and elastic any different here?

Re: Elasticsearch was never a database

#90

Post by Scaevolus »

deepsun wrote: post #88
Joker_vD wrote: post #81

Earlier quoted context omitted.

Just as any "plain blob storage" eventually evolves a hierarchical filesystem (but with silly quirks!) on top of it.

AFAIK, in Google it was the other way around -- their main blob storage (BigTable) is built on top of GFS (distributed filesystem).

You have that backwards. GFS was replaced by Colossus ca. 2010, and largely functions as blob storage with append-only semantics for modification. BigTable is a KV store, and the row size limits (256MB) make it unsuitable for blob storage. GCS is built on top of Spanner (metadata, small files) and Colossus (bulk data storage).

But that's besides the point. When people say "RDBMS" or "filesystem" they mean the full suite of SQL queries and POSIX semantics-- neither of which you get with KV stores like BigTable or distributed storage like Colossus.

The simplest example of POSIX semantics that are rapidly discarded is the "fast folder move" operation. This is difficult to impossible to achieve when you have keys representing the full path of the file, and is generally easier to implement with hierarchical directory entries. However, many applications are absolutely fine with the semantics of "write entire file, read file, delete file", which enables huge simplifications and optimizations!

Post reply on HN