Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.
Elasticsearch was never a database
11–20 of 109 posts
Re: Elasticsearch was never a database
#12Earlier quoted context omitted.
Dunno, I've had three node clusters running very stable for years. Which issues did you have that require a full team?
To be fair, I think it is chronically underprovisioned clusters that get overwhelmed by log forwarding. I wasn't on the team that managed the ELK stack a decade ago, but I remember our SOC having two people whose full time job was curating the infrastructure to keep it afloat. Now I work for a company whose log storage product has ES inside, and it seems to shit the bed more often than it should - again, could be bug…
I'm not even sure "get overwhelmed" is a problem, unless you need real time analytics. But yeah, sounds like a resources issue.
Re: Elasticsearch was never a database
#13”That means a recently acknowledged write may not show up until the next refresh.”
Which is why you supply the parameter
refresh: ”wait_for”
”schema migrations require moving the entire system of record into a new structure, under load, with no safety net”
Use index aliases. Create new index using the new mapping, make a reindex request from old index to new one. When it finishes, change the alias to point to the new index.
The other criticisms are more valid, but not entirely: for example, no database ”just works” without carefully tuning the memory-related configuration for your workload, schema and data.
Re: Elasticsearch was never a database
#14I work in infosec and several popular platforms use elasticsearch for log storage and analysis. I would never. Ever. Bet my savings on ES being stable enough to always be online to take in data, or predictable in retaining the data it took in. It feels very best-effort and as a consultant, I recommend orgs use some other system for retaining their logs, even a raw filesystem with rolling zips, before relying on ES un…
Dunno, I've had three node clusters running very stable for years. Which issues did you have that require a full team?
Even most toy databases "built in a weekend" can be very stable for years if:
- No edge-case is thrown at them
- No part of the system is stressed ( software modules, OS,firmware, hardware )
- No plug is pulled
Crank the requests to 11 or import a billion rows of data with another billion relations and watch what happens. The main problem isn't the system refusing to serve a request or throwing "No soup for you!" errors, it's data corruption and/or wrong responses.
Re: Elasticsearch was never a database
#15”That means a recently acknowledged write may not show up until the next refresh.” Which is why you supply the parameter refresh: ”wait_for” in your writes. This forces a refresh and waits for it to happen before completing the request. ”schema migrations require moving the entire system of record into a new structure, under load, with no safety net” Use index aliases. Create new index using the new mapping, make a r…
It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.
Re: Elasticsearch was never a database
#16”That means a recently acknowledged write may not show up until the next refresh.” Which is why you supply the parameter refresh: ”wait_for” in your writes. This forces a refresh and waits for it to happen before completing the request. ”schema migrations require moving the entire system of record into a new structure, under load, with no safety net” Use index aliases. Create new index using the new mapping, make a r…
It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.
Well, most people working on a car don’t have a car lift: it only makes sense when you need to safely work on a large volume of cars. If you only work on one or two, a jack and a pile of wood works just fine.
Re: Elasticsearch was never a database
#17Earlier quoted context omitted.
Dunno, I've had three node clusters running very stable for years. Which issues did you have that require a full team?
Even most toy databases "built in a weekend" can be very stable for years if: - No edge-case is thrown at them - No part of the system is stressed ( software modules, OS,firmware, hardware ) - No plug is pulled Crank the requests to 11 or import a billion rows of data with another billion relations and watch what happens. The main problem isn't the system refusing to serve a request or throwing "No soup for you!" err…
I'm talking about production loads, but thanks.
Re: Elasticsearch was never a database
#18I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…
> I really never understood how people could store very important information in ES like it was a database.
I agree.
Its been a while since I touched it, but as far as I can remember ES has never pretended to be your primary store of information. It was mostly juniors that reached for it for transaction processing, and I had to disabuse them of the notion that it was fit for purpose there.
ES is for building a searchable replica of your data. Every ES deployment I made or consulted sourced its data from some other durable store, and the only thing that wrote to it were replication processes or backfills.
Re: Elasticsearch was never a database
#19”That means a recently acknowledged write may not show up until the next refresh.” Which is why you supply the parameter refresh: ”wait_for” in your writes. This forces a refresh and waits for it to happen before completing the request. ”schema migrations require moving the entire system of record into a new structure, under load, with no safety net” Use index aliases. Create new index using the new mapping, make a r…
It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.
Modern JVMs are pretty effective in most scenarios right out of the box.
Re: Elasticsearch was never a database
#20> Elastic has been working on this gap. The more recent ES|QL introduces a similar feature called lookup joins, and Elastic SQL provides a more familiar syntax (with no joins). But these are still bound by Lucene’s underlying index model. On top of that, developers now face a confusing sprawl of overlapping query syntaxes (currently: Query DSL, ES|QL, SQL, EQL, KQL), each suited to different use cases, and with diffe…
... and then becomes an email client (https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski%27s_La...). A two-fer. lol.