Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch was never a database

paradedb.com

Post reply on HN
Print view View original

21–30 of 109 posts

Re: Elasticsearch was never a database

#21

Post by kentm »

unethical_ban wrote: post #3

I work in infosec and several popular platforms use elasticsearch for log storage and analysis. I would never. Ever. Bet my savings on ES being stable enough to always be online to take in data, or predictable in retaining the data it took in. It feels very best-effort and as a consultant, I recommend orgs use some other system for retaining their logs, even a raw filesystem with rolling zips, before relying on ES un…

Do you happen to know if ES was the only storage? Its been almost 8 years, but if I was building a log storage and analysis system, then I'd push the logs to S3 or some other object store and build an ES index off of that S3 data. From the consumer's perspective, it may look like we're using ES to store the data, but we have a durable backup to regenerate ES if necessary.

Re: Elasticsearch was never a database

#24

Post by cyberpunk »

unethical_ban wrote: post #3

I work in infosec and several popular platforms use elasticsearch for log storage and analysis. I would never. Ever. Bet my savings on ES being stable enough to always be online to take in data, or predictable in retaining the data it took in. It feels very best-effort and as a consultant, I recommend orgs use some other system for retaining their logs, even a raw filesystem with rolling zips, before relying on ES un…

Meh i run hundreds of es nodes, its gotten a lot more friendly these days, but yes it can be a bit unforgiving at times.

Turns out running complicated large distributed systems requires a bit more than a ./apply, who would have guessed it?

Re: Elasticsearch was never a database

#25

Post by vjerancrnjak »

PedroBatista wrote: post #6

I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…

They market it as a general purpose store. Successfully, even though hc cs wizards wouldn’t touch it ever, c suite likes it

Best example is IoT marketing, as if it can handle the load without bazillion shards, and since when does a text engine want telemetry

Re: Elasticsearch was never a database

#26

Post by _joel »

nkmnz wrote: post #15
cluckindan wrote: post #13

”That means a recently acknowledged write may not show up until the next refresh.” Which is why you supply the parameter refresh: ”wait_for” in your writes. This forces a refresh and waits for it to happen before completing the request. ”schema migrations require moving the entire system of record into a new structure, under load, with no safety net” Use index aliases. Create new index using the new mapping, make a r…

It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.

I just tend to use https://github.com/le0pard/pgtune

Re: Elasticsearch was never a database

#28

Post by CuriouslyC »

speedgoose wrote: post #11

Accenture managed to build a data platform for my company with Elasticsearch as the primary database. I raised concerns early during the process but their software architect told me they never had any issues. I assume he didn’t lie. I was only an user so I didn’t fight and decided to not make my work rely on their work.

Elastic feels about as much like a primary data store as Mongo, FWIW.

Re: Elasticsearch was never a database

#29

Post by xeraa »

toenail wrote: post #9

I think elastic always clearly documented to expect "eventual consistency", they never claimed to be a "database" in the sense that tfa defines.

First step of a marketing campaign: Claim something never said and then tell everyone why it's wrong ;)

Re: Elasticsearch was never a database

#30

Post by xeraa »

unethical_ban wrote: post #7
toenail wrote: post #4

Earlier quoted context omitted.

Dunno, I've had three node clusters running very stable for years. Which issues did you have that require a full team?

To be fair, I think it is chronically underprovisioned clusters that get overwhelmed by log forwarding. I wasn't on the team that managed the ELK stack a decade ago, but I remember our SOC having two people whose full time job was curating the infrastructure to keep it afloat. Now I work for a company whose log storage product has ES inside, and it seems to shit the bed more often than it should - again, could be bug…

There are no 2-node clusters (it needs a quorum). If your setup has 2-node clusters, someone is doing this horribly wrong.
Post reply on HN