Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch was never a database

paradedb.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: Elasticsearch was never a database

#101

Post by cluckindan »

aPoCoMiLogin wrote: post #75
cluckindan wrote: post #13

”That means a recently acknowledged write may not show up until the next refresh.” Which is why you supply the parameter refresh: ”wait_for” in your writes. This forces a refresh and waits for it to happen before completing the request. ”schema migrations require moving the entire system of record into a new structure, under load, with no safety net” Use index aliases. Create new index using the new mapping, make a r…

it's the other way around: `wait_for` waits for the next refresh (there is configurable refresh interval, 1s by default), `refresh: true` forces refresh without waiting for the next refresh interval. the difference is that waiting for refresh assures that the data will be available for search after the "insert" finishes. forcing refresh might be foot gun that will criple the servers.

Huh. Makes sense. Well, that’s the right option to use, anyway.

Re: Elasticsearch was never a database

#102

Post by dfunckt »

YetAnotherNick wrote: post #89
PedroBatista wrote: post #6

I really never understood how people could store very important information in ES like it was a database. Even if they don't understand what ES is and what a "normal" database is, I'm sure some of those people run into issues where their "db" got either corrupted of lost data even when testing and building their system around it. This is and was general knowledge at the time, it was no secret that from time to time t…

Neither the blogpost(beside consistency which most people don't care much about) nor your post describe any issue. > things got corrupted and indexes needed to be rebuilt. How is postgres and elastic any different here?

I’ve no experience with Elastic but what they’re getting at I think is indexes in Elastic is actually your data because that’s all it does due to the purpose it was built for, whereas in Postgres indexes are, well, indexes — that is, derived data, not the source of truth.

Re: Elasticsearch was never a database

#103

Post by nkmnz »

cluckindan wrote: post #100
nkmnz wrote: post #44

Earlier quoted context omitted.

Please don't move the goal post. Writing `no database ”just works” without (...)` is gatekeeping behavior, creating an image of complexity that for most use cases - especially for those starting out - just doesn't exist.

In fairness, it doesn’t exist for Elasticsearch either.

I have no clue about Elasticsearch, so you might be right – but on the other hand, you just contradicted your own statement about how difficult databases are, so I have no idea which of your statements I should trust.

Re: Elasticsearch was never a database

#105

Post by cluckindan »

nkmnz wrote: post #103
cluckindan wrote: post #100

Earlier quoted context omitted.

In fairness, it doesn’t exist for Elasticsearch either.

I have no clue about Elasticsearch, so you might be right – but on the other hand, you just contradicted your own statement about how difficult databases are, so I have no idea which of your statements I should trust.

Horses are great and run on grass. Formula cars are difficult to maintain. Cruise liners are even worse!

Re: Elasticsearch was never a database

#106

Post by YetAnotherNick »

dfunckt wrote: post #102
YetAnotherNick wrote: post #89

Earlier quoted context omitted.

Neither the blogpost(beside consistency which most people don't care much about) nor your post describe any issue. > things got corrupted and indexes needed to be rebuilt. How is postgres and elastic any different here?

I’ve no experience with Elastic but what they’re getting at I think is indexes in Elastic is actually your data because that’s all it does due to the purpose it was built for, whereas in Postgres indexes are, well, indexes — that is, derived data, not the source of truth.

But if data is corrupt, how is rebuilding index fixing anything. What kind of corruption are we talking about.

Re: Elasticsearch was never a database

#107

Post by deepsun »

Scaevolus wrote: post #90
deepsun wrote: post #88

Earlier quoted context omitted.

AFAIK, in Google it was the other way around -- their main blob storage (BigTable) is built on top of GFS (distributed filesystem).

You have that backwards. GFS was replaced by Colossus ca. 2010, and largely functions as blob storage with append-only semantics for modification. BigTable is a KV store, and the row size limits (256MB) make it unsuitable for blob storage. GCS is built on top of Spanner (metadata, small files) and Colossus (bulk data storage). But that's besides the point. When people say "RDBMS" or "filesystem" they mean the full su…

Thank you, yes my knowledge was very outdated, waay before Spanner.

Spanner for GCS actually explains how public Google Cloud was always ACID for object listing, while S3 only implemented it around 2020. I always suspected that there must be some very hard piece to implement that AWS didn't have until 2020. Makes sense now that that piece was Spanner.

Re: Elasticsearch was never a database

#108

Post by nkmnz »

cluckindan wrote: post #105
nkmnz wrote: post #103

Earlier quoted context omitted.

I have no clue about Elasticsearch, so you might be right – but on the other hand, you just contradicted your own statement about how difficult databases are, so I have no idea which of your statements I should trust.

Horses are great and run on grass. Formula cars are difficult to maintain. Cruise liners are even worse!

Then maybe you shouldn’t make a single statement about all means of transportation, claiming that you needed to be horse whisperer to ride any kind of bike, car, train, ship or plane.

Re: Elasticsearch was never a database

#109

Post by yencabulator »

unethical_ban wrote: post #3

I work in infosec and several popular platforms use elasticsearch for log storage and analysis. I would never. Ever. Bet my savings on ES being stable enough to always be online to take in data, or predictable in retaining the data it took in. It feels very best-effort and as a consultant, I recommend orgs use some other system for retaining their logs, even a raw filesystem with rolling zips, before relying on ES un…

> I work in infosec and several popular platforms use elasticsearch for log storage and analysis.

Storing logs in ElasticSearch is just stupid, as it does not preserve order:

https://logstash.jira.com/browse/LOGSTASH-192

Post reply on HN