Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch was never a database

paradedb.com

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Elasticsearch was never a database

#91

Post by tetha »

nkmnz wrote: post #15
cluckindan wrote: post #13

”That means a recently acknowledged write may not show up until the next refresh.” Which is why you supply the parameter refresh: ”wait_for” in your writes. This forces a refresh and waits for it to happen before completing the request. ”schema migrations require moving the entire system of record into a new structure, under load, with no safety net” Use index aliases. Create new index using the new mapping, make a r…

It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.

PostgreSQL has 2 memory-related parameters you need to set for larger instances - work_mem and shared_buffers, as these need to be set to a percentage of the VMs memory to utilize it well. However, pretty much every PostgreSQL setup guide names these two values, and on a managed PostgreSQL hosting I'd expect these to be set.

Outside of memory, log_duration, temp_file_limit, a good query plan visualizer and some backup and replication (e.g. PGBackrest and Patroni) are also generally recommended if self-hosting. Patroni doesn't even need an external config store anymore, which is great since you can just run it onto 3-4 nodes and get a high quality HA, easy to manage PostgreSQL cluster.

But those two parameters are pretty much all to have a PostgreSQL process thousands of transactions per second without further tuning. Even our larger DBs hosting simple REST-applications (opposed to ETL/Data warehousing) had to grow quite a lot until further configuration was necessary, if at all.

Checkpointing probably becomes the next issue then, but modern PostgreSQL actually has great logging there -- "Checkpoints occur too frequently, consider these 5 config parameters to look at". And don't touch VACUUM jobs, as a consultant once joked, he sometimes earns thousands of dollars to say "You killed a VACUUM job? Don't do that".

So yeah, actually running PostgreSQL takes a few considerations, but compared to 10 - 15 years ago, you can get a lot with little effort.

Re: Elasticsearch was never a database

#94

Post by lillesvin »

vedhant wrote: post #67

Ofcourse it is not meant as a primary database. What baffles me is that people use it as log storage. As an application scales, storage and querying logs become the bottleneck if elasticsearch is used. I was dealing with a system that could afford only 1 week of log retention!

What kind of storage do you have backing your Elasticsearch? And how have you configured sharding and phase rollover in your indices?

I work with a cluster that holds 500+ TB logs (where most are stored for a year and some for 5 years because of regulations) in searchable snapshots backed by a locally hosted S3 solution. I can do filtering across most of the data in less than 10 seconds.

Some especially gnarly searches may take around 60-90 seconds on the first run as the searchable snapshots are mounted and cached, but subsequent searches in the cached dataset are obviously as fast as any other search in hot data.

Obviously Elasticsearch isn't without its quirks and drawbacks, but I have yet to come across anything that performs better and is more flexible for logs — especially in terms of architectural freedom and bang-for-the-buck.

Re: Elasticsearch was never a database

#95

Post by lillesvin »

kentm wrote: post #21
unethical_ban wrote: post #3

I work in infosec and several popular platforms use elasticsearch for log storage and analysis. I would never. Ever. Bet my savings on ES being stable enough to always be online to take in data, or predictable in retaining the data it took in. It feels very best-effort and as a consultant, I recommend orgs use some other system for retaining their logs, even a raw filesystem with rolling zips, before relying on ES un…

Do you happen to know if ES was the only storage? Its been almost 8 years, but if I was building a log storage and analysis system, then I'd push the logs to S3 or some other object store and build an ES index off of that S3 data. From the consumer's perspective, it may look like we're using ES to store the data, but we have a durable backup to regenerate ES if necessary.

Searchable snapshots in Elasticsearch can be backed by S3 and they perform very well. No need to store the data on hot nodes any longer than it takes for the index to do a rollover, and from then it's all S3.

Re: Elasticsearch was never a database

#96

Post by nkmnz »

_joel wrote: post #26
nkmnz wrote: post #15

Earlier quoted context omitted.

It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.

I just tend to use https://github.com/le0pard/pgtune

I agree, but that doesn’t constitute „carefully tuning“ a config, at least not in my book. OP certainly didn’t mean to imply that it’s enough to follow along a 5min tutorial.

Re: Elasticsearch was never a database

#97

Post by nkmnz »

tetha wrote: post #91
nkmnz wrote: post #15

Earlier quoted context omitted.

It took me years before I started tuning the memory-related configuration of postgres for workload, schema and data, in any way. It "just works" for the first ten thousand concurrent users.

PostgreSQL has 2 memory-related parameters you need to set for larger instances - work_mem and shared_buffers, as these need to be set to a percentage of the VMs memory to utilize it well. However, pretty much every PostgreSQL setup guide names these two values, and on a managed PostgreSQL hosting I'd expect these to be set. Outside of memory, log_duration, temp_file_limit, a good query plan visualizer and some backu…

Exactly! Thank you for sharing your expertise!

Re: Elasticsearch was never a database

#100

Post by cluckindan »

nkmnz wrote: post #44
cluckindan wrote: post #16

Earlier quoted context omitted.

Well, most people working on a car don’t have a car lift: it only makes sense when you need to safely work on a large volume of cars. If you only work on one or two, a jack and a pile of wood works just fine.

Please don't move the goal post. Writing `no database ”just works” without (...)` is gatekeeping behavior, creating an image of complexity that for most use cases - especially for those starting out - just doesn't exist.

In fairness, it doesn’t exist for Elasticsearch either.
Post reply on HN