Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lakebase architecture delivers faster Postgres writes

databricks.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Lakebase architecture delivers faster Postgres writes

#31

Post by redwood »

faangguyindia wrote: post #30

Most problems with overgrowing data can be fixed by having data deletion rule. Many people just keep adding data and think "maybe it will be useful in future" till their system goes down. Many of your data is essentially useless for anything in future. You can simply have data retention policy and for most app this ensures your data does not grow top huge

This is orthogonal to many classes of write throughput. And in fact in many cases deletes themselves are effectively adding to the write workload

Re: Lakebase architecture delivers faster Postgres writes

#32

Post by hardwaresofton »

xyzzy_plugh wrote: post #29
hardwaresofton wrote: post #28

This is essentially a re-explanation of Neon’s architecture as a blog post. Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech. The Postgres ecosystem…

As far as I know Neon's open source repositories are no longer being updated/maintained.

You can't have it all, forever -- the tech is there for anyone to fork and improve, build new businesses (!), inspect, etc. F/OSS is basically a miracle as-is.

If we compare the current state of the world to one in which they were acquired and then continued to put out more F/OSS, things look bad (which I assume is your implication). I choose to instead make the comparison to the world where we never see this tech and it stays proprietary. Sure, eventually someone in F/OSS might have gotten around to building this solution, but they pulled forward the future and we get to see and build on the result for free.

Re: Lakebase architecture delivers faster Postgres writes

#33

Post by hasyimibhar »

faangguyindia wrote: post #30

Most problems with overgrowing data can be fixed by having data deletion rule. Many people just keep adding data and think "maybe it will be useful in future" till their system goes down. Many of your data is essentially useless for anything in future. You can simply have data retention policy and for most app this ensures your data does not grow top huge

In some cases you have no choice but to retain the data, e.g. due to compliance. But the good thing is it doesn't have to be in Postgres. You can periodically offload data to a lakehouse, then delete it from Postgres. If the table is partitioned, delete should be cheap.

I'm guessing with Neon, since their storage is a lakehouse, you get this for free.

Re: Lakebase architecture delivers faster Postgres writes

#34

Post by mixolydianagain »

nikita wrote: post #2

I'm a VP on Databricks and former CEO of Neon. Happy to answer performance related or any other questions here.

Hi Nikita. Can you share any of Neon's techniques for minimizing noisy neighbor issues in the multi tenant storage services? Thanks!

Re: Lakebase architecture delivers faster Postgres writes

#35

Post by philippemnoel »

hardwaresofton wrote: post #28

This is essentially a re-explanation of Neon’s architecture as a blog post. Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech. The Postgres ecosystem…

This a 100 times^. The Postgres ecosystem is remarkable and has managed to strike the balance between OSS and commercial successes in a way that most infra verticals have not.

Re: Lakebase architecture delivers faster Postgres writes

#36

Post by conradludgate »

hardwaresofton wrote: post #28

This is essentially a re-explanation of Neon’s architecture as a blog post. Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech. The Postgres ecosystem…

(Neon/databricks employee here)

Neon also only just disabled FPWs - so there is new substance here. We published a similar blog on Neon

https://neon.com/blog/turning-off-fpw-for-faster-writes

Re: Lakebase architecture delivers faster Postgres writes

#38

Post by mystraline »

nikita wrote: post #10
mystraline wrote: post #5

Im not a proper DBA, but oversee some basic postgres installs (read: logging, monitoring, upgrades). This appears to only have any effect with datalake style installs, where storage is separate from compute. Not going to have any effect on those small postgres installs for that generic one off app.

We provide you fully managed Postgres. Lots of our customers use it for lots of small instances of Postgres since using Lakebase is so lightweight. Small and large instances benefit from this performance optimization.

Again, when the users of this app are just 3 admins, over basic metrics from ArcGIS Monitor, I dont need to deviate from their recommended, tune, or much of anything.

This isnt some high throughput app where every IOPS matters.

And in my experiences, most apps that need a database arent serving 1000's of people per hour. Most are fine with a few users per hour.

The whole "scale to scale cause youre 1 minute away from HN hug of death" is just wishful thinking a budget bloat from the hyperscalers selling 'butwhatifs'. Ive called AWS to task on their 5 pillars crap, which basically says "pay double for redundancy". Gee, I wonder why AWS would recommend customers to pay double (eyeroll).

Re: Lakebase architecture delivers faster Postgres writes

#39

Post by nikita »

mixolydianagain wrote: post #34
nikita wrote: post #2

I'm a VP on Databricks and former CEO of Neon. Happy to answer performance related or any other questions here.

Hi Nikita. Can you share any of Neon's techniques for minimizing noisy neighbor issues in the multi tenant storage services? Thanks!

* Rate limiting on proxy in front of compute fleet

* Large tenants are broken up into shards, reducing hotspots

* Each shard is throttled to a fixed req/s rate

* We do not run pageservers at their redline in terms of CPU load, so there is some slack to take up bursts

* Capacity quotas which selectively throttle write traffic to the largest databases if they are competing with others for disk space, until the larger database is migrated away.

Post reply on HN