Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lakebase architecture delivers faster Postgres writes

databricks.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Lakebase architecture delivers faster Postgres writes

#21

Post by [deleted] »

gavinray wrote: post #18
nikita wrote: post #16

Earlier quoted context omitted.

Lakebase is referring to the fact that in addition to disaggregated storage s3 is authoritative storage for older data. Since data is on s3 (or lake) you can perform direct to s3 type operations like data loading, reading this data by engines that are not Postgres and more

> in addition to disaggregated storage s3 is authoritative storage for older data Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion. But if you say "Disaggregated Storage on S3" then you have the flexibility to change that to "Disaggregated Storage on FOOBAR" to avoid confusion.

[deleted]

Re: Lakebase architecture delivers faster Postgres writes

#22

Post by jeremyjh »

nikita wrote: post #2

I'm a VP on Databricks and former CEO of Neon. Happy to answer performance related or any other questions here.

In the blog article[1] that linked to, it says "Unified transactional and analytical workloads: Lakebase integrates seamlessly with the Lakehouse, sharing the same storage layer across OLTP and OLAP. This makes it possible to run real-time analytics, machine learning, and AI-driven optimization directly on transactional data without moving or duplicating it."

Is the "without moving or duplicating" part actually a true statement? If the actual table state is only reconstructed by the pageserver, its not like Spark can just read it from S3.

[1] https://www.databricks.com/blog/what-is-a-lakebase

Re: Lakebase architecture delivers faster Postgres writes

#23

Post by majormajor »

gavinray wrote: post #18
nikita wrote: post #16

Earlier quoted context omitted.

Lakebase is referring to the fact that in addition to disaggregated storage s3 is authoritative storage for older data. Since data is on s3 (or lake) you can perform direct to s3 type operations like data loading, reading this data by engines that are not Postgres and more

> in addition to disaggregated storage s3 is authoritative storage for older data Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion. But if you say "Disaggregated Storage on S3" then you have the flexibility to change that to "Disaggregated Storage on FOOBAR" to avoid confusion.

> Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion.

I've never seen "lake" or adjacent terminology refer to S3 specifically like that vs other object storage. A data lake on Ceph would still be a data lake.

(My quibble would be that "lake" often refers to inconsistent or unstructured, and itself has always been a bit handwavy compared to "warehouse," whereas this is very structured data on object storage.)

Re: Lakebase architecture delivers faster Postgres writes

#24

Post by andyferris »

majormajor wrote: post #23
gavinray wrote: post #18

Earlier quoted context omitted.

> in addition to disaggregated storage s3 is authoritative storage for older data Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion. But if you say "Disaggregated Storage on S3" then you have the flexibility to change that to "Disaggregated Storage on FOOBAR" to avoid confusion.

> Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion. I've never seen "lake" or adjacent terminology refer to S3 specifically like that vs other object storage. A data lake on Ceph would still be a data lake. (My quibble would be that "lake" often refers to inconsistent or unstructured,…

Yes.

Maybe I’m wrong, but AFAICT this is block (page) storage backed by S3, tuned for Postgres with some paxos-linked storage/caching servers sitting in front? Sounds good, but I’m not sure “lake” or “warehouse” is a word I’d choose… much closer to Litestream-with-reads, or the somewhat-famous “I ran out of RAM so I downloaded some more” blog article.

Re: Lakebase architecture delivers faster Postgres writes

#25

Post by nikita »

Veelox wrote: post #9
nikita wrote: post #2

I'm a VP on Databricks and former CEO of Neon. Happy to answer performance related or any other questions here.

Thanks for offering. In the graph labeled "Prod customer throughput: (higher is better)" eyeballing it within a week you are seeing ~2k qps peak increase over the previous week. Operationally, how do you handle landing that large of a perf improvement? If my data store changed that much in a week it could break something.

Generally the more throughput the system supports the better. In this case we were hitting limits (btw each operation is many queries of different sizes) and the customer observed higher latencies which is typical if the system can't sustain the throughput required.

After this change latencies are back to normal and throughput increased.

Re: Lakebase architecture delivers faster Postgres writes

#27

Post by Veelox »

nikita wrote: post #25
Veelox wrote: post #9

Earlier quoted context omitted.

Thanks for offering. In the graph labeled "Prod customer throughput: (higher is better)" eyeballing it within a week you are seeing ~2k qps peak increase over the previous week. Operationally, how do you handle landing that large of a perf improvement? If my data store changed that much in a week it could break something.

Generally the more throughput the system supports the better. In this case we were hitting limits (btw each operation is many queries of different sizes) and the customer observed higher latencies which is typical if the system can't sustain the throughput required. After this change latencies are back to normal and throughput increased.

Ahh, so it was a customer pain point of higher latency so they were happy to see latency go down and throughput go up. Good to hear.

Great write up, cheers to the people involved.

Re: Lakebase architecture delivers faster Postgres writes

#28

Post by hardwaresofton »

This is essentially a re-explanation of Neon’s architecture as a blog post.

Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech.

The Postgres ecosystem is pretty unique in its ability to produce a “boring” stable product, innovate, stay F/OSS, and create financial outcomes for participants.

Re: Lakebase architecture delivers faster Postgres writes

#29

Post by xyzzy_plugh »

hardwaresofton wrote: post #28

This is essentially a re-explanation of Neon’s architecture as a blog post. Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech. The Postgres ecosystem…

As far as I know Neon's open source repositories are no longer being updated/maintained.

Re: Lakebase architecture delivers faster Postgres writes

#30

Post by faangguyindia »

Most problems with overgrowing data can be fixed by having data deletion rule.

Many people just keep adding data and think "maybe it will be useful in future" till their system goes down.

Many of your data is essentially useless for anything in future.

You can simply have data retention policy and for most app this ensures your data does not grow top huge

Post reply on HN