Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing EBS and Rethinking Postgres Storage from First Principles

tigerdata.com

Post reply on HN
Print view View original

11–20 of 63 posts

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#11

Post by graveland »

hedora wrote: post #8
0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

Thanks for the summary. Note that those numbers are terrible vs. a physical disk, especially latency, which should be (That assumes async replication of the write ahead log to a secondary. Otherwise, write latency should be ~ 1 rtt, which is still Stacking storage like this isn’t great, but PG wasn’t really designed for performance or HA. (I don’t have a better concrete solution for ansi SQL that works today.)

(I'm on the team that made this)

The raw numbers are one thing, but the overall performance of pg is another. If you check out https://planetscale.com/blog/benchmarking-postgres-17-vs-18 for example, in the average QPS chart, you can see that there isn't a very large difference in QPS between GP3 at 10k iops and NVMe at 300k iops.

So currently I wouldn't recommend this new storage for the highest end workloads, but it's also a beta project that's still got a lot of room for growth! I'm very enthusiastic about how far we can take this!

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#12

Post by stefanha »

@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it?

Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#13

Post by electroly »

bradyd wrote: post #10
0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

> EBS only lets you resize once every 6–24 hours Is that even true? I've resized an EBS instance a few minutes after another resize before.

AWS documents it as "After modifying a volume, you must wait at least six hours and ensure that the volume is in the in-use or available state before you can modify the same volume" but community posts suggest you can get up to 8 resizes in the six hour window.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#14

Post by thesz »

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

What does EBS mean?

It is used in first line of the text but no explanation was given.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#15

Post by unsolved73 »

TimescaleDB was such a great project!

I'm really sad to see them waste the opportunity and instead build an nth managed cloud on top of AWS, chasing buzzword after buzzword.

Had they made deals with cloud providers to offer managed TimescaleDB so they can focus on their core value proposition they could have won the timeseries business, but ClickHouse made them irrelevant and Neon already has won the "Postgres for agents" business thanks to a better architecture than this.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#16

Post by karanbhangui »

thesz wrote: post #14
0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

What does EBS mean? It is used in first line of the text but no explanation was given.

https://aws.amazon.com/ebs/

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#17

Post by graveland »

stefanha wrote: post #12

@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it? Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.

There's some secret sauce there I don't know if I'm allowed to talk about yet, so I'll just address the existing tech that we didn't use: most things either didn't have a good enough license, cost too much, would take a TON of ramp-up and expertise we don't currently have to manage and maintain, but generally speaking, our stuff allows us to fully control it.

Entirely programmable storage so far has allowed us to try a few different things to try and make things efficient and give us the features we want. We've been able to try different dedup methods, copy-on-write styles, different compression methods and types, different sharding strategies... All just as a start. We can easily and quickly create a new experimental storage backends and see exactly how pg performs with it side-by-side with other backends.

We're a kubernetes shop, and we have our own CSI plugin, so we can also transparently run a pg HA pair with one pg server using EBS and the other running in our new storage layer, and easily bounce between storage types with nothing but a switchover event.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#18

Post by everfrustrated »

the8472 wrote: post #5

Though AWS instance-attached NVMe(oF?) still has less IOPS per TB than bare metal NVMe does. E.g. i8g.2xlarge, 1875 GB, 300k IOPS read vs. WD_BLACK SN8100, 2TB, 2300k IOPS read

You can't do those rates 24x7 on a WD_BLACK tho.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#19

Post by jasonthorsness »

bradyd wrote: post #10
0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

> EBS only lets you resize once every 6–24 hours Is that even true? I've resized an EBS instance a few minutes after another resize before.

The 6-hour counter is most certainly, painfully true. If you work with an AWS rep please complain about this in every session; maybe if we all do they will reduce the counter :P.
Post reply on HN