Replacing EBS and Rethinking Postgres Storage from First Principles • Hacker News Board

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#51

Post by 0xbadcafebee » Fri, Oct 31, 2025, 12:37 AM UTC

jread wrote: post #34

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

I'm working on graduate research evaluating AWS control and data plane performance. EBS volume attachment is typically ~11s for GP2/GP3 and ~20-25s for other types. 1ms read / 5ms write latencies seem high for 4k blocks. IO1/IO2 is typically ~0.5ms RW, and GP2/GP3 ~0.6ms read and ~0.94ms write. References: https://cloudlooking.glass/matrix/#aws.ebs.us-east-1--cp--at... https://cloudlooking.glass/matrix/#aws.ebs.*--dp…

You might want to add the bit from the blog about worst-case attach times to your research. From my own experience (though it was years ago), sometimes an EBS volume would fail and simply never return. Definitely won't be acceptable for some use cases

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#52

Post by tanelpoder » Fri, Oct 31, 2025, 12:58 AM UTC

If anyone is interested in reading about a similar ”local-NVMe made redundant & shared over network as block devices” engine, last year I did some testing of Silk’s cloud block storage solution (1.3M x 8kB IOPS and 20 GiB/s throughput when reading the block store from a single GCP VM). They’re using iSCSI with multipathing on the client side instead of a userspace driver:

https://tanelpoder.com/posts/testing-the-silk-platform-in-20...

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#53

Post by xpe » Fri, Oct 31, 2025, 1:18 AM UTC

thr0w wrote: post #3

Postgres for agents, of course! It makes too much sense.

Hard to say if the above comment is serious or sarcastic.

To my eye, seeing "Agentic Postgres" at the top of the page, in yellow, is not persuasive; it comes across as bandwagony. (About me: I try to be open but critical about new tech developments; I try out various agentic tooling often.).

But I'm not dismissing the product. I'm just saying this part is what I found persuasive:

> Agents spin up environments, test code, and evolve systems continuously. They need storage that can do the same: forking, scaling, and provisioning instantly, without manual work or waste.

That explains it clearly in my opinion.

* Seems to me, there are taglines that only work after someone in "on-board". I think "Agentic Postgres" is that kind of tagline. I don't have a better suggestion in mind at the moment, though, sorry.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#54

Post by jread » Fri, Oct 31, 2025, 1:18 AM UTC

0xbadcafebee wrote: post #51

jread wrote: post #34

Earlier quoted context omitted.

I'm working on graduate research evaluating AWS control and data plane performance. EBS volume attachment is typically ~11s for GP2/GP3 and ~20-25s for other types. 1ms read / 5ms write latencies seem high for 4k blocks. IO1/IO2 is typically ~0.5ms RW, and GP2/GP3 ~0.6ms read and ~0.94ms write. References: https://cloudlooking.glass/matrix/#aws.ebs.us-east-1--cp--at... https://cloudlooking.glass/matrix/#aws.ebs.*--dp…

You might want to add the bit from the blog about worst-case attach times to your research. From my own experience (though it was years ago), sometimes an EBS volume would fail and simply never return. Definitely won't be acceptable for some use cases

Yes, we've been testing volume attachments every 5m since start of the year, and have experienced 100-150 attachment failures per volume type in that time frame during multiple events (most recently last week):

https://cloudlooking.glass/dashboard/#aws.ebs.us-east-1--cp-...

Another interesting bit, is last March AWS changed something in the control plane which both triggered a multi-day LSE, and ultimately increased attachment times from 2-3s to 10-20s (also visible in the graphs).

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#55

Post by sgichohi » Fri, Oct 31, 2025, 3:16 AM UTC

tanelpoder wrote: post #52

If anyone is interested in reading about a similar ”local-NVMe made redundant & shared over network as block devices” engine, last year I did some testing of Silk’s cloud block storage solution (1.3M x 8kB IOPS and 20 GiB/s throughput when reading the block store from a single GCP VM). They’re using iSCSI with multipathing on the client side instead of a userspace driver: https://tanelpoder.com/posts/testing-the-silk…

[dead]

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#56

Post by namibj » Fri, Oct 31, 2025, 3:30 AM UTC

otterley wrote: post #48

I’m a bit confused about this: “The storage device driver exposes Fluid Storage volumes as standard Linux block devices mountable with filesystems such as ext4 or xfs. It...allows volumes to be resized dynamically while online.” Yet an xfs file system cannot be shrunk at all, and an ext4 filesystem cannot be shrunk without first unmounting it. Are you simply doing thin provisioning of these volumes, so they appear to…

..... They are block devices. Those allow resizing.

They can be used with, for example, the listed file systems.

No one claimed the listed file systems would (usefully) cooperate with (all aspects of) the block device's resizing.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#57

Post by otterley » Fri, Oct 31, 2025, 3:43 AM UTC

namibj wrote: post #56

otterley wrote: post #48

I’m a bit confused about this: “The storage device driver exposes Fluid Storage volumes as standard Linux block devices mountable with filesystems such as ext4 or xfs. It...allows volumes to be resized dynamically while online.” Yet an xfs file system cannot be shrunk at all, and an ext4 filesystem cannot be shrunk without first unmounting it. Are you simply doing thin provisioning of these volumes, so they appear to…

..... They are block devices. Those allow resizing. They can be used with, for example, the listed file systems. No one claimed the listed file systems would (usefully) cooperate with (all aspects of) the block device's resizing.

You can’t just shrink a block device any way you want. It has to be done in concert with the filesystem on top of it to ensure any allocated blocks in the space to be deallocated are relocated to a safe area in the volume. Otherwise, data could be lost and filesystem corruption could occur.

Put differently, there is no point in being able to shrink a volume if you can’t safely shrink the filesystem that uses it.

The usual solution to this problem is thin provisioning, where you put a translation layer between the blocks the filesystem thinks it’s using and the actual underlying blocks. With thin provisioning you can allocate only, say, 1GB to the physical storage, but the block device presents itself as much larger than that, so you can pretend to create a 1PB filesystem on top of it.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#58

Post by electroly » Fri, Oct 31, 2025, 4:06 AM UTC

otterley wrote: post #47

electroly wrote: post #30

EC2 instances have dedicated throughput to EBS via Nitro that you lose out on when you run your own EBS equivalent over the regular network. You only get 5Gbps maximum between two EC2 instances in the same AZ that aren't in the same placement group[1], and you're limited by the instance type's general networking throughput. Dedicated throughput to EBS from a typical EC2 instance is multiple times this figure. It's an…

That 5Gbps limit is per flow (e.g. TCP connection), not per instance pair. With enough concurrent flows, you can saturate the interface bandwidth between peers, even if it’s 200Gbps or more.

Ah, of course! Thank you for this correction.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#59

Post by antonkochubey » Fri, Oct 31, 2025, 1:23 PM UTC

the8472 wrote: post #5

Though AWS instance-attached NVMe(oF?) still has less IOPS per TB than bare metal NVMe does. E.g. i8g.2xlarge, 1875 GB, 300k IOPS read vs. WD_BLACK SN8100, 2TB, 2300k IOPS read

Don't compare customer SSDs, which quote burst IOPS, to data-center SSDs which quote sustained IOPS in their spec sheets.

E.g. Micron 7450 PRO 3.84 TB - IOPS 4K 735k lesend, 160k schreibend

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#60

Post by up2isomorphism » Fri, Oct 31, 2025, 1:46 PM UTC

If you are targeting customers on AWS, don’t challenge EBS, because it is a losing game to begin with. There are 100 ways for AWS to optimize but none of them are available to you.