Replacing EBS and Rethinking Postgres Storage from First Principles • Hacker News Board

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#61

Post by the8472 » Fri, Oct 31, 2025, 3:18 PM UTC

antonkochubey wrote: post #59

the8472 wrote: post #5

Though AWS instance-attached NVMe(oF?) still has less IOPS per TB than bare metal NVMe does. E.g. i8g.2xlarge, 1875 GB, 300k IOPS read vs. WD_BLACK SN8100, 2TB, 2300k IOPS read

Don't compare customer SSDs, which quote burst IOPS, to data-center SSDs which quote sustained IOPS in their spec sheets. E.g. Micron 7450 PRO 3.84 TB - IOPS 4K 735k lesend, 160k schreibend

The 7450 spec sheet[0] lists 1M iops read for the U.2 connector, 735k is for the M.2. And that's a PCIe Gen4 drive. If you look at the Micron 9550 Pro[1] (Gen5) the specsheet lists 3M IOPS for the 3.2TB model.

[0] https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b... [1] https://www.micron.com/products/storage/ssd/data-center-ssd/...

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#62

Post by antonkochubey » Fri, Oct 31, 2025, 4:07 PM UTC

the8472 wrote: post #61

antonkochubey wrote: post #59

Earlier quoted context omitted.

Don't compare customer SSDs, which quote burst IOPS, to data-center SSDs which quote sustained IOPS in their spec sheets. E.g. Micron 7450 PRO 3.84 TB - IOPS 4K 735k lesend, 160k schreibend

The 7450 spec sheet[0] lists 1M iops read for the U.2 connector, 735k is for the M.2. And that's a PCIe Gen4 drive. If you look at the Micron 9550 Pro[1] (Gen5) the specsheet lists 3M IOPS for the 3.2TB model. [0] https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b... [1] https://www.micron.com/products/storage/ssd/data-center-ssd/...

I have a strong suspicion that difference is due to heat dissipation of different form-factors, not due to connector. But yeah, it looks like whatever AWS is using is not on par with modern drives indeed, a factor of 10 to the drive you linked is a huge difference.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#63

Post by yencabulator » Thu, Nov 06, 2025, 5:14 PM UTC

graveland wrote: post #17

stefanha wrote: post #12

@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it? Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.

There's some secret sauce there I don't know if I'm allowed to talk about yet, so I'll just address the existing tech that we didn't use: most things either didn't have a good enough license, cost too much, would take a TON of ramp-up and expertise we don't currently have to manage and maintain, but generally speaking, our stuff allows us to fully control it. Entirely programmable storage so far has allowed us to try…

> would take a TON of ramp-up and expertise we don't currently have to manage and maintain

But you think you have resources to maintain a distributed strongly-consistent replicating block store?

The edge cases in RDB are literally why Ceph takes expertise to manage! Things like failure while recovering from failure while trying to maintain performance are inherently tricky.