Replacing EBS and Rethinking Postgres Storage from First Principles

#1

Post by mfreed » Wed, Oct 29, 2025, 3:49 PM UTC

Replacing EBS and Rethinking Postgres Storage from First Principles
tigerdata.com

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#2

Post by cpt100 » Wed, Oct 29, 2025, 9:46 PM UTC

pretty cool

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#3

Post by thr0w » Thu, Oct 30, 2025, 1:09 PM UTC

Postgres for agents, of course! It makes too much sense.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#4

Post by jacobsenscott » Thu, Oct 30, 2025, 1:31 PM UTC

thr0w wrote: post #3

Postgres for agents, of course! It makes too much sense.

The agent stuff is BS for the pointy hairs. This seems to address real problems I've had with PG though.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#5

Post by the8472 » Thu, Oct 30, 2025, 2:08 PM UTC

Though AWS instance-attached NVMe(oF?) still has less IOPS per TB than bare metal NVMe does.

    E.g. i8g.2xlarge, 1875 GB, 300k IOPS read
    vs. WD_BLACK SN8100, 2TB, 2300k IOPS read

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#6

Post by 0xbadcafebee » Thu, Oct 30, 2025, 2:17 PM UTC

There's a ton of jargon here. Summarized...

Why EBS didn't work:

  - EBS costs for allocation
  - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3)
  - EBS only lets you attach 24 volumes per instance
  - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously
  - Detaching and reattaching EBS volumes can take 10s for healthy volumes to 20m for failed ones, so failover takes longer

Why all this matters:

  - their AI agents are all ephemeral snapshots; they constantly destroy and rebuild EBS volumes

What didn't work:

  - local NVMe/bare metal: need 2-3x nodes for durability, too expensive; snapshot restores are too slow
  - custom page-server psql storage architecture: too complex/expensive to maintain

Their solution:

  - block COWs
  - volume changes (new/snapshot/delete) are a metadata change
  - storage space is logical (effectively infinite) not bound to disk primitives
  - multi-tenant by default
  - versioned, replicated k/v transactions, horizontally scalable
  - independent service layer abstracts blocks into volumes, is the security/tenant boundary, enforces limits
  - user-space block device, pins i/o queues to cpus, supports zero-copy, resizing; depends on Linux primitives for performance limits

Performance stats (single volume):

  - (latency/IOPS benchmarks: 4 KB blocks; throughput benchmarks: 512 KB blocks)
  - read: 110,000 IOPS and 1.375 GB/s (bottlenecked by network bandwidth
  - write: 40,000–67,000 IOPS and 500–700 MB/s, synchronousy replicated
  - single-block read latency ~1 ms, write latency ~5 ms

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#7

Post by maherbeg » Thu, Oct 30, 2025, 2:35 PM UTC

This has a similar flavor to xata.io's SimplyBlock based storage system * https://xata.io/blog/xata-postgres-with-data-branching-and-p... * https://www.simplyblock.io/

It's a great way to mix copy on write and effectively logical splitting of physical nodes. It's something I've wanted to build at a previous role.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#8

Post by hedora » Thu, Oct 30, 2025, 2:45 PM UTC

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

Thanks for the summary.

Note that those numbers are terrible vs. a physical disk, especially latency, which should be (That assumes async replication of the write ahead log to a secondary. Otherwise, write latency should be ~ 1 rtt, which is still Stacking storage like this isn’t great, but PG wasn’t really designed for performance or HA. (I don’t have a better concrete solution for ansi SQL that works today.)

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#9

Post by znpy » Thu, Oct 30, 2025, 2:48 PM UTC

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

Reminds me of about ten years ago when a large media customer was running NetApp on cloud to get most of what you just wrote on AWS (because EBS features sucked/sucks very bad and are also crazy expensive).

I did not set that up myself, but the colleague that worked on that told me that enabling tcp multipath for iscsi yielded significant performance gains.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#10

Post by bradyd » Thu, Oct 30, 2025, 2:52 PM UTC

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

> EBS only lets you resize once every 6–24 hours

Is that even true? I've resized an EBS instance a few minutes after another resize before.