Thanks for the writeup.
I'm curious whether you evaluated solutions like zfs/Gluster? Also curious whether you looked at Oracle Cloud given their faster block storage?
Replacing EBS and Rethinking Postgres Storage from First Principles
21–30 of 63 posts
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#22TimescaleDB was such a great project! I'm really sad to see them waste the opportunity and instead build an nth managed cloud on top of AWS, chasing buzzword after buzzword. Had they made deals with cloud providers to offer managed TimescaleDB so they can focus on their core value proposition they could have won the timeseries business, but ClickHouse made them irrelevant and Neon already has won the "Postgres for ag…
Thanks for the kind words about TimescaleDB :-)
We think we're still building great things, and our customers seem to agree.
Usage is at an all-time high, revenue is at an all-time high, and we’re having more fun than ever.
Hopefully we’ll win you back soon.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#23Postgres for agents, of course! It makes too much sense.
Thanks! We agree :-)
We just launched a bunch around “Postgres for Agents” [0]:
forkable databases, an MCP server for Postgres (with semantic + full-text search over the PG docs), a new BM25 text search extension (pg_textsearch), pgvectorscale updates, and a free tier.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#24There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…
Thanks for the summary. Note that those numbers are terrible vs. a physical disk, especially latency, which should be (That assumes async replication of the write ahead log to a secondary. Otherwise, write latency should be ~ 1 rtt, which is still Stacking storage like this isn’t great, but PG wasn’t really designed for performance or HA. (I don’t have a better concrete solution for ansi SQL that works today.)
A few datapoints that might help frame this:
- EBS typically operates in the millisecond range. AWS' own documentation suggests "several milliseconds"; our own experience with EBS is 1-2 ms. Reads/writes to local disk alone are certainly faster, but it's more meaningful to compare this against other forms of network-attached storage.
- If durability matters, async replication isn't really the right baseline for local disk setups. Most production deployments of Postgres/databases rely on synchronous replication -- or "semi-sync," which still waits for at least one or a subset of acknowledgments before committing -- which in the cloud lands you in the single-digit millisecond range for writes again.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#25Postgres for agents, of course! It makes too much sense.
The agent stuff is BS for the pointy hairs. This seems to address real problems I've had with PG though.
Yeah, I know what you mean. I used to roll my eyes every time someone said “agentic,” too. But after using Claude Code myself, and seeing how our best engineers build with it, I changed my mind. Agents aren’t hype, they’re genuinely useful, make us more productive, and honestly, fun to work with. I’ve learned to approach this with curiosity rather than skepticism.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#26Yes, EBS sucks, but plenty of cloud providers already implemented the same thing Tiger Data has a decade ago. Like Google.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#27Are they not using aws anymore? I found that confusing. It says they're not using ebs, not using attached nvme, but I didn't think there were other options in aws?
There weren’t, so they built one. (It is NVMe at the bottom, though.)
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#28Are they not using aws anymore? I found that confusing. It says they're not using ebs, not using attached nvme, but I didn't think there were other options in aws?
Tiger Cloud certainly continues to run on AWS. We have built it to rely on fairly low-level AWS primitives like EC2, EBS, and S3 (as opposed to some of the higher-level service offerings).
Our existing Postgres fleet, which uses EBS for storage, still serves thousands of customers today; nothing has changed there.
What’s new is Fluid Storage, our disaggregated storage layer that currently powers the new free tier (while in beta). In this architecture, the compute nodes running Postgres still access block storage over the network. But instead of that being AWS EBS, it’s our own distributed storage system.
From a hardware standpoint, the servers that make up the Fluid Storage layer are standard EC2 instances with fast local disks.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#29There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…
The 5ms write latency and 1ms write latency sounds like they are using S3 to store and retrieve data with some local cache. My guess is a S3 based block storage exposed as a network block device. S3 supports compare-and-swap operations (Put-If-Match), so you can do a copy-on-write scenario quite easily. May be somebody from TigerData can give a little bit more insight into this. I know slatedb supports S3 as a backend for their key-value store. We can build a block device abstraction using that.
Re: Replacing EBS and Rethinking Postgres Storage from First Principles
#30EC2 instances have dedicated throughput to EBS via Nitro that you lose out on when you run your own EBS equivalent over the regular network. You only get 5Gbps maximum between two EC2 instances in the same AZ that aren't in the same placement group[1], and you're limited by the instance type's general networking throughput. Dedicated throughput to EBS from a typical EC2 instance is multiple times this figure. It's an interesting tradeoff--I assume they must be IOPS-heavy and the throughput is not a concern.
[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...