Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing EBS and Rethinking Postgres Storage from First Principles

tigerdata.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#31

Post by huntaub »

electroly wrote: post #30

EC2 instances have dedicated throughput to EBS via Nitro that you lose out on when you run your own EBS equivalent over the regular network. You only get 5Gbps maximum between two EC2 instances in the same AZ that aren't in the same placement group[1], and you're limited by the instance type's general networking throughput. Dedicated throughput to EBS from a typical EC2 instance is multiple times this figure. It's an…

I believe this is also changing with instances that now allow you to adjust the ratio of throughput on the NIC that's dedicated to EBS vs. general network traffic (with the intention, I'm sure, that people would want more EBS throughput than the default).

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#32

Post by samlambert »

graveland wrote: post #11
hedora wrote: post #8

Earlier quoted context omitted.

Thanks for the summary. Note that those numbers are terrible vs. a physical disk, especially latency, which should be (That assumes async replication of the write ahead log to a secondary. Otherwise, write latency should be ~ 1 rtt, which is still Stacking storage like this isn’t great, but PG wasn’t really designed for performance or HA. (I don’t have a better concrete solution for ansi SQL that works today.)

(I'm on the team that made this) The raw numbers are one thing, but the overall performance of pg is another. If you check out https://planetscale.com/blog/benchmarking-postgres-17-vs-18 for example, in the average QPS chart, you can see that there isn't a very large difference in QPS between GP3 at 10k iops and NVMe at 300k iops. So currently I wouldn't recommend this new storage for the highest end workloads, but i…

it's a 70% difference at lower cost. i know math is hard but c'mon try and be serious.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#33

Post by _rs »

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

> Detaching and reattaching EBS volumes can take 10s for healthy volumes to 20m for failed ones

Is there a source for the 20m time limit for failed EBS volumes? I experienced this at work for the first time recently but couldn't find anything documenting the 20m SLA (and it did take just about 20 full minutes).

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#34

Post by jread »

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

I'm working on graduate research evaluating AWS control and data plane performance.

EBS volume attachment is typically ~11s for GP2/GP3 and ~20-25s for other types.

1ms read / 5ms write latencies seem high for 4k blocks. IO1/IO2 is typically ~0.5ms RW, and GP2/GP3 ~0.6ms read and ~0.94ms write.

References: https://cloudlooking.glass/matrix/#aws.ebs.us-east-1--cp--at... https://cloudlooking.glass/matrix/#aws.ebs.*--dp--rand-*&aws...

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#35

Post by [deleted] »

jread wrote: post #34
0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

I'm working on graduate research evaluating AWS control and data plane performance. EBS volume attachment is typically ~11s for GP2/GP3 and ~20-25s for other types. 1ms read / 5ms write latencies seem high for 4k blocks. IO1/IO2 is typically ~0.5ms RW, and GP2/GP3 ~0.6ms read and ~0.94ms write. References: https://cloudlooking.glass/matrix/#aws.ebs.us-east-1--cp--at... https://cloudlooking.glass/matrix/#aws.ebs.*--dp…

[deleted]

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#36

Post by DenisM »

IUUC they built a EBS replacement on top of NVME attached to a dynamically sized fleet of EC2 instances.

The advantage is that it’s allocating pages on demand from an elastic pool of storage so it appears as an infinite block device. Another advantage is cheap COW clones.

The downside is (probably) specialized tuning for Postgres access patterns. I shudder to think what went into page metadata management. Perhaps it’s similar to e.g. SQL Server buffer pool manager).

It’s not clear to me why it’s better than Aurora design - on the surface page servers are higher level concepts and should allow more holistic optimizations (and less page write traffic due to shipping log in lieu of whole pages). Is also not clear what stopped Amazon from doing the same (perhaps EBS serving more diverse access patterns?).

Very cool!

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#37

Post by samat »

0xbadcafebee wrote: post #6

There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…

Excellent tl;dr! Would pay to get them for every worthwhile tech article.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#38

Post by kristianp »

So they've built a competitor to EBS that runs on EC2 and nvme. Seems like their prices will need to be much higher than those of AWS to get decent profit margins. I really hate being in the high-cost ecosystem of the large cloud providers, so I wouldn't make use of this.

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#39

Post by NewJazz »

akulkarni wrote: post #22
unsolved73 wrote: post #15

TimescaleDB was such a great project! I'm really sad to see them waste the opportunity and instead build an nth managed cloud on top of AWS, chasing buzzword after buzzword. Had they made deals with cloud providers to offer managed TimescaleDB so they can focus on their core value proposition they could have won the timeseries business, but ClickHouse made them irrelevant and Neon already has won the "Postgres for ag…

Thanks for the kind words about TimescaleDB :-) We think we're still building great things, and our customers seem to agree. Usage is at an all-time high, revenue is at an all-time high, and we’re having more fun than ever. Hopefully we’ll win you back soon.

Does Tiger Cloud support multi-region clusters? We are using aurora postgresql currently but it is straining (our budget and itself).

Re: Replacing EBS and Rethinking Postgres Storage from First Principles

#40

Post by kjetijor »

stefanha wrote: post #12

@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it? Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.

I was struck by how similar this seems to Ceph/RADOS/RBD. I.e. how they implemented snapshotted block storage on top, sounds more or less exactly the same as how RBD is implemented on top of RADOS in ceph.
Post reply on HN