41–50 of 63 posts
There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…
Excellent tl;dr! Would pay to get them for every worthwhile tech article.
There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…
> Detaching and reattaching EBS volumes can take 10s for healthy volumes to 20m for failed ones Is there a source for the 20m time limit for failed EBS volumes? I experienced this at work for the first time recently but couldn't find anything documenting the 20m SLA (and it did take just about 20 full minutes).
The docs do say, however, "If the volume has been impaired for more than 20 minutes, you can contact the AWS Support Center." [0] which suggests its some expected cleanup/remount interval.
That is, it is something that we regularly encounter when EC2 instances fail, so we were sharing from personal experience.
[0] https://docs.aws.amazon.com/ebs/latest/userguide/work_volume...
Earlier quoted context omitted.
Thanks for the kind words about TimescaleDB :-) We think we're still building great things, and our customers seem to agree. Usage is at an all-time high, revenue is at an all-time high, and we’re having more fun than ever. Hopefully we’ll win you back soon.
Does Tiger Cloud support multi-region clusters? We are using aurora postgresql currently but it is straining (our budget and itself).
We're continuing to evaluate demand for multi-region clusters, love to hear from you.
EC2 instances have dedicated throughput to EBS via Nitro that you lose out on when you run your own EBS equivalent over the regular network. You only get 5Gbps maximum between two EC2 instances in the same AZ that aren't in the same placement group[1], and you're limited by the instance type's general networking throughput. Dedicated throughput to EBS from a typical EC2 instance is multiple times this figure. It's an…
There's a ton of jargon here. Summarized... Why EBS didn't work: - EBS costs for allocation - EBS is slow at restores from snapshot (faster to spin up a database from a Postgres backup stored in S3 than from an EBS snapshot in S3) - EBS only lets you attach 24 volumes per instance - EBS only lets you resize once every 6–24 hours, you can't shrink or adjust continuously - Detaching and reattaching EBS volumes can take…
The 5ms write latency and 1ms write latency sounds like they are using S3 to store and retrieve data with some local cache. My guess is a S3 based block storage exposed as a network block device. S3 supports compare-and-swap operations (Put-If-Match), so you can do a copy-on-write scenario quite easily. May be somebody from TigerData can give a little bit more insight into this. I know slatedb supports S3 as a backen…
The 5ms write latency is because the backend distributed block storage layer is doing synchronous replication to multiple servers for high availability and durability before ack'ing a write. (And this path has not yet been super-performance-optimized for latency, to be honest.)
@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it? Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.
There's some secret sauce there I don't know if I'm allowed to talk about yet, so I'll just address the existing tech that we didn't use: most things either didn't have a good enough license, cost too much, would take a TON of ramp-up and expertise we don't currently have to manage and maintain, but generally speaking, our stuff allows us to fully control it. Entirely programmable storage so far has allowed us to try…
You can probably hire people to maintain it.
Was it the ramp-up cost or expertise?
EC2 instances have dedicated throughput to EBS via Nitro that you lose out on when you run your own EBS equivalent over the regular network. You only get 5Gbps maximum between two EC2 instances in the same AZ that aren't in the same placement group[1], and you're limited by the instance type's general networking throughput. Dedicated throughput to EBS from a typical EC2 instance is multiple times this figure. It's an…
“The storage device driver exposes Fluid Storage volumes as standard Linux block devices mountable with filesystems such as ext4 or xfs. It...allows volumes to be resized dynamically while online.”
Yet an xfs file system cannot be shrunk at all, and an ext4 filesystem cannot be shrunk without first unmounting it.
Are you simply doing thin provisioning of these volumes, so they appear to be massive but aren’t really? I see later that you say you account for storage based on actual consumption.
@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it? Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.
DAOS seemed promising a couple of years ago. But in terms of popularity it seems to be stuck. No Ubuntu packages, no wide spread deployment, Optane got killed.
Yet the NVMe + metadata approach seemed promising.
Would love to see more databases fork it to do what you need from it.
Or if folks have looked at it and decided not to do it, an analysis of why would be super interesting.
@graveland Which Linux interface was used for the userspace block driver (ublk, nbd, tcmu-runner, NVMe-over-TCP, etc)? Why did you choose it? Also, were existing network or distributed file systems not suitable? This use case sounds like Ceph might fit, for example.
One of the problems with Ceph is that it doesn't operate at the highest possible throughput or the lowest possible latency point. DAOS seemed promising a couple of years ago. But in terms of popularity it seems to be stuck. No Ubuntu packages, no wide spread deployment, Optane got killed. Yet the NVMe + metadata approach seemed promising. Would love to see more databases fork it to do what you need from it. Or if fol…