Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cash App migrated 400TB of data to PlanetScale's cloud

planetscale.com

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Cash App migrated 400TB of data to PlanetScale's cloud

#11

Post by manacit »

znpy wrote: post #6
JTyQZSnP3cQGa8B wrote: post #3

Earlier quoted context omitted.

It would be interesting to know why a Bitcoin application requires 400 TB of disk space.

It's not much data. With current 22TB magnetic disks it's less than 20 disks, they would fit into a single machine (4u, likely). The Storinator XL60 from 45drives ( https://www.45drives.com/products/storinator-xl60-configurat... ) can hold 60 disks, for (advertised) ~1.4PB of data. (btw i've learned about 45drives through Linus Tech Tips channel so I think it's obligatory to say LTTSTORE.COM - for the meme)

Just to address the core of your comment, 20 magnetic disks would combine for about ~2,000 IOPS of capacity, provide for no redundancy, and allow only one machine to process the entirety of the queries coming in to power the application.

Even a full 60 disk server filled with magnetic disks would provide less I/O capacity for running a relational database than a single EBS volume.

It's might not look like a lot of data if you're talking about storing media files, but it's quite a bit of relational data to be queried in single-digit milliseconds at-scale.

Re: Cash App migrated 400TB of data to PlanetScale's cloud

#13

Post by znpy »

manacit wrote: post #11
znpy wrote: post #6

Earlier quoted context omitted.

It's not much data. With current 22TB magnetic disks it's less than 20 disks, they would fit into a single machine (4u, likely). The Storinator XL60 from 45drives ( https://www.45drives.com/products/storinator-xl60-configurat... ) can hold 60 disks, for (advertised) ~1.4PB of data. (btw i've learned about 45drives through Linus Tech Tips channel so I think it's obligatory to say LTTSTORE.COM - for the meme)

Just to address the core of your comment, 20 magnetic disks would combine for about ~2,000 IOPS of capacity, provide for no redundancy, and allow only one machine to process the entirety of the queries coming in to power the application. Even a full 60 disk server filled with magnetic disks would provide less I/O capacity for running a relational database than a single EBS volume. It's might not look like a lot of da…

I assumed people did not need to be explicitly reminded that you had to provision additional capacity for redundancy, and that you can use different layers of caching (ssd caches, ram caches etc).

And by the way, it was just posted today that you can get 60TB pci gen5 SSDs from Micron: https://news.ycombinator.com/item?id=42122434 : you can still fit all that dataset in a single machine and provide all the iops you need. You'd need just 7 of those.

So yeah, 400TB of data is not much data nowadays.

Re: Cash App migrated 400TB of data to PlanetScale's cloud

#14

Post by znpy »

telgareith wrote: post #8
znpy wrote: post #6

Earlier quoted context omitted.

It's not much data. With current 22TB magnetic disks it's less than 20 disks, they would fit into a single machine (4u, likely). The Storinator XL60 from 45drives ( https://www.45drives.com/products/storinator-xl60-configurat... ) can hold 60 disks, for (advertised) ~1.4PB of data. (btw i've learned about 45drives through Linus Tech Tips channel so I think it's obligatory to say LTTSTORE.COM - for the meme)

45drives is... bad. I cannot understand why people would use them- particularly for homelab. Three SC846 24x3.5" 4U can be had for no more than $400/ea and, theres 36bay versions. 60 drives in 6U is a crap ton of weight.

> Three SC846 24x3.5" 4U can be had for no more than $400/ea and, theres 36bay versions.

Because then you need three times the rack space. What you don't spend one-time for the hardware you'll spend extra every month for rack space, connectivity, cooling, power etc.

You might use that budget for redundancy, for example.

Re: Cash App migrated 400TB of data to PlanetScale's cloud

#15

Post by aaronyoung »

nickvanw wrote: post #10
prng2021 wrote: post #9

Earlier quoted context omitted.

The article says they used a forked version of Vitess with customizations. What were they and how did you address that when migrating?

Answering on behalf of PlanetScale (I'm the CTO!). I don't remember exactly what was different from upstream, but it wasn't a whole lot of changes. Fortunately, PlanetScale runs a well-maintained fork ourselves, so we're very used to taking custom changes and getting them deployed. In this case, we asked the Block team for all of their changes and went through them one by one to see what was needed to pull into our f…

Mostly the diffs were related to running against the on-prem MySQL instances smoothly: stuff like changes to split tooling or how you boot up the pieces. We have had unique vindexes or query planning changes in the past but we either deprecated or upstreamed them prior to migration.
Post reply on HN