Interesting read, but it could REALLY use some visualization to drive the point home.
How much is one terabyte of data?
11–20 of 48 posts
Re: How much is one terabyte of data?
#12> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s
It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3].
On the other points of the article, even if you had a huge disk array plugged into the machine, how many cores can you also plug into that computer? I suppose there will always be a (healthy, productive) race here between the vertical scaling of GPUs + NVMe SSDs and the horizontal scaling of CPUs and blob storage.
EDIT: formatting.
[1] First Google result is Tom's hardware: https://www.tomshardware.com/features/ssd-benchmarks-hierarc...
[2] https://cloud.google.com/compute/docs/disks/local-ssd#nvme_l...
[3] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisio...
[4] The ephemerality has two downsides. First, you have to get the data onto that local SSD from some other, probably slower, storage system (I haven't benchmarked GCS lately, but that's probably your best bet for quickly downloading a bunch of data?). Second, you need to use non-spot instances which are 3-6x the price.
Re: How much is one terabyte of data?
#131 million is 100^3 (i.e. a cube 1 metre across, contains a million 1 cm cubes) 1 billion is 1000^3 (1 m cube contains a billion 1 mm cubes) 1 trillion is 10,000^3 (I can't visualize 0.1 mm, so increase size: the big cube is now 10 m on side, with 1 mm parts) So the million and billion you could have on your desk made of parts that you can see and handle; the trillion is just a bit too big for that, you'd need a big r…
0.1mm is about the thickness of a sheet of paper.
Re: How much is one terabyte of data?
#14Earlier quoted context omitted.
I'm pretty sure they mean HDDs as they write hard disks in each paragraph below.
I don't think there are HDDs that can read 300MB/s
Spinning HDDs have ~3MB on the outer track and 1MB on the inner track. 7200rpm/60 * 3MB = 360MB/second for sequential outer track reads.
Source: I wrote S3's on-disk placement algorithm
Re: How much is one terabyte of data?
#15Earlier quoted context omitted.
I'm pretty sure they mean HDDs as they write hard disks in each paragraph below.
I don't think there are HDDs that can read 300MB/s
Current high capacity enterprise HDDs (Like an Exos 24 or an WD Gold 24), in ideal conditions, reach around 250-280MiB/s, so quite sort of 300 as a regular number.
Probably they are pulling info from an older article or just using a round number to make it more clear.
Edit note: As a sibling comment says, Seagate had for a while a "2x" Exos model that had dual actuators and claimed ~500Mbytes/s of max transfer speed. But seem discontinued or unavailable and didn't have much success (out of the datacenter at least)
Re: How much is one terabyte of data?
#16Earlier quoted context omitted.
I'm pretty sure they mean HDDs as they write hard disks in each paragraph below.
I don't think there are HDDs that can read 300MB/s
My 20TB WDs get pretty close, around 270MB/s sequential read[1]. Higher density means higher sequential speed, so if not the 22TB models then the 24TB should reach that.
[1]: https://www.kitguru.net/components/hard-drives/simon-crisp/w...
Re: How much is one terabyte of data?
#171 million is 100^3 (i.e. a cube 1 metre across, contains a million 1 cm cubes) 1 billion is 1000^3 (1 m cube contains a billion 1 mm cubes) 1 trillion is 10,000^3 (I can't visualize 0.1 mm, so increase size: the big cube is now 10 m on side, with 1 mm parts) So the million and billion you could have on your desk made of parts that you can see and handle; the trillion is just a bit too big for that, you'd need a big r…
There are ink pens with 0.1 mm tips. https://www.jetpens.com/Copic-Multiliner-SP-Pen-0.1-mm-Black...
There be wallet burning dragons here...
Re: How much is one terabyte of data?
#181 million is 100^3 (i.e. a cube 1 metre across, contains a million 1 cm cubes) 1 billion is 1000^3 (1 m cube contains a billion 1 mm cubes) 1 trillion is 10,000^3 (I can't visualize 0.1 mm, so increase size: the big cube is now 10 m on side, with 1 mm parts) So the million and billion you could have on your desk made of parts that you can see and handle; the trillion is just a bit too big for that, you'd need a big r…
0.1mm is about the thickness of a sheet of paper.
A 0.1mm cube is like a grain of fine sand, can't really see the size of it. I couldn't imagine stacking them by hand with tweezers, but just about doable for 1mm.
Re: How much is one terabyte of data?
#19> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s
It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3]. On the other points of the artic…
And "characteristic time": disk size / speed. In other words, time required to read/write the full disk.
Re: How much is one terabyte of data?
#20If you're looking for a visualization, a YouTube team of VFX artists did some neat ones last year in this video: https://youtu.be/J-K2yeQylCk