Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much is one terabyte of data?

github.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: How much is one terabyte of data?

#21

Post by mrits »

danking00 wrote: post #12
SushiHippie wrote: post #2

> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s

It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3]. On the other points of the artic…

I'm doing some consulting with a client with a few terabytes in SQL Server. He keeps talking about challenges in reprocessing data in a migration to clickhouse.

I find it interesting that the solution to a lot of the problems is to just reprocess the data and don't try to optimize anything. 10 TB is not a lot of data with NVMe.

Re: How much is one terabyte of data?

#22

Post by magicalhippo »

danking00 wrote: post #12
SushiHippie wrote: post #2

> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s

It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3]. On the other points of the artic…

With AMD Epycs having 128 PCIe 4.0 lanes, using say 96 for disks thats 192GB/s of aggregate bandwidth. With 16TB U.2 SSDs that's up to 1.5PB of storage if you use one lane per disk.

Not for your average homelab budget but...

Re: How much is one terabyte of data?

#24

Post by danking00 »

danking00 wrote: post #12
SushiHippie wrote: post #2

> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s

It'd be interesting to see a peak-sequential-bandwidth by cost-per-gigabyte plot. The number I keep in my head is 500 MiB/s, but you're right that there are much faster drives out there [1]. Of the public clouds: Google's "Local SSD" claims ~12,000 MiB/s but they're ephemeral and you need 12 TiB of disks to hit that bandwidth [2][4]. AWS has these io2 SSDs which claim 4,000 MiB/s [3]. On the other points of the artic…

I gathered a table for Google and Amazon's options. I do not have experience with on-prem solutions so I don't know how to compare these prices to the cost of owning and operating hardware. I'm sure its cheaper over time for the hardware but I imagine you need sufficient scale to amortize the personnel costs.

Storage

    | product              | price (USD/GiB-month) | price (USD/IOPS-month)            | claimed max read bandwidth (MiB/s) | minimum price to achieve bandwidth |
    |----------------------|-----------------------|-----------------------------------|------------------------------------|------------------------------------|
    | GCP NVMe Local SSD   | 0.1046 [1]            | 0.00                              | 12,480 [2]                         | 13,000 USD/month [3]               |
    | AWS io2 SSD          | 0.1250 [4]            | 0.065 [4,5]                       | 4,000 [6]                          | 1,042 USD/month [4,6,14]           |
    | AWS io1 SSD          | 0.1250 [4]            | 0.065 [4,5]                       | 500 [7]                            | 135 USD/month [4,7,15]             |
    | Google Cloud Storage | 0.0200 [8]            | 0.0004 per "1k Class B Op" [8,16] | 23,842 [9]                         | [10]                               |
    | AWS S3               | 0.0230 [11]           | 0.0004 per 1k GET [8,16]          | 11,921? [12]                       | [13]                               |
You could build a similar table for compute but it gets complicated. FLOP seems like a reasonable unit of compute, but there are things other than FLOPs (e.g. decoding your column-oriented compression scheme).

I've tried to do this comparison a few times but I usually find it hard to get clear aggregate FLOP numbers for GPUs. GPUs also require caretaker CPUs and I don't have experience using them so I'm not certain how to spec a VM that can practically saturate the compute of the GPUs. My gut instinct is that the big compute consumers must be able to arbitrage this to some extent by shifting some workloads to chase the cheapest FLOP.

EDIT(2x): Table formatting. We could really use some Markdown styling on HN.

EDIT3: Clarify incomparability of IOPS and GETs.

[1] https://cloud.google.com/compute/disks-image-pricing#localss...

[2] https://cloud.google.com/compute/docs/disks/local-ssd#nvme_l...

[3] For 12TiB. https://cloud.google.com/compute/docs/disks/local-ssd#nvme_l...

[4] https://aws.amazon.com/ebs/pricing/

[5] We only need need 16,000 IOPS for peak performance of io2, so I ignore the drop in price at higher volumes.

[6] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisio...

[7] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisio...

[8] https://cloud.google.com/storage/pricing#price-tables

[9] https://cloud.google.com/storage/quotas#bandwidth

[10] Honestly not sure. You can do 5,000 parallel "reads" per second to a single object. I'm not sure what kind of instance you need to receive 23,842 MiB/s or if a single object can actually deliver that much bandwidth. https://cloud.google.com/storage/quotas#objects

[11] It gets slightly cheaper as volume goes up. https://aws.amazon.com/s3/pricing/

[12] I could not find a clear answer, but it seems at least 100 Gbps (11,921 MiB/s) https://repost.aws/knowledge-center/s3-maximum-transfer-spee...

[13] As with Google [10], I'm not really sure.

[14] For 16GiB and 16k IOPS.

[15] For 40GiB and 2000 IOPS.

[16] Not really comparable to provisioned IOPS because you pay for the IOPS once per month whereas you pay for every individual GET request.

Re: How much is one terabyte of data?

#29

Post by wtallis »

thrwwycbr wrote: post #9
SushiHippie wrote: post #2

> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s

"Up to xyz MB/s" This very detail is the reason USB flash drives have useless performance numbers on them, which oftentimes lead to 1/100th of the advertised performance. Everything that's bigger than the sector size (4k) is usually far lower than those numbers.

Getting peak sequential read speed out of SSDs and USB flash drives usually requires issuing read commands that are much larger than 4kB, for two main reasons: the page size of the underlying flash memory is larger than 4kB, and SSDs and some USB flash drives stripe data access across multiple channels so you need to ask for at least a full strip of data. Queuing many commands for contiguous sequential reads can usually get close to the same performance, but has more overhead (and isn't an option for USB drives that don't support UASP).

Re: How much is one terabyte of data?

#30

Post by gilleain »

readingnews wrote: post #17
bumbledraven wrote: post #10

Earlier quoted context omitted.

There are ink pens with 0.1 mm tips. https://www.jetpens.com/Copic-Multiliner-SP-Pen-0.1-mm-Black...

There be wallet burning dragons here...

There absolutely are when it comes to materials like pens, but I bought a set of 3 pens (0.1 mm, 0.3 mm, 0.5 mm) for less than ten GBP. Specifically, Mitsubishi pencil co. unipen fine line.

Sure you can probably buy much more expensive examples, but it's not essential.

(Side note : don't try to use a gel ink pen across varnish, or it stops working until you clean it off with solvent)

Post reply on HN