How much is one terabyte of data? • Hacker News Board

Re: How much is one terabyte of data?

#41

Post by causi » Tue, Feb 20, 2024, 6:19 PM UTC

supertrope wrote: post #40

causi wrote: post #39

Earlier quoted context omitted.

Really wish games would give you a "cleanup" option in the settings. If I'm only going to use the Ultra textures it could delete all the others, and same for if someone is only going to use Medium, etc.

As much as I want to be able to un-select game modes, ultra textures, and foreign language audio, this would increase complexity and QA required.

Letting me or third-party software go in and delete the files manually without pitching a fit would be an acceptable compromise. I'd even purchase such a utility.

Re: How much is one terabyte of data?

#42

Post by supertrope » Tue, Feb 20, 2024, 6:21 PM UTC

causi wrote: post #41

supertrope wrote: post #40

Earlier quoted context omitted.

As much as I want to be able to un-select game modes, ultra textures, and foreign language audio, this would increase complexity and QA required.

Letting me or third-party software go in and delete the files manually without pitching a fit would be an acceptable compromise. I'd even purchase such a utility.

Some studios have tinkered with the idea of games that can be played while the download completes. There has to be a way to trace all assets that are needed for the first chapter. https://patents.google.com/patent/US11123634B1/

Re: How much is one terabyte of data?

#43

Post by textread » Tue, Feb 20, 2024, 6:29 PM UTC

kenm47 wrote: post #28

1 tb of searchable data is $25/mo if you do it right.

I am interested in the back of the envelope calculations you did to come to this conclusion. Would you please elaborate if possible?

I know of an early stage YC startup that has a 6TB Postgres DB. Would it be fair to say that the DB hosting (neglecting replica, engineering time) can be done at $150/month?

Re: How much is one terabyte of data?

#44

Post by araes » Tue, Feb 20, 2024, 6:36 PM UTC

breck-fresen wrote: post #14

SushiHippie wrote: post #4

Earlier quoted context omitted.

I don't think there are HDDs that can read 300MB/s

Spinning HDDs have ~3MB on the outer track and 1MB on the inner track. 7200rpm/60 * 3MB = 360MB/second for sequential outer track reads. Source: I wrote S3's on-disk placement algorithm

Do you happen to be the writer of this technical article on AWS? [1]

If not, someone on Seth Markle's team over at Amazon wrote a pretty fascinating article on the S3 architecture. Never actually heard the "airplane at 75 MPH over a grass lawn comparison" before.

This [2] aggregate workload graph/video is pretty sweet too. Very matrixy while still actually being fairly clear about what's going on.

Imagery implies S3 (at writing) had a 3.7 PB bucket size with 2.3M req/sec. Worked out to 143 hard drives (storage constrained result) and 19,000 hard drives for the I/O constrained result. Kind of an interesting counterpoint to the article. Here's what really large workload / access rate systems with spikey, intermittent workloads are structured like.

[1] https://www.allthingsdistributed.com/2023/07/building-and-op...

[2] https://www.allthingsdistributed.com/videos/fast-agg-graph-c...

Re: How much is one terabyte of data?

#45

Post by PlatinumBench » Tue, Feb 20, 2024, 8:41 PM UTC

supertrope wrote: post #42

causi wrote: post #41

Earlier quoted context omitted.

Letting me or third-party software go in and delete the files manually without pitching a fit would be an acceptable compromise. I'd even purchase such a utility.

Some studios have tinkered with the idea of games that can be played while the download completes. There has to be a way to trace all assets that are needed for the first chapter. https://patents.google.com/patent/US11123634B1/

Blizzard has had this for a number of years, at least for World Of Warcraft. You can start playing and it downloads the rest of the game in the background. I think the loading screens are a bit longer since my guess would be that the program checks to make sure the needed files are there for the zone and, if not, it downloads them as a priority.

Re: How much is one terabyte of data?

#46

Post by yreg » Tue, Feb 20, 2024, 8:47 PM UTC

supertrope wrote: post #42

causi wrote: post #41

Earlier quoted context omitted.

Letting me or third-party software go in and delete the files manually without pitching a fit would be an acceptable compromise. I'd even purchase such a utility.

Some studios have tinkered with the idea of games that can be played while the download completes. There has to be a way to trace all assets that are needed for the first chapter. https://patents.google.com/patent/US11123634B1/

World of Warcraft has worked that way for years. When you start, you can initially see other players fight phantoms because your game hasn't downloaded the assets for those mobs yet.

Re: How much is one terabyte of data?

#47

Post by pixl97 » Tue, Feb 20, 2024, 9:04 PM UTC

causi wrote: post #31

SushiHippie wrote: post #2

> The high-speed SSD reads 300 megabytes of data per second Aren't normal SSDs now at 500-600MB/s read? And then there are nvme SSDs which can read up to 7GB/s

I've never seen an SSD as slow as 300MB/s. My first SSD from a decade ago was over 500MB/s sequential read.

Are you measuring the NAND or its faster cache that can be a few GB in size?

All the different levels of caches make the raw performance numbers (also how full is the disk) hard to find.

Re: How much is one terabyte of data?

#48

Post by HPsquared » Tue, Feb 20, 2024, 10:07 PM UTC

hnuser123456 wrote: post #13

HPsquared wrote: post #7

1 million is 100^3 (i.e. a cube 1 metre across, contains a million 1 cm cubes) 1 billion is 1000^3 (1 m cube contains a billion 1 mm cubes) 1 trillion is 10,000^3 (I can't visualize 0.1 mm, so increase size: the big cube is now 10 m on side, with 1 mm parts) So the million and billion you could have on your desk made of parts that you can see and handle; the trillion is just a bit too big for that, you'd need a big r…

0.1mm is about the thickness of a sheet of paper.

Thinking again about it, you could have sheets of paper stacked into a cube. 1 m^2 sheets, with 0.1mm checkerboard pattern, stacked a metre high. That's some fine detail...