10 petabytes - visualized • Hacker News Board

Re: 10 petabytes - visualized

#31

Post by rarrrrrr » Sat, Jan 15, 2011, 11:08 PM UTC

cperciva wrote: post #27

Anyone want to guess how much data they actually have stored? Backblaze's pods are a data-loss nightmare -- lots of single points of failure which will wipe out many TB of data at a time -- and backblaze has stated that they replicate data across multiple pods. Given that the 10 PB seems to be the amount of raw storage backblaze has, I'm guessing that the amount of actual data stored is much less -- depending on what…

At SpiderOak we get a 3x replication equivalent for about 35% overhead, using Reed-Solomon at the cluster level (on top of RAID6 at the machine level.) Not nearly as expensive as outright replication.

Agree those SATA port multiplies are worrisome. In the beginning, our prototype machines used them to squeeze as many drives into a single machine as possible. They have unusually low tolerance for electrical interference and make it possible for one badly malfunctioning drive to take an entire array offline until manually serviced. We've seen occasions where just touching a cable attached to a port multiplier caused the Linux kernel to emit "dazed and confused" NMI events. I am not brave enough to try them again, even in a redundant setup.

Re: 10 petabytes - visualized

#32

Post by joanou » Sat, Jan 15, 2011, 11:40 PM UTC

That's a sizable operation and difficult to visualize. I'm guessing their failure points reduce potential damage to minimal levels. At AltDrive, we use ZFS RaidZ2 with a number of hot spares. Six drives would have to fail consecutively on a given machine before any data loss... we replace them as they occur. And we have inter-H/W duplication. ZFS is self healing and makes management easy. Additionally we periodically compare the integrity of the user's AES-256 CTR encrypted data against the database records and make corrections if necessary. From our customer's and white label provider's view, it just works. http://altdrive.com

Re: 10 petabytes - visualized

#33

Post by ghkluifge » Sat, Jan 15, 2011, 11:52 PM UTC

●------ http://u.ly/73I ------●

Gold autumn, personality Mes clothing + Shoes, Travel bag that grabs an eye coat + Chao packet Free transport

●------ http://u.ly/73I ------●

Re: 10 petabytes - visualized

#34

Post by sliverstorm » Sat, Jan 15, 2011, 11:55 PM UTC

rphlx wrote: post #19

quinndupont wrote: post #7

My 350GB of music is backed up on Backblaze (a wonderful service), but seeing those visualizations reminded me of the environmental impact of my digital packratting. That's a sizable data center keeping my data happy and backed up.

I wonder how many thousands of people are paying them to back up stuff they could easily redownload. Great business model if they de-dupe internally.

If you're implying they are crazy for doing so, have you ever lost everything and tried to redownload it all? It's a huge timesink, I tell you.

As for de-duping, Dropbox does this. When you drop a Windows ISO or the Office installer into your folder, it hashes the file and realizes 'hey we already have this!' and you don't have to upload it.

Re: 10 petabytes - visualized

#35

Post by cperciva » Sun, Jan 16, 2011, 12:21 AM UTC

rarrrrrr wrote: post #31

cperciva wrote: post #27

Anyone want to guess how much data they actually have stored? Backblaze's pods are a data-loss nightmare -- lots of single points of failure which will wipe out many TB of data at a time -- and backblaze has stated that they replicate data across multiple pods. Given that the 10 PB seems to be the amount of raw storage backblaze has, I'm guessing that the amount of actual data stored is much less -- depending on what…

At SpiderOak we get a 3x replication equivalent for about 35% overhead, using Reed-Solomon at the cluster level (on top of RAID6 at the machine level.) Not nearly as expensive as outright replication. Agree those SATA port multiplies are worrisome. In the beginning, our prototype machines used them to squeeze as many drives into a single machine as possible. They have unusually low tolerance for electrical interferen…

3x replication equivalent for about 35% overhead

How did you compute this "replication equivalent"?

Re: 10 petabytes - visualized

#36

Post by drhodes » Sun, Jan 16, 2011, 12:29 AM UTC

Or about 9 (American) football fields covered with 4GB DVDs: https://sites.google.com/site/10petabytes/viz

Re: 10 petabytes - visualized

#37

Post by kiujhygthyujik » Sun, Jan 16, 2011, 1:12 AM UTC

jacquesm wrote: post #6

I'll give you a much more compact way to envision 10 Pb: 10 Petabytes is 10,000,000,000,000,000 bytes or 80,000,000,000,000,000 bits, divided by 8,000,000,000 bits per full human genome (2 bits per base-pair) that's about 10,000,000 cells (not red blood cells because they don't contain DNA), or about 5 milliliters! (10 um diameter on average so about 500 cubic um, so 2 million or so per ml), and that includes all the…

We can already write individual atoms on a surface with a atomic force microscope. So at one bit per atom position we could encode 10Pb in about 0.1micromoles of xenon atoms on a silicon substrate.

Well under what nature can do.

Re: 10 petabytes - visualized

#38

Post by jacquesm » Sun, Jan 16, 2011, 1:16 AM UTC

kiujhygthyujik wrote: post #37

jacquesm wrote: post #6

I'll give you a much more compact way to envision 10 Pb: 10 Petabytes is 10,000,000,000,000,000 bytes or 80,000,000,000,000,000 bits, divided by 8,000,000,000 bits per full human genome (2 bits per base-pair) that's about 10,000,000 cells (not red blood cells because they don't contain DNA), or about 5 milliliters! (10 um diameter on average so about 500 cubic um, so 2 million or so per ml), and that includes all the…

We can already write individual atoms on a surface with a atomic force microscope. So at one bit per atom position we could encode 10Pb in about 0.1micromoles of xenon atoms on a silicon substrate. Well under what nature can do.

the 'on a silicon substrate' is a bit problematic here, a substrate typically is a lot larger than just a few atoms, the neat thing about DNA is that it is a three dimensional encoding, you are talking about a two dimensional encoding scheme, not three dimensional (an atomic force microscope typically moves in two dimensions and uses an electrical charge to deposit the atoms).

You could stack the atoms three dimensionally but then there would be no easy way to read them out again.

Re: 10 petabytes - visualized

#39

Post by ramchip » Sun, Jan 16, 2011, 1:40 AM UTC

epistasis wrote: post #24

ramchip wrote: post #12

Earlier quoted context omitted.

I see many debatable points: - Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit. - If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk. - There is no way to retrieve information in this system. This is a bit like packing a large amount of extremely high-density magnetic platters in a…

- There is no way to retrieve information in this system. DNA is natively a content addressable storage system, due to natural base-pairing. But to first address your question in your edit: think of DNA as a pair of singly-linked lists, each with an alphabet of four characters. Each singly-linked list is the "reverse complement" of the other: the reverse sequence with an A-T swap and a C-G swap. At each position ther…

> To probe for information in a DNA database, you construct the reverse-complement of the desired bit of information

To build the reverse-complement of the information, don't you need to have the information in the first place?

I know it's possible to store information in DNA through various means, but I don't believe it can be done at the density the OP calculated. If we're going to take into account only information storage while ignoring retrieval considerations, then we shouldn't compare naked cells with no DNA duplication to reliable whole hard drives.

Call it human pride, but I think we've beaten mother nature in several aspects ;)

Re: 10 petabytes - visualized

#40

Post by mdemare » Sun, Jan 16, 2011, 1:58 AM UTC

bane wrote: post #30

Amazing, a discussion of data without comparing the internationally accepted unit of "library of congress". I'm more interested in how many LoCs 10 Pb is? If we built a city of LoCs, how many acres would that city be? These are the questions that keep me up late at night.

The Library of Congress is 10TB. Zipped, it would fit on a single 3TB hard drive.