Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

91–100 of 146 posts

Re: Data centers contain 90% crap data

#91

Post by Pooge »

danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

Are you going to be able to find the data you're looking for if judges demand it?

Data that ends up in that sea of crap is very often poorly labeled.

Data that you cannot find again is useless.

If you take more than 3 minutes to find a picture you wanted to show me, it doesn't deserve to be showed anymore.

Re: Data centers contain 90% crap data

#93

Post by mrb »

Some fun math: according to some estimates there is 175 zettabytes of data worldwide. Assuming 20-terabyte harddrives, this could be stored on 8.8 billion drives. Assuming 10 drives per rack unit, 42 RU per rack cabinet, 16 square feet per cabinet (including aisle space), that means you need about 330 million square feet of data center space to host this data. If it was hosted in a single square data center, it would be 3.5 miles long and wide. (I always like to picture the physical space something would occupy.) And energy wise, assuming 5 watts per drive, it would consume 44 gigawatt, so it could be powered by about two large hydro dams similar to the Three Gorges Dam (22 gigawatts capacity). I am assuming a PUE close to 1.0 for simplicity. Of course one would not be able to spin up all these drives at once, since a drive spinning up consumes about three times more power (15 watts). So you would definitely staggered spin up when booting the servers :-)

If 90% of this data is "crap" and could be cut down, it would still be just a drop in the bucket compared to worldwide energy use.

Re: Data centers contain 90% crap data

#94

Post by otterley »

jodrellblank wrote: post #19

It's a problem for us when it comes to the GDPR and rights to be forgotten; companies will say they store your data carefully, will say they have shown you everything they have, will say they delete it, but "the company" in aggregate has no idea there's a thousand SharePoint sites and ex-employees mailboxes and filestores, and copies of old filservers from before a migration, and test databases containing copies of r…

The right to be forgotten was never a practically enforceable right to begin with. It’s a right that burdens others with perpetual effort and cost, unlike most rights, which only require others not to do things. I’m not sure why Europeans are so obsessed with it.

Re: Data centers contain 90% crap data

#95

Post by InsideOutSanta »

ghaff wrote: post #90
InsideOutSanta wrote: post #83

Earlier quoted context omitted.

The article touches on this issue by pointing out that: >The Cloud is what happens when the cost of storing data is less than the cost of figuring out what to do with the crap But I think that's wrong. The actual issue is that you often can't figure out "what to do with the crap" because the difference between useful data and crap data is determined at the point in time when you need it, not when you store it. I'm re…

I do cull photos and docs but it takes effort and you will make mistakes. On the other hand, intelligent culling makes it easier to find things. It’s a trade off especially with digital where there’s not that much cost associated with holding onto things relative to the physical world. My whole house is basically in storage because of a kitchen fire and I’m planning to be pretty selective about what I move back in.

>intelligent culling makes it easier to find things

That's true, but things get easier to find over time. I have thousands of digital photographs from the 1990s that I burned on CDs and then copied to a NAS later. Today, for the first time, they're actually searchable because they're now indexed in Immich. So they're sorted by the person in them and searchable by their contents.

If I had culled the photos back then, I would have lost many photos that have become searchable (and thus valuable) only recently.

Re: Data centers contain 90% crap data

#97

Post by Jean-Papoulos »

The other day I had to go through 15 years old PowerPoint files to grab the originals of pictures made by a guy extremely proefficient at creating detailed artwork from PowerPoint forms that's now retired. Was can now render them to full-hd PNGs instead of the 256x256 BMP files we were using before.

Storing "useless" data makes financial sense.

Re: Data centers contain 90% crap data

#98

Post by kkfx »

Not too related but... Did you ever tried to imaging a world where computing is back to interconnected desktops/homeservers? Well, if you see trends like {fog,edge}-computing or recent ideas to distributed LLM computing (BrianknowsAI DCI Network for instance) it's clear that giants well known they can't keep up the modern mainframe model preferring to makes users paying the iron and bandwidth while they handle the software stack.

Now it's clear the new deal could be implemented only in homes/sheds with domestic p.v. and storage, smart cities keeps to fail since the ancient Fordlandia, see Neom, Songdo, Masdar, PlanIT Valley, Lavasa, Ordos, Santander city, Toronto Quayside (Google Sidewalk Labs), Amazon HQ2, Egypt new Cairo still nameless, Modi's Indian 100-smart city program, Arkadag, Innopolis, Nusantara, Proton City, ... and can't be powered with a smart-grid at such scale.

So well, new well insulated buildings, with ventilation of course, with p.v. and storage, with room for a domestic rack(s), with FTTH. Anyone with such settlements could have his/her own "datacenter" at home, following the same trend for medical devices more and more cheaper and smaller. A LOM? Well a NanoKVM PCIe or an external JetKVM cost MUCH less than classic LOM and do much more. We have all the gear to makes such "datacenter at home" assemblies, anyone holding preferred crap and participating in distributed computing networks to pay at least a bit gear and bandwidth.

It's not for all of course, some will be trapped in dense cities while some large owner dream an obviously not possible conversion from offices to apartments and datacenters like https://finance.yahoo.com/news/southern-californias-hottest-... or https://www.euronews.com/next/2024/02/29/madrid-to-convert-u... and https://www.theguardian.com/society/2025/jan/05/office-to-ho... or https://czechdaily.cz/half-of-pragues-office-buildings-are-a... etc for all over the developed world. That's while we admit https://doi.org/10.1073/pnas.2304099120 we need a full-remote DISTRIBUTED shift.

Food, meds, general retail distributed by a single integrated logistic platform for maximum efficiency in a spread society, the IT evolution makes Distributism possible.

Doing so erase the big amount of concentrated energy, dense network, heat handling and water problem of datacenter and also reduce much the crap, because anyone keep it's own personal and being not free keeping it they'll learn to be storage continuous.

Re: Data centers contain 90% crap data

#99

Post by srmarm »

Some fair points, but some 'crap data' has storage requirements orders of magnitudes from each others.

One short video can equal a year worth of emails for someone. Similarly those many webpages that don't get viewed often probably require only a negligible amount of resources to keep online and might help someone who'd otherwise be faced with linkrot.

Best to focus on the low hanging fruit.

Re: Data centers contain 90% crap data

#100

Post by plasticeagle »

otterley wrote: post #75

90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted…

> 90% of libraries consist of books that are never opened

I'm reasonably certain that this statistic is completely made up. The best number I can find for the proportion of library books that are never borrowed was from a university library, and was 25%.

Post reply on HN