Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

11–20 of 146 posts

Re: Data centers contain 90% crap data

#11

Post by danpalmer »

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored.

On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money!

This is obviously sarcasm, but I think it's important to remember that much of the data is stored because we don't know what we will need later. Photos of kids? Maybe that one will be The One that we end up framing? Miscellaneous business records? Maybe those will be the ones we have to dig out for a tax audit? Web pages on government sites? Maybe there will suddenly be an interest in obscure pages on public health policy if a global pandemic happens.

Complaining that data is mostly junk is not a particularly interesting conclusion without acknowledging this. Is there wastage? Yeah sure, but accuracy on what needs storing is directly traded off with time spent figuring that out, and often it's cheaper to store the data.

Re: Data centers contain 90% crap data

#12

Post by tylerrobinson »

cactusplant7374 wrote: post #10

Fortunately with LLM's we can generate data and not store it. If you think about it that means articles could always be relevant with additional details added as time goes on.

No, the reality of this situation is that people need to save a snapshot of what the LLM said when they read it, just in case they need to substantiate it or blame someone for what they read at the time.

Re: Data centers contain 90% crap data

#14

Post by nameless912 »

There's another dimension to this, that storage is so cheap that being wasteful with it isn't really disincentivized. I know for example at work of a portal that accepts uploads of large files from external clients that stores both the initial upload and every subsequent transformation of the file (of which there are 4-6) permanently. It's extremely useful for debugging, as one of the bits of metadata we shove on the zip archive is the git hash of the code that was running, so it's trivial to pull down any failed step and diagnose what happened.

We are using 4-6 times as much storage as we need to, and these are often not small files (on the order of 100 MB - 5 GB, several dozen times a day) but fixing this overuse is so far down the priority list that I don't think it survived the great Jira purge of mid-2024.

Re: Data centers contain 90% crap data

#15

Post by manyturtles »

About a decade and a half ago I worked on a large data migration project at a FAANG. Multi-exabyte scale, many clusters across many countries. Once everyone was moved the old storage platform wasn't completely empty, because the number of migrations was large and users were (naturally) more focused on ensuring their data was in place and available on the target platform rather than ensuring every last thing was deleted on the legacy platform. We weren't initially concerned about it because it would all get deleted when we turned down the old setup.

As we were gearing up to declare victory and start turning down the several dozen legacy storage clusters someone mused that given some users were subject to litigation holds -- not allowed to delete any data -- that at least some of the leftover data on the old system might be subject to litigation hold, and we'd need to figure that out before we could delete it or incur legal risk. IIRC the leftover 'junk' data amounted to a few dozen petabytes spread across multiple clusters around the world, in different jurisdictions. We spent several months talking with the lawyers figuring that out. It was an interesting dance, because on the one hand we were quite confident that there was unlikely to be anything in the leftovers which was both meaningful and not migrated to the new platform, while on the other hand explaining that it wasn't practical to just "go and look" through a few dozen PB of data. I recall we ended up somewhere in between, coming up with ways to distinguish categories of data like caches and working data from various pipelines. It added over six months to the project, but was quite an interesting problem to work through that hadn't occurred to any of us earlier on, as we were thinking entirely in technical terms about infrastructure migration.

Re: Data centers contain 90% crap data

#17

Post by irjustin »

cactusplant7374 wrote: post #10

Fortunately with LLM's we can generate data and not store it. If you think about it that means articles could always be relevant with additional details added as time goes on.

At the moment, I don't like this for fear of revisionist history. The equvilent would be if wikipedia didn't have to have a source reference.

not saying that it's infallible but yeah we need some chain of audible at least 1 or 2 layers deep.

Re: Data centers contain 90% crap data

#18

Post by ein0p »

A gross underestimation, IMO. When I was in big data, fewer than 5% of data written was ever touched again, and only a single digit number of our large customers (out of tens of thousands) actually made real use of their "big data", and created most of the load. That's the trouble with "checkbox driven development" - 10 years ago you were required to have a "big data strategy" for anyone to take you seriously, even if your strategy boiled down to just ETL-ing a bunch of crap you're never going to need into the cloud and never touching it again. Now I'm in AI, and the same thing is happening to AI. It's great if you're selling shovels, so to speak, but not so great if you plan on selling them for an extended period of time.

This, by the way, has implications on storage systems design. You want something that's cheap yet dense to encode, potentially at the slight expense of decode speed. Normally people really lose sleep about decode speed first and foremost, which, while important, does not minimize the overall resource bill.

Re: Data centers contain 90% crap data

#19

Post by jodrellblank »

It's a problem for us when it comes to the GDPR and rights to be forgotten; companies will say they store your data carefully, will say they have shown you everything they have, will say they delete it, but "the company" in aggregate has no idea there's a thousand SharePoint sites and ex-employees mailboxes and filestores, and copies of old filservers from before a migration, and test databases containing copies of real data from a half-abandoned project where new management fired the contractors and then never got around to hiring new ones.

Re: Data centers contain 90% crap data

#20

Post by justsomehnguy »

HPsquared wrote: post #7
boznz wrote: post #2

Don't forget emails.. I have everything I ever sent or received, and I have it backed up. I expect 90% of my inbox is the jpg signature logo they attach to the bottom of my clients email rather than hyperlink.

It's probably deduplicated on the server though, so the millions and millions of messages with that logo likely share the same piece of disk space. Probably one reason why free providers don't tend to offer End-to-End encryption. It prevents deduplication (and otherwise compressing redundant information).

Nope.

Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file.

And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TBs of hashes to check. Until you have like 99% dedup efficiency, ie 99% of your incoming data is literally the same data you already have - it doesn't worth it.

https://techcommunity.microsoft.com/blog/exchange/dude-where...

Post reply on HN