Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

101–110 of 146 posts

Re: Data centers contain 90% crap data

#101

Post by mvc »

otterley wrote: post #75

90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted…

> 90% of libraries consist of books that are never opened.

Citation required. But don't bother because it's a meaningless statistic, or at least one designed to make it look like there's a lot more wastage in libraries than there actually is.

The statistic could be true, and yet still be the case that the vast majority of library books are well utilized.

Re: Data centers contain 90% crap data

#102

Post by ghaff »

InsideOutSanta wrote: post #95
ghaff wrote: post #90

Earlier quoted context omitted.

I do cull photos and docs but it takes effort and you will make mistakes. On the other hand, intelligent culling makes it easier to find things. It’s a trade off especially with digital where there’s not that much cost associated with holding onto things relative to the physical world. My whole house is basically in storage because of a kitchen fire and I’m planning to be pretty selective about what I move back in.

>intelligent culling makes it easier to find things That's true, but things get easier to find over time. I have thousands of digital photographs from the 1990s that I burned on CDs and then copied to a NAS later. Today, for the first time, they're actually searchable because they're now indexed in Immich. So they're sorted by the person in them and searchable by their contents. If I had culled the photos back then,…

And geoencoding on especially phone photos makes manual metadata entry less important. I still there’s value to culling redundant and bad photos.

Re: Data centers contain 90% crap data

#103

Post by Double_a_92 »

2muchcoffeeman wrote: post #59
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

Photos of kids? Maybe that one will be The One that we end up framing? Look through the pictures with your kids (and with your kids) and pick out the best ones. Delete the rest.

The problem with that is that what you might find important or beautiful changes with time.

That random photo of your messy living room might be trash right now, but beloved in 20 years when you want to remember your old home.

Re: Data centers contain 90% crap data

#104

Post by numpad0 »

ivraatiems wrote: post #27
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with…

  probably a similar percentage of people claim on their car insurance. 
  In that 0.01% of cases where the order was fraudulent, placed by mistake, 
  how much data do we need to store inefficiently, in a way that uses a lot of power and space?
I'm feeling, the real real question, as Sci-Fi as it gets, is, is the winning ticket data even data, OR is it more like a thumbnail of "the whole data" that is 98%+ worthless, than standalone piece of data?

The winning ticket ID, e.g. a "0x-636e-7461-4e7b", only makes sense in the context as one among the entire cohort of contestants; I can make one up like I did, but I can't walkout with the payout unless the rest of the lottery didn't exist.

Statistically, philosophically, technically, and all sorts of *-cally speaking, is the 2% data, the winning ticket datum, even data?

Re: Data centers contain 90% crap data

#106

Post by eloisius »

mrexroad wrote: post #77
eloisius wrote: post #35

Earlier quoted context omitted.

Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”

If you’re using Apple Photos, the feature is there. It will detect both exact copies as well as (nearly) identical visual duplicates. Look under utilities.

Maybe it's different on the desktop, but on the iPhone it only detects the same photo that may have been saved at different resolutions or things like that. That's useful, but I really want something that can say use an embedding or something to group all 10 photos I shot of friends around the campfire and help me select one keeper and delete the other 9.

Re: Data centers contain 90% crap data

#107

Post by andyp-kw »

mrb wrote: post #93

Some fun math: according to some estimates there is 175 zettabytes of data worldwide. Assuming 20-terabyte harddrives, this could be stored on 8.8 billion drives. Assuming 10 drives per rack unit, 42 RU per rack cabinet, 16 square feet per cabinet (including aisle space), that means you need about 330 million square feet of data center space to host this data. If it was hosted in a single square data center, it would…

I wonder how it compares to electric car usage, or street lights in the USA.

Re: Data centers contain 90% crap data

#109

Post by ghaff »

faust201 wrote: post #56
AStonesThrow wrote: post #36

Earlier quoted context omitted.

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it'…

> Especially deleting old emails and photos. Google or apple can put a big button delete everything in their phones/accounts but then some prankster will do it to a family member and this gives bad PR. Let's be pragmatic. > It's in their interest to retain everything at all costs, because more used storage can mean more profits. As an user, I get the reverse. When I had local NAS then I dumped anything and everything…

Assuming your somewhat methodical about labeling email in your non-primary tabs in Gmail, you can pretty much do periodic purges while not worrying much about your primary which, in my case at least, gets relative little traffic.

Re: Data centers contain 90% crap data

#110

Post by saalweachter »

Cthulhu_ wrote: post #72
duxup wrote: post #4

Earlier quoted context omitted.

I ended up working on some software and I was deemed the email guy (it's a very small % of my job but it is the biggest pain). "I need an email when this happens.. and when this happens." The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. The insidious problem is that it never solves…

I can see how that would work (email centric workflow), not unlike how some people now try to have a chat / Slack centric workflow.

Yeah, everyone knows the proper thing to do is orchestrate it all through Emacs.
Post reply on HN