Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

51–60 of 146 posts

Re: Data centers contain 90% crap data

#51

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

If I knew the data i was going to need in the future i'd be in the future predicting business.

But there's an important piece there that is about data that should not have been stored in the first place. All the big data bullshit made people believe every single piece of data you could possibly store from a user is data that you should store and this is both a huge waste of resources and a huge liability, because now a data leak with useless to you PII could completely bankrupt your business.

Re: Data centers contain 90% crap data

#52
post #49

When traveling, a funny thought I always have is watching other tourists take the same photos, from the same exact location, knowing it will be backed up to iCloud. I can't help but imagine how much disk space is taken up by duplicates of the exact same photo.

That and videos of concerts / fireworks shows.

For a fun example, every time I'm on the Las Vegas strip I see dozens of people taking videos of the Bellagio Water Show.

There are 30 shows per night, if 50 people take videos in 4K 60fps (default on new iPhones), that's around 60 GB of data per show or ~600 TB per year of just videos of the Bellagio Fountain Show!

Re: Data centers contain 90% crap data

#53

Earlier quoted context omitted.

Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”

Thing is, the tool needs you to pick the one instance that survives the purge. Which means looking through 10^x of mostly similar photos to pick one. Removing duplicates isn't that complex already, and these tools exist so anyone can try it. It's just a truely grueling process.

Yeah, there have been a number of companies that say they'll offer AI Photo Culling, but I have yet to find one that actually does it.

Re: Data centers contain 90% crap data

#55

Earlier quoted context omitted.

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with…

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it'…

I would love a feature like retention policies at the filesystem level. If it could somehow take data provenance into account, that would be even better, i.e. data that I've created directly is irreplaceable and should never be deleted unless I say so and should always be backed up (photographs, writing, project source code, art, game save files, my list of installed applications, etc.); data that I've created indirectly is high priority but may be deleted under certain circumstances or may be omitted from backups (browsing history, shell history, recently opened file lists, frequently used app lists); data that can be easily replaced is low priority and may be cleaned up at any time and need not be backed up at all, contingent on how replaceable it is (application files, cached data).

Re: Data centers contain 90% crap data

#56

Earlier quoted context omitted.

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with…

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it'…

> Especially deleting old emails and photos.

Google or apple can put a big button delete everything in their phones/accounts but then some prankster will do it to a family member and this gives bad PR. Let's be pragmatic.

> It's in their interest to retain everything at all costs, because more used storage can mean more profits.

As an user, I get the reverse. When I had local NAS then I dumped anything and everything - assuming.. this costs not much. Will clean up later. Once I moved to cloud that changed to If I put crap then it will cost me money! Keep it clean

Once upon a time Google had given generous unlimited to all education and workspace. They stopped it in the last 2 years and you can see most educational and companies are running a tight ship.

Due to backup costs in our organisation we are forcing people to use max of 100GB for emails.

> advanced filesystems don't include "expiration dates" or "purge dates" on file objects. Wouldn't it be logical, if an organization

Totally agree.

Outlook email service has some kind of keep only the last newsletter from this service etc.

> auto-delete could be configurable account-wide, for more than just our web histories

- features are designed with majority in mind - most users have some sort of nostalgia for reading or keeping old emails SMS etc

Re: Data centers contain 90% crap data

#58
post #46

Earlier quoted context omitted.

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it'…

I suspect for most cloud providers you deleting data is cheaper because the data is not charged by the byte. But then they like having data, maybe just to train their AI models or for bragging rights to their investors. For the expiration dates most modern file systems have the concept of arbitrary extended attributes per file. It's quite easy to add meta data like this yourself.

Unused services are always pure profit, and storage space is no exception. Providers can offer 100GB for like $24/year, because only like 2% of the subscribers will ever approach the limit, and so the extra space is never wasted but can be allocated to someone or something else.

It's like gym memberships, ISP/telco service bundles, and amenities at your apartment complex. Anyone not using every possible service is wasting money, but it's impossible to purchase a bespoke service, so essentially everyone will waste money because they're chipping in money for services that someone else uses more than they do.

Here at home I don't ever use the gym, the racquetball courts, the doggy-doo supplies, the laundry room, or a parking space, and yet my rent (everyone's rent) includes upkeep for all of those things. I'm subsidizing all my neighbors and all the wear-and-tear damage they put on those common amenities. Likewise, everyone who's paying $24/year for storage, or any business that purchases big multi-terabyte storage media, they're paying for unused storage space and giving profit. It's practically impossible to rightsize your storage media, and you never want it undersized, and you can't simply shrink them and reclaim the resources you invested, you just keep adding on new ones and replacing the malfunctioning ones. So nearly everyone always owns or rents more space than they can realistically utilize.

Furthermore, you'll notice that I specified "automatic" destruction of data by expiration date. Of course it is trivial to tag any arbitrary file with arbitrary metadata, but the challenge is to create a filesystem that executes automatic data purges on schedule, rather than pushing it into a rickety old handmade cronjob in userland. I've never ever seen a filesystem with such a feature, nor does it seem that anyone's interested in doing so.

And here I thought that computers were useful for automating business logic and easing the burden on human effort. And this is me, manually sifting through emails and photos in order to manually delete each one with 3 dialog boxes intervening. It takes hours, days, weeks.

Re: Data centers contain 90% crap data

#59

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

Photos of kids? Maybe that one will be The One that we end up framing?

Look through the pictures with your kids (and with your kids) and pick out the best ones. Delete the rest.

Post reply on HN