Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

141–146 of 146 posts

Re: Data centers contain 90% crap data

#141

Post by econ »

AStonesThrow wrote: post #58
nn3 wrote: post #46

Earlier quoted context omitted.

I suspect for most cloud providers you deleting data is cheaper because the data is not charged by the byte. But then they like having data, maybe just to train their AI models or for bragging rights to their investors. For the expiration dates most modern file systems have the concept of arbitrary extended attributes per file. It's quite easy to add meta data like this yourself.

Unused services are always pure profit, and storage space is no exception. Providers can offer 100GB for like $24/year, because only like 2% of the subscribers will ever approach the limit, and so the extra space is never wasted but can be allocated to someone or something else. It's like gym memberships, ISP/telco service bundles, and amenities at your apartment complex. Anyone not using every possible service is wa…

Ah yes, when doing the manual Cron job I often say out loud to myself this is not why we made computers!

There is no clear path forwards. Perhaps regulation could be a solution but I doubt it would turn out nice.

Re: Data centers contain 90% crap data

#142

Post by econ »

bob778 wrote: post #116
AStonesThrow wrote: post #58

Earlier quoted context omitted.

Unused services are always pure profit, and storage space is no exception. Providers can offer 100GB for like $24/year, because only like 2% of the subscribers will ever approach the limit, and so the extra space is never wasted but can be allocated to someone or something else. It's like gym memberships, ISP/telco service bundles, and amenities at your apartment complex. Anyone not using every possible service is wa…

There’s a whole concept of records management in enterprises that manages the disposal of data. It’s far more complex than just purge dates as there’s often regulatory requirements and legal discovery issues so For personal data, the concept would be simpler but still has requirements like say tax records need to be kept 7 years.

Maybe park the data with the regulator as leaving it in your hands longer than you need it is asking for trouble.

Re: Data centers contain 90% crap data

#143

Post by econ »

eloisius wrote: post #35
mrexroad wrote: post #22

Earlier quoted context omitted.

I have ~2.5TB of photos in iCloud, via Apple Photos. Excluding the various sized previews, I doubt many originals have been accessed in quite a while. I also have about 1TB Lightroom library archived to a different service; representing countless hours of photo processing work spanning over a decade. Haven’t touched that one in years. Neither are crap. (Yes, both have other backups and, yes, I’ve probably forgotten t…

Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”

I download them 200 at a time then delete the selection.

Re: Data centers contain 90% crap data

#144

Post by elmolino89 »

Without rules, metadata, systems to organize files even the most valuable data can be stored as a steaming pile of poo. In my case I was diving into a compressed tar containing compressed tar-s, which again may have giant uncompressed data files inside. The whole pile of crap had premature EOF so for sure some non trivial amount of data was lost. And on the top of it what was preserved should have a prefix: "unobtanium_". Storing several tar.gz several terabytes in size on some puny domestic use NAS made extraction of the files a crying game.

In another site I found a mix of some old version windows disk images with data. With more crap inside.

In the end: storage may be cheap. Storing piles of disorganized crap is very costly if you want to find something

Re: Data centers contain 90% crap data

#145

Post by 392 »

precommunicator wrote: post #137

We recently discovered we store 500MiB of email tokens (expired). Then a copy of them in history table. Then this data is replicated. Then there are total of 4 backups of it. Backups that are done every 2 hours for half of each day. And we store those backup for up to half a year. I don't even want to add that up...

If you believe adding it up linearly is valid, and you care about the results, you should be using more efficient storage software.

Re: Data centers contain 90% crap data

#146

Post by precommunicator »

392 wrote: post #145
precommunicator wrote: post #137

We recently discovered we store 500MiB of email tokens (expired). Then a copy of them in history table. Then this data is replicated. Then there are total of 4 backups of it. Backups that are done every 2 hours for half of each day. And we store those backup for up to half a year. I don't even want to add that up...

If you believe adding it up linearly is valid, and you care about the results, you should be using more efficient storage software.

It's "nice go have" i.e. cheaper to just pay for more storage than do it properly
Post reply on HN