Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: Data centers contain 90% crap data

#111

Post by tored »

Sprite_tm wrote: post #30
justsomehnguy wrote: post #20

Earlier quoted context omitted.

Nope. Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file. And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TB…

Not sure about Microsoft solutions, but modern file systems like zfs and btrfs can do this on the filesystem level, no support from the mail server needed.

NTFS and ReFS has both deduplication support (only on Windows Server).

Re: Data centers contain 90% crap data

#112

Post by mrweasel »

duxup wrote: post #4
boznz wrote: post #2

Don't forget emails.. I have everything I ever sent or received, and I have it backed up. I expect 90% of my inbox is the jpg signature logo they attach to the bottom of my clients email rather than hyperlink.

I ended up working on some software and I was deemed the email guy (it's a very small % of my job but it is the biggest pain). "I need an email when this happens.. and when this happens." The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. The insidious problem is that it never solves…

We had a similar request for a sales team once: "If this fails we want an email".... Okay, but we never seen this fail, so you'll never get the email. As expected, they never got an email, because no failures. So instead they wanted an email every time the job ran (once every night), so they'd know that the job had failed, if they DIDN'T get the email. Only time that happened was because the email got trapped in a spamfilter I think.

There must be thousands of copies of that email sitting in inboxes say: Job X ran successfully @ 04:30.

Re: Data centers contain 90% crap data

#113

Post by mrweasel »

otterley wrote: post #75

90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted…

Now we're also getting into the topic of "what is waste". Because the majority of the books that are opened are rehashes of the same murder mystery over and over and over.

I'd guess that 75% of all new books sold here are variations on "Someone is murdered in a brutal fashion. A old drunken cop from somewhere in Scandinavia is assigned the case. He's helped by a young woman, who may be his daughter or who he'll a father daughter relationship to. They solve the case, maybe, the end. You just tweak the details a little, but it's the same bloody story over and over.

That seems like such a waste of paper in my mind.

Re: Data centers contain 90% crap data

#114

Post by cactusplant7374 »

tylerrobinson wrote: post #12
cactusplant7374 wrote: post #10

Fortunately with LLM's we can generate data and not store it. If you think about it that means articles could always be relevant with additional details added as time goes on.

No, the reality of this situation is that people need to save a snapshot of what the LLM said when they read it, just in case they need to substantiate it or blame someone for what they read at the time.

No, it requires a shift in thinking. There are no guarantees with information one reads as it is.

Re: Data centers contain 90% crap data

#115

Post by otterley »

mvc wrote: post #101
otterley wrote: post #75

90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted…

> 90% of libraries consist of books that are never opened. Citation required. But don't bother because it's a meaningless statistic, or at least one designed to make it look like there's a lot more wastage in libraries than there actually is. The statistic could be true, and yet still be the case that the vast majority of library books are well utilized.

I should have been less specific than “library” as such. Libraries aren’t only institutional. Consider every book, magazine, and newspaper that sits today in everyone’s home, office, and in every institution in the world. The vast majority of them have been read once and then left on a shelf or in a box somewhere, taking up more or less valuable space.

Re: Data centers contain 90% crap data

#116

Post by bob778 »

AStonesThrow wrote: post #58
nn3 wrote: post #46

Earlier quoted context omitted.

I suspect for most cloud providers you deleting data is cheaper because the data is not charged by the byte. But then they like having data, maybe just to train their AI models or for bragging rights to their investors. For the expiration dates most modern file systems have the concept of arbitrary extended attributes per file. It's quite easy to add meta data like this yourself.

Unused services are always pure profit, and storage space is no exception. Providers can offer 100GB for like $24/year, because only like 2% of the subscribers will ever approach the limit, and so the extra space is never wasted but can be allocated to someone or something else. It's like gym memberships, ISP/telco service bundles, and amenities at your apartment complex. Anyone not using every possible service is wa…

There’s a whole concept of records management in enterprises that manages the disposal of data. It’s far more complex than just purge dates as there’s often regulatory requirements and legal discovery issues so For personal data, the concept would be simpler but still has requirements like say tax records need to be kept 7 years.

Re: Data centers contain 90% crap data

#117

Post by newAccount2025 »

makeitdouble wrote: post #42
ivraatiems wrote: post #27

Earlier quoted context omitted.

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with…

> The real question is, how much data do we need to store inefficiently, in a way that uses a lot of power and space? This is indeed the critical question, and it's far from being trivial. One issue we all hit is moving the data from the higher tier storage to the cheaper and more efficient one, which requires sync and paying for the transfer most of the time, but also handling two separate access and authorization p…

Seems like we should be solving this at the storage system layer, instead of per application?

Re: Data centers contain 90% crap data

#118

Post by AStonesThrow »

palata wrote: post #67
zanecodes wrote: post #55

Earlier quoted context omitted.

I would love a feature like retention policies at the filesystem level. If it could somehow take data provenance into account, that would be even better, i.e. data that I've created directly is irreplaceable and should never be deleted unless I say so and should always be backed up (photographs, writing, project source code, art, game save files, my list of installed applications, etc.); data that I've created indire…

Not at the filesystem level, but Android does something that feels like that: an app can write files in its own space, and when you delete the app, it deletes its files. Except when the app writes files outside of its own space, which is meant for stuff that should stay (like pictures). Of course, some apps store the pictures in their private space, and you lose the pictures when you remove the app. And some apps wri…

See now, that's quite unrelated what I'm envisioning, or the topic of the OP's article, but that's definitely the way we're headed in terms of file handling.

"File-type strictly associated with app" is good for security and it's good for people's mental organization. Unfortunately you're referring strictly to local file storage, I assume, and practically nothing on my phone is in local storage but "in the cloud". The only local storage I use is for ringtones and certain audiobooks/prayers which I enjoy listening whether or not I'm connected to a network.

It would seem that data centers and cloud services are a long way from "file types strictly associated with app" and also the sort of behavior you describe would be really horrible and undesirable in a typical data-center setting. I mean, I can't even imagine how it'd be implemented, given that data storage is often remote or at least decoupled from the "application server", and it's definitely never a 1:1 relationship between those abstracted components.

Can you imagine needing to "delete the /bin/tar application" in order to purge every file ending in ".tar" or ".tgz"?

But it's true that the new mobile OS paradigm has advantages for cleanup. If you've got a 3rd-party app and you've created some big crap files with it, and you uninstall that app, then it's a clean sweep when it takes along all the crap and you don't need to worry about lingering loose ends, or files you simply can't open, because their handler's gone.

Scoped storage is way more secure; it's also wonderful on my Androids that I can selectively grant access to files and media, rather than giving "the keys to the kingdom" to all these Social Media apps. I definitely never want to post my bank screenshots to Facebook and the possibility no longer looms!

These days I am sort of nonplussed when a web app demands an all-access pass to my Google Drive, because permissions can be appropriately scoped now, and the app can play in its own Drive "Sandbox" without having access to my spreadsheets and medical records. I just wish that there were some really good "cleanup" apps. Remember those "cleanup" programs that would come with antivirus for your PC, or Regclean, or whatever freeware utility would purport to liberate your 10MB HDD? Those were good times.

It's the same with social media posts--I often wish to really scrub my account, delete all posts and all comments and basically all activity, and while that's possible on Facebook it's arduous and it takes me weeks to scrub the major stuff after 4-5 years. But I only think of it on Shrovetide Monday or Fat Tuesday, and then I spend 3 weeks in Lent doing scrub, scrub, scrub. Sometimes the mouse/keyboard gestures are more repetetive & physically demanding than cleaning the toilet. Way too difficult, and that's by design.

Re: Data centers contain 90% crap data

#119

Post by manyturtles »

isaacremuant wrote: post #78
manyturtles wrote: post #15

About a decade and a half ago I worked on a large data migration project at a FAANG. Multi-exabyte scale, many clusters across many countries. Once everyone was moved the old storage platform wasn't completely empty, because the number of migrations was large and users were (naturally) more focused on ensuring their data was in place and available on the target platform rather than ensuring every last thing was delet…

That does sound very interesting. Any insights on what would you do differently if you had to do it again? Any way to accelerate things now that you know the pain or do you think it's quite unavoidable and "legal Time"?

Doing it in parallel, which of course is only an option if you know about it. And templating that into a general approach and having legal folks sign off on that.

Re: Data centers contain 90% crap data

#120

Post by vb-8448 »

mrb wrote: post #93

Some fun math: according to some estimates there is 175 zettabytes of data worldwide. Assuming 20-terabyte harddrives, this could be stored on 8.8 billion drives. Assuming 10 drives per rack unit, 42 RU per rack cabinet, 16 square feet per cabinet (including aisle space), that means you need about 330 million square feet of data center space to host this data. If it was hosted in a single square data center, it would…

Furthermore: a big amount of this data is stored on physical tapes and not hard drives. A LTO-9 tape is able to store up to 45TB of data and doesn't require any electricity when not mounted to be read.
Post reply on HN