Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

401–410 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#401

Post by tremon »

lostcolony wrote: post #174
diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

You car also doesn't drop from "alarm" to "empty" in thirty seconds. A HD on a VM with a bad behaving process can.

That's a quality problem. Your car can absolutely drop from "alarm" to "empty" in 30 seconds if there's a leak in the tank. We just don't build fuel tanks that don't spontaneously develop leaks, partly because the manufacturer can be held liable.

Re: All my servers have an 8 GB empty file on disk

#402

Post by davesque »

On the subject of "inverted" thinking like this, I recently added a test to a test suite that is intended to fail some day. The test will eventually fail when a bug (for which we developed a workaround and added the aforementioned test to confirm the fix) is fixed in one of our open source dependencies. When the test fails, we'll know to remove the workaround (and the test)!

Re: All my servers have an 8 GB empty file on disk

#404

Post by rincebrain »

throw0101a wrote: post #51
AlisdairO wrote: post #11

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.

ZFS has explicit reservations: > The minimum amount of space guaranteed to a dataset and its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations. * https://openzfs.github.io/openzfs…

ZFS, like ext[34], also has a reserved space allocation for the entire pool to allow you to still do certain operations when it's reporting "0" free space (...like deleting files or snapshots, to free space).

Unlike ext[34], that reservation is not available for root's general use, but it's there.

Re: All my servers have an 8 GB empty file on disk

#405

Post by chuckleMuscle »

skissane wrote: post #382
njovin wrote: post #335

Earlier quoted context omitted.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

Years ago I worked as a research assistant for a university. One day, my boss (a professor) pulled me aside for an impromptu meeting. "I have $5000 left in a research grant I need to spend this week or else it's gone forever – do you have any ideas of what I should spend it on?" Unfortunately I couldn't think of much. I suggested maybe we buy some more computers with it but I'm sure he'd already thought of that himse…

This as a fairly recent occurrence in my research group. It’s often quite tedious because you don’t want to waste the money and it’s never clear if there’s going to be a period where we’re short on cash at some point in the future. most of it’s spent on boring but expensive things to be used down the line. Would be far better if funding wasn’t quite so cyclical!

Re: All my servers have an 8 GB empty file on disk

#406

Post by osrec »

njovin wrote: post #335
victor9000 wrote: post #163

Earlier quoted context omitted.

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

Reminds me of this episode of The office: https://en.m.wikipedia.org/wiki/The_Surplus

Re: All my servers have an 8 GB empty file on disk

#407

Post by twistedpair »

ABeeSea wrote: post #300
twistedpair wrote: post #261

Earlier quoted context omitted.

I worked at a large company during a migration from Lotus to Outlook. We were told we'd get our current Lost email storage + 100MiB as a new email quota limit under Outlook. I made a bunch of 100MiB files of `/dev/random` noise (so they don't compress, compressed size was part of the quota) and emailed them to myself before the migration, to get a few GiB of quota buffer. My co-workers were constantly having to delet…

Email quotas aren’t just a cost thing. It forces deletion of files/communications that aren’t relevant anymore. The last thing the legal department wants is some executive’s laptop with 10 years of undeleted email to make it’s way to discovery.

That's why I did it. I'd always be trying to find an email from the prior year, that held a fix I needed to use again, but it had been deleted to stay in quota. Old email can be helpful.

Re: All my servers have an 8 GB empty file on disk

#408

Post by tremon »

vineyardmike wrote: post #211
copperfoil wrote: post #133

Earlier quoted context omitted.

> The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making findin…

>you'd have to solve the problem the same way if you didn't use this method. What if the solution required some amount of free space? (eg. installing a package or swap)

ramdisk? It wouldn't be the first time I'd extract a .deb to tmpfs to resolve a temporary issue.

Don't think I've ever encountered a critical issue where "add more swap" would be a serious disaster recovery solution. I've certainly seen situations where swap was nearing 100% full, and although I would have minutes off wall-clock time to formulate a strategy, those minutes have never allowed me to input more than a handful of characters or so.

Re: All my servers have an 8 GB empty file on disk

#410

Post by tremon »

cure wrote: post #236
jodrellblank wrote: post #176

Earlier quoted context omitted.

Yes they do with the "dyanmic allocation" the parent comment mentions; VMware datastore has 1TB total, you put VMs in with dynamically expanding disks they are sharing the same 1TB of free space and will fill it if they all want their max space at the same time and you've overprovisioned their max space. And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed…

I see, a VMware feature, thanks for clarifying. I suppose it's a nice idea in theory, but you'd have to be crazy to use that in production, or for any workload that you care about. It would just be a ticking time bomb.

Hyper-V can do that too, and so can you under Linux. It's called thinly-allocated disks, sparse files, or the dm-thin device mapper target. Professional SANs also allow you to overallocate the total size of the iSCSI volumes offered.

Yes, I've seen that time bomb go off on multiple occasions. Never on my watch though.

Post reply on HN