All my servers have an 8 GB empty file on disk • Hacker News Board

Re: All my servers have an 8 GB empty file on disk

#381

Post by zokier » Thu, Mar 25, 2021, 11:38 PM UTC

tshaddox wrote: post #321

vineyardmike wrote: post #203

Earlier quoted context omitted.

>You have an easy way to fix it, just delete the spacer file. But what good does that do? You solve the issue right then and there. Step 1. realize there is a space issue and get to terminal Step 2. free space so any solution has memory Step 3. Solve by doing

So if your alarm sounded when there was 8 GB of free disk space (instead of 0 GB), then you could still respond in the same amount of time and you would still have an additional 8 GB worth of padding while you determined the root cause. The only difference is that you wouldn't need to actually go in and delete the spacer file (and potentially have downtime in the time it takes you to delete the spacer file). Another…

The thing you are still missing is that no matter how you set your alarms, by the time you respond the disk can still be completely full.

Re: All my servers have an 8 GB empty file on disk

#382

Post by skissane » Thu, Mar 25, 2021, 11:38 PM UTC

njovin wrote: post #335

victor9000 wrote: post #163

Earlier quoted context omitted.

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

Years ago I worked as a research assistant for a university. One day, my boss (a professor) pulled me aside for an impromptu meeting. "I have $5000 left in a research grant I need to spend this week or else it's gone forever – do you have any ideas of what I should spend it on?"

Unfortunately I couldn't think of much. I suggested maybe we buy some more computers with it but I'm sure he'd already thought of that himself. I don't know what he ended up doing, but I'm sure he'd have decided to buy something with it rather than just losing it entirely.

Re: All my servers have an 8 GB empty file on disk

#383

Post by Aeolun » Thu, Mar 25, 2021, 11:43 PM UTC

njovin wrote: post #335

victor9000 wrote: post #163

Earlier quoted context omitted.

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place.

Definite case of misaligned incentives.

Re: All my servers have an 8 GB empty file on disk

#384

Post by IgorPartola » Thu, Mar 25, 2021, 11:48 PM UTC

tehjoker wrote: post #141

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

It is a shame that motorcycles have moved away from this model. My last bike had a manual petcock with a reserve setting. It was problematic because I’d forget to turn it from off to on, take off on what’s left in the carburetor bowl, and the engine would start sputtering just down the road. But I also never got stranded.

New bike has a vacuum-actuated fuel valve, no reserve. It does have a fuel gauge but since the tank is not a nice simple rectangle and the angle makes a difference the gauge is basically untrustworthy. So I go by the mileage and hope I don’t get it wrong. How hard would it be for them to add a reserve setting so it could just be between On and Reserve so I could just flip between them as needed?

Re: All my servers have an 8 GB empty file on disk

#385

Post by robbomacrae » Thu, Mar 25, 2021, 11:50 PM UTC

woko wrote: post #190

buffet_overflow wrote: post #140

Earlier quoted context omitted.

I still know people that pre-emptively buy toilet paper "because hoarders might buy the rest of it" with absolutely no introspection.

Hoarders will buy the rest of it. During the 2-month lockdown a year ago, I would purchase 4 frozen pizzas at a time when I had a chance to buy them, because I was so upset that I could not buy one when I wanted a single one during the 2 previous weeks, because of hoarders who had been faster than me. People think of toilet papers, but it is not just that. Pasta, rice, flour, yeast, plenty of useful things went missi…

I hear this derogatory term "hoarder" being thrown around a lot. When the pandemic first hit I bought a ton of food. I still ate and bought the same amount over time.. Nothing went to waste. I just wanted to make FEWER but BIGGER trips to the supermarket so I had reduced chance of getting infected.

In the same sentence you complain about "hoarders" being faster than you whilst justifying (reasonably well imho) why you yourself bought more.

People did what they thought they needed to and will always do so. The world was always still a bit of a jungle out there (any nation is three meals away from anarchy). I'm sure those with immune compromised or elderly family members were more excessive on "hoarding". We need to get out of this blame mentality and realize our system needs improving so it can respond to these challenges.

Re: All my servers have an 8 GB empty file on disk

#386

Post by bigiain » Thu, Mar 25, 2021, 11:51 PM UTC

chihuahua wrote: post #378

bigiain wrote: post #326

Earlier quoted context omitted.

Some older bikes, like my '99 Ducati Monster, don't have a petcock. It has a low fuel light that first failed in around 2002, and for which that part that fails (the in-tank float switch) stopped being available in about 2015 or so. No petcock _or_ warning light. (And that trip where the speedo cable fails so I couldn't even use thew trip meter to estimate fuel requirements was a fun one...)

Can you find someone who can adapt a float switch from a different bike? It seems like a very useful thing to have, even if it's not the original factory part.

I've just gotten used to it. I'm fairly reliable about always resetting the trip meter when I fill it up (and always fill it to full). I know it'll get 200km easy, maybe only 180 if I'm having _way_ too much fun. That's always about time I want to stop and stretch my legs anyway. It doesn't bother me enough to "solve the problem".

Re: All my servers have an 8 GB empty file on disk

#387

Post by dyingkneepad » Thu, Mar 25, 2021, 11:53 PM UTC

vineyardmike wrote: post #203

luckylion wrote: post #170

Earlier quoted context omitted.

Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the spacer file. But what good does that do? You're still in the mess where your database is really unhappy. On the other hand, if your monitoring was set up well, you got a notification and had time to react to it before it was at 100%. Granted, if you have a process…

>You have an easy way to fix it, just delete the spacer file. But what good does that do? You solve the issue right then and there. Step 1. realize there is a space issue and get to terminal Step 2. free space so any solution has memory Step 3. Solve by doing

Let me try to explain what the other person is saying.

If you have an 8gb spacer file, at some point the disk will get full and cause some errors, so you will have to log in, remove the spacer file and then deal with the problem.

If you have an alarm for 8gb remaining, you will receive the alarm before any application ever notices the disk is full. You will have basically the same amount of time to solve the problem for good, but if you're able to solve it before these 8gb also ends, you won't have to deal with any "app crashed/misbehaved because it reached a point where no space was available" issues.

Re: All my servers have an 8 GB empty file on disk

#388

Post by agumonkey » Thu, Mar 25, 2021, 11:53 PM UTC

chihuahua wrote: post #359

agumonkey wrote: post #233

Earlier quoted context omitted.

is there a model to solve these ?

I am far from an expert on game theory, but it seems that the cause of the tragedy of the commons is that people can use the shared resource for free. If there was a price to be paid, and the price was dynamically adjusted depending on conditions, then the overuse could be avoided. Similarly for file storage and "reserving" it by creating huge but useless files. If everyone was charged a fee per gigabyte per day, the…

I was wondering if not all of human society is game theory through time/space taxes and prices.

Re: All my servers have an 8 GB empty file on disk

#389

Post by kaminar » Thu, Mar 25, 2021, 11:54 PM UTC

rgj wrote: post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

Create empty disk partition not possible? Seems easier to manage, no?

Re: All my servers have an 8 GB empty file on disk

#390

Post by dyingkneepad » Thu, Mar 25, 2021, 11:54 PM UTC

zokier wrote: post #381

tshaddox wrote: post #321

Earlier quoted context omitted.

So if your alarm sounded when there was 8 GB of free disk space (instead of 0 GB), then you could still respond in the same amount of time and you would still have an additional 8 GB worth of padding while you determined the root cause. The only difference is that you wouldn't need to actually go in and delete the spacer file (and potentially have downtime in the time it takes you to delete the spacer file). Another…

The thing you are still missing is that no matter how you set your alarms, by the time you respond the disk can still be completely full.

The spacer solution guarantees you will always have one instance of "disk got full", even if the fill rate is slow. With an alarm there is a chance you end up solving the problem with zero instances of "disk got full" if the fill rate is not that big. That is the main difference.