All my servers have an 8 GB empty file on disk • Hacker News Board

Re: All my servers have an 8 GB empty file on disk

#321

Post by tshaddox » Thu, Mar 25, 2021, 10:34 PM UTC

vineyardmike wrote: post #203

luckylion wrote: post #170

Earlier quoted context omitted.

Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the spacer file. But what good does that do? You're still in the mess where your database is really unhappy. On the other hand, if your monitoring was set up well, you got a notification and had time to react to it before it was at 100%. Granted, if you have a process…

>You have an easy way to fix it, just delete the spacer file. But what good does that do? You solve the issue right then and there. Step 1. realize there is a space issue and get to terminal Step 2. free space so any solution has memory Step 3. Solve by doing

So if your alarm sounded when there was 8 GB of free disk space (instead of 0 GB), then you could still respond in the same amount of time and you would still have an additional 8 GB worth of padding while you determined the root cause. The only difference is that you wouldn't need to actually go in and delete the spacer file (and potentially have downtime in the time it takes you to delete the spacer file).

Another way to think of this is that you have the 8 GB spacer file, but when the disk fills up the spacer file is automatically deleted and your alarm goes off. Which is literally the same as having your alarm go off when free disk space reaches 8 GB.

Re: All my servers have an 8 GB empty file on disk

#322

Post by mypalmike » Thu, Mar 25, 2021, 10:34 PM UTC

rozap wrote: post #118

diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignor…

Same was true of most motorcycles until rather recently, though with motorcycles it was rare that there was a fuel gauge at all. A sputtering engine was how you knew it was low. And I believe that like with motorcycles, the "reserve tank" in an old Beetle is really the same tank - there are two hoses located in the tank at different heights.

Re: All my servers have an 8 GB empty file on disk

#323

Post by tshaddox » Thu, Mar 25, 2021, 10:35 PM UTC

busterarm wrote: post #110

tshaddox wrote: post #99

How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!

Sometimes your alarms are broken due to misconfiguration.

Both solutions assume that you will have some way of knowing when the disk is full. Whether the "alarm" is an automated health monitoring system, or an angry customer calling your cell phone, there's no point in discussing how to solve problems without assuming that you have some way of knowing there a problem exists.

Re: All my servers have an 8 GB empty file on disk

#324

Post by worik » Thu, Mar 25, 2021, 10:36 PM UTC

xupybd wrote: post #146

legulere wrote: post #125

[flagged]

This is the opposite of private property ownership. Property ownership comes at a cost. That cost increases as supply dries up. In this case there is no cost to the individual, only to the group.

There is nothing intrinsic about property that implies cast.

I think you are confusing property and scarcity.

Re: All my servers have an 8 GB empty file on disk

#325

Post by tshaddox » Thu, Mar 25, 2021, 10:36 PM UTC

_wldu wrote: post #108

tshaddox wrote: post #99

How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!

It requires far less configuration.

How so? You'll presumably need to configure some way to be notified when the disk space is full anyway.

Re: All my servers have an 8 GB empty file on disk

#326

Post by bigiain » Thu, Mar 25, 2021, 10:40 PM UTC

driverdan wrote: post #254

dnautics wrote: post #193

Earlier quoted context omitted.

Surprised there isn't a mechanism that mechanically switches the petcock over when you put a fuel nozzle up to the port

Some newer bikes, like mine, don't have a reserve petcock. They have a low fuel light. No forgetting about the petcock and an obvious warning light instead of sputtering.

Some older bikes, like my '99 Ducati Monster, don't have a petcock. It has a low fuel light that first failed in around 2002, and for which that part that fails (the in-tank float switch) stopped being available in about 2015 or so. No petcock _or_ warning light. (And that trip where the speedo cable fails so I couldn't even use thew trip meter to estimate fuel requirements was a fun one...)

Re: All my servers have an 8 GB empty file on disk

#327

Post by tshaddox » Thu, Mar 25, 2021, 10:40 PM UTC

ineedasername wrote: post #155

luckylion wrote: post #131

Earlier quoted context omitted.

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.

Then why not automate the removal of the 8 GB spacer file when the disk gets full? Or in other words, just sound your alarms when there is 8 GB of free disk space.

Re: All my servers have an 8 GB empty file on disk

#328

Post by sedachv » Thu, Mar 25, 2021, 10:41 PM UTC

yabones wrote: post #112

danielrhodes wrote: post #80

My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.

100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are…

OpenBSD default partition allocation is really well thought out:

https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION

At least put /tmp in its own partition as well.

Multiple partitions can also save a lot of recovery time when there is a multiple bad sector event that corrupts an entire partition beyond recovery.

Re: All my servers have an 8 GB empty file on disk

#329

Post by luckylion » Thu, Mar 25, 2021, 10:41 PM UTC

pvorb wrote: post #220

luckylion wrote: post #187

Earlier quoted context omitted.

Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room? And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.

Somebody will notify you. If the service is just for yourself, you don't need monitoring at all.

Yes, yes, but they will notify you after your service is down (because that's when they notice), in part thanks to a spacer file that eats up available disk space without being of any use. A monitoring service would notify you before your service is down, users grab pitchforks and start looking for torches.

I understand the benefit to be able to quickly delete some file to be able to run some command that would need space, though I find that highly theoretical. If it's your shell that requires space to start, you won't be able to run the command to remove the spacer, and once you're in the shell, I've never found it hard to clean up space; path autocompletion is the only noticeable victim usually. And at this point, the services are down anyhow, and you likely don't want to restart them before figuring out what the problem was, so I don't see the point of quickly being able to make some room.

It feels like "having two flat tires at the same it is highly unlikely, so I always drive with a flat tire just to make sure I don't get an unforeseen flat tire". It's cute, but I'd look for a new job if anyone in the company suggested that unironically.

Re: All my servers have an 8 GB empty file on disk

#330

Post by rcthompson » Thu, Mar 25, 2021, 10:42 PM UTC

tehjoker wrote: post #141

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

Motorboat fuel tanks have a reserve as well. It's just a raised area that splits the bottom of the tank into 2 separate concave areas. One of the concave areas contains the end of the fuel line, and the other doesn't. When you run out of gas, you tip the tank up to dump the remaining gas from the other basin into the main one, and then you restart the engine (or keep it from stopping at all if you're quick enough on the draw) and head for the docks.