Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

541–550 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#541

Post by herpderperator »

nottorp wrote: post #536
herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

> So log in as root and fix the issue. Simple. Except you can't ssh in as root any more. You have to ssh as a normal user then sudo in. Which may not be possible. A paid on call admin may not mind driving to the data center too much. Everyone else would.

sshd runs as root, so yes, it will be possible to login as any user -- the sshd daemon will be able to function. :-)

The only files that would be appended to on user login (by sshd, which is running as root) would be /var/log/{utmp,wtmp,btmp} to record the login (in practice I've only seen wtmp). After that you have sshd logging (e.g. /var/log/{messages,syslog,auth.log}) which is picked up by syslog-ng or its equivalents, which also run as root.

Regarding not being able to login via SSH as root by default: the sshd_config default is actually `PermitRootLogin prohibit-password` which means you can login if you use public-key cryptography, it just won't allow you to login with the root password (even if it's correct.) It's good practice these days to use public keys for SSH anyway, so I wouldn't say this is much of a setback.

Re: All my servers have an 8 GB empty file on disk

#542

Post by herpderperator »

cellularmitosis wrote: post #519
herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

> So log in as root and fix the issue. Simple. "just fix the issue" often takes longer than 'rm spacer.img', which allows non-root processes to begin working immediately. In practice, the 5% root reserve is not really an effective substitute for OP's technique.

Running `tune2fs -m 0 /dev/sdaX` as root instantly makes the reserved blocks available. You could even just lower it by 1% which would be more than sufficient in the meantime: `tune2fs -m 4 /dev/sdaX`. When you've fixed the issue and freed up space, you can increase it to 5% again.

So why would you bother making a blank file? :)

Re: All my servers have an 8 GB empty file on disk

#543

Post by aitchnyu »

tehjoker wrote: post #141
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

In the Honda CBF125 group on Facebook, a fellow Indian shared a photo of his bike. A British guy asked what's the switch, he's never seen one before. Same bike, same country of origin, but only certain markets get the switch and the recessed panel.

Re: All my servers have an 8 GB empty file on disk

#544

Post by luckylion »

geocrasher wrote: post #503
dilyevsky wrote: post #430

Earlier quoted context omitted.

That’s not perfect world this is standard industry practice to have host level monitoring

Of course! But do you put all your trust in your monitoring, 100%? You've never had monitoring fail for any reason at all? You've never had a server fill up before you can respond to the alert? This 8gb file idea isn't to replace monitoring. It's to offer a quick stopgap solution so you can do things in a hurry and give yourself a little extra "out" when things go awry. Because believe me, they WILL go awry. And if y…

> But do you put all your trust in your monitoring, 100%?

Yes. If I didn't feel that I can trust it, I would get another solution.

> You've never had a server fill up before you can respond to the alert?

I have. With the proposed hack in this article: it would fill up even faster: by that amount of time it would take the problem to write 8gb of data.

> Because believe me, they WILL go awry.

In my experience: not in any way that this would help. If your disk fills up, it's either slow (and your monitoring alerts you days or at least hours before it's a problem) or it's really, really fast. In the latter case, it's much faster than you can jump on your computer, ssh into the machine and delete your spacer file.

Invest in better monitoring, that's much, much, much, much better than adding spacer files to fill up your disk or changing the wall clock to give you more time.

Re: All my servers have an 8 GB empty file on disk

#545

Post by luckylion »

gitgud wrote: post #539
t0mmel wrote: post #535

What I don’t understand about this approach is why you think it actually does anything for you ? What you do instead of this is to setup an alert to monitor disk space at the right threshold for you, and then have a contingency plan for how to add more space to your environment. It seems like you have sort of done that, but in this case you are actually allowing your system to get into a bad state before you react. P…

This is true, but alerts and monitoring software can fail in many ways, due to network issues, process crashes... Etc. Due to the simplicity of this solution, there's not much that can go wrong.... unless you can't actually access the server anymore to delete the file... I see this more as a second line of defence

If the monitoring fails to alert you, you don't get the alert that you're running out of disk space and you can't access the server to delete the spacer file. All you've done in that case is lower your disk space by 8gb and make your server fill up more quickly.

Re: All my servers have an 8 GB empty file on disk

#546

Post by gardaani »

ryandrake wrote: post #147

A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens." As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pe…

Early Symbian apps are an excellent example how to write apps so that they don't crash when storage or memory becomes full. They just show an error dialog and the user can still use the system to free storage or memory. Modern phone apps either crash or the entire phone crashes in similar situations.

Re: All my servers have an 8 GB empty file on disk

#547

Post by jordinebot »

This reminded me of that joke about two guys who meet in the middle of the Savanna. One is carrying a phone booth and the other one an anvil. So, the one with the anvil asks:

- Why do you carry a phone booth around? - Oh, you see, it's for the lions. If I see a lion, I drop the booth, step inside and I'm safe. What's with the anvil? - It's for the lions too. If I see a lion, I drop the anvil and I can run way faster!

Good trick anyway!

Re: All my servers have an 8 GB empty file on disk

#548

Post by kaba0 »

sgregnt wrote: post #461
lostlogin wrote: post #454

Earlier quoted context omitted.

> This is similar to how some government agencies retain their budgets. The non-government sector isn’t immune to this.

If whatever actions turn a non government entity into something inefficient, then the entity wont survive for long and will go out of business (or at least that's the hope of a competitive free market economy)

> competitive free market economy

It was not even believed by Adam Smith. He writes that it only works that way in a controlled environment. That’s why European countries usually rank higher in market freedom than the US, because we don’t have companies getting so cancerously big that they have very real effects on law making (how lobbying is legal is still beyond me)

Re: All my servers have an 8 GB empty file on disk

#549

Post by ianmcgowan »

crdrost wrote: post #500
Aeolun wrote: post #383

Earlier quoted context omitted.

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place. Definite case of misaligned incentives.

Yep! The problem happens when you divide the safety buffer up in the first place. Safety buffers demand to be shared, when one part does not use all of its safety margin you want to transfer that to another system. Another surprising place where this happens is project scheduling. We budget time for each individual step of a project based on our guess of a 90% or 95% success rate, then our "old-timers' experience" ki…

If you haven't already, check out the oft-referenced book "The Principles of Product Development Flow". Based on this comment you'd really like it.

http://www.startuplessonslearned.com/2009/07/principles-of-p...

Re: All my servers have an 8 GB empty file on disk

#550

Post by eevilspock »

ksm1717 wrote: post #504
c3534l wrote: post #467

Earlier quoted context omitted.

Tragedy of the Commons effects are well-established to exist both in economics and outside it (in ecology, for instance). You seem to be attempting to shoehorn some misguided political take into the situation, even though Tragedy of the Commons is a decent characterization of this particular social pathology.

Their point was that tragedy of the commons need not be a given anywhere we see tragedies or commons. Last paragraph is lofty but I think the whole idea is we have the cognitive ability to deliberately prove its not a natural law. Understanding sociology as ecology at human scale is core to libertarianism.

Yes. Thank you.

Though did you mean, "Understanding sociology as Darwinian ecology at human scale is core to libertarianism."? Because the notion that ecology is characterized only by "the law of the jungle" is also strongly debated. Even "the selfish gene" is debatable simplistic reductionism. Individuals aren't the only actors; there are higher order emergent entities, e.g. species and ecosystems, that also evolve to perpetuate themselves and flourish, much like our own bodies are cooperative and interdependent systems of cells (with native and foreign DNA, the latter existing primarily in our GI tract) that originally evolved as single-celled "selfish" organisms.

And as you point out, "we have the cognitive ability" that nature lacks. We can do at least as well.

As to "lofty", I agree. But let's consider other things that were once considered lofty if not insanity:

- in ancient Greece, that democracy should be extended beyond the aristocracy

- in Medieval Europe, that democracy should exist at all, that the divine right of kings should be seen as a scam

- in the 19th century United States, that democracy should include women and blacks

- in the 1970's United States, that lesbians, gays, bisexuals, transexuals and queers should be treated with the same dignity as straights, should be able to marry, adopt children and serve in the military. And that we stop using "he/him" by default as you just did because that is an artifact of patriarchy as well as outmoded thinking about even binary gender.

- in India today, that when a woman is raped, she should be protected by law and the male rapist should be punished, not the other way around. The same proposition if proposed in America or Europe not all that long ago.

- I can make a really long list but you get it :)

Post reply on HN