Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

ubicloud.com

Post reply on HN
Print view View original

21–30 of 117 posts

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#21

Post by esafak »

h1fra wrote: post #19
esafak wrote: post #10

Earlier quoted context omitted.

GitHub is looking to add this feature to dependabot: https://github.com/dependabot/dependabot-core/issues/3651

In theory, that works in practice nope. You get a random update with a possible bug inside that is only fixed by a new version that you won't get until later. The other strategy is to wait for a package to be fully stable (no update), and in that case, some packages that receive daily/weekly updates are never updated

It does help, because major version updates are more likely to cause breakage than minor ones, so you benefit if you wait for a few minor version updates. That is not to say minor versions can't introduce bugs.

Windows is a well-known example; people used to wait for a service pack or two before upgrading.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#22

Post by ajmurmann »

esafak wrote: post #21
h1fra wrote: post #19

Earlier quoted context omitted.

In theory, that works in practice nope. You get a random update with a possible bug inside that is only fixed by a new version that you won't get until later. The other strategy is to wait for a package to be fully stable (no update), and in that case, some packages that receive daily/weekly updates are never updated

It does help, because major version updates are more likely to cause breakage than minor ones, so you benefit if you wait for a few minor version updates. That is not to say minor versions can't introduce bugs. Windows is a well-known example; people used to wait for a service pack or two before upgrading.

We could even wait for a patch version or the minor being out a certain amount of time. For a major I'd wait even longer and potentially for a second patch.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#23

Post by pwmtr »

vitus wrote: post #4

> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…

At the time of our investigation, we found few articles supporting that power caps could potentially cause hardware degradation, though I don't have the exact sources at hand. I see the child comment shared one example, and after some searching, I found a few more sources [1], [2].

That said, I'm not an electronics engineer, so my understanding might not be entirely accurate. It’s possible that the degradation was caused by power fluctuations rather than the power cap itself, or perhaps another factor was at play.

[1] https://electronics.stackexchange.com/questions/65837/can-el... [2] https://superuser.com/questions/1202062/what-happens-when-ha...

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#24

Post by Cthulhu_ »

esafak wrote: post #21
h1fra wrote: post #19

Earlier quoted context omitted.

In theory, that works in practice nope. You get a random update with a possible bug inside that is only fixed by a new version that you won't get until later. The other strategy is to wait for a package to be fully stable (no update), and in that case, some packages that receive daily/weekly updates are never updated

It does help, because major version updates are more likely to cause breakage than minor ones, so you benefit if you wait for a few minor version updates. That is not to say minor versions can't introduce bugs. Windows is a well-known example; people used to wait for a service pack or two before upgrading.

And then they went towards a more evergreen update strategy, causing some major outages when some releases caused issues.

I mean evergreen releases make sense imo, as the overhead of maintaining older versions for a long time is huge, but you need to have canary releases, monitoring, and gradual rollout plans; for something like Windows, this should be done with a lot of care. Even a 1% release rate will affect hundreds of thousands if not millions of systems.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#25

Post by jeffbee »

jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#26

Post by [deleted] »

Cthulhu_ wrote: post #24
esafak wrote: post #21

Earlier quoted context omitted.

It does help, because major version updates are more likely to cause breakage than minor ones, so you benefit if you wait for a few minor version updates. That is not to say minor versions can't introduce bugs. Windows is a well-known example; people used to wait for a service pack or two before upgrading.

And then they went towards a more evergreen update strategy, causing some major outages when some releases caused issues. I mean evergreen releases make sense imo, as the overhead of maintaining older versions for a long time is huge, but you need to have canary releases, monitoring, and gradual rollout plans; for something like Windows, this should be done with a lot of care. Even a 1% release rate will affect hundr…

[deleted]

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#27

Post by scottcha »

I’d like to see what cpu governor is running on those systems before assuming a power cap is in place. Lots of defaults installs of Linux ship with the power save governor running which is going to limit your max frequencies and through that the max power you can hit.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#30

Post by nickcw »

vitus wrote: post #4

> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…

Power = volts * amps

Volts is as supplied by the utility company.

Amps are monitored per rack and the usual data centre response to going over an amp limit is that a fuse blows or the data centre asks you for more money!

The only way you can decrease power used by a server is by throttling the CPUs.

The normal way of throttling CPUs is via the OS which requires cooperation.

I speculate this is possible via the lights out base band controller (which doesn't need the os to be involved), but I'm pretty sure you'd see that in /sys if it was.

Post reply on HN