Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
1–10 of 117 posts
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#2> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later.
This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#3[deleted]
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#4> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly.
Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components.
The only search results I found that claimed otherwise were indicating that if you're running into thermal throttling, then higher operating temperatures can cause components (e.g. capacitors) to degrade faster. But that's expressly not the case in the article, which looked at various temperature sensors.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#5> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.
Author of the blog post here.
Yeah, this is generally a good practice. The silver lining is that our suffering helped uncover the underlying issue faster. :)
This isn’t part of the blog post, but we also considered getting the servers and keeping them idle, without actual customer workload, for about a month in the future. This would be more expensive, but it could help identify potential issues without impacting our users. In our case, the crashes started three weeks after we deployed our first AX162 server, so we need at least a month (or maybe even longer) as a buffer period.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#6[dead]
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#7> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.
This is a wildly successfully pattern in nature, the old using the young and inexperienced, as enthusiastic test units.
In the wild for example in Forrest, old boars give safety squeaks to send the younglings ahead into a clearing they do not trust. The equivalent to that- would be to write a tech-blog entry that hypes up a technology that is not yet production ready.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#8> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.
Author of the blog post here. Yeah, this is generally a good practice. The silver lining is that our suffering helped uncover the underlying issue faster. :) This isn’t part of the blog post, but we also considered getting the servers and keeping them idle, without actual customer workload, for about a month in the future. This would be more expensive, but it could help identify potential issues without impacting our…
>The silver lining is that our suffering helped uncover the underlying issue faster.
Did you actually uncover the true root cause? Or did they finally uncap the power consumption without telling you, just as they neither confirmed nor denied having limited it?
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#9> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…
Yep, that's weird, I've always read that high power/temp can degrade electronics way faster. Any EE can shed a light here?
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#10> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.
GitHub is looking to add this feature to dependabot: https://github.com/dependabot/dependabot-core/issues/3651