Earlier quoted context omitted.
Customers are the best QA. And they pay you too, instead of the reverse!
I'm pretty sure they pay for QA. QA cannot always catch every possible bug.
these crashes should have been caught easily
Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
41–50 of 117 posts
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#42> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…
At the time of our investigation, we found few articles supporting that power caps could potentially cause hardware degradation, though I don't have the exact sources at hand. I see the child comment shared one example, and after some searching, I found a few more sources [1], [2]. That said, I'm not an electronics engineer, so my understanding might not be entirely accurate. It’s possible that the degradation was ca…
The power used by a computer isn't limited by giving it less voltage/current than it should have - if it was, the CPU would crash almost immediately. It's done by reducing the CPU's clock rate until the power it naturally consumes is less than the power limit.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#43Anyone got experience with Ubicloud's OpenStack stack?
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#44At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.
[deleted]
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#45I’d like to see what cpu governor is running on those systems before assuming a power cap is in place. Lots of defaults installs of Linux ship with the power save governor running which is going to limit your max frequencies and through that the max power you can hit.
schedutil on mine scheduled for mainboard replacement
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#46Similar thing happened to a AX102 I currently use, something related the network card which caused crashes. Thankfully hetzner support was helpful with replacement hardware. caused quite some grief but at least it was a good lesson in hardware troubleshooting. Worth it to me personally
Yep same here. AX102 crashes with almost no load, nothing in the logs, won't come on. Hetzner looked at it multiple times and found either nothing or replaced cpu paste or a PSU connector. I migrated to AX162 and so far so good
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#47> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…
One possibility is that at lower power settings, the CPUs don't get as hot, which means the fans don't spin up as much, which can mean that other components also get less airflow and then get hotter than they would otherwise. The fix for this is usually to monitor the temperature of those other components and include that as an input to the fan speed algorithm. No idea if that's what's actually going on here though.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#48> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly.
This was something I hadn't heard before, & a surprise to me.
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#49At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.
I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.
The term PROCHOT just brought me back to vivid memories of debugging exactly that at Facebook a while ago.
It was very non-obvious to debug since pretty much most emitted metrics, apart from mysterious errors/timeouts to our service, looked reasonable. Even the cpu usage and cpu temperature graphs looked normal since it was a bogus prochot and not actually a real thermal throttling
Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode
#50is there a provider that's like bare metal, but would detect these kinds of things mostly automatic? E.g. faulty or constantly crashing hardware.