Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

ubicloud.com

Post reply on HN
Print view View original

71–80 of 117 posts

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#71

Post by cbozeman »

wink wrote: post #28

> One of the providers we like is Hetzner because of their affordable and reliable servers. > In the days that followed, the crash frequency increased. I don't find the article conclusive whether they would still call them reliable.

Hetzner's reliable... until they aren't.

Since they don't do any sort of monitoring on their bare metal servers at all, at least insofar as I can tell having been a customer of theirs for ten years, you don't know there's as problem until there's a problem, or unless you've got your own monitoring solution in place.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#72

Post by porridgeraisin »

radicality wrote: post #49
jeffbee wrote: post #25

Earlier quoted context omitted.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

The term PROCHOT just brought me back to vivid memories of debugging exactly that at Facebook a while ago. It was very non-obvious to debug since pretty much most emitted metrics, apart from mysterious errors/timeouts to our service, looked reasonable. Even the cpu usage and cpu temperature graphs looked normal since it was a bogus prochot and not actually a real thermal throttling

And it brought me back to memories of debugging that on my friends laptop.

It kept going to 400mhz.. i suspected throttling and we got it cleaned thermal paste replaced and all that.

Still throttled. We replaced the windows with linux since it was atleast a bit more usable

At the time I didn't know about PROCHOT. And my googling skills clearly weren't sufficient.

One fine day during lunch at a place on campus, Id read about BD_PROCHOT recently. So i wrote a script to msrprobe or whatever it was and disabled it. "Extended" the lifespan of the thing.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#73

Post by cbozeman »

indulona wrote: post #59

i am so glad my sign up process with hetzner failed when i was so dumb that i wanted to give them a chance even with the internet full of horrific stories of bad experiences from their customers. lucky me.

Hetzner is fine for what it is, you just need to know that it's all on you and only YOU.

YOU do the monitoring.

YOU do the troubleshooting.

YOU etc., etc.

If that doesn't appeal to you, or if you don't have the requisite knowledge, which I admit is fairly broad and encompassing, then it's not for you. For those of you that meet those checkboxes, they're a pretty amazing deal.

Where else could I get a 4c/8t CPU with 32 GB of RAM and four (4) 6TB disks for $38 a month? I really don't know of many places with that much hardware for that little cost. And yes, it's an Intel i7-3770, but I don't care. It's still a hell of a lot of hardware for not much price.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#74

Post by rat9988 »

Tijdreiziger wrote: post #66
radicality wrote: post #51

Earlier quoted context omitted.

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf. To check, run `cat /sys/devices/system/cpu/cpu /cpufreq/scaling_governor`. It should be `performance`. If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/c…

However, eco-friendly power modes can reduce electricity usage, so they can be friendlier for our climate. https://www.rvo.nl/onderwerpen/energie-besparen-de-industrie...

I'm not sure why you are downvoted. Is it wrong?

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#75

Post by redleader55 »

vitus wrote: post #4

> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…

Every rack in a data center has a power budget, which is actually constrained by how much heat the HVAC system can pull out of the DC, rather than how much power is available. Nevertheless it is limited per rack to ensure a few high power servers don't bring down a larger portion of the DC.

I don't know for sure how the limiting is done, but a simple circuit breaker like the ones we have in our houses would be a simple solution for it. That causes the rack to loose power when the circuit breaks, which is not ideal because you loose the whole rack and affect multiple customers.

Another option would be a current/power limiter[0], which would cause more problems because P = U * I. That would make the voltage (U) drop and then the whole system to be undervolted - weird glitches happen here and it's a common way to bypass various security measures in chips. For example, Raspberry Pi ran this challenge [1] to look for this kind of bugs and test how well their chips can handle attacks, including voltage attacks.

[0] - https://en.m.wikipedia.org/wiki/Current_limiting [1] - https://www.raspberrypi.com/news/security-through-transparen...

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#76

Post by fdr »

vednig wrote: post #60
eitland wrote: post #40

Earlier quoted context omitted.

They are in the early stages. I think the website said they recently raised 16 million euros (or dollars). Making investments into data centers and hardware could burn through that really quick in addition to needing more engineers. By using rented servers (and only renting them when a customer signs up) they avoid this problem.

understood, would love to know about it from founders tho, and what went through in their decision

GP is more or less correct.

Building and owning an institution that finances, racks, services, networks, and disposes of servers, both takes time and increases the commitment level. Hetzner is month to month, with a fixed overhead for fresh leasing of servers: the set-up fee.

This is a lot to administer when also building a software institution, and a business. It was not certain at the outset, for example, that the GitHub Actions Runner product would be as popular as it became. In its earliest form, it was partially an engineering test for our virtual machines, and we went around asking friendly contacts that we knew would report abnormalities to use it. There's another universe where it only went as far as an engineering test, and our utilization and revenue pattern (that is, utility to other people) is different.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#77

Post by kjellsbells »

Tijdreiziger wrote: post #66
radicality wrote: post #51

Earlier quoted context omitted.

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf. To check, run `cat /sys/devices/system/cpu/cpu /cpufreq/scaling_governor`. It should be `performance`. If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/c…

However, eco-friendly power modes can reduce electricity usage, so they can be friendlier for our climate. https://www.rvo.nl/onderwerpen/energie-besparen-de-industrie...

Yes, but the point is that the customer has the agency to decide.

If I rent a server I want to be able to run it to the maximum capacity, since I'm paying for all of it. It's dishonest to make me pay for X and give me The flip side is that the provider should be also offering more climate friendly, lower power options. I'll still want to run them to the max, but the total energy consumed would be less than before.

Also not forgetting that code efficiency matters if we want to get the max results for the minimum carbon spend. Another reason why giant web frameworks and bloated OSes depress me a little.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#78

Post by KennyBlanken »

jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

No? Maybe you cloud kids don't know how this stuff works, but unmanaged just means you get silicon-level access and remote KVM.

It's still the hosting company's responsibility to competently own, maintain, and repair the physical hardware. That includes monitoring. In the old days you had to run a script or install a package to hook into their monitoring....but with IPMI et al being standard they don't need anything from you to do their job.

The only time a hosting company should be hands-off is when they're just providing rack space, power, and data. Anything beyond that is between you and them in a contract/agreement.

Every time I hear Hetzner come up in the last few years it's been a story about them being incompetent. If they're not detecting things like CPU fan failures of their own hardware and they deployed new systems without properly testing them first, then that's just further evidence they're still slipping.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#79

Post by nobankai »

cbozeman wrote: post #73
indulona wrote: post #59

i am so glad my sign up process with hetzner failed when i was so dumb that i wanted to give them a chance even with the internet full of horrific stories of bad experiences from their customers. lucky me.

Hetzner is fine for what it is, you just need to know that it's all on you and only YOU . YOU do the monitoring. YOU do the troubleshooting. YOU etc., etc. If that doesn't appeal to you, or if you don't have the requisite knowledge, which I admit is fairly broad and encompassing, then it's not for you. For those of you that meet those checkboxes, they're a pretty amazing deal. Where else could I get a 4c/8t CPU with…

[flagged]

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#80

Post by immibis »

wink wrote: post #28

> One of the providers we like is Hetzner because of their affordable and reliable servers. > In the days that followed, the crash frequency increased. I don't find the article conclusive whether they would still call them reliable.

Seems like this problem, was unforeseeable, is isolated to a particular current-generation/model of server motherboards (AX2), and doesn't usually happen. I had an AX41* previously with no such problem, so it's not all AXes, just all current-generation AXes (which is all of the AXes they give to new customers, so that's no consolation).
Post reply on HN