Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

ubicloud.com

Post reply on HN
Print view View original

81–90 of 117 posts

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#81

Post by immibis »

vednig wrote: post #29

as a CI/CD provider wouldn't it benefit if Ubicloud had their own servers?

Depends how many they need and how much control. Do they want to be a server company or an adapting-servers-to-run-your-CI/CD company or both? You can extract value from both parts of the equation, but theoretical economics tells us you can get the most value for the least effort by doing more of what you're best at and paying someone else to do what they're best at, rather than doing everything mediocrely yourself.

Sometimes that other company isn't actually very good and you can increase value by insourcing their part of your operation. But you can't assume that is always the case. It wouldn't have solved this particular problem - I think we can safely guess that your chance of getting a batch of faulty motherboards is at least as high as Hetzner's chance.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#82

Post by crishoj »

pwmtr wrote: post #5
V__ wrote: post #2

> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.

Author of the blog post here. Yeah, this is generally a good practice. The silver lining is that our suffering helped uncover the underlying issue faster. :) This isn’t part of the blog post, but we also considered getting the servers and keeping them idle, without actual customer workload, for about a month in the future. This would be more expensive, but it could help identify potential issues without impacting our…

Were you able to identify the manufacturer and model/revision of the failing motherboards? This would be extremely helpful when shopping for seconds hand servers.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#83

Post by anarazel »

chpatrick wrote: post #61
radicality wrote: post #51

Earlier quoted context omitted.

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf. To check, run `cat /sys/devices/system/cpu/cpu /cpufreq/scaling_governor`. It should be `performance`. If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/c…

Is there any downside to ondemand? If your servers aren't running at 100% then there's no point wasting watts, even if you aren't paying for them, right?

It performs terrible if you have an intermittent workload. Like e.g. a request response workload where request processing is cheap (so that the time to increase the frequency matters). I've seen cases it's a more than 2x request latency increase.

It can be pretty annoying, because it means that systems can perform better under higher load and that you get drastically different latency depending on whether a request is scheduled on a core that just processed another request (already at high freq) or one that was idle.

And because the frequency control isn't fun enough, this behavior also exists with cpu idle states. Even at high frequency Linux can enter idle states...

I've debugged several cases where this set of issues has caused unintuitive behavior. E.g.

a) switching to a more powerful servers drastically increased latency

b) optimized code resulting in higher latency / lower throughout because that provided enough idle cycles for a deeper idle time between requests

c) slightly increased IO latency leading to significantly worse overall performance, due to the IO getting long though to clock down

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#84

Post by marcusb »

KennyBlanken wrote: post #78
jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

No? Maybe you cloud kids don't know how this stuff works, but unmanaged just means you get silicon-level access and remote KVM. It's still the hosting company's responsibility to competently own, maintain, and repair the physical hardware. That includes monitoring. In the old days you had to run a script or install a package to hook into their monitoring....but with IPMI et al being standard they don't need anything…

> No? Maybe you cloud kids don't know how this stuff works, but unmanaged just means you get silicon-level access and remote KVM.

That's one way it can work. There are a great many hosted server options out there from fully managed to fully unmanaged with price points to match. Selling a cheap server under the conditions "call us when it breaks" is a perfectly reasonable offering.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#85

Post by wmf »

vitus wrote: post #4

> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…

Expert in server power management here. Your intuition is right and the comments/links to the contrary are wrong. Undervolting is unreliable but let's be clear: no one is undervolting servers. I don't even know if it's possible. Power limiting (e.g. RAPL) is completely safe to use because it keeps voltage, frequency, temperature, fan speed, etc within safe bounds.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#86

Post by trod1234 »

It would have been nice if they linked to the power metrics for the new servers.

I think it would be amusing if it turns out they just raised the power limits for those servers not showing the problem up to base that was originally advertised.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#87

Post by wmf »

urbandw311er wrote: post #53

Would anybody with data center experience be able to hazard a guess on what type of commercial resolution Hetzner would have reached with the Motherboard supplier here? Would we assume all mobos replaced free of charge plus compensation?

When you buy name-brand servers you'll definitely get any faulty hardware replaced. Compensation would only happen if you negotiated for that and you'd have to pay extra. You're probably better off buying some kind of business interruption insurance instead of trying to get vendors to pay you for downtime (even if it is their fault).

Hetzner is not a normal customer though. As part of their extreme cost optimization they probably buy the cheapest components available and they might even negotiate lower prices in exchange for no warranty. In that case they would have to buy replacement motherboards.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#88

Post by immibis »

redleader55 wrote: post #75
vitus wrote: post #4

> To increase the number of machines under power constraints, data center operators usually cap power use per machine. However, this can cause motherboards to degrade more quickly. Can anyone elaborate on this point? This is counter to my intuition (and in fact, what I saw upon a cursory search), which is that power capping should prolong the useful lifetime of various components. The only search results I found that…

Every rack in a data center has a power budget, which is actually constrained by how much heat the HVAC system can pull out of the DC, rather than how much power is available. Nevertheless it is limited per rack to ensure a few high power servers don't bring down a larger portion of the DC. I don't know for sure how the limiting is done, but a simple circuit breaker like the ones we have in our houses would be a simp…

Computers implement power limits by reducing their own speed until their power consumption falls under the limit. There's no risk of damage and it should actually extend the lifetime due to less heat, as well as increasing the efficiency (computation per watt).

No idea what the article is talking about with the damage. Computers like to run slow when possible. There's basically no downside except they take longer to do things.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#89

Post by PinkSheep »

radicality wrote: post #51
andai wrote: post #38

> Hetzner didn’t confirm or deny the possibility of power limiting What are the consequences of power limiting? The article says it can cause hardware to degrade more quickly, why? Hetzner's lack of response here (and UbiCloud's measurements) seems to suggest they are indeed limiting power, since if they weren't doing it, they'd say so, right?

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf. To check, run `cat /sys/devices/system/cpu/cpu /cpufreq/scaling_governor`. It should be `performance`. If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/c…

You can tune the ondemand (or any other) governor first to ramp up faster and clock down slower. "performance" should be seen as the nuclear option.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#90

Post by esafak »

TZubiri wrote: post #65
esafak wrote: post #10

Earlier quoted context omitted.

GitHub is looking to add this feature to dependabot: https://github.com/dependabot/dependabot-core/issues/3651

Being so deep into dependencies that you have to find more dependencies and features to make your dependency less of a clusterfuck is sad.

Are you referring to dependabot? You are free to update your dependencies manually.
Post reply on HN