Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

ubicloud.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#51

Post by radicality »

andai wrote: post #38

> Hetzner didn’t confirm or deny the possibility of power limiting What are the consequences of power limiting? The article says it can cause hardware to degrade more quickly, why? Hetzner's lack of response here (and UbiCloud's measurements) seems to suggest they are indeed limiting power, since if they weren't doing it, they'd say so, right?

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf.

To check, run `cat /sys/devices/system/cpu/cpu/cpufreq/scaling_governor`. It should be `performance`.

If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/cpu/cpufreq/scaling_governor`. If your workload is cpu hungry this will help. It will revert on startup, so can make it stick, with some cron/systemd or whichever.

Of course if you are the one paying for power or it’s your own hardware, make your own judgement for the scaling governor. But if it’s a rented bare metal server, you do want `performance`.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#52

Post by greggyb »

dangoodmanUT wrote: post #50

is there a provider that's like bare metal, but would detect these kinds of things mostly automatic? E.g. faulty or constantly crashing hardware.

Managed servers: https://www.hetzner.com/managed-server/

There are also others, but Hetzner is under discussion here.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#53

Post by urbandw311er »

Would anybody with data center experience be able to hazard a guess on what type of commercial resolution Hetzner would have reached with the Motherboard supplier here? Would we assume all mobos replaced free of charge plus compensation?

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#54

Post by frenchtoast8 »

aduffy wrote: post #33
wink wrote: post #28

> One of the providers we like is Hetzner because of their affordable and reliable servers. > In the days that followed, the crash frequency increased. I don't find the article conclusive whether they would still call them reliable.

To their credit they actually fixed the problem. Good luck getting this level of support from any of the big 3 public cloud providers.

For example, AWS's Mac machines frequently run into hardware failures. My current job runs a measly 5 mac1.metal hosts for internal testing, and we experience hardware failures on these machines a few times a year. Doesn't sound like a lot, but these machines are almost always completely idle, and we almost never get host failures for Linux hosts. To make matters worse, sometimes a brand new instance needs replacement before it even comes up for the first time, which is annoying because you are billed a minimum of 24 hours for these instances. People have been complaining about this for years and seemingly nothing is being done about it.

https://www.reddit.com/r/aws/comments/131v8md/beware_of_brok...

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#55

Post by jeffbee »

tryauuum wrote: post #31
jeffbee wrote: post #25

Earlier quoted context omitted.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

in my (limited) experience this only happened with GIGABYTE servers very weird behavior, I'd prefer my servers to crash instead of lowering frequency to 400MHz.

> I'd prefer my servers to crash instead of lowering frequency to 400MHz.

100% agreed. There is nothing worse than a slow server in your fleet. This behavior reeks of "pet" thinking.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#56

Post by bayindirh »

Dell has this problem sometimes. I remember getting the first batch one of their older servers when they were new. We had to replace motherboards' I/O (rear) section because the servers lost some devices on that part (e.g.: Ethernet controllers, iDRAC, sometimes BIOS) for some time. After shaking out these problems, they ran for almost a decade.

We recently retired them because we worn down everything on these servers. From RAID cards to power regulators. Rebooting a perfectly running server due to a configuration change and losing the RAID card forever because electron migration erode a trace inside the RAID processor is a sobering experience.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#57

Post by formerly_proven »

tryauuum wrote: post #31
jeffbee wrote: post #25

Earlier quoted context omitted.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

in my (limited) experience this only happened with GIGABYTE servers very weird behavior, I'd prefer my servers to crash instead of lowering frequency to 400MHz.

Stuff like this just comes up from time to time as soon as you run a four digit and up number of systems.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#60

Post by vednig »

eitland wrote: post #40
vednig wrote: post #29

as a CI/CD provider wouldn't it benefit if Ubicloud had their own servers?

They are in the early stages. I think the website said they recently raised 16 million euros (or dollars). Making investments into data centers and hardware could burn through that really quick in addition to needing more engineers. By using rented servers (and only renting them when a customer signs up) they avoid this problem.

understood, would love to know about it from founders tho, and what went through in their decision
Post reply on HN