Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

ubicloud.com

Post reply on HN
Print view View original

31–40 of 117 posts

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#31

Post by tryauuum »

jeffbee wrote: post #25
jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

in my (limited) experience this only happened with GIGABYTE servers

very weird behavior, I'd prefer my servers to crash instead of lowering frequency to 400MHz.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#32

Post by dijit »

tryauuum wrote: post #31
jeffbee wrote: post #25

Earlier quoted context omitted.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

in my (limited) experience this only happened with GIGABYTE servers very weird behavior, I'd prefer my servers to crash instead of lowering frequency to 400MHz.

I've seen it on nearly every brand, I have some Lenovo Servers in the basement that also down-clock if both PSU's aren't installed.

I have alerts on PSU's and frequency for this reason.

The servers are so cheap that overcommitting them by double is still significantly cheaper than using cloud hosting, which tends to have the same issue only monitoring it is harder. Though most people using cloud seem to be happy not to know and it's been a known thing that there's a 5x variation between instances of the same size on AWS.: https://www.brendangregg.com/Slides/AWSreInvent2017_performa...

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#33

Post by aduffy »

wink wrote: post #28

> One of the providers we like is Hetzner because of their affordable and reliable servers. > In the days that followed, the crash frequency increased. I don't find the article conclusive whether they would still call them reliable.

To their credit they actually fixed the problem. Good luck getting this level of support from any of the big 3 public cloud providers.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#34

Post by rikafurude21 »

Similar thing happened to a AX102 I currently use, something related the network card which caused crashes. Thankfully hetzner support was helpful with replacement hardware. caused quite some grief but at least it was a good lesson in hardware troubleshooting. Worth it to me personally

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#35

Post by rat9988 »

axus wrote: post #13
pwmtr wrote: post #5

Earlier quoted context omitted.

Author of the blog post here. Yeah, this is generally a good practice. The silver lining is that our suffering helped uncover the underlying issue faster. :) This isn’t part of the blog post, but we also considered getting the servers and keeping them idle, without actual customer workload, for about a month in the future. This would be more expensive, but it could help identify potential issues without impacting our…

Customers are the best QA. And they pay you too, instead of the reverse!

I'm pretty sure they pay for QA. QA cannot always catch every possible bug.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#36

Post by fdr »

V__ wrote: post #2

> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.

It varies by system. As the legendary (to some) Kelly Johnson of the Skunk Works had as one of his main rules:

> The inspection system as currently used by the Skunk Works, which has been approved by both the Air Force and the Navy, meets the intent of existing military requirements and should be used on new projects. Push more basic inspection responsibility back to the subcontractors and vendors. Don't duplicate so much inspection.

But this will be the only and last time Ubicloud does not burn in a new model, or even tranches of purchases (I also work there...and am a founder).

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#37

Post by throwaway984393 »

jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

[dead]

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#38

Post by andai »

> Hetzner didn’t confirm or deny the possibility of power limiting

What are the consequences of power limiting? The article says it can cause hardware to degrade more quickly, why?

Hetzner's lack of response here (and UbiCloud's measurements) seems to suggest they are indeed limiting power, since if they weren't doing it, they'd say so, right?

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#39

Post by nik736 »

Most other AX models (AX42, AX52 and AX102) also have serious reliability issues, where they will fail after some months. They are based on a faulty motherboard. Hetzner has to replace most, if not all, motherboards for servers built before a certain date over the next 12 months [0]

[0] https://docs.hetzner.com/robot/dedicated-server/general-info...

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#40

Post by eitland »

vednig wrote: post #29

as a CI/CD provider wouldn't it benefit if Ubicloud had their own servers?

They are in the early stages.

I think the website said they recently raised 16 million euros (or dollars).

Making investments into data centers and hardware could burn through that really quick in addition to needing more engineers.

By using rented servers (and only renting them when a customer signs up) they avoid this problem.

Post reply on HN