Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

ubicloud.com

Post reply on HN
Print view View original

61–70 of 117 posts

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#61

Post by chpatrick »

radicality wrote: post #51
andai wrote: post #38

> Hetzner didn’t confirm or deny the possibility of power limiting What are the consequences of power limiting? The article says it can cause hardware to degrade more quickly, why? Hetzner's lack of response here (and UbiCloud's measurements) seems to suggest they are indeed limiting power, since if they weren't doing it, they'd say so, right?

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf. To check, run `cat /sys/devices/system/cpu/cpu /cpufreq/scaling_governor`. It should be `performance`. If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/c…

Is there any downside to ondemand? If your servers aren't running at 100% then there's no point wasting watts, even if you aren't paying for them, right?

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#64

Post by TZubiri »

jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

I'm heavily against both relying on free dependencies and going for the cheapest option.

If you can't put yourself in the shoes for a second when evaluating a purchase and you just braindead try to make cost go lower and income go higher, your ngmi except in shady sales businesses.

Server hardware is incredibly cheap, if you are somewhat of a competent programmer you can handle most programs in a single server or even a virtual machine. Just give them a little bit of margin and pay 50$/mo instead of 25$/mo, it's not even enough to guarantee they won't go broke or make you a valuable customer, you'll still be banking on whales to make the whole thing profitable.

Also, if your business is in the US, find a US host ffs.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#65

Post by TZubiri »

esafak wrote: post #10
V__ wrote: post #2

> Looking back, waiting six months could have helped us avoid many issues. Early adopters usually find problems that get fixed later. This is really good advice and what I'm following for all systems which need to be stable. If there aren't any security issues, I either wait a few months or keep one or two versions behind.

GitHub is looking to add this feature to dependabot: https://github.com/dependabot/dependabot-core/issues/3651

Being so deep into dependencies that you have to find more dependencies and features to make your dependency less of a clusterfuck is sad.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#66

Post by Tijdreiziger »

radicality wrote: post #51
andai wrote: post #38

> Hetzner didn’t confirm or deny the possibility of power limiting What are the consequences of power limiting? The article says it can cause hardware to degrade more quickly, why? Hetzner's lack of response here (and UbiCloud's measurements) seems to suggest they are indeed limiting power, since if they weren't doing it, they'd say so, right?

Related and perhaps useful: I’ve seen this in multiple cloud offerings already, where the cpu scaling governor is set to some eco-friendly value, in benefit to the cloud provider and in zero benefit to you and much reduced peak cpu perf. To check, run `cat /sys/devices/system/cpu/cpu /cpufreq/scaling_governor`. It should be `performance`. If it’s not, set it with `echo performance | sudo tee /sys/devices/system/cpu/c…

However, eco-friendly power modes can reduce electricity usage, so they can be friendlier for our climate.

https://www.rvo.nl/onderwerpen/energie-besparen-de-industrie...

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#67

Post by gtirloni »

fdr wrote: post #58
gtirloni wrote: post #43

Anyone got experience with Ubicloud's OpenStack stack?

Ubicloud does not have an OpenStack dependency.

Thanks, I was under the impression it did but re-reading the posts I see it's not the case.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#68

Post by Tijdreiziger »

greggyb wrote: post #52
dangoodmanUT wrote: post #50

is there a provider that's like bare metal, but would detect these kinds of things mostly automatic? E.g. faulty or constantly crashing hardware.

Managed servers: https://www.hetzner.com/managed-server/ There are also others, but Hetzner is under discussion here.

Managed servers are quite a different product, closer to ‘old-school’ shared webhosting.

You don’t get root access, but you do get a preinstalled LAMP stack and a web UI for management.

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#69

Post by merb »

bayindirh wrote: post #56

Dell has this problem sometimes. I remember getting the first batch one of their older servers when they were new. We had to replace motherboards' I/O (rear) section because the servers lost some devices on that part (e.g.: Ethernet controllers, iDRAC, sometimes BIOS) for some time. After shaking out these problems, they ran for almost a decade. We recently retired them because we worn down everything on these server…

Dell has tons of issues. A faulty mini board of the front led can actually stop the server from booting/running at all (even drac will be dead)

Re: Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

#70

Post by bityard »

jeffbee wrote: post #25
jonatron wrote: post #18

At a previous company, devops would regularly find CPU fan failures on Hetzner. That's in addition to the usual expected HD/SSD failures. You've got to do your own monitoring, it's one of the reasons why unmanaged servers are cheaper than cloud instances.

I regularly find broken thermal solutions in azure and when I worked at Google it was also a low-level but constant irritant. When I joined Dropbox I said to my team on my first day that I could find a machine in their fleet running at 400MHz, and I was right: a bogus redundant PSU controller was asserting PROCHOT. These things happen whenever you have a lot of machines.

A laptop that I had would assert PROCHOT if it didn't like the power supply you plugged into it. It actually took an embarrassing amount of time for me to notice that this is what was causing Slack to be inexplicably slower at my desk than when I was out working in a common area in the building.
Post reply on HN