Fail-slow at scale: Evidence of hardware performance degradation in prod systems
1–10 of 14 posts
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#2> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance.
I've never heard of such thing, can vibrations from a fan really influence disk performance?
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#3> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
Yes. Loud noises and even yelling degrade disk performance as well.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#4> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
Yes. Mechanical disks are very sensitive to vibrations.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#5I had a large scale issue caused by a bug in the fan control system reading a temp that was deadlocked. This caused cpu performance to tank, since the cpu was thermal throttled.
To make matters worse the environmental sensors reported no issues, because they were all frozen at a single reading as well.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#6Blue screen errors are most often caused by RAM issues. I remember very well that I was facing blue screen error at my work once, & upon searching I found that all I need to do is remove the RAM, clean and dust it, and put it back into the same or another RAM socket.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#7> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's.
On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibrations, I think.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#8> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
Yes. Loud noises and even yelling degrade disk performance as well. https://www.youtube.com/watch?v=tDacjrSCeq4
This is one of the classics! I love it. Screaming at a HDD.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#9> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
Yes. Loud noises and even yelling degrade disk performance as well. https://www.youtube.com/watch?v=tDacjrSCeq4
I started screaming in my cubicle watching this.
Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems
#10> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's. On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibr…
I remember how I didn't dare to touch the computer during burning other than move the mouse a few pixels here and there so it has it peace and the screensaver do not kick in. CDs were expensive and burnig was a hour-long process.