Fail-slow at scale: Evidence of hardware performance degradation in prod systems • Hacker News Board

Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems

#11

Post by gpapilion » Mon, Feb 26, 2018, 5:46 PM UTC

LeonM wrote: post #2

> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?

I previously had a vendor replace hundreds of systems due to a poor fan design where they mounted it under a spinning disk. It is very surprising how sensitive they are.

Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems

#12

Post by jebeng » Mon, Feb 26, 2018, 8:22 PM UTC

janci wrote: post #10

meuk wrote: post #7

Earlier quoted context omitted.

In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's. On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibr…

I remember how I didn't dare to touch the computer during burning other than move the mouse a few pixels here and there so it has it peace and the screensaver do not kick in. CDs were expensive and burnig was a hour-long process.

That upgrade path from 2x to 52x was so nice though.

Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems

#13

Post by xkgt » Tue, Feb 27, 2018, 2:47 PM UTC

This is a HN gem! Wonder how it didn't make it to first page.

I once had a mobile app perform very poorly only for large downloads on a particular SSID. No other applications had such issue. It turned out to be due to a firmware bug which scrapped TCP SACK field, but the problem manifested only for large data transfers over unreliable, high latency connections. Got us all chasing our tails for a while though...

Re: Fail-slow at scale: Evidence of hardware performance degradation in prod systems

#14

Post by riza_on » Thu, Mar 01, 2018, 8:36 PM UTC

The authors are still collecting more stories for their longer journal version of the paper. If your institution has 10+ stories to contribute and are interested to be part of the journal paper, please go ahead contact the first two authors of the original paper.