Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every time we lift a pallet from the shipping room, the server times out (2006)

reddit.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#11

Post by Dagwoodie »

I had a similar thing happen to a friend of the family whose business I provided IT services for. He had 2 servers and 2 rack mounted UPS's and a handful of other devices that all fit into the same rack. One day I needed to do some maintenance that required shutting one of his servers down. This was a backup server that wasn't actually used for anything as long as the primary was up and functional.

The problem was that as soon as I shut the backup server down, his entire network stopped working. I was trying to the maintenance over a VPN, so immediately I lost access. I don't really have a clever way of telling this right now, but after a lot of frustration trying to figure out the problem remotely (and wondering if someone had pwn'd his servers and was using the backup server to MITM all his traffic) I drove out there and noticed the problem right away. The UPS that the backup server was connected to was faulty, what was happening was that once the server was no longer pulling electricity from the faulty UPS, it failed to power the other equipment that was plugged into it, one of those things being a critical switch. As soon as the server was powered back on, all the other equipment attached to it immediately powered up too.

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#12

Post by jsmthrowaway »

tyingq wrote: post #4

Similar story: http://patrickthomson.tumblr.com/post/2499755681/the-best-de... Oh, and back when Sun Microsystems said random server crashes were due to alpha particles. http://web.archive.org/web/20020202013942/http://www.compute...

Speaking of Sun alumni, there's also Brendan Gregg's one weird trick for inducing disk latency (spinning disks hate him!):

https://youtu.be/tDacjrSCeq4

Side note: that scream makes a superb notification tone for people you don't like, or PagerDuty.

I also heard a good yarn once where a customer's DSL service inexplicably stopped working at exactly the same time every day, and after weeks of troubleshooting and changing absolutely everything, with pretty much everyone involved ready to give up, a senior network engineer sitting in the guy's living room happened to notice the street light directly outside turning on at the same time. Turns out the lamp was poorly shielded and throwing enough RF into the poor customer's apartment to light up Pittsburgh. Never heard what came of that, though I imagine that call to public works was interesting.

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#13

Post by yellowapple »

Dagwoodie wrote: post #11

I had a similar thing happen to a friend of the family whose business I provided IT services for. He had 2 servers and 2 rack mounted UPS's and a handful of other devices that all fit into the same rack. One day I needed to do some maintenance that required shutting one of his servers down. This was a backup server that wasn't actually used for anything as long as the primary was up and functional. The problem was th…

Sounds like one of those surge protectors that control the power to other devices based on whether or not a central device is powered on. Some powerbars for home theater setups have a similar feature; if you turn off the TV, all the other stuff turns off with it, and if you turn on the TV, everything turns back on.

Weird that a rackmount UPS would have that sort of feature, but hey, it's possible.

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#14

Post by drfuchs »

hprotagonist wrote: post #3
rawdan wrote: post #2

haha, talk about hard code debugging. I think this post perfectly illustrates the major difference between "theory & practice".

"in theory, there's no difference between theory and practice. "

... but in practice, there is. - Jan van de Snepscheut

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#16

Post by yellowapple »

tyingq wrote: post #4

Similar story: http://patrickthomson.tumblr.com/post/2499755681/the-best-de... Oh, and back when Sun Microsystems said random server crashes were due to alpha particles. http://web.archive.org/web/20020202013942/http://www.compute...

"We just don't think a Linux partition on a mainframe makes a lot of sense. It's kind of like having a trailer park in the back of your estate."

That gave me more than a few giggles.

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#19

Post by kuahyeow »

This reminds me of the time when I managed to diagnose a packet storm issue after two days of methodically excluding software then hardware followed by tracing each Ethernet cable to the switch. Turned out that the re-connection of the cables caused a network loop within a dumb switch, ala packet storm !

Re: Every time we lift a pallet from the shipping room, the server times out (2006)

#20

Post by robert_tweed »

noxin wrote: post #17

Stories like this always remind me of the car allergic to vanilla ice cream. http://www.cgl.uwaterloo.ca/smann/IceCream/humor.html

Not forgetting the classic "magic / more magic" switch:

http://catb.org/esr/jargon/html/magic-story.html

Post reply on HN