The little ssh that (sometimes) couldn't
mina.naguib.ca
The little ssh that (sometimes) couldn't
1–10 of 66 posts
Re: The little ssh that (sometimes) couldn't
#2Weird connection problems like that sound like tcp timestamps breaking things. You can try turning it off across the board and see if your problems immediately clear up: http://prowiki.isc.upenn.edu/wiki/TCP_tuning_for_broken_fire...
Re: The little ssh that (sometimes) couldn't
#3Weird connection problems like that sound like tcp timestamps breaking things. You can try turning it off across the board and see if your problems immediately clear up: http://prowiki.isc.upenn.edu/wiki/TCP_tuning_for_broken_fire...
This was demonstrably link-level corruption, though, so that's irrelevant, right? That's about higher-level failures.
Re: The little ssh that (sometimes) couldn't
#4Awesome story.
Re: The little ssh that (sometimes) couldn't
#5Wonderful! Thank you, author and submitter both!
Re: The little ssh that (sometimes) couldn't
#6I love reading weird bug stories like this. Is there a place where lots of these types of stories are aggregated? Maybe a book about them?
Re: The little ssh that (sometimes) couldn't
#7Reminded me of this:
Re: The little ssh that (sometimes) couldn't
#8That is an awesome story. If you're in devops I would suggest you look at the sequence of events, especially the debugging decision tree. You can't always get access to all of the machines but you can create 'views' by going through them. Sort of like astronomers using a gravitational lens.
We had a similar issue at Blekko where a 10G switch we were using would not pass a certain bit pattern in a UDP packet fragment. Just vanished. Annoying as heck, the fix was to add random data to the packet on retries so that at least one datagram made it through intact.
Re: The little ssh that (sometimes) couldn't
#9This is why good DevOps people are worth their weight in platinum. As a developer who has done just enough administration to be dangerous, I can easily say that my job is always far more enjoyable when there are good DevOps folks around to keep my systems happy and shield me from the crazy place that is the Internet's wiring.
Re: The little ssh that (sometimes) couldn't
#10This is insane. The closest scenarios to this I've seen in my career:
1) A private frame relay network that one day stopped passing packets over a certain size. Worked around by lowering the MTU at both ends till I was able to convince the frame relay provider that yes, the problem was in their network. This was relatively straight-forward to diagnose, but it was still odd being able to ssh into a box, then have the connection hang once I did something that sent a full-size packet (cat a large file, ls -l in a big directory, etc).
2) A paging gateway program I wrote (email to SMS) that worked fine when testing on my Mac, but couldn't establish connections to a particular Verizon web site when I ran it from a Linux box. Turned out that the Linux TCP stack had ECN enabled and at the time the Verizon website was behind a buggy firewall that blocked any packets with ECN bits set.
3) A Solaris box that could randomly be connected to, but not always. Turned out someone had deleted its own MAC address from its ARP table (yes, you can do this with Solaris) so it wasn't replying to ARP packets for itself. As I recall, it could make outbound connections, and then you could connect to it from that same peer until the peer timed out the ARP entry. Then the peer couldn't reach the Solaris box again.
None of these are nearly as complex as the scenario in this story.