Live data from Hacker News

The little ssh that (sometimes) couldn't

mina.naguib.ca

11–20 of 66 posts

Re: The little ssh that (sometimes) couldn't

#11

I love reading weird bug stories like this. Is there a place where lots of these types of stories are aggregated? Maybe a book about them?

Does it count if they're weird misconfigurations which may or may not have been set by a human?

http://rachelbythebay.com/w/2011/08/16/window/ http://rachelbythebay.com/w/2011/07/02/ninja/ http://rachelbythebay.com/w/2012/01/08/blackhole/ http://rachelbythebay.com/w/2012/01/18/alertstorm/

There are many more. Unless specifically noted, they all happened to me at some point.

Re: The little ssh that (sometimes) couldn't

#12
I'm curious how the kernel was able to diagnose that a single one bit always being fixed on the 15th of 16th bytes in the packet was corruption. That sounds like some intense algorithmic profiling especially if its being applied to every packet.

Re: The little ssh that (sometimes) couldn't

#13

I love reading weird bug stories like this. Is there a place where lots of these types of stories are aggregated? Maybe a book about them?

Does it count if they're weird misconfigurations which may or may not have been set by a human? http://rachelbythebay.com/w/2011/08/16/window/ http://rachelbythebay.com/w/2011/07/02/ninja/ http://rachelbythebay.com/w/2012/01/08/blackhole/ http://rachelbythebay.com/w/2012/01/18/alertstorm/ There are many more. Unless specifically noted, they all happened to me at some point.

Thanks!

Re: The little ssh that (sometimes) couldn't

#14
post #12

I'm curious how the kernel was able to diagnose that a single one bit always being fixed on the 15th of 16th bytes in the packet was corruption. That sounds like some intense algorithmic profiling especially if its being applied to every packet.

The kernel is rejecting packets that fail the checksum verification. It just happens that the source of the bit errors was behaving in this predictable way.

Re: The little ssh that (sometimes) couldn't

#17
Reminds me of a problem I had with a T1 circuit corrupting packets.

Shortly after bringing up a second T1 into a remote location we discovered that some web pages would show broken JPG images at the remote site.

Some troubleshooting revealed that this only happened when traffic was routed over the new T1. The old T1 worked just fine. Pings, and other IP traffic seemed to work over either line but we kept seeing the broken image icon for some reason when traffic came over the new T1.

We tried several times to confirm with the telco that the T1 was provisioned correctly and that our equipment matched those telco parameters. Still had some mangled bits going over that new T1.

Finally had the telco check the parameters over every span in the new (long-distance) T1 circuit and they eventually found one segment that was configured for AMI instead of B8ZS (if I can remember correctly, certainly it was a misconfigured segment though).

The net result is that certain user-data patterns that didn't include sufficient 0/1 transitions would lead to loss of clock synchronization over that segment and corrupted packets. Those patterns were most likely to occur in JPGs.

Once they corrected the parameters on that segment, everything worked as expected.

Quite a bit of head scratching with that one and lots of frustration as the layer-1 telco culture just couldn't comprehend that layer-2/3 Internet folks could accurately diagnose problems with their layer-1 network.

Re: The little ssh that (sometimes) couldn't

#18
post #12

I'm curious how the kernel was able to diagnose that a single one bit always being fixed on the 15th of 16th bytes in the packet was corruption. That sounds like some intense algorithmic profiling especially if its being applied to every packet.

A (very attentive) human spotted that, not the kernel. The kernel just computed the checksum and silently discarded the packets as corrupt.
Post reply on HN