Well, while we're at it, here's my crazy MTU-related war story, although not as crazy as that one! I was troubleshooting with a user of an audio streaming application running over a LAN. The user could stream classical music but not rock music. Seriously. Classical was fine, but when streaming rock, the connection would drop after a few minutes. The application took chunks of audio, compressed them with a lossless co…
The little ssh that (sometimes) couldn't
51–60 of 66 posts
Re: The little ssh that (sometimes) couldn't
#52Reminded me of this: http://www.ibiblio.org/harris/500milemail.html
The numbers in this story do not add up. For a connection to be made within 3ms, there needs to be a round-trip within that time, which reduces the maximum possible radius by half.
Re: The little ssh that (sometimes) couldn't
#53Earlier quoted context omitted.
The numbers in this story do not add up. For a connection to be made within 3ms, there needs to be a round-trip within that time, which reduces the maximum possible radius by half.
Also, don't both photons in fiber and electrons in copper wire travel around 60% of the speed of light?
For electricity: Paraphrasing https://en.wikipedia.org/wiki/Speed_of_electricity
The 'Current' travels along a wire at anything between 97% and 60% of c (depending on insulation; more insulation == slower).
Individual electrons in copper wire travel much more slowly. The 'drift velocity' is roughly proportional to the voltage; for low voltage DC it is the order of millimeters per hour.
For AC voltage individual electrons don't have any net movement since they're oscillating back and forth with the alternating current.
Re: The little ssh that (sometimes) couldn't
#54About 3 visits in a row I went to look at problems (core dumps or errors) that the customer could reproduce at will, only for them to be unable to replicate the problem with me present on site.
I sat at one customer (in sunny Minneapolis) for 2 hours in the morning with the customer getting increasingly baffled as to why he couldn't get it to fail; it had been happily failing for him the previous evening when I was talking to him on the 'phone. We gave up and went for lunch (mmm, Khan's Mongolian Barbeque). A colleague of his called him midway through lunch to tell him that the software was failing again. Excellent I thought, we'll finally get to the bottom of it. Back to their office and ... no replication; it was working fine.
As a joke I said I should leave a clump of my hair taped to the side of the E450 it was running on. The customer took me up on that offer and, as far as I know (definitely for a few years at least), the software ran flawlessly at that customer.
It's the closest I've got to a "'more magic' switch" story of my own.
Re: The little ssh that (sometimes) couldn't
#55I suspected the VM code at the time, but it is very likely that my packets had to go through the same router (geography would support this).
I'm so glad somebody debugged this problem. Also, I'm quite glad that at least this time I'm not the only person with a weird issue (I have a knack for breaking things).
Re: The little ssh that (sometimes) couldn't
#56Well, while we're at it, here's my crazy MTU-related war story, although not as crazy as that one! I was troubleshooting with a user of an audio streaming application running over a LAN. The user could stream classical music but not rock music. Seriously. Classical was fine, but when streaming rock, the connection would drop after a few minutes. The application took chunks of audio, compressed them with a lossless co…
But this one is different -- it's not just the packet size that is the problem, it's that certain packets above a certain size get corrupted. Much more difficult to trace, so kudos to the authors. In fact, it seems I was bitten by the exact same problem, but could not trace it down, as I checked and ICMP packets of various sizes passed OK.
Re: The little ssh that (sometimes) couldn't
#57I've been wondering about something not entirely unrelated we see sporadically from a small but widespread number of users. We serve deep zoom images and the client appears to run normally but sends malformed image tile requests - e.g. in the URLs "service" is consistently garbled as "s/rvice", "dzi" as "d/i". I've seen this from IPs on every continent and user agents for most common browsers as well as both iOS and Android. My current theory is that it's some sort of tampering net filter as a fair number of the IPs have reverse DNS / Whois info suggesting educational institutions but have thus far failed to confirm this, particularly since none of the users have contacted us.
Re: The little ssh that (sometimes) couldn't
#58This is insane. The closest scenarios to this I've seen in my career: 1) A private frame relay network that one day stopped passing packets over a certain size. Worked around by lowering the MTU at both ends till I was able to convince the frame relay provider that yes, the problem was in their network. This was relatively straight-forward to diagnose, but it was still odd being able to ssh into a box, then have the…
> someone had deleted its own MAC address from its ARP table blink Two questions: - Is there ever a valid reason to do this? - How do you attain the skills required to do this while not also learning not to?
Upvoted for being one of the greatest ways I've ever seen to put what is a VERY common problem. I will quote this mercilessly in the future, if I may. Thanks.
Re: The little ssh that (sometimes) couldn't
#59A little background... I was brought up in the network ranks, I worked as a network / sys admin in high school, ended up working for an ISP as a junior network engineer in college (while I went to college at one of the first Cisco NetAcad baccalaureate programs - which was a combo of network study and Cisco curriculum and certifications) and have gone on to work in every major vertical since then for the past 10+ years; government, finance, healthcare, retail, telecomm, etc. I always tell clients and potential employers that having a network background generally gives me somewhat of an edge in the industry I primarily focus on: security, and I generally will study and take Juniper & Cisco tests and work on labs just to stay current. Most software devs and security folks I've run into (keep in mind there are a lot of really good folks who have a better grasp on network than a lot of seasoned engineers do) are generally overzealous in the thought that they truly do understand IP from a debugging and troubleshooting standpoint.
Case in point: I interviewed for a "Network Architect" position with a very well known online backup company (think top 4). The interview was the most bizarre I've ever had, not that it spanned more than 5 interviews, but that every time they positioned a complex network problem it was generally solvable within 5 to 10 minutes of pointed questions. The software dev who was interviewing me was baffled by how I came to a reasonable solution that took them over a week, in some cases, that quickly - and it was pretty simple in the fact that 1) I've seen something similar and 2) that's what I studied and still have a passion for over the course of 20+ years (when I found the Internet in 1991).
Most of the time when I run across a "magical" problem it's because someone hasn't looked at it from L1 up. As this article showcases you generally have two generic stack angles to approach it from - application back down to physical, or the inverse. Having been in network support - by the time you get a problem like this it's often so distorted with crazy outliers that really have nothing to do with the problem your best bet is to start from that L1 and go back up through the stack. Reading into the problem the author describes I think there were some key data that was missed and/or misinterpreted. There most surely would have been key indicators in TCP checksum errors and it was glossed over pretty lightly in the explanation - but it's interesting that those items of interest are often cast aside when digging into something like this. Nobody in this thread has indicated where a bit error test or even something as simple as iperf, or similar, would have been able to more accurately showcase/reproduce the problematic network condition.
But back to the labels remark - I don't believe, as some people have said, that this is a DevOps role largely. I don't mean to cut down on DevOps folks because I think, at some level, if you're a jack-of-all in any org then that's your role, it is what it is. However, this would be a problem most suited towards a professional network engineer - and you don't see much of that need in the startup space until people get into dealing with actual colo / DC type environments, otherwise it's often very simple and not architected with significant depth or specific use cases.
Long story short: network professionals are worth the money in the case of design, build, fix of potentially issues that may seem complex to others, but can be solved or found in minutes when you know what you're looking at. That being said, I'm impressed that the OP dug into it to get to a point where he could ask a specific person (who was probably a network engineer / tech of some level) to validate/fix his claim.
Re: The little ssh that (sometimes) couldn't
#60Well, while we're at it, here's my crazy MTU-related war story, although not as crazy as that one! I was troubleshooting with a user of an audio streaming application running over a LAN. The user could stream classical music but not rock music. Seriously. Classical was fine, but when streaming rock, the connection would drop after a few minutes. The application took chunks of audio, compressed them with a lossless co…
I love this sort of insane-sounding problem descriptions. Here's one that happened to me: WiFi disconnects when I visit Gmail, and doesn't reconnect until I reboot Linux. Cause: Gmail's chat thingy uses Flash, which for some reason does some webcam initialization, which triggers a bug (in combination with crappy hardware) in the uvcvideo kernel module which leads to a timeout which leads to the whole USB bus going do…