The journey of an internet packet: Exploring networks with traceroute

Yes. I've worked across a range of companies for small startups to Fortune 500s for nearly 30 years, interviewing for positions from system administrators to dev ops to programmers, beginners to architects. There's just not a lot of generalists out there. It always surprises me how far folks can get in their careers with a very narrow knowledge base.

A couple years ago I was bringing a Java programmer up to speed on some C code just to learn they had no idea what a call stack was or how it worked. They were familiar with the stack as a data structure, but had no idea how a CPU worked or that it has a stack pointer register.

As long as the abstractions don't leak, I guess everything is fine. I mean, I can't really tell you at a physics level exactly how semiconductors work. At best I can hand wave an explanation.

That said, the lack of even hand waving knowledge about how the internet works among professionals who use it every day continues to surprise me.

Traceroute doesn't see 90% of the machines your packet passes through.

When your packet leaves a router at some-pop-some-port-wherever, that fiber isn't usually the same piece of glass that plugs into the next hop. There's a whole chain of amplifiers and possibly multiplexers that handle it between here and there.

Some of those provide reliable transport service, giving you the illusion of a fiber that never breaks, despite backhoes doing what backhoes do. Some of those shift the wavelength of your signal, letting you use cheap optics without troubling in the nuances of DWDM that packs your signal alongside dozens of others onto the same long-haul fiber. Some of those just boost the signal, along with all those others on the same fiber.

But what all those machines have in common, is that none of them speak IP. None of them touch the payload. None of them are capable of decrementing a hop count. They're "part of the wire" as far as the packet is concerned.

In my experience, this leads to two types of network engineers, separated by their understanding of these underlying realities.

Wait until DNS comes up again. I don't know how many DevOps, SRE, Senior Architects, etc. that I've worked with that just don't understand how names are resolved. There's a wide array of specialists that don't understand general fundamentals.

~~ Security through obscurity does work.

As a layer.

In traceroute’s defense, it is traceroute — for sure it doesn’t tell you anything about the devices that don’t operate at the IP level. Those devices either don’t affect the IP “route” abstraction (e.g., signal boosters) or do so in ways that end up plausibly visible in the next hop.

There’s a reason the network layer abstraction is so strong, and an analogy to CPU ISAs here that have a similar strength.

TCP, similarly, doesn’t tell you when packets are deduplicated/resent/reordered/etc. — that’s just not part of the presented abstraction. Want that? Use UDP.