31–40 of 73 posts
This reminds me of one of the most interesting bugs I've faced: I was responsible for developing the component that provided away market data to the core trading system of a major US exchange (which allows the trading system to determine whether an order should be matched in-house or routed to another exchange with a better price). Throughputs were in the multiple tens of thousands of transactions per second and late…
> Path MTU discovery has not been enthusiastically embraced Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On Linux it still defaults to disabled! https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5E...
PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. The second worst offender is, IMO (data center and ISP) routers that generate ICMP replies in their CPU, meaning large packets hit a rate limited exception punt path out of the switch ASIC over to the cheapest CP…
This is an old Linux tcp offloading bug; large receive offload smooshes the inbound packet, then it's too big to forward.
I had to track down the other side of this. FreeBSD used to resend the whole send queue if it got a too big message, even if the size did not change. Sending all at once made it pretty likely for the broken forwarder to get packets close enough to do LRO, which resulted in large enough packet sending to show up as network problems.
I don't remember where the forwarder seemed to be, somewhere far away, IIRC.
> The speed of light in glass or fiber-optic cable is significantly slower, at approximately 194,865 kilometers per second. The speed of voltage propagation in copper is 224,844 kilometres per second. If I understand correctly, the speed of light in an electrical cable doesn't depend on the metal that carries current, but instead depends on the dielectric materials (plastic, air, etc.) between the two conductors?
For fibre the velocity factor depends on the refraction index of the fibre.
> Path MTU discovery has not been enthusiastically embraced Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On Linux it still defaults to disabled! https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5E...
PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. The second worst offender is, IMO (data center and ISP) routers that generate ICMP replies in their CPU, meaning large packets hit a rate limited exception punt path out of the switch ASIC over to the cheapest CP…
Passive PMTUD does NOT depend on ICMP messages.
> Path MTU discovery has not been enthusiastically embraced Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On Linux it still defaults to disabled! https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5E...
Would that help with UDP, or only TCP?
Earlier quoted context omitted.
Packets do not get “cut-off” normally. That is kind of the point. Some protocols allow transparent fragmentation, but the fragments need to encode enough information for reconstruction, so you can still detect “less data received than encoded on send”. You do not need bit error detection because you literally truncated the packet. The data is already lost. But in the process you learned it was due to MTU limits which…
I don’t think you understand what normally looks like if you start forwarding damaged frames like this because you can’t tell the difference. That was the point.
“Damaged” frames and frame integrity only matter if you need the contents of the entire packet to remain intact. Which you explicitly do not when truncating.
The only new problem that arises is that maybe the in-band length information or headers get corrupted resulting in misinterpreting the truncation that actually occurred. And again, you already need to be robust to garbage. And you can just change my proposal to recompute the integrity tag on the truncated data if you think that really matters.
I get the impression that the standard still allows hubs to exist, but that you just don't see them in practice.
I would be interested if anyone has ever used a 100mbit hub.
MTU discovery would be so much easier if the default behavior was truncate and forward when encountering a oversized packet. The endpoints can then just compare the bytes received against the size encoded inside of the packet to trivially detect truncation and thus get the inbound MTU size. This allows you to do MTU discovery as a endpoint protocol with all the authentication benefits that provides and allows you to…
Truncation for a dedicated probe packet type: you lose the information it's a probe when you go through a tunnel of some sort (VPN, L2TP, IPsec, MPLS, VPLS, VXLAN, PBB, q-in-q, whatever). You're also dealing with different layers e.g. a client could send an L3 packet probe and now you're expecting a layer 2 PBB/q-in-q node to recognize IP packet types and treat them specially (layering violation). Truncation for all…
Data in transit is almost never split for reasons other than fragmentation to avoid MTU problems. Any such split necessarily defines a fragmentation and reconstruction protocol so it still "preserves" the original send length information needed for truncation detection. If they have gone truly crazy and implemented a entire stream protocol transparently backing their flows then their transparent inner point-to-point layer would need to be aware of truncation in much the same way it would need to be aware of MTU limits anyways.
Forwarding generally corrupted packets should not be a problem unless your middleboxes are aggressively engaging in layering violations. From the perspective of a middlebox that is not engaging in layering violations you just have headers with blobs of data. Truncating the blob of data is basically uninteresting; at most you recalculate your integrity tags at your appropriate layer. You do not and should not recompute anything at higher layers. Furthermore, your endpoints must already be robust to blobs of garbage that pass your integrity tag checking because it is trivial for malicious actors to send you blobs of garbage with correctly calculated integrity tags. And, even if you were fully isolated, you can still get correlated bit errors that result in a correct integrity tag despite payload bit errors. Every client implementation that is not grossly incompetent must already be robust to getting garbage. You only get problems when your middleboxes start mucking around and trying to be too smart and violating your point-point transport abstraction.
You still get unidirectional protocols because you should manage truncation information out-of-band of any of your protocols. UDP or any other protocol should not communicate back to the sender that truncation happened. You do that some other way or even do not bother to do it at all. This is extra channel information that you can choose to communicate to let the other endpoint know about channel properties to make better data encoding decisions. You can transmit that in-band, out-of-band, on a different protocol, whatever. This is a higher level property of the communication channel between you and the other side.
Truncation is better authenticated because the packet reaches the other, known, authenticated endpoint who is the entity who can inform you, over a authenticated channel, that the transport channel has problems. You do not get nonsense like ICMP too large messages which come from unknown, unauthenticated entities. Furthermore, truncated messages can still be authenticated as long as you authentication tag the base header which should never be in the truncated section (you still need to have a minimum MTU below which you should always reject, but that number is small and much smaller than existing MTUs).
Can’t we accept to start a change that may take a decade or more to go forward? Instead of not starting that change.