41–50 of 73 posts
Earlier quoted context omitted.
Huh? Maybe I'm completely misreading the question, but when they say fiber-optic cable, they do mean optic. It's not an "electrical cable"; there is no metal needed in optic communication cables (perhaps for stiffness or whatnot, but not for the communication)
>The speed of voltage propagation in copper is 224,844 kilometres per second. This part?
Earlier quoted context omitted.
Truncation for a dedicated probe packet type: you lose the information it's a probe when you go through a tunnel of some sort (VPN, L2TP, IPsec, MPLS, VPLS, VXLAN, PBB, q-in-q, whatever). You're also dealing with different layers e.g. a client could send an L3 packet probe and now you're expecting a layer 2 PBB/q-in-q node to recognize IP packet types and treat them specially (layering violation). Truncation for all…
You truncate for all packet types. Data in transit is almost never split for reasons other than fragmentation to avoid MTU problems. Any such split necessarily defines a fragmentation and reconstruction protocol so it still "preserves" the original send length information needed for truncation detection. If they have gone truly crazy and implemented a entire stream protocol transparently backing their flows then thei…
Fragmentation is a specific (unrelated) term, it's not interchangeable with a split. You can have (depending on the protocols involved):
- A runt due to a collision
- A link drop during transmit
- A problem during cut-through type transport
You can do various things to combat some of these (such as fragment-free instead of cut-through in collision domains) but you can't guarantee every phy IP ends up riding over can or should avoid these constraints.
> Forwarding generally corrupted packets should not be a problem unless your middleboxes are aggressively engaging in layering violations. From the perspective of a middlebox that is not engaging in layering violations you just have headers with blobs of data.
If "delivery of something somewhere" is your only definition of a problem, perhaps :p.
> Furthermore, your endpoints must already be robust to blobs of garbage that pass your integrity tag checking because it is trivial for malicious actors to send you blobs of garbage with correctly calculated integrity tags.
Not only the endpoints to garbage in the data payloads but equally the gear to garbage in the network headers. Be it full authentication or just error detection, you don't want to just forward things with a corrupted network header and hope it doesn't cause an issue or security violation. Things like CRCs or HMACs are done per layer precisely for this kind of reason, going to truncation requires dropping that safe handling.
> Every client implementation
As a side note: the concerns have less to do with the clients, they have full context and control of their sessions in software land with little concerns from concerns in being the physical transport layer. Most all of these considerations need to be thought from the intermediate boxes doing the transport/truncation instead.
> You still get unidirectional protocols because you should manage truncation information out-of-band of any of your protocols
Unidirectional protocols cannot be expected to punt directionality to a separate session. In general, any time the answer to a network conundrum (such as the two generals) sounds as easy as "just move that to a separate channel which has the information" you have either duplicated the problem in that channel or added functionality which might not be physically available (or directionally available for security use case reasons, or scalably available for multicast, or something else for a use case that isn't 'inside out' from what might pop in mind as a 'standard' session).
> Truncation is better authenticated because the packet reaches the other, known, authenticated endpoint who is the entity who can inform you, over a authenticated channel, that the transport channel has problems.
I'm still not sure I follow - how is the message between endpoints still authenticated if middleboxes can modify the bytes, breaking an HMAC and/or CRC (if any), and it still gets delivered? Having authenticated an endpoint exists at an address you've sent a packet to before does not automatically authenticate any packet which arrives.
You also skipped over any of the implications for network tunnels (secure/insecure) - is MTU discovery just not supposed to work in those use cases?
I think you can absolutely make a domain specific protocol which is happy to use truncation for MTU discovery, I just don't think anything which is supposed to be as universally usable as IP can.
Earlier quoted context omitted.
You truncate for all packet types. Data in transit is almost never split for reasons other than fragmentation to avoid MTU problems. Any such split necessarily defines a fragmentation and reconstruction protocol so it still "preserves" the original send length information needed for truncation detection. If they have gone truly crazy and implemented a entire stream protocol transparently backing their flows then thei…
> Data in transit is almost never split for reasons other than fragmentation to avoid MTU problems Fragmentation is a specific (unrelated) term, it's not interchangeable with a split. You can have (depending on the protocols involved): - A runt due to a collision - A link drop during transmit - A problem during cut-through type transport You can do various things to combat some of these (such as fragment-free instead…
I said there were no “problems” if there are no layering violations because you argued that recalculating checksums would be a layering violation. Either we say layering violations are unacceptable at which point my argument stands. Or we say layering violations are par for the course and you can just recalculate the checksums if you need to.
Unidirectional protocols with no back channel must assume the network channel parameters such as MTU. Adding truncation information which can be picked up at a different layer is just strictly more information you can feed into your protocol if it is designed to handle that. You can just not use it and act as if truncation is dropping if you want to. This is just strictly more data you can use for decisions.
You can get still get authenticated transport in the presence of truncation if your protocol generates a authentication tag for the “original” length and puts it at the start of the message. Then you can authenticate the length field and verify truncation otherwise you can drop it.
I did not bother with tunnels because I do not see how it is a distinct problem. Tunnels already need to figure out how to manage their MTUs. Either the tunnel is transparently managing how it fragments data and can be enhanced to support truncation (though it does not need to, it can just drop truncation/malformed as they currently do) or it tells tunnel parameters to the endpoints so that the endpoints keep themselves in bound at which point the endpoints can detect whatever the MTU of the tunnel is.
And again, you can always just ignore truncated packets and act as if they are malformed which everybody already does. This is strictly more functionality which does not require changing all existing systems which can be used to support more efficient MTU discovery by systems and networks that supported it. And if they do not, you just fallback to the current, crusty way.
MTU discovery would be so much easier if the default behavior was truncate and forward when encountering a oversized packet. The endpoints can then just compare the bytes received against the size encoded inside of the packet to trivially detect truncation and thus get the inbound MTU size. This allows you to do MTU discovery as a endpoint protocol with all the authentication benefits that provides and allows you to…
Anti-DDoS middle boxes will be almost certainly unhappy with lone fragments and UDP in general, so it’s a bit of a thorny path.
The big question is what to do with IPv6, since the intermediary nodes will only drop. This bit unfortunately makes the whole exercise pretty theoretical, but it can be fun nonetheless to explore.
Feel free to contact me at my github userid at gmail, if this is a topic of interest.
MTU discovery would be so much easier if the default behavior was truncate and forward when encountering a oversized packet. The endpoints can then just compare the bytes received against the size encoded inside of the packet to trivially detect truncation and thus get the inbound MTU size. This allows you to do MTU discovery as a endpoint protocol with all the authentication benefits that provides and allows you to…
Truncation for a dedicated probe packet type: you lose the information it's a probe when you go through a tunnel of some sort (VPN, L2TP, IPsec, MPLS, VPLS, VXLAN, PBB, q-in-q, whatever). You're also dealing with different layers e.g. a client could send an L3 packet probe and now you're expecting a layer 2 PBB/q-in-q node to recognize IP packet types and treat them specially (layering violation). Truncation for all…
PMTUD somehow missed that packet networks ditching the OOB mechanisms of circuit switched networks was a good thing. By adding an OOB mechanism of attempted MTU discovery. Unauthenticated.
Yes, matching the 5-tuple from the original payload somewhat helps against the obvious security problem with this. (It was a fun 3-4 years while it was being added to systems across the ‘net while everyone was blocking the ICMP outright to avoid the exploitation. The burps of that one might still find in some security guidelines)
But the number of the network admins who understand what do they have to configure in their ACLs and why, is scarily small compared to the overall pool size.
Here’s another hurdle: for about two decades, to generate ICMP you have to punt the packet from hardware forwarding to the slow path. Which gets rate-limited. Which gives one a fantastic way to create extremely entertaining and hard to debug problems: a single misbehaving or malicious flow can disable the ICMP generation for everyone else.
Make hardware that can do it in fast path ? Even if you don’t punt - you still have to rate-limit to prevent the unauthenticated amplification attack (28 bytes of added headers is not comparable with some of the DNS or NTP scenarios, but not great anyway)
So - practically speaking, it can’t be relied on, other than a source for great stories.
PLPMTUD is a little better, in a sense that it attempts to limit itself to inband probes, but then there is the delicate dance of loss customarily being used to signal the congestion.
So this mechanism isn’t too reliable either, in very painful ways for the poor soul on call dealing with the outcomes. Ask me how I know.. ;-)
Now, let’s add to this the extremely pragmatic and evil hack that is the TCP MSS clamping, coming back from the first PPPoE days; which makes just enough of the eyeball traffic work to make this a “small problem with unimportant traffic that no one cares for anyway”.
So yes, safe minimums are a practical solution.
Until one start to build the tunnels, that is. A wireguard tunnel inside IPSec tunnel. Because policy. Inside VXLAN tunnel inside another IPSec tunnel, because SD-WAN. Which traverses NAT64, because transition and address scarcity.
At which point the previously safe minimums might not be safe anymore and we are back to square 1. I suspect when folks will start running QUIC over wireguard/ipsec/vxlan + IPv6 en masse we will learn that (surprise!) 1200 was not a safe value after all.
So, with this in mind, I posit it’s nice to attempt to at least fantasize about the universe where MTU determination would be done entirely inline, even if hypothetical - if we had the benefit of today’s hindsight and could time travel - could we have made it better ?
P.s. unidirectional protocols could be taken care of by fountain codes not unlike the I-, P- and B- frames in video world, with similar trade offs, moreover, I feel the unequal probability of loss depending on a place in the packet might allow for some interesting tricks.
MTU discovery would be so much easier if the default behavior was truncate and forward when encountering a oversized packet. The endpoints can then just compare the bytes received against the size encoded inside of the packet to trivially detect truncation and thus get the inbound MTU size. This allows you to do MTU discovery as a endpoint protocol with all the authentication benefits that provides and allows you to…
With IPv4, clearing the DF bit in all egress packets and hacking on top of QUIC could give just enough of a wiggle room to make it possible to explore this between a pair of cooperating hosts even in today’s Internet. Anti-DDoS middle boxes will be almost certainly unhappy with lone fragments and UDP in general, so it’s a bit of a thorny path. The big question is what to do with IPv6, since the intermediary nodes wil…
Of course nothing is stopping you from labbing it up at home. Firewalls and software routers can usually be made to do refragmentation.
Earlier quoted context omitted.
Truncation for a dedicated probe packet type: you lose the information it's a probe when you go through a tunnel of some sort (VPN, L2TP, IPsec, MPLS, VPLS, VXLAN, PBB, q-in-q, whatever). You're also dealing with different layers e.g. a client could send an L3 packet probe and now you're expecting a layer 2 PBB/q-in-q node to recognize IP packet types and treat them specially (layering violation). Truncation for all…
From the pragmatic standpoint: manually hard coding a safe minimum is the only approach which consistently works. PMTUD somehow missed that packet networks ditching the OOB mechanisms of circuit switched networks was a good thing. By adding an OOB mechanism of attempted MTU discovery. Unauthenticated. Yes, matching the 5-tuple from the original payload somewhat helps against the obvious security problem with this. (I…
With regard to the problems of out of band signaling in plain PMTUD I fully agree with all your well stated points, doubly so on PLPMTUD! PLPMTUD is my preferred variation of PMTUD and I was glad to see the datagram form utilized in QUIC (especially since it's really a generic secure network tunneling protocol, not just the HTTP variant). I'm also glad QUIC's security model naturally got rid of MSS clamping... it was somewhat pragmatic in one view... but concerning/problematic in others :D. Of course it's not like TCP/mss clamping have exactly gone away though :/.
Also fully agree on both PLPMTUD still not being as reliable/fast as one would like (though I still think it's the best of the options) + safe minimums never seeming to stay "safe". At least IPv6 attempted to hedge this by putting pressure on network admins, saying "everyone is expecting 1280". Of course... we all know that doesn't mean every client ends up with 1280, particularly if they are doing their own VPN tunnel or something, but at least it gives us network guys an extra wall of "well, the standard says we need to allow expectation of 1280 and the rate of bad things which happen will be much higher lower than that".
You seem to have some really neat perspectives on networking, do you mind if I ask about what you do/where you got your experience? I came up through the customer side and eventually over time morphed my way into NOS development at some network OEMs and it feels like I run into fewer and fewer folks who deal with the lower layers of networking as time has went on. I think the most "fun" parts are trying to design overlay/tunneling systems which are hardware compatible with existing ASICs or protocols but are able to squeeze some more cleverness out of the usage (or, as you put it, if we had the benefit of today’s hindsight and could time travel - could we have made it better). The area I'd say I've been least involved in, but would like to, is anything to do with time sensitive networking or lossless ethernet use cases.
Earlier quoted context omitted.
> Data in transit is almost never split for reasons other than fragmentation to avoid MTU problems Fragmentation is a specific (unrelated) term, it's not interchangeable with a split. You can have (depending on the protocols involved): - A runt due to a collision - A link drop during transmit - A problem during cut-through type transport You can do various things to combat some of these (such as fragment-free instead…
Your first point appears to be about physical layer concerns. My suggestion was not meant to operate at that level. The proposed model assumes the physical layer guarantees point-point delivery of a distinct packet between adjacent nodes in the network with MTU limits manifesting as either discarding or rejecting the trailing portions of the packet. I said there were no “problems” if there are no layering violations…
The proposal doesn't operate at that level but it must be compatible with the operations of that level. I.e. that the physical layer can also cause truncation of layers riding on top of it needs to be accounted for in the way those upper layers consider what truncation means. The same is true for possible intermediate layers (which sorta aligns with the later conversations regarding tunnels, which are basically just more complicated forms of intermediate layers).
> The proposed model assumes the physical layer guarantees point-point delivery of a distinct packet between adjacent nodes in the network with MTU limits manifesting as either discarding or rejecting the trailing portions of the packet.
Then proposed isn't applicable to IP since an upper layer protocol cannot make guarantees about the behavior of lower level protocols it may be transported on.
In addition, discarding trailing portions of the packet still results in the aforementioned problems with consistency checks and forwarding behavior limitations for lower level layers which did abide by this behavior.
> Unidirectional protocols with no back channel
One cannot guarantee bidirectional protocols will be able/allowed to form a back channel either, I just used unidirectional as a more clear-cut example.
> You can just not use it and act as if truncation is dropping if you want to. This is just strictly more data you can use for decisions.
Well sure, the same is true of the ICMP method or an active probing method. The concern is less with sessions you don't care to PMTUD in the first place and more with how the truncation design affects the designs of such other use cases.
> You can get still get authenticated transport in the presence of truncation if your protocol generates a authentication tag for the “original” length and puts it at the start of the message. Then you can authenticate the length field and verify truncation otherwise you can drop it.
I totally agree one can include an HMAC tag in your clientclient protocol to validate unmodified packets are authentic. This is regardless of whether truncation, ICMP packet too big, active PMTUD probing, or any other method is in place as, to this point, this is only about validating delivered packets which did fit in the MTU.
What isn't clicking is when a truncated message arrives how a (now invalid) HMAC helps you authenticate if this packet was completely spoofed by a malicious actor or really truncated by a middlebox. All you know is it was supposed to be longer and now something claims it needs to be shorter, how do you know that's not because of the same malicious actor who was supposed to be sending the fake ICMP packet too big rather than a middlebox really trying to signal the packet truly needed to be truncated?
> I did not bother with tunnels because I do not see how it is a distinct problem.
As highlighted earlier, tunnels may either encapsulate other protocols or encapsulate protocols which are expecting truncation. If the only things which existed in the world were client network interfaces it wouldn't be a problem, once more network devices become involved then you have to consider the impact on those too. The main thing to keep in mind is very few network middleboxes or tunnel protocols have the ability to do fragmentation on behalf of tunneled data, particularly if they are hardware based or based on protocols without such a feature (such as Ethernet) since this eats up TONS of hardware to do so (especially at high speeds). E.g. take an IPv6 VXLAN tunnel of an Ethernet frame on a 400 Gbps interface, how is an pure L3 intermediate carrier router doing truncation supposed to know not to update the UDP (a layer up the stack) checksum so the truncated Ethernet payload actually gets delivered to the client destination from the egress VTEP? It's not even that the egress VTEP needs some way to signal to the ingress VTEP how much the truncation was, it's that the original client which was VXLAN encapsulated by the ingressing VTEP needs its packet delivered to the remote client so the remote client can see the truncation and re-negotiate (in band or out of band) with the client to send smaller frames. This signaling will not occur because of the aforementioned UDP checksum being broken by an intermediate router. Just removing all checksums and allowing all modifications to headers and delivering whatever arrives would create not only high incidences of the propagation of deformed traffic but also security risks.
This brings us back to the example of secure tunnels, like IPsec, which have the same problem but in a much more succinct form. All parts of the payload of an IPsec tunnel are basically random noise after you truncate it, so there is no way to even attempt to consider sending the truncated payload to the intended destination. It's not the responsibility of the IPsec encapsulator to perform the encapsulation and the IPsec receiver usually doesn't have a path to communicate with the original client (not that it even knows who that is).
If you redesign everything about how network tunneling works under some severe limitations and assumptions then it may be possible to solve some (or maybe all if I can figure out what I'm missing regarding authentication of packets claiming MTU changes) of these problems but I'm not sure I could ever see the set of requirements needed as easier than the other MTU approaches. That doesn't necessarily mean I think there is an overall perfect answer all, just that I think PMTUD and its variants are definitely the easier path.
Earlier quoted context omitted.
From the pragmatic standpoint: manually hard coding a safe minimum is the only approach which consistently works. PMTUD somehow missed that packet networks ditching the OOB mechanisms of circuit switched networks was a good thing. By adding an OOB mechanism of attempted MTU discovery. Unauthenticated. Yes, matching the 5-tuple from the original payload somewhat helps against the obvious security problem with this. (I…
Agree wholeheartedly on the pragmatic standpoint of just using minimums. With regard to the problems of out of band signaling in plain PMTUD I fully agree with all your well stated points, doubly so on PLPMTUD! PLPMTUD is my preferred variation of PMTUD and I was glad to see the datagram form utilized in QUIC (especially since it's really a generic secure network tunneling protocol, not just the HTTP variant). I'm al…
This works great until there is an app that is expecting 1280 and there is an operator that gives you 1280, and you have to run this app over an encrypted GENÈVE tunnel that attempts to add half a kilobyte of metadata :-). RADIUS with EAP or DHCP with a bunch of options can be a good example of a user app like this. Unfortunately this is a real-world problem.
The smaller mismatch but nonetheless painful is the 20 byte difference between IPv4 and IPv6 header sizes. It trips up every NAT64 deployment.
> where you got your experience?
A long path along all the OSI layers :-). Fiber and UTP networks install between ~95 and 2000. CCIE R&S#5423 in ‘99 and from 2000 almost 10 years in TAC and one of the first CCIE in Europe. Then some years working on IPv6 transition. Large scale IPv6 WiFi. Some folks know me by “happy eyeballs”; some by a “nats are good” YouTube video (scariest thing it’s still funny a decade later). These days - relops at fd.io VPP + internal CI/CD pipeline for a bunch of projects using VPP; and as a side gig - full-cycle automation of the switched fleet (~500 boxes) at #CLEUR installations. One of the recent fun projects was [0] - probably industry first of this scale, for an event network: more than 15K WiFi clients on IPv6Mostly. Though we were benefitting from work of a lot of folks that pushed the standardization and did smaller/more controlled deployments, specifically to shout huge thanks to Jen Linkova and Ondřej Caletka.
If you like low level network stuff, you might like VPP - and given it’s Apache licensed, pretty easy to use it for your own project.
[0] https://www.ietf.org/proceedings/122/slides/slides-122-iepg-...