Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Size of Packets

potaroo.net

Post reply on HN
Print view View original

51–60 of 73 posts

Re: The Size of Packets

#51

Post by kazinator »

hliyan wrote: post #24

This reminds me of one of the most interesting bugs I've faced: I was responsible for developing the component that provided away market data to the core trading system of a major US exchange (which allows the trading system to determine whether an order should be matched in-house or routed to another exchange with a better price). Throughputs were in the multiple tens of thousands of transactions per second and late…

Another thing to look into in this kind of situation is enabling jumbo frames.

Re: The Size of Packets

#52

Post by ay »

zamadatix wrote: post #47
ay wrote: post #45

Earlier quoted context omitted.

With IPv4, clearing the DF bit in all egress packets and hacking on top of QUIC could give just enough of a wiggle room to make it possible to explore this between a pair of cooperating hosts even in today’s Internet. Anti-DDoS middle boxes will be almost certainly unhappy with lone fragments and UDP in general, so it’s a bit of a thorny path. The big question is what to do with IPv6, since the intermediary nodes wil…

Most carrier/enterprise/hardware IPv4 routers, particular those on the internet, will not actually perform IPv4 fragmentation on behalf of the client traffic even though it's allowed by the IPv4 standard. Typically fragmentation is reserved for boxes which already have another reason to care about it (such as needing to NAT or inspect the packets) or the client endpoints themselves. I.e. the internet will (sparing se…

Of course on the carrier boxes the fragmentation is done also not inline, so its behavior will depend on the aggressiveness of the CoPP configuration, and will be subject to the same pitfalls as the ICMP packet too big generation.

Thanks for keeping me straight here!

Based on the admittedly old study at [0] seems like some carriers just don’t bother to fragment, indeed - but by far not all of them.

Firewalls might do virtual reassembly, so the trick with the initial fragment won’t fly there.

This MTU subject is interesting for me because I have a little work in progress experiment: https://gerrit.fd.io/r/c/vpp/+/41914/1/src/plugins/pvti/pvti... (the code itself is already in, but has a few crashy bugs still and I need to take make it not suck performance wise, but that is my attempt to revisit the issue of MTU for tunnel use case. The thesis is that keeping the 5-tuple will make “chunking”/“de-chunking” at tunnel endpoints much much simpler on the endpoints of the tunnel.

The source of inspiration was a very practical setup at [1], which is, while looking horrible in theory (locally fragmented GRE over L2TP), actually gives a decent performance with 1500-byte end to end MTU over the tunnel.

The open question is which inner MTU will be sane, taking into account the increased probability of loss with bigger inner MTU… intuitively seems like something like ~2.5K should just double the loss probability (because it’s 2x packets) and might be a workable compromise in 2025….

One could also do the same trick over QUIC, of course, but i wanted something tiny and easier to experiment with - and the ability to go over IPSec or wireguard as well as a secured underlay.

[0] https://labs.ripe.net/author/emileaben/ripe-atlas-packet-siz...

[1] https://github.com/ayourtch/linode-ipv6-tunnel

Re: The Size of Packets

#53

Post by ay »

kazinator wrote: post #51
hliyan wrote: post #24

This reminds me of one of the most interesting bugs I've faced: I was responsible for developing the component that provided away market data to the core trading system of a major US exchange (which allows the trading system to determine whether an order should be matched in-house or routed to another exchange with a better price). Throughputs were in the multiple tens of thousands of transactions per second and late…

Another thing to look into in this kind of situation is enabling jumbo frames.

“You had an MTU problem. You enable jumbo frames. Now you have two MTU problems”

Unless you control the entire set of possible paths (can be many!) and set all the MTUs to match well, this (while maybe on surface helping with the problem, depending on many things) can set one up with a nasty footgun, whereby black hole will show in the most terrible moment of high traffic. See my PMTUD/PLPMTUD rant elsewhere in this thread.

Re: The Size of Packets

#54

Post by Veserv »

zamadatix wrote: post #49
Veserv wrote: post #44

Earlier quoted context omitted.

Your first point appears to be about physical layer concerns. My suggestion was not meant to operate at that level. The proposed model assumes the physical layer guarantees point-point delivery of a distinct packet between adjacent nodes in the network with MTU limits manifesting as either discarding or rejecting the trailing portions of the packet. I said there were no “problems” if there are no layering violations…

> Your first point appears to be about physical layer concerns. My suggestion was not meant to operate at that level. The proposal doesn't operate at that level but it must be compatible with the operations of that level. I.e. that the physical layer can also cause truncation of layers riding on top of it needs to be accounted for in the way those upper layers consider what truncation means. The same is true for poss…

I just do not understand the problems you are stating. Let me present a concrete example.

We have A B C. A wishes to transmit a packet of 0x1000 bytes containing a Ethernet, IPv4, and then bespoke protocol, P, which is a header containing a length, MAC on the length + header, MAC on entire packet, encrypted payload, in that order. A then prepares transmit descriptors pointing at the packet and with size 0x1000 bytes.

C prepares receive descriptors pointing to buffers with a maximum capacity of size 0x1000 bytes per packet. B prepares receive descriptors pointing to buffers with a maximum capacity of size 0x500 (1280) bytes per packet.

A transmits the packet to B. The physical coding layer transmits the bytes terminating in the FCS. B receives bytes and does a running computation of the FCS. Upon reaching 0x500 bytes, it stops storing data into memory, stores the current FCS into memory, then continues receiving the data and computing the FCS until the data stream ends. Upon determining that the FCS matches, it marks the descriptor as valid for consumption and stores that the descriptor contains 0x500 bytes of data. The transmit engine of B then configures a transmit descriptor pointing at the packet and with size 0x500 bytes.

C then receives the 0x500 byte packet from B and observes that the FCS matches the 0x500 byte FCS and marks the descriptor as valid for consumption and stores that the descriptor contains 0x500 bytes of data. C then processes the packet observing that the P header indicates a length of 0x1000 bytes, but only 0x500 bytes are available. It attempts to authenticate the P header MAC using a secret known only to A and C. As the truncation only hit the encrypted payload at the tail, the P header MAC and the header data it is authenticating have not been modified by the truncation process. As such, C is able use the higher layer secret it shares with A to successfully authenticate the header data and determine that the header containing a length field with the value of 0x1000 bytes could have only been written by A and has not been tampered with. It then rejects the rest of the packet, but stores that the inbound MTU is only 0x500 bytes.

Re: The Size of Packets

#56

Post by vlovich123 »

ay wrote: post #53
kazinator wrote: post #51

Earlier quoted context omitted.

Another thing to look into in this kind of situation is enabling jumbo frames.

“You had an MTU problem. You enable jumbo frames. Now you have two MTU problems” Unless you control the entire set of possible paths (can be many!) and set all the MTUs to match well, this (while maybe on surface helping with the problem, depending on many things) can set one up with a nasty footgun, whereby black hole will show in the most terrible moment of high traffic. See my PMTUD/PLPMTUD rant elsewhere in this…

Given this is a trading system where application latencies are measured in microseconds, the default would be to assume that jumbo frames are totally a valid approach.

Re: The Size of Packets

#57

Post by JackSlateur »

cryptonector wrote: post #3

> Path MTU discovery has not been enthusiastically embraced Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On Linux it still defaults to disabled! https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5E...

path mtu discovery is worthless because the sending host does not control the path used

So it is not compatible with anycast, for instance, which is massively used everywhere

In the end, having no answer is better than having a most likely wrong answer

Re: The Size of Packets

#58

Post by [deleted] »

vlovich123 wrote: post #56
ay wrote: post #53

Earlier quoted context omitted.

“You had an MTU problem. You enable jumbo frames. Now you have two MTU problems” Unless you control the entire set of possible paths (can be many!) and set all the MTUs to match well, this (while maybe on surface helping with the problem, depending on many things) can set one up with a nasty footgun, whereby black hole will show in the most terrible moment of high traffic. See my PMTUD/PLPMTUD rant elsewhere in this…

Given this is a trading system where application latencies are measured in microseconds, the default would be to assume that jumbo frames are totally a valid approach.

[deleted]

Re: The Size of Packets

#59

Post by ikiris »

Veserv wrote: post #37
ikiris wrote: post #20

Earlier quoted context omitted.

I don’t think you understand what normally looks like if you start forwarding damaged frames like this because you can’t tell the difference. That was the point.

I literally have no idea what you are talking about. You can send garbage packets that conform to no known protocol on the internet. You can get more bit errors or perfect bit errors that make your bit error detection pass while still forwarding corrupt payloads. Transport protocols and channels must be and are robust to this. “Damaged” frames and frame integrity only matter if you need the contents of the entire pac…

I agree you have no idea what I’m talking about

Re: The Size of Packets

#60

Post by zamadatix »

ay wrote: post #52
zamadatix wrote: post #47

Earlier quoted context omitted.

Most carrier/enterprise/hardware IPv4 routers, particular those on the internet, will not actually perform IPv4 fragmentation on behalf of the client traffic even though it's allowed by the IPv4 standard. Typically fragmentation is reserved for boxes which already have another reason to care about it (such as needing to NAT or inspect the packets) or the client endpoints themselves. I.e. the internet will (sparing se…

Of course on the carrier boxes the fragmentation is done also not inline, so its behavior will depend on the aggressiveness of the CoPP configuration, and will be subject to the same pitfalls as the ICMP packet too big generation. Thanks for keeping me straight here! Based on the admittedly old study at [0] seems like some carriers just don’t bother to fragment, indeed - but by far not all of them. Firewalls might do…

Very interesting! It's like the best of the fragment-pre-encrypt world (everything appears as single packet 5 tuples to middleboxes) and fragment-post-encrypt world (transported packet data remains untouched) debate seen on IPsec deployments.

Like you mention you could do this under QUIC but then you'd be hamstrung to some of the design mandates such as encryption. This is way better as it's just datagrams doing your one goal - hiding that you're transporting fragments.

Post reply on HN