Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why We Love QUIC and HTTP/3

fastly.com

Post reply on HN
Print view View original

91–100 of 116 posts

Re: Why We Love QUIC and HTTP/3

#91

Post by 420codebro »

toast0 wrote: post #88
drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

In addition to those downsides, the QUIC spec points out middleboxes tend to time out UDP streams pretty agressively, so it recommends a ping timer of 10 seconds. Additionally, since QUIC streams allow for client IP mobility, that creates an additional challenge for IP level load balancing as well as handling at the host level. In a well configured host, TCP packets for a given stream will always arrive at the same n…

I didn't consider that about QUIC. Ouch, that is going to be painful as it gains traction. ECMP hashing assumptions may need to be re-evaluated.

Re: Why We Love QUIC and HTTP/3

#92

Post by Misdicorl »

toast0 wrote: post #88
drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

In addition to those downsides, the QUIC spec points out middleboxes tend to time out UDP streams pretty agressively, so it recommends a ping timer of 10 seconds. Additionally, since QUIC streams allow for client IP mobility, that creates an additional challenge for IP level load balancing as well as handling at the host level. In a well configured host, TCP packets for a given stream will always arrive at the same n…

That's not really a fair comparison. In the case that the IP changes for quic, tcp would have to completely re-establish the connection. A cross core memory access is tiny in comparison.

Re: Why We Love QUIC and HTTP/3

#93

Post by the8472 »

londons_explore wrote: post #78
the8472 wrote: post #41

Earlier quoted context omitted.

your router shouldn't introduce noticable latency. If it does it either has a weak CPU or network queues that are too large. In the former case you need to upgrade hardware, in the latter you need a firmware that supports CAKE.

Some routers have hardware NAT for TCP, but use the CPU for UDP. Also, some routers prioritize all UDP packets because they treat them as VoIP. Then all TCP suffers. Some service providers traffic shape UDP because uTorrent uses it (UTP) on random port numbers.

If the router can't handle a scenario where you replace all TCP with UDP then it's a cheap plastic toy in my eyes. Don't blame the protocol, blame the router. Sending MTU-sized UDP flows to a dozen targets at most is not even the most extreme, non-malicious scenario you can encounter on networks.

Similar arguments go for the prioritization.

Re: Why We Love QUIC and HTTP/3

#94

Post by windexh8er »

seanwilson wrote: post #47
drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

> To serve the same amount of traffic with QUIC, you'd probably need multiple Xeon Gold CPUS. I guess that Google can afford this. Can you explain more about how the negatives you mention weigh up against the positives? There isn't a net benefit somewhere? If not, can something be changed to give a better balance like a hybrid solution?

Personally I believe that the majority of positive caters to privacy. That being said there are other positive things about IETF QUIC that will, likely, play into new functionality over time.

A good document outlining considerations can be found here: https://http3-explained.haxx.se/en/

Re: Why We Love QUIC and HTTP/3

#95

Post by windexh8er »

tialaramex wrote: post #75
windexh8er wrote: post #25

Earlier quoted context omitted.

It will be interesting because it's a problem for all middleboxes that do any sort of deep packet inspection. Most of the devices that fall into this category today leverage many performance gains made by the assumption that 1) the majority of network layer traffic is TCP and 2) they have access to certain levels of metadata for free. Things are changing and getting a lot more difficult with HTTP/3 (IETF QUIC) and TL…

In the cases I've noticed the middlebox vendor claims TLS 1.3 only meaning that now their product isn't critically insecure in the face of TLS 1.3. It can't actually speak TLS 1.3 it just knows to say "Sorry, TLS 1.2 only" without breaking everything. In my country we had many televisions labelled HD Ready when HD television first became available. Were these actually ready to play HD television? Er, no. They could h…

I haven't found any real implementations to test (I'd like to). But it seems Fortinet is making bold claims that, on the surface, feel like lip service thus far:

https://www.fortinet.com/blog/business-and-technology/tls-is...

Keep in mind this is just TLS1.3 and they make no claims around IETF QUIC / HTTP/3.

Re: Why We Love QUIC and HTTP/3

#96

Post by tialaramex »

windexh8er wrote: post #95
tialaramex wrote: post #75

Earlier quoted context omitted.

In the cases I've noticed the middlebox vendor claims TLS 1.3 only meaning that now their product isn't critically insecure in the face of TLS 1.3. It can't actually speak TLS 1.3 it just knows to say "Sorry, TLS 1.2 only" without breaking everything. In my country we had many televisions labelled HD Ready when HD television first became available. Were these actually ready to play HD television? Er, no. They could h…

I haven't found any real implementations to test (I'd like to). But it seems Fortinet is making bold claims that, on the surface, feel like lip service thus far: https://www.fortinet.com/blog/business-and-technology/tls-is... Keep in mind this is just TLS1.3 and they make no claims around IETF QUIC / HTTP/3.

They say

> The good news for Fortinet customers is FortiOS 6.2 fully supports TLS 1.3 for effective and high-performance MITM inspection.

and in contrast

> The latest version of FortiOS 6.0 not only fully supports TLS 1.2 MITM, but it also does not break TLS 1.3 when it has to negotiate down to TLS 1.2.

[ The "break TLS 1.3" they're talking about is the phenomenon I described in a cousin post in this thread, several of their competitors screwed up here ]

So that suggests that in Fortinet's case products running their 6.2 release (it's unclear to me if this is merely in Beta or actually a finished product) will actually do TLS 1.3. It's sad that they feel they can boast about the earlier 6.0 product actually working correctly (compatibility with TLS 1.3 by downgrading to TLS 1.2 is literally how everything would work if you just correctly implemented the specification, yet) when it was released in 2018, many years after the TLS 1.2 specification was finalised and in wide use.

Still you know "Better late than never".

Re: Why We Love QUIC and HTTP/3

#97

Post by drewg123 »

the8472 wrote: post #40
drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

> and sends it 1500b at a time sendmmsg (or the upcoming io_uring) let you send multiple UDP packets with a single syscall.

But I think each of these UDP packets will still travel separately from the syscall layer to the NIC (eg, no TSO). So you're still a factor of 40 or so behind TCP + TSO

Re: Why We Love QUIC and HTTP/3

#98

Post by manigandham »

drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

This is just normal technological progress. CPU time is cheap and scalable, and the protocol will keep getting more optimized with better software and hardware. Similar issues were brought up with HTTP2 using TLS everywhere and messing with proxies but that's no longer a problem.

QUIC/HTTP3 as a protocol is a great improvement to actual internet performance for users which is what really matters.

Re: Why We Love QUIC and HTTP/3

#99

Post by kev009 »

manigandham wrote: post #98
drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

This is just normal technological progress. CPU time is cheap and scalable, and the protocol will keep getting more optimized with better software and hardware. Similar issues were brought up with HTTP2 using TLS everywhere and messing with proxies but that's no longer a problem. QUIC/HTTP3 as a protocol is a great improvement to actual internet performance for users which is what really matters.

Picking your comment as the newest instance but this is one of the dumbest memes I see in this thread.

Things don't automatically get better. It is hard work, it sucks, and it's not for everyone. It will take years to undo the damage of this transition. We will still be working on it in a decade. There are some very subtle gains like HOL-blocking. I'm not convinced that outweighs current actualized improvements in TCP congestion control (BBR), and for any application I can think of the places that really need something message-oriented seem better covered by WebRTC.

What you are really talking about is Full Employment Theorem.

Re: Why We Love QUIC and HTTP/3

#100

Post by drewg123 »

Matthias247 wrote: post #58
drewg123 wrote: post #28

QUIC costs something like 2x to 4x as much CPU time to serve large files or streams per byte as compared to TCP. This is because the anti-middlebox protections also mean that modern network hardware and software offloads that greatly reduce CPU time cannot work with QUIC. When combined with the fact that QUIC is userspace, that's just deadly for performance. I'm talking about TSO, LRO (aka GRO), kTLS, and kTLS + hw e…

I think in general I agree. However the overhead numbers are exaggerated, and we should be fair with that. E.g. it was already mentioned that multiple UDP packets can be transmitted via a single syscall, and reasonable implementations can make use of it. I haven't read the Quic spec (yet), so I don't know how much data can be aggregated without waiting for ACKs or interleaving other data - but if it's anything compar…

Of course it is not going to read the entire file at once.

Having written the FreeBSD kernel TLS, I can assure you that there is no copy. Data is brought into the kernel via DMA from storage into a page in the VM page cache. When the IO is done, it is then encrypted into an connection-private page. That page is then sent and DMA'ed on the network adapter. So we have in the kernel tls case:

  - memory DMA to kernel mem from storage.
  - memory READ from kernel mem to read plaintext for crypto
  - memory write to another chunk of kernel mem to write encrypted data
  - memory DMA from kernel mem to NIC
In the case where the NIC supports inline TLS offload, the middle 2 steps are skipped, and it devolves to essentially the unencrypted case.

For QUIC you have:

  - memory DMA to kernel mem from storage
  - memory read from kernel mem via mmap
  - memory write to userspace mem to write encrypted data
  - memory read from userspace mem to copy to kernel
  - memory write to kernel mem
  - memory DMA from kernel mem to NIC
So you go from 3 "copies" to 4 "copies", which increases memory bandwidth demands by 33%.

Right now, we can just barely serve 100g from a Xeon-D because Intel limited the memory bandwidth to DDR4-2400. At an effective bandwidth limit of 60GB/sec, that's on the edge of being able to handle the kernel TLS data path. So even if everything else about QUIC was free, this extra memory copy from userspace would cut bandwidth by a third.

Post reply on HN