The curious case of slow downloads • Hacker News Board

Re: The curious case of slow downloads

#31

Post by scurvy » Wed, May 04, 2016, 1:43 PM UTC

Dylan16807 wrote: post #16

masklinn wrote: post #9

Earlier quoted context omitted.

Not really. nginx's implemented behaviour seems more or less what anyone writing socket code would do: write to buffer, then wait with timeout to write more, if timeout is hit consider the write failed. The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.

The TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.

nginx is also resetting the connection instead of closing it. That's also a bug (separately). The use of a TCP reset here violates RFC and Postel's robustness maxim. Be liberal on what you accept and be conservative in what you send.

Re: The curious case of slow downloads

#32

Post by Dylan16807 » Wed, May 04, 2016, 1:44 PM UTC

masklinn wrote: post #27

Dylan16807 wrote: post #22

Earlier quoted context omitted.

What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts.

> What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts. nginx called send(file), then polled the socket for write with the timeout , closing the connection if the timeout was hit. It has everything to do with poll/select timeouts, and especially with Linux letting poll hit timeouts when waiting on effectively writable sockets.

> writable sockets

The goal of NGINX here is not to measure whether a socket can consume any data whatsoever. A connection that consumes 1 byte every 15 seconds is still considered stalled, and should be killed. Using the time between sendfile invocations is not the goal, it's a means toward implementing a minimum rate, and they implemented a minimum rate the wrong way. It's an X Y problem and that's not the kernel's fault.

Re: The curious case of slow downloads

#33

Post by Dylan16807 » Wed, May 04, 2016, 1:48 PM UTC

scurvy wrote: post #25

I really wish people would stop misusing TCP resets. It's not just a fast way to close a connection. That's not what it's for at all. If you're ever thinking about sending a reset, please read RFC 793 and RFC 3360 first. I'm looking at you Arbor.

Well the goal here seems to be aborting a connection, not just closing it. They specifically don't want to empty the buffer, because time is up. Is that a misuse of reset?

Re: The curious case of slow downloads

#34

Post by brazzy » Wed, May 04, 2016, 1:50 PM UTC

BugsBunnySan wrote: post #12

Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess. Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sock…

> Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD

They should be the ones explaining why they don't have disk quotas set up on a shared system.

Re: The curious case of slow downloads

#35

Post by zzzcpan » Wed, May 04, 2016, 1:56 PM UTC

masklinn wrote: post #19

Dylan16807 wrote: post #16

Earlier quoted context omitted.

The TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.

> NGINX wanted to do something extra, and did it incorrectly. Having a timeout on a poll (or select) is not "something extra". it's something entirely normal and necessary for any non-trivial software, especially public-facing ones.

The problem is a fundamental one. Nginx, like many other event loop implementations of servers, presumes that kernel code behaves consistently, precisely and correctly, instead of treating it like a black box, that sometimes lies about things or behaves completely incorrectly, it's not the first time this happened. Any correct event loop implementation should take reported poll events under advisement only and try to schedule reads and writes independently from that.

Re: The curious case of slow downloads

#36

Post by scurvy » Wed, May 04, 2016, 1:56 PM UTC

Dylan16807 wrote: post #33

scurvy wrote: post #25

I really wish people would stop misusing TCP resets. It's not just a fast way to close a connection. That's not what it's for at all. If you're ever thinking about sending a reset, please read RFC 793 and RFC 3360 first. I'm looking at you Arbor.

Well the goal here seems to be aborting a connection, not just closing it. They specifically don't want to empty the buffer, because time is up. Is that a misuse of reset?

Yes, it is a misuse of a TCP reset. The RFCs are very clear about when to use a TCP reset.

You can't use a reset just because you want to be fast or don't feel like sending any more data. Resets are for something abnormal happening. A timeout isn't abnormal to a TCP connection. It's part of the process.

Re: The curious case of slow downloads

#37

Post by Dylan16807 » Wed, May 04, 2016, 2:00 PM UTC

scurvy wrote: post #36

Dylan16807 wrote: post #33

Earlier quoted context omitted.

Well the goal here seems to be aborting a connection, not just closing it. They specifically don't want to empty the buffer, because time is up. Is that a misuse of reset?

Yes, it is a misuse of a TCP reset. The RFCs are very clear about when to use a TCP reset. You can't use a reset just because you want to be fast or don't feel like sending any more data. Resets are for something abnormal happening. A timeout isn't abnormal to a TCP connection. It's part of the process.

So what are you supposed to do?

Re: The curious case of slow downloads

#38

Post by BugsBunnySan » Wed, May 04, 2016, 2:03 PM UTC

brazzy wrote: post #34

BugsBunnySan wrote: post #12

Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess. Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sock…

> Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD They should be the ones explaining why they don't have disk quotas set up on a shared system.

AFAIR, they had one, which was so flakey they had to disable it all the time.

But yeah, maybe them being somewhat at fault as well helped me out a bit :)

Re: The curious case of slow downloads

#39

Post by icebraining » Wed, May 04, 2016, 2:03 PM UTC

rakoo wrote: post #30

masklinn wrote: post #10

Earlier quoted context omitted.

> For example, Microsoft may be changing their image, but their core software is closed source. may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.

Cloudflare was allowed to patch their nginx with the correct solution. Would the Shared Source Initiative allow you to do that, ie patch and run the modified version ?

I'm guessing no, since the licenses on the SSI site include:

  Licensees may use the source code as a reference or view within a debugger,
  but may not modify the code or use it to create derivative works.

Re: The curious case of slow downloads

#40

Post by BugsBunnySan » Wed, May 04, 2016, 2:04 PM UTC

ajnin wrote: post #23

BugsBunnySan wrote: post #12

Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess. Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sock…

Read will return 0 when the peer has closed the connection, that's how you detect it. When there is no data, the call will block instead until some data arrives (unless the socket is set to non-block mode).

Hm yeah, it could've been that I disconnected from the server when I went home for the weekend and didn't check for that in the code. It was a long time ago :)