ah! got bit by this and had to workaround serving stuff from s3. nice to see there was an actual problem and wasn't just some issue with our stack.
The curious case of slow downloads
11–20 of 57 posts
Re: The curious case of slow downloads
#12Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess.
Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sockets then read what comes in, and write that to the quakeworld server.
To debug this, everytime someone sent some data over the network, it would write a simple message ("blah") into a file on my home drive, called "~/blah" (very sophisticated debugging, I know).
Now, the semantics of select(2), cousin of poll(2), are interesting.
My understanding: select(2) blocks until you can read data from the socket.
Actual behaviour: select(2) blocks until a subsequent read on the socket does not block.
Fun fact: if there's not data available on the socket, read does not block, but returns with an error immediatly.
Result: Over the weekend, the process was spinning in an endless loop, writing 'blah' into a file on the nfs-home drive as quickly as the filesystem allowed. This being a university scale solaris NFS server, that's pretty fast.
End Result: A test file called 'blah', filled with about 800 GB of 'blah'. Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD
EDIT: read(2) doesn't actually error on EOF, just returns 0, effect is the same though.
Re: The curious case of slow downloads
#13This is why open source (and controlling your whole stack) matters in big business. For example, Microsoft may be changing their image, but their core software is closed source. may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Please encourage your employer to fiscally support the open s…
> For example, Microsoft may be changing their image, but their core software is closed source. may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
What's that cost? How quickly can it be done? If I was a developer would I have to ask my manager to look into the details of our support plan? What about the ability to patch the source?
`git clone` is an incredible power. Companies should reward those who are generous with their code.
Re: The curious case of slow downloads
#14Re: The curious case of slow downloads
#15Earlier quoted context omitted.
There is a header that says that, but it actually was an NGINX problem.
This was a unix stack problem, but it could be fixed in nginx :)
NGINX decided to have a maximum time between calls to sendfile, no matter whether the data is actually transferring or not. In configuring this setup, someone had a bad intuition about how often NGINX would end up calling sendfile.
It is a problem entirely of NGINX's devising, not a unix stack problem.
Re: The curious case of slow downloads
#16Earlier quoted context omitted.
There is a header that says that, but it actually was an NGINX problem.
Not really. nginx's implemented behaviour seems more or less what anyone writing socket code would do: write to buffer, then wait with timeout to write more, if timeout is hit consider the write failed. The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.
The TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.
Re: The curious case of slow downloads
#17Earlier quoted context omitted.
> For example, Microsoft may be changing their image, but their core software is closed source. may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
What's that cost? How quickly can it be done? If I was a developer would I have to ask my manager to look into the details of our support plan? What about the ability to patch the source? `git clone` is an incredible power. Companies should reward those who are generous with their code.
> What's that cost? How quickly can it be done?
Careful with those goalposts, you're tearing up the pavement.
> Companies should reward those who are generous with their code.
So companies "should reward those who are generous with their code" but not compensate them for it?
Re: The curious case of slow downloads
#18Earlier quoted context omitted.
You're correct, but the title is way less helpful without the "nginx". Unless the reader knows that cloudflare uses nginx, he won't know the article pertains to nginx
Is it now? nginx seems to have pretty normal behaviour here, so the core issue is write-polling buffers on linux rather than nginx itself. It seems to me nginx was the messenger of sorts, but the issue doesn't really pertain to nginx. In a nearby comment, LoSboccacc notes that they've hit that issue [removed: on S3].
It sounded to me like LoSboccacc's solution was to serve files from S3, not that they experienced the same issue on S3.
Re: The curious case of slow downloads
#19Earlier quoted context omitted.
Not really. nginx's implemented behaviour seems more or less what anyone writing socket code would do: write to buffer, then wait with timeout to write more, if timeout is hit consider the write failed. The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.
The TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.
> NGINX wanted to do something extra, and did it incorrectly.
Having a timeout on a poll (or select) is not "something extra". it's something entirely normal and necessary for any non-trivial software, especially public-facing ones.
Re: The curious case of slow downloads
#20Earlier quoted context omitted.
> For example, Microsoft may be changing their image, but their core software is closed source. may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
What's that cost? How quickly can it be done? If I was a developer would I have to ask my manager to look into the details of our support plan? What about the ability to patch the source? `git clone` is an incredible power. Companies should reward those who are generous with their code.
Indeed the financial support of open source by many users is atrocious. Beyond that you're proselytizing.
This issue didn't get solved particularly quickly, either. Nor was "source to the whole stack" required -- kernel source made for a illustrative blog post but was not really part of the process.
Do you have source files for the CPU? The motherboard? Your DRAM controller? At some point you draw the line and make shit work. Tools plus docs plus code is great, but you can make do with just one of 'em in a pinch.