Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On incomplete HTTP reads and the requests library in Python

blog.petrzemek.net

Post reply on HN
Print view View original

11–20 of 32 posts

Re: On incomplete HTTP reads and the requests library in Python

#11

Post by llccbb »

Twirrim wrote: post #10
llccbb wrote: post #5

I am confused about the test used here. The `requests` docs very plainly state that `Response.ok` only checks the status code. Looking into the codebase proves that as well. Is there a status code for "I am going to send bad-length content"? This might be a UX problem and not a technical one. The author seems to think that `Response.ok` should be an all-purpose check.

Honestly? I'm with the author here. "Response.ok" should tell me that the response is ok. It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable.

I agree with your point that there should be some baked in capability for a more comprehensive check on the HTTP response. There are probably some edge cases around this regarding stream=True (load header only, not content) and iter_content() which pulls chunks from a buffer so the entire content isn't loaded into memory at once. Probably surmountable.

FWIW Response.status_code is an int, so it might need to be a member function on Response.

Re: On incomplete HTTP reads and the requests library in Python

#12

Post by greglindahl »

cryptonector wrote: post #2

I've written an HTTP service that offers "tail -f" as a service. When you GET a resource with a Range: bytes=0- (or some other starting offset with unspecified end offset) you get chunked transfer-encoding that doesn't terminate until the file is unlinked or renamed out of the way. This is incredibly handy, both for the usual things one might want to tail -f (log files), and as a cheap-but-very-functional pub/sub sys…

You could consider using websockets, which are intended to be used the way you're trying to use chunking.

Re: On incomplete HTTP reads and the requests library in Python

#13

Post by ubernostrum »

Twirrim wrote: post #10
llccbb wrote: post #5

I am confused about the test used here. The `requests` docs very plainly state that `Response.ok` only checks the status code. Looking into the codebase proves that as well. Is there a status code for "I am going to send bad-length content"? This might be a UX problem and not a technical one. The author seems to think that `Response.ok` should be an all-purpose check.

Honestly? I'm with the author here. "Response.ok" should tell me that the response is ok. It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable.

It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable.

Except that the status line is literally "HTTP 200 OK", and the 'ok' check is simply using the domain terminology. This is not a bad thing. And having 'request.ok' be a general "I have examined every possible aspect of this response for problems and found none" is probably impossible, despite being what you apparently expect.

The solution is not to rename the method, it's to also warn or error when there's something fishy in the response. Which is apparently what requests 3.x will do.

There's also the issue that the Content-Length header, while generally well-adopted, is optional (sending either Content-Length or Transfer-Encoding is a SHOULD in RFC 7230, not a MUST), and sending an incorrect Content-Length, while annoying, is not actually against the RFC (the only MUST NOT prohibitions on an incorrect Content-Length value in the response are when the request was a HEAD or a conditional GET).

Re: On incomplete HTTP reads and the requests library in Python

#14

Post by TheSwordsman »

ubernostrum wrote: post #13
Twirrim wrote: post #10

Earlier quoted context omitted.

Honestly? I'm with the author here. "Response.ok" should tell me that the response is ok. It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable.

It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable. Except that the status line is literally "HTTP 200 OK", and the 'ok' check is simply using the domain terminology. This is not a bad thing. And having 'request.ok' be a general "I have examined every possible aspect of this response for problems and found none" is probably impossible, despite…

I'm not sure I agree with your assessment that sending an incorrect Content-Length is permitted. This specific post quotes a section in RFC-2616 that deals with this directly[1]:

  When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly
  match the number of OCTETs in the message-body.
According to this, the Content-Length MUST match the data. Can you share an example of the opposite being stated?

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4....

Re: On incomplete HTTP reads and the requests library in Python

#15

Post by luhn »

Twirrim wrote: post #10
llccbb wrote: post #5

I am confused about the test used here. The `requests` docs very plainly state that `Response.ok` only checks the status code. Looking into the codebase proves that as well. Is there a status code for "I am going to send bad-length content"? This might be a UX problem and not a technical one. The author seems to think that `Response.ok` should be an all-purpose check.

Honestly? I'm with the author here. "Response.ok" should tell me that the response is ok. It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable.

If a HTTP response is malformed, the most natural expectation for an exception to be thrown.

I'm glad to see the requests 3.0 does this by default for invalid Content-Length.

Re: On incomplete HTTP reads and the requests library in Python

#16

Post by dagenix »

This is one of the most frustrating issues I've had with the requests/urllib3 libraries. I have no idea what I'm supposed to do with half of an HTML page or a PDF or a PNG. And even if it did, I certainly don't know how I'm supposed to handle those things when I don't get an indication that I'm likely dealing with an incomplete response. I can construct a use case where I might want this behavior (maybe a FLIF image), but it feels like such a huge outlier to me, that this being the default behavior of requests, which caters to making the common cases easy, has always felt a bit odd.

I don't begrudge the authors for this at all. After all, these are people donating their time to work on libraries solving hard problems for free that I can then use as part of my job. There is nothing that has stopped me from forking these libraries and changing the behavior myself, other than a lack of time and that, while I dislike this behavior, its only is a problem somewhat infrequently. All that being said, I really wish the response to this issue has been something other than what feels like a language-lawerly reading of the spec (requests isn't an "agent" and so that sentence in the spec doesn't apply) and the theory that someone _might_ be able to do something with an incomplete response _some_ of the time and as such, the vastly more common case should be made much more complicated.

But, anyway, I'm glad to hear that this issue is being addressed and I thank the authors for their work!

Re: On incomplete HTTP reads and the requests library in Python

#17

Post by ubernostrum »

TheSwordsman wrote: post #14
ubernostrum wrote: post #13

Earlier quoted context omitted.

It's the most natural expectation. If it's only going to check status code, Response.status_code.ok would be more reasonable. Except that the status line is literally "HTTP 200 OK", and the 'ok' check is simply using the domain terminology. This is not a bad thing. And having 'request.ok' be a general "I have examined every possible aspect of this response for problems and found none" is probably impossible, despite…

I'm not sure I agree with your assessment that sending an incorrect Content-Length is permitted. This specific post quotes a section in RFC-2616 that deals with this directly[1]: When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly match the number of OCTETs in the message-body. According to this, the Content-Length MUST match the data. Can you share an example of…

RFC2616 is obsoleted by RFC7230. Relevant section:

https://tools.ietf.org/html/rfc7230#section-3.3.2

Occurrences of MUST NOT for the server in that section are:

A sender MUST NOT send a Content-Length header field in any message that contains a Transfer-Encoding header field.

(so can't send both Content-Length and Transfer-Encoding)

A server MAY send a Content-Length header field in a response to a HEAD request (Section 4.3.2 of [RFC7231]); a server MUST NOT send Content-Length in such a response unless its field-value equals the decimal number of octets that would have been sent in the payload body of a response if the same request had used the GET method.

(if you send Content-Length on HEAD, it must match the length of the response you would have sent for GET)

A server MAY send a Content-Length header field in a 304 (Not Modified) response to a conditional GET request (Section 4.1 of [RFC7232]); a server MUST NOT send Content-Length in such a response unless its field-value equals the decimal number of octets that would have been sent in the payload body of a 200 (OK) response to the same request.

(you can send Content-Length on conditional GET; if you do, must match length of the response you would have sent for GET)

A server MUST NOT send a Content-Length header field in any response with a status code of 1xx (Informational) or 204 (No Content). A server MUST NOT send a Content-Length header field in any 2xx (Successful) response to a CONNECT request (Section 4.3.6 of [RFC7231]).

(some other situations that can't use Content-Length)

It may be an oversight, but nothing in that section requires that the Content-Length, in general, must match the size of the response body.

Re: On incomplete HTTP reads and the requests library in Python

#18

Post by tonyarkles »

dfox wrote: post #9

As for the compression and resulting discrepancy of length: if you are doing transparent HTTP compression, you are supposed to send Transfer-Encoding: gzip (which implies chunked mode and that Content-Length shoul not be present and should be ignored by UA). Content-Encoding: gzip means that the resource itself is gzip compressed (which means that the UA should decompress it for displaying to user, but should save it…

Ahhhhh... Thanks! I've always wondered why occasionally .tar.gz downloads end up ungzipped on disk (as just a .tar). Guess it was when using IE or Safari.

Re: On incomplete HTTP reads and the requests library in Python

#19

Post by wumpus »

dagenix wrote: post #16

This is one of the most frustrating issues I've had with the requests/urllib3 libraries. I have no idea what I'm supposed to do with half of an HTML page or a PDF or a PNG. And even if it did, I certainly don't know how I'm supposed to handle those things when I don't get an indication that I'm likely dealing with an incomplete response. I can construct a use case where I might want this behavior (maybe a FLIF image)…

Even after this is fixed the way you want it, webservers will still send you half of a PNG because it's a truncated file on the webserver. The webserver will give the truncated length as the length, the actual transfer will be the truncated length, requests will think all is well, and you'll have a problem.

Re: On incomplete HTTP reads and the requests library in Python

#20

Post by dagenix »

wumpus wrote: post #19
dagenix wrote: post #16

This is one of the most frustrating issues I've had with the requests/urllib3 libraries. I have no idea what I'm supposed to do with half of an HTML page or a PDF or a PNG. And even if it did, I certainly don't know how I'm supposed to handle those things when I don't get an indication that I'm likely dealing with an incomplete response. I can construct a use case where I might want this behavior (maybe a FLIF image)…

Even after this is fixed the way you want it, webservers will still send you half of a PNG because it's a truncated file on the webserver. The webserver will give the truncated length as the length, the actual transfer will be the truncated length, requests will think all is well, and you'll have a problem.

We can't fix every problem. And fixing every problem shouldn't be a gating function for fixing some problems. Web servers generally want to send a complete response. And sometimes some middle box chops off the response halfway through. And if that origin server set a correct Content-Length header and the response isn't that long, we can be fairly certain that the response is probably incomplete. And I really can't come up with a non-contrived use-case where we want those incomplete responses to be delivered silently to client code.

It seems like what you are saying is that because a file might be corrupt we shouldn't worry about a completely unrelated case where the file is fine but the transfer is incomplete. By that logic, why do we worry about trying to report errors at all?

Post reply on HN