Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CRLF is obsolete and should be abolished

fossil-scm.org

Post reply on HN
Print view View original

131–140 of 273 posts

Re: CRLF is obsolete and should be abolished

#131

Post by pdw »

FiloSottile wrote: post #40
michaelmior wrote: post #3

> various protocols (HTTP, SMTP, CSV) still "require" CRLF at the end of each line What would be the benefit to updating legacy protocols to just use NL? You save a handful of bits at the expense of a lot of potential bugs. HTTP/1(.1) is mostly replaced by HTTP/2 and later by now anyway. Sure, it makes sense not to require CRLF with any new protocols, but it doesn't seem worth updating legacy things. > Even if an est…

Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling. HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and…

HTTP is saved here because headers aren't allowed to contain control characters. A server that is strict enough to only recognize CRLF will hopefully also be strict enough to reject requests that contain invalid characters.

The situation is different with SMTP, see https://www.postfix.org/smtp-smuggling.html

Re: CRLF is obsolete and should be abolished

#132

Post by tptacek »

LegionMammal978 wrote: post #125
tptacek wrote: post #45

Earlier quoted context omitted.

This would be more persuasive if HTTP servers didn't already widely accept bare 0ah line termination. What's the first major public web site you can find that doesn't?

Going down a list of top websites, these URLs respond with HTTP 200 (possibly after redirections) when sent an ordinary HTTP/1.1 GET request with 0D0A line endings, but respond with HTTP 400 when sent the exact same request with 0A line endings: https://br.pinterest.com/ https://www.pinterest.co.uk/ https://apps.apple.com/ https://support.apple.com/ https://podcasts.apple.com/ https://music.apple.com/ https://geo.itu…

You sure about this? www.pinterest.com, for instance, does not appear to care whether I 0d0a or just 0a.

Re: CRLF is obsolete and should be abolished

#133

Post by rtpg »

FiloSottile wrote: post #40
michaelmior wrote: post #3

> various protocols (HTTP, SMTP, CSV) still "require" CRLF at the end of each line What would be the benefit to updating legacy protocols to just use NL? You save a handful of bits at the expense of a lot of potential bugs. HTTP/1(.1) is mostly replaced by HTTP/2 and later by now anyway. Sure, it makes sense not to require CRLF with any new protocols, but it doesn't seem worth updating legacy things. > Even if an est…

Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling. HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and…

Took me a second to get what was going on here, but basically the idea is that you middleware might not see `C:D`, but then your application _does_ see `C:D`.

And given your application might assume your middleware does some form of access control (for example, `X-ActualUserForReal` being treated as an internal-only header), you could get around some access control stuff.

Not a bytes-alignment thing but a "header values disagreement" thing.

This is an issue if one part of your stack parses headers differently than another in general though, not limited to newlines.

Re: CRLF is obsolete and should be abolished

#134

Post by TacticalCoder »

anonymousiam wrote: post #79

This article seems like it was written to troll people into a flame war. There is no such character as NL, and the article does not at all address that fact that the "ENTER" key on every keyboard sends a CR and not a LF. Things work fine the way they are.

> There is no such character as NL ...

More specifically the Unicode control character U+000a is, in the Unicode standard, named both LF and NL (and that comes from ASCII but in ASCII I think 0x0a was only called LF).

It literally has both names in Unicode: but LINEFEED is written in uppercase while newline is written in lowercase (not kidding you). You can all see for yourself that U+000a has both names (and eol too):

https://www.unicode.org/charts/PDF/U0000.pdf

> and the article does not at all address that fact that the "ENTER" key on every keyboard sends a CR and not a LF.

what a key on a keyboard sends doesn't matter though. What matters is what gets written to files / what is sent over the wire.

    ... $  cat > /tmp/anonymousiam
    
    

    ... $  hexdump /tmp/anonymousiam
    00000000  000a
When I hit ENTER at my Linux terminal above, it's LINEFEED that gets written to the file. Under Windows I take it the same still gets CRLF written to the file as in the Microsoft OSes of yore (?).

> Things work fine the way they are.

I agree

Re: CRLF is obsolete and should be abolished

#135

Post by cassepipe »

michaelmior wrote: post #3

> various protocols (HTTP, SMTP, CSV) still "require" CRLF at the end of each line What would be the benefit to updating legacy protocols to just use NL? You save a handful of bits at the expense of a lot of potential bugs. HTTP/1(.1) is mostly replaced by HTTP/2 and later by now anyway. Sure, it makes sense not to require CRLF with any new protocols, but it doesn't seem worth updating legacy things. > Even if an est…

It seems to me the author is not suggesting to update the protocols themselves but rather to stop sending them CR even if the spec requires it. And to patch the corresponding software to it accepts simple newlines.

Re: CRLF is obsolete and should be abolished

#136

Post by rtpg »

tptacek wrote: post #45
FiloSottile wrote: post #40

Earlier quoted context omitted.

Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling. HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and…

This would be more persuasive if HTTP servers didn't already widely accept bare 0ah line termination. What's the first major public web site you can find that doesn't?

Gunicorn expects `\r\n` for lines (see gunicorn/http/message.py:read_line), though it's possible that every middleware that is in front of gunicorn in practice normalizes lines to avoid this issue.

Re: CRLF is obsolete and should be abolished

#137

Post by tfehring »

michaelmior wrote: post #3

> various protocols (HTTP, SMTP, CSV) still "require" CRLF at the end of each line What would be the benefit to updating legacy protocols to just use NL? You save a handful of bits at the expense of a lot of potential bugs. HTTP/1(.1) is mostly replaced by HTTP/2 and later by now anyway. Sure, it makes sense not to require CRLF with any new protocols, but it doesn't seem worth updating legacy things. > Even if an est…

At least for CSV, there's a divergence between usage in practice and the spec. The spec requires CRLF, but all of the commonly used tools I've encountered for reading and writing CSVs can read files with CR, LF, or CRLF line endings, and when writing CSVs they'll default to either LF or platform-specific line endings. (Even Excel for Mac doesn't default to CRLF!) I think this divergence is bad and should be fixed.

But IMO the right resolution is to update the spec so that (1) readers MUST accept any of (CR, LF, CRLF), (2) writers MUST use one of (CR, LF, CRLF), and (3) writers SHOULD use LF. Removing compatibility from existing applications to break legacy code would be asinine.

Re: CRLF is obsolete and should be abolished

#138

Post by zulu-inuoe »

Kwpolska wrote: post #120
djha-skin wrote: post #5

> Let's make CRLF one less thing that your grandchildren need to know about or worry about. The struggle is real, the problem is real. Parents, teach your kids to use .gitattribute files[1]. While you're at it, teach them to hate byte order marks[2]. 1: https://stackoverflow.com/questions/73086622/is-a-gitattribu... 2: https://blog.djhaskin.com/blog/byte-order-marks-must-diemd/

Nope. Git should not mess with line endings, the remote repository not matching the code in your local clone can bite you when you least expect it. On Windows, one should disable the autocrlf misfeature (git config --global core.autocrlf false) and configure their text editor to default to LF.

3000% agree. I have been bitten endlessly by autocrlf. It is absolutely insane to me that anyone ever considered your having your SOURCE CONTROL tool get/set different content than what's in the repo

Re: CRLF is obsolete and should be abolished

#139

Post by somat »

mackal wrote: post #77
FiloSottile wrote: post #40

Earlier quoted context omitted.

Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling. HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and…

> massive SQLite fan, but this is giving me pause about using other software by the same author Even if I wanted to contribute code to SQLite, I can't. I acknowledge the fact God doesn't exist, so he doesn't want my contributions :P

He does not want your code anyway, sqlite is public domain. this has several implications. One of which is the author wants nothing from you. Note that public domain is fundamentally different than the usual method of releasing code, which is to issue a license to distribute a copyright protected work. Putting a thing into the public domain is to renounce any ownership over the thing.

I think that the proper spirit of the thing is that if you have patches to sqlite is to just maintain them yourself. if you are especially benevolent you will put the patches in the public domain as well. and if they are any good perhaps the original author will want them.

In fact the public domain is so weird, some countries have no legal understanding of it. originally the concept was just the stance of the US federal government that because the works of the government were for the people, these works were not protected by copyright, and could be thought of as collectively owned by the people, or in the public domain. Some countries don't recognize this. everything has to be owned by someone. and sqlite was legally unable to be distributed in these countries, it would default to copyright with no license.

Re: CRLF is obsolete and should be abolished

#140

Post by dwattttt »

tptacek wrote: post #84
dwattttt wrote: post #72

Earlier quoted context omitted.

But no one expects 0ah to be sufficient. Change that expectation, and now you have to wonder if your middleware and your backend agree on whether the middleware filtered out internal-only headers.

Yeah, I'm not certain that this is a real issue. It might be? Certainly, I'm read in to things like TECL desync. I get the concern, that any disagreement in parsing policies is problematic for HTTP because of middleboxes. But I think the ship may have sailed on 0ah, and that it may be the case that you simply have to build HTTP systems to be bare-0ah-tolerant if you want your system to be resilient.

But what's bare-0ah-tolerant? Accepting _or_ ignoring bare 0ah's means you need to ensure all your moving parts agree, or you end up in the "one bit thinks this is two headers, others think it's one header".

The only situation where you don't need to know two policies match is when one of the policies rejects one of the combinations outright. Probably. Maybe.

EDIT: maybe it's better phrased as "all parts need to be bare-0ah-strict". But then it's fine if it's bare-0ah-reject; they just need to all be strict, one way or the other.

Post reply on HN