Live data from Hacker News

CRLF is obsolete and should be abolished

fossil-scm.org

121–130 of 273 posts

Re: CRLF is obsolete and should be abolished

#121
They acted on these words, updating their HTTP server to serve just \n.

=> https://sqlite.org/althttpd/info/8d917cb10df3ad28 Send bare \n instead of \r\n for all HTTP reply headers.

While browser aren't effected, this broke compatibility with at least Zig's HTTP client.

=> https://github.com/ziglang/zig/issues/21674 zig fetch does not work with sqlite.org

Re: CRLF is obsolete and should be abolished

#122

Earlier quoted context omitted.

What? Which crazy non-binary format makes a distinction between CRLF(EOF) and just (EOF)? Apart from a plain text file, that is.

I won't mention telnet because you don't use it, but in CSV and similar data it is quite a trouble to normalize the data. So instead of 2 possibilities now we 3 to detect.

I have never had any issues with this using a standards compliant CSV parser.

Re: CRLF is obsolete and should be abolished

#123

Earlier quoted context omitted.

Apart from colloquially considering standards not-necessarily-normative being, in my opinion, nonsensical (see below), to the best of my knowledge at the very least the STD subseries of IETF standards documents are normative in nature: https://datatracker.ietf.org/doc/std > They are not the only tool, and they don't carry any moral force. Indeed there are countless other standards bodies in the world also producing n…

Ok, but just to be clear: the standards-track HTTP RFC says you can use a single LF. I don't think this issue is as clear as people seem to want it to be.

Sure. HTTP/1.1 isn't the only network protocol, though, IETF standardization or otherwise.

For SMTP (which this subthread started with):

   In addition, the appearance of "bare" "CR" or "LF" characters in text
   (i.e., either without the other) has a long history of causing
   problems in mail implementations and applications that use the mail
   system as a tool.  SMTP client implementations MUST NOT transmit
   these characters except when they are intended as line terminators
   and then MUST, as indicated above, transmit them only as a 
   sequence.
https://datatracker.ietf.org/doc/html/rfc5321#section-2.3.8

Re: CRLF is obsolete and should be abolished

#124

> various protocols (HTTP, SMTP, CSV) still "require" CRLF at the end of each line What would be the benefit to updating legacy protocols to just use NL? You save a handful of bits at the expense of a lot of potential bugs. HTTP/1(.1) is mostly replaced by HTTP/2 and later by now anyway. Sure, it makes sense not to require CRLF with any new protocols, but it doesn't seem worth updating legacy things. > Even if an est…

Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling. HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and…

What a weird reaction. Microsoft’s use of CRLF is an archaic pain in the ass. Taking a position that it should be deprecated isn’t radical or irresponsible — Microsoft makes gratuitous changes to things all of the time, why not this one?

Hipp is probably one of the better engineering leaders out there. His point of view carries weight because of who he is, but should be evaluated on its merits. If Microsoft got rid of this crap 30 years ago, when it was equally obsolete, we wouldn’t be having this conversation; if nobody does, our grandchildren will.

Re: CRLF is obsolete and should be abolished

#125
post #45

Earlier quoted context omitted.

Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling. HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and…

This would be more persuasive if HTTP servers didn't already widely accept bare 0ah line termination. What's the first major public web site you can find that doesn't?

Going down a list of top websites, these URLs respond with HTTP 200 (possibly after redirections) when sent an ordinary HTTP/1.1 GET request with 0D0A line endings, but respond with HTTP 400 when sent the exact same request with 0A line endings:

  https://br.pinterest.com/ https://www.pinterest.co.uk/
  https://apps.apple.com/ https://support.apple.com/ https://podcasts.apple.com/ https://music.apple.com/ https://geo.itunes.apple.com/
  https://ncbi.nlm.nih.gov/ https://www.salesforce.com/ https://www.purdue.edu/ https://www.playstation.com/
  https://llvm.org/ https://www.iana.org/ https://www.gnu.org/ https://epa.gov/ https://justice.gov/
  https://www.brendangregg.com/ http://heise.de/ https://www.post.ch/ http://hhs.gov/ https://oreilly.com/
  https://www.thinkgeek.com/ https://www.constantcontact.com/ https://sciencemag.org/ https://nps.gov/
  https://www.cs.mun.ca/ https://www.wipo.int/ https://www.unicode.org/ https://economictimes.indiatimes.com/
  https://science.org/ https://icann.org/ https://caniuse.com/ https://w3techs.com/ https://chrisharrison.net/
  https://www.universal-music.co.jp/ https://digiland.libero.it/ https://webaim.org/ https://webmd.com/
This URL responds with HTTP 505 on an 0A request:

  https://ed.ted.com/
These URLs don't respond on an 0A request:

  https://quora.com/
  https://www.nist.gov/
Most of these seem pretty major to me. There are other sites that are public but responded with an HTTP 403, probably because they didn't like the VPN or HTTP client I used for this test. (Also, www.apple.com is tolerant of 0A line endings, even though its other subdomains aren't, which is weird.)

Re: CRLF is obsolete and should be abolished

#126

Earlier quoted context omitted.

Apart from colloquially considering standards not-necessarily-normative being, in my opinion, nonsensical (see below), to the best of my knowledge at the very least the STD subseries of IETF standards documents are normative in nature: https://datatracker.ietf.org/doc/std > They are not the only tool, and they don't carry any moral force. Indeed there are countless other standards bodies in the world also producing n…

Ok, but just to be clear: the standards-track HTTP RFC says you can use a single LF. I don't think this issue is as clear as people seem to want it to be.

Can you provide a citation for this? I’ve read older RFCs that "recommend" recipients allow single LFs to terminate headers for robustness. I’ve also read newer RFCs that weaken that recommendation and merely say the recipient "MAY" allow single LFs. I’ve never noticed an HTTP RFC say you can send headers without the full CRLF sequence, but maybe I missed something.

https://datatracker.ietf.org/doc/html/rfc2616#section-19.3 https://datatracker.ietf.org/doc/html/rfc9112#section-2.2

Re: CRLF is obsolete and should be abolished

#127

Earlier quoted context omitted.

What? Which crazy non-binary format makes a distinction between CRLF(EOF) and just (EOF)? Apart from a plain text file, that is.

I won't mention telnet because you don't use it, but in CSV and similar data it is quite a trouble to normalize the data. So instead of 2 possibilities now we 3 to detect.

I don't get the CSV part. You can emit a new row after a line ending and on EOF with non-empty buffer. What's the tricky part or third option here? The crlf is never a part of the data.

Re: CRLF is obsolete and should be abolished

#128

Earlier quoted context omitted.

> I'm hoping this is satire. Me too. It's one thing to accept single LFs in protocols that expect CRLF, but sending single LFs is a bridge to far in my opinion. I'm really surprised most of the other replies to your comment currently seem to unironically support not complying with well-established protocol specifications under the misguided notion that it will somehow make things "simpler" or "easier" for developers.…

LF only? Huh. In its original terms for printing terminals, carriage return might be ambiguous. It could means either "just send the print head to column zero" or "print head to 0 and advance the line by one". The latter is what typewriters do for the Return key. But LF always meant Line Feed, moving the paper but not the print head. These are of course wildly out of date concepts. But it still strikes me as odd to s…

>The latter is what typewriters do for the Return key.

Minor correction: mechanical typewriters do not have a Return key, but they have both operations (line feed, as well as carriage return).

The carriage return lever is typically rigged to also do line feed at the same time, by a preset amount of lines (which can be set to 0), or you can push the carriage without engaging line feed.

Technically, the lever would do LF, and pushing on it further would do CR (tensioning the carriage spring).

It is, however, true that most of the time, the users would simply push the lever until it stops without thinking about it, producing CRLF operation —

— and that CR without LF was comparatively rare.

From a pure protocol UX perspective, it would make sense IMO to have a single command for (CR + LF) too, just like the typewriter effectively does it (push the lever here to do both at once).

It seems weird that the protocol is more limited than the mechanical device that it drives, but then again, designers probably weren't involved in deciding on terminal protocol specs.

Re: CRLF is obsolete and should be abolished

#129
post #58

Earlier quoted context omitted.

What's a "standard violation"? The original history of the IETF is a rejection of exactly this mode of thinking about the inviolability of standards, which was the ethos of the OSI.

When an implementation is noncomformant to a standard in question.

I've implemented a lot of protocols. Most implementations I've come across for most protocols not strictly standards conformant, for many reasons.

Big ones being:

* The standards are often not detailed enough, or contain enough loose verbage that there are many ways to understand how to implement some part, yet those ways are not interoperable.

* Many protocols allow vendor specifications in such a way that 2 implementations that are 100% compliant won't interoperate.

* Many protocol implementations are interoperable quite well, converging on behavior that isn't specified in any standard (often to the surprise of people who haven't read the relevant standards)

At least this is my experience for ietf rfc standards.

Re: CRLF is obsolete and should be abolished

#130
post #90
post #41

Counterpoint: Unix deciding on a non-standard line ending was always a mistake. It has produced decades of random incompatibility for no particular benefit. CRLF isn’t a convention: it’s two different pieces of the base terminal API. You have no idea how many programs rely on CR and LF working correctly.

It is a standard line ending. ANSI X3.4-1968 says: 10 LF (Line Feed). A format effector that advances the active position to the same character position on the next line. (Also applicable to display devices.) Where appropriate, this character may have the meaning “New Line” (NL), a format effector that advances the active position to the first character position on the next line. Use of the NL convention requires agr…

The first sentence is exactly what LF is in CRLF, and implies the necessity of CR. CR returns the cursor to the first character of the active line, LF moves it one line down without changing the horizontal position.

The second sentence is the UNIX interpretation of LF doing the equivalent of CRLF. But calling it a standard line ending when it's an alternative meaning defined in the standard as "requires agreement between sender and recipient of data" is a bit of a stretch. It's permissible by the standard, but it's not the default as per the standard

Post reply on HN