Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CRLF is obsolete and should be abolished

fossil-scm.org

Post reply on HN
Print view View original

261–270 of 273 posts

Re: CRLF is obsolete and should be abolished

#261

Post by inopinatus »

tptacek wrote: post #212
inopinatus wrote: post #207

Earlier quoted context omitted.

That was already motivated by Postel's Law. It's a step beyond to change what the strict form is; relying on the same to justify unilaterally transposing the form is asking too much of middlebox implementations of just about any line-oriented protocol, and possible violates Postel's Law itself by asserting the inverse.

I don't believe in Postel's Law, but I also don't believe in reverential adherence to standards documents. Make good engineering decisions on their own merits. This article is right: CRLF is dumb. You know who agrees with me about that? The IETF, in their (very old) informational RFC about the origins of CRLF in their protocols.

Yes, CRLF is dumb. Trying to justify the problem seems unnecessary, it's widely acknowledged. A productive inquiry looks at why fixing it didn't happen yet. Don't confuse that line of thought for calling for more failure.

This is unrealistic, though:

> I don't believe in Postel's Law

All the systems around us that work properly do believe in it, and they will continue to do so. No-one who writes MTAs or reverse proxies &c is gonna listen to the wolves howling at the moon for change when there's no better plan that "ram it through unilaterally". Irrespective of what any individual may believe, Postel's Law remains axiomatic in protocol design & implementation.

More constructively, it may be that line-oriented protocols will only move towards change when they can explicitly negotiate line termination preferences during the opening handshake/banner/key exchange etc, which inevitably means a protocol revision in every case and very careful consideration of when CRLF is passed through anyway (e.g. email body).

Re: CRLF is obsolete and should be abolished

#262

Post by Spooky23 »

theamk wrote: post #150
Spooky23 wrote: post #124

Earlier quoted context omitted.

What a weird reaction. Microsoft’s use of CRLF is an archaic pain in the ass. Taking a position that it should be deprecated isn’t radical or irresponsible — Microsoft makes gratuitous changes to things all of the time, why not this one? Hipp is probably one of the better engineering leaders out there. His point of view carries weight because of who he is, but should be evaluated on its merits. If Microsoft got rid o…

No one is talking about Microsoft and whatever it does on its platform, the parent comment is about network protocols (HTTP, SMTP and so on..). I understand that it is tempting to blame Microsoft for \r\n proliferation, but it does not seem to be the case - the \r\n is comes from the era of teletypes and physical VT terminals. You can still see the original "NL" in action (move down only, do not go back to start of l…

The author of the post specifically addressed this:

“Today, CR is represented by U+000d and both LF and NL are represented by U+000a. Almost all modern machines use U+000a to mean NL exclusively. That meaning is embedded in most programming languages as the backslash escape \n. Nevertheless, a minority of machines still insist on sending a CR together with their NLs”

Who is the “minority”?

He also takes the position that the legacy behavior is fine for a tty, as it’s emulating a legacy terminal.

Re: CRLF is obsolete and should be abolished

#263

Post by lynx23 »

samatman wrote: post #64
lynx23 wrote: post #48

Can OP please tell me how to abolsih CR while in Raw Mode? Did he forget about it, or am I just unimaginative?

Right, you don't need to search that hard for a device which interprets 0xA as a line feed, just set your terminal to raw mode, done. But given the very first sentence: > CR and NL are both useful control characters. I'm willing to conclude that he doesn't intend A Blaste Against The Useless Appendage of Carriage Return Upon a New Line, or Line Feed As Some Style It , to apply to emulators of the old devices which ma…

I know that we're technically emulating old devices... But that time is so long gone. I actually never worked on a hardware terminal in my entire career, which is already almost 30 years. I think it is about time to stop calling it emulation, because thats no longer what it is. Its simply the way how text mode applications do I/O. It has become so ubiquitous that ncurses is slowly going out of fashion, because you might as well just use the common ANSI escape sequences, because they're supported everywhere anyways. IOW, raw mode isn't just an emulation required to get a 50 year old peripheral device to work, its necessary for almost everything that sits between an CLI and a GUI.

Re: CRLF is obsolete and should be abolished

#264

Post by immibis »

Aeolun wrote: post #260
immibis wrote: post #237

Earlier quoted context omitted.

The exploit is that your request went through a proxy which followed the standard (but failed to reject the bare NL) and the client sent a header after a bare NL which you think came from the proxy but actually came from the client - such as the client's IP address in a fake X-Forwarded-For, which the proxy would have removed if it had parsed it as a header. This attack is even worse when applied to SMTP because the…

Thanks, that was actually a very clear description of the problem! The problem here is not to use one or the other, but to use a mix of both.

And the standard is CRLF, so you're either following the standard or using a mix.

Re: CRLF is obsolete and should be abolished

#265

Post by samatman »

lynx23 wrote: post #263
samatman wrote: post #64

Earlier quoted context omitted.

Right, you don't need to search that hard for a device which interprets 0xA as a line feed, just set your terminal to raw mode, done. But given the very first sentence: > CR and NL are both useful control characters. I'm willing to conclude that he doesn't intend A Blaste Against The Useless Appendage of Carriage Return Upon a New Line, or Line Feed As Some Style It , to apply to emulators of the old devices which ma…

I know that we're technically emulating old devices... But that time is so long gone. I actually never worked on a hardware terminal in my entire career, which is already almost 30 years. I think it is about time to stop calling it emulation, because thats no longer what it is. Its simply the way how text mode applications do I/O. It has become so ubiquitous that ncurses is slowly going out of fashion, because you mi…

I agree with this, and generally take pains to refer to the programs as terminals, not terminal emulators. But at the same time, when a user presses the enter key and stdin provides CR, if you're in raw mode, you can get NL semantics by emitting CRLF to stdout, and that does in fact emulate the behavior of DEC terminals, which do that because teletypes did.

> I actually never worked on a hardware terminal in my entire career

I used to look books up at the library using a VT220. In the late 1990s they replaced that with an ASPX web browser endpoint running on PC hardware, and it was terrible. But I'm also not quite old enough to have used them for programming.

You're completely correct that it's no longer emulation of hardware terminals, there are dozens of input and output sequences which no hardware terminal ever used or understood. In many ways it's now emulation of XTerm, but even that era is slowly being left behind.

Re: CRLF is obsolete and should be abolished

#266

Post by nunobrito »

nightpool wrote: post #122
nunobrito wrote: post #101

Earlier quoted context omitted.

I won't mention telnet because you don't use it, but in CSV and similar data it is quite a trouble to normalize the data. So instead of 2 possibilities now we 3 to detect.

I have never had any issues with this using a standards compliant CSV parser.

Absolutely correct: You never had issues. Other people have built other software that does not use modern CSV libraries. They work just fine today and don't need to update, wouldn't even be possible for most cases in either case.

Please do consider that many software products will not change and they will still be actively used on production environments that you will never have interest about.

Re: CRLF is obsolete and should be abolished

#267

Post by nightpool »

nunobrito wrote: post #266
nightpool wrote: post #122

Earlier quoted context omitted.

I have never had any issues with this using a standards compliant CSV parser.

Absolutely correct: You never had issues. Other people have built other software that does not use modern CSV libraries. They work just fine today and don't need to update, wouldn't even be possible for most cases in either case. Please do consider that many software products will not change and they will still be actively used on production environments that you will never have interest about.

Well, GP proposed removing the idea of trailing newlines from standard *nix tools entirely, so I think it's fair to ask what shop is going to update their RHEL deployment to upgrade to a version of `cat` or `sed` that doesn't have trailing newlines but can't spend 5 minutes to handle a newline properly in their CSV parser? That doesn't make a lot of sense.

And it was pretty clear from the context of norir's comment that they were not talking about legacy software, they were talking about writing new projects/file formats that used newlines as a separator. Just because you want to shoehorn your legacy projects into this discussion doesn't mean that they fit.

Re: CRLF is obsolete and should be abolished

#268

Post by Kwpolska »

djha-skin wrote: post #238
Kwpolska wrote: post #120

Earlier quoted context omitted.

Nope. Git should not mess with line endings, the remote repository not matching the code in your local clone can bite you when you least expect it. On Windows, one should disable the autocrlf misfeature (git config --global core.autocrlf false) and configure their text editor to default to LF.

Sure, don't use autocrlf. But some _windows_ tools need crlf, like for powershell or batch build scripts. Defaulting to lf in the editor will not save you. Don't use `auto`, full marks, but the gitattributes file is indispensable as a safety net when explicit entries are used in it. I mean, the whole point of the file is not everyone who is working on the project has their editors set to lf. Furthermore, not every to…

When LF doesn’t work (e.g. in cmd), you can always use something like .editorconfig to tell editors to use CRLF, and you should keep those files as CRLF in the repo.

Re: CRLF is obsolete and should be abolished

#269

Post by Kwpolska »

layer8 wrote: post #164
Kwpolska wrote: post #120

Earlier quoted context omitted.

Nope. Git should not mess with line endings, the remote repository not matching the code in your local clone can bite you when you least expect it. On Windows, one should disable the autocrlf misfeature (git config --global core.autocrlf false) and configure their text editor to default to LF.

This is impractical in many situations, because tools that process build-source files (for example XML files that control the build, or generated source files) inherently generate CRLF on Windows. These are many, many, many tools, not just one’s text editor. The correct solution is to use .gitattributes.

If everyone’s on Windows, or if the tool always generates/requires CRLF, then you should store the files with CRLF line endings in the repository. In a mixed Windows/Linux environment, I would still prefer to handle this myself rather than expecting Git to mangle line endings.

Re: CRLF is obsolete and should be abolished

#270

Post by layer8 »

dwattttt wrote: post #176
layer8 wrote: post #164

Earlier quoted context omitted.

This is impractical in many situations, because tools that process build-source files (for example XML files that control the build, or generated source files) inherently generate CRLF on Windows. These are many, many, many tools, not just one’s text editor. The correct solution is to use .gitattributes.

If you're using tools that only support one particular line ending, the solution isn't to convert your source automatically in the repo, it's to know your source is always stored in one format, and convert it back/forth when a tool needs it to be different. How would you handle two different tools only supporting disjoint line endings?

The tools generally don’t care about the line endings on input, but they do generate their output using the platform’s line endings. If you don’t normalize the line-endings upon commit, then using the tools on Unix and Windows will alternatingly switch the line endings in the repo between LF and CRLF. And you don’t want to have those constant changes in your history and in the diffs.

Having Git normalize line endings for the relevant file types upon check-in is by far the simplest solution to this problem, in particular since having .gitattributes in the repository means that all clients automatically and consistently perform the same normalization.

Post reply on HN