Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xterm(1) now UTF-8 by default on OpenBSD

undeadly.org

Post reply on HN
Print view View original

31–40 of 144 posts

Re: Xterm(1) now UTF-8 by default on OpenBSD

#31

Post by kazinator »

cokernel_hacker wrote: post #28
kazinator wrote: post #25

Earlier quoted context omitted.

That was baseless posturing. A famous study and its follow-up found that the utilities on GNU/Linux are more robust, and that was twenty years ago: ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf [1995] " This study parallels our 1990 study (that tested only the basic UNIX utilities); all systems that we compared between 1990 and 1995 noticeably improved in reliability, but still had significant rat…

I don't think Rob meant stability. Rob was probably referring to the reality that modern Linux hasn't innovated itself past SVR4 by any appreciable amount. We are still using X, still using terminals powered by control codes, etc. Rob probably sees things like LANG and LC_ALL as bugs. His fix was UTF-8 everywhere, always. Where is Linux? Still in bag-of-bytes-o-rama.

I agree with Rob's "UTF-8 everywhere". I took this approach in the TXR language. Its I/O streams output and input UTF-8, and only that. Period. (There is no virtual switch for alternative encodings.) Internally, everything is a wide character code point. I do not call the "fuck my C program function" known as setlocale, and no behavior related to character handling or localization is influenced by magic environment strings.

LANG and LC_ALL are the work of ISO C and POSIX; they are not the fault of Linux. Linux has these in the name of compliance; they were foisted upon the free word, essentially.

Re: Xterm(1) now UTF-8 by default on OpenBSD

#32

Post by klodolph »

kazinator wrote: post #19

Great! Now just drop the embarrassing man(1) page reference, and you can call it modernized. Wow, I'm surprised that the people whose buttons this pushes are able to make(1) a HN account, let alone have enough points to downvote. Think about it. There is only one man page for xterm. I fyou type "man xterm" with no section number you get that man page. If there existed an xterm(7) page, you'd still get the xterm(1) ma…

Don't take the downvotes personally, it's just uninteresting content getting moderated.

Re: Xterm(1) now UTF-8 by default on OpenBSD

#33

Post by kbd »

igravious wrote: post #11
kbd wrote: post #6

Earlier quoted context omitted.

I have a stupid little 'clip' program I wrote that has a dictionary of common texts that I can call by name and have added to the clipboard. $ clip lod $ pbpaste ಠ_ಠ Maybe you can do the same without needing to remember code points. Something like TextExpander would accomplish the same thing.

I don't think that's stupid. I think that's a great idea. I might have to use 'xclip' to go about making something equivalent.

> I might have to use 'xclip' to go about making something equivalent.

Right now my code just shells out to pbcopy on Mac, but you may be interested in pyperclip[1] which provides cross-platform access to the clipboard.

[1] https://github.com/asweigart/pyperclip

Re: Xterm(1) now UTF-8 by default on OpenBSD

#34

Post by deathanatos »

Repeat after me: UTF-8 is the sane default in this day and age. This is a good change.

The whole "the ISO 6429 C1 control code 'application program command'" thing is a bit surprising though. (I'm guessing this change doesn't actually avoid this directly? If you sent an APC it'd still do it, it's just that APC is multiple bytes in UTF-8, and hopefully a bit rarer?)

> Reinterpreting US-ASCII in an arbitrary encoding

This way will likely work — at least, I thought. The vast majority of encodings are a superset of ASCII, so reinterpreting ASCII as them is valid. The only one I know of that isn't is EBCDIC, and I've never seen it used. (Said differently, non-superset-of-ASCII codecs are incredible rare to encounter, so the above assumption usually holds.) (The reverse, reinterpreting arbitrary data as ASCII, is not going to work out as well.)

Though it is rather horrifying how easily it is to dump arbitrary data into a terminals stream. Unix does not make this easy for the program. The vast majority of programs, I'd say, really just want to output text. Yet, they're connected to a terminal. Or better, if perhaps a program could say, "I'm outputting arbitrary binary data", or even "I'm outputting a application/tar+gzip"; the terminal would then know immediately to not interpret this input. And in the case of tar+gzip, it would have the opportunity to do something truly magical: it could visualize the octets (since trying to interpret a gzip as UTF-8 is insane); it could even just note that the output was a tar, and list the tar's contents like tar -t. If the program declares itself aware, like "application/terminal.ansi", then okay, you know: it's aware; interpret away.

But it doesn't, so it can't. Part of the difficulty is probably that the TTY is both input and output (not that the input can't also declare a mimetype or something similar). And the vast majority of programs don't escape their user input before sending it to a terminal; it's like one giant "terminal-XSS" or "SQL-injection-for-your-terminal". And it is probably unreasonable to expect it; I don't really know of any good libraries around terminal I/O; most programs I see that do it assume the world is an xterm and just encode the raw bytes, right there, and pray w.r.t. user input.

catting the linux kernel's gzip into tmux can have consequences from "lol" to "I guess we need a new tmux session".

It was also just today that I discovered that neither GNU's `ps` nor `screen` support Unicode, at least, for characters outside the BMP.

Re: Xterm(1) now UTF-8 by default on OpenBSD

#35

Post by cplease »

kazinator wrote: post #30
setra wrote: post #26

Earlier quoted context omitted.

Then the commercial systems must have been horrible. Take a look at GNU code, then at plan 9 code. Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. Granted Plan 9 came out a little passed 1990.

Only by not using subordinate clauses did you just avoid saying "plan 9" and "commercial" in the same sentence! Where is Plan 9 deployed? Who are the customers? Plan 9 is a strawman representative of "commercial Unix". > Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. When you actually sit down and think of the cases that can occur, that translates into code. Handle this, handle tha…

Just replying to point out that the grayed out comment above is the correct one.

There is ugliness in coreutils, but it mature, functional, proven ugliness. A lot of it is even there for a reason.

It's not difficult to make an elegant toy in isolation.

Re: Xterm(1) now UTF-8 by default on OpenBSD

#36

Post by setra »

cplease wrote: post #35
kazinator wrote: post #30

Earlier quoted context omitted.

Only by not using subordinate clauses did you just avoid saying "plan 9" and "commercial" in the same sentence! Where is Plan 9 deployed? Who are the customers? Plan 9 is a strawman representative of "commercial Unix". > Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. When you actually sit down and think of the cases that can occur, that translates into code. Handle this, handle tha…

Just replying to point out that the grayed out comment above is the correct one. There is ugliness in coreutils, but it mature, functional, proven ugliness. A lot of it is even there for a reason. It's not difficult to make an elegant toy in isolation.

checkout the different implementations of echo across various operating systems: "https://gist.github.com/dchest/1091803".

Re: Xterm(1) now UTF-8 by default on OpenBSD

#37

Post by SixSigma »

cokernel_hacker wrote: post #28
kazinator wrote: post #25

Earlier quoted context omitted.

That was baseless posturing. A famous study and its follow-up found that the utilities on GNU/Linux are more robust, and that was twenty years ago: ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf [1995] " This study parallels our 1990 study (that tested only the basic UNIX utilities); all systems that we compared between 1990 and 1995 noticeably improved in reliability, but still had significant rat…

I don't think Rob meant stability. Rob was probably referring to the reality that modern Linux hasn't innovated itself past SVR4 by any appreciable amount. We are still using X, still using terminals powered by control codes, etc. Rob probably sees things like LANG and LC_ALL as bugs. His fix was UTF-8 everywhere, always. Where is Linux? Still in bag-of-bytes-o-rama.

That and getting rid of the TTY altogether.

We aren't using punched cards

EDIT: people hate when I say this, which amuses me. The TTY must die !!!!

Re: Xterm(1) now UTF-8 by default on OpenBSD

#38

Post by SixSigma »

kazinator wrote: post #30
setra wrote: post #26

Earlier quoted context omitted.

Then the commercial systems must have been horrible. Take a look at GNU code, then at plan 9 code. Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. Granted Plan 9 came out a little passed 1990.

Only by not using subordinate clauses did you just avoid saying "plan 9" and "commercial" in the same sentence! Where is Plan 9 deployed? Who are the customers? Plan 9 is a strawman representative of "commercial Unix". > Combine a few GNU core utils, and you have more code than the whole plan 9 kernel. When you actually sit down and think of the cases that can occur, that translates into code. Handle this, handle tha…

While it is only a few :

Sydney Olympics lighting system was Plan9 based.

Inferno was used by NASA JPL projects

Lucent use a real time version of plan9 in phone masts

Coraid use Plan9 on their NAS servers

Researchers at LANL and IBM use plan9 on the Blue Gene, and other, supercomputers

I have worked for two plan9 based companies - ok they didn't survive but we tried :)

The international plan9 conferences drew about 30 people. People from commercial enterprises used plan9 in their workflows. Plan9 was my desktop while building a successful recruitment website.

Re: Xterm(1) now UTF-8 by default on OpenBSD

#39

Post by pilif »

cokernel_hacker wrote: post #28
kazinator wrote: post #25

Earlier quoted context omitted.

That was baseless posturing. A famous study and its follow-up found that the utilities on GNU/Linux are more robust, and that was twenty years ago: ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf [1995] " This study parallels our 1990 study (that tested only the basic UNIX utilities); all systems that we compared between 1990 and 1995 noticeably improved in reliability, but still had significant rat…

I don't think Rob meant stability. Rob was probably referring to the reality that modern Linux hasn't innovated itself past SVR4 by any appreciable amount. We are still using X, still using terminals powered by control codes, etc. Rob probably sees things like LANG and LC_ALL as bugs. His fix was UTF-8 everywhere, always. Where is Linux? Still in bag-of-bytes-o-rama.

>Rob probably sees things like LANG and LC_ALL as bugs. His fix was UTF-8 everywhere, always

The problems solved by LANG or LC_ALL are not solved by UTF8 alone. Even if you use UTF8 for all your input and output, there is still the question of how to format numbers and dates to the user and how to collate strings.

These things are dependent on country and language, sometimes even varying between different places in a single country (in Switzerland, the German speaking parts use . As the decimal separator, while the French speaking part prefers ,)

These things are entirely independent of the encoding of your strings and they still need to be defined. Also, because it's a very common thing that basically needs to happen with every application, this is also something the user very likely prefers to set only once at one place.

Environment variables don't feel too bad a place.

Re: Xterm(1) now UTF-8 by default on OpenBSD

#40

Post by 2trill2spill »

kazinator wrote: post #25
thisrod wrote: post #21

This reminded me of a Rob Pike comment. I can't find the text, but it was along the lines of, "I recently tried Linux. It was as if every bug I fixed in the 1980s had reverted."

That was baseless posturing. A famous study and its follow-up found that the utilities on GNU/Linux are more robust, and that was twenty years ago: ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf [1995] " This study parallels our 1990 study (that tested only the basic UNIX utilities); all systems that we compared between 1990 and 1995 noticeably improved in reliability, but still had significant rat…

> The maintainers of the free BSD's have been carrying that torch, but they don't believe in features. Stepping into a BSD variant is like a trip back to the 1980's. Not exactly the real 1980's, but a parallel 1980's in which Unix is more robust---but the features are all rolled back, so it's just about as unpleasant to use.

Sorry but that's all BS. FreeBSD is definitely a modern system, has lot's of features and is emphatically, millions of times better than a 1980's Unix. Linux may have a larger community than the BSD's but saying the BSD's are like stepping into the 1980's is rather disingenuous.

Post reply on HN