Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenBSD removes support for non-UTF8 locales

marc.info

Post reply on HN
Print view View original

131–140 of 186 posts

Re: OpenBSD removes support for non-UTF8 locales

#131

Post by geofft »

skybrian wrote: post #55
kragen wrote: post #3

I wonder what the pros and cons weighed in the discussion were. Clearly not supporting Unicode text in non-UTF-8 locales (except through, like, some kind of compatibility function, like recode or iconv) is the Right Thing. One problem that I have is that current UTF-8 implementations typically are not "8 bit clean", in the sense that GNU and modern Unix tools typically attempt to be; they crash, usually by throwing a…

Reminds me of Go strings: they usually store UTF-8 but they're actually 8-bit clean: "It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes." https://blog.golang.org/strings

It's only 8-bit clean if you don't poke it very hard. Try either of the last two loop examples in that page after adding "\xff\x80" to the string; you get two (indistinguishable) U+FFFD REPLACEMENT CHARACTERs in the iteration. So the loop destroys data, which UTF-8B specifically does not.

Also, it's a little disappointing that Go doesn't have a type-level way to say that a string is in fact UTF-8, not Latin-1 or something, and preferably that all values that inhabit that type are guaranteed to be valid and well-formed UTF-8. This is the cause of plenty of subtle bugs in Python 2, C, etc., which are all technically the result of programmer error, but in this decade, type systems should be helping us avoid common, subtle programmer errors.

Re: OpenBSD removes support for non-UTF8 locales

#132

Post by kragen »

kazinator wrote: post #108
ptx wrote: post #60

Earlier quoted context omitted.

Kuhn's idea is also used in in Python 3, so that garbage bytes can (optionally!) be decoded to Unicode strings and later losslessly turned back into the same bytes, which ensures (e.g.) that filenames that can't be decoded can still be used: https://www.python.org/dev/peps/pep-0383/

Interesting; I implemented exactly the same thing in the TXR language. I can read an arbitrary file in /bin/ as UTF-8 to a string, and when that string is converted to UTF-8, it reproduces that file exactly. All invalid bytes go to DCXX, including the null character. The code U+DC00 is called "pnul" (pseudo-null) and can even be written like #\pnul in the language as a character constant. Thanks to pnul, you can easi…

You and the Python guys should get together and make your hack compatible, and then pressure everyone else to standardize on it, instead of the horrible nightmare where every string input and output operation potentially corrupts data or crashes.

Re: OpenBSD removes support for non-UTF8 locales

#133

Post by kragen »

masklinn wrote: post #39
kragen wrote: post #3

I wonder what the pros and cons weighed in the discussion were. Clearly not supporting Unicode text in non-UTF-8 locales (except through, like, some kind of compatibility function, like recode or iconv) is the Right Thing. One problem that I have is that current UTF-8 implementations typically are not "8 bit clean", in the sense that GNU and modern Unix tools typically attempt to be; they crash, usually by throwing a…

> One problem that I have is that current UTF-8 implementations typically are not "8 bit clean", in the sense that GNU and modern Unix tools typically attempt to be; they crash, usually by throwing an exception Crashing on invalid data sounds like a great idea. Leaving garbage through doesn't.

You're the reason I can't use an "'" in my password on Citibank's web site, aren't you? I finally caught you, you bastard.

Re: OpenBSD removes support for non-UTF8 locales

#134

Post by geofft »

masklinn wrote: post #39
kragen wrote: post #3

I wonder what the pros and cons weighed in the discussion were. Clearly not supporting Unicode text in non-UTF-8 locales (except through, like, some kind of compatibility function, like recode or iconv) is the Right Thing. One problem that I have is that current UTF-8 implementations typically are not "8 bit clean", in the sense that GNU and modern Unix tools typically attempt to be; they crash, usually by throwing a…

> One problem that I have is that current UTF-8 implementations typically are not "8 bit clean", in the sense that GNU and modern Unix tools typically attempt to be; they crash, usually by throwing an exception Crashing on invalid data sounds like a great idea. Leaving garbage through doesn't.

Is it really garbage? If we want to be true to UNIX's (questionable) "Write programs to handle text streams, because that is a universal interface" ethos, our definition of "text" has to admit all possible byte strings to be "universal". And the so-called C locale historically did.

Re: OpenBSD removes support for non-UTF8 locales

#135

Post by Asbostos »

blackbeard wrote: post #21
RexRollman wrote: post #17

Earlier quoted context omitted.

Personally, I wish everyone used the 24:00 clock. Maybe the military has messed me up, but I really prefer seeing something like 18:22 over 6:22pm. It just seems simplier.

Yes couldn't agree more with that. Also dates in numeric order I.e. yyyy/mm/dd you know like all the other numbers we deal with not dd/mm/yyyy or the crazy mm/dd/yy.

Yes. Putting the year first and with 4 digits is the only safe way because, afaik, nobody anywhere uses yyyy/dd/mm. It's nice that it's also in order of significance but the main advantage is unambiguousness.

You'd think people would have learnt from Y2K but somehow we still see 2-digit years which make dates like 03/04/05 impossible to even guess at. It's slightly better since 2012 where a 2-digit year can't also be a month, but we'll have to wait till 2032 for 2-digit years to unambiguously mean year.

This is an area where I believe localization makes things worse, not better. If every website showed dates with the year first, people would easily understand, regardless of whatever silly local convention they have. As it is, whenever I see a ##/##/## date, I have to think about what the website might be trying to do (do they know what country I'm from? What country I'm in now? Are they using their own local convention?) and what possible dates it might mean. "I think that happened around August, so 08/10/14 is probably not the 8th of October." Localized dates just make no sense at all on the internet.

Re: OpenBSD removes support for non-UTF8 locales

#136

Post by Asbostos »

fletchowns wrote: post #37
jaawn wrote: post #31

Earlier quoted context omitted.

I think it might not be as bad as we imagine. All speedometers I've seen show both miles-per-hour and kilometers-per-hour, all thermostats I've seen have an option for switching to Celsius, all scales I've owned can display both pounds and kilograms (my current one even has an option for stones), all measuring cups I've seen in a long time have both systems displayed, etc. Even lumber wouldn't necessarily be a huge i…

You're not thinking about all the designs, tooling, manufacturing facilities, etc to build all that stuff though. That's where the real cost is. > Would it be a huge deal to round that to 40mm by 90mm? It actually would be a big deal to round things like that I think. Whole designs would need to be updated to take into account the new dimensions of things.

More manufacturing is done in China, and the factories are used to the different international markets because they're often parts suppliers for foreign companies that have their designing done in various countries. So they're well equipped to make things in either inches or mm.

Re: OpenBSD removes support for non-UTF8 locales

#137

Post by skybrian »

geofft wrote: post #131
skybrian wrote: post #55

Earlier quoted context omitted.

Reminds me of Go strings: they usually store UTF-8 but they're actually 8-bit clean: "It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes." https://blog.golang.org/strings

It's only 8-bit clean if you don't poke it very hard. Try either of the last two loop examples in that page after adding "\xff\x80" to the string; you get two (indistinguishable) U+FFFD REPLACEMENT CHARACTERs in the iteration. So the loop destroys data, which UTF-8B specifically does not. Also, it's a little disappointing that Go doesn't have a type-level way to say that a string is in fact UTF-8, not Latin-1 or some…

If you're juggling a bunch of different types of strings and want to keep them straight, Go does support defining separate types for them. An example is the HTML type from the template library [1].

The question is which sanitized string types are worth defining in the standard library. Presumably UTF-8 sanitized strings didn't make the cut.

Not sure about UTF-8B. Suppose the input is already UTF-8B? Do you double-escape it somehow?

It looks like DecodeRuneInString returns RuneError if it can't decode something and RuneError is defined as U+FFFD. The example uses a hard-coded string where it can't happen, so technically it's not a bug that it doesn't check for the error. But a linter might want to flag it.

[1] http://golang.org/pkg/html/template/#HTML

Re: OpenBSD removes support for non-UTF8 locales

#138

Post by Asbostos »

protomyth wrote: post #69
0x0 wrote: post #58

Earlier quoted context omitted.

You still have plenty of arbitrary-ness in your definition of a (nano-)second. :)

I'll take one level over two levels. Besides, memorizing that metric fraction is just wrong. Plus, do we really want to go into space with a unit of measure whose origins are tied to the size of the Earth? Let's at least free part of the definition from an Earth-centric bias. I fear a Mars-centric meter might develop.

Unlikely. The metric units have a long history of never having been redefined in a way that breaks previous usage. As far as I'm aware, all redefinitions have just improved the precision, so they don't hurt previous users. Your proposal would make all the old feet wrong and anybody seeing "foot" would have to think about which type of foot it might me.

Re: OpenBSD removes support for non-UTF8 locales

#139

Post by [deleted] »

kuschku wrote: post #79
gpvos wrote: post #72

Earlier quoted context omitted.

Interesting! This one keeps the 7-day week, which means it has a chance of succeeding. But wouldn't inserting an entire extra week every few years cause problems with things like monthly salaries and mortgage payments?

The financial year assumes 30 days per month, 12 months per year, 360 days per year. That’s what’s the basis for your salary calculation.

[deleted]

Re: OpenBSD removes support for non-UTF8 locales

#140

Post by cbd1984 »

feld wrote: post #105
nawitus wrote: post #8

Earlier quoted context omitted.

UTC is (as everyone knows) a bit problematic due to leap seconds. Different software systems handle the leap seconds somewhat differently. Handling leap seconds is is actually quite difficult if you want to get it absolutely correct. In 99% of cases the problems are just ignored (e.g. "it doesn't matter if the chart is slightly odd looking when you look at the moment of the leap second"). There's also the problem tha…

Easy, we just kill off the idea of leap seconds. Someone please convince Russia and UK to agree so we can do it.

> Easy, we just kill off the idea of leap seconds.

OK, now everyone who cares how much time has passed in terms of the Earth's rotation needs to keep a time separate from everyone else. Astronomers come to mind, for example.

Post reply on HN