Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

131–140 of 287 posts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#131

Post by flohofwoe »

0x000xca0xfe wrote: post #123
flohofwoe wrote: post #121

Earlier quoted context omitted.

> Please stop pushing this UTF-8 everywhere nonsense. I was with you until this sentence. UTF-8 everywhere is great exactly because it is ASCII-compatible (e.g. all ASCII strings are automatically also valid UTF-8 strings, so UTF-8 is a natural upgrade path from ASCII) - both are just encodings for the same UNICODE codepoints, ASCII just cannot go beyond the first 127 codepoints, but that's where UTF-8 comes in and i…

I'm not advocating for ASCII-everywhere, I'm for bytes-everywhere. And bytes can conveniently fit both ASCII and UTF-8. If you want to restrict your programming language to ASCII for whatever reason, fine by me. I don't need "let wohnt_bei_Böckler_STRAẞE = ..." that much. But if you allow full 8-bit bytes, please don't restrict them to UTF-8. If you need to gracefully handle non-UTF-8 sequences graphically show the a…

> "let wohnt_bei_Böckler_STRAẞE"

This string cannot be encoded as ASCII in the first place.

> But if you allow full 8-bit bytes, please don't restrict them to UTF-8

UTF-8 has no 8-bit restrictions... You can encode any 21-bit UNICODE codepoint with UTF-8.

It sound's like you're confusing ASCII, Extended ASCII and UTF-8:

- ASCII: 7-bits per "character" (e.g. not able to encode international characters like äöü) but maps to the lower 7-bits of the 21-bits of UNICODE codepoints (e.g. all ASCII character codes are also valid UNICODE code points)

- Extended ASCII: 8-bits per "character" but the interpretation of the upper 128 values depends on a country-specific codepage (e.g. the intepretation of a byte value in the range between 128 and 255 is different between countries and this is what causes all the mess that's usually associated with "ASCII". But ASCII did nothing wrong - the problem is Extended ASCII - this allows to 'encode' äöü with the German codepage but then shows different characters when displayed with a non-German codepage)

- UTF-8: a variable-width encoding for the full range of UNICODE codepoints, uses 1..4 bytes to encode one 21-bit UNICODE codepoint, and the 1-byte encodings are identical with 7-bit ASCII (e.g. when the MSB of a byte in an UTF-8 string is not set, you can be sure that it is a character/codepoint in the ASCII range).

Out of those three, only Extended ASCII with codepages are 'deprecated' and should no longer be used, while ASCII and UTF-8 are both fine since any valid ASCII encoded string is indistinguishable from that same string encoded as UTF-8, e.g. ASCII has been 'retconned' into UTF-8.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#132

Post by zwnow »

dwb wrote: post #126
zwnow wrote: post #116

Earlier quoted context omitted.

[flagged]

You are in the wrong job if you don’t want to think about “nerd shit” while programming.

Idk it pays my bills and I have success at work so I must do something right.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#133

Post by birn559 »

CorrectHorseBat wrote: post #129
birn559 wrote: post #117

Earlier quoted context omitted.

I don't see why it should. I also believe parent is wrong as there are unambiguous rules about when to use ß or ss. Never thought of it but maybe there are rules that allow to visually present the code point for ß as ss? At least (from experience as a user) there seem to be a singular "ss" codepoint.

>also believe parent is wrong as there are unambiguous rules about when to use ß or ss. I never said it was ambiguous, I said it depends on the unicode version and the font you are using. How is that wrong? (Seems like the capital of ß is still SS in the latest unicode but since ẞ is the preferred capital version now this should change in the future)

> How is that wrong? Not sure where, how or if it's defined as part of Unicode, but so far I assumed that for a Unicode grapheme there exists a notion of what the visual representation should look like. If Unicode still defines capital of ß as SS that's an error in Unicode due to slow adaption of the changes in the German language.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#134

Post by birn559 »

guappa wrote: post #128
birn559 wrote: post #117

Earlier quoted context omitted.

I don't see why it should. I also believe parent is wrong as there are unambiguous rules about when to use ß or ss. Never thought of it but maybe there are rules that allow to visually present the code point for ß as ss? At least (from experience as a user) there seem to be a singular "ss" codepoint.

well I don't speak german, I was asking

I see, wasn't clear to me on what level you were asking. The letter ß has never been generally equivalent to ss in the German language.

From a user experience perspective though it might be beneficial to pretend that "ß" == "ss" holds when parsing user input.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#135

Post by tomsmeding »

guappa wrote: post #44
xigoi wrote: post #34

Earlier quoted context omitted.

In the case of Wordle, you know the exact set of letters you’re going to be using, which easily determines how to compute length.

No no, I want to create tomorrow's puzzle.

As the parent said:

> In the case of Wordle, you know the exact set of letters you’re going to be using

This holds for the generator side too. In fact, you have a fixed word list, and the fixed alphabet tells you what a "letter" is, and thus how to compute length. Because this concerns natural language, this will coincide with grapheme clusters, and with English Wordle, that will in turn correspond to byte length because it won't give you words with é (I think). In different languages the grapheme clusters might be larger than 1 byte (e.g. [1], where they're codepoints).

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#136

Post by tomsmeding »

zwnow wrote: post #116
jibal wrote: post #92

Earlier quoted context omitted.

The point is that those terms are ambiguous ... and if you mean the length in grapheme clusters, it can be quite expensive to calculate it, and isn't the right number if you're dealing with strings as objects that are chunks of memory.

[flagged]

If the validation rules don't specify (either explicitly or implicitly) what the "length" in the rule corresponds to (if it concerns a natural-language field, it's probably grapheme clusters), then either you should fix the rule, or you care only about checking the box of "I checked the validation rules", in which case it's a people problem and not a technology problem.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#137

Post by numpad0 »

flohofwoe wrote: post #121
0x000xca0xfe wrote: post #120

Earlier quoted context omitted.

Well I'm not American and I can tell you that we do not see English source code as imperialism. In fact it's awesome that we have one common very simple character set and language that works everywhere and can do everything. I have only encountered source code using my native language (German) in comments or variable names in highly unprofessional or awful software and it is looked down upon. You will always get an u…

> Please stop pushing this UTF-8 everywhere nonsense. I was with you until this sentence. UTF-8 everywhere is great exactly because it is ASCII-compatible (e.g. all ASCII strings are automatically also valid UTF-8 strings, so UTF-8 is a natural upgrade path from ASCII) - both are just encodings for the same UNICODE codepoints, ASCII just cannot go beyond the first 127 codepoints, but that's where UTF-8 comes in and i…

UTF-8 everywhere is not great and UTF-8 in practice is hardly ASCII-compatible. UTF-8 in source codes and file paths outside pure ASCII range breaks a lot of things especially on non-English systems due to legacy dependencies, ironically.

Sure, it's backward compatible, as in ASCII handling codes work on systems with UTF-8 locales, but how important is that?

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#138

Post by CorrectHorseBat »

birn559 wrote: post #133
CorrectHorseBat wrote: post #129

Earlier quoted context omitted.

>also believe parent is wrong as there are unambiguous rules about when to use ß or ss. I never said it was ambiguous, I said it depends on the unicode version and the font you are using. How is that wrong? (Seems like the capital of ß is still SS in the latest unicode but since ẞ is the preferred capital version now this should change in the future)

> How is that wrong? Not sure where, how or if it's defined as part of Unicode, but so far I assumed that for a Unicode grapheme there exists a notion of what the visual representation should look like. If Unicode still defines capital of ß as SS that's an error in Unicode due to slow adaption of the changes in the German language.

In unicode the default is still SS [1] while the Germans seem to have changed it to ẞ [2]. That means now it's the same on every system, but once the unicode standard changes and some systems get updated and others not there will be different behavior of len("ß".upper()) around.

I don't know how or if systems deal with this, but ß should be printed as ss if ß is unavailable in the font. It's possible this is completely up to the user.

[1] https://unicode.org/faq/casemap_charprop.html [2] https://www.rechtschreibrat.com/DOX/RfdR_Amtliches-Regelwerk...

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#139

Post by voidmain »

I haven't thought about this deeply, but it seems to me that the evolution of unicode has left it unparseable (into extended grapheme clusters, which I guess are "characters") in a forwards compatible way. If so, it seems like we need a new encoding which actually delimits these (just as utf-8 delimits code points). Then the original sender determines what is a grapheme, and if they don't know, who does?

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#140

Post by numpad0 »

simonask wrote: post #98
account42 wrote: post #94

Earlier quoted context omitted.

Actually, it would be great to have a lingua franca in every field that all participants can understand. Are you also going to complain that biologists and doctors are expected to learn some rudimentary Latin? English being dominant in computing is absolutely a strength and we gain nothing by trying to combat that. Having support for writing your code in other languages is not going to change that most libraries will…

Well, the problem is that what you are advocating is also that knowing Latin would be a prerequisite for studying medicine, which it isn't anywhere. That's the equivalent. Doctors learn a (very limited) Latin vocabulary as they study and work. You are severely underestimate how far you can get without any real command of the English language. I agree that you can't become really good without it, just like you can't d…

> Imagine learning programming using only your high school Spanish. Good luck.

This + translated materials + locally written books is how STEM fields work in East Asia, the odds of success shouldn't be low. There just needs to be enough population using your language.

Post reply on HN