Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

121–130 of 287 posts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#121

Post by flohofwoe »

0x000xca0xfe wrote: post #120
simonask wrote: post #86

Earlier quoted context omitted.

This is American imperialism at its worst. I'm serious. Lots of people around the world learn programming from sources in their native language, especially early in their career, or when software development is not their actual job. Enforcing ASCII is the same as enforcing English. How would you feel if all cooking recipes were written in French? If all music theory was in Italian? If all industrial specifications we…

Well I'm not American and I can tell you that we do not see English source code as imperialism. In fact it's awesome that we have one common very simple character set and language that works everywhere and can do everything. I have only encountered source code using my native language (German) in comments or variable names in highly unprofessional or awful software and it is looked down upon. You will always get an u…

> Please stop pushing this UTF-8 everywhere nonsense.

I was with you until this sentence. UTF-8 everywhere is great exactly because it is ASCII-compatible (e.g. all ASCII strings are automatically also valid UTF-8 strings, so UTF-8 is a natural upgrade path from ASCII) - both are just encodings for the same UNICODE codepoints, ASCII just cannot go beyond the first 127 codepoints, but that's where UTF-8 comes in and in a way that's backward compatible with ASCII - which is the one ingenious feature of the UTF-8 encoding.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#122

Post by josephg »

account42 wrote: post #95
josephg wrote: post #75

Earlier quoted context omitted.

It’s a bit of a niche use case, but I use the codepoint counts in CRDTs for collaborative text editing. Grapheme cluster counts can’t be used because they’re unstable across Unicode versions. Some algorithms use UTF8 byte offsets - but I think that’s a mistake because they make input validation much more complicated. Using byte offsets, there’s a whole lot of invalid states you can represent easily. Eg maybe insert “…

> Eg maybe insert “a” at position 0 is valid, but inserting at position 1 would be invalid because it might insert in the middle of a codepoint. You have the same problem with code points, it's just hidden better. Inserting "a" between U+0065 and U+0308 may result in a "valid" string but is still as nonsensical as inserting "a" between UTF-8 bytes 0xC3 and 0xAB. This makes code points less suitable than UTF-8 bytes a…

I hear your point, but invalid codepoint sequences are way less of a problem than strings with invalid UTF8. Text rendering engines deal with weird Unicode just fine. They have to since Unicode changes over time. Invalid UTF8 on the other hand is completely unrepresentable in most languages. I mean, unless you use raw byte arrays and convert to strings at the edge, but that’s a terrible design.

> This makes code points less suitable than UTF-8 bytes as mistakes are more likely to not be caught during development.

Disagree. Allowing 2 kinds of bugs to slip through to runtime doesn’t make your system more resilient than allowing 1 kind of bug. If you’re worried about errors like this, checksums are a much better idea than letting your database become corrupted.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#123

Post by 0x000xca0xfe »

flohofwoe wrote: post #121
0x000xca0xfe wrote: post #120

Earlier quoted context omitted.

Well I'm not American and I can tell you that we do not see English source code as imperialism. In fact it's awesome that we have one common very simple character set and language that works everywhere and can do everything. I have only encountered source code using my native language (German) in comments or variable names in highly unprofessional or awful software and it is looked down upon. You will always get an u…

> Please stop pushing this UTF-8 everywhere nonsense. I was with you until this sentence. UTF-8 everywhere is great exactly because it is ASCII-compatible (e.g. all ASCII strings are automatically also valid UTF-8 strings, so UTF-8 is a natural upgrade path from ASCII) - both are just encodings for the same UNICODE codepoints, ASCII just cannot go beyond the first 127 codepoints, but that's where UTF-8 comes in and i…

I'm not advocating for ASCII-everywhere, I'm for bytes-everywhere.

And bytes can conveniently fit both ASCII and UTF-8.

If you want to restrict your programming language to ASCII for whatever reason, fine by me. I don't need "let wohnt_bei_Böckler_STRAẞE = ..." that much.

But if you allow full 8-bit bytes, please don't restrict them to UTF-8. If you need to gracefully handle non-UTF-8 sequences graphically show the appropriate character "�", otherwise let it pass through unmodified. Just don't crash, show useless error messages or in the worst case try to "fix" it by mangling the data even more.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#124

Post by amake »

Ekaros wrote: post #110
amake wrote: post #70

Earlier quoted context omitted.

What does "no combinations" mean?

Like say Ä it might be either Ä a single byte, or combination of ¨ and A. Both are now supported, but if you can have more than two such things going in one thing it makes a mess.

That's fundamental to the mission of Unicode because Unicode is meant to be compatible with all legacy character sets, and those character sets already included combining characters.

So "no combinations" was never going to happen.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#126

Post by dwb »

zwnow wrote: post #116
jibal wrote: post #92

Earlier quoted context omitted.

The point is that those terms are ambiguous ... and if you mean the length in grapheme clusters, it can be quite expensive to calculate it, and isn't the right number if you're dealing with strings as objects that are chunks of memory.

[flagged]

You are in the wrong job if you don’t want to think about “nerd shit” while programming.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#127

Post by guappa »

Ekaros wrote: post #119
guappa wrote: post #114

Earlier quoted context omitted.

That would be extremely wasteful, every single text file would be 4x larger and I'm sure eventually it would not be enough anyway.

Maybe we should have just replaced ascii, horrible encoding were entire 25% of it is wasted. And maybe we could have gotten a bit more efficiency by saying instead of having both lower and uppercase letters just have one and then have a modifier before it. Saving lot of space as most text could just be lowercase.

yeah that's how ascii works… there's 1 bit for lower/upper case.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#128

Post by guappa »

birn559 wrote: post #117
guappa wrote: post #112

Earlier quoted context omitted.

should "ß" == "ss" evaluate as true?

I don't see why it should. I also believe parent is wrong as there are unambiguous rules about when to use ß or ss. Never thought of it but maybe there are rules that allow to visually present the code point for ß as ss? At least (from experience as a user) there seem to be a singular "ss" codepoint.

well I don't speak german, I was asking

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#129

Post by CorrectHorseBat »

birn559 wrote: post #117
guappa wrote: post #112

Earlier quoted context omitted.

should "ß" == "ss" evaluate as true?

I don't see why it should. I also believe parent is wrong as there are unambiguous rules about when to use ß or ss. Never thought of it but maybe there are rules that allow to visually present the code point for ß as ss? At least (from experience as a user) there seem to be a singular "ss" codepoint.

>also believe parent is wrong as there are unambiguous rules about when to use ß or ss.

I never said it was ambiguous, I said it depends on the unicode version and the font you are using. How is that wrong? (Seems like the capital of ß is still SS in the latest unicode but since ẞ is the preferred capital version now this should change in the future)

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#130

Post by program »

chrismorgan wrote: post #53

Previous discussions: • https://news.ycombinator.com/item?id=36159443 (June 2023, 280 points, 303 comments; title got reemojied!) • https://news.ycombinator.com/item?id=26591373 (March 2021, 116 points, 127 comments) • https://news.ycombinator.com/item?id=20914184 (September 2019, 230 points, 140 comments) I’m guessing this got posted by one who saw my comment https://news.ycombinator.com/item?id=44976046 today, thou…

I did post this. I found it by chance, coming from this other post https://tonsky.me/blog/unicode/
Post reply on HN