Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

231–240 of 287 posts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#231

Post by bobsmooth »

perching_aix wrote: post #224
deathanatos wrote: post #207

Earlier quoted context omitted.

> JavaScript is compelled to count UTF-16 code units because it actually does use UTF-16. Python's flexible string representation is a space optimization; it still fundamentally represents strings as a sequence of characters, without using the surrogate-pair system. Python's flexible string system has nothing to do with this. Python could easily have had len() return the byte count, even the USV count, or other vastl…

As the other comment says, Python considers strings to be a sequence of codepoints, hence the length of a string will be the number of codepoints in that string. I just relied on this fact yesterday, so it's kind of a funny timing. I wrote a little script that looks out for shenanigans in source files. One thing I wanted to explore was what Unicode blocks a given file references characters from. This is meaningless o…

I'm curious what you mean by "shenanigans" is that like emojis and zalgo text?

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#232

Post by ivanjermakov »

tralarpa wrote: post #37

Fascinating and annoying problem, indeed. In Java, the correct way to iterate over the characters (Unicode scalar values) of a string is to use the IntStream provided by String::codePoints (since Java 8), but I bet 99.9999% of the existing code uses 16-bit chars.

Codepoint is not cluster and cluster is not character. I bet there is "50 falsehoods about Unicode".

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#233

Post by ivanjermakov »

curtisf wrote: post #8
mrheosuper wrote: post #3

>We’ve seen four different lengths so far: Number of UTF-8 code units (17 in this case) Number of UTF-16 code units (7 in this case) Number of UTF-32 code units or Unicode scalar values (5 in this case) Number of extended grapheme clusters (1 in this case) We would not have this problem if we all agree to return number of bytes instead. Edit: My mistake. There would still be inconsistency between different encoding.…

"number of bytes" is dependent on the text encoding. UTF-8 code units _are_ bytes, which is one of the things that makes UTF-8 very nice and why it has won

I would say Unicode has won, but not UTF-8. UTF-16 is also widely used due to its efficiency on asian texts.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#234

Post by perching_aix »

bobsmooth wrote: post #231
perching_aix wrote: post #224

Earlier quoted context omitted.

As the other comment says, Python considers strings to be a sequence of codepoints, hence the length of a string will be the number of codepoints in that string. I just relied on this fact yesterday, so it's kind of a funny timing. I wrote a little script that looks out for shenanigans in source files. One thing I wanted to explore was what Unicode blocks a given file references characters from. This is meaningless o…

I'm curious what you mean by "shenanigans" is that like emojis and zalgo text?

I'm currently working with some local legacy code, so I primarily wanted to scan for incorrectly transcoded accented characters (central-european to utf-8 mishaps) - did find them.

Also good against data fingerprinting, homoglyph attacks in links (e.g. in comments), pranks (greek question mark vs. semicolon), or if it's a strictly international codebase, checking for anything outside ASCII. So when you don't really trust a codebase and want to establish a baseline, basically.

But I also included other features, like checking line ending consistency, line indentation consistency, line lengths, POSIX compliance, and encoding validity. Line lengths were of particular interest to me, having seen some malicious PRs recently to FOSS projects where the attacker would just move the payload out of sight to the side, expecting most people to have word wrap off and just not even notice (pretty funny tbf).

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#235

Post by aaron695 »

cmeacham98 wrote: post #5
bstsb wrote: post #2

ironic that unicode is stripped out the post's title here, making it very much wrong ;) for context, the actual post features an emoji with multiple unicode codepoints in between the quotes

Funny enough I clicked on the post wondering how it could possibly be that a single space was length 7.

[dead]

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#236

Post by degamad »

amake wrote: post #69
degamad wrote: post #52

Earlier quoted context omitted.

It's not just the variable byte length that causes an issue, in some ways that's the easiest part of the problem. You also have to deal with code points that modify other code points, rather than being characters themselves. That's a huge part of the problem.

That has nothing to do with UTF-8; that's a Unicode issue, and one that's entirely unescapable if you are the Unicode Consortium and your goal is to be compatible with all legacy charsets.

Yep, that's the point I was making - that choosing fixed 4-byte code-points doesn't significantly reduce the complexity of capturing everything that Unicode does.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#237

Post by degamad »

bawolff wrote: post #85
degamad wrote: post #52

Earlier quoted context omitted.

It's not just the variable byte length that causes an issue, in some ways that's the easiest part of the problem. You also have to deal with code points that modify other code points, rather than being characters themselves. That's a huge part of the problem.

That goes all the way back to the beginning Even ascii used to use "overstriking" where the backspace character was treated as a joiner character to put accents above letters.

Agreed, we just conveniently forget about those when speaking about how complex Unicode is.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#238

Post by koakuma-chan »

spyrja wrote: post #194
koakuma-chan wrote: post #151

Earlier quoted context omitted.

> You're also apparently insisting on three-letter variable names Why are the arguments not three-letter though? I would feel terrible if that was my code.

It's just a convention I use for personal projects. Back when I started coding in C, people often just opted to go with one or two character variable names. I chose three for locally-scoped variables because it was usually enough to identify them in a recognizable fashion. The fixed-width nature of it all also made for less eye-clutter. As for function arguments, the fact that they were fully spelled out made it easi…

Oh. The thing is, I really like my code formatted symmetrically, aligned evenly, etc. I go as far as adding empty comments to prevent the formatter from removing my custom line breaks. I thought you were the same, in a way ;)

e.g., https://github.com/mayo-dayo/app/blob/0.4/src/middleware.ts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#240

Post by eviks »

torstenvl wrote: post #182
account42 wrote: post #63

Earlier quoted context omitted.

> Number of code points when parsing. You shouldn't really ever care about the number of code points. If you do, you're probably doing something wrong.

I really wish people would stop giving this bad advice, especially so stridently. Like it or not, code points are how Unicode works. Telling people to ignore code points is telling people to ignore how data works. It's of the same philosophy that results in abstraction built on abstraction built on abstraction, with no understanding. I vehemently dissent from this view.

> Telling people to ignore code points is telling people to ignore how data works.

No, it's telling people that they're don't understand how data works otherwise they'd be using a different unit of measurement

Post reply on HN