Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

211–220 of 287 posts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#211

Post by jibal »

moefh wrote: post #201
flohofwoe wrote: post #131

Earlier quoted context omitted.

> "let wohnt_bei_Böckler_STRAẞE" This string cannot be encoded as ASCII in the first place. > But if you allow full 8-bit bytes, please don't restrict them to UTF-8 UTF-8 has no 8-bit restrictions... You can encode any 21-bit UNICODE codepoint with UTF-8. It sound's like you're confusing ASCII, Extended ASCII and UTF-8: - ASCII: 7-bits per "character" (e.g. not able to encode international characters like äöü) but ma…

I'm not GP, but I think you're completely missing their point. The problem they're describing happens because file names (in Linux and Windows) are not text: in Linux (so Android) they're arbitrary sequences of bytes, and in Windows they're arbitrary sequences of UTF-16 code points not necessarily forming valid scalar values (for example, surrogates can be present alone). And yet, a lot of programs ignore that and in…

This is correct, both about what happens and about what your P is not understanding. Your P's claim that "UTF-8 has no 8-bit restrictions" is nonsense. It's too bad that your GP wasn't clearer about what the problem is: not all byte strings are valid UTF-8. This is why people have had to invent hacks like WTF-8 (https://news.ycombinator.com/item?id=9611710)

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#212

Post by zahlman »

deathanatos wrote: post #207
zahlman wrote: post #170

There's an awful lot of text in here but I'm not seeing a coherent argument that Python's approach is the worst, despite the author's assertion. It especially makes no sense to me that counting the characters the implementation actually uses should be worse than counting UTF-16 code units, for an implementation that doesn't use surrogate pairs (and in fact only uses those code units to store out-of-band data via the…

> JavaScript is compelled to count UTF-16 code units because it actually does use UTF-16. Python's flexible string representation is a space optimization; it still fundamentally represents strings as a sequence of characters, without using the surrogate-pair system. Python's flexible string system has nothing to do with this. Python could easily have had len() return the byte count, even the USV count, or other vastl…

> or other vastly more meaningful metrics than "5", whose unit is so disastrous I can't put a name to it. It's not bytes, it's not UTF-16 code units, it's not anything meaningful, and that's the problem.

The unit is perfectly meaningful.

It's "characters". (Pedantically, "code points" — https://www.unicode.org/glossary/#code_point — because values that haven't been assigned to characters may be stored. This is good for interop, because it allows you to receive data from a platform that implements a newer version of the Unicode standard, and decide what to do with the parts that your local terminal, font rendering engine, etc. don't recognize.)

Since UTF-32 allows storing every code point in a single code unit, you can also describe it that way, despite the fact that Python doesn't use a full 4 bytes per code point when it doesn't have to.

The only real problem is that "character" doesn't mean what you think it does, and hasn't since 1991.

I don't understand what you mean by "USV count".

> but what is a character?

It's what the Unicode standard says a character is. https://www.unicode.org/glossary/#character , definition 3. Python didn't come up with the concept; Unicode did.

> …but "5" or "7"? Where do those even come from?

From the way that the Unicode standard dictates that this text shall be represented. This is not Python's fault.

> Again: "character in the implementation" is a meaningless concept.

"Character" is completely meaningful, as demonstrated by the fact the Unicode Consortium defines it, and by the fact that huge amounts of software has been written based on that definition, and referring to it in documentation.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#213

Post by nitwit005 »

simonask wrote: post #98
account42 wrote: post #94

Earlier quoted context omitted.

Actually, it would be great to have a lingua franca in every field that all participants can understand. Are you also going to complain that biologists and doctors are expected to learn some rudimentary Latin? English being dominant in computing is absolutely a strength and we gain nothing by trying to combat that. Having support for writing your code in other languages is not going to change that most libraries will…

Well, the problem is that what you are advocating is also that knowing Latin would be a prerequisite for studying medicine, which it isn't anywhere. That's the equivalent. Doctors learn a (very limited) Latin vocabulary as they study and work. You are severely underestimate how far you can get without any real command of the English language. I agree that you can't become really good without it, just like you can't d…

You don't need to become fluent in Greek and Latin, but if you want to be able to read your patent's diagnosis, you're absolutely going to need to know the terms used. The standard names are in those languages.

And frequently, there is no other name. There are a lot of diseases, and no language has names for all of them.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#214

Post by justsomehnguy »

bigstrat2003 wrote: post #150
DavidPiper wrote: post #11

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a…

I have never wanted any of the things you said. I have, on the other hand, always wanted the string length. I'm not saying that we shouldn't have methods like what you state, we should! But your statement that people don't actually want string length is untrue because it's overly broad.

Guessing from the other comments you missed the byte length for the codepoints.

When I'm comparing the human-readable strings I want the letgth. In all other cases I want sizeof(string) and it's... quite a variable thing.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#215

Post by zahlman »

jibal wrote: post #210
zahlman wrote: post #158

Earlier quoted context omitted.

The point is that not all sequences of characters ("code point" means the integer value, whereas "character" means the thing that number represents) are valid.

non sequitur ... I simply pointed out a mistaken claim and your comment is about something quite different. (Also that's not what "character" means in the Unicode framework--some code points correspond to characters and some don't.) P.S. Everything about the response to this comment is wrong, especially the absurd baseless claim that I misunderstood the claim that I quoted and corrected (that's the only claim I respo…

> I simply pointed out a mistaken claim and your comment is about something quite different.

My comment explains that you have misunderstood what the claim is. "Byte code format" was nonsensical (Unicode is not interpreted by a VM), but the point that comment was trying to make (as I understood it) is that not all subsequences of a valid sequence of (assigned) code points are valid.

> Also that's not what "character" means in the Unicode framework--some code points correspond to characters and some don't.

My definition does not contradict that. A code point is an integer in the Unicode code space which may correspond to a character. When it does, "character" trivially means the thing that the code point corresponds to, i.e., represents, as I said.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#216

Post by kiitos »

zahlman wrote: post #209
PapaPalpatine wrote: post #202

Earlier quoted context omitted.

I don’t know about advanced Unicode features… but I use them all the time as a backend developer to validate data input. I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. This seems to have always been known as the length of the string. This thread sounds like a bunch of scientists trying to make a simple concept a lot harder to understand.

> I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. > This seems to have always been known as the length of the string. Sure. And by this definition, the string discussed in TFA (that consists of a facepalm emoji with a skin tone set) objectively has 5 characters in it, and therefore a length of 5. And it has always had 5 characters in it, si…

"character" is not a well defined concept in the context of this discussion

do you mean "byte"? or "rune"?

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#217

Post by TZubiri »

DavidPiper wrote: post #11

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a…

How about for iterating every character in a string in order to find a specific character combination? I need (or the iterator needs) to know the length of the string and what the boundaries of each characters are.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#218

Post by int_19h »

PapaPalpatine wrote: post #202
zahlman wrote: post #154

Earlier quoted context omitted.

> I have, on the other hand, always wanted the string length. In an environment that supports advanced Unicode features, what exactly do you do with the string length?

I don’t know about advanced Unicode features… but I use them all the time as a backend developer to validate data input. I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. This seems to have always been known as the length of the string. This thread sounds like a bunch of scientists trying to make a simple concept a lot harder to understand.

If you restrict the input to ASCII, then it makes sense to talk about "string length" in this manner. But we're not talking about Unicode strings at all then.

If you do allow Unicode characters in whatever it is you're validating, then your approach is almost certainly wrong for some valid input.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#219

Post by int_19h »

guappa wrote: post #45
taneq wrote: post #26

Earlier quoted context omitted.

If you're playing at this level, you need to define: - letter - word - 5 :P

Eh in macedonian they have some letters that in russian are just 2 separate letters

That's not really any different than the distinction (or lack thereof) between "ae" and "æ". For that matter, in Russian there is a letter "ы" which is historically a digraph consisting of two separately letters "ъ" and "i" that just happens to be treated as a single letter for so long that few people would even recognize it as a digraph. This kind of stuff is all language-specific, which is why for Worlde etc you always need to be aware of the context, and this context will then unambiguously decide what constitutes a single letter.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#220

Post by int_19h »

zwnow wrote: post #46
DavidPiper wrote: post #11

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a…

I actually want string length. Just give me the length of a word. My human brain wants a human way to think about problems. While programming I never think about bytes.

Humans speak many different languages. Not all of them are English, and not all of them have writing systems which make it meaningful to talk about "string length" without disambiguating further.
Post reply on HN