Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

251–260 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#251

Post by runarberg »

speleding wrote: post #214
julik wrote: post #209

So many of these conversations could be easier if there would not be `length()` functions but `length_in_ ()` functions instead.

In ruby you have " ".codepoints.size == 5 and " ".bytes.size == 17 (It also has `length` which equals codepoints.size)

JavaScript is a weird one. To count UTF-16 bytes you write:

    " ".length
For unicode character count you write:

    [..." "].length
And for grapheme count (or language aware word/sentence count) you write:

    [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length
For word/sentence count you swap out the granularity option.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#252

Post by jameshart »

hateful wrote: post #225

The wife and I have a Google sheet that we use for our shared calendar - and we put an emoji before each "event" and in top row of each day I show the Emoji for that days entries. But I need to do: > =LEFT(F280,2) & LEFT(F281,2) & LEFT(F282,2) & LEFT(F283,2) Since the Emojis are actually 2 bytes.

And as this article indicates, many - and indeed an increasing number - of emojis are many more than 2 bytes. Flags, skin tones, groups, professions…

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#253

Post by ElectricalUnion »

rzwitserloot wrote: post #250

Java loaded full unicode code point semantics into its standard `java.lang.String` class. These _are not guaranteed_ to have `O(1)` performance characteristics, because the underlying storage format is dynamically either a UTF-16-esque variant (with surrogate pairs for characters that don't fit in 16 bit), or a single-byte-per-char format if the string does not contain any non-ASCII. This has the advantage of being v…

> The provided java snippet is vanilla valid

Did not you still need the `java --source 11 ${filename_without_java_extension_because_JEP_330}` to use it? And you still need a wrapper class with a static method main in it.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#254

Post by sundarurfriend »

omoikane wrote: post #56

Ruby gives you the choice to iterate over all types, via `each_byte`, `each_char`, `each_codepoint`, or `each_grapheme_cluster`. https://ruby-doc.org/3.2.2/String.html#class-String-label-Me...

In Julia, iterating over a string by default behaves like `each_char`.

`codeunits(str)` lets you access the underlying code units, which is bytes for the default UTF-8 encoding. (External packages implement UTF-16 and others, and there `codeunits` could return non-bytes, for eg. 16-bit values for UTF-16.)

The Unicode stdlib provides `graphemes(str)`, the equivalent of `each_grapheme_cluster`.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#256

Post by omoikane »

namaria wrote: post #122

I have read somewhere that you should learn 2 or 3 programming languages from the get go. If you learn one, you run the risk of letting it's shape dictate how you mentally model computation. At some point someone who learned a dynamically typed programming language first is bound to find out why data types matter.

I had a "programming languages" class that did that, where we did assignments in Python (scripting), OCaml (functional), and Prolog (logic). This is because most other classes used compiled imperative languages such as C++ and Java.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#258

Post by cylemons »

kps wrote: post #205
cylemons wrote: post #191

Earlier quoted context omitted.

I don't see what's inherently wrong with UTF-16 surrogates. If I am not wrong, a given UTF-16 codeunit is unambigously either a complete code point, a first surrogate, or a second surrogate. Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.

> Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF ar…

Is having a hole from U+D800 to U+DFFF such a big deal? The parent comment was specifically talking about surrogate pairs. That to me looks more like buggy implementation issue rather than standards issue.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#259

Post by masklinn »

MatthewWilkes wrote: post #242
masklinn wrote: post #113

Earlier quoted context omitted.

> Should this throw an error because there's only one displayed "character"? Why should it not? You’re literally breaking the content. Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. > Modern strings are complex objects that have evolved a bit past char[] or byte[]. And yet that’s exactly what you’re advocating, just with 21 bit chars.

One of the first things the author of the article does is breaks it down into the 5 code points and explains their individual meanings.

Your point being, what exactly?

If the user gives you what, as far as they're considered, is a glyph. And you return a completely different glyph. You've mangled their data.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#260

Post by tialaramex »

lxgr wrote: post #248
tialaramex wrote: post #223

Earlier quoted context omitted.

UCS-2 is dead. You can't express Unicode in UCS-2. If you have old UCS-2 data you can just treat it as UTF-16, maybe check for encoding irregularities but if it was really UCS-2 correct Unicode it'll be fine. UTF-16 length is only useful if you are moving UTF-16, perhaps for interop with other software that chose UTF-16. Remember to pass on your condolences and look forward to a day when we don't do that any more.

> UTF-16 length is only useful if you are moving UTF-16 [...] Remember to pass on your condolences and look forward to a day when we don't do that any more. Java/the JVM says hi! Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.

> Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.

Sure. On disk those strings are (modified) UTF-8 of course, is that what you meant ?

Post reply on HN