Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

181–190 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#181

Post by coldtea »

ant6n wrote: post #168
coldtea wrote: post #115

Earlier quoted context omitted.

> length is not ambiguous at all. Its the number of elements in the array That's because you defined it first as "the number of elements in the array". It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave. Not to mention there might not be any "array". A string (whatever the encoding / representation) is a c…

The python doc says "str" are immutable sequences of unicode code points. Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing). I couldn't find out in the documentation whether the __getitem__ is O(1), which I consider a deficiency -- this should definitely be well documented. It doesn't really matter how some people think "how people understand" something, the document…

>Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing)

Well, weren't we talking about things being "ambiguous"?

In Python we call what you describe a list. An array is something different. And people would expect something like the C (or the Java) data structure. In Python that would match the "array" lib package.

And that's just discussing the meaning of array - before we even get to whether a string is an array, and what this means.

>It doesn't really matter how some people think "how people understand" something, the documentation matters

In what universe? In practical use, clarity and non-ambiguous, least surprise names and semantics matter.

"But we clarify it in page 2000 of the documentation" is not an excuse. Nor is invoking moral or professional failings of those not reading the documentation. A good library design doesn't offload clearing ambiguity to the documentation.

>Any string in any language is some ordered sequence of atomic text-like objects

You'd be surprised. Especially since this isn't 1985 where strings were a bunch of 8-bit ascii characters, or even 1995, when widechar 16-bit arrays were "good enough" for Windows and Java, but we have not just non-ascii strings, but even variable length (e.g. utf-8) internal strings in mainstream languages.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#182

Post by preommr »

namaria wrote: post #122

I have read somewhere that you should learn 2 or 3 programming languages from the get go. If you learn one, you run the risk of letting it's shape dictate how you mentally model computation. At some point someone who learned a dynamically typed programming language first is bound to find out why data types matter.

> should learn 2 or 3 programming languages from the get go.

So... Java/C#/Kotlin?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#183

Post by Spivak »

adgjlsfhk1 wrote: post #17
hgs3 wrote: post #15

Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.

Unicode defines text as a number of different types of things. They are sequences of codepoints, sequences of graphemes, sequences of graphime clusters. Furthermore, codepoints are different depending on how you normalize them. Accented characters can be written two different ways and have a different number of codepoints depending on how you write them (and if normalization is used)

Grapheme are a made up human thing that, while useful, is locale dependent. Most people when they talk about grapheme clusters mean the default "locale-independent" graphemes but it's not the only one (in Hungarian for example 'ly' is a single letter). Having the same string be two different lengths in two countries is… let's go with surprising. The common denominator where everyone computes the same number is code points.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#184

Post by infinitedata »

chrismorgan wrote: post #127
xeeeeeeeeeeenu wrote: post #34

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are…

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates , and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as ot…

‘Would that it were only so.’

What?

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#185

Post by stephencanon »

pavlov wrote: post #179
eviks wrote: post #174

Earlier quoted context omitted.

Showing unintuitive results in a small number of cases always beats showing unintuitive results for all the cases. The Swift approach can also asymptotically reach perfection. The other approach will be broken forever. These are categorical improvements in design

The Swift approach can't reach perfection in isolation because data from the future can always break it. That's why in the article you see Swift running on Ubuntu 14.04 returning len==2 while the same code on Ubuntu 18.04 returns len==1 for the same emoji string. IMO that's a big philosophical question here: do we accept that "string length" means something you can't compute for arbitrary strings unless your code is…

Swift includes its own Unicode data tables with the standard library since last year, so it’s now tied to the stdlib version rather than some other library that may or may not be updated on the system.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#186

Post by PhilipRoman »

I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data.

Even for limiting input field sizes byte count is much better, as otherwise you are opening up yourself for unicode denial of service. I think the game Minecraft has such an exploit where you can fit in absurd amounts of utf-8 data (to the point of data corruption in multiplayer games) since it's limited by visual length.

My personal favorite dealing with UTF-8: pretend it's ascii and assume everything above 128 is an alphabetic character. It just works. For 99% of use cases it doesn't matter if the content is emojis, families of emojis, or ancient sumerian scripts. You can parse JSON and most other formats this way without caring about code points at all. The trend of unicodizing everything was a mistake, just treat strings as bytes and parse them as utf-8 only when you really need it (like when building a text editor or a browser engine from scratch).

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#188

Post by chrismorgan »

infinitedata wrote: post #184
chrismorgan wrote: post #127

Earlier quoted context omitted.

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates , and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as ot…

‘Would that it were only so.’ What?

“I wish it was that simple, that it was only an implementation detail.” I suppose “would that—” isn’t a particularly common form these days.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#189

Post by Obscurity4340 »

Can anyone comment as to whether there are any problems associated with using emojiis to enhance the entropy of passwords? For passwords you only need to autofill but never actually type, I feel like it would be an easy way to augment passwords but I don't know whether it would directly translate in every situation.
Post reply on HN