241–250 of 315 posts
Python 3's approach is the most correct: Unicode defines text as a sequence of code points. UTF-whatever is an implementation detail.
Earlier quoted context omitted.
5 makes perfect sense to me; the author's complaints seem kinda silly. An area this makes sense is, what do you expect to get if you do something like: emoji = " " print(emoji[:3]) Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes? Modern strings are complex objects that have evolved a bit past cha…
> Should this throw an error because there's only one displayed "character"? Why should it not? You’re literally breaking the content. Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue. > Modern strings are complex objects that have evolved a bit past char[] or byte[]. And yet that’s exactly what you’re advocating, just with 21 bit chars.
I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are…
But it’s not. That style is about tone and the article doesn’t exude that kind of tone.
Do you see the author scolding programmers for being ignorant Americans, for having unknown unknowns, or for not being “professionals”? Well, me neither.
These emoticons should never have been a part of Unicode in the first place. Second big mistake of that org after the Unihan fiasco.
I agree that they introduced unnecessary complexity for text encoding, and for font-rendering (which are expected to support multi-coloured emoticons now). I once started writing on a text editor, and then fell deep into Unicode handling. I have now spent more work on the Unicode parts than on anything else in the program. I think that the industry could have instead adopted the old web-forum convention of colon-word…
HN discards emojis in the title. The original emoji was https://emojipedia.org/man-facepalming-medium-light-skin-ton... which consists of 5 Unicode code points. Also please make sure to read the first heading after the title, which summarizes the whole point of this essay.
Ok, we've remojied the title above. Edit: there are always exceptions - https://news.ycombinator.com/item?id=34460417
So many of these conversations could be easier if there would not be `length()` functions but `length_in_ ()` functions instead.
In ruby you have " ".codepoints.size == 5 and " ".bytes.size == 17 (It also has `length` which equals codepoints.size)
Same goes for .each_byte.size, but for that you have the faster .bytesize method that avoids the intermediate Enumerator.
Earlier quoted context omitted.
UCS-2 length is also something that is still required today.
UCS-2 is dead. You can't express Unicode in UCS-2. If you have old UCS-2 data you can just treat it as UTF-16, maybe check for encoding irregularities but if it was really UCS-2 correct Unicode it'll be fine. UTF-16 length is only useful if you are moving UTF-16, perhaps for interop with other software that chose UTF-16. Remember to pass on your condolences and look forward to a day when we don't do that any more.
Java/the JVM says hi!
Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.
I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are…
> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates , and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as ot…
UTF-16 is a hack. Unicode originally thought 65,535 values should be enough to represent all human languages and so 16-bit fixed size characters would work. However, that proved incorrect. UTF-16 was a hack to try retrofit this onto systems that had already adopted this 16 bit character (Java, Windows NT, etc).
An entertaining summary of the situation is here:
void main() {
String x = "(that emoji here)";
System.out.println("Chars: " + x.length());
System.out.println("Codepoints: " + x.codePointCount(0, x.length()));
System.out.println("As stream of chars (= UTF16-esque with surrogate pairs):");
x.chars().forEach(System.out::println);
System.out.println("As a stream of codepoints:");
x.codePoints().forEach(System.out::println);
}
Chars: 7
Codepoints: 5
As stream of chars (= UTF16-esque with surrogate pairs):
55358
56614
55356
57340
8205
9794
65039
As a stream of codepoints:
129318
127996
8205
9794
65039
The fact that the codepoint counter is a very awkward `codePointCount` call has the dubious benefit of highlighting this method loops through and therefore would be quite slow on very large strings.