Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node's Unicode Dragon

cirw.in

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Node's Unicode Dragon

#31

Post by est »

stormbrew wrote: post #27
est wrote: post #26

Earlier quoted context omitted.

Because some of us are pissed that some BMP characters takes 3 bytes in UTF8, that's 50% more waste of storage space and 50% more time to read/write. I like the design of Python 3.3 encoding. ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes. http://www.python.org/dev/peps/pep-0393/

That's a reasonable replacement for ucs-4 for an internal representation, but it's not actually a character encoding like utf-8 and utf-16 are. It's just a tagged union of several encodings. As for the inflation issue, 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 and 2 bytes in utf-16. It tends to even out somewhat. And if you really…

> 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8

For latin alphabets, yes. For CJK, it's really bad. Things get worse if you dealt with non-BMP before, like iOS emoji, which force you to upgrade MySQL to support utf8mb4, which is totally bullshit. (why the hell do people even presume utf8 is max 3 bytes?)

Re: Node's Unicode Dragon

#32

Post by acdha »

est wrote: post #30
sillysaurus2 wrote: post #28

Earlier quoted context omitted.

Storage space is cheap, and the price continues to fall. Storage of text is virtually nothing. Bandwidth to send text is almost nothing. Also, most text is compressed, which virtually eliminates that concern. Programmer time is at least two orders of magnitude more expensive than storage space or bandwidth for text.

> Storage space is cheap True, but 1. time is precious. For example, you waste 50% more time for a fulltext indexing scan because utf8 is longer. 2. Memory. If you can't hold text in a single machine, you have bigger issues (e.g. clustering algorithms, persistency, redundancy, etc.) 3. Network transfer. If you can save 50% in a db connection rtt, you save a lot. It makes no sense to save BMP in 3 bytes anyway.

You'd have to have rather weird data for it to be anywhere near 50% larger for real text (i.e. even if you only use Chinese, if you have punctuation, arabic numerals, quotes or URLs, HTML, etc. the averages cancel more than you might think) and a completely incompetent search engine design for that to remotely approach 50% more time to query or index.

If you were assigned the task of indexing the UTF-8 worst case corpus, nothing would stop you from designing a custom internal encoding while enjoying the many technical advantages UTF-8 gives you in every other area. Y internal details like compression are much easier to change than dealing with external interfaces which must be coordinated (this is why JavaScript still has such painful Unicode support even though browsers handle almost everything well in markup)

Re: Node's Unicode Dragon

#33

Post by est »

acdha wrote: post #32
est wrote: post #30

Earlier quoted context omitted.

> Storage space is cheap True, but 1. time is precious. For example, you waste 50% more time for a fulltext indexing scan because utf8 is longer. 2. Memory. If you can't hold text in a single machine, you have bigger issues (e.g. clustering algorithms, persistency, redundancy, etc.) 3. Network transfer. If you can save 50% in a db connection rtt, you save a lot. It makes no sense to save BMP in 3 bytes anyway.

You'd have to have rather weird data for it to be anywhere near 50% larger for real text (i.e. even if you only use Chinese, if you have punctuation, arabic numerals, quotes or URLs, HTML, etc. the averages cancel more than you might think) and a completely incompetent search engine design for that to remotely approach 50% more time to query or index. If you were assigned the task of indexing the UTF-8 worst case cor…

> nothing would stop you from designing a custom internal encoding while enjoying the many technical advantages ...

That's exactly how those UTF-X and UCS-Y encodings were invented, right?

The point is, this beast is called unicode, how ironic.

Re: Node's Unicode Dragon

#34

Post by pcwalton »

sillysaurus2 wrote: post #24
stormbrew wrote: post #19

Wish I'd known about this when I was pointing out in another HN thread how utf-16 is a terrible encoding for, among other reasons, pushing the corner case where you find out your encoding/decoding is broken to the very edge of likelihood. It's ridiculous that v8 doesn't properly support utf16, but it's to be expected I suppose. UTF-8 does not have this problem. That's the way we should be moving.

This. Why doesn't everybody use UTF-8? Nobody seems to have any problems with UTF-8. It seems to work almost perfectly, and it's efficient.

1. Controversy over Han unification made Unicode adoption less universal than might have been hoped.

2. Interoperability with legacy systems that don't use UTF-8 (for example, JavaScript). For example, Rust needs support for the full range of string encodings, because we need that support for implementing a browser engine.

Re: Node's Unicode Dragon

#35

Post by erichurkman »

sillysaurus2 wrote: post #28
est wrote: post #26

Earlier quoted context omitted.

Because some of us are pissed that some BMP characters takes 3 bytes in UTF8, that's 50% more waste of storage space and 50% more time to read/write. I like the design of Python 3.3 encoding. ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes. http://www.python.org/dev/peps/pep-0393/

Storage space is cheap, and the price continues to fall. Storage of text is virtually nothing. Bandwidth to send text is almost nothing. Also, most text is compressed, which virtually eliminates that concern. Programmer time is at least two orders of magnitude more expensive than storage space or bandwidth for text.

> Storage space is cheap, and the price continues to fall.

At-rest storage is cheap. Memory is cheaper than it used to be, but CPU cache is not. At some point the text will have to cross the CPU where every byte still counts.

Re: Node's Unicode Dragon

#36

Post by acdha »

est wrote: post #33
acdha wrote: post #32

Earlier quoted context omitted.

You'd have to have rather weird data for it to be anywhere near 50% larger for real text (i.e. even if you only use Chinese, if you have punctuation, arabic numerals, quotes or URLs, HTML, etc. the averages cancel more than you might think) and a completely incompetent search engine design for that to remotely approach 50% more time to query or index. If you were assigned the task of indexing the UTF-8 worst case cor…

> nothing would stop you from designing a custom internal encoding while enjoying the many technical advantages ... That's exactly how those UTF- X and UCS- Y encodings were invented, right? The point is, this beast is called uni code, how ironic.

The difference is internal vs. external: if you use UTF-8 things simply work better any time you exchange data. If you had lots of high code points (>\U+8000) you would get better results in most cases using a full compression system rather than playing with 2-byte encoding hacks, which is really all you're talking about in both cases.

Re: Node's Unicode Dragon

#37

Post by est »

acdha wrote: post #36
est wrote: post #33

Earlier quoted context omitted.

> nothing would stop you from designing a custom internal encoding while enjoying the many technical advantages ... That's exactly how those UTF- X and UCS- Y encodings were invented, right? The point is, this beast is called uni code, how ironic.

The difference is internal vs. external: if you use UTF-8 things simply work better any time you exchange data. If you had lots of high code points (>\U+8000) you would get better results in most cases using a full compression system rather than playing with 2-byte encoding hacks, which is really all you're talking about in both cases.

I totally agree that UTF-8 is pretty good at exchanging data because UTF8 is better than UCS-* and other UTF-* overall, and because everyone is (and should be) using it.

But you know, there are other cases besides exchanging. Like I said, if your text data is mainly latin you are good, but not so good if you are stuck with non-latin BMPs.

Re: Node's Unicode Dragon

#38

Post by deathanatos »

est wrote: post #26
sillysaurus2 wrote: post #24

Earlier quoted context omitted.

This. Why doesn't everybody use UTF-8? Nobody seems to have any problems with UTF-8. It seems to work almost perfectly, and it's efficient.

Because some of us are pissed that some BMP characters takes 3 bytes in UTF8, that's 50% more waste of storage space and 50% more time to read/write. I like the design of Python 3.3 encoding. ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes. http://www.python.org/dev/peps/pep-0393/

The good point (in my opinion) is not that "ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes", but rather that the exposed API hides this from you, and exposes to you a sequence of code points. This, I hope, will reduce errors, as code points, not code units, is often a better abstraction to be working with. (For some random string processing function.)

So far as I know, Haskell is the only other language that I know of that exposes, as the defaultish-native interface, Unicode strings as a sequence or iterable of code points (by just using UTF-32). Java, C#, your-language-here all do code units. C++'s template are powerful enough that someone could make unicode_str, but I've not seen one.

See: http://www.unicode.org/glossary/#code_point http://www.unicode.org/glossary/#code_unit

Re: Node's Unicode Dragon

#39

Post by kps »

est wrote: post #31
stormbrew wrote: post #27

Earlier quoted context omitted.

That's a reasonable replacement for ucs-4 for an internal representation, but it's not actually a character encoding like utf-8 and utf-16 are. It's just a tagged union of several encodings. As for the inflation issue, 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 and 2 bytes in utf-16. It tends to even out somewhat. And if you really…

> 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 For latin alphabets, yes. For CJK, it's really bad. Things get worse if you dealt with non-BMP before, like iOS emoji, which force you to upgrade MySQL to support utf8mb4, which is totally bullshit. (why the hell do people even presume utf8 is max 3 bytes?)

    > ... MySQL ...
    > why the hell do people even presume utf8 is max 3 bytes?
I think you answered your own question before you even asked.

Re: Node's Unicode Dragon

#40

Post by justin_vanw »

Man, I'm starting to think there is a cult around JSON.

If you need to accept arbitrary binary data, JSON is a profoundly bad choice. At a minimum, you would expect them to base64 encode the data and put that into a JSON string.

If you are looking at error reports, how is it even remotely acceptable to have them silently modified to include invalid unicode replacement characters?

The lesson here isn't some crappy hack workaround they found, it's a case study in the lengths you'll have to go to when you insist on making technology choices without considering the problem you want to solve.

Post reply on HN