Live data from Hacker News

UTF-8 history (2003)

cl.cam.ac.uk

41–50 of 58 posts

Re: UTF-8 history (2003)

#41

Nice post, quite interesting. I think Rob Pike may have one of the coolest email addresses in the world (r@google.com).

Back when I worked at Google, a friend of mine got an unpleasant email from r@ because his Python script had a bug: it sent alerts to the individual characters of my-friend@google.com

Re: UTF-8 history (2003)

#42
post #31

Nice post, quite interesting. I think Rob Pike may have one of the coolest email addresses in the world (r@google.com).

I've heard his email causes issues for tons of internal systems at Google because a lot are coded with the expectation of a minimum of 3 characters for the left part of the email. May just be urban legend though.

Do "+" suffixes work for google.com addresses? Is r+pike@google.com the same account?

Re: UTF-8 history (2003)

#43
With all due respect, it seems to me that the documents contradict the claim in the initial email.

Rob's initial claim is that Ken came up with UTF-8 entirely from scratch, without even looking at IBM's proposal.

But Ken's oldest document from Sep 2, 1992 has his changes simply appended after the original FSS-UTF document from IBM (starting at "We define 7 byte types", as mentioned in the email), and notes the changes from original spec.

Then the final document that he sent out (Sep 8) is basically the FSS-UTF document with his changes applied (this is also mentioned in the email!).

There are two changes, basically:

1. Use 10 instead of 1 as the prefix for continuation bytes, so you can synchronize from an arbitrary location.

2. Once the bits are reassembled, use the value as-is instead of adding a bias constant, which simplifies the code at the cost of a tiny bit of packing efficiency.

Given the documentation provided, I would say that the fairest description of the development of UTF-8 is that IBM came up with the initial design, and Plan 9 made two improvements on it to produce the final UTF-8 design.

Re: UTF-8 history (2003)

#45
post #41

Nice post, quite interesting. I think Rob Pike may have one of the coolest email addresses in the world (r@google.com).

Back when I worked at Google, a friend of mine got an unpleasant email from r@ because his Python script had a bug: it sent alerts to the individual characters of my-friend@google.com

[deleted]

Re: UTF-8 history (2003)

#46

Earlier quoted context omitted.

Yes, the ultra-compact encoding will not be self-sequencing. But the bit-6-continuation variant yields more bits per byte, which would give better compression in many languages. Regarding efficiency, you still have to read every character, with the difference being one check every 1, 2, 3, or 4 characters in a more complex algorithm vs a check on every character in a simpler algorithm (I haven't checked to see which…

> huge data wastage when certain languages are encoded Is there a language that consistently uses codepoints with more than 2 bytes? It bothers me a bit that UTF-8 is not an infinitely extendable encoding. But that also isn't an important objection, because it is finite, but huge.

> Is there a language that consistently uses codepoints with more than 2 bytes?

Kanji, hiragana, katakana etc. At least 3 bytes with some 4-byte sequences. This compares unfavorably to for example Shift-JIS.

Re: UTF-8 history (2003)

#47

I believe these are photos of the exact diner. I can't confirm though. https://www.flickr.com/photos/ajstarks/albums/72157631470798... . Source: https://news.ycombinator.com/item?id=19565980

Fixed flickr url: https://www.flickr.com/photos/ajstarks/sets/7215763147079887...

Thank you.

Re: UTF-8 history (2003)

#48
post #31

Earlier quoted context omitted.

I've heard his email causes issues for tons of internal systems at Google because a lot are coded with the expectation of a minimum of 3 characters for the left part of the email. May just be urban legend though.

Do "+" suffixes work for google.com addresses? Is r+pike@google.com the same account?

Yes and no. Yes, they belong to the same employee; no, each google.com mailbox needs to be approved, they are not automatically redirected unlike regular GMail addresses.

Re: UTF-8 history (2003)

#49
Markus Kuhn did a great deal of work for unicode for unix, see the rest of the same folder, https://www.cl.cam.ac.uk/~mgk25/ucs/

such as examples/UTF-8-demo.txt a beautiful demo file for testing terminal emulators

or the wcwidth.c file I discovered was the origin of the same c function found in all OS's, i forked & maintain in python form as the public "wcwidth" module,

anyway just wanted to point out the treasure trove of the parent folder!

Re: UTF-8 history (2003)

#50
post #43

With all due respect, it seems to me that the documents contradict the claim in the initial email. Rob's initial claim is that Ken came up with UTF-8 entirely from scratch, without even looking at IBM's proposal. But Ken's oldest document from Sep 2, 1992 has his changes simply appended after the original FSS-UTF document from IBM (starting at "We define 7 byte types", as mentioned in the email), and notes the change…

One thing I was confused about. The document says there are 7 byte types, but I thought UTF-8 was variable width up to only 4 bytes. Did I misunderstand something?
Post reply on HN