Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

medium.com

Post reply on HN
Print view View original

81–90 of 170 posts

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#81

Post by bmn__ »

amelius wrote: post #64
chrismorgan wrote: post #17

Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane, and so stupid hacks like MySQL’s utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2002–2004) are now obviously insufficient for full Unicode support, and enough people care about them that adoption of this basic level of Unicode is driven, and so non-English speakers are inadverten…

> Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane When can we have user-defined glyphs/emoji, where SVG is embedded in the unicode string?

You can have that now, but it doesn't work the way you think. The SVG is embedded in the font, not the string. Demo: https://eosrei.github.io/emojione-color-font/full-demo.html

To create user-defined characters, you can occupy any code point in the reserved areas. https://en.wikipedia.org/wiki/Private_Use_Areas

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#82

Post by bradleyjg »

pwagland wrote: post #71
bradleyjg wrote: post #60

Earlier quoted context omitted.

With utf-8 you give up any hope of O(1) string operations. With utf-16 you can get O(1) string operations at a cost of doing them on code points rather than graphemes. If greater than 95% of the strings your language/program will ever see are going to be a single code point that trade off seems worth it. I think a lot of people totally discount the enormous cost that Latin-1 users are paying for CJK (etc) support the…

Except that with utf-16 you still have to give up any hope of O(1) string operations, unless you don't care about accuracy. Only UTF-32 allows O(1) index based operations. For example, 🧐 (aka face with monocle) needs 2 UTF-16 'characters', since the UTF code point is U+1F9D0, or, in UTF-8: F0 9F A7 90. And this ignored the point that others have made about combining characters, such as ‍ woman farmer Unicode: U+1F46…

In 2004 no one cared about face with monocle or woman farmer (I still don’t). The question was why did people pick UTF-16 back then. The answer is they were willing to make accuracy trade-offs for performance. Especially where they thought those trade-offs would only impact a small percentage of their users.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#83

Post by bradleyjg »

viraptor wrote: post #69
bradleyjg wrote: post #60

Earlier quoted context omitted.

With utf-8 you give up any hope of O(1) string operations. With utf-16 you can get O(1) string operations at a cost of doing them on code points rather than graphemes. If greater than 95% of the strings your language/program will ever see are going to be a single code point that trade off seems worth it. I think a lot of people totally discount the enormous cost that Latin-1 users are paying for CJK (etc) support the…

They'd kind of what happened in python 3.3. The internal representation is flexible, depending on the string contents. It may be just 1byte ASCII, or it may be utf32 if needed: https://www.python.org/dev/peps/pep-0393/

Java 11 is going to a similar model.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#84

Post by drb91 »

skywhopper wrote: post #74

This is a fascinating tale. But the #1 takeaway by the author is dangerous: "Database systems have subtle bugs and oddities, and you can avoid a lot of bugs by avoiding database systems." This is correct as far as it goes, but it's poor advice. You may as well say "Software has subtle bugs and oddities, and you can avoid a lot of bugs by avoiding software." True! But not helpful! If you don't need a relational databa…

Dangerous? How do you figure?

Many people assume a DB is the only way to store data. Now that’s dangerous.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#86

Post by sattoshi »

drb91 wrote: post #84
skywhopper wrote: post #74

This is a fascinating tale. But the #1 takeaway by the author is dangerous: "Database systems have subtle bugs and oddities, and you can avoid a lot of bugs by avoiding database systems." This is correct as far as it goes, but it's poor advice. You may as well say "Software has subtle bugs and oddities, and you can avoid a lot of bugs by avoiding software." True! But not helpful! If you don't need a relational databa…

Dangerous? How do you figure? Many people assume a DB is the only way to store data. Now that’s dangerous.

Because filesystems don't contain oddities...

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#87

Post by Piskvorrr »

bradleyjg wrote: post #60
chrismorgan wrote: post #33

Earlier quoted context omitted.

While I’m contemplating why people might have thought utf8mb3 was wise in 2004, I’ll ask a similar question that I’d love to know the answer to, of events from my early childhood: why did UTF-16 gain popularity? I get UCS-2 existing, but once it became evident it wasn’t enough, why make UTF-16 and ruin Unicode with surrogate pairs (which still cause trouble today, leading to things like WTF-8 to make it possible for…

With utf-8 you give up any hope of O(1) string operations. With utf-16 you can get O(1) string operations at a cost of doing them on code points rather than graphemes. If greater than 95% of the strings your language/program will ever see are going to be a single code point that trade off seems worth it. I think a lot of people totally discount the enormous cost that Latin-1 users are paying for CJK (etc) support the…

How exactly is Latin-1 privileged in this regard? Note its suffix - writing this from a region where other single-byte encodings proliferated in the 1990s, before sort-of settling to Latin-2 and/or Windows-1250 in the aughts (no, they're not the same, and the mapping is subtly broken).

If you mean "ASCII was good for our grandfathers", say so - but don't pretend that Latin-n somehow was not a bastardized set of hacks extended upon ASCII (like all "regional" encodings, single-byte or not), and shouldn't have died at least a decade ago: suddenly a wild Latin-10 string appears, and you're down to spinning up a conversion engine, praying that something didn't slip through the cracks (and inevitably something does, eventually). If your data never crosses a process boundary - by all means, keep it in Linear B, see if I care. Caveat: maintenance of your back-and-forth conversion engine will eat you alive, or iconv will obliterate the benefits of the "enormous" savings. (I also had the great idea "we'll just use a clever(tm) scheme and only have the necessary encoding parts," sometime around 2005: Memory and processing speed improvement was marginal, burden to keep juggling One Special Encoding (Latin-2 in my case) was not.)

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#88

Post by amelius »

bmn__ wrote: post #81
amelius wrote: post #64

Earlier quoted context omitted.

> Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane When can we have user-defined glyphs/emoji, where SVG is embedded in the unicode string?

You can have that now, but it doesn't work the way you think. The SVG is embedded in the font, not the string. Demo: https://eosrei.github.io/emojione-color-font/full-demo.html To create user-defined characters, you can occupy any code point in the reserved areas. https://en.wikipedia.org/wiki/Private_Use_Areas

That's nice, but I meant something else.

I was thinking: what if users of Whatsapp and other chat services could define their own emoji, as they type, and share them with other users? If a friend sends you a new emoji, you could simply copy it and use it in messages of your own. This functionality doesn't work with fixed codepoints.

Of course there's a million way to do this (even with today's Unicode standard), but it would be nice if this kind of functionality would be part of a standard, perhaps even at the level of Unicode.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#89

Post by jillesvangurp »

pwagland wrote: post #75
bhaak wrote: post #47

Earlier quoted context omitted.

I would disagree with the statement that UTF-16 gained popularity. UTF-16 is mostly used where it provided an easy upgrade path from the 16 bit restricted UCS-2 to enabling full Unicode coverage. Switching from UCS-2 to UTF-8 would have been a pain either because of dropping backward compatibility or ridiculous wrappers to get it working. Microsoft was one of the first companies to implement Unicode, so they didn't h…

Java 1.0 was made back in the days when "65K characters would be enough for anyone!" hence they chose UCS-2. Java only supported UTF-16 as of Java 5. I suspect that if they could turn back time, they would have chosen UTF-8 from the get-go, but, as you say, that was a battle lost long ago.

Java 9 released last year changed the internal string representation to not be UTF-16 anymore Up until then characters took at least 2 bytes in Java. Of course the internal representation and what is written to IO streams are two things. UTF-8 has been supported pretty much from day 1 along with a whole range of legacy encodings that used to be common.

Using UTF-8 instead of UTF-16 internally is mostly a memory optimization and does not actually break any apis that I'm aware off (I'm sure there are corner cases, but am not aware of any issues here). Using UTF-16 was always somewhat valid in non western scripts where the most used characters would be represented using two bytes instead of just 1 like for most western scripts. But you are right it was a bit of a regrettable choice.

If you consider that, e.g. asians mix western script with their local script, it is a lot less clear cut. Throw things like html in the mix and you are looking at a lot of ascii with bits of asian script here and there.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#90

Post by bradleyjg »

Piskvorrr wrote: post #87
bradleyjg wrote: post #60

Earlier quoted context omitted.

With utf-8 you give up any hope of O(1) string operations. With utf-16 you can get O(1) string operations at a cost of doing them on code points rather than graphemes. If greater than 95% of the strings your language/program will ever see are going to be a single code point that trade off seems worth it. I think a lot of people totally discount the enormous cost that Latin-1 users are paying for CJK (etc) support the…

How exactly is Latin-1 privileged in this regard? Note its suffix - writing this from a region where other single-byte encodings proliferated in the 1990s, before sort-of settling to Latin-2 and/or Windows-1250 in the aughts (no, they're not the same, and the mapping is subtly broken). If you mean "ASCII was good for our grandfathers", say so - but don't pretend that Latin-n somehow was not a bastardized set of hacks…

> How exactly is Latin-1 privileged in this regard?

Latin-1 is sufficient for languages which countries representing more than 50% of global GDP speak. It might not be fair but I’m not sure this is a question of fairness to begin with.

Post reply on HN