Live data from Hacker News

In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

medium.com

131–140 of 170 posts

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#131
> "Once MySQL published this invalid character set, it could never fix it: that would force every user to rebuild every database. MySQL finally released UTF-8 support in 2010, with a different name: “utf8mb4”."

Had MySQL defined UTF-8 as up to 6 bytes initially, would they later force users to rebuilt their databases when UTF-8 was redefined to only 4 bytes?

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#132
post #101
post #97

Earlier quoted context omitted.

It was widely used before it was the default.

Certainly, but let’s not rewrite history here. What innodb could/couldn’t do was not the motivation.

Why not? Even as not the default, it was an official bundled storage engine, and the cutoff just happens to occur right at its boundaries, that doesn't sound like a coincidence.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#133

Earlier quoted context omitted.

> How exactly is Latin-1 privileged in this regard? Latin-1 is sufficient for languages which countries representing more than 50% of global GDP speak. It might not be fair but I’m not sure this is a question of fairness to begin with.

Oh. In that case, you do mean ASCII, methinks (which also works), with Latin-1 you're shooting yourself in both feet. (Looks for link to "falsehoods programmers believe about encodings")

To my understanding, ASCII basically only covers English and Italian well, whereas Latin-1 covers most Western European languages — which includes the main languages of the Americas (Spanish, English, French, Portuguese).

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#134

Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane, and so stupid hacks like MySQL’s utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2002–2004) are now obviously insufficient for full Unicode support, and enough people care about them that adoption of this basic level of Unicode is driven, and so non-English speakers are inadverten…

There’s nothing wrong with slicing at the byte level if you’re slicing with a delimiter like \n or “. UTF-8 sets high bits so continuation bytes never look like ASCII characters.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#135

Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane, and so stupid hacks like MySQL’s utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2002–2004) are now obviously insufficient for full Unicode support, and enough people care about them that adoption of this basic level of Unicode is driven, and so non-English speakers are inadverten…

> not Basic Multilingual Plane, and so stupid hacks like the utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2004) are now obviously insufficient for full Unicode support From [0]: UTF-8 was designed with a much larger limit of 231 (2,147,483,648) code points (32,768 planes), and can encode 221 (2,097,152) code points (32 planes) even if limited to 4 bytes What is wrong with U…

In case anyone is confused (since I know a lot of people see weird numbers and are just like, "Ugh, math hates me"), there's missing formatting in the above comment. It's supposed to read 2^31 and 2^21, not 231 and 221.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#136
post #88
post #81

Earlier quoted context omitted.

You can have that now, but it doesn't work the way you think. The SVG is embedded in the font, not the string. Demo: https://eosrei.github.io/emojione-color-font/full-demo.html To create user-defined characters, you can occupy any code point in the reserved areas. https://en.wikipedia.org/wiki/Private_Use_Areas

That's nice, but I meant something else. I was thinking: what if users of Whatsapp and other chat services could define their own emoji, as they type, and share them with other users? If a friend sends you a new emoji, you could simply copy it and use it in messages of your own. This functionality doesn't work with fixed codepoints. Of course there's a million way to do this (even with today's Unicode standard), but…

That sounds like a cool idea, but a complete pain to implement.

There's been bugs that crash iPhones with combining Arabic, or freeze Android devices by spamming LtR override.

There's the Zalgo meme, where you drown out your words in combining diacritics.

Layout engines are already struggling to keep up with the standard, even without the ability to render a UINT_MAX x UINT_MAX canvas of infinitely repeating fractal penises.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#137
post #10

> If you need a database, don’t use MySQL or MariaDB. Use PostgreSQL. This is stupid. There are pros and cons to every database. For example, MySQL allows you to set a trigger on an op, PostgreSQL requires you to write a function first.

> This is stupid. Talk about stupid, run this query on both Mysql and Postgres: select 0 = 'banana'; Make your own conclusion.

  mysql> select 0 = 'banana';
  +--------------+
  | 0 = 'banana' |
  +--------------+
  |            1 |
  +--------------+
  1 row in set, 1 warning (0.00 sec)
vs

  pg# select 0 = 'banana';
  ERROR:  invalid input syntax for integer: "banana"
  LINE 1: select 0 = 'banana';

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#138
post #11

Given that the article says that everyone should use utf8mb4, then why don't/haven't they just fixed it? Should be easy enough, and probably safe, too.

This can't silently be changed as the setting has impact on disk format for data and indexes and so and will be confusing when using different server versions, i.e. in replication.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#139

Earlier quoted context omitted.

Right, code units not code points. My mistake. Nonetheless that’s what you get back from e.g. java’s charAt.

You can do O(1) string operations in UTF-8 too, if you do them at the code unit level. It's just as wrong, but it's more obvious that it's wrong because it only works for ASCII instead of only working for BMP.

Bringing things full circle in this thread, this is absolutely why it is a big deal that emoji have become so popular and that people care about them, and that they are mostly homed in the Astral Plane. Now there's a giant corpus of UTF-16 data people are interacting with daily that absolutely makes it clear that you can't treat UTF-16 like UCS-2, and if you are still doing bad string operations in 2018 you have fewer excuses and more unhappy users ("why is my emoji broken?!").

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#140

Earlier quoted context omitted.

Right, code units not code points. My mistake. Nonetheless that’s what you get back from e.g. java’s charAt.

You can do O(1) string operations in UTF-8 too, if you do them at the code unit level. It's just as wrong, but it's more obvious that it's wrong because it only works for ASCII instead of only working for BMP.

Same thing I said above except even more so. What’s the most used language that can’t be represented with the BMP? Bengali is one possibility but AFAIK there’s a widely used Arabic form in use as well as the traditional script.
Post reply on HN