In MySQL, don’t use “utf8”, use “utf8mb4” (2016) • Hacker News Board

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#161

Post by chrismorgan » Fri, Jun 15, 2018, 3:20 AM UTC

zyx321 wrote: post #136

amelius wrote: post #88

Earlier quoted context omitted.

That's nice, but I meant something else. I was thinking: what if users of Whatsapp and other chat services could define their own emoji, as they type, and share them with other users? If a friend sends you a new emoji, you could simply copy it and use it in messages of your own. This functionality doesn't work with fixed codepoints. Of course there's a million way to do this (even with today's Unicode standard), but…

That sounds like a cool idea, but a complete pain to implement. There's been bugs that crash iPhones with combining Arabic, or freeze Android devices by spamming LtR override. There's the Zalgo meme, where you drown out your words in combining diacritics. Layout engines are already struggling to keep up with the standard, even without the ability to render a UINT_MAX x UINT_MAX canvas of infinitely repeating fractal…

It actually wouldn’t be as bad as you might imagine: pick whichever of the SVG-in-font specifications wins, and then all you need to do is apply that subset of SVG to the text, effectively allowing inline definition of the font. Roughly no new attack surface exposed in the text layout and rendering engine, given the existing ability to load untrusted fonts.

On further reflection, you would probably want some way to efficiently signal the dimensions of the SVG-mode glyph, so that the text layout engine doesn’t need the SVG parser, and only the renderer does. That would probably add at least one more code point reservation, depending on how you chose to do it.

But I’m only saying that it’s actually not that hard to do; I firmly believe it’s a bad idea, and I’m fairly confident it will never happen.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#162

Post by gpvos » Fri, Jun 15, 2018, 12:47 PM UTC

Piskvorrr wrote: post #108

bradleyjg wrote: post #90

Earlier quoted context omitted.

> How exactly is Latin-1 privileged in this regard? Latin-1 is sufficient for languages which countries representing more than 50% of global GDP speak. It might not be fair but I’m not sure this is a question of fairness to begin with.

Oh. In that case, you do mean ASCII, methinks (which also works), with Latin-1 you're shooting yourself in both feet. (Looks for link to "falsehoods programmers believe about encodings")

> with Latin-1 you're shooting yourself in both feet

Can you be more specific about that? As I wrote in a sibling comment, I'm not convinced. I mean, nowadays you shouldn't use any of the ISO-8859-* encodings anymore of course, but we're talking about the 1990s here.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#163

Post by Piskvorrr » Fri, Jun 15, 2018, 2:27 PM UTC

gpvos wrote: post #124

Piskvorrr wrote: post #108

Earlier quoted context omitted.

Oh. In that case, you do mean ASCII, methinks (which also works), with Latin-1 you're shooting yourself in both feet. (Looks for link to "falsehoods programmers believe about encodings")

Not really if you can make a good living writing software for only the US/UK, France, Germany, Italy and a few more Western European countries, which is perfectly feasible. Latin1 works perfectly fine there, and if you wanted, and you didn't have to interact too much with other systems, you could just completely ignore all other encodings until UTF-8 started to get traction in the later 1990s. It even got into many R…

Oh, you could have done that in 1990s and well into the aughts, no doubt about that - that's exactly what happened :) I thought you proposed Latin-1 as useful today.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#164

Post by Piskvorrr » Fri, Jun 15, 2018, 2:28 PM UTC

kilburn wrote: post #129

bradleyjg wrote: post #90

Earlier quoted context omitted.

> How exactly is Latin-1 privileged in this regard? Latin-1 is sufficient for languages which countries representing more than 50% of global GDP speak. It might not be fair but I’m not sure this is a question of fairness to begin with.

I beg to differ, considering that it doesn't contain the € sign.

The list of what it doesn't contain, with a 256 character space, is...long ;)

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#165

Post by gpvos » Fri, Jun 15, 2018, 4:20 PM UTC

Piskvorrr wrote: post #163

gpvos wrote: post #124

Earlier quoted context omitted.

Not really if you can make a good living writing software for only the US/UK, France, Germany, Italy and a few more Western European countries, which is perfectly feasible. Latin1 works perfectly fine there, and if you wanted, and you didn't have to interact too much with other systems, you could just completely ignore all other encodings until UTF-8 started to get traction in the later 1990s. It even got into many R…

Oh, you could have done that in 1990s and well into the aughts, no doubt about that - that's exactly what happened :) I thought you proposed Latin-1 as useful today .

Oh no, no way. I'm not 100% sure about bradleyjg though... Maybe I misunderstood them.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#166

Post by mmt » Fri, Jun 15, 2018, 7:37 PM UTC

gmac wrote: post #137

neverminder wrote: post #112

Earlier quoted context omitted.

> This is stupid. Talk about stupid, run this query on both Mysql and Postgres: select 0 = 'banana'; Make your own conclusion.

mysql> select 0 = 'banana'; +--------------+ | 0 = 'banana' | +--------------+ | 1 | +--------------+ 1 row in set, 1 warning (0.00 sec) vs pg# select 0 = 'banana'; ERROR: invalid input syntax for integer: "banana" LINE 1: select 0 = 'banana';

My conclusion (not just based on this example), is that MySQL favors (initial) ease [1], whereas PostgreSQL favors correctness [2].

To suggest the former approach is "stupid" is to fail to explain how it became so wildly popular. I allege that it is actually brilliant.. marketing.

MySQL marketed itself to developers by being not just free, but easy to work with. Need a database? Want to learn SQL? We won't bore you or slow you down with Codd and those pesky rules maaaan. It'll just work!

For the vast majority of use cases, it did exactly that.

Meanwhile, Postgres was esssentially sending the stodgy, old-fashioned message that databases are a hard problem that require careful up-front consideration so that one doesn't run into problems down the road, be they scale/performance or more severe like (even non-catastrophic) data loss.

That this message was, and still is, correct, just isn't very compelling in a rapid-protoyping fail-fast world. Unfortunately for those of us whose job it is to deal with the eventual consequences, it's too late by the time we're brought in [3].

I think we've seen a similar effect with many of the "NoSQL" datastores, as well. They gain initial popularity due to being easy, lightweight, and lightnight fast, but, as they mature and gain real-world use, we see articles that are shocked (shocked!) that there is data loss going on in this establishment and that, yes, if you need certain tedious features (like data integrity) from that stodgy old database world, you'll have to sacrifice that lightning fast performance.

[1] aka flexibility [2] aka standards adherence or even pedantry [3] though I've never quite understood the reasoning of why it would be so impractical to move RDBMSes when there's so often an ORM and no MySQL-specific features in use

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#167

Post by bradleyjg » Fri, Jun 15, 2018, 8:52 PM UTC

gpvos wrote: post #165

Piskvorrr wrote: post #163

Earlier quoted context omitted.

Oh, you could have done that in 1990s and well into the aughts, no doubt about that - that's exactly what happened :) I thought you proposed Latin-1 as useful today .

Oh no, no way. I'm not 100% sure about bradleyjg though... Maybe I misunderstood them.

I think Latin-1 compatible strings are common enough to be worth optimizing for with separate code paths. At least in large projects like OSes and programming languages. That doesn’t mean I think Unicode support should be omitted.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#168

Post by chrismorgan » Sat, Jun 16, 2018, 2:00 PM UTC

ithkuil wrote: post #160

chrismorgan wrote: post #67

Earlier quoted context omitted.

You’d just define two code points that switches into and out of SVG mode. (The second is only necessary for performance, so you can scan quickly without needing any sort of XML parser!)

What if nested element also containers an inline emosvg? :-)

Concerning the SVG table in OpenType fonts, https://docs.microsoft.com/en-au/typography/opentype/spec/sv... declares:

> any SVG and elements within a glyph description must be ignored and not rendered

The same limitations would be applied to this hypothetical SVG mode.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#169

Post by im3w1l » Sat, Jun 16, 2018, 7:17 PM UTC

morgo wrote: post #23

im3w1l wrote: post #19

Earlier quoted context omitted.

Do you have a collation were no two different strings compare equal? Have you considered imposing an arbitrary order where you don't know which one "should" go first?

Yes - there is a binary collation. w.r.t. arbitrary: The ordering must be deterministic so that indexes can maintain sorted order. It's possible to create a custom collation to have your own ordering - but I would say most users don't do that.

Sorry if I was unclear I meant some ordering that sorts by natural language but breaks ties in some arbitrary (but deterministic) manner, e.g. by binary.

Re: In MySQL, don’t use “utf8”, use “utf8mb4” (2016)

#170

Post by gpvos » Sun, Jun 17, 2018, 7:31 AM UTC

bradleyjg wrote: post #167

gpvos wrote: post #165

Earlier quoted context omitted.

Oh no, no way. I'm not 100% sure about bradleyjg though... Maybe I misunderstood them.

I think Latin-1 compatible strings are common enough to be worth optimizing for with separate code paths. At least in large projects like OSes and programming languages. That doesn’t mean I think Unicode support should be omitted.

In such cases I would think the optimization is basically for ASCII, and then extend that to Latin1 because it happens to be the first 256 characters of Unicode, which means processing is trivial and you don't waste the other 128 byte values that way. But I figure that 99.9% of those strings would be ASCII-only.