The dangers of streaming across versions of glibc: A cautionary tale (2014) • Hacker News Board

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#31

Post by barrkel » Tue, Jan 08, 2019, 5:24 PM UTC

leeter wrote: post #22

barrkel wrote: post #17

About 20 years ago, I implemented a hash table that used binary trees for buckets. It was supposed to support Unicode strings as keys, and I used the Windows comparison functions to figure out how to compare the keys, in order to handle case insensitive lookup IIRC. I tested the table with randomly generated strings, and was puzzled to discover expected lookups would fail with some table constructions. I narrowed it…

IIRC the newer Unicode collation sensitive comparison functions hadn't been implemented until Vista Looks like that's the case: https://docs.microsoft.com/en-us/windows/desktop/api/stringa...

The thing I most remember is the surprise of intransitive collation order. I also recall implementing case insensitive lookup.

It might have been an early version of this one: http://codecentral.embarcadero.com/Item/15171 - from 2001 - but I've written quite a few hash tables over the years, and may be blending the different recollections. I never used Vista, and I'm pretty sure my experience predated Windows 7.

I also wrote JclStrHashMap to support JclExprEval: https://github.com/project-jedi/jcl/blob/master/jcl/source/c...

I also wrote the Delphi runtime library TDictionary generic implementation, but that was more recent.

Update I found it: original discussion, from newsgroups, has been ported to the web: http://www.delphigroups.info/2/62/478610.html

It was from 2004, so more like 15 years ago. It's mildly painful to read myself from back then too :)

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#32

Post by jjnoakes » Tue, Jan 08, 2019, 5:26 PM UTC

Someone wrote: post #26

gumby wrote: post #12

This isn't a glibc issue; really it's a story that "character collations are not stable over time" (mentioned downthread in the article in a message quoting Unicode Technical Standard #10). Which is a cautionary tale the solution for which is to use icu, the solution adopted by Postgres.

ICU isn’t the solution. If a bug in its collation routines is found, it likely will be fixed. You don’t want that if retrieving your data requires perfect stability of string comparisons. The only real solution is to have your own collation routines, test the hell out of them for every release (you don’t want a compiler bug or the fixing of a compiler bug to introduce a subtle change in your collation code. The truly…

I imagine the database would use versioned collation, so a fixed bug would have to be opted-into by whomever owns the tables, no?

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#33

Post by cryptonector » Tue, Jan 08, 2019, 5:28 PM UTC

rraval wrote: post #16

gumby wrote: post #12

This isn't a glibc issue; really it's a story that "character collations are not stable over time" (mentioned downthread in the article in a message quoting Unicode Technical Standard #10). Which is a cautionary tale the solution for which is to use icu, the solution adopted by Postgres.

> the solution for which is to use icu We ran headfirst into this issue at my company and we've actually been recommending the opposite (use the "C" locale on the database, treat collation as a render level concern). I have a whole write up explaining the technical motivations behind that recommendation: https://gist.github.com/rraval/ef4e4bdc63e68fe3e83c9f98f56af...

Using the C locale certainly helps, but do watch out: you still need to normalize. That means you still need something like ICU.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#34

Post by j16sdiz » Tue, Jan 08, 2019, 5:31 PM UTC

Someone wrote: post #26

gumby wrote: post #12

This isn't a glibc issue; really it's a story that "character collations are not stable over time" (mentioned downthread in the article in a message quoting Unicode Technical Standard #10). Which is a cautionary tale the solution for which is to use icu, the solution adopted by Postgres.

ICU isn’t the solution. If a bug in its collation routines is found, it likely will be fixed. You don’t want that if retrieving your data requires perfect stability of string comparisons. The only real solution is to have your own collation routines, test the hell out of them for every release (you don’t want a compiler bug or the fixing of a compiler bug to introduce a subtle change in your collation code. The truly…

IBM use ICU for DB2 collation. It have versioned collation that do bug-to-bug backward compatibly.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#35

Post by jakobegger » Tue, Jan 08, 2019, 5:55 PM UTC

anarazel wrote: post #27

jakobegger wrote: post #15

Earlier quoted context omitted.

Even if you use ICU, you need to make sure you link with the same version of ICU! Edit: Why the downvote? New versions of Unicode come out every year, and ICU collations are explicitely versioned. If you use ICU 51.1 and ICU 52.1 you are going to have the same kind of issues

ICU collations in PG are versioned, and ICU has support for accessing the different collation versions. So you might be stuck on an older collation version without explicit action, but it'll not yield wrong results.

Really? PostgreSQL can use an index created with an older version of ICU? I didn't know that.

Edit: The docs say indices need to be rebuilt when ICU version changes: https://www.postgresql.org/docs/11/sql-altercollation.html

ICU offers a way to detect the collation version has changed, but I thought there was no way to access older collation versions.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#36

Post by alexchamberlain » Tue, Jan 08, 2019, 6:36 PM UTC

May I ask a stupid question? What's the difference between collation and encoding? Some people in this thread have suggested "use byte strings"; is it "enough" to "just" use UTF-8, like you should "just" use UTC (unless you know/Should know better)?

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#37

Post by BeeOnRope » Tue, Jan 08, 2019, 6:39 PM UTC

alexchamberlain wrote: post #36

May I ask a stupid question? What's the difference between collation and encoding? Some people in this thread have suggested "use byte strings"; is it "enough" to "just" use UTF-8, like you should "just" use UTC (unless you know/Should know better)?

Collation is basically a fancy way of saying "sorting" and should be independent of encoding (or course, certain encodings lend themselves more naturally to certain collation algorithms). In general collation needs to be locale-aware while encoding (in Unicode) largely doesn't care.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#38

Post by BeeOnRope » Tue, Jan 08, 2019, 7:19 PM UTC

rraval wrote: post #16

gumby wrote: post #12

This isn't a glibc issue; really it's a story that "character collations are not stable over time" (mentioned downthread in the article in a message quoting Unicode Technical Standard #10). Which is a cautionary tale the solution for which is to use icu, the solution adopted by Postgres.

> the solution for which is to use icu We ran headfirst into this issue at my company and we've actually been recommending the opposite (use the "C" locale on the database, treat collation as a render level concern). I have a whole write up explaining the technical motivations behind that recommendation: https://gist.github.com/rraval/ef4e4bdc63e68fe3e83c9f98f56af...

It's easy to say "treat collation as a render level concern", but this doesn't really work efficiently when the rendering component wants to query the database using this index, does it?

That is, to do anything you want to do a locale sensitive way, such as querying the database for a given case insensitive string, or pagination, you'll need to have the DB index be locale/collation aware or else return every possible value and the renderer sort it out.

As an example, how you would you repeatedly return a range of results based on a string in a locale-aware way, e.g., to display paged results, if you defer the work to the renderer?

The only general solution I'm aware of which lets you "bypass" the DB collation is to use a locale-aware collation library like ICU to generate a binary sort key, which can be compared using plain binary comparison and storing those in the DB. This still means the overall index is locale aware, but the DB doesn't need to be aware of any collation rules: only the code that generates queries and handles the results needs to do the sort key transformation.

It means that you have a single library that does all the conversion, which you can probably control more easily, rather than delegating this to the database, where you might need to support several vendors or at least various versions (and problems can arise even within a single DB version as this postgres issue shows).

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#39

Post by jrochkind1 » Tue, Jan 08, 2019, 7:35 PM UTC

gumby wrote: post #12

This isn't a glibc issue; really it's a story that "character collations are not stable over time" (mentioned downthread in the article in a message quoting Unicode Technical Standard #10). Which is a cautionary tale the solution for which is to use icu, the solution adopted by Postgres.

you'd of course need to make sure everything in the system is using the same version of ICU/unicode.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#40

Post by jrochkind1 » Tue, Jan 08, 2019, 7:50 PM UTC

BeeOnRope wrote: post #28

greglindahl wrote: post #24

Earlier quoted context omitted.

Normalization is a separate issue, you can normalize and then use the C collation order.

Sure but either you are talking about a fixed normalization algorithm which is not locale aware, in which case it doesn't solve the locale-specific unique key issue, or it is locale-aware and hence suffers the same problem with time-varying behavior.

You are using string/text values as a pk and trying to sort on em? I'd say this is another reason not to do that.