Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The dangers of streaming across versions of glibc: A cautionary tale (2014)

postgresql.org

Post reply on HN
Print view View original

41–50 of 82 posts

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#41

Post by nine_k »

barrkel wrote: post #17

About 20 years ago, I implemented a hash table that used binary trees for buckets. It was supposed to support Unicode strings as keys, and I used the Windows comparison functions to figure out how to compare the keys, in order to handle case insensitive lookup IIRC. I tested the table with randomly generated strings, and was puzzled to discover expected lookups would fail with some table constructions. I narrowed it…

Case transformations are locale-dependent. That is, in French lower case of "I" would be "i", and upper case of "i" would be "I". In Turkish, which uses largely the same letters, lower case of "I" would be "ı", and upper case of "i" would be "İ". Also, in German, upper case of "s" is "S", but upper case of "ß" would be "SS", and you have to guess what lower case of "SS" would be.

Universal case insensitivity is hard if not impossible. It's best to preserve both a "canonical case" version and the raw data in a search index, if different.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#42

Post by nine_k »

BeeOnRope wrote: post #38
rraval wrote: post #16

Earlier quoted context omitted.

> the solution for which is to use icu We ran headfirst into this issue at my company and we've actually been recommending the opposite (use the "C" locale on the database, treat collation as a render level concern). I have a whole write up explaining the technical motivations behind that recommendation: https://gist.github.com/rraval/ef4e4bdc63e68fe3e83c9f98f56af...

It's easy to say "treat collation as a render level concern", but this doesn't really work efficiently when the rendering component wants to query the database using this index, does it? That is, to do anything you want to do a locale sensitive way, such as querying the database for a given case insensitive string, or pagination, you'll need to have the DB index be locale/collation aware or else return every possible…

If you build a plain index over Unicode strings, chances are you're doing it wrong.

Normal DB indexes are mostly for numbers and (short) ASCII strings. (Something like canonicalized UTF-8 is an edge case.) For strings that have encodings and locales, you likely need a full-text index, provided by your DB or by something like Solr / ElasticSearch. It addresses the oddities of human-oriented texts better.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#43

Post by krylon »

nine_k wrote: post #41
barrkel wrote: post #17

About 20 years ago, I implemented a hash table that used binary trees for buckets. It was supposed to support Unicode strings as keys, and I used the Windows comparison functions to figure out how to compare the keys, in order to handle case insensitive lookup IIRC. I tested the table with randomly generated strings, and was puzzled to discover expected lookups would fail with some table constructions. I narrowed it…

Case transformations are locale-dependent. That is, in French lower case of "I" would be "i", and upper case of "i" would be "I". In Turkish, which uses largely the same letters, lower case of "I" would be "ı", and upper case of "i" would be "İ". Also, in German, upper case of "s" is "S", but upper case of "ß" would be "SS", and you have to guess what lower case of "SS" would be. Universal case insensitivity is hard…

There is an uppercase ß, actually: https://en.wikipedia.org/wiki/Capital_%E1%BA%9E

Not that it has any impact on your point. But as a German who only learned about it fairly recently, I have rather ambivalent feelings about it. ;-)

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#45

Post by antirez »

To rely on anything non trivial provided by libc is always a risk, but here the problem is that collation is not the right tool for the job (implementing data structures were there is to compare elements). In Redis memcmp() is used, because byte-to-byte comparison is a much more stable story. This leaves the user with the problem of finding a suitable representation that is lexicographically sortable by the first bytes in a way that makes sense for the application, and that also contains, in the farest bytes, what the user really wants to return. Not viable for something higher level than Redis I guess.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#46

Post by waynecochran »

butterisgood wrote: post #44

Static linking can be good.

Of course if you have two applications communicating with each other that are statically linked with different version of the library you can still have inconsistency problems.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#47

Post by nine_k »

krylon wrote: post #43
nine_k wrote: post #41

Earlier quoted context omitted.

Case transformations are locale-dependent. That is, in French lower case of "I" would be "i", and upper case of "i" would be "I". In Turkish, which uses largely the same letters, lower case of "I" would be "ı", and upper case of "i" would be "İ". Also, in German, upper case of "s" is "S", but upper case of "ß" would be "SS", and you have to guess what lower case of "SS" would be. Universal case insensitivity is hard…

There is an uppercase ß, actually: https://en.wikipedia.org/wiki/Capital_%E1%BA%9E Not that it has any impact on your point. But as a German who only learned about it fairly recently, I have rather ambivalent feelings about it. ;-)

Nice! German has funnier problems, though, because a vowel with an umlaut can be represented as that vowel + e, that is, "fuer" is a legit representation of "für". How do you normalize that? Turning to one canonical representation works most of the time, but sometimes you also need letter-to-letter correspondence.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#48

Post by cbsmith »

antirez wrote: post #45

To rely on anything non trivial provided by libc is always a risk, but here the problem is that collation is not the right tool for the job (implementing data structures were there is to compare elements). In Redis memcmp() is used, because byte-to-byte comparison is a much more stable story. This leaves the user with the problem of finding a suitable representation that is lexicographically sortable by the first byt…

It's not that simple. SQL in particular actually requires lexicographical sorting, so you need to do it somehow. The problem is relying on lexicographical sorting remaining fixed when it isn't. Per the unicode standard, lexicographical sorting rules are subject to change. So now you need a mechanism to detect/manage those changes.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#49

Post by cbsmith »

nine_k wrote: post #42
BeeOnRope wrote: post #38

Earlier quoted context omitted.

It's easy to say "treat collation as a render level concern", but this doesn't really work efficiently when the rendering component wants to query the database using this index, does it? That is, to do anything you want to do a locale sensitive way, such as querying the database for a given case insensitive string, or pagination, you'll need to have the DB index be locale/collation aware or else return every possible…

If you build a plain index over Unicode strings, chances are you're doing it wrong. Normal DB indexes are mostly for numbers and (short) ASCII strings. (Something like canonicalized UTF-8 is an edge case.) For strings that have encodings and locales, you likely need a full-text index, provided by your DB or by something like Solr / ElasticSearch. It addresses the oddities of human-oriented texts better.

I'm sorry but no.

It's a pretty common use case (so common that it is usually taught in an introduction to databases) that you might want an index that allows you to efficiently search by non-ASCII strings like say, a person's last name.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#50

Post by cbsmith »

cryptonector wrote: post #30

PostgreSQL should use its own internationalization library, or ICU, but not the C library's I18N components. This would allow PG to ensure that any one version of PG also uses the same version of Unicode.

Or, you know, you could just track the versioning of the library that you use for collation...
Post reply on HN