Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The dangers of streaming across versions of glibc: A cautionary tale (2014)

postgresql.org

Post reply on HN
Print view View original

71–80 of 82 posts

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#71

Post by jancsika »

ben509 wrote: post #54
alexchamberlain wrote: post #36

May I ask a stupid question? What's the difference between collation and encoding? Some people in this thread have suggested "use byte strings"; is it "enough" to "just" use UTF-8, like you should "just" use UTC (unless you know/Should know better)?

To get a bit into the theory: a string is a sequence of symbols drawn from some "alphabet", and we can define operations like concatenation, counting, matching and substring on it. In particular, a natural operation on strings is lexicographic comparison, whereby you compare the first characters of two strings, if they are different, the two strings compare that way, if they match, you proceed to look at the second c…

> Collation represents a string as an expression (usually also a string) that is trivially comparable. For instance, supposed I wanted an English-friendly way to compare band names. I might map all codepoints that look like A or E to "A", then K, S and C to C, etc., discarding any accents and maybe throwing out leading junk like "The" or "A".

So how does collation algorithmic instability break an application?

Edit: What I mean is-- what are a set of reasonable assumptions a dev would make in the application layer that would be good practice on the one hand, yet also be wholly dependent on collation algorithm stability to keep from breaking?

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#72

Post by nine_k »

BeeOnRope wrote: post #63
nine_k wrote: post #57

Earlier quoted context omitted.

I'd posit that this is not universally true. It may be true for English names. It may be implementable for e.g. French names: a name like "François" may be stored as UTF-8 with "ç" always represented as a composite pair (or always a single character), and the application layer knows and uses this. It must be a dubious idea for German names when you may need to see "Müller" and "Mueller" as the same name, but also kee…

Of course it's hairy - that's kind of the point of this whole discussion! People who use databases naturally end up wanting to store Unicode strings, and people naturally want to query them in locale-sensitive ways. Your Mueller example is a fine example of this. Databases have deep support for this, and you can also build your own compromise, so to speak, on top of the database with ICU or another library if it does…

Storing is fine!

Indexing is full of caveats, though.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#73

Post by ben509 »

jancsika wrote: post #71
ben509 wrote: post #54

Earlier quoted context omitted.

To get a bit into the theory: a string is a sequence of symbols drawn from some "alphabet", and we can define operations like concatenation, counting, matching and substring on it. In particular, a natural operation on strings is lexicographic comparison, whereby you compare the first characters of two strings, if they are different, the two strings compare that way, if they match, you proceed to look at the second c…

> Collation represents a string as an expression (usually also a string) that is trivially comparable. For instance, supposed I wanted an English-friendly way to compare band names. I might map all codepoints that look like A or E to "A", then K, S and C to C, etc., discarding any accents and maybe throwing out leading junk like "The" or "A". So how does collation algorithmic instability break an application? Edit: W…

All the assumptions that make a database index work will break.

If you think about any kind of binary tree based structure, it will have invariants like "the left child node will be less than the parent node, and the right child node will be greater than the parent node."

If those are violated just a little bit, at best data will simply disappear from the index. It's possible for updates to delete stuff, or for operations on the index to hang.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#74

Post by blattimwind »

voltagex_ wrote: post #69
leeter wrote: post #23

Earlier quoted context omitted.

It has always amused me that NTFS and NT Kernel are both case sensitive when it comes to filenames. But WIN32 emulates the case insensitive legacy behavior because people won't fix, or can't fix legacy code. That said applications can properly opt into POSIX semantics.

Are you talking about WSL or some other way that applications can use POSIX on Windows?

FILE_FLAG_POSIX_SEMANTICS

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#75

Post by CamperBob2 »

waynecochran wrote: post #46
butterisgood wrote: post #44

Static linking can be good.

Of course if you have two applications communicating with each other that are statically linked with different version of the library you can still have inconsistency problems.

That's more a protocol design problem than anything else.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#76

Post by tedunangst »

jcranmer wrote: post #66
chousuke wrote: post #53

Earlier quoted context omitted.

Is that really legit in German? I know Finnish umlauts (ä, ö) get sometimes mangled to ae or oe, but they are definitely not valid alternative spellings nor are they pronounced even close to similar.

This brings up something that is perhaps easy to forget: the interpretation of diacritics is far from universal. The diacritic in the character 'ä' can be one of two semantically different diacritics. It can be a diaeresis, a diacritic whose function is to mark that vowel starts the next syllable rather than existing as part of a diphthong; or it can be an umlaut, whose purpose is to indicate that it is a different v…

Pretty much the only place you see diaeresis in English is in the New Yorker, whenever they use a word like coördination.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#77

Post by jandrese »

oconnor663 wrote: post #19
barrkel wrote: post #17

About 20 years ago, I implemented a hash table that used binary trees for buckets. It was supposed to support Unicode strings as keys, and I used the Windows comparison functions to figure out how to compare the keys, in order to handle case insensitive lookup IIRC. I tested the table with randomly generated strings, and was puzzled to discover expected lookups would fail with some table constructions. I narrowed it…

> I used the Windows comparison functions to figure out how to compare the keys, in order to handle case insensitive lookup IIRC. Wouldn't you need to do case normalization before hashing, to make that work?

Case normalization is it's own minefield in Unicode.

This article discusses some of the challenges: https://stackoverflow.com/questions/6162484/why-does-modern-...

In the first section points 16, 22, and 23 are relevant, in the second section look at points 8, 9, 10, 11, 12, 13, and 40.

Sorting based on Unicode strings is tricky, especially if you want cross platform compatibility. It's better today but there are still a lot of edge cases to consider if you have any hope of getting the same sort twice from "interesting" data.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#78

Post by wrs »

theamk wrote: post #13

This, BTW, is a great example why many filesystems are case sensitive -- ignoring case requires collation support, and this can change all the time. Treating filenames as opaque byte strings, on the other hand, makes the filesystems, databases and so on Always Work. Of course, PostgreSQL is one the few programs which actually cares about collation and case-insensitivity, so it has to work the hard way.

IIRC, the NTFS filesystem has its own collation tables on disk that are fixed when the volume is formatted, so its indexes are forever consistent but don’t necessarily match the ongoing changes to collations. Thus the shell has to re-sort everything before showing it, even though you’d think the filenames were already sorted by NTFS.

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#79

Post by zaarn »

nine_k wrote: post #41
barrkel wrote: post #17

About 20 years ago, I implemented a hash table that used binary trees for buckets. It was supposed to support Unicode strings as keys, and I used the Windows comparison functions to figure out how to compare the keys, in order to handle case insensitive lookup IIRC. I tested the table with randomly generated strings, and was puzzled to discover expected lookups would fail with some table constructions. I narrowed it…

Case transformations are locale-dependent. That is, in French lower case of "I" would be "i", and upper case of "i" would be "I". In Turkish, which uses largely the same letters, lower case of "I" would be "ı", and upper case of "i" would be "İ". Also, in German, upper case of "s" is "S", but upper case of "ß" would be "SS", and you have to guess what lower case of "SS" would be. Universal case insensitivity is hard…

The handling of "ß" was recently changed; "ẞ" is the new uppercase form of "ß" since 2017 (Unicode has it since 2007)

Re: The dangers of streaming across versions of glibc: A cautionary tale (2014)

#80

Post by cbsmith »

nine_k wrote: post #72
BeeOnRope wrote: post #63

Earlier quoted context omitted.

Of course it's hairy - that's kind of the point of this whole discussion! People who use databases naturally end up wanting to store Unicode strings, and people naturally want to query them in locale-sensitive ways. Your Mueller example is a fine example of this. Databases have deep support for this, and you can also build your own compromise, so to speak, on top of the database with ICU or another library if it does…

Storing is fine! Indexing is full of caveats, though.

Indexing is always full of caveats. :-)
Post reply on HN