When I say “alphabetical order”, I mean “alphabetical order”

The International Components for Unicode library implements the Unicode Collation Algorithm, which depends on the language code and region of the locale, and looks up the quirks for each locale in the Common Locale Data Repository.

It's a much better idea to just use the standard ICU library or platform specific libraries (which are often build on ICU like JavaScript's Intl.Collator), instead of trying to hot dog it by rolling your own.

International Components for Unicode

https://en.wikipedia.org/wiki/International_Components_for_U...

>ICU provides the following services: Unicode text handling, full character properties, and character set conversions; Unicode regular expressions; full Unicode sets; character, word, and line boundaries; language-sensitive collation and searching; normalization, upper and lowercase conversion, and script transliterations; comprehensive locale data and resource bundle architecture via the Common Locale Data Repository (CLDR); multiple calendars and time zones; and rule-based formatting and parsing of dates, times, numbers, currencies, and messages.

Unicode Collation Algorithm

https://en.wikipedia.org/wiki/Unicode_collation_algorithm

>The Unicode collation algorithm (UCA) is an algorithm defined in Unicode Technical Report #10, which is a customizable method to produce binary keys from strings representing text in any writing system and language that can be represented with Unicode. These keys can then be efficiently compared byte by byte in order to collate or sort them according to the rules of the language, with options for ignoring case, accents, etc.[1]

>Unicode Technical Report #10 also specifies the Default Unicode Collation Element Table (DUCET). This data file specifies a default collation ordering. The DUCET is customizable for different languages,[1][2] and some such customizations can be found in the Unicode Common Locale Data Repository (CLDR).[3]

Common Locale Data Repository

https://en.wikipedia.org/wiki/Common_Locale_Data_Repository

>The Common Locale Data Repository (CLDR) is a project of the Unicode Consortium to provide locale data in XML format for use in computer applications. CLDR contains locale-specific information that an operating system will typically provide to applications. CLDR is written in the Locale Data Markup Language (LDML).

>Among the types of data that CLDR includes are the following:

  Translations for language names
  Translations for territory and country names
  Translations for currency names, including singular/plural modifications
  Translations for weekday, month, era, period of day, in full and abbreviated forms
  Translations for time zones and example cities (or similar) for time zones
  Translations for calendar fields
  Patterns for formatting/parsing dates or times of day
  Exemplar sets of characters used for writing the language
  Patterns for formatting/parsing numbers
  Rules for language-adapted collation
  Rules for spelling out numbers as words
  Rules for formatting numbers in traditional numeral systems (such as Roman and Armenian numerals)
  Rules for transliteration between scripts, much of it based on BGN/PCGN romanization

sv-SE (Swedish): å, ä, ö are separate letters at the end of the alphabet, not variants of a or o.

de-DE (German): ä, ö, ü may sort as ae, oe, ue in some contexts, or as distinct letters. ß sometimes sorts as ss.

tr-TR (Turkish): dotted i (i) and dotless ı are different letters; I sorts with ı, not with i.

es-ES (Spanish): traditionally ch and ll were treated as single letters with their own place in the alphabet.

cs-CZ (Czech): ch still counts as a unique letter, sorted after h.

da-DK / no-NO (Danish/Norwegian): ø comes after z.

is-IS (Icelandic): þ (“thorn”) is part of the alphabet, after z.

fr-FR (French): accents usually ignored in sorting, so é = e, but not always depending on collation settings.

el-GR (Modern Greek): tonos accents, final sigma ς vs. σ, etc.

nl-NL (Dutch): the digraph “ij” is often treated as a single letter, and capitalized as “IJ”. In dictionaries and phone books it often sorts as a single letter under “I”, but sometimes is listed after “X” depending on tradition.

Then you get into non-Latin languages like, Chinese, Japanese, and Korean collation, which gets hairy with radicals, kana order, and stroke count.

Also different locales have different ways of representing numbers, like switching between "," and "." as separators and decimal points.

ICU supports integer only "natural" numeric collation, so anything more complicated like versions, floating point, negative numbers, hex, thousands separators, fractions, roman numerals, etc, you'd have to build on top of ICU.

ICU doesn't support incomprehensible dead languages like Latin or Ancient Greek (it does however support French ;). It does support Roman numeral formatting, but not collation, which would be pretty tricky and ambiguous.

https://www.youtube.com/watch?v=sKWvTlLMB-Y

A nuanced but common example that ICU/UCA/CLDR helps with is a menu to select the current locale: you have to translate each language's name into the current locale, and also sort them in the current locale. On top of different collations they can also have totally different spellings, like "United States of America" is "Verenigde Staten van Amerika" in Dutch. This makes it challenging for users to find their own language when the locale is set wrong! You just can't win.

Not to mention emojis! Which comes first: The chicken or the egg? The taco or the poop?

Also, the Mac Finder switches ":" and "/" for historical reasons (HFS used to use ":" as a directory separator instead of "/"), so you can create a file name like "9/11 Attack" in the Finder, which actually gets the underlying Unix filename "9:11 Attack". Don't believe me? Rename a file in the Finder to include a slash, which you know is impossible to represent as a Unix file name. Then go "ls" the directory in the shell.

The Mac Finder weirdly collates "/" after "9" because under the hood it’s really storing it as ":", which sorts before "0". But it also has other punctuation collating inconsistencies, sorting "," and ";" and others after "0" too. Definitely not ASCII order -- I'm not sure what rules it uses, but it's different than "ls".

However, while it's generally true you can't have "/" in Unix file names, NFS used to trustingly let clients rename Unix files to include a "/" in their name, which the Gator Box AppleTalk/Ethernet gateway let you do with the Mac Finder (pre OS/X), which would silently corrupt your "dump" backups on the Unix NFS server, so you would not learn about it until you tried to retrieve your files and "restore" crashed.

https://news.ycombinator.com/item?id=31821646

>Another reason that NFS sucks: Anyone remember the Gator Box? It enabled you to trick NFS into putting slashes into the names of files and directories, which seemed to work at the time, but came back to totally fuck you later when you tried to restore a dump of your file system.

>The NFS protocol itself didn't disallow slashes in file names, so the NFS server would accept them without question from any client, silently corrupting the file system without any warning. Thanks, NFS!

---

In all file managers, I miss an API point where one can give a userdefined sorting function for the file and folder list.

Call lexicographic order "sort by name" as it's called now, and call dumb character-by-character sort "plain" or something like that. I'm not a designer, maybe there are more intuitive names, but come on. This isn't an intractable problem.

  for i in $(seq 2 10) ; do
    touch img_$i-hn.txt
  done

  ls img_* | sort -V
  img_2-hn.txt
  img_3-hn.txt
  img_4-hn.txt
  img_5-hn.txt
  img_6-hn.txt
  img_7-hn.txt
  img_8-hn.txt
  img_9-hn.txt
  img_10-hn.txt

Edit: formatting and add sort -h