21–30 of 272 posts
The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.
The lowercase of "DON'T FUSS ABOUT FUSSBALL" is "don't fuss about fußball". Unless you're in Switzerland.
Earlier quoted context omitted.
>It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! These are mostly unicode or linguistics problems.
The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).
But there are always dragons. It's strings. The mere assumption that they can be transformed int-by-int, irrespective of encoding, is wrong. As is the assumption that a sensible transformation to lower case without error handling exists.
Earlier quoted context omitted.
No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.
> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.
It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…
> There are so many ways to do something and every way is freaking wrong! That's life! The perfect way does not exist. The best you can do is be aware of the tradeoffs, and languages like C++ absolutely throw them in your face at every single opportunity. It's fatiguing, and writing in javascript or python allows us to uphold the facade that everything is okay and that we don't have to worry about a thing.
It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…
Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.
But isn't it also dependent on the available glyphs in the font used? So f.e. it needs to be ensured that U+1E9E exists?
The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.
Different parts of a string can be in different languages too[1]. The lowercase of "DON'T FUSS ABOUT FUSSBALL" is "don't fuss about fußball". Unless you're in Switzerland. [1] https://en.wikipedia.org/wiki/Code-switching
It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…
I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…
It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…
> There are so many ways to do something and every way is freaking wrong! That's life! The perfect way does not exist. The best you can do is be aware of the tradeoffs, and languages like C++ absolutely throw them in your face at every single opportunity. It's fatiguing, and writing in javascript or python allows us to uphold the facade that everything is okay and that we don't have to worry about a thing.
Swift has been developed in the modern times, and it's able to tackle Unicode properly, e.g. makes distinction between codepoints and grapheme clusters, and steers users away from random-access indexing and having a single (incorrect) notion of a string length.