A popular but wrong way to convert a string to uppercase or lowercase

Post by fluoridation » Tue, Oct 08, 2024, 7:06 PM UTC

bayindirh wrote: post #68

SAI_Peregrinus wrote: post #65

Earlier quoted context omitted.

> Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons). Because there are more than 65,535 characters. That's just writing systems, not Unicode's fault. Most of the unnecessary complexity of Unicode is legacy compatibility: UTF-16 & UTF-32 are bad ideas that increas…

I mean, I already know some Unicode internals and linguistics (since I developed a language-specific compression algorithm back in the day), but I have never seen a single character requiring four bytes (and I know Emoji chaining for skin color, etc.). So, seeing this just moved the complexity of Unicode one notch up in my head, and I respect the guys who designed and made it work. It was not whining or complaining o…

Post by zahlman » Tue, Oct 08, 2024, 7:08 PM UTC

kccqzy wrote: post #91

zahlman wrote: post #82

Earlier quoted context omitted.

Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worr…

Still breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information. > (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.) Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.

Post by kccqzy » Tue, Oct 08, 2024, 7:20 PM UTC

zahlman wrote: post #94

kccqzy wrote: post #91

Earlier quoted context omitted.

Still breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information. > (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.) Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.

I'm not seeing anything in the Swift documentation about strings carrying language metadata, either, though?

Post by Rygian » Tue, Oct 08, 2024, 7:41 PM UTC

cyxxon wrote: post #17

Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.

Post by squeaky-clean » Tue, Oct 08, 2024, 7:48 PM UTC

fluoridation wrote: post #86

SleepyMyroslav wrote: post #73

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…

>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :) I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. What if your game needs to talk to a server and do s…