Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

11–20 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#11

Post by bayindirh »

blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop.

On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship.

Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mammoth now, everyone expects this huge pony to learn new tricks, but everybody has a different idea of the "new tricks", so more features are added on top of its already impressive and very long list of features and capabilities.

You want libICU built-in? There must be other folks who want that too. So you may need to find them and work with them to make your dream a reality.

So, C++ is doing fine. It's not that they omitted Unicode during the design phase. Unicode has arrived later, and it has to be integrated by other means. This is what libraries for.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#12

Post by 9dev »

vardump wrote: post #7
fhars wrote: post #4

Earlier quoted context omitted.

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

It's funny how software developers live in bubbles so much. Whether you deal with human language a lot or almost not at all depends entirely on your specific domain. Anyone working on user interfaces of any kind must accommodate for proper encoding, for example; that includes pretty much every line-of-business app out there, which is a lot of code.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#13

Post by vardump »

ahartmetz wrote: post #8

...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unli…

You just want a banana, but you also get the gorilla. And the jungle.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#14

Post by pistoleer »

daemin wrote: post #6
fhars wrote: post #4

Earlier quoted context omitted.

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

> I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII.

Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying localization problems matter a lot. That includes sorting the last name "van den Berg" just after "Bakker", stylizing it as "Berg, van den", and making sure this capitalization is correct and not "Van Den Berg". There is no built in standard library function in any language that does any of that. It's so much larger than ascii and even larger than unicode.

Another user said that the main takeaway is that you can't process strings until you know their language (locale), and that is exactly correct.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#15

Post by BoringTimesGang »

daemin wrote: post #6
fhars wrote: post #4

Earlier quoted context omitted.

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

Now double all of that effort, so you can get it to work with Windows' UTF-16 wstrings.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#16

Post by heisenzombie »

daemin wrote: post #6
fhars wrote: post #4

Earlier quoted context omitted.

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux).

For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

Re: A popular but wrong way to convert a string to uppercase or lowercase

#17

Post by cyxxon »

Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#18

Post by vardump »

heisenzombie wrote: post #16
daemin wrote: post #6

Earlier quoted context omitted.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings.

So use standard string processing libraries on path names at your own peril.

It's a good idea to consider file paths as a bag of bytes.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#19

Post by tralarpa »

BoringTimesGang wrote: post #9
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

>It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! These are mostly unicode or linguistics problems.

The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#20

Post by pornel »

bayindirh wrote: post #11
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

Being developed in, and having to stay compatible with, ancient times is a real problem of C++.

The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++.

Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but there's a lot that a language can do to at least not silently destroy the encoding.

Post reply on HN