Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

241–250 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#241

Post by account42 »

pjmlp wrote: post #185
PaulDavisThe1st wrote: post #171

Earlier quoted context omitted.

He may be right, but approximately 75% of the problems he describes are all Microsoft-ecosystem specific. In Unix-land we don't use wchar_t or UTF-16, and his article is a good demonstrations of why not.

UNIX land is even worse in international languages support. As in, there isn't even something on POSIX at the level other operaring systems support for localisation. Yes there is some locale stuff, however not enough for all stuff, hence why every modern programming language happens to have this as part of their standard library.

Is there a platform where you can't use ICU?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#242

Post by account42 »

pjmlp wrote: post #69
cyxxon wrote: post #17

Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.

Lowering case is even better, because a Swiss user would expect the two-character sequence “SS“ to be converted into “ss“ and not “ß“. And thus we add country specific locale to the party.

Not just a Swiss user as there are many German words that use ss and not ß. And having an ss where there should be an ß will be a lot less disruptive as the inverse because people are used to ASCII limitations.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#243

Post by Netch »

cheema33 wrote: post #169
SleepyMyroslav wrote: post #73

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…

> In gamedev there is simple rule: don't try to do any of that. I am not in gamedev, but I frequently have to develop middleware that takes in user entered data and formats it in a way that will import into a 3rd party system without errors. And that sometimes means changing the case on strings. In my experience as a developer, this is very very common requirement. Luckily I am not forced to use a low level language…

> In C# I can simply do this: "hello world".ToUpper();

Hmm still actual: https://www.moserware.com/2008/02/does-your-code-pass-turkey...

Re: A popular but wrong way to convert a string to uppercase or lowercase

#244

Post by Night_Thastus »

blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

As a C++ dev, I have never run into the problem the post is describing. Upper and lowercase conversion has always worked just fine. Though then again, I don't fiddle with mixed unicode and non-unicode situations.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#245

Post by Netch »

relaxing wrote: post #28
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

> There’s no excuse.

Until mid-2000s there was no certainty Unicode will eventually defeat competitors. In real it havenʼt fully yet - GB2312 and Tron are still locally prevailing, and IBM still jogs with EBCDIC. But at its early times nobody was reasonably sure, and Java attempt could have failed as well. (More so Java approach for UCS-2 was wrong - already commented nearby.)

Re: A popular but wrong way to convert a string to uppercase or lowercase

#246

Post by account42 »

vardump wrote: post #109
commandlinefan wrote: post #105

Earlier quoted context omitted.

for (int i = 0; i

Surely you meant: s[i] &= ~0x20; We're talking about converting to upper case after all! As an added benefit, every space character (0x20) is now a NUL byte!

Free strtok!

Re: A popular but wrong way to convert a string to uppercase or lowercase

#247

Post by account42 »

lilyball wrote: post #148
PoignardAzur wrote: post #117

So I'm going to be that guy and say it: Man, I'm happy we don't need to deal with this crap in Rust, and we can just use String::to_lowercase. Not having to worry about things makes coding fun.

While certainly much better, you still need to be aware that doing case conversion absent any locale information will never be perfect. If you want proper locale-aware conversion you can use the icu crate ( https://docs.rs/icu/latest/icu/ ).

Exactly, simple "unicode-aware" case conversions are a trap. You are always going to need much more.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#248

Post by Netch »

blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

> Makes you wonder why this isn't part of the C++ standard library itself.

Plainly no need if there is a separate easily attachable library (and with permissible license). What C++ had to do - provide character (char{8,16,32}_t) and string types - it has done.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#249

Post by account42 »

ahartmetz wrote: post #8

...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unli…

QString is how you ensure you cannot open/delete some files you WILL eventually encounter.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#250

Post by account42 »

ahartmetz wrote: post #57
cannam wrote: post #52

Earlier quoted context omitted.

QString::toUpper/toLower are not locale-aware ( https://doc.qt.io/qt-6/qstring.html#toLower ) Qt does have a locale-aware equivalent (QLocale::toUpper/toLower) which calls out to ICU if available. Otherwise it falls back to the QString functions, so you have to be confident about how your build is configured. Whether it works or not has very little to do with the design of QString.

I don't see a problem with that. You can have it done locale-aware or not and "not" seems like a sane default. QString will uppercase 'ü' to 'Ü' just fine without locale-awareness whereas std::string doesn't handle non-ASCII according to the article. The cases where locale matters are probably very rare and the result will probably be reasonable anyway.

That attitude is how you end up with exploits because your case folding is different from some other system you interact with.
Post reply on HN