Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

101–110 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#101

Post by gpderetta »

relaxing wrote: post #28
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

Java ended up picking UCS-2 and getting screwed.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#102

Post by gpderetta »

pjmlp wrote: post #67
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

Because it is a fight to put anything on a ISO managed language, and only the strongest persevere long enough to make it happen. Regardless of what ISO language we are talking about.

Yes, significantly smaller libraries had an hard time getting onto the standard. Getting the equivalent of ICU would be almost impossible. And good luck keeping it up to date.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#103

Post by aetherspawn »

ahartmetz wrote: post #8

...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unli…

I will admit I don’t love the Qt licensing model, but most things in Qt just work as they are supposed to, and on every platform too.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#104

Post by fluoridation »

squeaky-clean wrote: post #100
fluoridation wrote: post #86

Earlier quoted context omitted.

>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :) I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. What if your game needs to talk to a server and do s…

> They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. I'm confused now. The article specifically mentions issues with UTF-16 and UTF-32 unicode characters outside the basic multilingual plane (BMP).

I'm referring to the people who call case conversion in general "a simple text operation". Say you have an std::string and you want to make it lower case. If you assume it contains just ASCII that's a simpler operation than if you assume it contains UTF-8, but C++ doesn't provide a single function that does either of them. A person can rightly complain that the former is a basic functionality that the language should include; personally, I would agree. And you could say "wow, doesn't this person realize that case conversion in Unicode is actually complicated? They must be really inexperienced." It could be that the other person really doesn't know about Unicode, or it could mean that you and them are thinking about entirely different problems and you're being judgemental a bit too eagerly.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#106

Post by johnnyjeans »

high_na_euv wrote: post #60

In cpp basic things are hard

nothing about working with locales, or text in general, is basic. we were decades into working with digital computers before we moved past switchboards and LEDs. don't take for granted just how high of a perch upon the shoulders of giants you have. that's exactly how the mistakes in the blog post get made.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#107

Post by IncreasePosts »

tedunangst wrote: post #95
zahlman wrote: post #82

Earlier quoted context omitted.

Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worr…

But that's wrong. The upper case for ß is ẞ.

That was only adopted in Germany like 7 years ago!

Re: A popular but wrong way to convert a string to uppercase or lowercase

#108

Post by Muromec »

sebstefan wrote: post #66
Muromec wrote: post #64

Earlier quoted context omitted.

If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.

First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's li…

>First off: And exclude 70% of the world?

Guess what, I'm part of this 70% and I also work in a bank and I know exactly how.

Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway.

So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in addition to canonical form in Ukrainian Cyrillic. That ASCII-rendering is happily accepted by all kinds of systems that only speak ASCII.

People still can't pronounce it and it got misspelled like yesterday when dictated over the phone.

Now regarding the accents, it's illegal to not support them per GDPR (as per case law, discussed here few years ago).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#109

Post by vardump »

commandlinefan wrote: post #105
flareback wrote: post #97

He gave 4 examples of how it's done incorrectly, but zero actual examples of doing it correctly.

for (int i = 0; i

Surely you meant:

  s[i] &= ~0x20;
We're talking about converting to upper case after all! As an added benefit, every space character (0x20) is now a NUL byte!

Re: A popular but wrong way to convert a string to uppercase or lowercase

#110

Post by squeaky-clean »

fluoridation wrote: post #104
squeaky-clean wrote: post #100

Earlier quoted context omitted.

> They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. I'm confused now. The article specifically mentions issues with UTF-16 and UTF-32 unicode characters outside the basic multilingual plane (BMP).

I'm referring to the people who call case conversion in general "a simple text operation". Say you have an std::string and you want to make it lower case. If you assume it contains just ASCII that's a simpler operation than if you assume it contains UTF-8, but C++ doesn't provide a single function that does either of them. A person can rightly complain that the former is a basic functionality that the language should…

For ascii in C++ isn't there std::tolower / std::toupper? If you're not dealing with unsigned char types there isn't a simple case conversion function, but that's for a good reason as the article lays out.
Post reply on HN