Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

51–60 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#51

Post by [deleted] »

bayindirh wrote: post #41
blenderob wrote: post #35

Earlier quoted context omitted.

But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripher…

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

[deleted]

Re: A popular but wrong way to convert a string to uppercase or lowercase

#52

Post by cannam »

ahartmetz wrote: post #8

...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unli…

QString::toUpper/toLower are not locale-aware (https://doc.qt.io/qt-6/qstring.html#toLower)

Qt does have a locale-aware equivalent (QLocale::toUpper/toLower) which calls out to ICU if available. Otherwise it falls back to the QString functions, so you have to be confident about how your build is configured. Whether it works or not has very little to do with the design of QString.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#53

Post by bayindirh »

blenderob wrote: post #47
bayindirh wrote: post #41

Earlier quoted context omitted.

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

Thanks for the reply! Really appreciate the time you have taken to write down a thoughtful reply.

No problems! If you want a slightly longer write-up, here's a classic I constantly share with people:

https://blog.codinghorror.com/whats-wrong-with-turkey/

Re: A popular but wrong way to convert a string to uppercase or lowercase

#54

Post by BoringTimesGang »

cm2187 wrote: post #48
pornel wrote: post #20

Earlier quoted context omitted.

Being developed in, and having to stay compatible with , ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but…

That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).

Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions.

Or perhaps I should say naïve.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#55

Post by sebstefan »

Muromec wrote: post #44
sebstefan wrote: post #37

Earlier quoted context omitted.

Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.

If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.

From experience anything banking adjacent will usually fuck it up as well

For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII

Re: A popular but wrong way to convert a string to uppercase or lowercase

#56

Post by pistoleer »

daemin wrote: post #43
pistoleer wrote: post #14

Earlier quoted context omitted.

> I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying loc…

I would maintain that your program has more string manipulation for error messages and logging than for generating localised formatted names. Further I do say that if you're creating text for presenting to the user then the most common operation would be replacement of some field in pre-defined text. In your case I would design it so that the correctly capitalised first name, surname, and variations of those for sort…

I agree, but the logging formatters don't really do much beyond trivially pasting in placeholders.

And as for data entry... Maybe in an ideal world. In the current world, marred by importing previously mangled datasets, a common solution in the few companies I've worked at is to just not do anything, which leaves ugly edges, yet is "good enough".

Re: A popular but wrong way to convert a string to uppercase or lowercase

#57

Post by ahartmetz »

cannam wrote: post #52
ahartmetz wrote: post #8

...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unli…

QString::toUpper/toLower are not locale-aware ( https://doc.qt.io/qt-6/qstring.html#toLower ) Qt does have a locale-aware equivalent (QLocale::toUpper/toLower) which calls out to ICU if available. Otherwise it falls back to the QString functions, so you have to be confident about how your build is configured. Whether it works or not has very little to do with the design of QString.

I don't see a problem with that. You can have it done locale-aware or not and "not" seems like a sane default. QString will uppercase 'ü' to 'Ü' just fine without locale-awareness whereas std::string doesn't handle non-ASCII according to the article. The cases where locale matters are probably very rare and the result will probably be reasonable anyway.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#58

Post by InfamousRece »

Muromec wrote: post #44
sebstefan wrote: post #37

Earlier quoted context omitted.

Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.

If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.

Some systems are still using EBCDIC.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#59

Post by inexcf »

vardump wrote: post #31
inexcf wrote: post #23

Earlier quoted context omitted.

Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?

Here's an example. Hypothetically say you want to build an HTML parser. You might encounter tags like , , , etc., but you want to perform a hash table lookup. So first you're going to normalize to either lower- or uppercase.

Ah, i see, we disagree on what is "human language". An abbreviation like HTML and it's different capitalisations to me sound a lot like a feature of human language.
Post reply on HN