Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

41–50 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#41

Post by bayindirh »

blenderob wrote: post #35
bayindirh wrote: post #33

Earlier quoted context omitted.

> There’s no excuse. I politely disagree. None of the programming languages which started integrating Unicode was targeting from bare metal to GUI, incl. embedded and OS development at the same time. C++ has a great target area when compared to other programming languages. There are widely used libraries which compile correctly on PDP-11s, even if they are updated constantly. You can't just say "I'll be just making e…

But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripher…

> Converting one Unicode string to another is a purely in-memory, in-CPU operation.

...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding.

Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules.

Esp, this I/ı and İ/i pairs break tons of applications in very unexpected ways. I don't remember how many bugs I reported, and how many workarounds I have implemented in my systems.

Adding a type is nice, but the surrounding machinery is so big, it brings tons of work with itself. Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#42

Post by vardump »

kergonath wrote: post #39
vardump wrote: post #7

Earlier quoted context omitted.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

I said use a Unicode library if input data is actual human language. Which names and addresses are.

99% case being ASCII data generated by other software of unknown provenance. (Or sometimes by humans, but it's still data for machines, not for humans.)

Re: A popular but wrong way to convert a string to uppercase or lowercase

#43

Post by daemin »

pistoleer wrote: post #14
daemin wrote: post #6

Earlier quoted context omitted.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

> I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying loc…

I would maintain that your program has more string manipulation for error messages and logging than for generating localised formatted names.

Further I do say that if you're creating text for presenting to the user then the most common operation would be replacement of some field in pre-defined text.

In your case I would design it so that the correctly capitalised first name, surname, and variations of those for sorting would be generated at the data entry point (manually or automatically) and then just used when needed in user facing text generation. Therefore the only string operation needed would be replacement of placeholders like the fmt and standard library provide. This uses more memory and storage but these are cheaper now.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#44

Post by Muromec »

sebstefan wrote: post #37
vardump wrote: post #2

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.

Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.

If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway.

The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#45

Post by Muromec »

kergonath wrote: post #39
vardump wrote: post #7

Earlier quoted context omitted.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

Who and why still tries to lowercase/uppercase names? Please tell them to stop.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#46

Post by Muromec »

vardump wrote: post #31
inexcf wrote: post #23

Earlier quoted context omitted.

Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?

Here's an example. Hypothetically say you want to build an HTML parser. You might encounter tags like , , , etc., but you want to perform a hash table lookup. So first you're going to normalize to either lower- or uppercase.

But but, I want to have a custom web component and register it under my own name, which can only be properly written in Ukrainian Cyrillic. How dare you not let me have it.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#47

Post by blenderob »

bayindirh wrote: post #41
blenderob wrote: post #35

Earlier quoted context omitted.

But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripher…

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

Thanks for the reply! Really appreciate the time you have taken to write down a thoughtful reply.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#48

Post by cm2187 »

pornel wrote: post #20
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

Being developed in, and having to stay compatible with , ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but…

That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#49

Post by Muromec »

cyxxon wrote: post #17

Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.

But what if you need to uppercase the historical record in a vital records registry from 1950ies, but and OCRed last week? Now you need to not just be locale-aware, but you locale should be versioned.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#50

Post by elpocko »

vardump wrote: post #7
fhars wrote: post #4

Earlier quoted context omitted.

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Every search feature everywhere has to be case-insensitive or it's unusable. Search seems like a pretty ubiquitous feature in a lot of software, and has to work regardless of locale/encoding.
Post reply on HN