Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

221–230 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#221

Post by account42 »

relaxing wrote: post #28
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

Java has 16-bit character types. It is in no way better at modern Unicode than C++ while being needlessly less efficient for mostly-ASCII text like XML-like markup.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#222

Post by account42 »

bayindirh wrote: post #41
blenderob wrote: post #35

Earlier quoted context omitted.

But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripher…

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

> Germans have their ß to S (or capital ß depending on the year)

FYI, it's never S. If there is no better option then SS and ss are the proper capital and lowercase substitutions.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#223

Post by fluoridation »

squeaky-clean wrote: post #205
fluoridation wrote: post #155

Earlier quoted context omitted.

I know I can simply iterate. The point is that it's a function that should be included, not that it's impossible without it. It's one of the most common string operations.

To me that feels like the JS community asking for left-pad or is-even in a module. Why have a dedicated function for 2 lines of code? And it's a huge footgun. There is no ascii type in C++. People will use the generalized tolower for UTF8 encoded in narrow strings and have issues. You could say the generalized tolower should support all the different width/encoding combinations and sort it out. But that's still highl…

>Why have a dedicated function for 2 lines of code?

Then why does std::max() exist?

>People will use the generalized tolower for UTF8 encoded in narrow strings and have issues.

tolower() and toupper() work correctly on UTF-8 strings, because UTF-8 was specifically designed so that non-ASCII characters were represented by sequences of purely non-ASCII bytes.

>Generalized string conversion is a very complex

Hence why people who say C++ should have a tolower() that operates on strings are not asking more complex Unicode support.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#224

Post by TorKlingberg »

bayindirh wrote: post #68
SAI_Peregrinus wrote: post #65

Earlier quoted context omitted.

> Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons). Because there are more than 65,535 characters. That's just writing systems, not Unicode's fault. Most of the unnecessary complexity of Unicode is legacy compatibility: UTF-16 & UTF-32 are bad ideas that increas…

I mean, I already know some Unicode internals and linguistics (since I developed a language-specific compression algorithm back in the day), but I have never seen a single character requiring four bytes (and I know Emoji chaining for skin color, etc.). So, seeing this just moved the complexity of Unicode one notch up in my head, and I respect the guys who designed and made it work. It was not whining or complaining o…

Lots of emoji are outside the Basic Multilingual Plane and need 4 bytes in UTf-8 and UTF-16. That's without going into skin color and other modifiers and combinations.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#225

Post by account42 »

SAI_Peregrinus wrote: post #65
bayindirh wrote: post #41

Earlier quoted context omitted.

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

> Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons). Because there are more than 65,535 characters. That's just writing systems, not Unicode's fault. Most of the unnecessary complexity of Unicode is legacy compatibility: UTF-16 & UTF-32 are bad ideas that increas…

The number of characters is not the problem, the mess due to legacy compatibility is - case folding and normaltization could be much simpler if the codepoints were laid out with that in mind. Also the fact the Unicode can't make up its mind if it wants to encode glyphs (turkish I and i, han unification) or semantic characters (e.g. cyrillic vs. latin letters) or just "ideas" (emojis).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#226

Post by PaulDavisThe1st »

pjmlp wrote: post #185
PaulDavisThe1st wrote: post #171

Earlier quoted context omitted.

He may be right, but approximately 75% of the problems he describes are all Microsoft-ecosystem specific. In Unix-land we don't use wchar_t or UTF-16, and his article is a good demonstrations of why not.

UNIX land is even worse in international languages support. As in, there isn't even something on POSIX at the level other operaring systems support for localisation. Yes there is some locale stuff, however not enough for all stuff, hence why every modern programming language happens to have this as part of their standard library.

When the notion of what a "string" is differs so much from language to language, i18n is never going to be an effective part of POSIX.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#227

Post by account42 »

throwaway2037 wrote: post #195
gpderetta wrote: post #101

Earlier quoted context omitted.

Java ended up picking UCS-2 and getting screwed.

Pretty much all Unicode early adopters went for 16-bit chars. Qt and Win32 API are another pair.

Qt really has no excuse for still using 16-bit characters since unlike the other two they have had multiple ABI breaks since then.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#228

Post by account42 »

Muromec wrote: post #40
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

Well, the only time you can do str lower where unicode locale awareness will be a problem is when you do it on the user input, like names. How about you just dont? If it's a constant in your code, you probably use ASCII anyway or can do a static mapping. If it's user user input -- just don't str lower / str upper it.

Yes, except when it is not your choice. If the requirements are to display some strings in lower/uppercase then you need to find a way to do that. That doesn't have to be using the standard library though.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#229

Post by account42 »

pjmlp wrote: post #67
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

Because it is a fight to put anything on a ISO managed language, and only the strongest persevere long enough to make it happen. Regardless of what ISO language we are talking about.

If anything it should be harder to add things to the language. Too many new additions have been half-arsed like and needed to be changed or deprecated soon after.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#230

Post by account42 »

BoringTimesGang wrote: post #22
tralarpa wrote: post #19

Earlier quoted context omitted.

The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).

to_lower is in the std namespace but is actually just part of the C89 standard, meaning it predates both UTF8 and UTF16. Is the alternative that it should be made unusable, and more existing code broken? A modern user has to include one of the c-prefix headers to use it, already hinting to them that 'here be dragons'. But there are always dragons. It's strings. The mere assumption that they can be transformed int-by-…

> Is the alternative that it should be made unusable, and more existing code broken?

It should be marked [[deprecated]], yes. There is no good reason to use std::tolower/toupper anywhere - they can neither do unicode properly nor are they anywhere close to efficient for ASCII. And their behavior depends on the process-global locale.

Post reply on HN