Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

211–220 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#211

Post by squeaky-clean »

account42 wrote: post #204
dwattttt wrote: post #192

Earlier quoted context omitted.

> The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant). 7-bit ASCII code points are fine, but outside that it's not guara…

It's not guaranteed for 7-bit ASCII either because tolower/toupper are locale-dependent and with the tr_TR lowercase I (U+0049) is ı (U+0131, aka dotless i) wich encodes as two bytes in UTF-8.

That's not ascii then. It's byte width compatible (to a certain degree as you point out). But it's not ascii. ascii defines 128 code points and the handling of an escape character. It doesn't handle locales.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#212

Post by squeaky-clean »

gpderetta wrote: post #207
squeaky-clean wrote: post #146

Earlier quoted context omitted.

std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…

std::u8string, std::u16string and std::u32string are supposed to be the portable unicode string types, but a lot of machinery is missing and some that has been added has since been deprecated. > there's no way to implement such a module without being very opinionated about something. indeed! Boost.Nowide[1] is such an opinionated library. [1] https://www.boost.org/doc/libs/master/libs/nowide/doc/html/i...

Yep, there's also ICU and utf8cpp, and many others. They all have trade-offs. So I just don't think the stdlib should cover this because there is no objectively best way to handle it.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#213

Post by account42 »

blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

> Makes you wonder why this isn't part of the C++ standard library itself.

Because the C++ standard library cares about binary size and backwards compatiblity, both of with are incompatible with a full Unicode implementation. Putting this in the stdlib means everyone has to pay for it even when you don't need it.

Libraries are fine, not everything needs to be defined by the language itself.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#214

Post by account42 »

pornel wrote: post #20
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

Being developed in, and having to stay compatible with , ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but…

No, strong backwards compatiblity a real strength of C++. In fact, it's probably it's main strength these days.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#215

Post by account42 »

cm2187 wrote: post #112
tedunangst wrote: post #95

Earlier quoted context omitted.

But that's wrong. The upper case for ß is ẞ.

C#'s "ToUpper" takes an optional CultureInfo argument if you want to play around with how to treat different languages. Again, solved problem decades ago.

This is not a locale issue, it's a Unicode version issue. Which hightlights another problem with adding this to the base standard library.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#216

Post by account42 »

extraduder_ire wrote: post #153
kccqzy wrote: post #119

Earlier quoted context omitted.

Well languages and conventions change. The € sign was added not that long ago and it was somewhat painful. The Chinese language uses a single character to refer to chemical elements so when IUPAC names new elements they will invent new characters. Etc.

Does unicode have space set aside for those new symbols to slot into? I know it's very rare, but it could get messy.

Unicode is already messy. Chinese characters especially so due to han unificiation.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#217

Post by account42 »

Towaway69 wrote: post #176
tedunangst wrote: post #95

Earlier quoted context omitted.

But that's wrong. The upper case for ß is ẞ.

Isn't uppercase for ß just ß - i.e. it's its own uppercase character?

Not generally no, but some applications used it that way because of ambiguity of upppercasing ß to SS - which is why ẞ was added.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#218

Post by account42 »

bratwurst3000 wrote: post #210
Towaway69 wrote: post #176

Earlier quoted context omitted.

Isn't uppercase for ß just ß - i.e. it's its own uppercase character?

there shouldn’t be an uppercase version of ß because there is no word in the german language that uses it as the first letter. the german language didnt think of allcaps. please correct me if I am wrong. If written in uppercase it should be converted to SZ or the new uppercase ß…. which my iphone doesn’t have… and converting anything to uppercase SS isn’t something germany wants …

> there shouldn’t be an uppercase version of ß because there is no word in the german language that uses it as the first letter. the german language didnt think of allcaps.

Allcaps (and smallcaps) has always existed in signage everywhere. Before the computing age, letters where just arbitrary metal stamps -- and just whatever you could draw before that. Historically, language was not as standardized as it is today.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#219

Post by account42 »

tialaramex wrote: post #127
kccqzy wrote: post #75

Earlier quoted context omitted.

Almost no programming language, perhaps other than Swift, solved that problem. Just use the article's examples as test cases. It's just as wrong as the C++ version in the article, except it's wrong with nicer syntax.

Rust will cheerfully: assert_eq!("ὀδυσσεύς", "ὈΔΥΣΣΕΎΣ".to_lowercase()); [Notice that this is in fact entirely impossible with the naive strategy since Greek cares about position of symbols] Some of the latter examples aren't cases where a programming language or library should just "do the right thing" but cases of ambiguity where you need locale information to decide what's appropriate, which isn't "just as wrong a…

Is this

    assert_eq!("\u1F41δυσσεύς", "ὈΔΥΣΣΕΎΣ".to_lowercase());
or

    assert_eq!("\u03BF\u0314δυσσεύς", "ὈΔΥΣΣΕΎΣ".to_lowercase());
For display it doesn't matter but most other applications really want some kind of normalizatin which does much much more so having a convenient to_lowercase() doesn't buy you as much as you think and can be actively misleading.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#220

Post by account42 »

squeaky-clean wrote: post #211
account42 wrote: post #204

Earlier quoted context omitted.

It's not guaranteed for 7-bit ASCII either because tolower/toupper are locale-dependent and with the tr_TR lowercase I (U+0049) is ı (U+0131, aka dotless i) wich encodes as two bytes in UTF-8.

That's not ascii then. It's byte width compatible (to a certain degree as you point out). But it's not ascii. ascii defines 128 code points and the handling of an escape character. It doesn't handle locales.

ASCII is an encoding, it doesn't say anything about locale. The point is that tolower/toupper is not guaranteed to be safe even if the input is 7-bit.
Post reply on HN