Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

261–270 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#261

Post by account42 »

throwaway2037 wrote: post #260
account42 wrote: post #227

Earlier quoted context omitted.

Qt really has no excuse for still using 16-bit characters since unlike the other two they have had multiple ABI breaks since then.

"no excuse" -- I would respectfully disagree here. There are lots of very smart people who have worked on Qt. Really, some insanely good C++ programmers have worked on that project. I have no doubt that they have discussed changing class QString to use UTF-8 internally. To be clear, probably QChar would also need to change, or a new class (QChar8?) would be needed, in parallel to QChar. I guess they concluded the API…

Ah yes, appeal to authority. No better way to admit that you are talking out of your arse.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#262

Post by gpderetta »

account42 wrote: post #220
squeaky-clean wrote: post #211

Earlier quoted context omitted.

That's not ascii then. It's byte width compatible (to a certain degree as you point out). But it's not ascii. ascii defines 128 code points and the handling of an escape character. It doesn't handle locales.

ASCII is an encoding, it doesn't say anything about locale. The point is that tolower/toupper is not guaranteed to be safe even if the input is 7-bit.

I don't think there is any possibility of doing locale specific lower/upper casing in ASCII. It is really designed for (a subset of) American english.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#263

Post by gpderetta »

squeaky-clean wrote: post #202
dwattttt wrote: post #192

Earlier quoted context omitted.

> The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant). 7-bit ASCII code points are fine, but outside that it's not guara…

Yeah If you're using narrow strings for UTF8 you're making a mistake. wstrings also are not a good representation because of the platform differences, unless you don't care about Windows in which case it's fine but still not great semantically. In most type definitions you cannot convert UTF8 via simple iteration because the type generally represents a code point and not a character. You can have a library where UTF8…

I'm not 100% sure what you mean by narrow string, but if you refer to std::string vs std::wstring, then std::string is perfectly fine for encoding UTF8, as that uses 8 bit code units which are guaranteed to fit in a char. On the other hand, std::wstring would be a bizarre choice for UTF8 on any platform.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#264

Post by sebstefan »

account42 wrote: post #235
sebstefan wrote: post #66

Earlier quoted context omitted.

First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's li…

You are not being excluded just because you need to use a romanized version of your name. Clear example of a first world problem.

>first world problem

? The more first world you are the more your alphabet is taken into consideration

Hint: You use the word """romanized"""

Re: A popular but wrong way to convert a string to uppercase or lowercase

#265

Post by crote »

neonsunset wrote: post #253
crote wrote: post #187

Earlier quoted context omitted.

The problem is that such third-party requirements are usually wrong . Two decades ago some developer probably went "Yeah, obviously all names start with capital letters!", not realizing that there are in fact plenty of names which start with a lowercase letter. So they added an input validation test which checks for capitals, which meant everyone feeding that system had to format their data. A whole ecosystem grew ar…

> So no, you cannot "simply" use ToUpper() / ToLower(). They might work well enough of basic ASCII for languages like English, but they have a habit of making a mess out of everything else. You're supposed to use CultureInfo.TextInfo.ToUpperCase() and explicitly specify what locale the text is in so that it can use the right converter. Which is of course essentially impossible in general-purpose text fields. Have you…

Yes. Now try applying it to something like this very HN comment section, which is mixing words belonging to different cultures inside a single comment - and in some cases even inside the same word.

Sure, you can now do case conversion for a specific culture, but which one?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#266

Post by kergonath »

Factory wrote: post #151
kergonath wrote: post #39

Earlier quoted context omitted.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

"The vast majority of the population does not read or write any English in their day to day lives." This is doubtful: https://en.wikipedia.org/wiki/List_of_languages_by_total_num... While English speakers are not a majority, it is the most popular language. And one should also note that given English is the lingua franca of programming, I'd suspect that English as a second language is actually a majority for programm…

> "The vast majority of the population does not read or write any English in their day to day lives." This is doubtful: https://en.wikipedia.org/wiki/List_of_languages_by_total_num... While English speakers are not a majority, it is the most popular language.

That is the number of English-speaking people, as in people who can speak English. Not necessarily people who use it every day. In any case, ASCII only works for a subset of even English if you ignore all loan words and diacritics in things like proper names.

> So any code that deals solely with programmers as users can easily just use standard ASCII as default, and never see any problems.

That would not be much code at all, given that most code deals with user interfaces or user-provided data. That is the point: it’s not because the code is in basic English simplified enough to fit in ASCII that you can ignore Unicode and don’t need to consider text encoding.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#267

Post by kergonath »

numpad0 wrote: post #147
kergonath wrote: post #39

Earlier quoted context omitted.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

> That’s why I still get mail with my name mangled Which is why you always type out addresses in ASCII representations in any foreign transactions even if it's not going to match your identity documents, unless the other party specifically demands it in UTF-8 and insists that they can handle it. > it’s better if a Chinese user name does not break your reporting or logging systems You should not be just casually dumpi…

> Which is why you always type out addresses in ASCII representations in any foreign transactions even if it's not going to match your identity documents, unless the other party specifically demands it in UTF-8 and insists that they can handle it.

That is not always possible and the translation from local writing system to ASCII is often not unique and ambiguous. There really is no excuse for this sort of thinking. Even American programmers have to realise at some point that programs serve some purpose and that their failure to represent how the world works is just that: a failure. There is no excuse for programs to not support UTF-8 from user input to any output, including all the processing in between.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#268

Post by kergonath »

account42 wrote: post #237
kergonath wrote: post #61

Earlier quoted context omitted.

I am really not sure about this 99%. A lot of programs deal with quite a lot of user-provided data, which you don’t control.

User-provided data, yes, but also data where you can treat non-ASCII bytes as garbage in -> garbage out. E.g. the config file might be typed by a human but if you need to support case-insensitive keys you still don't need to worry about Unicode.

Exactly. But in this case, don’t try to upper-case or otherwise transform anything.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#269

Post by Netch »

neonsunset wrote: post #254
Netch wrote: post #243

Earlier quoted context omitted.

> In C# I can simply do this: "hello world".ToUpper(); Hmm still actual: https://www.moserware.com/2008/02/does-your-code-pass-turkey...

> 2008 This is completely irrelevant because culture-sensitive case conversion relies on ICU/NLS.

But at least a programmer shall be aware to call it (whatever API is used).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#270

Post by [deleted] »

cm2187 wrote: post #48
pornel wrote: post #20

Earlier quoted context omitted.

Being developed in, and having to stay compatible with , ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but…

That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).

[deleted]
Post reply on HN