Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

141–150 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#141

Post by numpad0 »

SAI_Peregrinus wrote: post #65
bayindirh wrote: post #41

Earlier quoted context omitted.

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

> Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons). Because there are more than 65,535 characters. That's just writing systems, not Unicode's fault. Most of the unnecessary complexity of Unicode is legacy compatibility: UTF-16 & UTF-32 are bad ideas that increas…

It's because Unicode don't allow for language switching.

It takes up to eight bytes per character in Unicode if you want to support both Chinese and Japanese in a single font using IVS(and I don't think there's any font that actually supports this).

AFAICS(As far as I can search), Simplified(PRC) and Traditional(Taiwan) Chinese encoding are respectively called GB2312 and Big5, and they're both two byte encodings with good practical coverage. Same applies for Japanese Shift_JIS. If e.g. :flag_cc: were allowed to be used as start-of-language marker, one could theoretically cut that back down to two bytes per character without losing much and actually improving language supports.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#142

Post by numpad0 »

Muromec wrote: post #108
sebstefan wrote: post #66

Earlier quoted context omitted.

First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's li…

>First off: And exclude 70% of the world? Guess what, I'm part of this 70% and I also work in a bank and I know exactly how. Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway. So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in add…

Why can't these people understand that that 70% of the world consider ASCII to be "the computer language", not English, and UTF-8 to be "whatever soup that only works inside files and forms and can't be program manipulated"?

Maybe it needs to be communicated more often, like way more often, until it sticks.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#143

Post by steveklabnik »

wslh wrote: post #136
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

Me too, how is case conversion perfectly done in modern languages such as Zig [1], Rust, or Swift? [1] Ended up looking at https://github.com/JakubSzark/zig-string

In Rust, the APIs are clear if they're ASCII only or unicode aware.

https://doc.rust-lang.org/stable/std/primitive.str.html#meth...

> ‘Lowercase’ is defined according to the terms of the Unicode Derived Core Property Lowercase.

https://doc.rust-lang.org/stable/std/primitive.str.html#meth...

> ASCII letters ‘A’ to ‘Z’ are mapped to ‘a’ to ‘z’, but non-ASCII letters are unchanged.

Now, "perfectly" is very strong. For example, the Turkish i problem. That is not solved. But 99% of Unicode stuff is handled correctly by default.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#144

Post by wakawaka28 »

MBCook wrote: post #130
BoringTimesGang wrote: post #54

Earlier quoted context omitted.

Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions. Or perhaps I should say naïve.

So what? That doesn’t prevent adding a new function that converts an entire string to upper or lowercase in a Unicode aware way. What would be wrong with adding new correct functions to the standard library to make this easy? There are already namespaces in C++ so you don’t even have to worry about collisions. That’s the problem I see. It’s fine if you have a history of stuff that’s not that great in hindsight. But w…

The reason that wasn't done is because Unicode is not really in older C++ standards. I think it may have been added to C++23 but I am not familiar with that. There are many partial solutions in older C++ but if you want to do it well then you need to get a library for it from somewhere, or else (possibly) wait for a new standard.

Unicode and character encodings are pretty esoteric. So are fonts. The stuff is technically everywhere and fundamental, but there are many encodings, technical details, etc. And most programmers only care about one language, or else only use UTF-8 with the most basic chars (the ones that agree with ASCII). That isn't terrible. You only need what you actually need. Most programs don't strictly have to be built for multiple random languages, and there is kind of a standard methodology to learn before you can do that.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#145

Post by wakawaka28 »

relaxing wrote: post #28
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

Java was built from scratch as a heavy language with a whole portability layer that C++ does not have. Also, libraries have been around to do this stuff in C++ but maybe some people saw it better to not require C++ to support Unicode, presumably.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#146

Post by squeaky-clean »

fluoridation wrote: post #115
squeaky-clean wrote: post #110

Earlier quoted context omitted.

For ascii in C++ isn't there std::tolower / std::toupper? If you're not dealing with unsigned char types there isn't a simple case conversion function, but that's for a good reason as the article lays out.

Those functions take and return single characters. What's missing is functions that operate on strings. You can use them in combination with std::transform(), but as the article points out, even if you're just dealing with ASCII you can easily do it wrong. I've been using C++ for over 20 years and I didn't know tolower() and toupper() were non-addressable. There's really no excuse for the library not having simple ca…

std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside.

The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place.

If you are operating on wide strings, there is no suitable single solution, partly because wstring is a terrible type. It's different widths on different platforms, and no string encoding format uses a generalized wsring, they have mandatory min/max character byte widths. So a wstring tells you nothing about the actual encoded string contents semantic representation.

The C++ stdlib could include a fully unicode aware string type set, and surrounding library. But personally I think C++ isn't the kind of language to provide an opinionated stdlib module for such a complex task. And there's no way to implement such a module without being very opinionated about something.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#147

Post by numpad0 »

kergonath wrote: post #39
vardump wrote: post #7

Earlier quoted context omitted.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

> That’s why I still get mail with my name mangled

Which is why you always type out addresses in ASCII representations in any foreign transactions even if it's not going to match your identity documents, unless the other party specifically demands it in UTF-8 and insists that they can handle it.

> it’s better if a Chinese user name does not break your reporting or logging systems

You should not be just casually dumping Chinese usernames into logs without warnings, in fact, you should not be using Chinese characters for usernames at all. Lots of Chinese online services exclusively use numeric IDs and e-mails for login IDs. "Usernames in natural human language" is a valid concept only in ASCII cultural sphere.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#148

Post by lilyball »

PoignardAzur wrote: post #117

So I'm going to be that guy and say it: Man, I'm happy we don't need to deal with this crap in Rust, and we can just use String::to_lowercase. Not having to worry about things makes coding fun.

While certainly much better, you still need to be aware that doing case conversion absent any locale information will never be perfect. If you want proper locale-aware conversion you can use the icu crate (https://docs.rs/icu/latest/icu/).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#149

Post by numpad0 »

vardump wrote: post #18
heisenzombie wrote: post #16

Earlier quoted context omitted.

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.

Just using UTF-8 for username at all is problematic. That has been a major PSA item for Windows users in my language literally since 90s and still is. Microsoft switched home folder names from Microsoft Account username to shortened user email for that reason.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#150

Post by lmm »

fluoridation wrote: post #86
SleepyMyroslav wrote: post #73

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…

>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :) I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. What if your game needs to talk to a server and do s…

> What if your game needs to talk to a server and do some string manipulation in between requests?

What conceivable reason would there be to ever need to do that? If the server takes commands in upper case, then have them in upper case from the start. If the server takes commands in lower case, have them in lower case from the start. If the server specifies that you need to invert the case of its response to use in the next request, find a server developed by someone not crazy.

Post reply on HN