Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

231–240 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#231

Post by account42 »

alkonaut wrote: post #178

Handle text in two ways: either it's controlled by you and you can do simple, efficient, and naive processing, or it's not (it's translated resources, or user input) and you can't. For the former case, you don't need any complex logic. A very typical example would be: i'm serializing a field or constructing a url so I want the variable name "Someproperty" as a lower case string. The lowercase transform is completely…

> For the former case, you don't need any complex logic. A very typical example would be: i'm serializing a field or constructing a url so I want the variable name "Someproperty" as a lower case string. The lowercase transform is completely naive. I know exactly what the range of possible characters are and they aren't going to be Turkish or emoji, not least because I have asserted they won't be. And THIS is what the regular programming functions for upper/lower case are for. They are important, and they are most often correct. Because for all the other cases (i18n, user input, ...) you probably don't want to do toUpper/toLower at all to begin with!

C++ std::tolower/toupper (which are really just C tolower/toupper) are the wrong tool for that too though because they depend on the process locale which makes them a) horribly inefficient and b) prone to blow your program up in interesting ways on customer systems. Not quite as bad as the locale-dependent standard number parsing functions that want . in some localses and , in others but still should never be used.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#232

Post by account42 »

vardump wrote: post #2

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.

> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters.

And std::tolower/toupper is the wrong tool for that as well.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#233

Post by account42 »

hinkley wrote: post #132
crazygringo wrote: post #116

Earlier quoted context omitted.

> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. I think it's more the exact opposite. The only times I'm dealing with 7-bit ASCII is for internal identifiers like variable names or API endpoints. Which is a lot of the time, but I can't ever think of when I've needed my code to change their case. It might literally be never. On the other han…

And you could argue that if the internal identifiers need to be capitalized or lower-cased, you've already lost. On an enterprise app these little string manipulations are a drop in the bucket. In a game they might not be. Sort that stuff out at compile time, or commit time.

You can't always control the case you get but often you can not care about anything outside ASCII. Scripts and configuration or text-based data formats are common examples.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#234

Post by account42 »

sebstefan wrote: post #37
vardump wrote: post #2

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.

Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.

Cool, I will.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#235

Post by account42 »

sebstefan wrote: post #66
Muromec wrote: post #64

Earlier quoted context omitted.

If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.

First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's li…

You are not being excluded just because you need to use a romanized version of your name. Clear example of a first world problem.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#236

Post by account42 »

sumeno wrote: post #139
MajimasEyepatch wrote: post #138

Earlier quoted context omitted.

It’s totally reasonable to assume your users are in the US if your business only sells to people in the US. I work in the health insurance sector; there’s absolutely no chance my company ever sells these products internationally. We can’t even sell them in every state.

It's not reasonable to assume that users in the US have names that only use 7-bit ASCII

It's reasonable to assume that all users can deal with having to encode their names in 7-bit ASCII. Otherwise you might as well demand that computer systems need to support arbitrary drawings in the name field at which point you might as well not have a name field at all because even most humans won't be able to deal with what you want to put in there.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#237

Post by account42 »

kergonath wrote: post #61
vardump wrote: post #42

Earlier quoted context omitted.

I said use a Unicode library if input data is actual human language. Which names and addresses are. 99% case being ASCII data generated by other software of unknown provenance. (Or sometimes by humans, but it's still data for machines, not for humans.)

I am really not sure about this 99%. A lot of programs deal with quite a lot of user-provided data, which you don’t control.

User-provided data, yes, but also data where you can treat non-ASCII bytes as garbage in -> garbage out. E.g. the config file might be typed by a human but if you need to support case-insensitive keys you still don't need to worry about Unicode.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#238

Post by account42 »

elpocko wrote: post #50
vardump wrote: post #7

Earlier quoted context omitted.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Every search feature everywhere has to be case-insensitive or it's unusable. Search seems like a pretty ubiquitous feature in a lot of software, and has to work regardless of locale/encoding.

Search needs a whole lot more normalization than just case folding.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#239

Post by account42 »

heisenzombie wrote: post #16
daemin wrote: post #6

Earlier quoted context omitted.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

Yes and most importantly, that interpretation is for display purposes ONLY. If your file manager won't let me delete a file because the name includes invalid UTF-16/UTF-8 then it is simply broken.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#240

Post by account42 »

BoringTimesGang wrote: post #15
daemin wrote: post #6

Earlier quoted context omitted.

I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and…

Now double all of that effort, so you can get it to work with Windows' UTF-16 wstrings.

Better to just convert WTF-16 (Windows filenames re not guaranteed to be valid UTF-16) to/from WTF-8 at the API boundary and then do the same processing internally on all platforms.
Post reply on HN