Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

31–40 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#31

Post by vardump »

inexcf wrote: post #23
vardump wrote: post #7

Earlier quoted context omitted.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?

Here's an example. Hypothetically say you want to build an HTML parser.

You might encounter tags like , , , etc., but you want to perform a hash table lookup.

So first you're going to normalize to either lower- or uppercase.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#32

Post by Someone »

vardump wrote: post #18
heisenzombie wrote: post #16

Earlier quoted context omitted.

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.

> It's a good idea to consider file paths as a bag of bytes

(Nitpick: sequence of bytes)

Also very limiting. If you do that, you can’t, for example, show a file name to the user as a string or easily use a shell to process data in your file system (do you type “/bin” or “\x2F\x62\x69\x6E”?)

Unix, from the start, claimed file names where byte sequences, yet assumed many of those to encode ascii.

That’s part of why Plan 9 made the choice “names may contain any printable character (that is, any character outside hexadecimal 00-1F and 80-9F)” (https://9fans.github.io/plan9port/man/man9/intro.html)

Re: A popular but wrong way to convert a string to uppercase or lowercase

#33

Post by bayindirh »

relaxing wrote: post #28
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

> There’s no excuse.

I politely disagree. None of the programming languages which started integrating Unicode was targeting from bare metal to GUI, incl. embedded and OS development at the same time.

C++ has a great target area when compared to other programming languages. There are widely used libraries which compile correctly on PDP-11s, even if they are updated constantly.

You can't just say "I'll be just making everything Unicode aware, backwards compatibility be damned, eh".

Re: A popular but wrong way to convert a string to uppercase or lowercase

#34

Post by daemin »

vardump wrote: post #18
heisenzombie wrote: post #16

Earlier quoted context omitted.

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.

That's what I mean, you treat filesystem paths as bags of bytes separated by known ASCII characters, as the only path manipulation that you generally need to do is to append a path, remove a path, change extension, things that only care about those ASCII characters. You only modify the path strings at those known characters and leave everything in between as is (with some exceptions using OS API specific functions as needed).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#35

Post by blenderob »

bayindirh wrote: post #33
relaxing wrote: post #28

Earlier quoted context omitted.

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

> There’s no excuse. I politely disagree. None of the programming languages which started integrating Unicode was targeting from bare metal to GUI, incl. embedded and OS development at the same time. C++ has a great target area when compared to other programming languages. There are widely used libraries which compile correctly on PDP-11s, even if they are updated constantly. You can't just say "I'll be just making e…

But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything.

But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripherals. So one can dream that such a type along with its conversion routines could be added to an updated standard library without breaking existing code that compiles correctly on PDP-11s.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#36

Post by netsharc »

vardump wrote: post #18
heisenzombie wrote: post #16

Earlier quoted context omitted.

File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization

File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.

IIRC, the FAT filesystem (before Windows 95) allowed lowercase letters, but there's a layer in the filesystem driver that converted everything to uppercase, e.g. if you did the command "more readme.txt", the more command would ask the filesystem for "readme.txt" and it would search for "README.TXT" in the file allocation table.

I think I once hex-edited the FA-table to change a filename to have a lowercase name (or maybe it was disk corruption), trying to delete that file didn't work because it would be trying to delete "FOO", and couldn't find it because the file was named "FOo".

Re: A popular but wrong way to convert a string to uppercase or lowercase

#37

Post by sebstefan »

vardump wrote: post #2

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.

Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#39

Post by kergonath »

vardump wrote: post #7
fhars wrote: post #4

Earlier quoted context omitted.

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me…

In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages.

ASCII is mostly fine within your programs like the parser you mention in your other comment. But even then, it’s better if a Chinese user name does not break your reporting or logging systems or your parser, so it’s still a good idea to take Unicode seriously. Otherwise, anything that comes from a user or gets out of the program needs to behave.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#40

Post by Muromec »

blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

Well, the only time you can do str lower where unicode locale awareness will be a problem is when you do it on the user input, like names.

How about you just dont? If it's a constant in your code, you probably use ASCII anyway or can do a static mapping. If it's user user input -- just don't str lower / str upper it.

Post reply on HN