Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

61–70 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#61

Post by kergonath »

vardump wrote: post #42
kergonath wrote: post #39

Earlier quoted context omitted.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

I said use a Unicode library if input data is actual human language. Which names and addresses are. 99% case being ASCII data generated by other software of unknown provenance. (Or sometimes by humans, but it's still data for machines, not for humans.)

I am really not sure about this 99%. A lot of programs deal with quite a lot of user-provided data, which you don’t control.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#62

Post by kergonath »

Muromec wrote: post #45
kergonath wrote: post #39

Earlier quoted context omitted.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

Who and why still tries to lowercase/uppercase names? Please tell them to stop.

Hell if I know. I don’t know what kind of abomination e-commerce websites run on their backend, I just see the consequences.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#64

Post by Muromec »

sebstefan wrote: post #55
Muromec wrote: post #44

Earlier quoted context omitted.

If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.

From experience anything banking adjacent will usually fuck it up as well For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII

If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#65

Post by SAI_Peregrinus »

bayindirh wrote: post #41
blenderob wrote: post #35

Earlier quoted context omitted.

But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripher…

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

> Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons).

Because there are more than 65,535 characters. That's just writing systems, not Unicode's fault. Most of the unnecessary complexity of Unicode is legacy compatibility: UTF-16 & UTF-32 are bad ideas that increase complexity, but they predate UTF-8 which actually works decently well so they get kept around for backwards compatibility. Likewise with the need for multiple normalization forms.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#66

Post by sebstefan »

Muromec wrote: post #64
sebstefan wrote: post #55

Earlier quoted context omitted.

From experience anything banking adjacent will usually fuck it up as well For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII

If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.

First off: And exclude 70% of the world?

Usually they'll accept it, but some parts of the backend are still running code from the 60's.

So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's like. Mine is just accented latin characters so it usually drops the accents ; not sure how it would work if your name was in an entirely different alphabet

Re: A popular but wrong way to convert a string to uppercase or lowercase

#67

Post by pjmlp »

blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

Because it is a fight to put anything on a ISO managed language, and only the strongest persevere long enough to make it happen.

Regardless of what ISO language we are talking about.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#68

Post by bayindirh »

SAI_Peregrinus wrote: post #65
bayindirh wrote: post #41

Earlier quoted context omitted.

> Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules…

> Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons). Because there are more than 65,535 characters. That's just writing systems, not Unicode's fault. Most of the unnecessary complexity of Unicode is legacy compatibility: UTF-16 & UTF-32 are bad ideas that increas…

I mean, I already know some Unicode internals and linguistics (since I developed a language-specific compression algorithm back in the day), but I have never seen a single character requiring four bytes (and I know Emoji chaining for skin color, etc.).

So, seeing this just moved the complexity of Unicode one notch up in my head, and I respect the guys who designed and made it work. It was not whining or complaining of any sort. :)

Re: A popular but wrong way to convert a string to uppercase or lowercase

#69

Post by pjmlp »

cyxxon wrote: post #17

Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.

Lowering case is even better, because a Swiss user would expect the two-character sequence “SS“ to be converted into “ss“ and not “ß“.

And thus we add country specific locale to the party.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#70

Post by thiht »

zarzavat wrote: post #21
appointment wrote: post #5

The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.

Different parts of a string can be in different languages too[1]. The lowercase of "DON'T FUSS ABOUT FUSSBALL" is "don't fuss about fußball". Unless you're in Switzerland. [1] https://en.wikipedia.org/wiki/Code-switching

I thought the German language deprecated the use of ß years ago, no? I learned German for a year and that's what the teacher told us, but maybe it's not the whole story
Post reply on HN