Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

251–260 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#251

Post by Muromec »

numpad0 wrote: post #142
Muromec wrote: post #108

Earlier quoted context omitted.

>First off: And exclude 70% of the world? Guess what, I'm part of this 70% and I also work in a bank and I know exactly how. Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway. So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in add…

Why can't these people understand that that 70% of the world consider ASCII to be "the computer language", not English, and UTF-8 to be "whatever soup that only works inside files and forms and can't be program manipulated"? Maybe it needs to be communicated more often, like way more often, until it sticks.

Well, it's much easier to understand the difference when one and another are using different alphabets.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#252

Post by PaulDavisThe1st »

oguz-ismail wrote: post #173
PaulDavisThe1st wrote: post #170

Earlier quoted context omitted.

that is neither up-casing nor-downcasing, but (de)capitalization, which is a significantly more complex task (which ultimately requires up- or down-casing, but a whole lot more before then).

So it doesn't. If Unicode doesn't cover non-trivial forms of case-folding, 99% of Unicode doesn't mean anything.

I am not aware of a Unicode concept of "the latin letter o followed by an apostrophe followed by another latin letter". Unicode would identify the glyphs for such a concept, but I don't see how Unicode is involved in this in anyway as far the process of deciding what "capitalized o'reilly" means.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#253

Post by neonsunset »

crote wrote: post #187
cheema33 wrote: post #169

Earlier quoted context omitted.

> In gamedev there is simple rule: don't try to do any of that. I am not in gamedev, but I frequently have to develop middleware that takes in user entered data and formats it in a way that will import into a 3rd party system without errors. And that sometimes means changing the case on strings. In my experience as a developer, this is very very common requirement. Luckily I am not forced to use a low level language…

The problem is that such third-party requirements are usually wrong . Two decades ago some developer probably went "Yeah, obviously all names start with capital letters!", not realizing that there are in fact plenty of names which start with a lowercase letter. So they added an input validation test which checks for capitals, which meant everyone feeding that system had to format their data. A whole ecosystem grew ar…

> So no, you cannot "simply" use ToUpper() / ToLower(). They might work well enough of basic ASCII for languages like English, but they have a habit of making a mess out of everything else. You're supposed to use CultureInfo.TextInfo.ToUpperCase() and explicitly specify what locale the text is in so that it can use the right converter. Which is of course essentially impossible in general-purpose text fields.

Have you ever read the documentation? https://learn.microsoft.com/en-us/dotnet/fundamentals/runtim...

Re: A popular but wrong way to convert a string to uppercase or lowercase

#254

Post by neonsunset »

Netch wrote: post #243
cheema33 wrote: post #169

Earlier quoted context omitted.

> In gamedev there is simple rule: don't try to do any of that. I am not in gamedev, but I frequently have to develop middleware that takes in user entered data and formats it in a way that will import into a 3rd party system without errors. And that sometimes means changing the case on strings. In my experience as a developer, this is very very common requirement. Luckily I am not forced to use a low level language…

> In C# I can simply do this: "hello world".ToUpper(); Hmm still actual: https://www.moserware.com/2008/02/does-your-code-pass-turkey...

> 2008

This is completely irrelevant because culture-sensitive case conversion relies on ICU/NLS.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#255

Post by Towaway69 »

account42 wrote: post #217
Towaway69 wrote: post #176

Earlier quoted context omitted.

Isn't uppercase for ß just ß - i.e. it's its own uppercase character?

Not generally no, but some applications used it that way because of ambiguity of upppercasing ß to SS - which is why ẞ was added.

On the other hand, the German language has existed for several hundred years without having a capital ß but now it needs one?

True capitalisation has always existed but even that didn’t seem to have required a capital ß - why now?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#256

Post by Towaway69 »

bratwurst3000 wrote: post #210
Towaway69 wrote: post #176

Earlier quoted context omitted.

Isn't uppercase for ß just ß - i.e. it's its own uppercase character?

there shouldn’t be an uppercase version of ß because there is no word in the german language that uses it as the first letter. the german language didnt think of allcaps. please correct me if I am wrong. If written in uppercase it should be converted to SZ or the new uppercase ß…. which my iphone doesn’t have… and converting anything to uppercase SS isn’t something germany wants …

I don’t think that Germany wants a capital ß or the German language requires one rather technology needs one to dot the eyes and cross the tees.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#257

Post by elpocko »

account42 wrote: post #238
elpocko wrote: post #50

Earlier quoted context omitted.

Every search feature everywhere has to be case-insensitive or it's unusable. Search seems like a pretty ubiquitous feature in a lot of software, and has to work regardless of locale/encoding.

Search needs a whole lot more normalization than just case folding.

Okay.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#258

Post by sumeno »

account42 wrote: post #236
sumeno wrote: post #139

Earlier quoted context omitted.

It's not reasonable to assume that users in the US have names that only use 7-bit ASCII

It's reasonable to assume that all users can deal with having to encode their names in 7-bit ASCII. Otherwise you might as well demand that computer systems need to support arbitrary drawings in the name field at which point you might as well not have a name field at all because even most humans won't be able to deal with what you want to put in there.

Nice slippery slope you've got there

Re: A popular but wrong way to convert a string to uppercase or lowercase

#259

Post by nitwit005 »

relaxing wrote: post #28
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

Java embraced Unicode, and ended up with a mess as Unicode changed underneath it.

You can actually end up in a cleaner state in C++, as there is no obligation to use the standard library string classes, but it's pretty much required in Java.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#260

Post by throwaway2037 »

account42 wrote: post #227
throwaway2037 wrote: post #195

Earlier quoted context omitted.

Pretty much all Unicode early adopters went for 16-bit chars. Qt and Win32 API are another pair.

Qt really has no excuse for still using 16-bit characters since unlike the other two they have had multiple ABI breaks since then.

"no excuse" -- I would respectfully disagree here. There are lots of very smart people who have worked on Qt. Really, some insanely good C++ programmers have worked on that project. I have no doubt that they have discussed changing class QString to use UTF-8 internally. To be clear, probably QChar would also need to change, or a new class (QChar8?) would be needed, in parallel to QChar. I guess they concluded the API breakage would be too severe. I assume Java and Win32/DotNet decided the same. Finally, you can Google for old mailing list discussions about QString using UTF-16. Many before have asked "can we just change to UTF-8?".
Post reply on HN