Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

81–90 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#81

Post by ectospheno »

bayindirh wrote: post #11
blenderob wrote: post #3

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with its…

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

> Any tool which is old enough will have a thousand ways to do something.

Only because of the strange desire of programmers to never stop. Not every program is a never ending story. Most are short stories their authors bludgeon into a novel.

Programming languages bloat into stupidity for the same reason. Nothing is ever removed. Programmers need editors.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#82

Post by zahlman »

kccqzy wrote: post #75
cm2187 wrote: post #74

Earlier quoted context omitted.

Well pretty much every other more recent language solved that problem.

Almost no programming language, perhaps other than Swift, solved that problem. Just use the article's examples as test cases. It's just as wrong as the C++ version in the article, except it's wrong with nicer syntax.

Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worried about surrogate pairs, and it knows a few things about localized text casing:

    >>> 'ß'.upper()
    'SS'
    >>> 'ß'.lower()
    'ß'
    >>> 'ß'.casefold()
    'ss'
There are a lot of really complicated tasks for Unicode strings. String casing isn't really one of them.

(No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)

Re: A popular but wrong way to convert a string to uppercase or lowercase

#83

Post by CamperBob2 »

47282847 wrote: post #71
thiht wrote: post #70

Earlier quoted context omitted.

I thought the German language deprecated the use of ß years ago, no? I learned German for a year and that's what the teacher told us, but maybe it's not the whole story

Incorrect. ẞ is still a thing.

Going by what you and the grandparent wrote, it's not just a thing, but two different things: ẞ ß

It is probably time for an Esperanto advocate to show up and set us all straight.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#84

Post by zahlman »

SleepyMyroslav wrote: post #73

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…

>If text is coming from user - then change design until its not needed to 'convert'

In games, you can possibly get away with this. Most other people need to worry about things like string collation (locale-aware sorting) for user-supplied text.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#86

Post by fluoridation »

SleepyMyroslav wrote: post #73

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…

>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :)

I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters.

What if your game needs to talk to a server and do some string manipulation in between requests? Are you really going to architect everything so that the client doesn't need to handle any of that ever?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#87

Post by zahlman »

recursive wrote: post #77
criddell wrote: post #72

Earlier quoted context omitted.

Strings in C++ standard library do suck (and C++ is my favorite language). As for UTF-16, well, I don't know that UTF-8 is a whole lot more intuitive: > And for UTF-8 data, you have the same issues discussed before: Multibyte characters will not be converted properly, and it breaks for case mappings that alter string lengths.

UTF-16 has all the complexity of UTF-8 plus surrogate pairs.

Surrogate pairs aren't more complex than UTF-8's scheme for determining the number of bytes used to represent a code point. (Arguably the logic is slightly simpler.) But the important point is that UTF-16 pretends to be a constant-length encoding while actually having the surrogate-pair loophole - that's because it's a hack on top of UCS-2 (which originally worked well enough for Microsoft to get married to; but then the BMP turned out not to be enough code points). UTF-8 is clearly designed from scratch to be a multi-byte encoding (and, while the standard now makes the corresponding sequences illegal, the scheme was designed to be able to support much higher code points - up to 2^42 if we extend the logic all the way; hypothetical 6-byte sequences starting with values FC or FD would neatly map up to 2^31).

Re: A popular but wrong way to convert a string to uppercase or lowercase

#88

Post by fluoridation »

ectospheno wrote: post #81
bayindirh wrote: post #11

Earlier quoted context omitted.

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mam…

> Any tool which is old enough will have a thousand ways to do something. Only because of the strange desire of programmers to never stop. Not every program is a never ending story. Most are short stories their authors bludgeon into a novel. Programming languages bloat into stupidity for the same reason. Nothing is ever removed. Programmers need editors.

So how do you design a language that accommodates both the people who need a codebase to be stable for decades and the people who want the bleeding edge all the time, backwards compatibility be damned?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#90

Post by the_gorilla »

fluoridation wrote: post #88
ectospheno wrote: post #81

Earlier quoted context omitted.

> Any tool which is old enough will have a thousand ways to do something. Only because of the strange desire of programmers to never stop. Not every program is a never ending story. Most are short stories their authors bludgeon into a novel. Programming languages bloat into stupidity for the same reason. Nothing is ever removed. Programmers need editors.

So how do you design a language that accommodates both the people who need a codebase to be stable for decades and the people who want the bleeding edge all the time, backwards compatibility be damned?

You don't. Any language that tries to do both turns into an unusable abomination like C++. Good languages are stable and the bleeding edge is just the "new thing" and not necessarily better than the old thing.
Post reply on HN