Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular but wrong way to convert a string to uppercase or lowercase

devblogs.microsoft.com

Post reply on HN
Print view View original

151–160 of 272 posts

Re: A popular but wrong way to convert a string to uppercase or lowercase

#151

Post by Factory »

kergonath wrote: post #39
vardump wrote: post #7

Earlier quoted context omitted.

> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.

Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me… In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages. ASCII…

"The vast majority of the population does not read or write any English in their day to day lives." This is doubtful: https://en.wikipedia.org/wiki/List_of_languages_by_total_num... While English speakers are not a majority, it is the most popular language. And one should also note that given English is the lingua franca of programming, I'd suspect that English as a second language is actually a majority for programmers. So any code that deals solely with programmers as users can easily just use standard ASCII as default, and never see any problems.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#152

Post by moron4hire »

rty32 wrote: post #131
beeboobaa3 wrote: post #122

Earlier quoted context omitted.

> There are major issues just to show user back what he entered! Because the font for editing and displayed text could be different. Not even mentioning RTL and other issues. Your web browser is doing it right now as you are reading this comment.

And web development is not game development? And chances are that games don't ship chromium with them?

Actually...

  https://github.com/baikety/uWebKit
  https://zenfulcrum.com/browser/docs/Readme.html
  https://github.com/roydejong/chromium-unity-server
There are a lot more, I just got bored at 3.

And it's not just Unity. Several exist for Unreal as well.

Why? Specifically because 2D layout and text rendering suck so much in game engines. What's ~50MB matter when you're shipping several GB of game assets?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#153

Post by extraduder_ire »

kccqzy wrote: post #119
IncreasePosts wrote: post #107

Earlier quoted context omitted.

That was only adopted in Germany like 7 years ago!

Well languages and conventions change. The € sign was added not that long ago and it was somewhat painful. The Chinese language uses a single character to refer to chemical elements so when IUPAC names new elements they will invent new characters. Etc.

Does unicode have space set aside for those new symbols to slot into? I know it's very rare, but it could get messy.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#154

Post by fluoridation »

lmm wrote: post #150
fluoridation wrote: post #86

Earlier quoted context omitted.

>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :) I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. What if your game needs to talk to a server and do s…

> What if your game needs to talk to a server and do some string manipulation in between requests? What conceivable reason would there be to ever need to do that? If the server takes commands in upper case, then have them in upper case from the start. If the server takes commands in lower case, have them in lower case from the start. If the server specifies that you need to invert the case of its response to use in t…

Case conversion is not the only string manipulation that's locale sensitive.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#155

Post by fluoridation »

squeaky-clean wrote: post #146
fluoridation wrote: post #115

Earlier quoted context omitted.

Those functions take and return single characters. What's missing is functions that operate on strings. You can use them in combination with std::transform(), but as the article points out, even if you're just dealing with ASCII you can easily do it wrong. I've been using C++ for over 20 years and I didn't know tolower() and toupper() were non-addressable. There's really no excuse for the library not having simple ca…

std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…

I know I can simply iterate. The point is that it's a function that should be included, not that it's impossible without it. It's one of the most common string operations.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#156

Post by lmm »

fluoridation wrote: post #154
lmm wrote: post #150

Earlier quoted context omitted.

> What if your game needs to talk to a server and do some string manipulation in between requests? What conceivable reason would there be to ever need to do that? If the server takes commands in upper case, then have them in upper case from the start. If the server takes commands in lower case, have them in lower case from the start. If the server specifies that you need to invert the case of its response to use in t…

Case conversion is not the only string manipulation that's locale sensitive.

No reasonable server API should require locale sensitive string manipulation.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#157

Post by thayne »

appointment wrote: post #5

The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.

Not quite.

islower is actually supposed to account for the user's "locale", which includes their language.

The key takeway is that lowercasing a string needs to be done on the whole string, not individual characters, even if std::string had a way to iterate over codepoints instead of bytes (or code units, in the case of wstring).

And there isn't a standard way to do that, you either meed to use a platform specific API, like the windows function mentioned, or use a library like ICU.

Re: A popular but wrong way to convert a string to uppercase or lowercase

#158

Post by oguz-ismail »

steveklabnik wrote: post #143
wslh wrote: post #136

Earlier quoted context omitted.

Me too, how is case conversion perfectly done in modern languages such as Zig [1], Rust, or Swift? [1] Ended up looking at https://github.com/JakubSzark/zig-string

In Rust, the APIs are clear if they're ASCII only or unicode aware. https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ‘Lowercase’ is defined according to the terms of the Unicode Derived Core Property Lowercase. https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ASCII letters ‘A’ to ‘Z’ are mapped to ‘a’ to ‘z’, but non-ASCII letters are unchanged. Now, "perfectly" is very strong. For e…

> 99% of Unicode stuff

Does that include context-dependent conversion rules like o'reilly -> O'Reilly?

Re: A popular but wrong way to convert a string to uppercase or lowercase

#159

Post by makeitdouble »

zahlman wrote: post #84
SleepyMyroslav wrote: post #73

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…

>If text is coming from user - then change design until its not needed to 'convert' In games, you can possibly get away with this. Most other people need to worry about things like string collation (locale-aware sorting) for user-supplied text.

TBF, if you are caring about string collation, you're already at the entrance of the rabbit hole and probably should go down to the deep end anyway.

I'd assume SleepyMyroslav doesn't apply to devs willing to spend weeks at time to handle all the complexity in full.

Post reply on HN