191–200 of 272 posts
I'm not sure about other languages, but Swift has pretty intense String support[0], and can go quite a long ways.
Someone actually wrote a whole book about just Swift Strings[1].
[0] https://docs.swift.org/swift-book/documentation/the-swift-pr...
Earlier quoted context omitted.
Those functions take and return single characters. What's missing is functions that operate on strings. You can use them in combination with std::transform(), but as the article points out, even if you're just dealing with ASCII you can easily do it wrong. I've been using C++ for over 20 years and I didn't know tolower() and toupper() were non-addressable. There's really no excuse for the library not having simple ca…
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…
Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant).
7-bit ASCII code points are fine, but outside that it's not guaranteed that one UTF-8 byte converts into exactly one UTF-8 byte when converting case.
In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to…
Game UI is the place I’d expect to most likely come across horrific abuses of localization precisely because game UI is such a cobbled together layer of hacks on hacks.
Earlier quoted context omitted.
> In gamedev there is simple rule: don't try to do any of that. I am not in gamedev, but I frequently have to develop middleware that takes in user entered data and formats it in a way that will import into a 3rd party system without errors. And that sometimes means changing the case on strings. In my experience as a developer, this is very very common requirement. Luckily I am not forced to use a low level language…
The problem is that such third-party requirements are usually wrong . Two decades ago some developer probably went "Yeah, obviously all names start with capital letters!", not realizing that there are in fact plenty of names which start with a lowercase letter. So they added an input validation test which checks for capitals, which meant everyone feeding that system had to format their data. A whole ecosystem grew ar…
Yes we can simply ToUpper(). We just can’t ToUpper().ToLower(), but that’s useless cause we have the original string if we need it and fine if we don’t need it.
The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.
If you're printing a name, you're probably printing the name for the current user, not for the person who entered it at some point. If you're going to try to convert back like that, you also need to store a timestamp with every string in case a language changes its rules (such as permitting ẞ instead of SS when capitalising ß). And even then, someone might intend to use the new spelling rules, or they might not, who knows!
This article probably boils down to "programmers don't realise graphemes aren't characters and characters aren't bytes even though they usually are in US English". The core problem, "text processing looks easy as long as you only look at your own language", is one that doesn't just affect computers.
Your best bet is to just avoid the entire problem by not processing input further than basic input sanitisation, such as removing whitespace prefixes/suffixes and maybe stripping out invalid unicode so it can't be used as a weird stored attack.
Earlier quoted context omitted.
In Rust, the APIs are clear if they're ASCII only or unicode aware. https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ‘Lowercase’ is defined according to the terms of the Unicode Derived Core Property Lowercase. https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ASCII letters ‘A’ to ‘Z’ are mapped to ‘a’ to ‘z’, but non-ASCII letters are unchanged. Now, "perfectly" is very strong. For e…
> 99% of Unicode stuff Does that include context-dependent conversion rules like o'reilly -> O'Reilly?
Earlier quoted context omitted.
Java ended up picking UCS-2 and getting screwed.
Pretty much all Unicode early adopters went for 16-bit chars. Qt and Win32 API are another pair.
By not committing to UCS-2 early C++ left the road open to UTF-8. I'll concede that UTF8 has risen as the clear winner for more than a decade and C++ is well past the point that it should have at least basic builtin support. The problem is that there is at least one important C++ platform that only very recently added full support for the encoding in their native API.
Earlier quoted context omitted.
I mean, I already know some Unicode internals and linguistics (since I developed a language-specific compression algorithm back in the day), but I have never seen a single character requiring four bytes (and I know Emoji chaining for skin color, etc.). So, seeing this just moved the complexity of Unicode one notch up in my head, and I respect the guys who designed and made it work. It was not whining or complaining o…
Cuneiform codepoints are 17 bits long. If you're using UTF-16 you'll need two code units to represent a character.