A popular but wrong way to do Unicode
> wchar_t
A popular but wrong way to convert a string to uppercase or lowercase
201–210 of 272 posts
Re: A popular but wrong way to convert a string to uppercase or lowercase
#202Earlier quoted context omitted.
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…
> The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant). 7-bit ASCII code points are fine, but outside that it's not guara…
Yeah If you're using narrow strings for UTF8 you're making a mistake. wstrings also are not a good representation because of the platform differences, unless you don't care about Windows in which case it's fine but still not great semantically.
In most type definitions you cannot convert UTF8 via simple iteration because the type generally represents a code point and not a character.
You can have a library where UTF8 characters are a native type and code points are a mostly-hidden internal element. But again, that's highly opinionated for C++.
Re: A popular but wrong way to convert a string to uppercase or lowercase
#203Earlier quoted context omitted.
I'm referring to the people who call case conversion in general "a simple text operation". Say you have an std::string and you want to make it lower case. If you assume it contains just ASCII that's a simpler operation than if you assume it contains UTF-8, but C++ doesn't provide a single function that does either of them. A person can rightly complain that the former is a basic functionality that the language should…
For ascii in C++ isn't there std::tolower / std::toupper? If you're not dealing with unsigned char types there isn't a simple case conversion function, but that's for a good reason as the article lays out.
std::tolower / std::toupper are rubbish functions that can't do proper Unicode but still pull in the bloated locale machinery for what should be a simple conditional integer addition if all you care about is ASCII. Both have no valid use case and should be marked [[deprecated]] and erased from all teaching materials.
Re: A popular but wrong way to convert a string to uppercase or lowercase
#204Earlier quoted context omitted.
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…
> The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant). 7-bit ASCII code points are fine, but outside that it's not guara…
It's not guaranteed for 7-bit ASCII either because tolower/toupper are locale-dependent and with the tr_TR lowercase I (U+0049) is ı (U+0131, aka dotless i) wich encodes as two bytes in UTF-8.
Re: A popular but wrong way to convert a string to uppercase or lowercase
#205Earlier quoted context omitted.
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…
I know I can simply iterate. The point is that it's a function that should be included, not that it's impossible without it. It's one of the most common string operations.
To me that feels like the JS community asking for left-pad or is-even in a module. Why have a dedicated function for 2 lines of code?
And it's a huge footgun. There is no ascii type in C++. People will use the generalized tolower for UTF8 encoded in narrow strings and have issues.
You could say the generalized tolower should support all the different width/encoding combinations and sort it out. But that's still highly opinionated as far as performance is concerned.
Generalized string conversion is a very complex problem and you really cannot simplify it in a way that will satisfy most C++ users. Just use ICU or utf8cpp if you want to do string operations and don't care what's going on under the hood. But even then I can't recommend just 1 library, because no perfect 3rd party library exists. A perfect first party library definitely could not exist.
Re: A popular but wrong way to convert a string to uppercase or lowercase
#206Earlier quoted context omitted.
Word censoring? Ease of use? Console commands (i.e. from Quake to minecraft)?
Those sound exactly like the newcomer detectors GP was referring to. What you want is a case-insensitive string comparison, and outside ASCII that's not equivalent to just turning both strings to lowercase and checking equality (or doing a substring search or whatever the task requires)
Exactly and where you want case-insensitive comparison you almost always also want other kinds of Unicode normalization.
Re: A popular but wrong way to convert a string to uppercase or lowercase
#207Earlier quoted context omitted.
Those functions take and return single characters. What's missing is functions that operate on strings. You can use them in combination with std::transform(), but as the article points out, even if you're just dealing with ASCII you can easily do it wrong. I've been using C++ for over 20 years and I didn't know tolower() and toupper() were non-addressable. There's really no excuse for the library not having simple ca…
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside. The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place. If you are operating on wide stri…
std::u8string, std::u16string and std::u32string are supposed to be the portable unicode string types, but a lot of machinery is missing and some that has been added has since been deprecated.
> there's no way to implement such a module without being very opinionated about something.
indeed! Boost.Nowide[1] is such an opinionated library.
[1] https://www.boost.org/doc/libs/master/libs/nowide/doc/html/i...
Re: A popular but wrong way to convert a string to uppercase or lowercase
#208Earlier quoted context omitted.
For ascii in C++ isn't there std::tolower / std::toupper? If you're not dealing with unsigned char types there isn't a simple case conversion function, but that's for a good reason as the article lays out.
std::tolower / std::toupper are rubbish functions that can't do proper Unicode but still pull in the bloated locale machinery for what should be a simple conditional integer addition if all you care about is ASCII. Both have no valid use case and should be marked [[deprecated]] and erased from all teaching materials.
[deleted]
Re: A popular but wrong way to convert a string to uppercase or lowercase
#209Earlier quoted context omitted.
The problem is that such third-party requirements are usually wrong . Two decades ago some developer probably went "Yeah, obviously all names start with capital letters!", not realizing that there are in fact plenty of names which start with a lowercase letter. So they added an input validation test which checks for capitals, which meant everyone feeding that system had to format their data. A whole ecosystem grew ar…
It’s a lossy operation, but it does work. By this logic jpeg and mpeg don’t work either. But were watching them videos daily. Yes we can simply ToUpper(). We just can’t ToUpper().ToLower(), but that’s useless cause we have the original string if we need it and fine if we don’t need it.
The point is that what ToUpper does depends on locale AND Unicode version. This for many applications it only appears to work until it will fail spectacularly in production.
Re: A popular but wrong way to convert a string to uppercase or lowercase
#210Earlier quoted context omitted.
But that's wrong. The upper case for ß is ẞ.
Isn't uppercase for ß just ß - i.e. it's its own uppercase character?
there shouldn’t be an uppercase version of ß because there is no word in the german language that uses it as the first letter. the german language didnt think of allcaps. please correct me if I am wrong. If written in uppercase it should be converted to SZ or the new uppercase ß…. which my iphone doesn’t have… and
converting anything to uppercase SS isn’t something germany wants …