Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

81–90 of 230 posts

Re: The string type is broken

#81

Post by jeltz »

jmpe wrote: post #30
billpg wrote: post #23

Do people really need to reverse strings in the real world? I don't think I've ever written code to do that outside of homework assignments and interviews.

Had such a case a few months back. Strings of single-byte characters are Endian-agnostic but multi-byte character encoding is affected by Endianness. To cope with it I read the sequence as single byte, then reversed, then changed the encoding to proper encoding and reversed again. The data came from a binary dump where I only needed a section that contained a few strings. I admit it's dirty but it was throwaway code…

For this you do not need to reverse the string in a unicode aware way. You need to operate on the raw bytes.

Re: The string type is broken

#82

Post by gutnor »

this_user wrote: post #35

The string type isn't broken. If anything these "X is broken" posts are broken. Taking one special case, finding problems with that case and deducing that the whole concept must therefore be discarded is just silly. Strings work fine for the vast majority of use cases. No technology is free of flaws and engineering decisions are almost always based on weighting the pros and cons and choosing a solution that on balanc…

Vast majority of use cases in the English-speaking world.

In other countries like China, Japan, India, ... those edge cases are common enough to represent a significant portion of use cases and make X truly broken.

The article is maybe a bit provocative, but you know what, that's exactly what is needed to raise awareness of mainly US-centric developers who would completely ignore the technical issues until they face a clone in China whose only innovative feature is not breaking on Chinese text.

Re: The string type is broken

#83

Post by regularfry »

this_user wrote: post #35

The string type isn't broken. If anything these "X is broken" posts are broken. Taking one special case, finding problems with that case and deducing that the whole concept must therefore be discarded is just silly. Strings work fine for the vast majority of use cases. No technology is free of flaws and engineering decisions are almost always based on weighting the pros and cons and choosing a solution that on balanc…

Conflating the responsibilities of "character list" with "byte array" is always going to go badly.

Re: The string type is broken

#84

Post by jhull »

lelf wrote: post #76
judofyr wrote: post #6

In many languages it's difficult fixing the string type without breaking existing code. In Ruby: String#upcase only handles ASCII (by spec), #length counts codepoints, #reverse reverses codepoints. You can use UnicodeUtils if you need "full" Unicode support: >> UnicodeUtils.upcase("baffle") => "BAFFLE" >> graphemes = UnicodeUtils.each_grapheme("noe\u0308l").to_a >> graphemes.reverse.join => "lëon" >> graphemes.size =>…

> String#upcase only handles ASCII (by spec) Bad for Ruby > You can use UnicodeUtils if you need "full" Unicode support: Oh, sure Betty:~ lelf$ ruby -r unicode_utils/u -e 'puts UnicodeUtils.each_grapheme("A‮͜CB‬D").to_a.reverse.join' D‬BC͜‮A So, "full" (it's not) Unicode support won't help you if you have little idea about what you're doing (like indexing stringه҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈…

what are these characters printing here?

Re: The string type is broken

#85

Post by [deleted] »

jeltz wrote: post #69
jerf wrote: post #65

Earlier quoted context omitted.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine. [1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

Have you checked how your database counts? Does it count code points or does it try to count graphemes? I assume the former, but I guess you would still have to cut the input at a grapheme border when truncating the input.

[deleted]

Re: The string type is broken

#86

Post by ygra »

delinka wrote: post #79

Now let's take the lower case of "BAFFLE" - should we get "baffle" or should the string class/function/wtfe attempt to recognize that a ligature can replace "ffl" and return to us "baffle"? More generally, should the string library ever attempt to replace letter with ligatures? Should this be yet another option? And as I type this, another issue manifests: the spelling correction can't even recognize baffle as a properly…

Uppercasing and lowercasing is inherently lossy. E.g. the German ß becomes SS when uppercased, yet there is no way to know whether SS should be lowercased to ss or ß again. That's a reason why those things should be used, if at all, only as display transformations. Same goes for ligatures, but even those actually shouldn't be applied automatically, depending on the language. E.g. in German ligatures cannot span syllables and few layout engines can detect that.

Re: The string type is broken

#87

Post by preek »

jhull wrote: post #84
lelf wrote: post #76

Earlier quoted context omitted.

> String#upcase only handles ASCII (by spec) Bad for Ruby > You can use UnicodeUtils if you need "full" Unicode support: Oh, sure Betty:~ lelf$ ruby -r unicode_utils/u -e 'puts UnicodeUtils.each_grapheme("A‮͜CB‬D").to_a.reverse.join' D‬BC͜‮A So, "full" (it's not) Unicode support won't help you if you have little idea about what you're doing (like indexing stringه҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈…

what are these characters printing here?

It's an awesome little gadget - looks like one character, but is a really big messy bunch of bytes:

"\xD9\x87\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xC C\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\ xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x8 8\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xC C\xBF\n"

Re: The string type is broken

#88

Post by pilif »

jeltz wrote: post #77
pilif wrote: post #57

A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th posit…

> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…

.NET uses UCS-2 because the Windows API uses UCS-2 (so when you use Visual Studio out of the box, you will get UCS-2). ECMAScript (JS) uses UCS-2 because that's all there was when the spec was written.

Other scripting languages I know for certain are

- PHP doesn't care and treats strings as arrays of bytes. All the str functions operate on these byte arrays and thus happily destroy your strings if they are encoded as anything but the old 8-bit encodings. If you need to support utf-8, you have to use different library functions (mb_*) and a special syntax in their regex support (/u modifier).

- Python - Ruby = 1.9 lets you chose your internal encoding. Most people use utf-8, but you don't have to.

- Perl I don't know enough about, but I hear it as an UTF-8 mode that is actually well-supported by the language itself and gets almost everything right.

These are the more common scripting languages.

Of the compiled languages, I know for certain about Go (utf-8; good library support), C (OS dependent, but the standard string API treats strings as byte arrays), C++ (dito) and Delphi (UCS-2 since 2010, byte arrays before that)

I would say that there are so many exceptions to the UTF-8 rule that I wouldn't say "most" languages are using UTF-8.

Re: The string type is broken

#89

Post by antocv »

edent wrote: post #33

˙ƃuᴉuɐǝɯ ⅋ 'spɹoʍ 'sɥdʎlƃ 'sɹǝʇɔɐɹɐɥɔ uǝǝʍʇǝq 'ɹǝʌǝʍoɥ 'ǝɔuǝɹǝɟɟᴉp ɐ sᴉ ǝɹǝɥ┴ ˙ʇxǝʇ ɥʇᴉʍ punoɹɐ ƃuᴉsɹɐ oʇ sǝɯoɔ ʇᴉ uǝɥʍ sǝᴉʇᴉlᴉqᴉssod ƃuᴉʇsǝɹǝʇuᴉ ǝɯos sɹǝɟɟo ǝpoɔᴉu∩

Awesome way to exercise the brain.

Re: The string type is broken

#90

Post by jheriko »

this hits on one of my biggest problems with native android and ios development. the wcs/wchar functions are largely broken or unusable... it caused me a real headache from not knowing upfront.

the idea of the string type is just fine though (or a character array) broken implementations don't invalidate it, they just invalidate the myth of '3rd party libraries must be good because hundreds of programmers worked on them for years' - which is exactly a myth. it doesn't just apply to strings but everything. (not brokeness, just that you shouldn't expect them to work beyond what you can measure, and certainly shouldn't expect that they are flawless or even good implementations)

Post reply on HN