Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

141–150 of 230 posts

Re: The string type is broken

#141

Post by lelf »

wbond wrote: post #137
lelf wrote: post #128

Earlier quoted context omitted.

Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”. PS: Python 3.3.2 (default, Nov 27 2013, 20:04:48) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 'öo̧'[1:] '̈o̧' And sorry, those new regexes don't even support \X (grapheme matching) Edit: python version

Yes, I did, and you did not provide a single example. You just said "“oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!". Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output. Python 3.3.0 (default, Mar 11 2013, 00:32:12) [GCC 4.7.2] o…

3.3.2. No, it is not. Use 'o\u0308o\u0327'

Re: The string type is broken

#142

Post by al2o3cr »

If anybody hasn't seen it, Glitchr's twitter is a fantastic example of how bizarro things can get with "140 characters".

https://twitter.com/glitchr_

Note: may freak out browsers with a flaky Unicode implementation. For instance, scrolling that stream on the iOS Twitter client can get very laggy.

Re: The string type is broken

#143

Post by mbq »

There is a one more issue -- the easier it is to manipulate strings in some language the greater chance that they will be used as an internal data structure for things that certainly aren't texts. And this almost always causes substantial performance loss and awful bugs that are either untraceable due to a dependence on subtle configuration details or form security holes. Or both.

Re: The string type is broken

#144

Post by pfortuny »

tomp wrote: post #13

The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise. Some examples: (1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that…

The funny thing is that, according to "the rules" (the Real Academia de la Lengua Española), in Spanish we should be always using \u0130, but of course no one does...

Re: The string type is broken

#145

Post by ademarre »

I think a lot of programmers don't properly understand character encoding simply because their programming languages don't give them the proper treatment. We need more APIs that force developers to acknowledge character encodings, probably in the type system.

Re: The string type is broken

#146

Post by ianburrell »

XorNot wrote: post #10
unwind wrote: post #5

Earlier quoted context omitted.

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters. If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothin…

To compare strings case-insensitively, you want case-folding instead of lowercase or uppercase. Unicode defines case-folding for comparing strings. There are enough complexities with case, like characters that don't have other case or multiple mappings, that it can't be correctly used for comparison.

Re: The string type is broken

#147

Post by glhaynes »

al2o3cr wrote: post #140
jerf wrote: post #65

Earlier quoted context omitted.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine. [1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

"I have a database field limited to 100 "characters"." Well there's your problem right there... "The user sent me a form submission with 150. I need to do something to resolve that." Any software that defines "do something" here as "silently discard 1/3rd of the user input" is software I'm going to throw in the trash. If you must have fixed-length fields, surely telling the user "much characters, wow overflow" is bet…

What if it's a batch ETL process where there is no "user" to tell that it went wrong?

The point that when you're worrying about string length, it's often an indicator of a separate problem is a good one. But some things really do need the ability to measure/truncate strings and not every situation allows just throwing the software in the trash as an option.

Re: The string type is broken

#148

Post by AsymetricCom »

Now, look over here! When I substitute this context with that, ka-pow! now it's an array of characters!

Big deal. I don't understand what the point of this article is when it shows the shortcomings of half a dozen different string implementations in random languages. Yes, if you don't understand the language, then your assumptions about how it works may be wrong. Big surprise, that doesn't mean every string implementation needs to conform to your expectations...

Re: The string type is broken

#149

Post by jerf »

al2o3cr wrote: post #140
jerf wrote: post #65

Earlier quoted context omitted.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine. [1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

"I have a database field limited to 100 "characters"." Well there's your problem right there... "The user sent me a form submission with 150. I need to do something to resolve that." Any software that defines "do something" here as "silently discard 1/3rd of the user input" is software I'm going to throw in the trash. If you must have fixed-length fields, surely telling the user "much characters, wow overflow" is bet…

Since this seems to be confusing people, I'm providing a small hypothetical example here.

"Any software that defines "do something" here as "silently discard 1/3rd of the user input" is software I'm going to throw in the trash."

You are reading far more in than I put in. I merely said somehow you need to resolve this; you put a particular resolution in my mouth, then attacked.

I did choose the web for one reason, which is that you can't avoid this case; you can try to limit the UI to generating 100 characters only (and I still haven't defined "characters"...), but it's 15 seconds for a user to pull open Firebug and smash 150 characters into your form submission anyhow. Somehow, you better resolve this, and as quickly as you mounted the high horse when faced with the prospect of mere truncation, throwing the entire request out for that will cause somebody else to mount an equally high horse....

Post reply on HN