Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

201–210 of 230 posts

Re: The string type is broken

#201

Post by jodrellblank »

mercurial wrote: post #67
this_user wrote: post #35

The string type isn't broken. If anything these "X is broken" posts are broken. Taking one special case, finding problems with that case and deducing that the whole concept must therefore be discarded is just silly. Strings work fine for the vast majority of use cases. No technology is free of flaws and engineering decisions are almost always based on weighting the pros and cons and choosing a solution that on balanc…

In my experience, the world is full of software which "work fine for the majority of use cases" until the point where you take the wrong code path and things go south.

Much like human brains, and business processes.

Re: The string type is broken

#202

Post by Pitarou »

riquito wrote: post #47
Pitarou wrote: post #2

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

The ffl ligature passes $ python3 Python 3.3.2+ (default, Oct 9 2013, 14:50:09) [GCC 4.8.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "baffle".upper() 'BAFFLE' Strange that the article claims that no languages passes it. It seems from another post that perl passes it too

Embarrassing. I should have checked for myself.

And it is strange. Maybe the author needs to check his locale settings?

Re: The string type is broken

#203

Post by berdario »

pitterpatter wrote: post #183
berdario wrote: post #164

Earlier quoted context omitted.

Python Ruby >1.8 lets you choose the encoding .NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on) Java UCS2 Clojure UCS2 Scala UCS2 QT UCS2 Haskell String UCS4 Haskell Data.Text UTF-16 (yes, not a naive UCS-2) Rust UCS4 (last time I checked) Javascript UCS2 Dart UCS2 PHP Unico…

> Rust UCS4 (last time I checked) Rust chars are 32bit Unicode codepoints. But strings themselves are utf-8. That is the string type, ~str, is basically just ~[u8], a vector of bytes and not ~[char]. `.len()` [O(1)] gives you byte length while `.char_len()` [O(n)] gives you the number of codepoints. So strings in rust are just vectors of bytes with the invariant that it's valid utf-8.

Thanks, I didn't know that

Re: The string type is broken

#204

Post by mef »

shioyama wrote: post #4

Happy to see that ruby (2.0 at least) passes all the tests except the "baffle" one. Edit: sadly, it doesn't.

If you're using Rails, the ActiveSupport::Multibyte::Chars library (#mb_chars on any string) passes all these tests except the upcase of baffle:

  irb(main):001:0> RUBY_VERSION
  => "1.9.3"
  irb(main):002:0> Rails.version
  => "3.2.14"
  irb(main):003:0> # example 1
  irb(main):004:0* example1 = "noe\u0308l".mb_chars
  => noël
  irb(main):005:0> example1.reverse
  => lëon
  irb(main):006:0> example1.compose.slice(0,3)
  => noë
  irb(main):007:0> example1.g_length #grapheme_length
  => 4
  irb(main):008:0> example1.compose.length
  => 4
  irb(main):009:0> # example 2
  irb(main):010:0* example2 = "😸😾".mb_chars
  => 😸😾
  irb(main):011:0> example2.length
  => 2
  irb(main):012:0> example2.slice(1,1)
  => 😾
  irb(main):013:0> example2.reverse
  => 😾😸
  irb(main):014:0> # example 3
  irb(main):015:0* example3 = "baffle".mb_chars
  => baffle
  irb(main):016:0> example3.upcase
  => BAfflE
  irb(main):017:0> # example 4
  irb(main):018:0* example4 = "noël".mb_chars
  => noël
  irb(main):019:0> example4 == example1
  => false
  irb(main):020:0> example4 == example1.compose
  => true

Re: The string type is broken

#205

Post by microtherion »

pilif wrote: post #95
fauigerzigerk wrote: post #94

Earlier quoted context omitted.

>In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go. I'm not sure what "internally using unicode" means. Pyhon's internal representation of strings has changed a lot. It hasn't even been stable in Python 3. Now they are apparently using an internal representation that varies depending on the "widest" character stored. The only solution that isn't driving me ins…

In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses in…

Mac OS and Windows use UCS-2

Which parts of Mac OS? You'd have a lot of problems with Emoji support if that were true. To the best of my knowledge, it's UTF-16 everywhere.

Or do you actually mean Mac OS as in Mac OS 9, and not OS X?

Re: The string type is broken

#206

Post by gnaritas »

zokier wrote: post #127
TorKlingberg wrote: post #78

Earlier quoted context omitted.

For that use case it is better to compare case insensitively with "yes" instead of converting the input to lower case first.

How do you do case-insensitive comparison without normalizing the case of the operands?

Most string compare routines in the library offer a case insensitive compare option already, you don't have to normalize it.

Re: The string type is broken

#207

Post by rolux »

TillE wrote: post #162
zokier wrote: post #133

Earlier quoted context omitted.

I feel like I should learn German only so that I would be able to comment on the ß issue every time a Unicode thread pops up. From my uninformed point of view it is not really clear if ß should really be handled as a separate character/grapheme, or just as a ligature in rendering phase and stored as 'ss'. Or even if current-day orthography should be held at such a sacrosanct position that it shouldn't be changed to s…

> or just as a ligature in rendering phase and stored as 'ss'. Probably. > to save significant amount of collective effort I've seen this kind of suggestion a number of times on HN, and I find it highly amusing. When confronted with a difficult challenge in representing the world on a computer, apparently the answer is to instead change the world. OK, but then how are you going to handle hundreds of years of legacy t…

In German, 'ß' is definitely not just a ligature of 'ss'.

Consider 'Masse' (mass) vs. 'Maße' (dimensions).

Uppercasing these words will necessarily produce ambiguity.

It would be equally tempting -- and wrong -- to treat the German characters 'ä', 'ö' and 'ü' as ligatures of 'ae', 'oe' and 'ue'. They're pronounced the same, and the latter forms commonly occur as substitutions in informal writing, but they also occur in proper names, where it would be incorrect to substitute them with the former. However, if you want to sort German strings, 'ä', 'ö' and 'ü' sort as 'ae', 'oe' and 'ue'.

Re: The string type is broken

#208

Post by ceras »

afandian wrote: post #113
antocv wrote: post #89

Earlier quoted context omitted.

Awesome way to exercise the brain.

Interesting. I had no problem reading that (except for 'arsing' which I thought I'd misread). The ability of the brain to pattern-match upside down is amazing.

Varies by person. I'm nearly 100% incapable of reading upside down text.

Re: The string type is broken

#210

Post by tekacs »

wbond wrote: post #151
lelf wrote: post #141

Earlier quoted context omitted.

3.3.2. No, it is not. Use 'o\u0308o\u0327'

Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true. A little googling turned this up. https://mail.python.org/pipermail/python-ideas/2013-July/021...

TL;DR of parent comment here for those skimming:

  x = 'o\u0308o\u0327'
  len(x) == 4
  x == "öo̧"
  x[:2] == "ö"
  x[2:] == "o̧"
  x[0] == x[2] == "o"
Post reply on HN