Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

121–130 of 230 posts

Re: The string type is broken

#121

Post by wbond »

fauigerzigerk wrote: post #109
pilif wrote: post #95

Earlier quoted context omitted.

In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses in…

>In Python 3, you don't care about what they use internally. You don't need to. I do need to know and I always care. My requirements may be different than those of most others because I write text analysis code and I need to optimize the hell out of every single step. I shiver at the thought that any representation could be chosen for me automatically. Of course, nothing is stopping me from simply using the bytes typ…

So in your one, specific, performance-limited situation, Python 3's implementation of unicode doesn't work for you. Mostly because you are trying to optimize based on implementation details.

I don't see how this equates to a general purpose language failing at strings, especially when the language isn't particularly focused on performance and optimization. And if memory usage is of concern, I would certainly think anything like Python and Ruby would be out of the running?

Re: The string type is broken

#122

Post by Dewie »

What do you guys think about String in Haskell, where it is a list of char? Should it have some other default implementation, or should it have been more, um, decoupled from its implementation (don't know the correct terminology)?

Re: The string type is broken

#123

Post by riffraff »

drdaeman wrote: post #114
riffraff wrote: post #80

Earlier quoted context omitted.

doesn't every statically typed imperative language do this, and recommend it?

Not to my knowledge. C++ and Java are statically typed and they, as far as I know, don't have distinction between string interface and implementation, just a standard string type. You can't make your own string implementation and make others (given that - would it exist - they use standard string interface) transparently accept them instead of language's standard string implementation. Even Haskell (with standard Pre…

I was referring to "efforts to separate a type from its implementations", not String specifically, and thinking of containers & co.

Although, for example, even java has CharSequence which only gives you access to codepoints in a char sequence, you can inherit from that and create your own.

Re: The string type is broken

#124

Post by zokier »

jheriko wrote: post #93
zokier wrote: post #27

Earlier quoted context omitted.

I think (2) is an issue with Unicode specifically. They should have specified Turkish alphabet to use ı and a diacritic to make the dotted one. That would have made (in this case) capitalization locale-independent.

isn't this solving the wrong side of the problem? how about not having to think about such things at all and just accepting that uppercase/lowercase conversion is never going to be language agnostic. thats futureproof and powerful, rather than extra thinking and work...

Most likely case-changes need to be locale-aware, that is true. But still I think minimizing number of locale-specifics is a reasonable goal and in that light I dislike the common usage of turkish i as a example because it is such a obviously fixable (if legacy stuff wasn't concern) flaw in Unicode rather than fundamental issue.

Re: The string type is broken

#125

Post by zokier »

panzi wrote: post #107
pilif wrote: post #95

Earlier quoted context omitted.

In Python 3, you don't care about what they use internally. You don't need to. If you want to work with strings, you work with strings. If you want to work with bytes, you work with bytes. If you want to convert bytes into strings (maybe because it's user input that you want to work with), then you tell Python what encoding these bytes are in and you have it create a string for you. You don't care what Python uses in…

Agreed with most of it except: "because their string API is correct" Apparently they have a bug in their UTF-7 parser that can lead to invalid unicode strings. Don't know if it's already fixed.

That would be an implementation flaw, not an API issue.

Re: The string type is broken

#126

Post by adamtj »

rayiner wrote: post #56

This is why the U.S. dominates the software world. Back when everyone was figuring out how to express their languages, we had the option to punt on complexity and just use ASCII.

ASCII doesn't make the U.S. special. ASCII is special because it's from the U.S.

Lots of people speak languages that trivially fit in 8 bits with no real "figuring out" to do. Before Unicode, we all had our different codepages or encodings. Including the U.S.

The U.S. is pretty central to computing. Because of that, and because ASCII only uses 7 bits, some other 8-bit cultures use it as a subset for their native 8-bit encodings. Even in the U.S, we use extensions to ASCII so we can represent text in languages that are close cousins to English. I doubt you actually use ASCII much. You've probably been using either ISO 8859-1 (aka Latin-1), which is a superset of ASCII, or Windows-1252, which is a superset of Latin-1.

http://msdn.microsoft.com/en-us/library/cc194884.aspx

This mess of incompatible codepages and culture specific encodings is one of the main problems that Unicode was invented to solve. It also happens to help languages which need more than 8 bits.

Re: The string type is broken

#127

Post by zokier »

TorKlingberg wrote: post #78
XorNot wrote: post #10

Earlier quoted context omitted.

The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters. If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothin…

For that use case it is better to compare case insensitively with "yes" instead of converting the input to lower case first.

How do you do case-insensitive comparison without normalizing the case of the operands?

Re: The string type is broken

#128

Post by lelf »

wbond wrote: post #115
lelf wrote: post #108

Earlier quoted context omitted.

> Python 3 gets so much of this right What does it gets right????? It's all broken as nearly everything else! It's sad 99% comments there are “oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!” The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code p…

Can you provide some examples of Python 3 getting strings wrong? Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition ( http://docs.python.org/3.3/library/unicodedata.html ) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on http://docs.python.org/3.3/howto/…

Did you read the comment you're replying to at all? You can start at “It's sad 99% comments”.

PS:

  Python 3.3.2 (default, Nov 27 2013, 20:04:48)
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> 'öo̧'[1:]
  '̈o̧'
And sorry, those new regexes don't even support \X (grapheme matching)

Edit: python version

Re: The string type is broken

#129

Post by danbruc »

panzi wrote: post #105
kvb wrote: post #100

Earlier quoted context omitted.

.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.

One unicode character at a time or one unicode codepoint at a time? (see character composition)

The .NET StringInfo class provides methods to work at the grapheme level, not code points.

Re: The string type is broken

#130

Post by zokier »

daGrevis wrote: post #46

Logically equivalent doesn't mean equivalent for computers. While you can't define why reverse of “noël“ is “lëon“ by set of rules that computer can follow, computer just can't know.

Umm. For that case you definitely can define a valid reversing algorithm. The key is using grapheme clusters as the indivisible base unit. Sure, there are probably some weird languages that will not reverse properly with such algorithm, but it would still be a significant improvement over the current situation.
Post reply on HN