Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken

mortoray.com

Post reply on HN
Print view View original

61–70 of 230 posts

Re: The string type is broken

#61

Post by ygra »

jeltz wrote: post #52
ygra wrote: post #32

Earlier quoted context omitted.

Substrings exhibit similar problems and those are used quite often. It's just that in this case the effect of seeing it fail is a little more dramatic (i.e., l̈ – which doesn't even seem to render properly here).

"l̈" renders just fine for me, maybe your font does not include it.

Verdana doesn't seem to properly support U+0308, apparently. It's wrong (with that font) in Chrome, IE 10, Firefox and Word 2010. Other operating systems might substitute a different font that works better, perhaps.

Re: The string type is broken

#62

Post by jeltz »

rlpb wrote: post #38
lmm wrote: post #16

I think the mistake here is seeing a string as an extension of an array or vector. What I would prefer is a string type that didn't support all the operations of vectors. The length of a string is not inherently a meaningful question (and for the cases where it is, what you want is something like a vector of grapheme clusters - which is a useful type to have, but not so useful that every string in your program should…

I'm with you here; but in that case, I'd like an ascii_string type, which most languages don't provide specifically. This type _would_ support string reversal, substring slices, and so on, but be limited to 7-bit ASCII only. I think there are many use cases that are purely internal, and don't need i8n. It's handy to be able to do things, including operations on strings, for internal things. Filename handling where yo…

I think this might just confuse new programmers and the filename thing is especially dangerous since at some point you might want to support i18n there. I think it would be better to have two types of string: 1) unicode strings and 2) arrays of 8 byte data with some string like functions (essentially C strings). The second case is essentially binary data strings.

Re: The string type is broken

#63

Post by Pxtl »

jeltz wrote: post #50
abcd_f wrote: post #45

Earlier quoted context omitted.

May be not reversing, but trimming a Unicode string to certain character count is a close relative and it is a very common operation.

What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

Truncating with ellipsis in the GUI in a desktop app. I can measure rendered length on a desktop, so I can truncate down to the desired number of pixels, round down to the nearest char, and then tack on "...". I would hate to see a semantically-important accent mark lost this way.

Re: The string type is broken

#64

Post by ashashwat »

jeltz wrote: post #50
abcd_f wrote: post #45

Earlier quoted context omitted.

May be not reversing, but trimming a Unicode string to certain character count is a close relative and it is a very common operation.

What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

You have a search query and you want to remove stopwords and normalize the query.

Re: The string type is broken

#65

Post by jerf »

jeltz wrote: post #50
abcd_f wrote: post #45

Earlier quoted context omitted.

May be not reversing, but trimming a Unicode string to certain character count is a close relative and it is a very common operation.

What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine.

[1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

Re: The string type is broken

#66

Post by ygra »

monkeyninja wrote: post #48
justincormack wrote: post #41

Earlier quoted context omitted.

Use UTF8, no endian issues. Thats yet another reason why UTF16 and UTF32 are broken.

language will not store unicode string internally with UTF8. Yes, we use it as input and output, but in memory, utf8 is terrible for random access characters. endian is only an issue (normally) for input and output, not really an issue for internal storage. especially when using UTF16 and UTF32 you know exactly the size of items.

UTF-16 is just as bad as UTF-8 regarding variable-width code points. The only thing you always have (unless using compression schemes like SCSU) is random access to code units. Only UTF-32 also allows random access to code points. However, that's still of questionable value because when dealing with text you often want to handle graphemes, not code points, code units or bytes.

Re: The string type is broken

#67

Post by mercurial »

this_user wrote: post #35

The string type isn't broken. If anything these "X is broken" posts are broken. Taking one special case, finding problems with that case and deducing that the whole concept must therefore be discarded is just silly. Strings work fine for the vast majority of use cases. No technology is free of flaws and engineering decisions are almost always based on weighting the pros and cons and choosing a solution that on balanc…

In my experience, the world is full of software which "work fine for the majority of use cases" until the point where you take the wrong code path and things go south.

Re: The string type is broken

#68

Post by jmdavis »

monkeyninja wrote: post #48
justincormack wrote: post #41

Earlier quoted context omitted.

Use UTF8, no endian issues. Thats yet another reason why UTF16 and UTF32 are broken.

language will not store unicode string internally with UTF8. Yes, we use it as input and output, but in memory, utf8 is terrible for random access characters. endian is only an issue (normally) for input and output, not really an issue for internal storage. especially when using UTF16 and UTF32 you know exactly the size of items.

UTF16 is variable length just like UTF8 is (so don't assume 2 bytes == 1 character).

Re: The string type is broken

#69

Post by jeltz »

jerf wrote: post #65
jeltz wrote: post #50

Earlier quoted context omitted.

What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is incredibly common. Truncation to a defined size is routine. [1]: I'm leaving "characters" undefined here, because no matter what Unicode-aware definition you apply here, you've got trouble.

Have you checked how your database counts? Does it count code points or does it try to count graphemes? I assume the former, but I guess you would still have to cut the input at a grapheme border when truncating the input.

Re: The string type is broken

#70

Post by jeltz »

ygra wrote: post #61
jeltz wrote: post #52

Earlier quoted context omitted.

"l̈" renders just fine for me, maybe your font does not include it.

Verdana doesn't seem to properly support U+0308, apparently. It's wrong (with that font) in Chrome, IE 10, Firefox and Word 2010. Other operating systems might substitute a different font that works better, perhaps.

Yes, I am running Debian without having installed the Microsoft core fonts so Verdana is substituted for DejaVu Sans.
Post reply on HN