Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicorn: C++ Unicode string library

github.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Unicorn: C++ Unicode string library

#21

Post by jstimpfle »

rspeer wrote: post #19
fantasticfears wrote: post #12

Earlier quoted context omitted.

It's hard but needed to differentiate between UTF-16 and UChar byte array. UChar byte array are not essentially an well-formed UTF-16 string. Beyond, why bother use UnicodeString? It's fairly easy to use. It covers the detail from your sight. It's indeed super cool to see a modern Unicode C++ library. But anyway, is it really useful for production usage? The answer could be no. In contrast, ICU was old, battle-tested…

I'm talking about using UTF-8 as the string representation, not UChars. UChars are an artifact of UTF-16, and thus require converting all text on input and output, unless you work in a Windows API world where I/O is UTF-16. Modern programming languages such as Rust gain efficiency by working with unmodified UTF-8. All you lose is constant-time arbitrary indexing, which is a bad idea in most cases anyway.

Why is a bad idea? Because Unicode has too complicated semantics to split a Unicode string at arbitrary points?

Re: Unicorn: C++ Unicode string library

#22

Post by skystrife »

rspeer wrote: post #19
fantasticfears wrote: post #12

Earlier quoted context omitted.

It's hard but needed to differentiate between UTF-16 and UChar byte array. UChar byte array are not essentially an well-formed UTF-16 string. Beyond, why bother use UnicodeString? It's fairly easy to use. It covers the detail from your sight. It's indeed super cool to see a modern Unicode C++ library. But anyway, is it really useful for production usage? The answer could be no. In contrast, ICU was old, battle-tested…

I'm talking about using UTF-8 as the string representation, not UChars. UChars are an artifact of UTF-16, and thus require converting all text on input and output, unless you work in a Windows API world where I/O is UTF-16. Modern programming languages such as Rust gain efficiency by working with unmodified UTF-8. All you lose is constant-time arbitrary indexing, which is a bad idea in most cases anyway.

It's not super convenient, but one can operate on UTF-8 buffers with ICU via UText (see e.g. http://userguide.icu-project.org/strings/utext#TOC-Example:-...)

Not everything is doable this way, but quite a lot actually is.

Re: Unicorn: C++ Unicode string library

#23

Post by cmrdporcupine »

captaincrowbar wrote: post #8
cmrdporcupine wrote: post #5

The unicode portion looks reasonable, but why is it necessary for it to include its own flags, file io, file management, and environment classes? Why is it so many C++ libraries fall into this habit of trying to build one big framework. I'm perfectly happy with gflags -- a unicode library would be nice for my project, but now I won't consider this library.

Because the whole point is to handle anything that needs Unicode support. A library that only manipulated Unicode strings would be incomplete if you still couldn't use Unicode in command line options, file names, etc.

I would recommend breaking them off into separate additional libraries. I don't need unicode for flags, so paying for it at compile and link time seems unwise. Or provide adapter classes that can be used over other frameworks. Just a suggestion.

Re: Unicorn: C++ Unicode string library

#24

Post by imron »

jstimpfle wrote: post #21
rspeer wrote: post #19

Earlier quoted context omitted.

I'm talking about using UTF-8 as the string representation, not UChars. UChars are an artifact of UTF-16, and thus require converting all text on input and output, unless you work in a Windows API world where I/O is UTF-16. Modern programming languages such as Rust gain efficiency by working with unmodified UTF-8. All you lose is constant-time arbitrary indexing, which is a bad idea in most cases anyway.

Why is a bad idea? Because Unicode has too complicated semantics to split a Unicode string at arbitrary points?

Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element.

This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP).

Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data.

Edit: It's not too difficult to handle these cases and make sure you only split at valid positions, but you do need to be careful and there are a number of edge cases you might not think through or even encounter unless you have the right sort of data to test with - which leads to lots of faulty implementations. e.g. for years MySQL couldn't handle utf8 characters outside the BMP.

Re: Unicorn: C++ Unicode string library

#25

Post by [deleted] »

imron wrote: post #24
jstimpfle wrote: post #21

Earlier quoted context omitted.

Why is a bad idea? Because Unicode has too complicated semantics to split a Unicode string at arbitrary points?

Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element. This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP). Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data. Edit: It's not…

[deleted]

Re: Unicorn: C++ Unicode string library

#26

Post by jstimpfle »

imron wrote: post #24
jstimpfle wrote: post #21

Earlier quoted context omitted.

Why is a bad idea? Because Unicode has too complicated semantics to split a Unicode string at arbitrary points?

Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element. This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP). Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data. Edit: It's not…

My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level.

I do know that Unicode has combining code points (confusingly called combining characters) and nasty things like rtl switching code points. I guess it's turtles all the way down.

Re: Unicorn: C++ Unicode string library

#27

Post by vardump »

jstimpfle wrote: post #26
imron wrote: post #24

Earlier quoted context omitted.

Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element. This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP). Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data. Edit: It's not…

My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. I do know that Unicode has combining code points (confusingly called combining characters) and nasty things like rtl switching code points. I guess it's turtles all the way down.

> My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level.

You need UTF-32 for (random) indexing of code points. UTF-16 has 16-bit code units. Some UTF-16 code points are 32-bits, using a surrogate pair.

So it's the same trade-off as with UTF-8. Thus no reason not to just simply use UTF-8 in the first place and take advantage of the memory savings.

Re: Unicorn: C++ Unicode string library

#28

Post by jstimpfle »

vardump wrote: post #27
jstimpfle wrote: post #26

Earlier quoted context omitted.

My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. I do know that Unicode has combining code points (confusingly called combining characters) and nasty things like rtl switching code points. I guess it's turtles all the way down.

> My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. You need UTF-32 for (random) indexing of code points. UTF-16 has 16-bit code units . Some UTF-16 code points are 32-bits, using a surrogate pair. So it's the same trade-off as with UTF-8. Thus no reason not to just simply use UTF-8 in the first place and take advantage of the memory savings.

Again, my original parent's statement was not about encoding or memory savings. The statement was that it was a bad idea to index into an (abstract) unicode string (of unicode code points -- not compositions thereof whatsoever).

I didn't question that, but hoped to get some inspiration for sane usage of unicode handling (which I'm not sure is humanly possible except for treating it as a rather black box and make no promises).

Re: Unicorn: C++ Unicode string library

#29

Post by imron »

jstimpfle wrote: post #28
vardump wrote: post #27

Earlier quoted context omitted.

> My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. You need UTF-32 for (random) indexing of code points. UTF-16 has 16-bit code units . Some UTF-16 code points are 32-bits, using a surrogate pair. So it's the same trade-off as with UTF-8. Thus no reason not to just simply use UTF-8 in the first place and take advantage of the memory savings.

Again, my original parent's statement was not about encoding or memory savings. The statement was that it was a bad idea to index into an (abstract) unicode string (of unicode code points -- not compositions thereof whatsoever). I didn't question that, but hoped to get some inspiration for sane usage of unicode handling (which I'm not sure is humanly possible except for treating it as a rather black box and make no p…

Your original parent was all about encodings, and mentioned it was a bad idea to arbitrarily index in to utf8 strings, (no mention of abstract strings of unicode codepoints).

> languages such as Rust gain efficiency by working with unmodified UTF-8. All you lose is constant-time arbitrary indexing

So it's saying Rust mostly benefits from using utf8, but in doing so, it loses the ability to arbitrarily index a character in a string (in constant time).

If it was abstract strings of unicode codepoints then there is no problem - except you'd then be using 32bits per codepoint.

Re: Unicorn: C++ Unicode string library

#30

Post by imron »

jstimpfle wrote: post #26
imron wrote: post #24

Earlier quoted context omitted.

Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element. This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP). Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data. Edit: It's not…

My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. I do know that Unicode has combining code points (confusingly called combining characters) and nasty things like rtl switching code points. I guess it's turtles all the way down.

Actually, they are not combining code points. Take for example the character 𪚥 (4 dragons).

The codepoint is U+2A6A5, but in UTF16 it requires combining 2 utf16 characters (\uD869 and \uDEA5) in order to reference it.

The codepoint however is still exactly the same (U+2A6A5).

Post reply on HN