11–20 of 188 posts
How could we avoid acronyms like 'utf-8'? We can do better than that. Unicode8?
In the article he puts in in terms of std::string, but more generally I think this is what he means.
Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?
As jwz said about backups: "Shut up. I know things. You will listen to me. Do it anyway."
Disagree "UTF-16 is the worst of both worlds—variable length and too wide" Really, the author tries to convince the reader, but it's not that clean cut. One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32) "It's not fix…
What does this mean/prove? I certainly hope you don't intend this to mean "so just pretend in your code that it will always be fixed width". And if that's not what you mean, then I don't know what you gain from that statement.
ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?
I'd love to hear any specific counter-arguments.
Disagree "UTF-16 is the worst of both worlds—variable length and too wide" Really, the author tries to convince the reader, but it's not that clean cut. One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32) "It's not fix…
First of all, if you don't know the encoding, then you don't know the encoding, and you will need to figure out if it's UTF-8, UTF-16, ISO-8859-1, etc. If you happen to know that it's UTF-16, you still need to figure out if it's UTF-16BE or LE.
> "It's not fixed width" But for most text, it is.
This is a dangerous way of thinking. One of the big problems with UTF-16 is that for most text, it is fixed width; so many people make that assumption, and you never notice the problem until someone tries to use an obscure script or an emoji character. This means that bugs can easily be lurking under the surface; while with UTF-8, anything besides straight ASCII will break if you assume fixed width, making it much more obvious.
> Sure, you can do UTF-32 and it may not be a bad idea (today)
UTF-32 isn't really meaningfully fixed width either. Sure, each code point is represented in a fixed number of bytes, but code points are not necessarily the interesting unit you want to index by. A glyph could be composed of several code points. Most of the time, you actually want to deal with text in longer units such as words or tokens, which are going to be variable width anyhow. The actual width of individual code points is only really of interest to low-level text processing libraries, not most applications.
How could we avoid acronyms like 'utf-8'? We can do better than that. Unicode8?
Disagree "UTF-16 is the worst of both worlds—variable length and too wide" Really, the author tries to convince the reader, but it's not that clean cut. One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32) "It's not fix…
still, text for today's computer capabilities is a non issue even if using UTF-32 That obviously depends entirely on what kind of application we're talking about. Keeping large amounts of text data in memory as efficiently as possible is one of my greatest concerns. Many people are processing lots of text nowadays, more than ever before. "It's not fixed width" But for most text, it is. True, so ignoring it means that…
Depends on what you consider efficiency. If it's size, sure, store it using UTF-8. But if you're worried about speed, then UTF-16 or 32 may be the way to go, since you're dealing with data that fits a CPU register. For example, on ARM comparing one byte is much more work than comparing one 32-bit value.
"True, so ignoring it means that your code will be correct ... most of the time."
No, not going to ignore it! But on UTF-16 more code points match the UTF-16 encoding of it (easier for debugging)