Can someone explain the issue at hand? I'm not sure I have enough context to understand the problem
An Urgent Notice from AssemblyScript
11–20 of 55 posts
Re: An Urgent Notice from AssemblyScript
#12> On August 3rd, the WebAssembly CG will poll on whether JavaScript string semantics/encoding are out of scope of the Interface Types proposal. This decision will likely be backed by Google (C++), Mozilla (Rust) and the Bytecode Alliance (WASI), who appear to have a common interest to exclusively promote C++, Rust respectively non-Web semantics and concepts in WebAssembly. > If the poll passes, which is likely, Assem…
Yes, it seems they want to use UTF-16 strings.
I’m confused why they can’t just switch their (nascent) language to UTF-8, and if so why the alarmist attitude? I didn’t think they were mature enough to claim no breaking changes, for example.
I probably prefer we drag the web (and .Net and Java) platforms towards UTF-8, to be honest… but maybe that’s just me.
Re: An Urgent Notice from AssemblyScript
#13> On August 3rd, the WebAssembly CG will poll on whether JavaScript string semantics/encoding are out of scope of the Interface Types proposal. This decision will likely be backed by Google (C++), Mozilla (Rust) and the Bytecode Alliance (WASI), who appear to have a common interest to exclusively promote C++, Rust respectively non-Web semantics and concepts in WebAssembly. > If the poll passes, which is likely, Assem…
Yes, it seems they want to use UTF-16 strings. I’m confused why they can’t just switch their (nascent) language to UTF-8, and if so why the alarmist attitude? I didn’t think they were mature enough to claim no breaking changes, for example. I probably prefer we drag the web (and .Net and Java) platforms towards UTF-8, to be honest… but maybe that’s just me.
What should Blazor and TeaVM do when existing code allows for isolate pairs? If they perform implicit conversions to utf-8 they have the option to either trap, or perform lossy conversion which has immense security and data integrity implications.
Re: An Urgent Notice from AssemblyScript
#14This seems to be the discussion thread related to this.
Re: An Urgent Notice from AssemblyScript
#15UTF-16 was always a mistake[1]. Good riddance. Time to get it out of LSP specification[2] as well. [1] http://utf8everywhere.org/ [2] https://github.com/microsoft/language-server-protocol/issues...
So basically even in UTF8 you can create malformed string. For example ðŸŒ. It miss one byte and may cause to problems in some editors / text viewers which doesn't handle or pre-verify such cases . Valid UTF8 has a specific binary format. If it's a single byte UTF8 character, then it is always of form '0xxxxxxx', where 'x' is any binary digit. If it's a two byte UTF8 character, then it's always of form '110xxxxx10xxxxxx'. Similarly for three and four byte UTF8 characters it starts with '1110xxxx' and '11110xxx' followed by '10xxxxxx' one less times as there are bytes.
So it's not just UTF16 that has problems and can cause security problems. I just wanted to emphasize that
Re: An Urgent Notice from AssemblyScript
#16This is an unfortunate consequence of the poor choice of keeping UCS-2 alive as UTF-16 for way too long. The plug in 16 bit encodings should have been pulled a long time ago, but some people were and still are so focused on backwards compatibility that they didn't see they were just pushing the issue to another decade. UTF-8 has won, completely. UTF-16 is basically a zombie nobody wants anymore, kept artificially alive by the fear of big 90s frameworks of clean breaks with the past.
We must get rid of legacy encodings no matter the cost, I'm tired of seeing Java and Qt apps wasting millions of CPU cycles mindlessly converting stuff back and forth from UTF-16. It's plain madness, and sometimes you just need the courage to destroy everything and start again.
Re: An Urgent Notice from AssemblyScript
#17This is an unfortunate consequence of the poor choice of keeping UCS-2 alive as UTF-16 for way too long. The plug in 16 bit encodings should have been pulled a long time ago, but some people were and still are so focused on backwards compatibility that they didn't see they were just pushing the issue to another decade. UTF-8 has won, completely. UTF-16 is basically a zombie nobody wants anymore, kept artificially ali…
I love reading stuff like this, because it reminds me that there are two entire universes of IT, and both are mostly filled with people blissfully unaware of the other.
UTF-8 is a great hack that works wonderfully on Linux and BSD, because neither actually supported internationalisation properly until recently. They clung to 8-bit ASCII with white knuckles until they could bear it no longer, but then UTF-8 came to the rescue and there was much rejoicing. "It's the inevitable future!" cried millions of Linux devs... in English. I mention this because UTF-8 is a bit... shit... if you're from Asia.
Meanwhile, in the other universe, UCS-2 or UTF-16 have been around for forever because in that Universe people do things for money and had to take internationalisation seriously. Not just recently, but decades ago. Before some Linux developers were born. In this Universe, an ungodly amount of Real Important Code was written by Big Business and Big Government. The type of code that processes trillions of dollars, not the type used to call MySQL unreliably from some Python ML bullshit running in a container or whatever the kids are doing these days.
So, yes. Clearly UTF-16 has to "die" because it's inconvenient for C developers that never figured out how to deal with strings based on more than encoding.
PS: There are several Unicode compression formats that blow UTF-8 out of the water if used in the right way. If you can support those, then you can support UTF-16. If you can't, then you can't claim that you chose UTF-8 because you care about performance.
Re: An Urgent Notice from AssemblyScript
#18This is an unfortunate consequence of the poor choice of keeping UCS-2 alive as UTF-16 for way too long. The plug in 16 bit encodings should have been pulled a long time ago, but some people were and still are so focused on backwards compatibility that they didn't see they were just pushing the issue to another decade. UTF-8 has won, completely. UTF-16 is basically a zombie nobody wants anymore, kept artificially ali…
Are you sure UTF-8 is the ideal format? After all, we have grapheme clusters that cannot be rendered as text units using UTF-8. Maybe UTF-8 is already obsolete and never took over the world? I am more than sure that soon we will see the new Unicode format ;)
Re: An Urgent Notice from AssemblyScript
#19UTF-16 was always a mistake[1]. Good riddance. Time to get it out of LSP specification[2] as well. [1] http://utf8everywhere.org/ [2] https://github.com/microsoft/language-server-protocol/issues...
So basically even in UTF8 you can create malformed string. For example ðŸŒ. It miss one byte and may cause to problems in some editors / text viewers which doesn't handle or pre-verify such cases . Valid UTF8 has a specific binary format. If it's a single byte UTF8 character, then it is always of form '0xxxxxxx', where 'x' is any binary digit. If it's a two byte UTF8 character, then it's always of form '110xxxxx10xxx…
The point is that UTF-16 is the worst of both worlds. It's not ASCII compatible like UTF-8, but it still has the disadvantaged of being a variable length encoding.
Every problem that UTF-8, it shares with UTF-16. It also shares every problem with UTF-32.
Re: An Urgent Notice from AssemblyScript
#20This seems to be the discussion thread related to this. https://github.com/WebAssembly/interface-types/issues/13
The tone of this discussion makes it seem like there is long-running interpersonal conflicts between discussants. Anyone know the context here?