UTF-8 Everywhere
utf8everywhere.org
UTF-8 Everywhere
1–10 of 289 posts
Re: UTF-8 Everywhere
#2Even Microsoft is finally giving up UTF-16!
They recommend now to use the UTF-8 "code page" in new code.
Re: UTF-8 Everywhere
#3Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.
Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.
Re: UTF-8 Everywhere
#4However, sometimes you're in a layer when ASCII was fine and you should just be explicit about that.
Server Name Indication (in RFC 3546) is flawed in several ways, it's a classic unused extension point for example because it has an entire field for what type of server name you mean, with only a single value for that field ever defined. But one that stands out is it uses UTF-8 encoding rather than insisting on ASCII for the server name.
You can see the reasoning - international domain names are a big deal, we should embrace Unicode. But IDNA already needed to handle all this work, the DNS A-labels are already ASCII even for IDNs.
Essentially choosing UTF-8 here only made things needlessly more complicated in a critical security component. Users, the people who IDNs were for, don't know what SNI is, and don't care how it's encoded.
Re: UTF-8 Everywhere
#5Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.
Have they fixed all the bugs with that pseudocodepage?
Re: UTF-8 Everywhere
#6Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.
Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about?
Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.
Re: UTF-8 Everywhere
#7Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.
Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.
The closest I could find to a recommendation for UTF-8 is in UWP design guidelines: https://docs.microsoft.com/en-us/windows/uwp/design/globaliz...
However it's not quite unequivocal. Windows still uses UTF-16 in the kernel (or actually an array of 16bit integers, but UTF-16 is a very strong convention). The code page will often allow the Win32 API to perform the conversion back and forth instead of your application doing it.
Re: UTF-8 Everywhere
#8Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames.
Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.
Re: UTF-8 Everywhere
#9Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.
Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.
I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.