UTF-8 Encoding Debugging Chart
i18nqa.com
UTF-8 Encoding Debugging Chart
1–10 of 12 posts
Re: UTF-8 Encoding Debugging Chart
#2Lol, biggest bug is developer ignoring that latin1 & unicode encoded in UTF8 can coexists in the same stream of data :
- HTTP 1.1 headers are ISO-8859-1 (CERN legacy) while content can be UTF8 - SIP being based on HTTP RFC have the same flaw.
The CTO of my last VoIP company is still wondering why some callerIDs are breaking his nice python program assuming everything is UTF8 and still does not understand this...
Yes, encoding can change, I also saw it while using regionalisation with C# .net in logs.
Re: UTF-8 Encoding Debugging Chart
#3Lol, biggest bug is developer ignoring that latin1 & unicode encoded in UTF8 can coexists in the same stream of data : - HTTP 1.1 headers are ISO-8859-1 (CERN legacy) while content can be UTF8 - SIP being based on HTTP RFC have the same flaw. The CTO of my last VoIP company is still wondering why some callerIDs are breaking his nice python program assuming everything is UTF8 and still does not understand this... Yes,…
According to newer HTTP specs clients should ignore weird ISO-8859 characters. https://tools.ietf.org/html/rfc7230#section-3.2.4:
Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.
Re: UTF-8 Encoding Debugging Chart
#4Lol, biggest bug is developer ignoring that latin1 & unicode encoded in UTF8 can coexists in the same stream of data : - HTTP 1.1 headers are ISO-8859-1 (CERN legacy) while content can be UTF8 - SIP being based on HTTP RFC have the same flaw. The CTO of my last VoIP company is still wondering why some callerIDs are breaking his nice python program assuming everything is UTF8 and still does not understand this... Yes,…
According to newer HTTP specs clients should ignore weird ISO-8859 characters. https://tools.ietf.org/html/rfc7230#section-3.2.4 : Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOU…
IETF should just publish an RFC that says "all text without a field specifying its encoding shall be UTF-8, even if this conflicts with a previous RFC." The only real objection to doing this is that it would break things, but almost all of those things are already broken.
Re: UTF-8 Encoding Debugging Chart
#5I previously wrote about this common double encoding issue at http://www.pixelbeat.org/docs/unicode_utils/ which references tools and techniques to fix up such garbled data
Re: UTF-8 Encoding Debugging Chart
#6The fortunate thing is, almost all of the broken sequences are unambiguous enough to be signs that the text should be encoded and then re-decoded as UTF-8. (This is not the case with any arbitrary encoding mixup -- if you mix up Big5 with EUC-JP, you might as well throw out your text and start over -- but it works for UTF-8 and the most common other encodings because UTF-8 is well-designed.)
So if you want a Python library that can do this automatically with an extremely low rate of false positives: https://github.com/LuminosoInsight/python-ftfy
Re: UTF-8 Encoding Debugging Chart
#7Earlier quoted context omitted.
According to newer HTTP specs clients should ignore weird ISO-8859 characters. https://tools.ietf.org/html/rfc7230#section-3.2.4 : Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOU…
IETF should just publish an RFC that says "all text without a field specifying its encoding shall be UTF-8, even if this conflicts with a previous RFC." The only real objection to doing this is that it would break things, but almost all of those things are already broken.
The problem is browser vendors wouldn't want to implement that spec because if updating your browser breaks the website, no matter how much you explain it to the user, it's your fault, not the website owner's. It's why we have Quirks Mode even after 15 years. It's why Linus is so adamant about patches breaking userspace;[1] if your update broke it, it's your fault, no matter how bad the truly broken thing is.
Re: UTF-8 Encoding Debugging Chart
#8Missing: how defaults are wrong between UTF8 and EBCDIC. E.g. where a character in UTF8 outside the MES2 subset ('latin1') will be mapped to the x3F 'unknown' character of EBCDIC, which will be mapped back to the x1A character ('CTRL-z') of UTF8...
Re: UTF-8 Encoding Debugging Chart
#9Lol, biggest bug is developer ignoring that latin1 & unicode encoded in UTF8 can coexists in the same stream of data : - HTTP 1.1 headers are ISO-8859-1 (CERN legacy) while content can be UTF8 - SIP being based on HTTP RFC have the same flaw. The CTO of my last VoIP company is still wondering why some callerIDs are breaking his nice python program assuming everything is UTF8 and still does not understand this... Yes,…
According to newer HTTP specs clients should ignore weird ISO-8859 characters. https://tools.ietf.org/html/rfc7230#section-3.2.4 : Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOU…
Well there are still situations where coders put mix different codeset / encoding.
Willingly or not.
And SIP still exists.
Re: UTF-8 Encoding Debugging Chart
#10Currently down. Here’s a snapshot from January: