Fixing a bug with byte order marks
alexwlchan.net
Fixing a bug with byte order marks
1–10 of 11 posts
Re: Fixing a bug with byte order marks
#2Ugh, why are BOMs even still a thing in the 21st century? E.g. when will Windows finally arrive in the late 1990s and switch to UTF-8 for everything?
The UTF-8 BOM is especially bizarre because UTF-8 is completely endian-agnostic (so this "UTF-8 Byte Order Mark" is at most an indicator that this file is UTF-8 encoded, but guess what? Outside the Windows bubble, all text files are UTF-8 anyway).
Re: Fixing a bug with byte order marks
#3The csv.DictReader version of this bug is the one that got me. Open a UTF-8-with-BOM CSV as plain utf-8 and your first column name comes back with an invisible U+FEFF glued to the front of it, so every lookup on that one field fails while the other columns behave perfectly. You end up suspecting the source data, the header row, anything except the encoding.
json.loads has a similar tell: a leading BOM gives you "Expecting value: line 1 column 1 (char 0)", which reads like malformed JSON when the JSON is fine.
Same fix, utf-8-sig at the open, and I think the lesson generalises past subtitles: deal with encoding once at the I/O boundary so nothing downstream ever needs to know a BOM existed. Mid-file BOMs like yours are what it looks like when that leaks.
Re: Fixing a bug with byte order marks
#4I hate the BOM so much. It causes so many subtle issues and I don't even understand why it's needed.
Re: Fixing a bug with byte order marks
#5> A byte order mark is a special use of the zero width no-break space character U+FEFF at the beginning of a text file
Isnt BOM allowed to appear anywhere in the file, because of file concatenation?
Re: Fixing a bug with byte order marks
#6> A byte order mark is a special use of the zero width no-break space character U+FEFF at the beginning of a text file Isnt BOM allowed to appear anywhere in the file, because of file concatenation?
No, it's only allowed at the start. How would that even work? If an app sees a UTF-32 BOM halfway through a file should it interpret the rest of the file as UTF-32? Are there any apps that handle text files like this?
Re: Fixing a bug with byte order marks
#7> A byte order mark is a special use of the zero width no-break space character U+FEFF at the beginning of a text file Isnt BOM allowed to appear anywhere in the file, because of file concatenation?
No, a ZWNBSP that’s not at the start is just a regular ZWNBSP. Can be used to break up ligatures for example
Re: Fixing a bug with byte order marks
#8Ugh, why are BOMs even still a thing in the 21st century? E.g. when will Windows finally arrive in the late 1990s and switch to UTF-8 for everything? The UTF-8 BOM is especially bizarre because UTF-8 is completely endian-agnostic (so this "UTF-8 Byte Order Mark" is at most an indicator that this file is UTF-8 encoded, but guess what? Outside the Windows bubble, all text files are UTF-8 anyway).
Windows bought into Unicode hard before Unicode expanded beyond 16 bits. The consequence is that its native strings are theoretically UTF-16, but often behave like UCS-2, allowing random halves of surrogate pairs. Windows APIs reflect that, Windows filenames reflect that, and translation into UTF-8 has to reflect that -- somehow. It's not great.
Personally I'd like a world where Unicode gave up all the compromises that come with supporting UTF-16. But presumably burning all your early adopters is not a winning strategy.
Re: Fixing a bug with byte order marks
#9Ugh, why are BOMs even still a thing in the 21st century? E.g. when will Windows finally arrive in the late 1990s and switch to UTF-8 for everything? The UTF-8 BOM is especially bizarre because UTF-8 is completely endian-agnostic (so this "UTF-8 Byte Order Mark" is at most an indicator that this file is UTF-8 encoded, but guess what? Outside the Windows bubble, all text files are UTF-8 anyway).
Windows bought into Unicode hard before Unicode expanded beyond 16 bits. The consequence is that its native strings are theoretically UTF-16, but often behave like UCS-2, allowing random halves of surrogate pairs. Windows APIs reflect that, Windows filenames reflect that, and translation into UTF-8 has to reflect that -- somehow. It's not great. Personally I'd like a world where Unicode gave up all the compromises th…
> But presumably burning all your early adopters is not a winning strategy.
This topic is so old that most early adaptors probably have retired by now ;)
Re: Fixing a bug with byte order marks
#10Earlier quoted context omitted.
Windows bought into Unicode hard before Unicode expanded beyond 16 bits. The consequence is that its native strings are theoretically UTF-16, but often behave like UCS-2, allowing random halves of surrogate pairs. Windows APIs reflect that, Windows filenames reflect that, and translation into UTF-8 has to reflect that -- somehow. It's not great. Personally I'd like a world where Unicode gave up all the compromises th…
> But presumably burning all your early adopters is not a winning strategy. This topic is so old that most early adaptors probably have retired by now ;)
Fair. If only they'd retired those APIs, too.
I remember when Windows gained experimental support for a UTF-8 locale usable with its 8-bit APIs. I used to run it at work; it had some issues at first, but most got shaken out pretty quickly. I don't know its current state, but that's probably the best route to UTF-8 on Windows if MS would lean into it.