Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

domm.plix.at

Post reply on HN
Print view View original

51–60 of 66 posts

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#51

Post by panzi »

magicalhippo wrote: post #22
heinrich5991 wrote: post #14

Earlier quoted context omitted.

No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.

I was AFK so couldn't check, but yeah you're right. Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD. So yeah, don't mess with the normalization of filenames.

They're both valid UTF-16, though. Can you create a filename with only half of a surrogate pair in it?

I don't use Windows, so I can't check. Linux literally allows any arbitrary byte except for 0x00 and 0x2F ('/' in ASCII/UTF-8). It's a problem for programming languages that want to only use valid Unicode strings, like Python. Rust has a separate type "OsString" to handle that, with either lossy conversion to "String" or a conversion method that can fail. Python uses the custom use Unicode range to represent invalid byte sequences in filenames. It's all a mess. JavaScript doesn't give a damn about the validity of their UTF-16 strings.

(Note that Rust's OsString is different from it's CString type. Well, I guess under Unix they're the same, but under Windows OsString is UTF-16 (or "WTF-16", because it isn't actually valid UTF-16 in all cases).)

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#52

Post by mort96 »

lisper wrote: post #49
mort96 wrote: post #47

Earlier quoted context omitted.

Reading a discussion thread where each message is just a link to some pastebin with the actual message isn't very nice. Besides, I wasn't going to write the message again after HN removed arbitrary parts of it, hence the edit; I think people got the gist. You may feel that discussion about Unicode doesn't belong on HN but I feel otherwise.

Reading discussion threads full of silly emojis isn't "very nice" either, at least for a certain kind of audience. It's a tradeoff, and the powers that be at HN have decided to optimize for sober discussion over expressivity. It's a defensible decision. Keeping HN from degenerating into Reddit is already hard enough.

I'm sure you'd have survived my unsensored message.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#53

Post by mort96 »

AlienRobot wrote: post #50
mort96 wrote: post #19

Earlier quoted context omitted.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag . EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the…

More sites should have this bug. Whenever I see a colored icon in the middle of black text I get inexplicably angry.

Might I suggest therapy?

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#54

Post by tingletech »

quink wrote: post #37
tingletech wrote: post #31

MARC can do all kinds of crazy things. I used to work with folks who had been hacking on MARC since the 1960s. If I remember correctly, at one point it got punched onto dangling chad cards (and of course was used to print the cards in the card catalog in the library). > The real problem is that USMARC uses an int with 4 digits to store the size of a field, followed by 5 digits for the offset. A colleague told me they…

Well, until some system comes along that relies on the directory for the tags only and just splits the record using the separator characters. Which is a valid enough approach to either work around bad encoding or if your record is on something other than a magnetic band and you don't need to know what exact offset to move to. Hidden is a very relative term there.

I never really worked with MARC much (except for a script for generating patron records once a quarter to load new students and staff into III and somehow we marked obsolete users to change their status) but I used to work at the successor organization to the University of California Division of Library Automation (nee University Library Automation Program), and one of the folks telling MARC war stories was describing doing this with with a tool he created specifically for creating pathological MARC records. They aggregated records from local systems into the systemwide "Melvyl" (during ULAP they produced microfiche binders of the union catalog) -- I don't know that they ever redistributed the MARC to other display systems.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#55

Post by lisper »

mort96 wrote: post #52
lisper wrote: post #49

Earlier quoted context omitted.

Reading discussion threads full of silly emojis isn't "very nice" either, at least for a certain kind of audience. It's a tradeoff, and the powers that be at HN have decided to optimize for sober discussion over expressivity. It's a defensible decision. Keeping HN from degenerating into Reddit is already hard enough.

I'm sure you'd have survived my unsensored message.

Of course. It's everyone else's emojis that would get annoying.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#56

Post by tingletech »

quink wrote: post #39
tingletech wrote: post #30

Earlier quoted context omitted.

I do wish the community had leaned into the MODS direction vs going over to RDF.

Yeah I don't know what the sell is there... throw away the fidelity of your data when WEMI/FRBR/BIBFRAME/semantic web is coming any ~year~ decade now, soon (lol), while re-learning everything, definitely going out to tender because your current system won't do it and shift your processes and integrations. All so you can end up halfway to DC, yeah no. The reason libraries don't do cataloguing any longer anywhere near…

The fidelity seems pretty good, at least if you convert to MARCXML and then use the XSLT from Library of Congress to generate it. IIRC it has record types for all the FRBR levels. It is also not flat like DC. It was a joy to work with from a record aggregator perspective, especially if you were generating it from MARC. You can even put the full table of contents into it. At the time that one of my colleagues wrote the "MARC Must Die" article (at least if I remember correctly) the teams working on RDA and MODS had a lot of overlap and MODS was being designed with the era's cataloging theory in mind. There was a moment in time where it seemed like a "new MARC" might go in that direction.

Having catalogers or metadata librarians write directly in MODS XML by hand never made sense (although some folks tried this), but as far as something usable to ship around I'd rather get MODS than MARC or dublin core. I really don't want to have to query a triple store to aggregate records.

Catalogers ideally would have tools that make it easy for them follow RDA/AACR2 descriptive practices without having to think about the details of MARC or MODS or linked data.

I've been out of the business for a couple of years, so I have not been following Library of Congress' BIBFRAME transition.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#57

Post by gmueckl »

panzi wrote: post #48
nordsieck wrote: post #34

Earlier quoted context omitted.

> Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here. Just because the worst case can't improve doesn't mean that making the average case better is worthless.

I saw something about Arabic text, where that naive truncation at codepoint boundaries turns one word into a different word! Like the sequence of codepoints generate something that is represented as a single glyph in fonts, but truncated its totally different glyphs. I don't remember more details, I don't know any Arabic, but grapheme clusters aren't just about adding diacritics to latin characters. In other language…

Randomly truncating words can have the same effect in any language. It's outright trivial to find examples in English or German. I don't understand why one has to invoke Arab script for a good example.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#58

Post by asabil »

gmueckl wrote: post #57
panzi wrote: post #48

Earlier quoted context omitted.

I saw something about Arabic text, where that naive truncation at codepoint boundaries turns one word into a different word! Like the sequence of codepoints generate something that is represented as a single glyph in fonts, but truncated its totally different glyphs. I don't remember more details, I don't know any Arabic, but grapheme clusters aren't just about adding diacritics to latin characters. In other language…

Randomly truncating words can have the same effect in any language. It's outright trivial to find examples in English or German. I don't understand why one has to invoke Arab script for a good example.

Yes, but you don’t end up with different glyphs. Arabic script has letter shaping, that means a letter can have up to 4 shapes based on its position within the word. If you chop off the last letter, the previous one which used to have a “middle” position shape suddenly changes into “terminal” position shape.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#59

Post by simonw »

AlotOfReading wrote: post #41
simonw wrote: post #38

Earlier quoted context omitted.

I pasted your comment here into GPT-4o and asked for the Python equivalent, it suggested this which seems to work well: import regex as re def grapheme_clusters(text): # \X is the regex pattern that matches a grapheme cluster pattern = re.compile(r'\X') return [ match.group(0) for match in pattern.finditer(text) ] https://chatgpt.com/share/481c9c94-0431-4fcb-82aa-a44a4f3c21...

Note that regex is not the re module from the stdlib, it's a separate third party module that exposes the more powerful capabilities of PCRE like grapheme clustering directly.

That's a good callout, here's the docs for \X in that regex module: https://github.com/mrabarnett/mrab-regex?tab=readme-ov-file#...

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#60

Post by magicalhippo »

panzi wrote: post #51
magicalhippo wrote: post #22

Earlier quoted context omitted.

I was AFK so couldn't check, but yeah you're right. Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD. So yeah, don't mess with the normalization of filenames.

They're both valid UTF-16, though. Can you create a filename with only half of a surrogate pair in it? I don't use Windows, so I can't check. Linux literally allows any arbitrary byte except for 0x00 and 0x2F ('/' in ASCII/UTF-8). It's a problem for programming languages that want to only use valid Unicode strings, like Python. Rust has a separate type "OsString" to handle that, with either lossy conversion to "Strin…

Yeah seems to be same with Win32.

I tried using U+13161 EGYPTIAN HIEROGLYPH G029[1], which resulted in a string of length 2 as expected.

Using both chars (code units) and just the first char (code unit) worked equally fine. In Windows Explorer the first one shows the stork as expected, while the second shows that "invalid character" rectangle.

So yeah, treating filenames as nearly-opaque byte sequences is probably the best approach.

[1]: https://en.wiktionary.org/wiki/%F0%93%85%A1

Post reply on HN