Show HN: Mojibake – A low-level Unicode library written in C • Hacker News Board

Re: Show HN: Mojibake – A low-level Unicode library written in C

#11

Post by throwaway2037 » Fri, Jul 17, 2026, 5:45 AM UTC

I assume the submitter is also the author. If so, can you share about your motivation to write this library? For example, do you use it professionally or in hobby projects? Did you look at other libraries and think that you could do better? These are honest questions -- no trolling from me. I browsed the code and it looks very clean.

The API documention is so nice! It looks like index.html from https://mojibake.zaerl.com/ uses JavaScript to generate the page. Very cool, indeed.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#12

Post by program » Fri, Jul 17, 2026, 5:55 AM UTC

throwaway2037 wrote: post #11

I assume the submitter is also the author. If so, can you share about your motivation to write this library? For example, do you use it professionally or in hobby projects? Did you look at other libraries and think that you could do better? These are honest questions -- no trolling from me. I browsed the code and it looks very clean. The API documention is so nice! It looks like index.html from https://mojibake.zaerl…

Yes, I'm the author. I began writing this library because I needed something to handle Unicode for another project of mine (that I didn't finish), and I didn't like ICU4C or the other libraries I found.

It's a hobby project I wrote because I find the Unicode standard (sometimes unnecessary) complexity fascinating. And for other people to use, if needed.

The HTML page is static and generated at compile time. Every function has a little form to test the WASM function.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#13

Post by program » Fri, Jul 17, 2026, 6:02 AM UTC

lifthrasiir wrote: post #4

How does it compare with utf8proc [1]? I'm aware that Mojibake does a bit more than utf8proc (e.g. bi-di) but that seems marginal to me. [1] https://juliastrings.github.io/utf8proc/

Mojibake handle UTF-8, UTF-16, UTF-16LE, UTF-16BE, UTF-32, UTF-32LE, UTF-32BE in input and output. Can be used as a simple file and handle a wider number of algorithms, such as the "confusable" one.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#14

Post by program » Fri, Jul 17, 2026, 6:04 AM UTC

digg99 wrote: post #2

Love the amalgamation approach—the C/C++ ecosystem desperately needs cleaner, lightweight Unicode support without pulling in massive dependencies... thanks for sharing

Thank you very much. I've also started writing a smaller C++ wrapper so the user can use std::string_view for this, but I'm not the best C++ guy out there.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#15

Post by program » Fri, Jul 17, 2026, 6:07 AM UTC

CharlesW wrote: post #7

Not to bikeshed, but isn't the word "mojibake" synonymous with "when character encoding breaks"?

Mojibake is a Japanese term, and yes, it's used there when a page displays a lot of gibberish text. I chose it for fun.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#16

Post by program » Fri, Jul 17, 2026, 6:09 AM UTC

tjwebbnorfolk wrote: post #5

what's performance like compared to python ftfy module?

Mojibake is a C library, not Python, so comparing the two isn't ideal. You can see the performance in the GitHub workflows results. At the end of the logs, there is an "execution time" line.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#17

Post by avadodin » Fri, Jul 17, 2026, 7:18 AM UTC

adrianN wrote: post #6

avadodin wrote: post #3

I have come to the conclusion that the only Unicode support needed in C is supporting pointers to char and arrays but lightweight C libraries are always welcome.

I guess you never have to deal with text if you think that’s enough? What kind of software do you write in C?

Software that treats Unicode(UTF-8) text as a bag of bytes, obviously. Even better if it is in a fixed length box of bytes.

A cursory look at the services provided by this library should dissuade people from attempting to work with Unicode text in C.

You can of course use or build a text processing engine's DSL that does all sorts of things people may want to do with Unicode text but C is hardly the best fit.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#18

Post by program » Fri, Jul 17, 2026, 7:57 AM UTC

avadodin wrote: post #3

I have come to the conclusion that the only Unicode support needed in C is supporting pointers to char and arrays but lightweight C libraries are always welcome.

(Author here) Dealing with encodings is already a big step when you aim to handle multiple ones and multiple OSes. You can see what I am talking about in the tests/ folder.

With Mojibake, I wanted to help people handle text by providing the smallest possible C/C++ library, without requiring them to use a +20MB library just to normalize a string, handle a flag emoji, or perform similar tasks.

See the CONFORMANCE_REQUIREMENTS.md file if you are interested in what the +17 versions of the Unicode standard have introduced.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#19

Post by benj111 » Fri, Jul 17, 2026, 9:14 AM UTC

avadodin wrote: post #3

I have come to the conclusion that the only Unicode support needed in C is supporting pointers to char and arrays but lightweight C libraries are always welcome.

So how do you uppercase or lowercase an arbitrary character?

How do you check if it's a valid character?

How do you deal with combining characters?

Most of the world don't use just the Latin subset in ASCII or follow the same assumptions about letters, words, cases etc etc.

Re: Show HN: Mojibake – A low-level Unicode library written in C

#20

Post by avadodin » Fri, Jul 17, 2026, 9:25 AM UTC

program wrote: post #18

avadodin wrote: post #3

I have come to the conclusion that the only Unicode support needed in C is supporting pointers to char and arrays but lightweight C libraries are always welcome.

(Author here) Dealing with encodings is already a big step when you aim to handle multiple ones and multiple OSes. You can see what I am talking about in the tests/ folder. With Mojibake, I wanted to help people handle text by providing the smallest possible C/C++ library, without requiring them to use a +20MB library just to normalize a string, handle a flag emoji, or perform similar tasks. See the CONFORMANCE_REQUI…

Don't take my comment as dismissive of your project.

The people using it will probably have an easier time navigating Unicode text than they would have if they had used other existing libraries or tried to roll their own.

It's more a comment on the users who need to be warned that "no, you probably don't want your C program to know if that string actually fits in the 80 column terminal".