Show HN: Mandarin Word Segmenter with Translation • Hacker News Board

Re: Show HN: Mandarin Word Segmenter with Translation

#21

Post by routerl » Sat, Feb 08, 2025, 11:44 AM UTC

greyman wrote: post #18

OP, thank you for your work, I will continue to watch it. I tried to built something similar, but what I didn't discover and think is crucial is the proper FE: yes, word segmenting is useful, but if I have to click on each word to see its meaning, for how I learn Chinese by reading texts, I still find Zhongwen Chrome extension to be more useful, since I see English meaning quicker, just by hover cursor over the word.…

Thanks for the kind words, and the bug report!

The (awful and incorrect) translation you've pointed out comes from the segmenter being too greedy, not finding the (non-existent) word in any dictionary, and therefore dispatching the word to be machine translated, without context. This is the final fallback in the segmentation pipeline, to avoid displaying nothing at all, and my priority right now is making the segmentation pipeline more robust so this rarely (or never) happens, since it sometimes produces hilariously bad results!

Re: Show HN: Mandarin Word Segmenter with Translation

#22

Post by routerl » Sat, Feb 08, 2025, 11:48 AM UTC

rahimnathwani wrote: post #11

This is cool. If you haven't already, you might like to take a look at Du Chinese and The Chairman's Bao. They might provide ideas or inspiration. Also the 'clip reader' feature in Pleco is decent. Also, supporting simplified as well as traditional might increase your potential audience.

It supports traditional and simplified, as well as pinyin and bopomofo :)

It's already possible to switch instantly between pinyin and bopomofo, and I'm working on letting users switch between simplified/traditional, but this is also a non-trivial problem. For now, the app will follow the user's lead: if you enter traditional text, it will return traditional text, and same goes for simplified.

Re: Show HN: Mandarin Word Segmenter with Translation

#23

Post by routerl » Sat, Feb 08, 2025, 11:51 AM UTC

carom wrote: post #17

Did you find the library jieba? That is what I am using for segmentation. It seems to work fine on simplified despite not advertising it.

I did! Jieba is the first step in my segmentation pipeline. As far as I can tell, Jieba's default config tends to work better for simplified, but in my case the custom dictionary I feed it has significantly more traditional entries than simplified entries, especially for historical terms and slang.

Re: Show HN: Mandarin Word Segmenter with Translation

#24

Post by routerl » Sat, Feb 08, 2025, 11:52 AM UTC

georgeplusplus wrote: post #19

Have you used the app Pleco? That app has been invaluable as someone learning Chinese. that app breaks down mandarin sentences into individual characters. I believe it’s made by a Taiwanese developer too. I tried your app with a few sentences and it works really well!

Thank you, and thanks for checking it out!

I use Pleco almost every day :)

Re: Show HN: Mandarin Word Segmenter with Translation

#25

Post by ipnon » Sat, Feb 08, 2025, 1:42 PM UTC

cannam wrote: post #20

This was my attempt at doing something a little bit like it, 27 years ago. It's mostly interesting as a historical artifact - certainly yours is a lot more sophisticated and much much prettier! This one just does greedy matching against CEDICT. https://all-day-breakfast.com/chinese/ What is kind of interesting is that the script itself (a single Perl CGI script) has survived the passage of time better than the text d…

Awesome! Your Chinese must be pretty good now. Do you still have the Perl source? How did you even devise such a project in 1998?

Re: Show HN: Mandarin Word Segmenter with Translation

#26

Post by gs17 » Sat, Feb 08, 2025, 3:12 PM UTC

routerl wrote: post #21

greyman wrote: post #18

OP, thank you for your work, I will continue to watch it. I tried to built something similar, but what I didn't discover and think is crucial is the proper FE: yes, word segmenting is useful, but if I have to click on each word to see its meaning, for how I learn Chinese by reading texts, I still find Zhongwen Chrome extension to be more useful, since I see English meaning quicker, just by hover cursor over the word.…

Thanks for the kind words, and the bug report! The (awful and incorrect) translation you've pointed out comes from the segmenter being too greedy, not finding the (non-existent) word in any dictionary, and therefore dispatching the word to be machine translated, without context. This is the final fallback in the segmentation pipeline, to avoid displaying nothing at all, and my priority right now is making the segment…

What are you using for machine translation? I'm surprised anything could mistranslate 气功师 like that.

Re: Show HN: Mandarin Word Segmenter with Translation

#27

Post by routerl » Sat, Feb 08, 2025, 3:48 PM UTC

gs17 wrote: post #26

routerl wrote: post #21

Earlier quoted context omitted.

Thanks for the kind words, and the bug report! The (awful and incorrect) translation you've pointed out comes from the segmenter being too greedy, not finding the (non-existent) word in any dictionary, and therefore dispatching the word to be machine translated, without context. This is the final fallback in the segmentation pipeline, to avoid displaying nothing at all, and my priority right now is making the segment…

What are you using for machine translation? I'm surprised anything could mistranslate 气功师 like that.

For anonymous users, I'm using OpenNMT, via Argos. Logged in users get DeepL translations, which correctly translates 气功师.

Re: Show HN: Mandarin Word Segmenter with Translation

#28

Post by AlchemistCamp » Sat, Feb 08, 2025, 4:07 PM UTC

gwd wrote: post #16

> Thus, learning Mandarin by reading requires first memorizing hundreds or thousands of words, before you can even know where one word ends and the next word begins. That's not been my experience at all: As long as the content I'm reading is at the right level, I've been able to learn to segment as my vocabulary has grown, and there's always only a few new words that I haven't learned how to recognize in-context yet.…

I had yet another experience. Before Chrome existed, I learned thousands of words by hearing them before I could read them.

A lot of the pain I see in foreigners learning Chinese is that they try to tackle the written language too early. Actually, I think that's sub-optimal in any language, but it's even more expensive in a language like Chinese or Thai where word segmentation isn't a part of the writing system. I totally get it since characters are cool and curiosity about them draws many learners to the language, but it's a lot easier to take on one challenge at a time!

Re: Show HN: Mandarin Word Segmenter with Translation

#29

Post by AlchemistCamp » Sat, Feb 08, 2025, 4:11 PM UTC

georgeplusplus wrote: post #19

Have you used the app Pleco? That app has been invaluable as someone learning Chinese. that app breaks down mandarin sentences into individual characters. I believe it’s made by a Taiwanese developer too. I tried your app with a few sentences and it works really well!

I don't think it's a Taiwanese developer. If so, he got a lot of stroke orders wrong for traditional characters as per the MOE standards.

Anytime I need to look up a character or word, I go to https://www.moedict.tw/ first. Pleco is still great for having so many add-ons (including dictionaries) and, from what I've heard, some decent graded readers.

Re: Show HN: Mandarin Word Segmenter with Translation

#30

Post by thaumasiotes » Sat, Feb 08, 2025, 7:29 PM UTC

wonnage wrote: post #14

thaumasiotes wrote: post #12

> Thus, learning Mandarin by reading requires first memorizing hundreds or thousands of words, before you can even know where one word ends and the next word begins. That's not true at all; you can go a long way just by clicking on characters in Pleco, and Pleco's segmentation algorithm is awful. (Specifically, it's greedy "find the longest substring starting at the selected character for which a dictionary entry exi…

That's not terrible for a dictionary where you control what you're looking up. I found poor segmentation to be way more annoying when trying to do stuff like selecting words for highlighting on my Kindle - it invariably breaks up a chengyu or selects nonsensical groups of words. Same with the Mac 3d touch quick look.

But that's the same interface every time. You select a character and a lookup pops up. The Pleco controls are "touch a character to pop up an entry", "move end of selection window left", "move end of selection window right", "jump to previous 'word'", and "jump to next 'word'". The popup will display an entry for whatever is the longest prefix of the selection window that has an entry. How do your touch interfaces differ?