This is great. I'd love it for flashcard creation - paste in a block of text I'm reading and extract vocabulary from it.
Show HN: Mandarin Word Segmenter with Translation
31–36 of 36 posts
Re: Show HN: Mandarin Word Segmenter with Translation
#32Earlier quoted context omitted.
Thanks for the kind words! I'm using Jieba[0] because it hits a nice balance of fast and accurate. But I'm initializing it with a custom dictionary (~800k entries), and have added several layers of heuristic post-segmentation. For example, Jieba tends to split up chengyu into two words, but I've decided they should be displayed as a single word, since chengyu are typically a single entry in dictionaries. [0] https://…
Great project! It's fascinating how hard segmentation is and how many approaches there are. I thought I'd mention a trick that can let you segment without a backend. When you double click Chinese text in the browser, it will highlight an entire word. For example, try double clicking on the text here: 一步登天：走一步就到天堂美好境地。 It highlights/segments the first 4 characters as a chengyu, and the others as one or two character w…
That’s neat!
Re: Show HN: Mandarin Word Segmenter with Translation
#33This is cool. If you haven't already, you might like to take a look at Du Chinese and The Chairman's Bao. They might provide ideas or inspiration. Also the 'clip reader' feature in Pleco is decent. Also, supporting simplified as well as traditional might increase your potential audience.
It supports traditional and simplified, as well as pinyin and bopomofo :) It's already possible to switch instantly between pinyin and bopomofo, and I'm working on letting users switch between simplified/traditional, but this is also a non-trivial problem. For now, the app will follow the user's lead: if you enter traditional text, it will return traditional text, and same goes for simplified.
i've been working on a new script converter (https://github.com/creolio/mandarinTamer) for simplified and traditional (taiwanese) mandarin. almost finished--needs a few tests yet, but should be more accurate than opencc and google translate. would love to chat more about your tool. i'll send you an email.
Re: Show HN: Mandarin Word Segmenter with Translation
#34This was my attempt at doing something a little bit like it, 27 years ago. It's mostly interesting as a historical artifact - certainly yours is a lot more sophisticated and much much prettier! This one just does greedy matching against CEDICT. https://all-day-breakfast.com/chinese/ What is kind of interesting is that the script itself (a single Perl CGI script) has survived the passage of time better than the text d…
Awesome! Your Chinese must be pretty good now. Do you still have the Perl source? How did you even devise such a project in 1998?
There's a link to the Perl code hidden in the third para of text ("The [Perl source] for this script is available...") Of course a big reason it still works is that it was written for Perl 5, which is still current!
What that link doesn't give you is the dictionary files I used as input for the preprocessing step - which of course were also 1998 vintage. There are copies on the server (https://all-day-breakfast.com/chinese/cedict.b5_saved, https://all-day-breakfast.com/chinese/big5-PY.tit)
My Chinese got somewhat better, then a lot worse, then a little bit better again - obviously mostly to do with whether I was actually using it, which on the whole I haven't been. But back then I was really working on it and I just wanted something to help - there were a few useful resources I knew of (CEDICT obviously, and Rick Harbaugh's zhongwen.com was mindblowing at the time) and this seemed like a way to glue them together that I actually knew how to do.
Writing learning tools is obviously not the same thing as learning though.
Re: Show HN: Mandarin Word Segmenter with Translation
#35You should add it to Built with Django - https://builtwithdjango.com/projects/new/
Re: Show HN: Mandarin Word Segmenter with Translation
#36This is great. I'd love it for flashcard creation - paste in a block of text I'm reading and extract vocabulary from it.
OP here, I'm adding a feature that will allow users to save specific words to lists, and export the lists in formats that can be imported to flashcard apps.