Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

ezfurigana.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#11

Post by tkgally »

epitrochoid413 wrote: post #9
tkgally wrote: post #8

Earlier quoted context omitted.

Very, very nice. Since January, I’ve been having Claude build a static Japanese-English dictionary in which all of the kanji and jukugo can be displayed either with or without furigana: https://www.tkgje.jp/index.html I haven’t spotted any mistakes in the furigana myself, though there must be some. I have a scheduled routine running multiple times a day to have Claude check and polish existing entries; it should be c…

Thanks for sharing this. It looks like a really cool project, and making the data public domain is especially generous. I especially like the dictionary + example sentence format. I haven’t found a really good Japanese-English dictionary for learners, and yours looks promising. I’m curious how token-intensive the repeated Claude polishing runs are.

Quite token-intensive. I pay for the Max plan, and the regular dictionary runs consume maybe 20 percent of my weekly quota.

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#12

Post by k-taro56 »

I’m Japanese. I was surprised that it was able to answer correctly even when I entered commonly seen difficult-to-read place names. However, there seem to be cases where it may incorrectly read “今日” when it should be read as “こんにち.” Example: 今日の日本社会では、少子高齢化が大きな課題となっている。

Also, it’s disappointing that Japanese does not appear even when I select it.

Please let me know if there’s anything I can do to help.

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#14

Post by epitrochoid413 »

sollniss wrote: post #13

Uh, in 田中さんは今何をしている, 今何 comes out as こんなに.

Thanks, good catch. This is a known class of edge case I am trying to improve: adjacent tokens sometimes get over-merged into a phrase reading when they should stay separate. 今 and 何 should be handled separately here, not as こんなに. I appreciate this concrete example. That helps a lot.

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#15

Post by epitrochoid413 »

k-taro56 wrote: post #12

I’m Japanese. I was surprised that it was able to answer correctly even when I entered commonly seen difficult-to-read place names. However, there seem to be cases where it may incorrectly read “今日” when it should be read as “こんにち.” Example: 今日の日本社会では、少子高齢化が大きな課題となっている。 Also, it’s disappointing that Japanese does not appear even when I select it. Please let me know if there’s anything I can do to help.

Thank you this is very helpful, especially from a native speaker.

今日 is a tradeoff I made intentionally: I disabled the fallback model for it because most cases are きょう, while こんにち is much rarer. But yes, this is one of the cases that gets lost with that choice.

And agreed on Japanese dictionary support. I plan to add Japanese soon. Thanks again.

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#16

Post by k-taro56 »

epitrochoid413 wrote: post #15
k-taro56 wrote: post #12

I’m Japanese. I was surprised that it was able to answer correctly even when I entered commonly seen difficult-to-read place names. However, there seem to be cases where it may incorrectly read “今日” when it should be read as “こんにち.” Example: 今日の日本社会では、少子高齢化が大きな課題となっている。 Also, it’s disappointing that Japanese does not appear even when I select it. Please let me know if there’s anything I can do to help.

Thank you this is very helpful, especially from a native speaker. 今日 is a tradeoff I made intentionally: I disabled the fallback model for it because most cases are きょう, while こんにち is much rarer. But yes, this is one of the cases that gets lost with that choice. And agreed on Japanese dictionary support. I plan to add Japanese soon. Thanks again.

Certainly, you’re right that reading it as “konnichi” is very rare. It’s rare enough that I only sometimes misread it that way when it appears in the news or similar contexts. So I think that was a good decision.

By the way, I tried testing it further while thinking back to the kinds of tests I had when I was in school. The accuracy is still excellent. My guess is that “一日” and “分別” are being handled in a similar way to “今日.” “分別” is very rare, but I don’t think “一日” is all that uncommon.

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#18

Post by epitrochoid413 »

uasi wrote: post #17

Got an incorrect result on my first try. Input was 振り仮名変換器の性能が如何程か試してみよう. It returned 如何(どう)程(ほど) instead of 如何(いか)程(ほど). Regardless, I'm impressed with the tool!

Thanks, this kind of report is very useful.

如何 is context-dependent, and I hadn’t come across this case yet. I’ll add it to the model soon. Really appreciate the report and the kind words.

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#19

Post by [deleted] »

tkgally wrote: post #8
epitrochoid413 wrote: post #2

I built a context-aware furigana converter for Japanese text, files, and web pages. The main problem I wanted to solve was that simple dictionary-based furigana works well for common cases, but breaks on words where the reading depends on context: * 市場: いちば or しじょう * 大分: おおいた or だいぶ * 人気: にんき or ひとけ * 最中: さいちゅう or さなか or もなか * 方: かた or ほう The engine is a hybrid system: * Sudachi for tokenization, base forms, POS, and…

Very, very nice. Since January, I’ve been having Claude build a static Japanese-English dictionary in which all of the kanji and jukugo can be displayed either with or without furigana: https://www.tkgje.jp/index.html I haven’t spotted any mistakes in the furigana myself, though there must be some. I have a scheduled routine running multiple times a day to have Claude check and polish existing entries; it should be c…

[deleted]

Re: Show HN: Context-aware Japanese furigana using Sudachi and ModernBERT

#20

Post by epitrochoid413 »

k-taro56 wrote: post #16
epitrochoid413 wrote: post #15

Earlier quoted context omitted.

Thank you this is very helpful, especially from a native speaker. 今日 is a tradeoff I made intentionally: I disabled the fallback model for it because most cases are きょう, while こんにち is much rarer. But yes, this is one of the cases that gets lost with that choice. And agreed on Japanese dictionary support. I plan to add Japanese soon. Thanks again.

Certainly, you’re right that reading it as “konnichi” is very rare. It’s rare enough that I only sometimes misread it that way when it appears in the news or similar contexts. So I think that was a good decision. By the way, I tried testing it further while thinking back to the kinds of tests I had when I was in school. The accuracy is still excellent. My guess is that “一日” and “分別” are being handled in a similar way…

[dead]
Post reply on HN