Show HN: Nihongo – Study Japanese using authentic text from games, songs, etc.

IMHO your goal in creating a spaced repetition algorithm is to balance the cost of learning against the cost of forgetting. I generated a lot of data for myself of how I used JLDrill. What I found was that it takes me a little over a minute (on average) to memorize (or re-memorize) a word, but about 2 seconds to review it if I already remember it. Just to keep the math easy, let's assume that it takes 100 seconds of review to memorize something and 2 to review it if you already remember it.

What I found was that there isn't much point in reducing review frequency beyond every 2-3 months. Remember that while the forgetting curve has a gamma distribution, the section of the curve between 90-100% recall is nearly linear. So you can approximate the drop off linearly.

Assume you have an item which is scheduled to be reviewed after 1 year. Further assume that this corresponds to a 90% recall rate. This means that 10% of the time you will forget it. Since the average re-learning time is 100 seconds, that means that the average cost of forgetting is 10 seconds. If you were to review it every 6 months instead, you would review it twice in a year (an extra 2 seconds of review time), but the recall rate would increase to 95%. This means that the forgetting cost is only 5 seconds. So you have increased your review cost by 2 seconds, but decreased your forgetting cost by 5 seconds. If you were to double the frequency again (every 3 months), then you have increased the review cost by 4 seconds and decreased the forgetting cost by 7.5 seconds. The sweet spot seems to be about every 2 months according to the data for me.

So basically, I would say that your level 4 is counter productive and most of level 3 is at the edge of being useful.

One of the questions I struggled with was, "At what point do you prune vocabulary from the drill?" When do you know it "well enough"? Unfortunately, this is a difficult question because it depends on how often you use the language outside of study. Word frequency lists are illuminating for this question, though. I have not been able to find a good list of Japanese frequencies (the one in Edict is based on newspapers and therefore not good for general study). However, take a look at this English word frequency list: https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG...

The linked list shows you the most common 10K words. The first word has a frequency of about 5% (so read 100 words and you will see it 5 times on average). Even at only the 1000th most frequent word we're already down to 0.008%. So I would have to read about 12k words before I see it on average.

Consider that a 5 year old child has a vocabulary of 5k word families (not just words of vocabulary, but all permutations of that word -- so police and policeman belong to the same word family). The frequency of the 5000th word is 0.0013%, so I have to read close to 80k words before I see it on average.

Consider that in English a book has about 400 words per printed page (reference books have more, children's books have less, of course). This means that in order to be exposed to the 5000th most frequently used word once you would have to read about 200 pages of text (a short novel). Studies have also shown that you need 95% comprehension of a text in order to correctly infer the vocabulary and grammar that you don't know. So it means that you probably want to be exposed to that word 2 or 3 times a year. At 5000 words of vocabulary, you need to be reading 2-3 short novels per year in order to maintain your vocabulary.

This gives you some idea of the amount of natural language that you need to be exposed to if you want to "naturally" remember something indefinitely. A 5 year old child's proficiency in language is not terribly good (I invite you to test this out on friends and relatives ;-) ) If you want to be at a 10 year old's level you need 10k words (last word has a frequency of 0.0004% or every 250k words).

Anyway, the long and the short of it is that permanently expiring a word form study is potentially useful for beginners, but by the time you reach intermediate level, you almost certainly want to hang on to every word. Unless you get a lot of outside exposure to the language, you are unlikely to encounter these words in normal usage. In fact, I came to the conclusion that expiring words is essentially useless because you only want to do it for the most common 2k words and those will only correspond to about 4 extra hours of study per year. Allowing the user to manually expire a word is probably useful, though.

Finally, for words that you are just learning: There is some new-ish research (i.e., the last 20 years or so) that is very interesting.

There is a phenomenon known as the "spacing effect" (which is different from the confusingly similarly named "spaced repetition" and even more confusingly similarly name "spaced learning") -- to be clear, all three of these things are different ;-).

The spacing effect suggests that if you wait until you forget something and then relearn it, it will be easier to remember the second time. The interesting thing is that if you learn something, then wait until you forget, then learn it again, it turns out that you will remember it better than if you had remembered it consistently the whole time.

The spacing effect is essentially the opposite of spaced repetition. In spaced repetition we want to review something before the person forgets it. In spacing, you want to review just after the person forgets it, so that they have a chance to relearn it. The papers I have read on the subject indicate that the difference is very pronounced.

With "spaced learning" and the related "interleaving" approach, instead of spending a large chunk of time studying, you have much smaller sessions that are separated by time. The idea is to have enough time to forget something. With interleaving, you interleave study material of different subjects so that your mind is forced to think about something else (and in my understanding, forget the previous material as fast as possible). This reduces the time you need to wait until you can relearn, which increases the speed at which you can learn the material.

So having said all that, my suggestion is to space your level 1 items much further apart. A review after 1-3 days is good for the first review and then you want to wait for another few days. The goal is to allow the user to forget and relearn their material once or twice before it is on the "long term" track of review (hopefully without forgetting).

Hope you found the above interesting. There is lots of research on the topic at the moment so if you enjoy reading psychology papers, there are many to choose from ;-)