"But they didn't, and come to think of it, why should they know about something so far outisde their specialty?"
Outside ;)
How to write a spelling corrector (2016)
11–20 of 87 posts
Re: How to write a spelling corrector (2016)
#12I've used http://php.net/manual/en/function.pspell-suggest.php in one of my projects. It's simple and scalable;
Re: How to write a spelling corrector (2016)
#13I wish articles about NLP wouldn't assume by default the language to be English. Many points of this article can also apply to other languages, but may require some more thought. Either other languages should be mentioned, or the title should be "How to Write an English Spelling Corrector".
Re: How to write a spelling corrector (2016)
#14This has been an area of interest for me as part of working on Typesense[1]. If you are looking to implement spelling correction, you cannot but not stumble on this excellent post by Peter Norvig (most of it written on a bored flight journey!).
While it's clever and concise, in terms of raw speed, indexing your vocabulary in a Trie, and then doing a traversal on it using a Levenshtein distance is way faster[2]. By using a trie, you eliminate lot of unwanted look-ups that Peter Norvig's brute-force approach takes.
The other interesting thing for me personally about this problem is that there is a statistical angle as well. You will often find words that are of the same edit distance from a given target word -- you will need to rely on some form of "popularity metric" to rank those tied corrections. E.g. how many times did people type this word and then change it to another word - that's precisely the kind of data that Google has and which makes its search suggestions and spell checking so intuitive.
[1]: https://typesense.org/ [2]: http://stevehanov.ca/blog/index.php?id=114
Re: How to write a spelling corrector (2016)
#15What I would like to know is why the spelling correction suggestion of Apple iOS are so bad. Why can they get along with this ? I used to disable it.
It can't even make a suggestion when there is only one missing character.
Re: How to write a spelling corrector (2016)
#16A S-P-E-L-L-I-N-G C-O-R-R-E-C-T-O-R.
Re: How to write a spelling corrector (2016)
#17Can mods add (2016) to the title?
Re: How to write a spelling corrector (2016)
#18This has been an area of interest for me as part of working on Typesense[1]. If you are looking to implement spelling correction, you cannot but not stumble on this excellent post by Peter Norvig (most of it written on a bored flight journey!). While it's clever and concise, in terms of raw speed, indexing your vocabulary in a Trie, and then doing a traversal on it using a Levenshtein distance is way faster[2]. By us…
>indexing your vocabulary in a Trie
fst (finite state transducer) is even smaller. And for some languages like portuguese where a lot of suffixes are extremely common, the size reduction is dramatic!
>The other interesting thing for me personally about this problem is that there is a statistical angle as well. You will often find words that are of the same edit distance from a given target word
Indeed.
http://www.ling.helsinki.fi/~klinden/pubs/PirinenLrec2010.pd...
Re: How to write a spelling corrector (2016)
#19While doing a bunch of research on exactly this problem space recently for a project, I stumbled onto this improvement on the Norvig corrector idea http://blog.faroo.com/2012/06/07/improved-edit-distance-base... that's one of those things that's so deceptively simple you kick yourself for not thinking of it: it turns out you can model the same "generate all the variations" effect but generating only the deletes, rath…
The latest version of SymSpell (by the author of that blog post) handles compound words too, so it's pretty capable along with speed.
https://github.com/wolfgarbe/SymSpellCompound/blob/master/RE...
I'm certainly not knocking that achievement, but the two issues I ran into fairly quickly were: 1. It (currently) doesn't handle words that are genuine words but which are contextually wrong 2. You need a high quality dictionary that's also well aligned with your domain or you'll have poor corrections (this last point is merely a matter of effort, so less of a concern)
Re: How to write a spelling corrector (2016)
#20"But they didn't, and come to think of it, why should they know about something so far outisde their specialty?" Outside ;)
Nice catch!