How to Write a Spelling Corrector

#1

Post by sprt » Thu, Dec 26, 2019, 8:57 AM UTC

How to Write a Spelling Corrector
norvig.com

Re: How to Write a Spelling Corrector

#2

Post by tgv » Thu, Dec 26, 2019, 10:50 AM UTC

> We need a lot of data to do this well.

Yup. And once you've got a sufficiently large tagged error corpus, spelling correction will be as simple as a lookup: almost all misspellings (in terms of frequency) will be present in the corpus, and you can drop the (rather simplistic) algorithmic part.

Re: How to Write a Spelling Corrector

#3

Post by brudgers » Thu, Dec 26, 2019, 1:41 PM UTC

Some previous discussions,

https://news.ycombinator.com/item?id=16918757

https://news.ycombinator.com/item?id=12453535

https://news.ycombinator.com/item?id=8749518

https://news.ycombinator.com/item?id=7487341

Re: How to Write a Spelling Corrector

#4

Post by techbio » Thu, Dec 26, 2019, 6:58 PM UTC

tgv wrote: post #2

> We need a lot of data to do this well. Yup. And once you've got a sufficiently large tagged error corpus, spelling correction will be as simple as a lookup: almost all misspellings (in terms of frequency) will be present in the corpus, and you can drop the (rather simplistic) algorithmic part.

By way of word2vec, or ?

Re: How to Write a Spelling Corrector

#5

Post by colejohnson66 » Thu, Dec 26, 2019, 11:26 PM UTC

tgv wrote: post #2

> We need a lot of data to do this well. Yup. And once you've got a sufficiently large tagged error corpus, spelling correction will be as simple as a lookup: almost all misspellings (in terms of frequency) will be present in the corpus, and you can drop the (rather simplistic) algorithmic part.

But what about “they’re”, “there”, and “their”? And other homonyms people frequently misspell?