Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write a spelling corrector (2016)

norvig.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: How to write a spelling corrector (2016)

#81

Post by rasz »

crispyporkbites wrote: post #37
rietta wrote: post #35

Earlier quoted context omitted.

Not only there. Automatic word replacement is my absolute least favorite iOS mobile feature. I cannot tell you how many times I painstakingly, painfully thumb typed exactly what I meant just to have it changed out from under me. And I have gone back to see nonsense messages where I know I didn't type what was sent. Very frustrating.

I think this could be corrected if the spell corrector factored in the time taken to write the word out. If the user takes longer than average to write the word they are probably deliberately spelling it out, compared to just mashing the screen to get the words through.

Throw RNN at it, like this toy example http://karpathy.github.io/2015/05/21/rnn-effectiveness/ and make it figure out which spelling correction is the most probable.

You could even train it on past messages making it more tailored towards particular user.

Re: How to write a spelling corrector (2016)

#82

Post by cup-of-tea »

rietta wrote: post #35
shdon wrote: post #25

Spelling correction rather than spell checking and suggestions really bugs me. As soon as you have a somewhat international audience (and given that this is the internet - you probably do) or have any non-English content, automatic correction can be an actively user-hostile "feature". Especially on search engines, sometimes I am searching for words in another language, mixed-language results, or even doing an intenti…

Not only there. Automatic word replacement is my absolute least favorite iOS mobile feature. I cannot tell you how many times I painstakingly, painfully thumb typed exactly what I meant just to have it changed out from under me. And I have gone back to see nonsense messages where I know I didn't type what was sent. Very frustrating.

I distinctly remember the time when all of a sudden many of my friends started sending me sms messages with really odd mistakes in them. I didn't jump on the smartphone wagon for a number of years so I just couldn't understand how these errors were being made at the time.

Re: How to write a spelling corrector (2016)

#83

Post by hellofunk »

abecedarius wrote: post #73
hellofunk wrote: post #23

Here is Rich Hickey's port of this Norvig exercise to Clojure: https://en.wikibooks.org/wiki/Clojure_Programming/Examples/N...

It turns out Hickey added some functions to Clojure to be able to write that code. Nothing wrong with that -- they're useful functions! -- but I found it kind of amusing. It can be good to be a king.

I don't see any functions in that code that wouldn't have existed in Clojure 1.0. It's all pretty standard Clojure stuff, very basic 'for' constructs and generalized lazy sequences with 'range' and others.

Which function were you thinking were explicitly added to the language that are in use here?

Re: How to write a spelling corrector (2016)

#84

Post by abecedarius »

hellofunk wrote: post #83
abecedarius wrote: post #73

Earlier quoted context omitted.

It turns out Hickey added some functions to Clojure to be able to write that code. Nothing wrong with that -- they're useful functions! -- but I found it kind of amusing. It can be good to be a king.

I don't see any functions in that code that wouldn't have existed in Clojure 1.0. It's all pretty standard Clojure stuff, very basic 'for' constructs and generalized lazy sequences with 'range' and others. Which function were you thinking were explicitly added to the language that are in use here?

"Along the way, Clojure got subs(tring), slurp (a file), max-key and min-key..." at https://groups.google.com/forum/#!topic/clojure/YDQdmAsUaFU

Re: How to write a spelling corrector (2016)

#85

Post by hellofunk »

abecedarius wrote: post #84
hellofunk wrote: post #83

Earlier quoted context omitted.

I don't see any functions in that code that wouldn't have existed in Clojure 1.0. It's all pretty standard Clojure stuff, very basic 'for' constructs and generalized lazy sequences with 'range' and others. Which function were you thinking were explicitly added to the language that are in use here?

"Along the way, Clojure got subs(tring), slurp (a file), max-key and min-key..." at https://groups.google.com/forum/#!topic/clojure/YDQdmAsUaFU

Thanks, that is definitely interesting. We have Norvig to thank for every-day Clojure functions.

Re: How to write a spelling corrector (2016)

#86

Post by inteleng »

djtriptych wrote: post #76
falsedan wrote: post #31

The unit tests worry me: assert len(WORDS) == 32192 assert sum(WORDS.values()) == 1115504 assert WORDS.most_common(10) == [ ('the', 79808), ('of', 40024), ('and', 38311), ('to', 28765), ('in', 22020), ('a', 21124), ('that', 12512), ('he', 12401), ('was', 11410), ('it', 10681)] assert WORDS['the'] == 79808 Those aren't testing the file open, or Counter, or read, but instead are tightly-coupling the tests to the exact…

1. Read blog post. 2. Opine on how blog code is not ready for production. 3. Post to HN. 4. Propser.

This is a 1% engineer? The guy who can't spell "prosper"?

Re: How to write a spelling corrector (2016)

#87

Post by djtriptych »

inteleng wrote: post #86
djtriptych wrote: post #76

Earlier quoted context omitted.

1. Read blog post. 2. Opine on how blog code is not ready for production. 3. Post to HN. 4. Propser.

This is a 1% engineer? The guy who can't spell "prosper"?

hahaha man. Thank you. I upvoted :)
Post reply on HN