How to write a spelling corrector (2016) • Hacker News Board

Re: How to write a spelling corrector (2016)

#51

Post by scott_s » Wed, Apr 25, 2018, 2:46 PM UTC

madacoo wrote: post #38

lb1lf wrote: post #32

I know a spelling corrector is not the same thing as a spelling checker, but this is too good an opportunity to pass to promote Martha Snow's hilarious poem 'Spell Chequer': Eye halve a spelling chequer It came with my pea sea It plainly marques four my revue Miss steaks eye kin knot sea. Eye strike a quay and type a word And weight four it two say Weather eye am wrong oar write It shows me strait a weigh. As soon as…

Thanks for sharing this. I found it difficult to parse the initial couple of lines because I was constantly attempting to read by attaching meaning to the spellings: "Eye halve" is a bit frightening in that sense. But then I realised I can read the text as sounds and almost ignore the spellings. Listening to what the sounds made in my head allowed for a much faster pace of comprehension because I didn't have to keep…

I think this is how all poems should be read, and I didn't realize until just now that I automatically did that.

Re: How to write a spelling corrector (2016)

#52

Post by currymj » Wed, Apr 25, 2018, 2:46 PM UTC

sp332 wrote: post #29

A couple of years ago, Emily Short made an "interactive fiction" game [that's what they call text adventures these days] with letter-removal as a major mechanic. It's called Counterfeit Monkey and it's a lot of fun. http://emshort.com/counterfeit_monkey/ Install one of the interpreters at the bottom of the page, then download the "story" file and open it in said interpreter.

it is an extremely fun game!

especially for all the oldheads around here, if you ever played Zork or Hitchhiker's Guide or whatever, give it a shot. it's the same style, just as funny, with helpful features to make it far less frustrating than the old games could be, and it really takes advantage of not having to run on a TRS-80.

Re: How to write a spelling corrector (2016)

#53

Post by scott_s » Wed, Apr 25, 2018, 2:55 PM UTC

falsedan wrote: post #31

The unit tests worry me: assert len(WORDS) == 32192 assert sum(WORDS.values()) == 1115504 assert WORDS.most_common(10) == [ ('the', 79808), ('of', 40024), ('and', 38311), ('to', 28765), ('in', 22020), ('a', 21124), ('that', 12512), ('he', 12401), ('was', 11410), ('it', 10681)] assert WORDS['the'] == 79808 Those aren't testing the file open, or Counter, or read, but instead are tightly-coupling the tests to the exact…

His tests are appropriate because they are task-oriented. He is not writing a general spellchecker, but a spellchecker designed to work with a specific corpus. His tests ensure that the corpus remains unchanged during development (say, doesn't accidentally remove some words). What's implicit here is that if the corpus changes, he may need to change his approach.

Re: How to write a spelling corrector (2016)

#54

Post by astine » Wed, Apr 25, 2018, 3:20 PM UTC

scott_s wrote: post #51

madacoo wrote: post #38

Earlier quoted context omitted.

Thanks for sharing this. I found it difficult to parse the initial couple of lines because I was constantly attempting to read by attaching meaning to the spellings: "Eye halve" is a bit frightening in that sense. But then I realised I can read the text as sounds and almost ignore the spellings. Listening to what the sounds made in my head allowed for a much faster pace of comprehension because I didn't have to keep…

I think this is how all poems should be read, and I didn't realize until just now that I automatically did that.

There is apparently a big divide between people who subvocalize when they read and those who don't. Those who don't tend to read much faster than those who do which is why speedreading techniques tend to focus on eliminated subvocalization. The problem is that people who subvocalize tend to need to do so in order to understand the text.

https://en.wikipedia.org/wiki/Subvocalization

Re: How to write a spelling corrector (2016)

#55

Post by sp332 » Wed, Apr 25, 2018, 3:22 PM UTC

currymj wrote: post #52

sp332 wrote: post #29

A couple of years ago, Emily Short made an "interactive fiction" game [that's what they call text adventures these days] with letter-removal as a major mechanic. It's called Counterfeit Monkey and it's a lot of fun. http://emshort.com/counterfeit_monkey/ Install one of the interpreters at the bottom of the page, then download the "story" file and open it in said interpreter.

it is an extremely fun game! especially for all the oldheads around here, if you ever played Zork or Hitchhiker's Guide or whatever, give it a shot. it's the same style, just as funny, with helpful features to make it far less frustrating than the old games could be, and it really takes advantage of not having to run on a TRS-80.

Here's her blog post about which interpreters to use to play the game. (Spoilers in the comments but not in the main post.) https://emshort.blog/2012/12/31/counterfeit-monkey/ Also there's a cheat/hint guide [PDF] https://emshort.blog/2013/01/24/making-of-counterfeit-monkey....

There's a spoiler-filled post about the design of the puzzles here. https://emshort.blog/2013/01/24/making-of-counterfeit-monkey...

It won Best Game, Best Setting, Best Puzzles, Best Individual Player Character, and Best Implementation in 2012! http://www.ifwiki.org/index.php/Counterfeit_Monkey There's a nod to this game in https://xkcd.com/1975/ Right-click the image, go to games -> advent.exe and start exploring :)

Re: How to write a spelling corrector (2016)

#56

Post by sp332 » Wed, Apr 25, 2018, 3:22 PM UTC

sp332 wrote: post #29

A couple of years ago, Emily Short made an "interactive fiction" game [that's what they call text adventures these days] with letter-removal as a major mechanic. It's called Counterfeit Monkey and it's a lot of fun. http://emshort.com/counterfeit_monkey/ Install one of the interpreters at the bottom of the page, then download the "story" file and open it in said interpreter.

There's an updated version here - I didn't realize that page had an old release on it. https://emshort.blog/2017/12/26/counterfeit-monkey-release-7...

Re: How to write a spelling corrector (2016)

#57

Post by scott_s » Wed, Apr 25, 2018, 3:34 PM UTC

astine wrote: post #54

scott_s wrote: post #51

Earlier quoted context omitted.

I think this is how all poems should be read, and I didn't realize until just now that I automatically did that.

There is apparently a big divide between people who subvocalize when they read and those who don't. Those who don't tend to read much faster than those who do which is why speedreading techniques tend to focus on eliminated subvocalization. The problem is that people who subvocalize tend to need to do so in order to understand the text. https://en.wikipedia.org/wiki/Subvocalization

Thanks for the term - I think I typically don't subvocalize, but I do when I read poems.

Re: How to write a spelling corrector (2016)

#58

Post by lcuff » Wed, Apr 25, 2018, 3:42 PM UTC

lb1lf wrote: post #32

I know a spelling corrector is not the same thing as a spelling checker, but this is too good an opportunity to pass to promote Martha Snow's hilarious poem 'Spell Chequer': Eye halve a spelling chequer It came with my pea sea It plainly marques four my revue Miss steaks eye kin knot sea. Eye strike a quay and type a word And weight four it two say Weather eye am wrong oar write It shows me strait a weigh. As soon as…

A fun story in the same vein, Ladle Rat Rotten Hut, by H.L. Chace, can be found here: http://www.exploratorium.edu/files/exhibits/ladle/

Re: How to write a spelling corrector (2016)

#59

Post by meitham » Wed, Apr 25, 2018, 3:44 PM UTC

No PEP8! Code rejected

Most elegant code I've actually seen, and can see the lisp thinking apparent on the style.

Re: How to write a spelling corrector (2016)

#60

Post by falsedan » Wed, Apr 25, 2018, 4:25 PM UTC

cdancette wrote: post #34

falsedan wrote: post #31

The unit tests worry me: assert len(WORDS) == 32192 assert sum(WORDS.values()) == 1115504 assert WORDS.most_common(10) == [ ('the', 79808), ('of', 40024), ('and', 38311), ('to', 28765), ('in', 22020), ('a', 21124), ('that', 12512), ('he', 12401), ('was', 11410), ('it', 10681)] assert WORDS['the'] == 79808 Those aren't testing the file open, or Counter, or read, but instead are tightly-coupling the tests to the exact…

This code is not written for production, it's just written to make you understand how the basic of this technology works. So I would say the unit tests have the exact same purpose: make the reader understand what the functions are doing (and not a real unit testing). Since you speak of value, the value Peter Norvig is trying to provide is making readers understand the principles, he's not trying to provide some monet…

> the value Peter Norvig is trying to provide is making readers understand the principles, he's not trying to provide some monetary value

I think he's trying to show how to approach solving this kind of problem. Juniors will copy it, and will then write tests in a business-logic environment which have dubious value.

The code shows how to solve the problem very well. The tests do not.