Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Write a Spelling Corrector

norvig.com

Post reply on HN
Print view View original

91–100 of 133 posts

Re: How to Write a Spelling Corrector

#91

Post by schoen »

fallous wrote: post #89
agd wrote: post #18

Earlier quoted context omitted.

In a way, I'm surprised I can read this so easily. Do our brains read by converting text to sounds, and then parsing the sounds?

I think many people do have brains that operate this way, probably because their understanding of language is grounded in conversation as a first experience. I suspect this is the reason you see confusion with "your, you're" and "there, their, they're." I don't parse language in quite that manner, and even in speaking I have a different "mouth feel" for those homonyms. This creates a modest inversion of the idea of s…

> I don't parse language in quite that manner, and even in speaking I have a different "mouth feel" for those homonyms.

I believe in the (somewhat controversial) non-subvocalization-based text processing and even think I do it myself, but I'm wondering if you could describe more about the "mouth feel" issue. Do you mean that you believe that you pronounce them using different phonology (that another person would potentially be able to hear), that your muscles are doing something different but not in a way that makes an auditory difference, or simply that you're subjectively aware of the spelling while speaking but not necessarily in a way that makes a physically-observable difference? Or is it not quite clear which of these is the case?

I think this is an interesting question in the philosophy of language and also in the psychology of reading. (I've thought about this myself but haven't studied the academic literature about it.) If your answer is the first one, I wonder if you'd be willing to make an audio recording of yourself pronouncing these words that might show what difference you experience.

By the way, there are documented cases where spelling differences have created new pronunciation differences that didn't previously exist in the spoken language. Maybe something like that has been happening in your idiolect?

Re: How to Write a Spelling Corrector

#92

Post by majewsky »

bsenftner wrote: post #22
Tloewald wrote: post #20

Earlier quoted context omitted.

Interesting. I had to write a badwords filter for a random name generator (for a procedural galaxy generator...) and it's quite a task just for English.

If you've not used a Trie https://en.wikipedia.org/wiki/Trie I suggest checking them out. Very useful for fuzzy searching.

Indeed. I did a spell checker for a Haskell university course, and the trie was the recommended data structure. Makes it really easy to eliminate large chunks of the dictionary at once when edit distance gets too large during traversal.

Re: How to Write a Spelling Corrector

#93

Post by majewsky »

Animats wrote: post #33

That's not how Google does it. Try spelling errors with Google Search. Google does multi-word spelling correction.

I guess that Google looks at each word individually. And what Google does is intriguingly simple: When they see someone do two similar searches from the same person, where the second search has way more results, they record the first search as "wrong spelling", and the second one as "correction".

Re: How to Write a Spelling Corrector

#94

Post by lb1lf »

fallous wrote: post #90
dsr_ wrote: post #47

Earlier quoted context omitted.

Early reading is taught that way. You may have heard of "Phonics" as an element of literacy? Full literacy in alphabetic-phonetic languages includes the steps of (1) no longer sounding out words aloud and (2) no longer sounding out most words in your head, but rather grasping the shape of the word immediately.

It's the jump from word-as-sound to word-as-symbol, which I think a lot of people never make, partially because it's not particularly useful in their lives. I suspect those involved in programming skew very much towards the word-as-symbol, but whether that is causative or correlative I can't say.

I definitely belong to the word-as-symbol camp when reading in my native Norwegian. Besides, I am a programmer (A shoddy one in all but assembly language, though...)

Interestingly, perhaps, is that I recently made the transition from word-as-sound to word-as-symbol in another language - morse code; that felt very odd while it was going on - I have parsed words character by character for a couple of decades, and suddenly, I found my decoding lagging further behind - I had started hearing words as units, rather than composites of characters. Funnily enough, it was through no conscious effort - just happened, over the course of a few hours.

Re: How to Write a Spelling Corrector

#95

Post by nommm-nommm »

misiti3780 wrote: post #45
leblancfg wrote: post #27

Earlier quoted context omitted.

Came here to say this. I've combed through his pieces many times over just to glean the way he structures his code. Here, he got my head spinning for a minute with return set(w for w in words if w in WORDS) and made a mental note to use that idiom in the future. As for doc strings, well, this is just a toy piece of code after all. The main purpose for the code is to be read, instead of actually used. something someth…

that is a great idiom. i remember the first time i saw it (possibly in is very essay), i thought it was awesome. I use it all over the place now. it is similiar to the javascript ternary operator [1], which I also find to be very useful var varX = (boolean) ? 'X' : 'Y' [1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Pretty much every language has a ternary conditional operator: C++, Java, Python, C#, Ruby....

Re: How to Write a Spelling Corrector

#96

Post by leblancfg »

xapata wrote: post #74
benhoyt wrote: post #39

Earlier quoted context omitted.

Nothing's wrong with them. Though it is more usual to use triple quotes for docstrings even if they fit on one line (see PEP 257, "Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.").

Easy in the sense of more pleasant version control diffs.

Couldn't it cause issues with "object definition" code in REPLs and editors, and interfere with documentation generators, i.e. [Sphinx](http://www.sphinx-doc.org/en/stable/)?

Re: How to Write a Spelling Corrector

#97

Post by D-Coder »

drauh wrote: post #65
ChicagoBoy11 wrote: post #52

I couldn't help but read this and think about all the "coding" initiatives I've seen in K-12 and shake my head. What Norvig is doing is what we should be teaching. He is tackling this seemingly REALLY hard problem by thinking about it methodically, translating some intuition into code, carefully constructing an argument about how to solve it, and ways that it could be extended. This is what actual engineers look like…

By that reasoning, we should not teach kids how to spell, or about punctuation, and just aim for them writing essays/stories/novels.

That does turn out to be more effective in teaching them to write coherently.

Spelling and punctuation details can come later. If the kids start out being wrong all the time, they just quit.

Re: How to Write a Spelling Corrector

#98

Post by petercooper »

lb1lf wrote: post #3

Spell chequer Martha Snow Eye halve a spelling chequer It came with my pea sea It plainly marques four my revue Miss steaks eye kin knot sea. Eye strike a quay and type a word And weight four it two say Weather eye am wrong oar write It shows me strait a weigh. As soon as a mist ache is maid It nose bee fore two long And eye can put the error rite It's rare lea ever wrong. Eye have run this poem threw it I am shore y…

I think it's cool how you can tell the writer's accent from this - something you can rarely do with written English. "halve" and "kin knot" sound nothing like "have" or "cannot" in my native accent, for example.

Similarly, rhymes in Shakespeare's sonnets have been used to work out how Shakespeare spoke the same words, as we no longer pronounce the words the same way (and many of the rhymes have been broken).

Re: How to Write a Spelling Corrector

#99

Post by imgabe »

danso wrote: post #55
ChicagoBoy11 wrote: post #52

I couldn't help but read this and think about all the "coding" initiatives I've seen in K-12 and shake my head. What Norvig is doing is what we should be teaching. He is tackling this seemingly REALLY hard problem by thinking about it methodically, translating some intuition into code, carefully constructing an argument about how to solve it, and ways that it could be extended. This is what actual engineers look like…

I agree that what Norvig demonstrates here and in all of his notebooks is an ideal of how we use programming to explore (nevermind implement ) concepts. But how do we get there without teaching people how to program, including syntax? I think everyone agrees that kids should be able to understand the themes of "A Modest Proposal" and perhaps even write with such depth, but they have to learn their ABCs and be compell…

The compiler/interpreter will correct your syntax. Explain it once and show where the documentation is, then let kids loose. There's no need to do a worksheet with 100 "spot the syntax error" problems or to have to write it out by hand on a piece of paper to learn it. That kind of pain is only going to teach kids that programming is a boring game of "find the missing semi-colon".

Re: How to Write a Spelling Corrector

#100

Post by shanusmagnus »

drauh wrote: post #65
ChicagoBoy11 wrote: post #52

I couldn't help but read this and think about all the "coding" initiatives I've seen in K-12 and shake my head. What Norvig is doing is what we should be teaching. He is tackling this seemingly REALLY hard problem by thinking about it methodically, translating some intuition into code, carefully constructing an argument about how to solve it, and ways that it could be extended. This is what actual engineers look like…

By that reasoning, we should not teach kids how to spell, or about punctuation, and just aim for them writing essays/stories/novels.

"If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea."
Post reply on HN