Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A spellchecker used to be a major feat of software engineering (2008)

prog21.dadgum.com

Post reply on HN
Print view View original

101–110 of 154 posts

Re: A spellchecker used to be a major feat of software engineering (2008)

#101

Post by MHard »

wolfgarbe wrote: post #98

I get the point of the article that today's improved hardware allows things that weren't possible before. But neither spell checking nor its more advanced siblings spelling correction or automatic spelling correction is solved. Here is an example: "I want to by Apple" "I want to be Apple" "I want to buy Apple" Because "by" is a genuine English word, a spellchecker based on a simple hashtable will not detect the error…

That would be a grammar checker though. And there are many solutions available today to do that. Even open source solutions. Grammarly would be one example, language tool another.

The ones I looked at work by encoding common spelling mistakes into a grammar (as in Chomsky grammar) and then running that over the text.

Re: A spellchecker used to be a major feat of software engineering (2008)

#102

Post by wolfgarbe »

MHard wrote: post #101
wolfgarbe wrote: post #98

I get the point of the article that today's improved hardware allows things that weren't possible before. But neither spell checking nor its more advanced siblings spelling correction or automatic spelling correction is solved. Here is an example: "I want to by Apple" "I want to be Apple" "I want to buy Apple" Because "by" is a genuine English word, a spellchecker based on a simple hashtable will not detect the error…

That would be a grammar checker though. And there are many solutions available today to do that. Even open source solutions. Grammarly would be one example, language tool another. The ones I looked at work by encoding common spelling mistakes into a grammar (as in Chomsky grammar) and then running that over the text.

No, that's not grammar checking, but spelling mistakes that turn out to be real words!

If you misspelled "college" as "collage", or you misspelled "three" as "tree" the word you typed incorrectly happens to be an actual word itself! Correcting these types of errors is called "real word spelling correction".

Real-word spelling errors are words in a text that, although correctly spelled words in the dictionary, are not the words that the writer intended

https://www.researchgate.net/publication/221628953_Real-Word...

And even if there are several approaches evaluated, the problem is far from solved.

Re: A spellchecker used to be a major feat of software engineering (2008)

#103

Post by unpopularopp »

jiggawatts wrote: post #39
Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

To be fair, Hunspell was written specifically for Hungarian because it's such a difficult language to handle efficiently in a spell checker. It just happens to also work for other languages. If you only need English, the complexity of Hunspell is not required. I still haven't found a decent Hungarian spell checker, they get confused by rare words that have the same letters as a very common word, but different accents…

>but not "le es"

First of all it's a verb with -ik ending so you can't write "es", the root lexical word is "esik".

Second you never use anything like "le es" because generaly you always have to use the verb and prefix together ("leesett", "leesik" etc.) unless you use a commanding form ("ess le") or a modal verb ("le akart esni, "le fog esni" etc.)

"le es" is strictly wrong because it's the wrong verb (should be "esik") and you have to write together ("leesik") so obviously a spellchecker picks that up.

Re: A spellchecker used to be a major feat of software engineering (2008)

#105

Post by mellavora »

oneearedrabbit wrote: post #46
Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

I suppose, it is worth giving a shout-out to a recent Hunspell port to Python by Zverok: https://github.com/zverok/spylls , this description from Github sums it up nicely: > Hunspell is a long-living, complicated, almost undocumented piece of software, and it was our feeling that the significant part of human knowledge is somehow "locked" in a form of a large C++ project. That's how Spylls was born: as an attempt to…

my hovercraft is full of eels.

Re: A spellchecker used to be a major feat of software engineering (2008)

#106

Post by bjoli »

adrianN wrote: post #74
powersnail wrote: post #21

Earlier quoted context omitted.

Consider that the most often typo I make is "ture" for "true", and "flase" for "false", I'd say this isn't going to catch some common mistakes.

I have a bunch of iab in my vimrc for typos like that because I make them so often.

I have abbrev mode i Emacs for that. No more "teh" instead of the. I found most of them went away when I switched to Dvorak, since most of my errors like that are when the letters are in one hand. "teh" however became the all time high. 4000 corrections this year according to abbrev mode. And English isn't even the language I write the most.

Re: A spellchecker used to be a major feat of software engineering (2008)

#107

Post by WalterBright »

DanBC wrote: post #5

Here's an article that people might be interested in. It gives a bit more detail: https://web.archive.org/web/20100706052342/http://www.spelli... I'm particularly interested in this one, and I'm curious about how useful something like this would be to use. > The second does not use a dictionary at all (Morris & Cherry 1975). Like the previous method, it divides the text into trigrams, but it creates a table of these,…

I've thought of a variation on that, if you're spell checking a large text file. Create an index of all the unique words in the file, along with how many each appears. Any unique words with a count of 1 are likely to be misspelled.

Re: A spellchecker used to be a major feat of software engineering (2008)

#109

Post by coopierez »

jamil7 wrote: post #97
flohofwoe wrote: post #89

...and yet the first thing I switch off in every tool is the spellchecker, because they get utterly confused with the mix of English, German and my local dialect that I'm using when communicating with different people. I'd say doing a spellchecker "right" is apparently still a major feat of software engineering ;)

Yeah, to add to it a lot of Germans will use American English but some British English and my spellchecker can't handle the mixture. Off topic but despide how difficult German has been to learn I've found spelling in German to be far more predictable and straightforward than my native language English.

This is at least partly because all of the German speaking countries got together in 1996 to reform the spelling of the language.

See here: https://en.wikipedia.org/wiki/German_orthography_reform_of_1...

Re: A spellchecker used to be a major feat of software engineering (2008)

#110

Post by RegW »

I wrote a spelling checker in the 1980's

In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker.

For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word list down to get the whole thing onto a single 360K floppy.

After I left Tasman, I was lying in bed one night still thinking about it and realised where I had gone wrong. The tokenising thing, which someone else had put me onto had blinded me. I had stared at word lists for months and hadn't pinned down the obvious pattern. All but 26 words in the 70,000 word list shares the bulk of their characters with the word before it.

So the solution is use 5 bits of the first byte as a count of chars from the word before, 3 bits indicate commons enddings (ship, s, ing) or that the following bytes are tokens for the rest of the word. With this I got the word list compressed to less than 2 bytes per word.

I took this back to Tasman. They put all 70,000 words and the spelling checker onto a 175K floppy for the ZX Spectrum +3.

[Edited for typos]

Post reply on HN