Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A spellchecker used to be a major feat of software engineering (2008)

prog21.dadgum.com

Post reply on HN
Print view View original

81–90 of 154 posts

Re: A spellchecker used to be a major feat of software engineering (2008)

#81

Post by tgv »

staplung wrote: post #68

Really, there are two components to a modern spell checker. First, identify the words that are misspelled. Second, offering (good) corrections. This article is really talking about how hard even the first task was when memory was scarce. Offering good suggestions is still non-trivial.

Third: identifying words that appear in the dictionary, but are not used properly. And that's not looking at compounds and agglutinative languages. But there's no need to look that far: capitalization isn't even 100% solved. The article does software development and linguistics a bad service by suggesting spell checking is just loading a dictionary into memory.

Re: A spellchecker used to be a major feat of software engineering (2008)

#83

Post by aaron695 »

How much better are they today scored as a number? Like Elo.

I wish we had spell checking (with corrections) competitions.

Computer based chess is very cool.

But a little time spent on a computer based spellchecking competition would help a lot of native and non native speakers every day.

And I'm just talking English, to start.

Re: A spellchecker used to be a major feat of software engineering (2008)

#84

Post by indeyets »

oneearedrabbit wrote: post #46
Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

I suppose, it is worth giving a shout-out to a recent Hunspell port to Python by Zverok: https://github.com/zverok/spylls , this description from Github sums it up nicely: > Hunspell is a long-living, complicated, almost undocumented piece of software, and it was our feeling that the significant part of human knowledge is somehow "locked" in a form of a large C++ project. That's how Spylls was born: as an attempt to…

Exactly. It's a very important argument against calling it a solved problem. I believe the problem with hunspell is that it's very hard to get into fixing it (it's good, but it's not perfect) and spylls makes it feasible again

Re: A spellchecker used to be a major feat of software engineering (2008)

#86

Post by Aissen »

Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

If french is one of your language, state of the art has been Druide's Antidote for more than 20 years. And it has had a Linux version available for almost 15 years, too.

Re: A spellchecker used to be a major feat of software engineering (2008)

#87

Post by efitz »

powersnail wrote: post #21
DanBC wrote: post #5

Here's an article that people might be interested in. It gives a bit more detail: https://web.archive.org/web/20100706052342/http://www.spelli... I'm particularly interested in this one, and I'm curious about how useful something like this would be to use. > The second does not use a dictionary at all (Morris & Cherry 1975). Like the previous method, it divides the text into trigrams, but it creates a table of these,…

Consider that the most often typo I make is "ture" for "true", and "flase" for "false", I'd say this isn't going to catch some common mistakes.

If you just want to fix typos then it seems to me that Damerau-Levenshtein distance would be the best approach - a DL distance of 1 from a dictionary word is a high likelihood of a transposition or a missing or extra character.

Re: A spellchecker used to be a major feat of software engineering (2008)

#89

Post by flohofwoe »

...and yet the first thing I switch off in every tool is the spellchecker, because they get utterly confused with the mix of English, German and my local dialect that I'm using when communicating with different people. I'd say doing a spellchecker "right" is apparently still a major feat of software engineering ;)

Re: A spellchecker used to be a major feat of software engineering (2008)

#90

Post by vbezhenar »

Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

I would argue that it's not a hard engineering problem. It's just a problem that should be solved. Linguists made a lot of formalizations about language. There are rules, there are known lists of exceptions, there are dictionaries. You just have to implement those things carefully with lots of edge-cases, implement it for every language, but it's just mundane engineering work, nothing that requires breakthrough.

Currently spell checking just checks words from a dictionary. Its 1% of work. Or even less.

Microsoft Word used to have awesome spell checking, grammar checking, etc back in 2000 for Russian language. It seems to be degraded since then. But the fact is, this problem was solved. It checked for spelling, for grammar, for punctuation. I would expect that kind of functionality working in every OS textbox by now.

Post reply on HN