Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A spellchecker used to be a major feat of software engineering (2008)

prog21.dadgum.com

Post reply on HN
Print view View original

111–120 of 154 posts

Re: A spellchecker used to be a major feat of software engineering (2008)

#111

Post by Aachen »

Avamander wrote: post #92
flohofwoe wrote: post #89

...and yet the first thing I switch off in every tool is the spellchecker, because they get utterly confused with the mix of English, German and my local dialect that I'm using when communicating with different people. I'd say doing a spellchecker "right" is apparently still a major feat of software engineering ;)

I wish it just struggled with the mix of multiple languages, it can't even handle my native one. A plural compound word in translative case with a clitic? I can only dream about it, it can't even do all the cases reliably.

What language is that?

I guess what I'm wondering is how obscure it has to get (or perhaps how low the GDP of the people speaking a language has to be) before custom spell checking rules aren't considered worth it to bigcorps anymore. Though I'm also kinda interested in what this weird language thing is you're talking about.

Re: A spellchecker used to be a major feat of software engineering (2008)

#112

Post by brianmcc »

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

What an awesome insight, thanks for sharing. I love hearing these kinds of "unobvious" solutions. Simple and elegant but all too easily missed!

Re: A spellchecker used to be a major feat of software engineering (2008)

#113

Post by berkes »

lacker wrote: post #64
jeroenhd wrote: post #50

Earlier quoted context omitted.

While it's not bad, it's not that great a universal tool either. My native language makes use of a lot of compound words and Google's spell checker often gets confused when I combine words according to the standard grammar. I can see and understand the technical limitations, but tools like Microsoft Word seem to do a much better job than Google's spell checker, even in things like Google Docs. Google search will ofte…

Yeah, exactly! - this is why building the perfect spell checker is still a challenging and interesting engineering problem, in 2020. The nature of misspelling varies from language to language, especially with non-letter-based languages like CJK, or with input methods that lead to different sorts of typos than a regular keyboard. Out of curiosity, what is your native language, Dutch? Can you give an example of somethi…

I'm Dutch, and here are some funny things suggested wrong by spellcheckers:

* hoofdaannemer - hoofd aannemer : main contracter - person who takes heads

* wegomlegging - weg om legging : road closure, a road around a legging(pants)

* dagverse vis - dag verse vis : catch of the day - goodbye, fresh fish

This twitter has many such errors in the wild.

https://mobile.twitter.com/spatiegebruik

Not all spellcheckers suggest it wrong, but almost all that rely on statistics, will get it wrong: hoofdaannemer (or hoofdanything), will occur far less than hoofd or aannemer.

This, I fear, is forming our language use. As people rely on tech, they'll follow the suggestions and wrong use of space will turn an actual problem.

Edit: formatting.

Re: A spellchecker used to be a major feat of software engineering (2008)

#114

Post by dathinab »

Given how terrible bad most spell checkers are I would say it's still a major fear in 2020.

(Through it's a feat of properly incorporating linguistic knowledge into your spell checker.)

It just happens that for the English language terrible bad spell checkers are often still good enough.

Re: A spellchecker used to be a major feat of software engineering (2008)

#115

Post by ivanhoe »

WalterBright wrote: post #107
DanBC wrote: post #5

Here's an article that people might be interested in. It gives a bit more detail: https://web.archive.org/web/20100706052342/http://www.spelli... I'm particularly interested in this one, and I'm curious about how useful something like this would be to use. > The second does not use a dictionary at all (Morris & Cherry 1975). Like the previous method, it divides the text into trigrams, but it creates a table of these,…

I've thought of a variation on that, if you're spell checking a large text file. Create an index of all the unique words in the file, along with how many each appears. Any unique words with a count of 1 are likely to be misspelled.

It'd probably be better to look for words with small Levenshtein distance that look like a case of typo, because people tend to miss-spell some words repeatedly, especially in larger texts.

Re: A spellchecker used to be a major feat of software engineering (2008)

#116

Post by jamil7 »

coopierez wrote: post #109
jamil7 wrote: post #97

Earlier quoted context omitted.

Yeah, to add to it a lot of Germans will use American English but some British English and my spellchecker can't handle the mixture. Off topic but despide how difficult German has been to learn I've found spelling in German to be far more predictable and straightforward than my native language English.

This is at least partly because all of the German speaking countries got together in 1996 to reform the spelling of the language. See here: https://en.wikipedia.org/wiki/German_orthography_reform_of_1...

Right, I vaguely remember hearing about this. Another benefit is you usually have a fairly good idea how a word is pronounced, which is not the case for English or French in my exprience. In some ways I guess American English attempts to standardise and fix some rough edges of English.

Re: A spellchecker used to be a major feat of software engineering (2008)

#117

Post by jiggawatts »

unpopularopp wrote: post #103
jiggawatts wrote: post #39

Earlier quoted context omitted.

To be fair, Hunspell was written specifically for Hungarian because it's such a difficult language to handle efficiently in a spell checker. It just happens to also work for other languages. If you only need English, the complexity of Hunspell is not required. I still haven't found a decent Hungarian spell checker, they get confused by rare words that have the same letters as a very common word, but different accents…

>but not "le es" First of all it's a verb with -ik ending so you can't write "es", the root lexical word is "esik". Second you never use anything like "le es" because generaly you always have to use the verb and prefix together ("leesett", "leesik" etc.) unless you use a commanding form ("ess le") or a modal verb ("le akart esni, "le fog esni" etc.) "le es" is strictly wrong because it's the wrong verb (should be "es…

Okay, I admit, my Hungarian is not so good these days.

But if you're right (seems like you are), then why do spell checkers not pick up on "es" as misspelled?

Re: A spellchecker used to be a major feat of software engineering (2008)

#118

Post by Symbiote »

jamil7 wrote: post #116
coopierez wrote: post #109

Earlier quoted context omitted.

This is at least partly because all of the German speaking countries got together in 1996 to reform the spelling of the language. See here: https://en.wikipedia.org/wiki/German_orthography_reform_of_1...

Right, I vaguely remember hearing about this. Another benefit is you usually have a fairly good idea how a word is pronounced, which is not the case for English or French in my exprience. In some ways I guess American English attempts to standardise and fix some rough edges of English.

It tried, but I'm not sure how much it succeeded.

American English changes a few cases, but there are often exceptions (color, favor, but four, glamour; -ize, but still advertise, compromise etc).

There are too many differences for me to summarize (American and Oxford English spelling there), but I think the problems with English spelling are much deeper than American English can fix.

https://en.wikipedia.org/wiki/American_and_British_English_s...

Re: A spellchecker used to be a major feat of software engineering (2008)

#120

Post by greenkey »

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

That’s awesome!

In comparison, here a quote from the OP’s blog entry:

“Fast forward to today. A program to load /usr/share/dict/words into a hash table is 3-5 lines of Perl or Python, depending on how terse you mind being. Looking up a word in this hash table dictionary is a trivial expression, one built into the language. And that's it. Sure, you could come up with some ways to decrease the load time or reduce the memory footprint, but that's icing and likely won't be needed. The basic implementation is so mindlessly trivial that it could be an exercise for the reader in an early chapter of any Python tutorial.

That's progress.”

But is a simpler, less efficient method progress? Sure it allows more words to be added/removed with ease, and I don’t want to advocate over-optimization, but the solution you made for the Spectrum seems better because words don’t change much. Why don’t we use a similar specialized hash and compressed dictionary format to increase spellchecking speed and allow more words in less space? We could still produce that format using /usr/share/dict/words and similar.

Post reply on HN