A spellchecker used to be a major feat of software engineering (2008) • Hacker News Board

Re: A spellchecker used to be a major feat of software engineering (2008)

#121

Post by hoseja » Fri, Dec 04, 2020, 12:25 PM UTC

Aachen wrote: post #111

Avamander wrote: post #92

Earlier quoted context omitted.

I wish it just struggled with the mix of multiple languages, it can't even handle my native one. A plural compound word in translative case with a clitic? I can only dream about it, it can't even do all the cases reliably.

What language is that? I guess what I'm wondering is how obscure it has to get (or perhaps how low the GDP of the people speaking a language has to be) before custom spell checking rules aren't considered worth it to bigcorps anymore. Though I'm also kinda interested in what this weird language thing is you're talking about.

Based on the features; Hungarian, Finnish, or Estonian.

Re: A spellchecker used to be a major feat of software engineering (2008)

#122

Post by warpech » Fri, Dec 04, 2020, 12:29 PM UTC

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

Thanks for this nice detailed memory!

How did you get the word list from Collins? Did they license it in a digital form?

Re: A spellchecker used to be a major feat of software engineering (2008)

#123

Post by noneeeed » Fri, Dec 04, 2020, 12:31 PM UTC

My spelling has always been bad. I'm not dyslexic, but I did get extra help as a kid. In theory spellcheckers should be great for me. But in reality I seem to make spelling mistakes that leave some checkers unable to make useful suggestions. I can be just one or two vowel substitutions away from the correct spelling, but they will fail to suggest the right answer.

I should start keeping notes when that happens as I'd really like to understand what I'm doing that causes them issues, it might help me when I'm struggling to spell a word and get stuck.

Re: A spellchecker used to be a major feat of software engineering (2008)

#124

Post by RobAley » Fri, Dec 04, 2020, 12:47 PM UTC

greenkey wrote: post #120

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

That’s awesome! In comparison, here a quote from the OP’s blog entry: “Fast forward to today. A program to load /usr/share/dict/words into a hash table is 3-5 lines of Perl or Python, depending on how terse you mind being. Looking up a word in this hash table dictionary is a trivial expression, one built into the language. And that's it. Sure, you could come up with some ways to decrease the load time or reduce the m…

> Why don’t we

Because we don't need to and we have much more interesting problems to take up our time.

Re: A spellchecker used to be a major feat of software engineering (2008)

#125

Post by donbrae » Fri, Dec 04, 2020, 12:50 PM UTC

flohofwoe wrote: post #89

...and yet the first thing I switch off in every tool is the spellchecker, because they get utterly confused with the mix of English, German and my local dialect that I'm using when communicating with different people. I'd say doing a spellchecker "right" is apparently still a major feat of software engineering ;)

I often write in a mixture of Scots and English, and spellcheckers are nae guid for that use case. The problem is solved, however, on iOS and Android thanks to SwiftKey[1].

[1] https://dsl.ac.uk/our-publications/scots-predictive-keyboard...

Re: A spellchecker used to be a major feat of software engineering (2008)

#126

Post by nenadst » Fri, Dec 04, 2020, 12:54 PM UTC

greenkey wrote: post #120

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

That’s awesome! In comparison, here a quote from the OP’s blog entry: “Fast forward to today. A program to load /usr/share/dict/words into a hash table is 3-5 lines of Perl or Python, depending on how terse you mind being. Looking up a word in this hash table dictionary is a trivial expression, one built into the language. And that's it. Sure, you could come up with some ways to decrease the load time or reduce the m…

I always thought that we still use a trie or (to save memory) ternary search trees for that..

Re: A spellchecker used to be a major feat of software engineering (2008)

#127

Post by darkwater » Fri, Dec 04, 2020, 1:03 PM UTC

RobAley wrote: post #124

greenkey wrote: post #120

Earlier quoted context omitted.

That’s awesome! In comparison, here a quote from the OP’s blog entry: “Fast forward to today. A program to load /usr/share/dict/words into a hash table is 3-5 lines of Perl or Python, depending on how terse you mind being. Looking up a word in this hash table dictionary is a trivial expression, one built into the language. And that's it. Sure, you could come up with some ways to decrease the load time or reduce the m…

> Why don’t we Because we don't need to and we have much more interesting problems to take up our time.

But GP already solved the problem (at least for English and other Latin script languages). Why throw away those findings?

Re: A spellchecker used to be a major feat of software engineering (2008)

#128

Post by greenkey » Fri, Dec 04, 2020, 1:12 PM UTC

nenadst wrote: post #126

greenkey wrote: post #120

Earlier quoted context omitted.

That’s awesome! In comparison, here a quote from the OP’s blog entry: “Fast forward to today. A program to load /usr/share/dict/words into a hash table is 3-5 lines of Perl or Python, depending on how terse you mind being. Looking up a word in this hash table dictionary is a trivial expression, one built into the language. And that's it. Sure, you could come up with some ways to decrease the load time or reduce the m…

I always thought that we still use a trie or (to save memory) ternary search trees for that..

How many operations and objects? The method he’s talking about would seem more efficient for the purpose vs all of the strings still being created even if never used in the plain hash version.

Re: A spellchecker used to be a major feat of software engineering (2008)

#129

Post by nafey » Fri, Dec 04, 2020, 1:27 PM UTC

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

Wouldn't using something like trie be useful here?

Re: A spellchecker used to be a major feat of software engineering (2008)

#130

Post by throw93 » Fri, Dec 04, 2020, 1:42 PM UTC

I was recently tasked with finding alternative to our age old spellchecker. Main criteria was it should be a separate api in cloud(we are not completely in cloud yet). Also I couldn’t use any commercial cloud spell checkers because of data privacy and security. After putting up a demo with an open source spellchecker we ultimately decided to abandon the effort. We found out that most of our competitors were recommending their users to use browser provided spellchecker like the one provided by chrome. We followed the same suite because we felt that being a relatively small development team our time was best spent writing business logic than to to create and maintain a spellcheck service.