Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A spellchecker used to be a major feat of software engineering (2008)

prog21.dadgum.com

Post reply on HN
Print view View original

131–140 of 154 posts

Re: A spellchecker used to be a major feat of software engineering (2008)

#131

Post by possiblelion »

donbrae wrote: post #125
flohofwoe wrote: post #89

...and yet the first thing I switch off in every tool is the spellchecker, because they get utterly confused with the mix of English, German and my local dialect that I'm using when communicating with different people. I'd say doing a spellchecker "right" is apparently still a major feat of software engineering ;)

I often write in a mixture of Scots and English, and spellcheckers are nae guid for that use case. The problem is solved, however, on iOS and Android thanks to SwiftKey[1]. [1] https://dsl.ac.uk/our-publications/scots-predictive-keyboard...

+1 to SwiftKey. I use a mix of English and Estonian and SwiftKey handles it amazingly. Almost never suggests the wrong language when I start typing.

Honestly I can't use the default keyboard in iOS anymore because it lacks multilingual support. How it still isn't a thing in 2020 boggles me.

Re: A spellchecker used to be a major feat of software engineering (2008)

#132

Post by devenblake »

darkwater wrote: post #127
RobAley wrote: post #124

Earlier quoted context omitted.

> Why don’t we Because we don't need to and we have much more interesting problems to take up our time.

But GP already solved the problem (at least for English and other Latin script languages). Why throw away those findings?

Problems tend to have more than one solution. GP's solution should be documented, yes, but the alternate solution that won out was computers being capable of storing a million words or so in plaintext very easily, and doing the same using their compression scheme just isn't really worth the space saved nowadays.

Re: A spellchecker used to be a major feat of software engineering (2008)

#133

Post by jshprentz »

The venerable Ispell program dates back to 1971 with the most recent update in 2015 [1]. Its Contributors files traces earlier spell checkers back to 1959 [2].

[1] https://www.cs.hmc.edu/~geoff/ispell.html

[2] http://web.mit.edu/~mkgray/jik/sipbsrc/src/ispell-3.1/Contri...

Re: A spellchecker used to be a major feat of software engineering (2008)

#134

Post by kiwijamo »

lacker wrote: post #64
jeroenhd wrote: post #50

Earlier quoted context omitted.

While it's not bad, it's not that great a universal tool either. My native language makes use of a lot of compound words and Google's spell checker often gets confused when I combine words according to the standard grammar. I can see and understand the technical limitations, but tools like Microsoft Word seem to do a much better job than Google's spell checker, even in things like Google Docs. Google search will ofte…

Yeah, exactly! - this is why building the perfect spell checker is still a challenging and interesting engineering problem, in 2020. The nature of misspelling varies from language to language, especially with non-letter-based languages like CJK, or with input methods that lead to different sorts of typos than a regular keyboard. Out of curiosity, what is your native language, Dutch? Can you give an example of somethi…

German is another one that seems to match the description.

Re: A spellchecker used to be a major feat of software engineering (2008)

#135

Post by vesinisa »

hoseja wrote: post #121
Aachen wrote: post #111

Earlier quoted context omitted.

What language is that? I guess what I'm wondering is how obscure it has to get (or perhaps how low the GDP of the people speaking a language has to be) before custom spell checking rules aren't considered worth it to bigcorps anymore. Though I'm also kinda interested in what this weird language thing is you're talking about.

Based on the features; Hungarian, Finnish, or Estonian.

Microsoft has an okay Finnish spellchecker. It's really a shame no Finnish university has developed an acceptable open source Finnish spellchecker though AFAIK.

Word-list based checkers actually work for English, which gives an impression that they would work for all types of languages. For agglutinative languages (which I think is about 50% of world languages) not so much, since the list of valid words is basically infinite.

But combined with a high-quality root word list it's possible to write a computer program that returns if a given input word is a valid form of some known root word, i.e. a spellchecker.

In English, you can think the problem same as writing a spellchecker that can tell if an input word is a valid chemical name for a molecule, like 1,3,7-trimethylxanthine (caffeine) or adenosine-5'-triphosphate (ATP). Clearly, we can not write an exhaustive list of valid words here. But it's pretty easy to write a computer program that can tell if the input word at least seems like a valid chemical name devoid of typos.

Re: A spellchecker used to be a major feat of software engineering (2008)

#136

Post by unpopularopp »

jiggawatts wrote: post #117
unpopularopp wrote: post #103

Earlier quoted context omitted.

>but not "le es" First of all it's a verb with -ik ending so you can't write "es", the root lexical word is "esik". Second you never use anything like "le es" because generaly you always have to use the verb and prefix together ("leesett", "leesik" etc.) unless you use a commanding form ("ess le") or a modal verb ("le akart esni, "le fog esni" etc.) "le es" is strictly wrong because it's the wrong verb (should be "es…

Okay, I admit, my Hungarian is not so good these days. But if you're right (seems like you are), then why do spell checkers not pick up on "es" as misspelled?

Most likely because it's used as a suffix too.

Mostly for numerals:

"I arrive with the train at 7" [as in 7 o'clock] = "A 7-es vonattal érkezem"

"M7 motorway" = "M7-es autópálya"

"50m² room" = "50m²-es szoba"

But it's also used with foreign names and full names.

Re: A spellchecker used to be a major feat of software engineering (2008)

#138

Post by jeltz »

Aissen wrote: post #86
Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

If french is one of your language, state of the art has been Druide's Antidote for more than 20 years. And it has had a Linux version available for almost 15 years, too.

It is similar for Swedish. As far as I know nobody has surpassed stava despite it being about 25 years old.

Re: A spellchecker used to be a major feat of software engineering (2008)

#139

Post by [deleted] »

RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

[deleted]

Re: A spellchecker used to be a major feat of software engineering (2008)

#140

Post by ignoramous »

nafey wrote: post #129
RegW wrote: post #110

I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word…

Wouldn't using something like trie be useful here?

You may be interested in any or all of these:

Minimal Acyclic Finite State Automata: http://stevehanov.ca/blog/?id=115 (related: https://web.archive.org/web/20120302104036/http://siganakis....)

Succinct Tries: http://stevehanov.ca/blog/?id=120 (related: https://alexbowe.com/succinct-debruijn-graphs/)

Compression and completion using GPT-2: https://bellard.org/textsynth/index.html (related: https://ed-von-schleck.github.io/shoco/)

Search and compression with Finite State Transducers: http://blog.burntsushi.net/transducers/ (related: https://swtch.com/~rsc/regexp/)

Collection of succinct string representations: https://github.com/simongog/sdsl-lite (related: http://pizzachili.dcc.uchile.cl/)

Search with compressed Radix Trie: https://cr.yp.to/critbit.html related: http://reports-archive.adm.cs.cmu.edu/anon/2020/CMU-CS-20-10...

Post reply on HN