A spellchecker used to be a major feat of software engineering (2008) • Hacker News Board

Re: A spellchecker used to be a major feat of software engineering (2008)

#151

Post by ASalazarMX » Fri, Dec 04, 2020, 10:48 PM UTC

devenblake wrote: post #132

darkwater wrote: post #127

Earlier quoted context omitted.

But GP already solved the problem (at least for English and other Latin script languages). Why throw away those findings?

Problems tend to have more than one solution. GP's solution should be documented, yes, but the alternate solution that won out was computers being capable of storing a million words or so in plaintext very easily, and doing the same using their compression scheme just isn't really worth the space saved nowadays.

Also, compressing could actually be slower for modern computers. Remember when compressing your hard disk made your PC faster, up until disks became faster, then it actually made it slower?

Today's CPUs are very fast, so the trend could have flipped again, that would be an interesting benchmark.

Re: A spellchecker used to be a major feat of software engineering (2008)

#152

Post by anthk » Sat, Dec 05, 2020, 2:56 AM UTC

vbezhenar wrote: post #90

Const-me wrote: post #15

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

I would argue that it's not a hard engineering problem. It's just a problem that should be solved. Linguists made a lot of formalizations about language. There are rules, there are known lists of exceptions, there are dictionaries. You just have to implement those things carefully with lots of edge-cases, implement it for every language, but it's just mundane engineering work, nothing that requires breakthrough. Curr…

>You just have to implement those things carefully with lots of edge-cases,

Say hello to homophones, ambiguous phrases and double meaning.

In Spanish a lot of verbs share the singular first and third persons. "Estaba" can mean "I was" or "He/she was.". Or "fuimos", (we were); but also "we went".

Re: A spellchecker used to be a major feat of software engineering (2008)

#153

Post by garfieldnate » Mon, Dec 07, 2020, 4:40 PM UTC

This dictionary listing thing might work okay for a language with little morphology like English, but surely something smarter is required for Finnish, Turkish, etc.; Sometimes you can have a million word forms for one base, so you can't really just put them all into a file.

Re: A spellchecker used to be a major feat of software engineering (2008)

#154

Post by gnramires » Thu, Dec 10, 2020, 1:02 PM UTC

meetups323 wrote: post #60

jiggawatts wrote: post #43

Earlier quoted context omitted.

I often make this mistake in technical documentation: "the database sever was updated". However, a surgeon might want to write "the next procedure is to sever the artery." I wonder if GPT-3 could be used to determine the "context" and determine the spelling correctness "weights"?

Yes. Up until last two sections is prompt, then I let GPT-3 fill in the responses. Non-standard English: Please provide me with a short brief of the design youre looking four and some examples or previous projects youve done would be helpful. Standard American English: Please provide me with a short brief of the design you’re looking for and some examples or previous projects you’ve done would be helpful. Non-standar…

Indeed, I think most linguistic tasks could benefit from the neural (transformer) approach, even a tiny spellchecker. The combination of semi-regular rules, semantics and logic makes a highly effective "manual" solution too expensive to obtain (in terms of research and programming time).