Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write a spelling corrector (2016)

norvig.com

Post reply on HN
Print view View original

21–30 of 87 posts

Re: How to write a spelling corrector (2016)

#21

Post by mjburgess »

graup wrote: post #13

I wish articles about NLP wouldn't assume by default the language to be English. Many points of this article can also apply to other languages, but may require some more thought. Either other languages should be mentioned, or the title should be "How to Write an English Spelling Corrector".

The assumption is about the audience, not the language.

The author is an English speaker, internal to an English community, writing about spell checking.

Its unreasonable to wish every group on the planet to factor in the concerns of every other group, or add their group identifier to everything they write.

Re: How to write a spelling corrector (2016)

#22

Post by Jaruzel »

richardboegli wrote: post #8
laurencei wrote: post #7

I always wish I could just plugin a Google spelling check API into things like Microsoft Word instead of using the inbuilt ones. i.e. the amount of times I cant spell a word, and MS Word has no idea what I'm trying to say, so I "copy and paste" the wrong word into Google and it instantly knows what I meant and shows me the correct spelling. I appreciate part of that is probably related to my previous searches on Goog…

You could make a very simple AutoHotkey script to do this.

...or a Word macro that you assign to a toolbar button, that takes the currently selected word and 'googles' it for the correction.

Re: How to write a spelling corrector (2016)

#24

Post by mannykannot »

graup wrote: post #13

I wish articles about NLP wouldn't assume by default the language to be English. Many points of this article can also apply to other languages, but may require some more thought. Either other languages should be mentioned, or the title should be "How to Write an English Spelling Corrector".

As this is an issue that concerns you, perhaps you could take a fraction of the time the author spent on this article to share some insights on the problems specific to other languages. I, for one, would be interested to read about them.

Re: How to write a spelling corrector (2016)

#25

Post by shdon »

Spelling correction rather than spell checking and suggestions really bugs me. As soon as you have a somewhat international audience (and given that this is the internet - you probably do) or have any non-English content, automatic correction can be an actively user-hostile "feature".

Especially on search engines, sometimes I am searching for words in another language, mixed-language results, or even doing an intentional search for misspelt words. In my native Dutch his first example would already cause problems: "spelling" means the same as in English, but "speling" means margin/give (the noun) and I wouldn't want one to be auto-corrected to the other. The fuzzy search for names on Facebook makes the search pretty much worthless especially in case you have an unusual variant of a common name in your search - you can be sure the actual person you're looking for is listed after half a billion bogus results.

The whole trend of second-guessing the user and saying "you tried to do A, but we assume you meant B, so we're gonna do B instead" is one of my biggest pet peeves.

Re: How to write a spelling corrector (2016)

#26

Post by crispyporkbites »

I wrote a spell checker for my undergraduate to help with automation research back in 2011, it allowed the researcher to do things like insert words that would show as incorrect when they were actually correct, and see if people would notice with varying levels of difficulty. The idea was to try to measure how some automated systems can improve or reduce net performance.

I actively tried to avoid using libraries, and ended up writing most of it in vanilla JavaScript - including a bunch of DOM manipulations, which was suicide at the time. I even wrote my own XML-based DSL for defining the input text.

Would have loved to actually follow through with the research but phd's don't pay bills.

Re: How to write a spelling corrector (2016)

#27

Post by iask »

Thanks for sharing! I just started researching how to analyze text for a project. I am tasked with analyzing content to see if they match my client products or something similar. Matching my the product name is easy, but matching by similar is what’s throwing me off.

Is there an api service for these type of text analysis?

Re: How to write a spelling corrector (2016)

#29

Post by sp332 »

A couple of years ago, Emily Short made an "interactive fiction" game [that's what they call text adventures these days] with letter-removal as a major mechanic. It's called Counterfeit Monkey and it's a lot of fun. http://emshort.com/counterfeit_monkey/

Install one of the interpreters at the bottom of the page, then download the "story" file and open it in said interpreter.

Re: How to write a spelling corrector (2016)

#30

Post by globuous »

shdon wrote: post #25

Spelling correction rather than spell checking and suggestions really bugs me. As soon as you have a somewhat international audience (and given that this is the internet - you probably do) or have any non-English content, automatic correction can be an actively user-hostile "feature". Especially on search engines, sometimes I am searching for words in another language, mixed-language results, or even doing an intenti…

Agreed !! Actually, it's a real problem that's not limited to spelling. Look at Siri for instance, granted I haven't used it in a few years, back in the days it was impossible to use for most of my contacts. Indeed, I have both English and French contacts and it's a nightmare to use with Siri.

My Siri is in English because I figured it was more reliable for everyday us then in French. But my mom's contact is as 'maman', can't use Siri to call her. Or I have to try and pronounce 'maman' with an english accent and hope Siri know what I'm talking about. And that's an easy one, try a name like 'Jean Brentois'.... But I assume setting up Siri in French will lead to the same problem, but with my non French contacts now. I have not tried however.

So obviously I get the challenge, but it makes it very hard to use (I can't use Siri like tools in parts because of this). Especially since so many of my contacts are international.

EDIT: a quick fix for contact pronounciation might be to ask the user how he pronounces a particular contact when adding it.

Similarly, why not allow for a special syntax allowing for multiple languages in a sentence ? Say my first language is English, my second is French, and my third is Spanish. Why not something like that: "Let's go to the $$casa $demain $avec $toi, what do you think ?"

Post reply on HN