Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Text Correcter

atpaino.com

Post reply on HN
Print view View original

11–20 of 71 posts

Re: Deep Text Correcter

#11

Post by babuskov »

jmiserez wrote: post #7

Interesting idea. I went ahead and tested: > Alex went to the kitchen to store the milk in the fridge. Corrected: > Alex went to the kitchen to the store the milk in the fridge. Gathering a large, high quality dataset from the internet is probably not so easy. A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Movie dialogues don…

> A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself).

Corrected:

> A lot of the content on the HN/Reddit/forums is of the quality grammatically and non-native written by written English speakers (such myself). as UNK

Yeah. It's got a long way to go. No idea where "as UNK" came from.

Re: Deep Text Correcter

#12

Post by atpaino »

daveytea wrote: post #5

This is really cool. If you're looking for more datasets to train your model, here are a few relevant ones: - https://archive.org/details/stackexchange - http://trec.nist.gov/data/qamain.html - http://opus.lingfil.uu.se/OpenSubtitles2016.php - http://corpus.byu.edu/full-text/wikipedia.asp OR https://en.wikipedia.org/wiki/Wikipedia:Database_download#En... - http://opus.lingfil.uu.se/ I'd love to see how good your mode…

Thanks for the links! I'll have to try out some of these. The data is definitely the limiting factor at this point.

Re: Deep Text Correcter

#13

Post by tannerc »

jmiserez wrote: post #7

Interesting idea. I went ahead and tested: > Alex went to the kitchen to store the milk in the fridge. Corrected: > Alex went to the kitchen to the store the milk in the fridge. Gathering a large, high quality dataset from the internet is probably not so easy. A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Movie dialogues don…

> A tool that could understand my intent and then propose structure or similar sentences and words

I'm working on this exact thing right now fwiw, in my app called Prompts. I imagine machine assisted apps are going to percolate up over the coming years, or months, in a way we haven't seen yet. It's pretty exciting, if we can get it right. Written language seems like a decent place to start.

Re: Deep Text Correcter

#14

Post by smoyer »

'Kvothe went to the market'

Off-topic but I've been waiting for the third and final book in the trilogy for a long time ... I've come to the conclusion that Rothfuss can't find a way to tie all the plot threads together.

I'm also wondering if anyone else thinks Rothfuss looks like Longfellow in a lot of his publicity shots.

Re: Deep Text Correcter

#15

Post by atpaino »

jmiserez wrote: post #7

Interesting idea. I went ahead and tested: > Alex went to the kitchen to store the milk in the fridge. Corrected: > Alex went to the kitchen to the store the milk in the fridge. Gathering a large, high quality dataset from the internet is probably not so easy. A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Movie dialogues don…

Yep, it definitely has room to improve. The work thus far has primarily been a proof-of-concept for the methodology used to generate training samples (i.e. starting with grammatically correct text and introducing errors). Next step is to try to include more high quality data, after which I may try out comment data from HN, etc. I think it would be interesting to see what the effect of somewhat noisier data like that would have on the model.

Re: Deep Text Correcter

#16

Post by wiz21c »

topynate wrote: post #8

> Unfortunately, I am not aware of any publicly available dataset of (mostly) grammatically correct English. How about books?

Guess who's got that massive amount of data nowadays ? Google, facebook and co. So good luck if you want to work on AI stuff...

Re: Deep Text Correcter

#17

Post by aembleton »

jmiserez wrote: post #7

Interesting idea. I went ahead and tested: > Alex went to the kitchen to store the milk in the fridge. Corrected: > Alex went to the kitchen to the store the milk in the fridge. Gathering a large, high quality dataset from the internet is probably not so easy. A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Movie dialogues don…

Dropping `in the fridge` works.

"Alex went to kitchen to store the milk" corrects to "Alex went to the kitchen to store the milk"

Re: Deep Text Correcter

#19

Post by mortehu »

wiz21c wrote: post #16
topynate wrote: post #8

> Unfortunately, I am not aware of any publicly available dataset of (mostly) grammatically correct English. How about books?

Guess who's got that massive amount of data nowadays ? Google, facebook and co. So good luck if you want to work on AI stuff...

And Project Gutenberg: https://www.gutenberg.org/

The language in many of those books may be a bit archaic, though.

See also Distributed Proofreaders: http://www.pgdp.net/c/

Re: Deep Text Correcter

#20

Post by UhUhUhUh »

Re. intent but with regards to spelling. I often wonder if there could be rules to correct errors due to key strokes in the immediate vicinity of the intended letter (e.g. "keu" vs. "key"). Would check combination of the surrounding letters, first oin (that's an unintended addition here) horizontal axis. That happens to me all the time, probably because I'm not a good typist but still.
Post reply on HN