Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Text Correcter

atpaino.com

Post reply on HN
Print view View original

1–10 of 71 posts

Re: Deep Text Correcter

#3

Post by hbornfree »

This is terrific! Very coincidentally, we were thinking of implementing a sentence de-noiser using sequence-to-sequence models only today evening. I work in the NLP domain writing Machine Translation systems. But NLP parsers are accurate for grammatically correct sentences only which necessitates the need for something like deep text correcter. Thank you for this. Will try this out this week and let you know how it goes.

Re: Deep Text Correcter

#5

Post by daveytea »

This is really cool. If you're looking for more datasets to train your model, here are a few relevant ones: - https://archive.org/details/stackexchange - http://trec.nist.gov/data/qamain.html - http://opus.lingfil.uu.se/OpenSubtitles2016.php - http://corpus.byu.edu/full-text/wikipedia.asp OR https://en.wikipedia.org/wiki/Wikipedia:Database_download#En... - http://opus.lingfil.uu.se/

I'd love to see how good your model gets.

Re: Deep Text Correcter

#6

Post by stephanheijl »

Looks like a cool project, I would love to see this as a browser plugin of some sort. As for the corpus, I suspect that using articles from Wikipedia would be appropriate. Especially large articles are routinely checked and cleaned up. It has the added benefit of being available in multiple languages.

(https://en.wikipedia.org/wiki/Wikipedia:Database_download)

EDIT: I see this has already been suggested, along with a large amount of other source in another comment by daveytea.

Re: Deep Text Correcter

#7

Post by jmiserez »

Interesting idea. I went ahead and tested:

> Alex went to the kitchen to store the milk in the fridge.

Corrected:

> Alex went to the kitchen to the store the milk in the fridge.

Gathering a large, high quality dataset from the internet is probably not so easy. A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Movie dialogues don't necessarily consist of grammatically correct sentences like the ones you'd write in a letter. Perhaps there is some public domain contemporary literature available that could be used instead or alongside the dialogues?

EDIT: Unrelated to this project, I have this general fear of language recommendation tools trained on just low-quality comments or emails. A simple thesaurus and a grammar-checker are often enough to find the right words when writing. But a tool that could understand my intent and then propose restructured or similar sentences and words that convey the same meaning could be a true killer application.

Post reply on HN