Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 100K sentences mined from Wikipedia to help non-native English learners

buildmyvocab.in

Post reply on HN
Print view View original

91–95 of 95 posts

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#91

Post by [deleted] »

abhas9 wrote: post #10
wodenokoto wrote: post #3

How do you decide on which sentences to use? I'm interested in generating example sentences myself, but in a way, that chooses sentences that are simple, easy to understand and support the word, they are supposed to exemplify. For example "She got a car for her birthday, while she was traveling in Italy eating pizza" does not tell the reader anything about what a car is, or how the word should be used. However "He dr…

That's a great question. Optimizing for sentence selection is important for teaching. For now, I have a simple check that filters out sentences which are longer than 160 characters. Also, I believe that this is one thing which humans can do better. I, therefore, plan to add upvote & downvote buttons to rate the quality of sentences.

[deleted]

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#93

Post by nandemo »

unhammer wrote: post #80
microcolonel wrote: post #59

Earlier quoted context omitted.

The blessing of language is that word frequency follows a power law. The first couple hundred words cover much of the language anyway.

This is https://en.wikipedia.org/wiki/Zipf%27s_law The other side of the coin is that the long tail is long : the frequency of any word in a corpus is inversely proportional to its rank in the frequency table. For large corpora, about 40% to 60% of the words are hapax legomena [words appearing exactly once], and another 10% to 15% are dis legomena[words appearing exactly twice]. Thus, in the Brown Corpus of American…

> (I understood 8/10 words! And I even know that word 4 is a past tense verb! But not at all the meaning).

That's an important point. Knowing 80% of the words in a text doesn't mean you understand 80% of its meaning, that is, you probably wouldn't get high marks on a basic comprehension test.

I've seen some studies that indicate you need to know at least (around) 95% of the words in a text in order to understand it "enough" of it. (I don't have the links right now but could look it up at home if you're interested).

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#94

Post by wodenokoto »

PeterisP wrote: post #78
wodenokoto wrote: post #3

How do you decide on which sentences to use? I'm interested in generating example sentences myself, but in a way, that chooses sentences that are simple, easy to understand and support the word, they are supposed to exemplify. For example "She got a car for her birthday, while she was traveling in Italy eating pizza" does not tell the reader anything about what a car is, or how the word should be used. However "He dr…

There's the linguistic principle of "You shall know a word by the company it keeps", so for any particular word you can identify which other words are most specifically related to that word, the simplest measure that can be used for that is freq(both words together)/freq(that other word in general). That would allow you to prioritize sentences containing "getting a car" over "driving a car" - even if "getting a car"…

Hmm, maybe I've been overcomplicating the problem in my mind. You've given me some good ideas.

Bigrams, as your own example shows, are too simple: in both examples, "car" will get related to "a", instead of "getting" and "driving".

Maybe if I parse all sentences with dependency and built dependency bigrams, and score sentences with frequency/inverse_freq and length of sentence (short sentences are better).

Post reply on HN