Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 100K sentences mined from Wikipedia to help non-native English learners

buildmyvocab.in

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#71

Post by carlmungz »

camillomiller wrote: post #30
miles wrote: post #13

This list is to help non-native English learners? Many native English speakers might have trouble with a few of these: abeyance, abscission, accretion, amalgamate, anodyne, antediluvian, apposite, arabesque, atavism, and avuncular.

As an Italian native with a classical studies background, this kind of words are easy for me. They're almost all Latin-derived and they usually sound very similar to the Italian equivalent. You wanna know what's hard for us? The street talk. You know like when you shoot the breeze before you really spill the beans about your shenanigans while riding shotgun on a friend's old jalopy.

> You know like when you shoot the breeze before you really spill the beans about your shenanigans while riding shotgun on a friend's old jalopy.

That sounds like quite the night out...

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#72

Post by [deleted] »

sametmax wrote: post #62
wodenokoto wrote: post #3

How do you decide on which sentences to use? I'm interested in generating example sentences myself, but in a way, that chooses sentences that are simple, easy to understand and support the word, they are supposed to exemplify. For example "She got a car for her birthday, while she was traveling in Italy eating pizza" does not tell the reader anything about what a car is, or how the word should be used. However "He dr…

That's the geek inside you talking. But the hard part was the word stats. Now that you get 1K words, writing a thousand sentences manually to illustrate them is not really hard. It's a one day manual work. Less than the work needing to figure out automation, with way better results.

[deleted]

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#74

Post by mijoharas »

SilasX wrote: post #60
schoen wrote: post #56

Earlier quoted context omitted.

I have this book and I can tell you that it does use illustrations quite a bit (although most vocabulary is ultimately probably defined using other words). The very first chapter presents a labeled map of the Mediterranean region and begins: "Rōma in Italiā est. Italia in Eurōpā est. Graecia in Eurōpā est. Italia et Graecia in Eurōpā sunt. Hispānia quoque in Eurōpā est. Hispānia et Italia et Graecia in Eurōpā sunt. A…

I want that, for every language I want to learn! Edit: Before anyone says it, in my original comment, that should have been an e.g. instead of an i.e.

Thanks to you I just learnt the difference between those two. Thanks! (Also writing this comment reminded me of this https://xkcd.com/1053/ )

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#75

Post by csa »

nandemo wrote: post #34
cschmidt wrote: post #4

I went to an interesting talk once at the Boston Python meetup, where a guy figured out how to order sentences so could learn them in an order where you already knew the "other" words in the sentence. Basically, making a directed graph of vocabulary. He was doing it to learn Latin, but you could do it for any language.

How about this? 1. Get a frequency list. The most common word's rank is 1, the second is 2, etc. [0] 2. Then use your favorite Spaced Repetition Software (such as anki) to learn the words in that order. 3. Define a sentence's difficulty as the maximum rank over all its words. You could refine it by adding tie-breakers but I think it doesn't matter. Then sort the sentences in order of difficulty. [0] See https://en.wi…

Great idea but...

The word "run" is a relatively high frequency word. How many different meanings for "run" does a learner need to know? Is that in isolation? With collocations? As phrasal verbs?

In many languages, the most frequently appearing words also have the most varied meanings. Interestingly, many highly vernacular languages also use relatively few words, but those words have a lot of meanings that are clearly known by the speech communities.

FWIW, some theoretical linguists consider this a non-issue. People in the field (i.e., people might die if I get this wrong) know otherwise.

While your idea sounds nice in principle, I hope you will accept the idea that reality may be slightly more complex.

* Collins Cobuild has 50+ meanings for "run" if phrasal verbs are included. Most non-native speakers are not even aware of the potential breadth of meanings it offers.

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#76

Post by [deleted] »

nandemo wrote: post #34
cschmidt wrote: post #4

I went to an interesting talk once at the Boston Python meetup, where a guy figured out how to order sentences so could learn them in an order where you already knew the "other" words in the sentence. Basically, making a directed graph of vocabulary. He was doing it to learn Latin, but you could do it for any language.

How about this? 1. Get a frequency list. The most common word's rank is 1, the second is 2, etc. [0] 2. Then use your favorite Spaced Repetition Software (such as anki) to learn the words in that order. 3. Define a sentence's difficulty as the maximum rank over all its words. You could refine it by adding tie-breakers but I think it doesn't matter. Then sort the sentences in order of difficulty. [0] See https://en.wi…

[deleted]

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#77

Post by [deleted] »

nandemo wrote: post #34
cschmidt wrote: post #4

I went to an interesting talk once at the Boston Python meetup, where a guy figured out how to order sentences so could learn them in an order where you already knew the "other" words in the sentence. Basically, making a directed graph of vocabulary. He was doing it to learn Latin, but you could do it for any language.

How about this? 1. Get a frequency list. The most common word's rank is 1, the second is 2, etc. [0] 2. Then use your favorite Spaced Repetition Software (such as anki) to learn the words in that order. 3. Define a sentence's difficulty as the maximum rank over all its words. You could refine it by adding tie-breakers but I think it doesn't matter. Then sort the sentences in order of difficulty. [0] See https://en.wi…

[deleted]

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#78

Post by PeterisP »

wodenokoto wrote: post #3

How do you decide on which sentences to use? I'm interested in generating example sentences myself, but in a way, that chooses sentences that are simple, easy to understand and support the word, they are supposed to exemplify. For example "She got a car for her birthday, while she was traveling in Italy eating pizza" does not tell the reader anything about what a car is, or how the word should be used. However "He dr…

There's the linguistic principle of "You shall know a word by the company it keeps", so for any particular word you can identify which other words are most specifically related to that word, the simplest measure that can be used for that is freq(both words together)/freq(that other word in general).

That would allow you to prioritize sentences containing "getting a car" over "driving a car" - even if "getting a car" is more frequent, driving is more specific according to such a measure.

Re: Show HN: 100K sentences mined from Wikipedia to help non-native English learners

#80

Post by unhammer »

microcolonel wrote: post #59
nandemo wrote: post #34

Earlier quoted context omitted.

How about this? 1. Get a frequency list. The most common word's rank is 1, the second is 2, etc. [0] 2. Then use your favorite Spaced Repetition Software (such as anki) to learn the words in that order. 3. Define a sentence's difficulty as the maximum rank over all its words. You could refine it by adding tie-breakers but I think it doesn't matter. Then sort the sentences in order of difficulty. [0] See https://en.wi…

The blessing of language is that word frequency follows a power law. The first couple hundred words cover much of the language anyway.

This is https://en.wikipedia.org/wiki/Zipf%27s_law

The other side of the coin is that the long tail is long:

the frequency of any word in a corpus is inversely proportional to its rank in the frequency table. For large corpora, about 40% to 60% of the words are hapax legomena [words appearing exactly once], and another 10% to 15% are dis legomena[words appearing exactly twice]. Thus, in the Brown Corpus of American English, about half of the 50,000 words are hapax legomena within that corpus. https://en.wikipedia.org/wiki/Hapax_legomenon

So the chance of seeing a hapax in any given sentence is really high.

What's worse, the more rare words in a sentence are often the important content words. When I try to decipher sentences in some language I don't know much of, what I end up understanding is often something like "And then he XXXXX-ed the YYYYY just like that hahaha" (I understood 8/10 words! And I even know that word 4 is a past tense verb! But not at all the meaning).

(Not that you shouldn't study the most frequent first, that's still a good rule.)

Post reply on HN