Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data-Mining Wikipedia for Fun and Profit

billpg.com

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Data-Mining Wikipedia for Fun and Profit

#71

Post by bawolff »

jancsika wrote: post #64
squeaky-clean wrote: post #46

Earlier quoted context omitted.

The infoboxes aren't standardized at all. The HTML they generate is.

Hehe-- I am going to rankly speculate nearly all of them follow an obvious standard of key/value pairs where the key is a string. And then there are like two or three subcultures on Wikipedia that put rando stuff in there and would troll to the death before being forced to change the their infobox class to "rando_box" or whatever negligible effort it would take them if a standard were to be enforced. Am I anywhere cl…

I think you'll have to more clearly define what you mean by "key-value" pairs.

Re: Data-Mining Wikipedia for Fun and Profit

#72

Post by bawolff »

learc83 wrote: post #20
traceroute66 wrote: post #16

Earlier quoted context omitted.

IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every…

>IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. They said please don't, not don't do it or they'll sue you. But content license and site terms of use are different things. From their terms of use you aren’t allowed to > [Disrupt] the services by placing an undue bur…

Yes, but they aren't going to care for just 2400 pages.

As a general rule, make your scraper non-paralell, and put a user-agent that has contact details in the event of an issue, and you're probably all good.

After all wikipedia is meant to be used. Don't be unduly disruptive, don't scrape 20 million pages, but scraping a couple thousand is totally acceptable.

Source: used to work for wikimedia, albeit not in the sre dept. My opinions are of course totally my own.

Re: Data-Mining Wikipedia for Fun and Profit

#73

Post by bawolff »

kemayo wrote: post #70
ahP7Deew wrote: post #51

Has anyone found an easy way to expand their templates without using their whole stack? I tried getting Lua templates working from Python but didn't get very far...

Parsoid[1] is what you'd want for that, most likely. It's the new wikitext parser that MediaWiki is gradually switching over to, but it has the virtue of being usable entirely outside of MediaWiki if you need to. [1]: https://www.mediawiki.org/wiki/Parsoid

Yes, but it implements lua by calling into the old stack, which doesn't really solve OPs problem.

Re: Data-Mining Wikipedia for Fun and Profit

#74

Post by learc83 »

bawolff wrote: post #72
learc83 wrote: post #20

Earlier quoted context omitted.

>IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. They said please don't, not don't do it or they'll sue you. But content license and site terms of use are different things. From their terms of use you aren’t allowed to > [Disrupt] the services by placing an undue bur…

Yes, but they aren't going to care for just 2400 pages. As a general rule, make your scraper non-paralell, and put a user-agent that has contact details in the event of an issue, and you're probably all good. After all wikipedia is meant to be used. Don't be unduly disruptive, don't scrape 20 million pages, but scraping a couple thousand is totally acceptable. Source: used to work for wikimedia, albeit not in the sre…

I don’t think the op was talking specifically to the content author, but to all the people who read the article and get the idea to scrape Wikipedia.

Re: Data-Mining Wikipedia for Fun and Profit

#75

Post by bawolff »

permo-w wrote: post #59

>The graph was interesting but this wasn’t the primary objective of this exercise. I wanted to write “He is the n-times great-father of his current successor Queen Elizabeth.” on King Alfred’s Wikipedia page. Wouldn’t that contravene Wikipedia’s rules on original research?

I also think its of questionable relavence to the topic at hand. Just because a fact about something is true doesn't mean it should be in the article on the topic.

But yes, citing your own blog isn't a valid source, and neither is citing wikipedia - https://en.wikipedia.org/wiki/Wikipedia:Verifiability#Self-p...

Re: Data-Mining Wikipedia for Fun and Profit

#76

Post by Mathnerd314 »

permo-w wrote: post #59

>The graph was interesting but this wasn’t the primary objective of this exercise. I wanted to write “He is the n-times great-father of his current successor Queen Elizabeth.” on King Alfred’s Wikipedia page. Wouldn’t that contravene Wikipedia’s rules on original research?

It's not original research, more like synthesis. Reading the policy he's probably OK under the routine calculation exception.

The edit got reverted because there was no clear criteria for mentioning Queen Elizabeth III but not the other descendants. If he made an info graphic or info box with a clear inclusion criteria and stuck it in the Alfred article it would probably stick.

Re: Data-Mining Wikipedia for Fun and Profit

#77

Post by bawolff »

learc83 wrote: post #74
bawolff wrote: post #72

Earlier quoted context omitted.

Yes, but they aren't going to care for just 2400 pages. As a general rule, make your scraper non-paralell, and put a user-agent that has contact details in the event of an issue, and you're probably all good. After all wikipedia is meant to be used. Don't be unduly disruptive, don't scrape 20 million pages, but scraping a couple thousand is totally acceptable. Source: used to work for wikimedia, albeit not in the sre…

I don’t think the op was talking specifically to the content author, but to all the people who read the article and get the idea to scrape Wikipedia.

Honestly i'd rather people err on the side of scrapping wikipedia too much than live in fear of being disruptive and not do cool things as a result. Wikipedia is meant to be used to spread knowledge. That includes data mining projects such as the one in this blog.

(Before anyone takes this out of context - no im not saying its ok to be intentionally disruptive, or do things without exercising any care at all. Also always set a unique descriptive user-agent with an email address if you're doing anything automated on wikipedia).

Re: Data-Mining Wikipedia for Fun and Profit

#78

Post by k12sosse »

billpg wrote: post #37
lizhang wrote: post #14

Where's the profit?

For three minutes, something I wrote was cited on Wikipedia.

Ah yes, every student in my district had this distinction too, until they lifetime banned our egress IPs over it.

The person who did the revert seems like a bundle of joy, though. Congrats on your efforts.

Re: Data-Mining Wikipedia for Fun and Profit

#79

Post by bawolff »

powera wrote: post #4
papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

Definitely. There's even a public query service ( https://query.wikidata.org/ ) which can do a lot of this (though SQL is not good with searching for chains).

Note: that's the sparql endpoint. Much better at searching chains than SQL (but easy to make slow queries that timeout).

The SQL endpoint is at https://quarry.wmflabs.org/ however it doesn't have the actual data so much as metadata (mostly) so its not super useful.

Re: Data-Mining Wikipedia for Fun and Profit

#80

Post by kemayo »

bawolff wrote: post #73
kemayo wrote: post #70

Earlier quoted context omitted.

Parsoid[1] is what you'd want for that, most likely. It's the new wikitext parser that MediaWiki is gradually switching over to, but it has the virtue of being usable entirely outside of MediaWiki if you need to. [1]: https://www.mediawiki.org/wiki/Parsoid

Yes, but it implements lua by calling into the old stack, which doesn't really solve OPs problem.

Okay, yeah, I guess that Lua modules in particular are going to make everything fall down.
Post reply on HN