For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
It’s so sad that almost nobody knows or uses SPARQL…
Why should more people know SPARQL?
Data-Mining Wikipedia for Fun and Profit
21–30 of 100 posts
Re: Data-Mining Wikipedia for Fun and Profit
#22Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps.
The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them).
The rendered HTML, on the other hand, is designed for humans, and so what you see is what you get.
Re: Data-Mining Wikipedia for Fun and Profit
#23Earlier quoted context omitted.
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
(Author of original article here.)
That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
Re: Data-Mining Wikipedia for Fun and Profit
#24Earlier quoted context omitted.
That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
(Author of original article here.) That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
The HTML is more volatile and subject to change than other sources though
Re: Data-Mining Wikipedia for Fun and Profit
#25Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every…
"Who's gonna stop me" is kind of a crappy attitude to take with a cooperative project like Wikipedia.
I mean, sure, you can do a lot of things you shouldn't with freely available services. There's even an economics term that describes this: the Tragedy of the Commons.
Individual fish poachers' hauls are also, individually, de-minimis.
Re: Data-Mining Wikipedia for Fun and Profit
#26Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
Unlike APIs, html class/tag names or whatever provide no stability guarantees. The site owner can break your parser whenever they want for any reason. They can do that with an API, but usually won't since some guarantee of stability is the point of an API.
Re: Data-Mining Wikipedia for Fun and Profit
#27Re: Data-Mining Wikipedia for Fun and Profit
#28Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed…
Still, I guess you could get the dumps and do a local Wikimedia setup based on them, and then crawl that instead?
Re: Data-Mining Wikipedia for Fun and Profit
#29Earlier quoted context omitted.
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed…
Still, I guess you could get the dumps and do a local Wikimedia setup based on them, and then crawl that instead?
You could, and if he was doing this on the entire corpus that'd be the responsible thing to do.
But, his project really was very reasonable:
- it fetched ~2,400 pages
- he cached them after first fetch
- Wikipedia aggressively caches anonymous page views (eg the Queen Elizabeth page has a cache age of 82,000 seconds)
English Wikipedia does about 250,000,000 pageviews/day. This guy's use was 0.001% of traffic on that day.
I get the slippery slope arguments, but to me, it just doesn't apply. As someone who has donated $1,000 to Wikipedia in the past, I'm totally happy to have those funds spent supporting use cases like this, rather than demanding that people who want to benefit from Wikipedia be able to set up a MySQL server, spend hours doing the import, install and configure a PHP server, etc, etc.
Re: Data-Mining Wikipedia for Fun and Profit
#30Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
Scrapping, especially on a large scale, can put a noticeable strain on servers.
Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages.
It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).