[deleted]
Data-Mining Wikipedia for Fun and Profit
41–50 of 100 posts
Re: Data-Mining Wikipedia for Fun and Profit
#42Earlier quoted context omitted.
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed…
> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?
The problem lies in parsing them.
Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT
It's a whole little clever language (https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa...) for making complex diagrams out of rather simple pictograms (https://commons.wikimedia.org/wiki/Template:Bsicon).
Re: Data-Mining Wikipedia for Fun and Profit
#43Earlier quoted context omitted.
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed…
> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?
Even just being able to download a tarball of the HTML of the infoboxes would be really powerful, setting aside the difficulty of, say, translating them into a consistent JSON format.
That plus a few other key things (categories, opening paragraph, redirects, pageview data) enable a lot of powerful analysis.
That actually might be kind of a neat thing to publish. Hmmmm.
Re: Data-Mining Wikipedia for Fun and Profit
#44Earlier quoted context omitted.
It’s so sad that almost nobody knows or uses SPARQL…
At least the last times I checked, the WikiData SPARQL server was extremely slow, frequently timing out.
seems to depend on the query. I can issue straight forward queries that visit a few hundred thousand triples easily. But when i write a query that visits tens of millions of triples it times out.
Re: Data-Mining Wikipedia for Fun and Profit
#45Earlier quoted context omitted.
(Author of original article here.) That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
The HTML is more volatile and subject to change than other sources though
Don't remember the last time wikipedia changed the infobox though
Re: Data-Mining Wikipedia for Fun and Profit
#46Earlier quoted context omitted.
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed…
> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?
The infoboxes aren't standardized at all. The HTML they generate is.
Re: Data-Mining Wikipedia for Fun and Profit
#47For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
Like people on the other comment have said, if you've actually tried getting data from wikidata/wikipedia you very quickly learn the HTML is much easier to parse than the results wikidata gives you.
Re: Data-Mining Wikipedia for Fun and Profit
#48Earlier quoted context omitted.
That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
Can make it even harder, use Puppeteer to take screenshots then pass it to an OCR to get the text.
Re: Data-Mining Wikipedia for Fun and Profit
#49Earlier quoted context omitted.
> This guy's use was 0.001% of traffic on that day For 1 person consuming from one of the most popular sites on the web, this really reads big.
He was probably one of the biggest users that day, so that makes sense. The 2,400 pages, assuming a 50 KB average gzipped size, equate to 120 MB of transfer. I'm assuming CPU usage is negligible due to CDN caching, and so bandwidth is the main cost. 120 MB is orders of magnitude less transfer than the 18.5 GB dump. Instead of the dumps, he could have used the API -- but would that have significantly changed the costs…
I don't think I agree. Cache has a cost too.
In theory, you'd want to cache more popular pages and let the rarely visited ones go through the uncached flow.
Crawling isn't user-behavior, so the odds are that a large percentage of the crawled pages were not cached.
Re: Data-Mining Wikipedia for Fun and Profit
#50For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
Yes:
* http://www.entitree.com/en/family_tree/Elizabeth_II
* https://family.toolforge.org/ancestors.php?q=Q187114
Tools found on this page: https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/...
---
Some SPARQL queries: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/...
---
Out of topic: I wish wikipedia would provide an API to get the infoboxes (made using Lua or wikidata).