Has anyone found an easy way to expand their templates without using their whole stack? I tried getting Lua templates working from Python but didn't get very far...
Data-Mining Wikipedia for Fun and Profit
51–60 of 100 posts
Re: Data-Mining Wikipedia for Fun and Profit
#52Earlier quoted context omitted.
He was probably one of the biggest users that day, so that makes sense. The 2,400 pages, assuming a 50 KB average gzipped size, equate to 120 MB of transfer. I'm assuming CPU usage is negligible due to CDN caching, and so bandwidth is the main cost. 120 MB is orders of magnitude less transfer than the 18.5 GB dump. Instead of the dumps, he could have used the API -- but would that have significantly changed the costs…
I don't think I agree. Cache has a cost too. In theory, you'd want to cache more popular pages and let the rarely visited ones go through the uncached flow. Crawling isn't user-behavior, so the odds are that a large percentage of the crawled pages were not cached.
That's true. On the other hand, pages with infoboxes are likely well-linked and will end up in the cache either due to legitimate popularity or due to crawler visits.
Checking a random sample of 50 pages from this guy's dataset, 70% of them were cached.
Re: Data-Mining Wikipedia for Fun and Profit
#53If the task is to answer the question "Does a royal lineage exist between Alfred the Great and Queen Elizabeth?" then this works fine, but the results only show 1 of possibly 100s of such paths. If you're of European descent, it's 100% possible for you to find a similar connection to Alfred the Great as well :)
Re: Data-Mining Wikipedia for Fun and Profit
#54Earlier quoted context omitted.
It’s so sad that almost nobody knows or uses SPARQL…
Because the syntax is relatively complex and it is difficult to judge which endpoints and definitions to use.
I learned SPARQL recently, and would agrre its complicated to get info out of Wikidata.
However, having read the article, they didnt have an easy time with scraping Wikipedia either.
So I'd probably still recommend people look into wikidata and SPARQL if they want to do this kind of thing.
Theres a few tools that generate queries for you, and some cli tools as well:
https://github.com/maxlath/wikibase-cli#readme
It makes Wikipedia better too, in a virtuous cycle, with some infoboxes like those that he scraped being converted to be automatically populated from wikidata.
Re: Data-Mining Wikipedia for Fun and Profit
#55Earlier quoted context omitted.
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
Unlike APIs, html class/tag names or whatever provide no stability guarantees. The site owner can break your parser whenever they want for any reason. They can do that with an API, but usually won't since some guarantee of stability is the point of an API.
True, but the analysis was done on files downloaded over the span of two or three days. If someone had decided to change the CSS class of an infobox during that time, I'd have noticed, investigated and adjusted my code appropriately.
Re: Data-Mining Wikipedia for Fun and Profit
#56Earlier quoted context omitted.
>IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. They said please don't, not don't do it or they'll sue you. But content license and site terms of use are different things. From their terms of use you aren’t allowed to > [Disrupt] the services by placing an undue bur…
> [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website; Two things: 1) The English wikipedia *alone* gets 250 million page views per day ! So you would have to be doing an awful lot to cause "undue burden". 2) The Wikipedia robots.txt page openly implies that crawling (and therefore scraping) is acceptable *as long as* you do it in a rate-l…
1. You'd be surprised what kind of traffic scrapers can generate. I've seen scraping companies employing botnets to get around rate limiting that could easily cost enough in extra server fees to cause an "undue burden".
At a previous company we had the exact problem that we published all of our content as machine readable xml, but we had scrapers costing us money by insisting on using our search interface to access our content.
2. No one is going to jail for scraping a few thousand or even a few million pages, but just because low speed web crawlers are allowed to index the site, doesn't mean scraping for every possible use is permitted.
Re: Data-Mining Wikipedia for Fun and Profit
#57For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.
Yup, I had the same situation some months before, even though I knew wikidata was a thing.
I know javascript and had the pages at hand.
I looked at wikidata and some pages about, but still had no clear idea how to use it and no motivation to digg into it. Because js just worked with a small custom script, to retrieve some pages and data.
Re: Data-Mining Wikipedia for Fun and Profit
#58For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
Yes: * http://www.entitree.com/en/family_tree/Elizabeth_II * https://family.toolforge.org/ancestors.php?q=Q187114 Tools found on this page: https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/... --- Some SPARQL queries: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... --- Out of topic: I wish wikipedia would provide an API to get the infoboxes (made using Lua or wikidata).
There is a tool to get infobox data from Wikipedia into Wikidata:
https://pltools.toolforge.org/harvesttemplates/
The easily parsable Infobox data can probably already be found in Wikidata (assuming there is a property).
Re: Data-Mining Wikipedia for Fun and Profit
#59>The graph was interesting but this wasn’t the primary objective of this exercise. I wanted to write “He is the n-times great-father of his current successor Queen Elizabeth.” on King Alfred’s Wikipedia page.
Wouldn’t that contravene Wikipedia’s rules on original research?
Re: Data-Mining Wikipedia for Fun and Profit
#60Earlier quoted context omitted.
(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.
Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew. When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.
> Don't worry about the haters
To be fair, the original comment just made a valid observation in a casual way, he didn't criticize the approach of the OP, nor was he impolite.
But I know it's pretty common to see haters nitpicking things all around ;)