31–40 of 100 posts
Here's a demo: https://www.youtube.com/watch?v=SyhaxCjrZFw
Where's the profit?
Earlier quoted context omitted.
Still, I guess you could get the dumps and do a local Wikimedia setup based on them, and then crawl that instead?
You could, and if he was doing this on the entire corpus that'd be the responsible thing to do. But, his project really was very reasonable: - it fetched ~2,400 pages - he cached them after first fetch - Wikipedia aggressively caches anonymous page views (eg the Queen Elizabeth page has a cache age of 82,000 seconds) English Wikipedia does about 250,000,000 pageviews/day. This guy's use was 0.001% of traffic on that…
For 1 person consuming from one of the most popular sites on the web, this really reads big.
For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.
I only thought of it myself because you mentioned the problem with deducing which parent is the mother and which is the father, and I remember in wikidata those are separate fields.
Earlier quoted context omitted.
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
Where's the profit?
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed…
How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?
Earlier quoted context omitted.
You could, and if he was doing this on the entire corpus that'd be the responsible thing to do. But, his project really was very reasonable: - it fetched ~2,400 pages - he cached them after first fetch - Wikipedia aggressively caches anonymous page views (eg the Queen Elizabeth page has a cache age of 82,000 seconds) English Wikipedia does about 250,000,000 pageviews/day. This guy's use was 0.001% of traffic on that…
> This guy's use was 0.001% of traffic on that day For 1 person consuming from one of the most popular sites on the web, this really reads big.
The 2,400 pages, assuming a 50 KB average gzipped size, equate to 120 MB of transfer. I'm assuming CPU usage is negligible due to CDN caching, and so bandwidth is the main cost. 120 MB is orders of magnitude less transfer than the 18.5 GB dump.
Instead of the dumps, he could have used the API -- but would that have significantly changed the costs to the Wikimedia foundation? I think probably not. In my experience, the happy path (serving anonymous HTML) is going to be aggressively optimized for costs, eg caching, CDNs, negotiated bandwidth discounts.
If we accept that these kinds of projects are permissible (which no one seems to be debating, just the manner in which he did the project!), I think the way this guy went about doing it was not actually as bad as people are making it out to be.
Earlier quoted context omitted.
IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every…
>IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. They said please don't, not don't do it or they'll sue you. But content license and site terms of use are different things. From their terms of use you aren’t allowed to > [Disrupt] the services by placing an undue bur…
Two things:
1) The English wikipedia *alone* gets 250 million page views per day ! So you would have to be doing an awful lot to cause "undue burden".
2) The Wikipedia robots.txt page openly implies that crawling (and therefore scraping) is acceptable *as long as* you do it in a rate-limited fashion, e.g.:
>Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please.
> There are a lot of pages on this site, and there are some misbehaved spiders out there that go _way_ too fast.
>Sorry, wget in its recursive mode is a frequent problem. Please read the man page and use it properly; there is a --wait option you can use to set the delay between hits, for instance.