11–20 of 100 posts
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
In an act of divine justice, my website is down.
https://web.archive.org/web/20210711201037/https://billpg.co...
(I'll send you a donation. Thank you!)
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every second).
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
It’s so sad that almost nobody knows or uses SPARQL…
Yes, I made some improvements ( https://www.wikidata.org/wiki/Special:Contributions/Mateusz_... ).
But overall I would not encourage using it, if I would know how much work it takes to get usable data I would not bother with it.
Queries as simple as "is this entry describing event, bridge or neither" are requiring extreme effort to get right in a reliable way, including maintaining private list of patches and exemptions.
And bots creating millions of known duplicated entries and expecting people to resolve this manually is quite discouraging. Creating Wikidata entries for Cebuano Wikipedia 'articles' was accepted, despite that Cebuano botpedia is nearly completely bot-generated.
And that is without unclear legal status. Yes, they can legally import databases covered by database rights - but they should either make clear that Wikidata is a legal quagmire in EU or forbid such imports. But Wikidata community did neither.
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps ( https://dumps.wikimedia.org ) to the fully-featured API with a nice sandbox ( https://en.wikipedia.org/wiki/Special:ApiSandbox ) and Wikidata ( https://wikidata.org ).
IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every…
They said please don't, not don't do it or they'll sue you.
But content license and site terms of use are different things.
From their terms of use you aren’t allowed to
> [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website;
Wikipedia is also well within their rights to implement scraping countermeasures.