Data-Mining Wikipedia for Fun and Profit

#1

Post by billpg » Thu, Aug 19, 2021, 2:17 PM UTC

Data-Mining Wikipedia for Fun and Profit
billpg.com

Re: Data-Mining Wikipedia for Fun and Profit

#2

Post by papercrane » Thu, Aug 19, 2021, 2:49 PM UTC

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

Re: Data-Mining Wikipedia for Fun and Profit

#3

Post by billpg » Thu, Aug 19, 2021, 2:57 PM UTC

papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

(Author here.)

Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley.

Also, it was only my reading your comment just now that told me wikidata was even a thing.

Re: Data-Mining Wikipedia for Fun and Profit

#4

Post by powera » Thu, Aug 19, 2021, 2:57 PM UTC

papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

Definitely. There's even a public query service ( https://query.wikidata.org/ ) which can do a lot of this (though SQL is not good with searching for chains).

Re: Data-Mining Wikipedia for Fun and Profit

#5

Post by humanistbot » Thu, Aug 19, 2021, 2:58 PM UTC

Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps (https://dumps.wikimedia.org) to the fully-featured API with a nice sandbox (https://en.wikipedia.org/wiki/Special:ApiSandbox) and Wikidata (https://wikidata.org).

Re: Data-Mining Wikipedia for Fun and Profit

#6

Post by brodo » Thu, Aug 19, 2021, 3:00 PM UTC

papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

It’s so sad that almost nobody knows or uses SPARQL…

Re: Data-Mining Wikipedia for Fun and Profit

#7

Post by mistrial9 » Thu, Aug 19, 2021, 3:02 PM UTC

billpg wrote: post #3

papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

earth calling ivory tower, earth calling ivory tower

superior RDF triples are like martian language to millions of humans

over

Re: Data-Mining Wikipedia for Fun and Profit

#8

Post by shock-value » Thu, Aug 19, 2021, 3:03 PM UTC

billpg wrote: post #3

papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475).

Re: Data-Mining Wikipedia for Fun and Profit

#9

Post by udp » Thu, Aug 19, 2021, 3:03 PM UTC

mistrial9 wrote: post #7

billpg wrote: post #3

Earlier quoted context omitted.

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over

The ivory tower is working on it: https://github.com/w3c/EasierRDF

Re: Data-Mining Wikipedia for Fun and Profit

#10

Post by 3pt14159 » Thu, Aug 19, 2021, 3:04 PM UTC

billpg wrote: post #3

papercrane wrote: post #2

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew.

When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.