Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data-Mining Wikipedia for Fun and Profit

billpg.com

Post reply on HN
Print view View original

61–70 of 100 posts

Re: Data-Mining Wikipedia for Fun and Profit

#61

Post by tasogare »

udp wrote: post #9
mistrial9 wrote: post #7

Earlier quoted context omitted.

earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over

The ivory tower is working on it: https://github.com/w3c/EasierRDF

RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.

Re: Data-Mining Wikipedia for Fun and Profit

#62

Post by jancsika »

ceejayoz wrote: post #42
jancsika wrote: post #38

Earlier quoted context omitted.

> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?

The problem lies in parsing them. Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT It's a whole little clever language ( https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa... ) for making complex diagrams out of rather simple pictograms ( https://commons.wikimedia.org/wiki/Template:Bsicon ).

Oh wow.

But every other infobox I've seen has key/value pairs where the key was always a string.

So what's the spec for an info box? Is it simply to have a starting `` and an ending ``?

Re: Data-Mining Wikipedia for Fun and Profit

#63

Post by jancsika »

cldellow wrote: post #43
jancsika wrote: post #38

Earlier quoted context omitted.

> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?

Even just being able to download a tarball of the HTML of the infoboxes would be really powerful, setting aside the difficulty of, say, translating them into a consistent JSON format. That plus a few other key things (categories, opening paragraph, redirects, pageview data) enable a lot of powerful analysis. That actually might be kind of a neat thing to publish. Hmmmm.

Better yet-- what is the set of wikipedia articles which have an info box that cannot be sensibly interpreted as key/value pairs where the key is a simple string?

Re: Data-Mining Wikipedia for Fun and Profit

#64

Post by jancsika »

squeaky-clean wrote: post #46
jancsika wrote: post #38

Earlier quoted context omitted.

> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?

The infoboxes aren't standardized at all. The HTML they generate is.

Hehe-- I am going to rankly speculate nearly all of them follow an obvious standard of key/value pairs where the key is a string. And then there are like two or three subcultures on Wikipedia that put rando stuff in there and would troll to the death before being forced to change the their infobox class to "rando_box" or whatever negligible effort it would take them if a standard were to be enforced.

Am I anywhere close to being correct?

Re: Data-Mining Wikipedia for Fun and Profit

#65

Post by liminal »

udp wrote: post #9
mistrial9 wrote: post #7

Earlier quoted context omitted.

earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over

The ivory tower is working on it: https://github.com/w3c/EasierRDF

I just looked at the linked repo. Have they made any progress? It looks like _very_ early days.

Re: Data-Mining Wikipedia for Fun and Profit

#66

Post by epaulson »

guidovranken wrote: post #27
brodo wrote: post #6

Earlier quoted context omitted.

It’s so sad that almost nobody knows or uses SPARQL…

At least the last times I checked, the WikiData SPARQL server was extremely slow, frequently timing out.

There's some mix between "it's slow" and "it sets its timeout threshold too low" - a lot of queries would be OK if they just had a bit more time to run. And unfortunately, the time wasted on the badput of the killed queries just slows down everyone else. (They really need a batch queue)

The Wikidata folks are well aware of the limits on their SPARQL service. They just posted an update the other day:

https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.w...

Re: Data-Mining Wikipedia for Fun and Profit

#67

Post by bawolff »

mistrial9 wrote: post #7
billpg wrote: post #3

Earlier quoted context omitted.

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over

Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that.

In real life you use tools for both.

Re: Data-Mining Wikipedia for Fun and Profit

#68

Post by bawolff »

cldellow wrote: post #52
NicoJuicy wrote: post #49

Earlier quoted context omitted.

I don't think I agree. Cache has a cost too. In theory, you'd want to cache more popular pages and let the rarely visited ones go through the uncached flow. Crawling isn't user-behavior, so the odds are that a large percentage of the crawled pages were not cached.

That's true. On the other hand, pages with infoboxes are likely well-linked and will end up in the cache either due to legitimate popularity or due to crawler visits. Checking a random sample of 50 pages from this guy's dataset, 70% of them were cached.

Note - there's several levels of caching at wikipedia. Even if those pages aren't in cdn (varnish) cache, they may be in parser cache (an application level cache of most of the page).

This amount of activity really isn't something to worry about, especially when taking the fast path of logged out user viewing a likely to be cached page.

Re: Data-Mining Wikipedia for Fun and Profit

#69

Post by bawolff »

jancsika wrote: post #62
ceejayoz wrote: post #42

Earlier quoted context omitted.

The problem lies in parsing them. Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT It's a whole little clever language ( https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa... ) for making complex diagrams out of rather simple pictograms ( https://commons.wikimedia.org/wiki/Template:Bsicon ).

Oh wow. But every other infobox I've seen has key/value pairs where the key was always a string. So what's the spec for an info box? Is it simply to have a starting ` ` and an ending ` `?

En wikipedia has some standards. Generally though they are user-created tables and its up to the users to make them consistent (if they so desire). En Wikipedia generally does, but its not exactly a hard garuntee.

If you want machine readable use wikidata (if you hate rdf you can still scrape the html preview of the data)

Re: Data-Mining Wikipedia for Fun and Profit

#70

Post by kemayo »

ahP7Deew wrote: post #51

Has anyone found an easy way to expand their templates without using their whole stack? I tried getting Lua templates working from Python but didn't get very far...

Parsoid[1] is what you'd want for that, most likely. It's the new wikitext parser that MediaWiki is gradually switching over to, but it has the virtue of being usable entirely outside of MediaWiki if you need to.

[1]: https://www.mediawiki.org/wiki/Parsoid

Post reply on HN