Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data-Mining Wikipedia for Fun and Profit

billpg.com

Post reply on HN
Print view View original

91–100 of 100 posts

Re: Data-Mining Wikipedia for Fun and Profit

#91

Post by dalf »

beardyw wrote: post #88
zaik wrote: post #58

Earlier quoted context omitted.

There is a tool to get infobox data from Wikipedia into Wikidata: https://pltools.toolforge.org/harvesttemplates/ The easily parsable Infobox data can probably already be found in Wikidata (assuming there is a property).

So am I right in thinking Wikidata is sort of scraped from Wikipedia internally? The other way round seems better, but obviously too late.

A lot of items don't have a wikipedia page, but all wikipedia, wikitravel, wiki* pages have a wikidata item (see "Wikidata item" link of the left side of the pages).

Some wikipedia infoxes are based on wikidata. I can't find an example, but here some links:

* https://commons.wikimedia.org/wiki/Commons:Wikidata_infobox_...

* https://commons.wikimedia.org/wiki/Template:Wikidata_Infobox

* https://en.wikipedia.org/wiki/Template:Infobox_person/Wikida...

There are lexeme too, and it is not based on wiktionary. Search the prefix "L:" (without quote).

Example:

* https://www.wikidata.org/w/index.php?search=L%3Acat&search=L...

* https://www.wikidata.org/wiki/Lexeme:L7

Also, there are a lot of tools on toolforge.org. One is reasonator which produce sentences from a wikidata item: https://reasonator.toolforge.org/?q=Q1339

Re: Data-Mining Wikipedia for Fun and Profit

#92

Post by pphysch »

namedgraph wrote: post #82
pphysch wrote: post #21

Earlier quoted context omitted.

Why should more people know SPARQL?

E.g. to get a job in FAANG, finance or pharma where SPARQL is used extensively on enterprise Knowledge Graphs? Check the jobs here: http://sparql.club/

The few openings I flipped through all mention SPARQL in an offhand manner, in the sense of "familiarity with query languages and data ontology".

Re: Data-Mining Wikipedia for Fun and Profit

#93

Post by tasogare »

namedgraph wrote: post #83
tasogare wrote: post #61

Earlier quoted context omitted.

RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.

Let us see your proposal of the superior model? RDF was designed primarily for data interchange and there's nothing that beats it at that.

Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to it and was simplier to use in their original format.

And for the model: property graph. But yeah, enjoy your Stockholm syndrome with your model where reification is required to annotate an edge. Also even your nickname is an aknowledgment of RDF failure: named graphs (n-quads) were created because RDF triples aren't good enough for modeling data.

Re: Data-Mining Wikipedia for Fun and Profit

#94

Post by boxed »

shock-value wrote: post #8
billpg wrote: post #3

Earlier quoted context omitted.

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475 ).

I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project (https://relatedhow.kodare.com/) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell.

I ended up loading the full nightly db dump and filtering it streaming from the zip instead. Faster and it actually worked.

The code to do that is at https://github.com/boxed/relatedhow

Re: Data-Mining Wikipedia for Fun and Profit

#95

Post by namedgraph »

tasogare wrote: post #93
namedgraph wrote: post #83

Earlier quoted context omitted.

Let us see your proposal of the superior model? RDF was designed primarily for data interchange and there's nothing that beats it at that.

Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to i…

Yes, let us see how you do data interchange without global identifiers. Such as URIs, which RDF has built-in natively and property graphs do not.

You're right about bioinformatics, but lets do a quick check on http://sparql.club/ on who else is looking for RDF/SPARQL specialists. Oh look: automotive industry, finance, publishing, medical, research etc.

Re: Data-Mining Wikipedia for Fun and Profit

#97

Post by learc83 »

bawolff wrote: post #77
learc83 wrote: post #74

Earlier quoted context omitted.

I don’t think the op was talking specifically to the content author, but to all the people who read the article and get the idea to scrape Wikipedia.

Honestly i'd rather people err on the side of scrapping wikipedia too much than live in fear of being disruptive and not do cool things as a result. Wikipedia is meant to be used to spread knowledge. That includes data mining projects such as the one in this blog. (Before anyone takes this out of context - no im not saying its ok to be intentionally disruptive, or do things without exercising any care at all. Also al…

Having been on the other side of this, I’d rather we encourage people to make use of formats/interfaces designed for machines and use the right tool for the job instead of scraping everything.

It’s incredibly easy for careless scrapers to disrupt a site and cost real money without having a clue what they’re doing.

I want people to think twice and consider what they are doing before they scrape a site.

Re: Data-Mining Wikipedia for Fun and Profit

#98

Post by shock-value »

boxed wrote: post #94
shock-value wrote: post #8

Earlier quoted context omitted.

Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475 ).

I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project ( https://relatedhow.kodare.com/ ) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell. I ended up loading the full nightly db dump and filtering it streaming from the zip in…

There's an alternate Wikidata query engine available here: https://qlever.cs.uni-freiburg.de/wikidata (from https://github.com/ad-freiburg/QLever)

Currently it doesn't support some SPARQL features, but I've found it to generally be quite a bit faster for most queries.

Re: Data-Mining Wikipedia for Fun and Profit

#99

Post by matkoniecz »

namedgraph wrote: post #85
matkoniecz wrote: post #19

Earlier quoted context omitted.

In my experience SPARQL is really hard to use, and Wikidata data quality is really low. To the point that one of my larger project is trying to filter data to make it usable for my usecase. Yes, I made some improvements ( https://www.wikidata.org/wiki/Special:Contributions/Mateusz_... ). But overall I would not encourage using it, if I would know how much work it takes to get usable data I would not bother with it. Q…

Who knew that a global machine-readable knowledge base would involve some complexity?

"is this entry describing (a) bridge (b) event" should have some reasonable way to answer.

So far I have not found way to achieve this without laboriously maintaining my own database of errata, and new exceptions keep appearing.

Re: Data-Mining Wikipedia for Fun and Profit

#100

Post by vagab0nd »

Loosely related: I had some fun working on a database with all events parsed from Wikipedia [0]. I parsed the entire wiki texts using a language model, extracting date/time/location and what happened then and there. The location data was then fed into a local openstreetmap instance to get the coordinates. I then built a front end to allow querying events near by, in time or in space. The UI is a bit clunky because my database is too large and I had little experience working with them. But building it was fun.

0: https://whataday.info/

Post reply on HN