[deleted]
Data-Mining Wikipedia for Fun and Profit
81–90 of 100 posts
Re: Data-Mining Wikipedia for Fun and Profit
#82Earlier quoted context omitted.
It’s so sad that almost nobody knows or uses SPARQL…
Why should more people know SPARQL?
E.g. to get a job in FAANG, finance or pharma where SPARQL is used extensively on enterprise Knowledge Graphs? Check the jobs here: http://sparql.club/
Re: Data-Mining Wikipedia for Fun and Profit
#83Earlier quoted context omitted.
The ivory tower is working on it: https://github.com/w3c/EasierRDF
RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
Let us see your proposal of the superior model?
RDF was designed primarily for data interchange and there's nothing that beats it at that.
Re: Data-Mining Wikipedia for Fun and Profit
#84Earlier quoted context omitted.
earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over
Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that. In real life you use tools for both.
Triples are already structured, machine-readable data. HTML is not.
Re: Data-Mining Wikipedia for Fun and Profit
#85Earlier quoted context omitted.
It’s so sad that almost nobody knows or uses SPARQL…
In my experience SPARQL is really hard to use, and Wikidata data quality is really low. To the point that one of my larger project is trying to filter data to make it usable for my usecase. Yes, I made some improvements ( https://www.wikidata.org/wiki/Special:Contributions/Mateusz_... ). But overall I would not encourage using it, if I would know how much work it takes to get usable data I would not bother with it. Q…
Who knew that a global machine-readable knowledge base would involve some complexity?
Re: Data-Mining Wikipedia for Fun and Profit
#86Earlier quoted context omitted.
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
Unlike APIs, html class/tag names or whatever provide no stability guarantees. The site owner can break your parser whenever they want for any reason. They can do that with an API, but usually won't since some guarantee of stability is the point of an API.
"html class/tag names or whatever provide no stability guarantees"
Not quite. Many Wikipedia infoboxes (and some other templates) use standardised class names from microformats such as hCard:
Re: Data-Mining Wikipedia for Fun and Profit
#87For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
+1 use WikiData’s SPARQL endpoint.
Still, it was a cool article and a good example of scraping information.
Re: Data-Mining Wikipedia for Fun and Profit
#88Earlier quoted context omitted.
Yes: * http://www.entitree.com/en/family_tree/Elizabeth_II * https://family.toolforge.org/ancestors.php?q=Q187114 Tools found on this page: https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/... --- Some SPARQL queries: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... --- Out of topic: I wish wikipedia would provide an API to get the infoboxes (made using Lua or wikidata).
There is a tool to get infobox data from Wikipedia into Wikidata: https://pltools.toolforge.org/harvesttemplates/ The easily parsable Infobox data can probably already be found in Wikidata (assuming there is a property).
So am I right in thinking Wikidata is sort of scraped from Wikipedia internally?
The other way round seems better, but obviously too late.
Re: Data-Mining Wikipedia for Fun and Profit
#89I have a basically English heritage. If I counted unrelated ancestors back through 45 generations I reckon I should have had 35,184,372,088,832! Since the population of England back in the 800s was probably a few million, what are the chances I am NOT a descendent of King Alfred?
Re: Data-Mining Wikipedia for Fun and Profit
#90Earlier quoted context omitted.
The ivory tower is working on it: https://github.com/w3c/EasierRDF
RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
I mean, imho RDF isn't the problem. RDF itself is very simple. As you correctly point out, the stack is overcomplicated.
> Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers
neo4j is basically this. You can also load RDF into neo4j using neosemantics and query it using Cypher instead of using a conventional triplestore with SPARQL, which is nice.