Ask HN: Has anyone had luck using the Wikipedia data dumps? • Hacker News Board

Re: Ask HN: Has anyone had luck using the Wikipedia data dumps?

#11

Post by pklee » Sat, Mar 16, 2019, 1:31 PM UTC

yk66 wrote: post #3

I use Kiwix to do that. Much simpler. Plus they provide other dumps too. So allows you to play with say wikipedia and stackoverflow simultaneously.

That is really cool.. had not seen this before.. I do think the intention is api access to this data not browser.

Re: Ask HN: Has anyone had luck using the Wikipedia data dumps?

#12

Post by StrangeDoctor » Sat, Mar 16, 2019, 7:59 PM UTC

I built tools to parse the compressed XML dumps. My computer was pretty underpowered at the time (MacBook air) so I had to very careful to make everything a streaming algorithm. Looking back I basically recreated a shitty map reduce in Python.

Re: Ask HN: Has anyone had luck using the Wikipedia data dumps?

#13

Post by usgroup » Mon, Mar 18, 2019, 8:32 PM UTC

Dependent on what you’re doing, consider using wikidata instead. Has a SPARQL interface that’s easy to query.

Re: Ask HN: Has anyone had luck using the Wikipedia data dumps?

#14

Post by zepearl » Mon, Mar 18, 2019, 10:06 PM UTC

I used Python to load the contents of the articles into a DB (potentially wrong extract of veeery old code - I have something like 20 different versions lying around therefore I'm not 100% sure that this did work well):

===

  import xml.dom.pulldom as pulldom
  from lxml import etree
  from xml.etree import ElementTree as ET
  sInputFileName = "/my/input/wiki_file.xml"
  context = etree.iterparse(sInputFileName, events=('end',), tag='doc')

  for event, elem in context:
    iThisArticleCharLength = len(elem.text)
    sPageURL = elem.get("url")[0:4000]
    sPageTitle = elem.get("title")[0:4000]
    SPageContents = elem.text

===