Live data from Hacker News

SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

seqlite.puny.engineering

11–13 of 13 posts

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#11
post #4

I liked the idea. I think SQLite is very powerful way of storing and sharing data. But this site is only about Stack Exchange Network. Do you know if is there a place to host other of sqlite dumps? I mean from other websites? Recently I dumped the whole hackernews api and I got thinking about it.

Datasette is built to serve SQLite databases. For querying purposes, not bulk download.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#12

I built one of these myself that I keep on my laptop. Never had real need to use it, but glad I have . I keep meaning to do the same thing with Wikipedia. Although the Wikipedia dumps are so inscrutably named and seemingly undocumented it seems the organization does not want me to pursue the idea.

I've had the same problem with Fandom née Wikia dumps. Just gigabytes of XML with questionable adherence to schemas. Fandom also has a ton of custom-to-Fandom tags which are a further pain to handle. Pulling useful content out of the dumps has been an exercise in frustration. I'm sure I could figure something out if I had a bunch of time to dedicate to the effort. If I just had sqlite dumps they'd be trivial to work…

Those sites are so ad infested, I am amazed they offer dumps to get the content. Now I am similarly interested in pursuing this idea, but possibly with the exact same amount of tolerance for pain that you have reported.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#13

Earlier quoted context omitted.

I've had the same problem with Fandom née Wikia dumps. Just gigabytes of XML with questionable adherence to schemas. Fandom also has a ton of custom-to-Fandom tags which are a further pain to handle. Pulling useful content out of the dumps has been an exercise in frustration. I'm sure I could figure something out if I had a bunch of time to dedicate to the effort. If I just had sqlite dumps they'd be trivial to work…

Those sites are so ad infested, I am amazed they offer dumps to get the content. Now I am similarly interested in pursuing this idea, but possibly with the exact same amount of tolerance for pain that you have reported.

The ad cancer was part of my original motivation for downloading dumps. I've now found a a lot of Fandom wikis have links to their dumps you can't actually download because they're on S3 buckets that require keys or have download limits. It's infuriating and I think maybe I'm lucky I grabbed a few dumps when I did.

Fandom is usually the first example I think of whenever I hear the word "enshitification". First Wikia ate all the independent wikis because they offered free/managed MediaWiki hosting. Then slowly started making Wikia worse until the full Fandomization. Now the site is literally unusable without an ad blocker and all of that GFDL content on the site is locked behind obfuscation and incompetence. I desperately miss the old Wookiepedia and MemoryAlpha.

Post reply on HN