Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

seqlite.puny.engineering

Post reply on HN
Print view View original

1–10 of 13 posts

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#4

Post by panqueca »

I liked the idea. I think SQLite is very powerful way of storing and sharing data. But this site is only about Stack Exchange Network.

Do you know if is there a place to host other of sqlite dumps? I mean from other websites? Recently I dumped the whole hackernews api and I got thinking about it.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#6

Post by JasonPunyon »

panqueca wrote: post #4

I liked the idea. I think SQLite is very powerful way of storing and sharing data. But this site is only about Stack Exchange Network. Do you know if is there a place to host other of sqlite dumps? I mean from other websites? Recently I dumped the whole hackernews api and I got thinking about it.

This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#7

Post by wolfgang42 »

roozbeh18 wrote: post #5

Can someone tell me how this is collected in SQLite

I wrote a blog post a while back about reading these dumps: https://search.feep.dev/blog/post/2021-09-04-stackexchange

Presumably they have a script that does something similar to that process, and then writes the resulting data into a predefined table structure.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#8

Post by 0cf8612b2e1e »

I built one of these myself that I keep on my laptop. Never had real need to use it, but glad I have .

I keep meaning to do the same thing with Wikipedia. Although the Wikipedia dumps are so inscrutably named and seemingly undocumented it seems the organization does not want me to pursue the idea.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#9

Post by JasonPunyon »

wolfgang42 wrote: post #7
roozbeh18 wrote: post #5

Can someone tell me how this is collected in SQLite

I wrote a blog post a while back about reading these dumps: https://search.feep.dev/blog/post/2021-09-04-stackexchange Presumably they have a script that does something similar to that process, and then writes the resulting data into a predefined table structure.

Nice post!

Yep, my process is similar. It goes...

  - decompress (users|posts)  
  - split into batches of 10,000  
  - xsltproc the batch into sql statements  
  - pipe the batches of statements into sqlite in parallel using flocks for coordination
On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.

Re: SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

#10

Post by giantrobot »

0cf8612b2e1e wrote: post #8

I built one of these myself that I keep on my laptop. Never had real need to use it, but glad I have . I keep meaning to do the same thing with Wikipedia. Although the Wikipedia dumps are so inscrutably named and seemingly undocumented it seems the organization does not want me to pursue the idea.

I've had the same problem with Fandom née Wikia dumps. Just gigabytes of XML with questionable adherence to schemas. Fandom also has a ton of custom-to-Fandom tags which are a further pain to handle.

Pulling useful content out of the dumps has been an exercise in frustration. I'm sure I could figure something out if I had a bunch of time to dedicate to the effort.

If I just had sqlite dumps they'd be trivial to work with and I'd be much happier with them.

Post reply on HN