Reddit is the current era’s equivalent of Usenet, and we don’t have a robust archive of that either.
wayback machine's archive of reddit isn't perfect but it works. just give the IA more money
Newsgroups and the Internet Archive: I Made a Difference
61–65 of 65 posts
Re: Newsgroups and the Internet Archive: I Made a Difference
#62Reddit is the current era’s equivalent of Usenet, and we don’t have a robust archive of that either.
wayback machine's archive of reddit isn't perfect but it works. just give the IA more money
Yes, but much of the Wayback Machine’s reddit content was specifically targeted and scraped by ArchiveTeam, who are volunteers that seek out at-risk content from the web and make sure that it gets into the Wayback. In the past few years we’ve specifically tried to go after sub-reddits that we thought were newsworthy and/or at high risk for deletion. But there’s no way we can get all of it.
But you can help! If you have extra server space/bandwidth or you can spare $40/month, we can add more pipelines: https://www.archiveteam.org/index.php/ArchiveBot
Source: am ArchiveTeam member, run various pipelines, have scraped sub-reddits ranging from The_Donald to the cryptocurrency worlds to darknet markets.
Re: Newsgroups and the Internet Archive: I Made a Difference
#63Earlier quoted context omitted.
It's my fault (not the IA), I thought I'd got all newsgroups but must have missed some. I just checked the main newsgroup list and it's incomplete, for some reason. My plan though, is to dust off the old code, get a complete list of groups, get them, and then make it searchable. Sorry about that, I didn't check. This was all done in 2013. Basically, I wanted to build a search engine but indexing the newsgroup posts (…
No apologies necessary: creating the archive in the first place was awesome! Like I said, this just shows how massive and complex Usenet is (or maybe, was), and how it's not easy at all to create a comprehensive archive. It's far better to have some of it than none of it!
From what I can gather, I downloaded one tenth of Usenet - 11,000 groups. This means if I'd done all 110,000 groups it would have taken me about a year to download them, and an 8TB drive to store them (in 2013!). That wasn't really feasible...
Re: Newsgroups and the Internet Archive: I Made a Difference
#64For all of you who are misty-eyed pondering those wonderful, probably-lost-forever Usenet posts from the 90s, I dare you to read Kibo's .signature (last updated 5/5/94 4:52AM http://archive.birdhouse.org/etc/kibosig.txt to cure yourselves of misplaced misty-eyed nostalgia for those long-ago times when the Internet was something else. Kibo for President!
What did I just read. It’s like trying to understand satire from a hundred years ago. I was convinced it was hopelessly corrupted by the archiving process, but some of the ASCII art still works (mostly).
It's concentrated 90's memes, straight from the Gen X tap. Can be hazardous in large doses.
Re: Newsgroups and the Internet Archive: I Made a Difference
#65It's great to see some Usenet archives out there to partly make up for the disappointment of Google Groups. But I'm sad that this archive seems to be incomplete, even within its stated date range. Back in the day, I was active on rec.arts.books.tolkien and alt.fan.tolkien: in this archive, I can't find any trace of the massive "alt." hierarchy at all, and the list of files for the "rec." hierarchy doesn't include the…
It's my fault (not the IA), I thought I'd got all newsgroups but must have missed some. I just checked the main newsgroup list and it's incomplete, for some reason. My plan though, is to dust off the old code, get a complete list of groups, get them, and then make it searchable. Sorry about that, I didn't check. This was all done in 2013. Basically, I wanted to build a search engine but indexing the newsgroup posts (…
This is much appreciated....:-)
I should like to note that in the case of the comp.sys.amiga.* news groups, that these were first established Jan 8, 1991. The first posts in the mbox archive for comp.sys.amiga.programmer, however, are from May 31, 1994. It looks like the first three years might be missing, at least for this group. I haven't yet checked the other amiga groups yet.