Sounds like a great excuse to test those "Unlimited Diskspace!" and "Unlimited Bandwidth!" claims of shared providers.
The Geocities Torrent (~1TB of awesomeness)
51–60 of 89 posts
Re: The Geocities Torrent (~1TB of awesomeness)
#52Earlier quoted context omitted.
I've been doing that for the last 12 months, and the collection is significantly larger than that. http://www.reocities.com/
I'd love to know what sort of infrastructure you're running this on. I'm in a course for my Masters for library school that deals with similar sorts of problems in maintaining and preserving for the "long term" digital materials. From your site it sounds like you wrote a script using wget to harvest the files and another to check them against versions that were still up. What do you do on the server end now to ensure…
I have a large database table that stores the md5 hashes of all the files and there is a script that can compare all of the contents of the site with the hashes in the files (and with a second copy if that's what it would come to).
Some bitrot is inevitable but I think it's under control for now.
As for help, yes, but right now I'm pretty swamped in other stuff, the next round of work on reocities will likely come after the new year.
Re: The Geocities Torrent (~1TB of awesomeness)
#53I like the concept of "Digital Heritage" as they call it. I think preserving digital heritage may become an important issue in the next few years. As the world begins to recognize the historic legacy of the web, it may someday become as important as preserving physical historical landmarks. Its time to accept that the internet may be the greatest legacy 21st century civilization leaves behind. It likely isn't going a…
Charles Stross has an excellent novel, "Glasshouse", a major plot point of which is that due to shoddy digital preservation practices, far-future historians know virtually nothing about ≈1950 through ≈2050 or so. What little they know is pieced together from fragmentary bits and pieces of evidence, with results that are at turns hilarious and horrifying when "put into practice" by historical re-enactors. Kind of reminded me of David Macauly's "Motel of the Mysteries" in some ways.
Re: The Geocities Torrent (~1TB of awesomeness)
#54Earlier quoted context omitted.
I'd love to know what sort of infrastructure you're running this on. I'm in a course for my Masters for library school that deals with similar sorts of problems in maintaining and preserving for the "long term" digital materials. From your site it sounds like you wrote a script using wget to harvest the files and another to check them against versions that were still up. What do you do on the server end now to ensure…
I have a large database table that stores the md5 hashes of all the files and there is a script that can compare all of the contents of the site with the hashes in the files (and with a second copy if that's what it would come to). Some bitrot is inevitable but I think it's under control for now. As for help, yes, but right now I'm pretty swamped in other stuff, the next round of work on reocities will likely come af…
Have you considered using something like MogileFS? It'd be perfect for this sort of situation.
Let me know if you're interested in this or have any questions -- I've dealt a good bit with systems like this in the past, and would love to give you a hand.
Re: The Geocities Torrent (~1TB of awesomeness)
#551TB of internet nostalgia.
Argh, they're gonna revive and store the stuff I was so ashamed for that I hoped it would never resurface.
On reocities.com you can request to have information removed.
If you want sites removed from this torrent... better ask now, before the torrent's up.
Re: The Geocities Torrent (~1TB of awesomeness)
#56Earlier quoted context omitted.
I've been doing that for the last 12 months, and the collection is significantly larger than that. http://www.reocities.com/
Ah ha! So here's where you've all been hiding. http://www.reocities.com/TimesSquare/1266/
That page is by a young 'hacker' named Simon Liu.
I wonder if it is this Simon Liu:
http://www.computer.org/portal/web/csdl/doi/10.1109/MITP.200...
"Surviving Distributed Denial of Service Attacks"
See? This is why reocities is awesome. The Simon Lius of the world should have access to their old Descent scores.
Re: The Geocities Torrent (~1TB of awesomeness)
#57Re: The Geocities Torrent (~1TB of awesomeness)
#58Earlier quoted context omitted.
I have a large database table that stores the md5 hashes of all the files and there is a script that can compare all of the contents of the site with the hashes in the files (and with a second copy if that's what it would come to). Some bitrot is inevitable but I think it's under control for now. As for help, yes, but right now I'm pretty swamped in other stuff, the next round of work on reocities will likely come af…
Have you considered using something like MogileFS? It'd be perfect for this sort of situation. Let me know if you're interested in this or have any questions -- I've dealt a good bit with systems like this in the past, and would love to give you a hand.
You know where to find me :)
And yes, of course I'm interested. But right now no time.
Re: The Geocities Torrent (~1TB of awesomeness)
#59Wow. This site is amazing. Tons of gems: http://www.textfiles.com/underconstruction/
Was under construction the first web meme? All these people couldn't have arrived at the same metaphor individually (as obvious as it is).
Keep in mind that the idea of a document that might be different every time you pull it up was still kinda new.
So maybe the "construction" part was a meme, but I think the idea that you needed to warn people that this homepage wasn't "finished" yet was universal.
Re: The Geocities Torrent (~1TB of awesomeness)
#60The a51 tables used for cracking gsm crypto are 2TB of awesomeness.