Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wayback Machine: Now with 240,000,000,000 URLs

blog.archive.org

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Wayback Machine: Now with 240,000,000,000 URLs

#31

Post by citricsquid »

arscan wrote: post #8
thomasvendetta wrote: post #2

Seriously, the Wayback Machine is awesome. Just last week I used it to find a website I made twelve years ago at the ripe age of 10. If there are any maintainers/developers reading this, thank you. The fact that they were able to preserve a masterpiece like this means a lot to me: http://web.archive.org/web/20010124071800/http://expage.com/...

Agreed. Back in '96 I was a teenager who spent all his free time running a modest gaming fan website [1]. I dropped it when I went to college a year later, but its nice to know all that hard work will forever be memorialized within the wayback machine. It helps me remember why I became a software engineer in the first place. Thanks for that. [1] http://web.archive.org/web/19970414022225/http://www.scorche...

Do you still own that domain? It's fantastic.

Re: Wayback Machine: Now with 240,000,000,000 URLs

#32

Post by Tichy »

mtrn wrote: post #9

I'm ten percent into an implementation of a 'personal web archive', mostly as a fun side-project. I just wonder, if historical data gets more interesting as the web ages?

The other day it occurred to me that the digital age might result in some serious data loss. I was wondering about historic prices for cars. Not sure how you would have gone about it in previous times, but I suppose you could find old catalogs, adverts in newspapers and stuff life that. But what if vendors only advertise prices on their web sites? Those sites with old prices will be all gone once new prices are set up, same with the advertisements.

Not even sure if archives can help - with some algorithmically created content it might be impossible to index it all.

Just one example - there are surely more. I used to think digital data would be easier to preserve for the future, but now I am not so sure anymore.

Not even mentioning Facebook, which presumably can not be archived because of the walled garden thing.

Re: Wayback Machine: Now with 240,000,000,000 URLs

#34

Post by arscan »

citricsquid wrote: post #31
arscan wrote: post #8

Earlier quoted context omitted.

Agreed. Back in '96 I was a teenager who spent all his free time running a modest gaming fan website [1]. I dropped it when I went to college a year later, but its nice to know all that hard work will forever be memorialized within the wayback machine. It helps me remember why I became a software engineer in the first place. Thanks for that. [1] http://web.archive.org/web/19970414022225/http://www.scorche...

Do you still own that domain? It's fantastic.

Thanks! Sadly, I brought on a volunteer who bought the domain in his name. Not a very smart idea in retrospect... but I was young, and not many people understood the value of a good domain name back in 1996 ;-)

We were up to 20k daily uniques when I quit (not bad for 1997). I wrote the forum & related software myself in perl, which was an amazing learning experience.

Re: Wayback Machine: Now with 240,000,000,000 URLs

#36

Post by agatto2 »

ato wrote: post #21
ddorian43 wrote: post #7

Does anyone know what database they use? Or just files and folders?

The index for Wayback is a massive sorted text file (called a CDX) containing a line for each URL and timestamp. For very large installations this index is sharded across multiple servers and queried in parallel. The lookups are done using plain old binary search. http://archive.org/web/researcher/cdx_file_format.php Each CDX record maps a URL-timestamp pair to a byte offset into an ARC or WARC file. These are essent…

How do you get a hold of the list of urls?

Re: Wayback Machine: Now with 240,000,000,000 URLs

#37

Post by thomasvendetta »

catshirt wrote: post #25
thomasvendetta wrote: post #2

Seriously, the Wayback Machine is awesome. Just last week I used it to find a website I made twelve years ago at the ripe age of 10. If there are any maintainers/developers reading this, thank you. The fact that they were able to preserve a masterpiece like this means a lot to me: http://web.archive.org/web/20010124071800/http://expage.com/...

when i was a kid using my parents AOL connection Homestead was blocked by the default AOL parental controls. so i installed a keylogger so i could sneak onto my dad's screen name and make websites when he went to work. in retrospect i probably could have just asked him to unblock homestead so i can make websites. i realize this is only tangentially related, but your page compelled me to share. :)

Ahhh! Homestead. Thank you for reminding me. I need to check there for archives, too!

Re: Wayback Machine: Now with 240,000,000,000 URLs

#38

Post by attabi »

Please I need help,

Hi, please I am using wayback-1.6 on my tomcat-5.28 (java-1.7 , ubuntu-11.04) to display all my arc.gz files but I have got this error, however this folder contains all my arc.gz files /tmp/wayback/files1/IA.arc.gz

Resource Not In Archive

The Resource you requested is not in this archive.

Post reply on HN