Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wayback Machine: Now with 240,000,000,000 URLs

blog.archive.org

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Wayback Machine: Now with 240,000,000,000 URLs

#21

Post by ato »

ddorian43 wrote: post #7

Does anyone know what database they use? Or just files and folders?

The index for Wayback is a massive sorted text file (called a CDX) containing a line for each URL and timestamp. For very large installations this index is sharded across multiple servers and queried in parallel. The lookups are done using plain old binary search.

http://archive.org/web/researcher/cdx_file_format.php

Each CDX record maps a URL-timestamp pair to a byte offset into an ARC or WARC file. These are essentially just gzipped HTTP responses concatenated together:

http://archive.org/web/researcher/ArcFileFormat.php http://www.digitalpreservation.gov/formats/fdd/fdd000236.sht...

The document is retrieved, uncompressed, URLs are rewritten, the navigation banner javascript injected and the result is sent to the client.

The code is here: https://github.com/internetarchive/wayback

Re: Wayback Machine: Now with 240,000,000,000 URLs

#23

Post by ithkuil »

scottbartell wrote: post #18
binarycrusader wrote: post #15

My only gripe with the wayback machine is that when old sites go offline and some random domain squatter picks up the domain when it expires, they apply the current robots.txt to all of the old content making archive.org useless. robots.txt should have a limit; it shouldn't be applied retroactively so aggressively.

I suppose it's a pretty good approach in order to help avoid upsetting website owners and possible lawsuits. While I'm not too sure I agree with it, I can appreciate that they have provided a very easy way for websites to opt-out.

A possible pragmatic solution would be to track the site and spot ownership changes and freeze the robots.txt when it happens.

Reliable ownership change detection can be tricky though, but it's doable IMHO.

Re: Wayback Machine: Now with 240,000,000,000 URLs

#24

Post by jackalope »

scottbartell wrote: post #18
binarycrusader wrote: post #15

My only gripe with the wayback machine is that when old sites go offline and some random domain squatter picks up the domain when it expires, they apply the current robots.txt to all of the old content making archive.org useless. robots.txt should have a limit; it shouldn't be applied retroactively so aggressively.

I suppose it's a pretty good approach in order to help avoid upsetting website owners and possible lawsuits. While I'm not too sure I agree with it, I can appreciate that they have provided a very easy way for websites to opt-out.

It's pretty shocking how many web designers, even experienced professionals, assume a site isn't being crawled because it hasn't "gone live" yet in their minds (i.e., no press release). If you have a site active on an IP without any access controls, you can almost be sure it is being indexed by someone. If it's not the default site, expect one of your users to leak the virtual host name. If it's SSL-protected, it might even be revealed in the certificate. I respect the work the Internet Archive is doing, but I'm also grateful that they will immediately retroactively apply robots.txt if you discover you foolishly exposed a site prematurely.

Re: Wayback Machine: Now with 240,000,000,000 URLs

#25

Post by catshirt »

thomasvendetta wrote: post #2

Seriously, the Wayback Machine is awesome. Just last week I used it to find a website I made twelve years ago at the ripe age of 10. If there are any maintainers/developers reading this, thank you. The fact that they were able to preserve a masterpiece like this means a lot to me: http://web.archive.org/web/20010124071800/http://expage.com/...

when i was a kid using my parents AOL connection Homestead was blocked by the default AOL parental controls. so i installed a keylogger so i could sneak onto my dad's screen name and make websites when he went to work. in retrospect i probably could have just asked him to unblock homestead so i can make websites.

i realize this is only tangentially related, but your page compelled me to share. :)

Re: Wayback Machine: Now with 240,000,000,000 URLs

#26

Post by catshirt »

hayksaakian wrote: post #13
arscan wrote: post #8

Earlier quoted context omitted.

Agreed. Back in '96 I was a teenager who spent all his free time running a modest gaming fan website [1]. I dropped it when I went to college a year later, but its nice to know all that hard work will forever be memorialized within the wayback machine. It helps me remember why I became a software engineer in the first place. Thanks for that. [1] http://web.archive.org/web/19970414022225/http://www.scorche...

> Best viewed in 800x600 resolution. that's great ;-)

i remember questioning whether it was ok to drop 8x6 support. the golden days.

Re: Wayback Machine: Now with 240,000,000,000 URLs

#27

Post by arscan »

catshirt wrote: post #26
hayksaakian wrote: post #13

Earlier quoted context omitted.

> Best viewed in 800x600 resolution. that's great ;-)

i remember questioning whether it was ok to drop 8x6 support. the golden days.

Oh, this was back when we were tired of supporting 640x480, hence the message of "best viewed on 800x600" :)

Re: Wayback Machine: Now with 240,000,000,000 URLs

#30

Post by yuhong »

catshirt wrote: post #26
hayksaakian wrote: post #13

Earlier quoted context omitted.

> Best viewed in 800x600 resolution. that's great ;-)

i remember questioning whether it was ok to drop 8x6 support. the golden days.

Personally, my favorite is how NT4 and 2000 Terminal Servers were limited to 8-bit color depth (without Citrix, of course).
Post reply on HN