Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every BBC Essential Mix, 1993 to 2010

thenine.ca

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Every BBC Essential Mix, 1993 to 2010

#31

Post by aidenn0 »

lsb wrote: post #24
brighton36 wrote: post #5

I may regret posting this .... But this is what you probably want to do: user@host:/path/to/EssentialMixRip09.04.10$ wget -rl 0 -np -A .mp3 http://thenine.ca/essential/

Using Coral CDN: wget -rl 0 -np -A .mp3 http://thenine.ca.nyud.net/essential/

Nogood, checkout the robots.txt

Re: Every BBC Essential Mix, 1993 to 2010

#37

Post by _delirium »

aidenn0 wrote: post #31
lsb wrote: post #24

Earlier quoted context omitted.

Using Coral CDN: wget -rl 0 -np -A .mp3 http://thenine.ca.nyud.net/essential/

Nogood, checkout the robots.txt

Since Coral considers itself a distributed caching proxy rather than a crawler, it doesn't respect robots.txt. It does respect the relevant cache-control headers, like "no-cache", but this site doesn't appear to set them.

However, the main problem here is that Coral doesn't cache any files >50mb, so almost none of these are cached. The few http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20E...

Re: Every BBC Essential Mix, 1993 to 2010

#38

Post by _delirium »

petercooper wrote: post #11

For "open directories" like this, it's good form to use the Coral Cache in order to not overwhelm the original source: http://thenine.ca.nyud.net/essential/

Unfortunately, Coral Cache doesn't cache anything >50mb, so for almost all of these it'll just redirect you to the original URL. (There are a handful of <50mb mixes, which it does serve up without redirecting.)
Post reply on HN