31–40 of 63 posts
Earlier quoted context omitted.
No, you do NOT have the right to pound my site with requests and serve data that I decided to pull down.
Do you also want to steal into libraries in the night and set fire to their microfiche collection?
Earlier quoted context omitted.
No, you do NOT have the right to pound my site with requests and serve data that I decided to pull down.
"serve data that I decided to pull down." If it's on their bandwidth and power, why not?
My server returns 410 GONE to robots.txt requests. The robots exclusion protocol is a ridiculous anachronism. I don't use it and neither should you.
And what do you do about sites with an infinite number of pages?
Spiders have to be robust against sites with unlimited numbers of internal links anyway, or else an attacker could trap a web spider with a malicious site, or a 13 year old writing a buggy PHP add could take down Google's entire spidering system.
Earlier quoted context omitted.
Do you also want to steal into libraries in the night and set fire to their microfiche collection?
Can we get an explanation of how not wanting to have your servers handling more requests than necessary compares to breaking into a library and setting it on fire?
Earlier quoted context omitted.
Do you also want to steal into libraries in the night and set fire to their microfiche collection?
Can we get an explanation of how not wanting to have your servers handling more requests than necessary compares to breaking into a library and setting it on fire?
Earlier quoted context omitted.
Yes, I know, I'm a member of Archive Team, and I use "wget -e robots=off --mirror …" quite a bit, and then I upload those WARC's to the IA. But major content providers like the Washington Post that explicitly choose to block their entire website and its history should be named and shamed. Authors don't get the right to go around removing their novels from public libraries just because they would rather the books be a…
Unfair to name & shame a private entity that doesn't want it's content to be archived.
Earlier quoted context omitted.
Can we get an explanation of how not wanting to have your servers handling more requests than necessary compares to breaking into a library and setting it on fire?
I'm assuming this was in response to the "and serve data that I decided to pull down." part.
fun fact: robots.txt can also be used by attackers to find admin interfaces or other sensitive tidbits that you don't want search engines to crawl lots of target-detection crawlers will look at robots.txt as the first thing they do to see if there's any fun pages you don't want the other crawlers to see
That said, obscurity is not really security. Your admin pages should be behind a password, which, if coded properly, will exclude spiders, bots, and bad guys.
Earlier quoted context omitted.
Do you also want to steal into libraries in the night and set fire to their microfiche collection?
Can we get an explanation of how not wanting to have your servers handling more requests than necessary compares to breaking into a library and setting it on fire?
Perhaps individual private websites, such as pekk's, should have the right to say "No, you do NOT have the right to pound my site with requests and serve data that I decided to pull down."
However, in theory, the Washington Post's articles online are also (eventually) placed on microfiche. Saying there's no right to serve data that WP decided to pull down would in some sense require WP to "steal into libraries in the night and set fire to their microfiche collection".
Earlier quoted context omitted.
And what do you do about sites with an infinite number of pages?
By not writing bad software. State shouldn't be stored in URLs, it should be stored in cookies. Spiders have to be robust against sites with unlimited numbers of internal links anyway, or else an attacker could trap a web spider with a malicious site, or a 13 year old writing a buggy PHP add could take down Google's entire spidering system.
GAH!! So it's you who writes those horrible sites?
I want to be able to middle click on two different URLs and browse two pages with completely different state at the same time.
I HATE sites that store state in cookies, the two different tabs start getting completely mixed up about where I am in the site.
The only thing that should be in a cookie is stuff like a shopping cart. But that's only because the action "add to cart" is like a transaction and should be remembered.
Viewing a page and changing the sort is ephemeral and should have no effect on anything else.
> Spiders have to be robust
Who cares about the spider? What about your site that got hit with an unending stream of completely useless page views?
Your position about robots.txt is simply wrong and you need to change your mind.