Viewing profile — throwawayscrapd

And maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.

I mean given that you've had this problem repeatedly, I'd call it "past-proofing", but I suppose you know your codebase better than I do.

Did you ever consider refactoring the code so that you don't have to do shotgun surgery every time you make this kind of change?

I work on the kind of big online scientific database that this article is about. 100% of our data is available from a clearly marked "Download" page. We still have scraper bots run…