I'm sitting here going through some of our competitors robots.txt files wondering if they were doing anything interesting or innovative but they all seem pretty run of the mill. HN can you share some robots.txts that do something unusual or innovative?
Ask HN: Examples of interesting robots.txt
1–3 of 3 posts
Re: Ask HN: Examples of interesting robots.txt
#2How "innovative" do you expect robots.txt to be?
Re: Ask HN: Examples of interesting robots.txt
#3How "innovative" do you expect robots.txt to be?
Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.