StackOverflow disallows all crawlers in robots.txt file
stackoverflow.com
StackOverflow disallows all crawlers in robots.txt file
1–9 of 9 posts
Re: StackOverflow disallows all crawlers in robots.txt file
#2Someone pointed this out on Twitter - it looks like StackOverflow recently updated their robots.txt file to explicitly disallow all crawlers. Obviously, this won't stop those that don't respect robots.txt, but I found this decision strange. Not even Google or Bing's crawlers (which respect robots.txt) will be able to crawl StackOverflow, which could be the final nail in the coffin for SO, since (presumably) most StackOverflow traffic comes from search engines.
Re: StackOverflow disallows all crawlers in robots.txt file
#3- With the advent of LLMs, sites like StackOverflow are effectively obsolete—robots.txt or not.
- It’s the inevitable consequence when companies cease to innovate.
Re: StackOverflow disallows all crawlers in robots.txt file
#4- With the advent of LLMs, sites like StackOverflow are effectively obsolete—robots.txt or not. - It’s the inevitable consequence when companies cease to innovate.
[deleted]
Re: StackOverflow disallows all crawlers in robots.txt file
#5- With the advent of LLMs, sites like StackOverflow are effectively obsolete—robots.txt or not. - It’s the inevitable consequence when companies cease to innovate.
[dead]
Re: StackOverflow disallows all crawlers in robots.txt file
#6Someone pointed this out on Twitter - it looks like StackOverflow recently updated their robots.txt file to explicitly disallow all crawlers. Obviously, this won't stop those that don't respect robots.txt, but I found this decision strange. Not even Google or Bing's crawlers (which respect robots.txt) will be able to crawl StackOverflow, which could be the final nail in the coffin for SO, since (presumably) most Stac…
I just tested it with ChatGPT agent mode and it returned a 502 bad gateway, so even agents are not going to be a source of traffic.
They must really want only organic traffic for some reason.
Re: StackOverflow disallows all crawlers in robots.txt file
#7- With the advent of LLMs, sites like StackOverflow are effectively obsolete—robots.txt or not. - It’s the inevitable consequence when companies cease to innovate.
Was stackexchange, a web forum company, supposed to innovate an LLM?
Re: StackOverflow disallows all crawlers in robots.txt file
#8Someone pointed this out on Twitter - it looks like StackOverflow recently updated their robots.txt file to explicitly disallow all crawlers. Obviously, this won't stop those that don't respect robots.txt, but I found this decision strange. Not even Google or Bing's crawlers (which respect robots.txt) will be able to crawl StackOverflow, which could be the final nail in the coffin for SO, since (presumably) most Stac…
https://meta.stackexchange.com/questions/404687/our-robots-t...
>After this change, any trusted crawler will continue to see the old copy of robots.txt.
I think this still works for crawlers like Google
Re: StackOverflow disallows all crawlers in robots.txt file
#9- With the advent of LLMs, sites like StackOverflow are effectively obsolete—robots.txt or not. - It’s the inevitable consequence when companies cease to innovate.
Was stackexchange, a web forum company, supposed to innovate an LLM?
I never claimed Stack Exchange should’ve pioneered LLMs. But it’s not merely a web forum company—just as Google isn’t just a search company. Stack Exchange is a 15-year-old tech firm that has stagnated. Innovation has been virtually nonexistent; their idea of progress has long been superficial UI tweaks on a legacy platform.
Even now, Stack Exchange resists adaptation. This very post highlights their `robots.txt` policy, which actively blocks crawlers—a clear signal of protectionism over transparency. They market themselves as community-driven, but the reality is far more corporate and insular.
Stack Overflow’s situation is telling: while search traffic is down a modest –5% to –14% (per their own data), engagement metrics are in freefall. Weekly posts have dropped 16%; monthly questions are down as much as 66% from their peak. That’s not a dip—it’s systemic decay.