Live data from Hacker News

Reddit's robots.txt disallows all web crawlers

Post reply on HN

Print view View original

1–2 of 2 posts

Reddit's robots.txt disallows all web crawlers

#1

Post by rawfael » Thu, Jul 04, 2024, 9:19 AM UTC

Reddit's robots.txt disallows all web crawlers
old.reddit.com

Re: Reddit's robots.txt disallows all web crawlers

#2

Post by rawfael » Thu, Jul 04, 2024, 9:54 AM UTC

Seems that they might be showing a different robots.txt to Google: https://x.com/pedrodias/status/1808799403798409216

Post reply on HN