Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

nerdcrawler.com

Post reply on HN
Print view View original

21–23 of 23 posts

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#21

Post by catoc »

jsnell wrote: post #13
catoc wrote: post #5

Can large companies not be faulted for ignoring robots.txt? Seems like something GDPR could enforce for personal(ly owned) sites?

Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider. https://www.nerdcrawler.com/robots.txt The domain serving the images is allowing everything: https://res.cloudinary.com/robots.txt

I just assumed that would have been the first thing they tried…

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#22

Post by chptung »

jsnell wrote: post #13
catoc wrote: post #5

Can large companies not be faulted for ignoring robots.txt? Seems like something GDPR could enforce for personal(ly owned) sites?

Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider. https://www.nerdcrawler.com/robots.txt The domain serving the images is allowing everything: https://res.cloudinary.com/robots.txt

So I decided to google around and I think TikTok does ignore robots.txt. A few posts from around:

- https://www.reddit.com/r/flask/comments/161fqml/what_to_do_a...

- https://www.reddit.com/r/sysadmin/comments/1ahhzdg/has_anyon...

Post reply on HN