Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

nerdcrawler.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#11

Post by chptung »

chasd00 wrote: post #7

Is returning a 403 based on the user agent worth a blog post? Also, can't Bytespider just change their user agent to Byte-Spider? Or, just make their user agent a random string? It will be a forever arms race and require constant code updates to keep chasing that bot by user agent. You're probably better off whitelisting the known user agents and blocking everything else. Also, does it really require a specific "gem"…

It might not be, but I couldn't find much about the topic so I figured I'd write it up and share. And you're right that this may be a bit of whack-a-mole, but for now I've cut my bandwidth down which means I may be able to downgrade my cloudinary plan to a lower tier, which is a big win for me since it accounts for like 20-30% of my total operating cost

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#13

Post by jsnell »

catoc wrote: post #5

Can large companies not be faulted for ignoring robots.txt? Seems like something GDPR could enforce for personal(ly owned) sites?

Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider.

https://www.nerdcrawler.com/robots.txt

The domain serving the images is allowing everything:

https://res.cloudinary.com/robots.txt

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#14

Post by braden_e »

chptung wrote: post #10
braden_e wrote: post #4

This is the worst behaved bot I have ever seen, I suspect it is AI related. I recently decided to block all the AI crawlers - unlike search engines I get nothing from them.

Same! Any chance you can share a list of the bots you're blocking so I can add them too?

ClaudeBot (second worst crawler) and GPTBot are the only ones that identify themselves in an obvious way. The rest I am blocking by network. I assume AI when the crawler is very aggressively downloading images - it must be costing them an absolute fortune!

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#15

Post by phartenfeller »

chasd00 wrote: post #7

Is returning a 403 based on the user agent worth a blog post? Also, can't Bytespider just change their user agent to Byte-Spider? Or, just make their user agent a random string? It will be a forever arms race and require constant code updates to keep chasing that bot by user agent. You're probably better off whitelisting the known user agents and blocking everything else. Also, does it really require a specific "gem"…

For me the interesting part is that the crawler is going bezerk. Never ever should a single crawler be the cause of 80% of traffic.

Also why should they not respect the 403? Crawlers just go to anything they can find. It is not a targeted attack.

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#16

Post by chptung »

braden_e wrote: post #14
chptung wrote: post #10

Earlier quoted context omitted.

Same! Any chance you can share a list of the bots you're blocking so I can add them too?

ClaudeBot (second worst crawler) and GPTBot are the only ones that identify themselves in an obvious way. The rest I am blocking by network. I assume AI when the crawler is very aggressively downloading images - it must be costing them an absolute fortune!

Thanks for these!

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#17

Post by gizmo686 »

I don't get what Bytedance is doing here. Clearly they are not actively trying to evade blocks, as they are idenifying their bot with a user agent sites can block.

However, surely they have enough smart engineers there to realize that running a bot at full speed (and, based on other reports, completely ignoring robots.txt) will get them blocked by a lot of sites.

If they just had a well behaved spider, almost no one would mind. Getting crawled is a fact of life on the internet, and most website owners recognize it as an essential cost of doing busses. Once you get a reputation as a bad spider, though, that is very hard to shake.

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#18

Post by nickthegreek »

chptung wrote: post #8
mmaunder wrote: post #3

Is it just me or is that site a bit broken? Weirdly dark. Edit: Nice try on the vote brigade guys. lol

Yeah...I suck at optimizing for dark mode and I think I'm about to get too much traffic from this post so I can't fix it right now. Probably a tomorrow task haha

quickest fix is to change the text to a light gray.

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#19

Post by chptung »

nickthegreek wrote: post #18
chptung wrote: post #8

Earlier quoted context omitted.

Yeah...I suck at optimizing for dark mode and I think I'm about to get too much traffic from this post so I can't fix it right now. Probably a tomorrow task haha

quickest fix is to change the text to a light gray.

Thank you! Will do!

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#20

Post by chptung »

nickthegreek wrote: post #18
chptung wrote: post #8

Earlier quoted context omitted.

Yeah...I suck at optimizing for dark mode and I think I'm about to get too much traffic from this post so I can't fix it right now. Probably a tomorrow task haha

quickest fix is to change the text to a light gray.

Just pushed a fix out; it's not perfect but i hope it works for now. Let me know if it's better for you!
Post reply on HN