How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

This is the worst behaved bot I have ever seen, I suspect it is AI related. I recently decided to block all the AI crawlers - unlike search engines I get nothing from them.

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#5

Post by catoc » Wed, May 22, 2024, 6:04 PM UTC

Can large companies not be faulted for ignoring robots.txt? Seems like something GDPR could enforce for personal(ly owned) sites?

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#6

Post by jd20 » Wed, May 22, 2024, 6:04 PM UTC

I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#7

Post by chasd00 » Wed, May 22, 2024, 6:04 PM UTC

Is returning a 403 based on the user agent worth a blog post? Also, can't Bytespider just change their user agent to Byte-Spider? Or, just make their user agent a random string? It will be a forever arms race and require constant code updates to keep chasing that bot by user agent. You're probably better off whitelisting the known user agents and blocking everything else.

Also, does it really require a specific "gem"? This is HTTP request filtering, the router (as in the real router, like the metal box with network cables) can probably do it by itself these days.

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#8

Post by chptung » Wed, May 22, 2024, 6:04 PM UTC

mmaunder wrote: post #3

Is it just me or is that site a bit broken? Weirdly dark. Edit: Nice try on the vote brigade guys. lol

Yeah...I suck at optimizing for dark mode and I think I'm about to get too much traffic from this post so I can't fix it right now. Probably a tomorrow task haha

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#9

Post by chptung » Wed, May 22, 2024, 6:05 PM UTC

jd20 wrote: post #6

I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?

Not super sure so this felt like a faster way to plug it ASAP.

Re: How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%

#10

Post by chptung » Wed, May 22, 2024, 6:05 PM UTC

braden_e wrote: post #4

This is the worst behaved bot I have ever seen, I suspect it is AI related. I recently decided to block all the AI crawlers - unlike search engines I get nothing from them.

Same! Any chance you can share a list of the bots you're blocking so I can add them too?