31–40 of 291 posts
Is it just me or does Ars keep on serving videos about the sound design of Callisto Protocol in the middle of everything? Why do they keep on promoting these videos about a game from 2022? They've been doing this for months now.
Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!
Individuals do too. Tools like https://github.com/unclecode/crawl4ai make it simple to obtain public content but also paywalled content including ebooks, forums and more. I doubt these folks are trying to do a DDoS though.
Earlier quoted context omitted.
It depends on whether the information on the website is supposed to be publicly available or not. "This information is publicly available except to people from Israel" sends a really terrible message.
It sends a great message to crack down on these companies, as long as you mention why it’s blocked.
I can't help but wonder if big AI crawlers belonging to the LLMs wouldn't be doing some amount of local caching with Squid or something.
Maybe it's beneficial somehow to let the websites tarpit them or slow down requests to use more tokens.
I've had a number of content sites I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst. These were created 20 years ago and updated over the years. I use to get traffic but that's been slowed to 1,000 or less legitimate visitors over the last year. But now I have to deal with server down emails caused by these aggressive bots that don't respect the…
> Alexa seems like the worst. Many of the bots disguise themselves as coming from Amazon or other big company. Amazon has a page where you can check some details to see if it’s really their crawler or someone imitating it.
I've had a number of content sites I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst. These were created 20 years ago and updated over the years. I use to get traffic but that's been slowed to 1,000 or less legitimate visitors over the last year. But now I have to deal with server down emails caused by these aggressive bots that don't respect the…
It's funny how every time this topic comes up someone says "I've had this happen and x is the worst" With x being any of the big AI providers. Just a couple minutes ago I read the same in another thread and it was Anthropic. A couple weeks back it was Meta. My conclusion is that they're all equally terrible then.