431–440 of 466 posts
That cannot be an efficient use of their money, maybe they used their own AI to write the scraper code.
Earlier quoted context omitted.
That’s a problem of the serverless environment, not of not being a good netizen. Seriously, my toaster from 20 years ago could serve 1req/s
What would you recommend I do instead? Deploying a Docker container on Cloud Run sorta seemed like the logical way to deploy my micro app. Also for more context, this was the app in question (now moved to streamlit cloud): https://jreadability-demo.streamlit.app/
The whole 'cloud serverless buzzword-here' thing is ridiculous for most use cases.
Heck you can serve quite a few static req/s on a $2 ESP32 microcontroller.
For any self-hosting enthusiasts out here. Check your network traffic if you have a Gitea instance running. My network traffic was mostly just AmazonBot and some others from China hitting every possible URL constantly. My traffic has gone from 2-5GB per day to a tenth of that after blocking the bots.
Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…
There's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners. https://github.com/ai-robots-txt/ai.robots.txt
After some digging, I also found a great way to surprise bots that don't respect robots.txt[1] :)
I would be interested in people's thoughts here on my solution: https://www.tela.app . The answer to bot spam: payments, per message. I will soon be releasing a public forum system based on this model. You have to pay to submit posts.
I see this proposed 5-10 times a year for the last 20 years. There's a reason none of them have come to anything.
AI companies go on forums to scrape content for training models, which are surreptitiously used to generate content posted on forums, from which AI companies scrape content to train models, which are surreptitiously used to generate content posted on forums... It's a lot of traffic, and a lot of new content, most of which seems to add no value. Sigh.
I swear that 90% of the posts I see on some subreddits are bots. They just go through the most popular posts of the last year and repost for upvotes. I'm looked at the post history and comments of some of them and found a bunch of accounts where the only comments are from the same 4 accounts and they all just comment and upvote each other with 1 line comments. It's clearly all bots but reddit doesn't care as it looks like more activity and they can charge advertisers more to advertise to bots I guess.
Earlier quoted context omitted.
People hate collective punishment because it works so well.
Putting everyone in jail also works well to prevent crime.
Earlier quoted context omitted.
You know, at this point, I wonder if an allowlist would work better.
I have thought about writing such a thing... 1. A proxy that looks at HTTP Headers and TLS cipher choices 2. An allowlist that records which browsers send which headers and selects which ciphers 3. A dynamic loading of the allowlist into the proxy at some given interval New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so peo…
Earlier quoted context omitted.
If 90% of your problem users come from 1-2 countries, seems pretty sensible to block that country. I know I have 0 paying users in those countries, so why deal with it? Let them go fight it out doing bot wars in local sites
Keep in mind, this is literally why stereotypes and racism exists. It’s the exact same process/reasoning.
Blocking Chinese (or whatever) IPs because they are responsible for a huge amount of malicious behavior is not racist.
Frankly I don’t care what the race of the Chinese IP threat actor is.
Earlier quoted context omitted.
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.
You rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail. I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run…