131–140 of 290 posts
The article claims that using this will "cause your site to disappear from all search results", but the generated pages don't have the traditional "meta" tags that state the intention to block robots. Are any search engines respecting that classic meta tag?
Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
httpunch() {
local url=$1
local connections=${2:-${HTTPUNCH_CONNECTIONS:-100}}
local action=$1
local keepalive_time=${HTTPUNCH_KEEPALIVE:-60}
local silent_mode=false
# Check if "kill" was passed as the first argument
if [[ $action == "kill" ]]; then
echo "Killing all curl processes..."
pkill -f "curl --no-buffer"
return
fi
# Parse optional --silent argument
for arg in "$@"; do
if [[ $arg == "--silent" ]]; then
silent_mode=true
break
fi
done
# Ensure URL is provided if "kill" is not used
if [[ -z $url ]]; then
echo "Usage: httpunch [kill | ] [number_of_connections] [--silent]"
echo "Environment variables: HTTPUNCH_CONNECTIONS (default: 100), HTTPUNCH_KEEPALIVE (default: 60)."
return 1
fi
echo "Starting $connections connections to $url..."
for ((i = 1; i What's a reasonable way forward to deal with more bots than humans on the internet?
The arms race between AI bots and bot-protection is only going to get worse, leading to increasing infra costs while negatively impacting the UX and performance (captchas, rate limiting, etc.). What's a reasonable way forward to deal with more bots than humans on the internet?
Earlier quoted context omitted.
It would be more efficient for them to spin up a team to study this robots.txt thing. They've ignored that low hanging fruit, so they won't do the more sophisticated thing any time soon.
You can't make money out of studying robots.txt, but you can avoid costs skipping bad web sites.
The arms race between AI bots and bot-protection is only going to get worse, leading to increasing infra costs while negatively impacting the UX and performance (captchas, rate limiting, etc.). What's a reasonable way forward to deal with more bots than humans on the internet?
Earlier quoted context omitted.
Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…
why try this hard for a private company that doesn't employ you?
The arms race between AI bots and bot-protection is only going to get worse, leading to increasing infra costs while negatively impacting the UX and performance (captchas, rate limiting, etc.). What's a reasonable way forward to deal with more bots than humans on the internet?
Earlier quoted context omitted.
What is the https://chatgpt.com/backend-api/attributions endpoint doing (or responsible for when not crushing websites).
When ChatGPT cites web sources in it's output to the user, it will call `backend-api/attributions` with the URL and the API will return what the website is about. Basically it does HTTP request to fetch HTML ` ` tag. They don't check length of supplied `urls[]` array and also don't check if it contains the same URL over and over again (with minor variations). It's just bad engineering all around.
Bot detection is fairly sophisticated these days. No one bypasses it by accident. If they are getting around it then they are doing it intentionally (and probably dedicating a lot of resources to it). I'm pro-scraping when bots are well behaved but the circumvention of bot detection seems like a gray-ish area.
And, yes, I know about Facebook training on copyrighted books so I don't put it above these companies. I've just never seen it confirmed that they actually do it.