331–340 of 466 posts
Earlier quoted context omitted.
People hate collective punishment because it works so well.
Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.
On a side note. CGTalk ( One of the largest forum for 3D / VFX Content creation ) was shut down earlier this year citing resources reasons. And for whatever reason they didn't allow anyone to take a copy of the data, run the forum. They could have sold that Data to OpenAI. I wonder how many Forums shut down due to traffics like this? Most of the reason why forums moved to Slack, Discord etc was that they no longer ha…
Earlier quoted context omitted.
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.
My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.
Earlier quoted context omitted.
This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.
People hate collective punishment because it works so well.
Earlier quoted context omitted.
I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file. https://news.ycombinator.com/item?id=42551628
What is causing you to be so unnecessarily aggressive?
Earlier quoted context omitted.
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.
The amateurs at home are going to give the big companies what they want: an excuse for government regulation.
Earlier quoted context omitted.
Well, Facebook actually releases their models instead of seeking rent off them, so I’m sort of inclined to say Facebook is one of the less evil ones.
> releases their models Some of them, and initially only by accident. And without the ingredients to create your own. Meta is trying to kill OpenAI and any new FAANG contenders. They'll commoditize their complement until the earth is thoroughly salted, and emerge as one of the leading players in the space due to their data, talent, and platform incumbency. They're one of the distribution networks for AI, so they're g…
I mean, the comment with a direct download link in their GitHub repo stayed up even despite all the visibility (it had tons of upvotes).
OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…
I have zero faith that OpenAI respects attempts to block their scrapers
they aren’t blocking them. they’re giving them different content instead.
CommonCrawl is supposed to help for this, i.e. crawl once and host the dataset for any interested party to download out of band. However, data can be up to a month stale, and it costs $$ to move the data out of us-east-1. I’m working on a centralized crawling platform[1] that aims to reduce OP’s problem. A caching layer with ~24h TTL for unauthed content would shield websites from redundant bot traffic while still pr…
You can do so by adding `https://data.commoncrawl.org/` instead of `s3://commoncrawl/` before each of the WARC/WAT/WET paths.
Earlier quoted context omitted.
It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.
I'd rather no-one read it and die forgotten than help "usher in the AI era"