101–110 of 290 posts
Earlier quoted context omitted.
yea, it comes across as an extremely entitled mobster take. heads i win, tails you lose. we own all your content, and you better behave. i can bet this is incentive-speak.
[flagged]
Earlier quoted context omitted.
If you’re unable to throttle your own outgoing requests you shouldn’t be making any
I assume it'll be hard for them to notice because it's all coming from Azure IP ranges. OpenAI has very big credit card behind this Azure account so this vulnerability might only be limited by Azure capacity. I noticed they switched their crawler to new IP ranges several times, but unfortunately Microsoft CERT / Azure security team didn't answer to my reports. If this vulnerability is exploited, it hits your server w…
Earlier quoted context omitted.
If it means it makes your own content safe when you deploy it on a corner of your website: mission accomplished!
[flagged]
Maybe you don't want your your stuff to get thrown into the latest silicon valley commercial operation without getting paid for it. That seems like a valid position to take. Or maybe you just don't want Claude's ridiculously badly behaved scraper to chew through your entire budget.
Regardless, scrapers that don't follow the rules like robots.txt pretty quickly will discover why those rules exist in the first place as they receive increasing amounts of garbage.
Earlier quoted context omitted.
> No, put up a loginwall or paywall, authenticate users, and go private. We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they'll jump through hoops to take it anyway. https://www.theguardian.com/technology/2025/jan/10/mark-zuck... If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.
[flagged]
Nepenthes also allows scraping and crawling, for as long as you like.
There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.
Earlier quoted context omitted.
Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.
> And yea, this kind of thing should be trivially preventable if they cared at all. Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial. As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.
Earlier quoted context omitted.
> And yea, this kind of thing should be trivially preventable if they cared at all. Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial. As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.
The technical flaws are quite trivial to spot, if you have the relevant experience: - urls[] parameter has no size limit - urls[] parameter is not deduplicated (but their cache is deduplicating, so this security control was there at some point but is ineffective now) - their requests to same website / DNS / victim IP address rotate through all available Azure IPs, which gives them risk of being blocked by other hoste…
I would guess that this is intentional, intended to prevent IP level blocks from being effective. That way blocking them means blocking all of Azure. Too much collateral damage to be worth it.
Earlier quoted context omitted.
I assume it'll be hard for them to notice because it's all coming from Azure IP ranges. OpenAI has very big credit card behind this Azure account so this vulnerability might only be limited by Azure capacity. I noticed they switched their crawler to new IP ranges several times, but unfortunately Microsoft CERT / Azure security team didn't answer to my reports. If this vulnerability is exploited, it hits your server w…
Note I said outgoing, as in the crawlers should be throttling themselves
I agree it should be throttled. Maybe they don't need to throttle because they don't care about cost.
Funny thing is that servers from AWS were trying to connect to my system when I played around with this - I assume OpenAI has not moved away from AWS yet.
Also many different security scanners hitting my IP after every burst of incoming requests from the ChatGPT crawler Azure IP ranges. Quite interesting to see that there are some proper network admins out there.
There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.
> Unknown websites will get very few crawls per day whereas popular sites millions. we're hosting some pretty unknown very domain specific sites and are getting hammered by Claude and others who, compared to old-school search engine bots also get caught up in the weeds and request the same pages all over. They also seem to not care about response time of the page they are fetching, because when they are caught in the…
There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.