241–250 of 291 posts
Earlier quoted context omitted.
So you're willing to pay my hosting bills?
Re-read what I posted and stop projecting.
Earlier quoted context omitted.
>But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run? Maybe. But even if that turns out to be true, what good is it for the source website? The "AI" will surely not share any money (or anything else that may help the source website) with the source anyways. Why would they, they already got the content and trained on it.
What good is it? If "AI" doesn't know about you down the line, you won't be discovered. Be it in LLM weights or via crawling (perplexity, jina reader etc.), you won't get any organic traffic. It's not about sharing profits.
Crawling, incidentally, I think is the biggest issue with making a new search engine these days. Websites flat out refuse to support any crawler [other] than Google, and Cloudflare and other protection services and CDN's flat out deny access to incumbents. It is not a level playing field. I wrote the above some time ago. I think its even more true today. Its practically impossible to crawl the way the bigger players…
Earlier quoted context omitted.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
If we are able to detect AI scrapers then I would welcome a more strategic solution: feed them garbage data instead of the real content. If enough sites did that then the inference quality would take a hit and eventually the perpetrators, too. But of course this is the more expensive option that can't really be asked of sites that already provide public services (even if those are paid for by ads).
Earlier quoted context omitted.
[flagged]
You might say that, but claiming everyone is a pedophile is such a tired political play at this point. Its primary purpose is to dehumanize people so that blatant wrongdoing can be justified. The visceral reaction might be genuine, but the actual feelings are probably not. I have yet to see someone who actually "cares about the children". The vast majority of accusations e.g. democrats running a pedophile ring turn o…
An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week. This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actua…
I have considered this before, but then if the content can be cached why wouldn't the website just do this themselves? They have the incentive, it is relatively easy and I don't think there's a huge benefit to centralisation (especially since it will basically be centralised to one of the big providers of caching anyways)
I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
At which level of DDos one can claim damages from them?
Earlier quoted context omitted.
When was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?
Crawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.