21–30 of 290 posts
Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.
The reality of web crawling is that the web is already extremely adversarial and any crawler will get every imaginable nonsense thrown at it, ranging from various TCP tar pits, compression and XML bombs, really there's no end to what people will put online.
A more resource effective technique to block misbehaving crawlers is to have a hidden link on each page, to some path forbidden via robots.txt, randomly generated perhaps so they're always unique. When that link is fetched, the server immediately drops the connection and blocks the IP for some time period.
Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
I am not sure. How would crawlers filter this?
There's a ton of these types of of things online, you can't e.g. exhaustively crawl every wikipedia mirror someone's put online.
Earlier quoted context omitted.
It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."
Real search engines respect robots.txt so you could just tell them not to enter Markov Chain Hell.
I have a very vague concept for this, with a different implementation. Some, uh, sites (forums?) have content that the AI crawlers would like to consume, and, from what I have heard, the crawlers can irresponsibly hammer the traffic of said sites into oblivion. What if, for the sites which are paywalled, the signup, which invariably comes with a long click-through EULA, had a legal trap within it, forbidding ingestio…
For example, a EULA might have buried in it that by agreeing, you will become their slave for the next 10 years of your life (or something equally ridiculous). Were it to actually go to court for "violating the agreement", it would be obvious that no rational person would ever actually agree to such an agreement.
It basically boiled down to a claim that the entire process of EULAs are (mostly) pointless because it's understood that no one reads them, but companies insist upon them because a false sense of protection, and the ability to threaten violators of (whatever activity) is better than nothing. A kind of "paper threat".
As it's coming back to me, I think one of the real world examples they used was something like this:
If you go to a golf course and see a sign that says, "The golf course is not responsible for damage to your car from golf balls." The sign is essentially meant as false deterrent - It's there to keep people from complaining by, "informing them of the risk", and make it seem official, so employees will insist it's true if anyone complains, but if you were actually to take it to court, the golf course might still be found culpable because they theoretically could have done something to prevent damage to customers cars and they were aware of the damage that could be caused.
Basically, just because a sign (or the EULA) says it, doesn't make it so.
There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.
> There are already “infinite” websites like these on the Internet. Cool. And how much of the software driving these websites is FOSS and I can download and run it for my own (popular enough to be crawled more than daily by multiple scrapers) website?
I have a very vague concept for this, with a different implementation. Some, uh, sites (forums?) have content that the AI crawlers would like to consume, and, from what I have heard, the crawlers can irresponsibly hammer the traffic of said sites into oblivion. What if, for the sites which are paywalled, the signup, which invariably comes with a long click-through EULA, had a legal trap within it, forbidding ingestio…
This doesn't work like you think it does but even if it did, do you have the money to sustain several years long legal battle against OpenAI?
Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one.
Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chinese bots? Should they even? I don't know.