91–100 of 234 posts
Sounds like you should give the bots exactly what they want... a 512MB file of random data.
Scraper sinkhole of randomly generated inter-linked files filled with AI poison could work. No human would click that link, so it leads to the "exclusive club".
Earlier quoted context omitted.
Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.
No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.
Oh, now I understand why Americans can't see a problem here.
Earlier quoted context omitted.
Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.
No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.
The title is confusing, should be "commented-out".
Agree, I thought maybe this was going to be a script to block AI scrapers or something like that.
Earlier quoted context omitted.
Scraper sinkhole of randomly generated inter-linked files filled with AI poison could work. No human would click that link, so it leads to the "exclusive club".
Outbound traffic normally costs more than inbound one, so the asymmetry is set up wrong here. Data poisoning is probably the way.
That's what zip bombs are for.
Earlier quoted context omitted.
Seriously. Did you see what that web server was wearing? I mean, sure it said "don't touch me" and started screaming for help and blocked 99.9% of our IP space, but we got more and they didn't block that so clearly they weren't serious. They were asking for it. It's their fault. They're not really victims.
Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.
A vast hoard of personal information exists and most of it never had or will have proper consent, knowledge, or protection.
Earlier quoted context omitted.
How would recommend doing it? If I was just trying to pull tag links out I feel like treating it like text and using regex would be way more efficient than a full on HTML parser like JSDom or something.
You don't need javascript to parse HTML. Just use an HTML parser. They are very fast. HTML isn't a regular language, so you can't parse it with regular expressions. Obligatory: https://stackoverflow.com/questions/1732348/regex-match-open...
Earlier quoted context omitted.
Any analogy is flawed and you can kill most analogies very fast. They are meant to illustrate a point hopefully efficiently, not to be mathematically true. They are not to everyone's taste, me included in most cases. They are mostly fine as long as they are not used to make a point, but only to illustrate it. I agree with this criticism of this analogy, I actually had this flaw in mind from the start. There are other…
People need to have a better mental model of what it means to host a public web site, and what they are actually doing when they run the web server and point it at a directory of files. They're not just serving those files to customers. They're not just serving them to members. They're not just serving them to human beings. They're not even necessarily serving files to web browsers. They're serving files to every IP…
I know a thing or two about web scraping.
There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi).
Zip bombs are also not for me. Reading header content length is enough to not read the page/file. I provide byte limit to check if response is not too big for me. For other cases reading timeout is enough.
Oh, and did you know that requests timeout is not really timeout a timeout for page read? So server can spoonfeed you bytes, one after another, and there will be no timeout.
That is why I created my own crawling system to mitigate these problems, and have one consistent mean of running selenium.
https://github.com/rumca-js/crawler-buddy
Based on library
Earlier quoted context omitted.
Reminds me of the shortcut that works for the happy path but is utterly fucked by real data. This is an interesting trap, can it easily be avoided without walking the dom?
Yes, parse out HTML comments which is also kind of trivial if you've ever done any sort of parsing, listen for " ". But then again, these people are using AI to build scrapers, so I wouldn't put too much pressure on them to produce high-quality software.
But I think it's moot, parsing HTML is not very expensive if you don't have to actually render it.