281–290 of 312 posts
> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…
What I really enjoy about this thread is all of the completely different perspectives. Lots of people doing anti-abuse research bemoaning that this stuff exists, and lots of people working against what are from their perspective ham-handed anti-abuse tech blocking legitimate useful automation trading tips on how to do it better. I guess the other sides of those we don't see much. People doing actual black-hat work pr…
When these companies endeavor to stop abuse, they trample all over our freedoms. Suddenly we can't have non-browser user agents anymore. Suddenly we can't root our smartphones anymore. They want nothing to do with us unless it's 100% on their terms with us completely under their control.
Earlier quoted context omitted.
Depends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.
Price data can be protected by copyright as a compliation of data. https://www.bitlaw.com/copyright/database.html
What I really enjoy about this thread is all of the completely different perspectives. Lots of people doing anti-abuse research bemoaning that this stuff exists, and lots of people working against what are from their perspective ham-handed anti-abuse tech blocking legitimate useful automation trading tips on how to do it better. I guess the other sides of those we don't see much. People doing actual black-hat work pr…
When these companies endeavor to stop abuse, they trample all over our freedoms. Suddenly we can't have non-browser user agents anymore. Suddenly we can't root our smartphones anymore. They want nothing to do with us unless it's 100% on their terms with us completely under their control.
The solution to the automation problem is to do what most companies do and have registered API integrations.
Earlier quoted context omitted.
When these companies endeavor to stop abuse, they trample all over our freedoms. Suddenly we can't have non-browser user agents anymore. Suddenly we can't root our smartphones anymore. They want nothing to do with us unless it's 100% on their terms with us completely under their control.
Yeah, but you're not entitled to use their servers. If your use of their servers is something they don't like, their freedom is to blackhole your packets
If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…
> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get…
> Why is Google the only one who can do it?
Because site operators explicitly allow them automated access. If you want the same treatment you have to ask for it.
Earlier quoted context omitted.
When these companies endeavor to stop abuse, they trample all over our freedoms. Suddenly we can't have non-browser user agents anymore. Suddenly we can't root our smartphones anymore. They want nothing to do with us unless it's 100% on their terms with us completely under their control.
There seems to be wildly different perspectives on "bad actors means we can't have nice things" -- one group says that this is a fact of life, and the other says that this is an affront to freedom. A non-tech example is I've had guys on Tinder get legit angry at me for insisting that our first few dates have to be in public places where we drive separate -- "oh so you think I'm some creepy stalker?" And like I am tot…
That's totally understandable though. I insist on meeting in public as well... It's the real world, safety is most important.
The thing with websites is they already allow me to make thousands of requests from my browser. What harm does it do if I make a bunch of requests from a script? I don't see it.
> The solution to the automation problem is to do what most companies do and have registered API integrations.
Yeah, those are pretty great. I always use those whenever possible. Many of the sites I use lack those though. Some have APIs so badly designed that scraping their web site actually results in fewer requests and less overhead.
Earlier quoted context omitted.
> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get…
But you have no freedom when contacting my servers. I can send you 403's just because I don't like your face. There is zero entitlement that you have any access to my servers in any way I don't permit. If I say no automated access then on what grounds to you have to do it anyway? > Why is Google the only one who can do it? Because site operators explicitly allow them automated access. If you want the same treatment y…
Sure. At least then you're being honest. If you hate me, it doesn't matter what user agent I use to access your site. Browsers, scripts, they are all me.
Earlier quoted context omitted.
But you have no freedom when contacting my servers. I can send you 403's just because I don't like your face. There is zero entitlement that you have any access to my servers in any way I don't permit. If I say no automated access then on what grounds to you have to do it anyway? > Why is Google the only one who can do it? Because site operators explicitly allow them automated access. If you want the same treatment y…
> I can send you 403's just because I don't like your face. Sure. At least then you're being honest. If you hate me, it doesn't matter what user agent I use to access your site. Browsers, scripts, they are all me.