191–200 of 358 posts
Trying to login to a linkedin account using a google account from an automated browser(like puppeteer+puppeteer-stealth or fakebrowser), will open a white empty window instead of the normal google login window, I could be a limitation of those libraries, but I doubt it, smells like something they detect, maybe looking into might lead some interesting insights on how to limit modern bots.
Earlier quoted context omitted.
It depends on your audience and regions you're most interested in. But if you're aiming for the EU, gcore labs may be interesting. Akamai is not bad, but a bit enterprisey - I don't think they even had an official api the last time I used them?
None of those are free, though.
We also use some simple heuristics to reject obvious bot traffic.
One of the simplest is to have a form field that is hidden via CSS. Humans don't see it and it stays blank. Bots fill it in.
Bots tend to fill in every form fields with random garbage, even checkboxes. Validating checkboxes rather than checking they have a value is another good way to detect bots.
Many bots have a hard time with CSRF tokens in hidden fields.
Many bots also don't handle session cookies properly. If someone submits a registration form without an existing session, we reject it. (So we don't get as far as checking the CSRF token.)
After a certain number of failed attempts to register or login, we block the IP for a period of time.
"The rest are forced to build web services with no interactivity, or seek shelter behind something like Cloudflare, which discriminates against specific browser configurations and uses IP reputation to selectively filter traffic." Interactivity is not a must-have. The world's first general purpose computer, ENIAC, was not built for "interactivity". It was built to calculate ballistic trajectories, which were otherwis…
>>> Automation is not reserved for those providing "services". It also should be available to those using them. Yes ! I call this software literacy. And yes - no matter how cool the JS on a major site, the fact that the sites goals are to keep me there and clicking and my goals are to get what I want with minimal action are in conflict. I would suggest that bots are actually not a problem. For most things I would lik…
Isn't that simply your calendar? Sure, you want it automated; but it doesn't need internet access, it doesn't need to crawl or search, I don't know why you refer to it as a 'bot'.
To my mind, the idea of personal 'bots' was that you could give it some general instructions such as "Let me know when the content at any of these URLs changes", and then leave it running. Were they also called agents?
I don't really understand, is that a lot? 15qps does not sound like a lot, especially for a blocking/rejection function.
For my forum with 500k users a month I just added a registration captcha related to my niche. E.g. for a Dark Souls forum it would say "what game is this forum about?" And if you got it wrong the validation would include "tip it's just two words D rk S*ls". This reduced spam by over 99% and didn't annoy people with recaptcha. If someone was unable to get past that captcha (it still happens I have logs!) I figured the…
re: someone was unable to get past that captcha - this reminded me a story I heard back in ICQ times about some human who couldn't pass anti-bot question: "What planet do we live on?"
> If Marginalia Search didn't use Cloudflare, it couldn't serve traffic. There has been upwards of 15 queries per second from bots. 15 RPS is very far from an apocalypse.
It's bad if it's your dead-average Wordpress site that has 10 PHP workers, each page load being >1s. Easy DoS.
I traced it once, and I got to admit there was not an obvious bottleneck (this was 2015 or so). Just millions upon millions of calls into deeper and deeper layers for things like translations or themes. Wrapping mysql_query in a function that caches the result (to avoid doing identical queries) helped a few % I think, but aside from major changes like patching out the entire translation system for single-language sites, I didn't spot an obvious way to fix it. You'd need to spend a lot of time to optimize away the complexity that grew from suiting a million different needs, contributed by thousands of people across many years.
I wonder if proof-of-work would help. Suppose every form submission requires an expensive calculation, calibrated to take about 1 second on a typical modern computer/smartphone. For human users, this happens in the background, although it makes the website feel slower. But for bots, it dramatically limits how many submissions each botnet host can make to random websites.
Newer variations (such as argon2) are tunable so you can include memory footprint and cpu-parallelism. There also are time-lock puzzles or verifiable delay functions that negate any parallelism because there's a single answer which can't be arrived at sooner by throwing more cores at the problem.
"The rest are forced to build web services with no interactivity, or seek shelter behind something like Cloudflare, which discriminates against specific browser configurations and uses IP reputation to selectively filter traffic." Interactivity is not a must-have. The world's first general purpose computer, ENIAC, was not built for "interactivity". It was built to calculate ballistic trajectories, which were otherwis…
I run a website about immigration. I'd love to reinstate comments and get valuable feedback from people who just tried my advice. Bots just make it too time-consuming.
EDIT: bad joke but maybe someone will get a chuckle.