For the row "Long-lived sessions after sign-in" the author mentions that this solution is for social media automation i.e. you build a tool to automate social media accounts to manage ads more efficiently. I am curious by what the author means by automating social media accounts to manage ads more efficiently
Avoiding bot detection: How to scrape the web without getting blocked?
221–230 of 312 posts
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#222Earlier quoted context omitted.
Yeah, there’s a central service that all Flight search is connected to, irregardless of airline. The airlines are charged per search to that api, so they monitor their ”look to book” rationvery closely. That ratio remains quite stable im the absence of bots, but skyrockets with any bot activity. Hence, they know from that metric how big of a bot problem they have and how much money they are losing. Major flight searc…
In fact the airlines are charged per book, but if and only if the look to book stays within reasonable bounds. If it rockets up, they’re on the hook for the penalties
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#223Earlier quoted context omitted.
I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…
At a minimum any scraper that doesn't execute JS needs to impersonate a screen reader user agent. Locking out disabled people has to be many levels of illegal in most countries.
Social media platforms has made physical appearance as the first class citizen of the reputation economy. I'm not even talking about those platforms which outright bury content from the disabled as a policy, I'm talking about those platforms whose algorithms favor selfies, videos over text/URLs and thereby putting those with accessibility issues in severe disadvantage.
Why would you use such platforms one might say, Do something which has nothing to do with the reputation economy they might add; Well have you looked at LinkedIn lately? LinkedIn has become ubiquitous with professional job search and 30 second video intro is the very first thing on the profile, not the skills which the platform was meant to be when it was launched. One must be naive to claim that the physical appearance on that video or profile picture doesn't affect the job prospects(Several studies have stated otherwise).
It's not just the physical appearance, The action of creating videos or posting photos itself is hard as a time-constrained person[1] and so I think it's reasonable to ask the platform to allow bots to post deep-fake videos of the user doing silly things which these platform expects from an average user.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#224Re: Avoiding bot detection: How to scrape the web without getting blocked?
#225Earlier quoted context omitted.
thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)
2FA isn't limited to one device, or specific 2FA mobile apps. For example I use oathtool for most 2FA things; you just need to store the secret (often in the form of a QR code, but many services will also offer a text version, and if not you can decode the QR). 100% reliance on a phone which is easily lost, broken, stolen, etc. without backup is really bad IMO. My bank (Revolut) only had a mobile app, and no way to c…
1. https://blog.revolut.com/introducing-the-revolut-web-app/
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#226Earlier quoted context omitted.
> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…
Not exactly. Some people are in the business of gathering and selling valid credit cards. They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#227Earlier quoted context omitted.
Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser. Travel information is also one of those services where it’s not weird for a significa…
By default Selenium exposes a few things in JS that are pretty trivial to detect, so you need to disable/hide that for starters. I don't know how easy or hard that is, but stock Selenium is a poor way to get around anti-bot stuff.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#228Earlier quoted context omitted.
This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs. This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/
Brightdata is formerly known as as Luminati who is owned by same company as Hola VPN. Similarly NordVPN owns Oxylabs (who mostly hack routers and cameras and sell those as residential IP’s).
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#229Earlier quoted context omitted.
You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…
>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.
You must have missed the first 20 or so years of its existence, if that's your position.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#230> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…
2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.