Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

241–250 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#241

Post by ensiferum »

fragmede wrote: post #157
jameshart wrote: post #116

Earlier quoted context omitted.

Bots aren't just trying credential stuffing. They are: - committing clickfraud to game ad and referral revenue systems - posting fake or spam reviews and comments - generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments - validating stolen credit card details - screwing with your metrics collection if you can't identify them as bots All of that is…

> -validating stolen credit card details To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they t…

My bank called as they had red flagged a suspicious charge on on my credit card. The sum was $0. The bank rep told me that this often flies under the radar and doesn't show up in some records and it indeed did not show up in my transaction history that i could see online in my banking details. But yeah the point was exactly the same. The fraudsters testing whether the charge goes through and the card is alive/valid.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#242

Post by herbst »

marginalia_nu wrote: post #235
oefrha wrote: post #172

Earlier quoted context omitted.

> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs. Tell that to any Cloudflare site on security level High or "I'm Under Attack!" year round.

Usually means they have large ongoing problems with aggressive bots.

Or they using it as lazy approach to fix long-dos attacks by competitors.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#243

Post by shapefrog »

kingcharles wrote: post #190
IceWreck wrote: post #184

Earlier quoted context omitted.

> Unfortunately, the days of reliable non-JavaScript capable scraping are over. Not really. In a lot of cases websites use JavaScript to call some API along with some on the fly generated token to prevent abuse. As long as that token isn't captcha you can reverse engineer the site to do scraping without javascript and that is so much faster than browser based scraping.

I agree with this. This is what I see on a lot of sites I scrape. Reverse engineering the JS to figure out how the fuck the token was generated is a bitch though.

So then you use headless browsers to render the js and that is even hackier, but totally worth it to hit another full webpage request to get the token, so you can go back to plain requests.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#244

Post by shapefrog »

at_compile_time wrote: post #208
Mister_Snuggles wrote: post #151

Earlier quoted context omitted.

I had never heard of this, but it looks like a reasonable option. My go-to for this type of thing would be Python+Selenium+Firefox, but only due to familiarity with those.

I'm glad that I'm not the only one not using Selenium for its stated purpose.

Is there another - stated purpose - that I missed?

Primarily it is for automating web applications for testing purposes

Here was I thinking it was just a tool to scrape all the js I am too lazy to reverse engineer.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#245

Post by egberts1 »

A couple of things for unblockable scraping

1. plenty of VPS with many IP addresses (this is easier with IPv6 subnet)

2. HTTP header rearranging

3. Fuzzing user-agent

4. Pseudo-PKBOE algorithm

5. office hours, break-time, lunch-time activity emulation

6. ????

7. profit

I am looking at you, SSH port bashers.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#246

Post by rndgermandude »

maven29 wrote: post #207
rndgermandude wrote: post #187

Earlier quoted context omitted.

All very valid reasons (except to a degree the "screwing with metrics" one). There a lot of websites which do not really face any of the aforementioned issues, simply because they do not sell you anything, are not an ad network or run referral programs, do not even have user-generated content, etc. And even the sites that do, it's usually a rather small part of the "surface" that needs such protections e.g. the actua…

It doesn't help that VPN providers in the consumer space pay small ISPs to front their traffic so that netflix et al doesn't drop their traffic coming from a datacenter and they still get to keep up the false pretenses and get valued at billions of dollars. Smaller ISPs are also more likely to have issues with CPE getting compromised and routers running botnets within the comfort of your home. There are services whic…

Yeah, but that's not what happens with this small German ISP my parents use, which was started because some people and local businesses really got disgruntled at the pricing policy and lack of true broadband by the Deutsche Telekom. This thing is essentially run like a non-profit, the little they make in profit meant to be invested into the company again not to make some investors happy, and is majority owned by the city-owned municipal utilities company, with the rest of the ownership I believe in the hands of some local businesses (some of which quite large) who needed broadband but couldn't get it or only at astronomical prices; they are their own customers and thus not very much inclined to fuck themselves.

They are reportedly very proactive when it comes to CPE security, as well, up to giving customers a proactive phone call when they see somebody is using equipment with known vulnerabilities (customers are allowed to operate their own equipment as long as it is deemed compatible, most will use remotely managed equipment, tho, I believe; my dad used to use his own DSL router and once got such a call if I remember correctly. He switched over to their fiber now and managed equipment).

Their AS is indeed identified as an "ISP" in peeringDB.

While I would be extremely surprised if the company was doing shady things, you surely got a point that a small ISP like that could suffer more in reputation from some few customers being up to shady things, including sub-lending the line. I am pretty sure that is against the ToS, but enforcement is a problem of course. Especially detecting such traffic without violating German privacy laws is probably a difficult task, but it's not impossible.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#247

Post by jonatron »

aembleton wrote: post #239
jonatron wrote: post #80

Earlier quoted context omitted.

If you look at Ryanair, they've had legal battles with scrapers for more than 10 years, so it's not that simple.

I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices. For example: curl "https://www.ryanair.com/api/booking/v4/en-gb/availability?ADT=1&CHD=0&DateIn=&DateOut=2021-11-15&Destination=BER&Disc=0&INF=0&Origin=MAN&TEEN=0&promoCode=&IncludeConnectingFlights=false…

I haven't tried to scrape Ryanair, you could be right that it's trivial. It's the legal side that has a long and interesting history. Personally I wouldn't scrape them unless it was when working for a company that can afford lawyers.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#248

Post by ivanhoe »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Saying that Anti-bot software is nonsense is like saying that door locks are snake oil too. We've all seen Lockpicking Lawyer on Youtube opening with ease any lock out there, so how come that all of us haven't got robbed yet?

Well, because protection is not a binary thing - either being 100% safe or 100% not working - instead it's a proportion between the skill/effort/time needed to break in, and the reward you get for it.

To stop majority of attacks you don't have to be absolutely unbreakable, you just need to make it hard enough for majority of attackers so that it doesn't payout for them compared to the value of the data you're protecting. And that's where anti-bot SW has it's place, it slows done spiders and global attacks, forcing for custom tailored scraping that is constantly being fine-tuned, infrastructure to hide your IPs, and that makes the operation way more expensive and harder to run continuously...

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#249

Post by shapefrog »

aembleton wrote: post #239
jonatron wrote: post #80

Earlier quoted context omitted.

If you look at Ryanair, they've had legal battles with scrapers for more than 10 years, so it's not that simple.

I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices. For example: curl "https://www.ryanair.com/api/booking/v4/en-gb/availability?ADT=1&CHD=0&DateIn=&DateOut=2021-11-15&Destination=BER&Disc=0&INF=0&Origin=MAN&TEEN=0&promoCode=&IncludeConnectingFlights=false…

{"code":"TermsOfUseAreNotAccepted","message":"TermsOfUseAreNotAccepted"}
Post reply on HN