Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

11–20 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#11

Post by devit »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

If users using weak/reused passwords is your problem, just don't let users choose a password (generate it for them), or don't use passwords at all (send link by e-mail that adds a cookie), or use oauth login.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#12

Post by yamakadi »

IceWreck wrote: post #10

Half of the short-links to cutt.ly aren't working. Why use short links in markdown ?

It’s most likely for tracking clicks. Better to just search for the company names instead of clicking on the links in case they lead to unexpected places.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#13

Post by al2o3cr » 

    You use this software at your own risk. Some of them contain malwares just fyi
LOL why post LINKS to them then? Flat-out irresponsible...

    you build a tool to automate social media accounts to manage ads more efficiently
If by "manage" you mean "commit click fraud"

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#14

Post by adinosaur123 »

Are there any court cases that provide precedence regarding the legality of web scraping?

I'm currently looking for ways to get real estate listings in a particular area and apparently the only real solution is the scrape the few big online listing sites.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#15

Post by oxymoron »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing for a lot of companies, even if it's not 100% effective.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#16

Post by Grimm1 »

adinosaur123 wrote: post #14

Are there any court cases that provide precedence regarding the legality of web scraping? I'm currently looking for ways to get real estate listings in a particular area and apparently the only real solution is the scrape the few big online listing sites.

https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

That’s one of the bigger ones. Unfortunately recent events means scraping is still a gray area.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#17

Post by curun1r »

There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the major search engines. They don’t want their content scraped, but they want the traffic that comes from search. In those cases, it’s often possible to scrape the search engines instead using specialized queries designed to get the content you want into the blurb for each search result.

This kind of indirect scraping can be useful for getting almost all the information you want from sites like LinkedIn that do aggressive scraping detection.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#18

Post by amelius »

curun1r wrote: post #17

There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the major search engines. They don’t want their content scraped, but they want the traffic that comes from search. In those cases, it’s often possible to scrape the search engines instead using specialized queries designed to get the content you want into the blu…

But won't the search engines block you after some limit has been reached?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#20

Post by rp1 »

It's very easy to install Chrome on a linux box and launch it with a whitelisted extension. You can run Xorg using the dummy driver and get a full Chrome instance (i.e. not headless). You can even enable the DevTools API programmatically. I don't see how this would be detectable, and probably a lot safer than downloading a random browser package from an unknown developer.
Post reply on HN