Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

161–170 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#161

Post by hattmall »

bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I've always thought credential stuffing and most password hacking attempts could be defeated by simply logging into randomly generated dummy accounts if the password is wrong. Just make it so that the same username / password combo takes you to the same random info. Real users should notice things were wrong immediately but bots would have no way to tell unless they already knew some of the real information.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#162

Post by S_A_P »

Mister_Snuggles wrote: post #31
bsamuels wrote: post #2

> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks. If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository. None of these anti-bot providers give a shit about…

I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…

I’m not sure what rate you are trying to get but the electric market in the US has 5 minute settlement periods. So for your region you would need to grab the price for each period and average that to get a power rate. Take that rate and add transmission fees, taxes and various other fees your provider tacks on then multiply that by usage. In Texas you can go directly to the ERCOT site and get these prices and not worry about counter measures. I’m not sure where you are but there is likely a similar whole sale site that you can access.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#163

Post by [deleted] »

nieve wrote: post #121
toomuchtodo wrote: post #75

Earlier quoted context omitted.

A link is generated, emailed to the user, and clicking the link logs them in.

I.e. what Facebook does if you don't log in for long enough. Two days ago I got a pair of messages to the same address with links to completely bypass login and verbiage about how they'd seen I was having trouble logging in followed an sms message with the same to a phone number they're not supposed to be using. It looks a lot like phishing, but it comes out of Facebook's servers and they've done it to me before.

[deleted]

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#164

Post by softwaredoug »

A lot of web scraping is annoying often because there’s *an explicit API built for the scrapers needs*. Instead of looking for an API, many think to first use web scraping. This in turn puts load and complexity on the user facing web app that must now tell scraper from real users.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#165

Post by kodah »

tyingq wrote: post #92
kodah wrote: post #78

Earlier quoted context omitted.

When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs tha…

There's also a fair amount of scraping for things like... - Reselling aggregated data - Competitive pricing and inventory data - "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve - Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it. - And, sort of in the space you mentioned, searching for e…

Most of these I threw under "disrupting business processes". My company considered most of these to be threats to the integrity of their business.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#166

Post by 3np »

nocturnial wrote: post #87
kodah wrote: post #78

Earlier quoted context omitted.

When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs tha…

> I'm sure there are folks who use bots and scrapers for home automation I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.

I don't think anyone mentioned crawling (in my mind, "crawling" refers to a long-running process over several pages or sites and "scraping" could be a single fetch of a single URI)

I'm sure there are others, just from the top of my head:

* Electricity prices (as OP mentioned). Especially for people with solar panels or multiple options for heating.

* Watching for availability/prices of products or new homes one might be on the lookout for. Notifications at price drops/availability

* Public transport: next bus/trains from closest station, delays and interruptions

* IMDB/tvdb/etc for monitored shows and movies. Common with sonarr.

* Air quality, covid outbreaks, whatnot

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#167

Post by fragmede »

softwaredoug wrote: post #164

A lot of web scraping is annoying often because there’s *an explicit API built for the scrapers needs*. Instead of looking for an API, many think to first use web scraping. This in turn puts load and complexity on the user facing web app that must now tell scraper from real users.

But if there's an API, then the overall load is the same, no?

Or to put it another way, naively, having api.example.com and realpeople.example.com separated out into separate sandboxes seems reasonable, but due to the aforementioned problem, its not. But then it also turns out to be the wrong axis for this anyway, and you need your monitoring to work for you.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#168

Post by zarzavat »

dpryden wrote: post #88

It always amazes me how people believe they have a right to retrive data from a website. The HTTP protocol calls it a request for a reason: you are asking for data. The server is allowed to say no, for any reason it likes, even a reason you don't agree with. This whole field of scraping and anti-bot technology is an arms race: one side gets better at something, the other side gets better at countering it. An arms rac…

Because often that data is only available through scraping.

Nobody wants to scrape, it's messy and fickle and a general pain in the backside. But sometimes the data you need exists only in that form.

If you run a website and you have a problem with scrapers, then make all that data available through an API and say what acceptable rate limits are. If cost is an issue, then charge a proportionate fee, my time writing a scraper is worth much more than paying a few dollars for an API.

If you just say "No" to everything then you lose all control over the process and the only outcome will be such an arm race.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#170

Post by fragmede »

marginalia_nu wrote: post #65
EMIRELADERO wrote: post #58

Earlier quoted context omitted.

On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.

There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.

For those that are interested in the specifics, Jamie Williams wrote a piece for the EFF[0] in the wake of hiQ vs Linkedin which dealt with this exact question.

It depends on who the server operator is. If it's your server, yeah, anyone I don't want to be there should go away. If it's your enemy's server, the argument that they're sending that page to the rest of the Internet turns out to be a decent one.

[0] https://www.eff.org/deeplinks/2018/04/scraping-just-automate...

Post reply on HN