Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

51–60 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#51

Post by foxylion »

I'm curious what others use to scrape modern (javascript based) web applications.

The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work.

For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved from the server.

This worked fine but required some effort to implement. Is there an open source solution which is as helpful as Scrapy but solves the issues provided by modern javascript websites/applications?

With tools like Chrome headless this should now be possible, right?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#52

Post by chatmasta »

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#53

Post by brianwawok »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Splash https://github.com/scrapy-plugins/scrapy-splash

Runs a little headless browser.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#54

Post by [deleted] »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

[deleted]

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#56

Post by janci »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Many times it's actually much easier to scrape an JS-based app. You just find the right API calls and you get nicely formatted data (JSON mostly).

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#57

Post by xur17 »

chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

Definitely agree. I'd also recommend using mobile sites for scraping as they tend to be simpler and load faster (assuming they contain all of the data you are interested in).

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#58

Post by sergiotapia »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

I use Elixir and Hound because it has a nice clean API that's not difficult to mess around with. It's really straightforward.

https://github.com/HashNuke/hound

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#59

Post by foxylion »

brianwawok wrote: post #53
foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Splash https://github.com/scrapy-plugins/scrapy-splash Runs a little headless browser.

Interesting, is there a variant which uses chrome? This would also eliminate most scraping protections.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#60

Post by brilliantcode »

cookiecaper wrote: post #46
cryptarch wrote: post #31

Earlier quoted context omitted.

I'm afraid that as long as explicit agreement is not required to make a TOS binding we'll be dealing with this crap.

Yes, but see QVC v. Resultly , where the robots.txt was considered binding, not the human-readable TOS. We are getting small wins, but it's going to be slow going until we can get Congress to adjust both the CFAA and the Copyright Act, or until we can get SCOTUS to seriously alter the way these acts have been interpreted with reference to internet access.

Not quite. https://www.law360.com/articles/757906/qvc-website-crash-sui...

Basically, all damage claims are null because QVC & Resultly never entered into a mutual agreement. You can write whatever the fuck you want in your ToS but it's not law binding.

> Judge Beetlestone also rejected QVC’s claims that Resultly violated the Computer Fraud and Abuse Act by knowingly and intentionally harming the retailer when Resultly caused the shopping network's website to crash, reasoning that the tech company and QVC both could only earn money if the site was operational.

I see you are back on the FUD train surrounding web scraping but there's only very specific case where your fears materialize: "When you receive C&D from said website, do not continue scraping". Such was the case for Craigslist vs 3Taps.

Please do not cite legal resources and grossly twist realities to spread FUD. If you don't want to be web scraped, simply do not put it online.

Post reply on HN