Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

71–80 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#71

Better solution: pay target-site.com to start building an API for you. Pros: * You'll be working with them rather than against them. * Your solution will be far more robust. * It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper. * You're eliminating the possibility that you'll have to deal with legal antagonism * Good anti-scraper defenses are far more sophisticated t…

> pay target-site.com to start building an API for you.

When has that ever worked?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#72

Earlier quoted context omitted.

Splash https://github.com/scrapy-plugins/scrapy-splash Runs a little headless browser.

Interesting, is there a variant which uses chrome? This would also eliminate most scraping protections.

Splash is not chromium I believe. Therefore it's buggy as hell and doesn't render websites that Chrome can as smoothly and easily.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#73
post #63

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

You capture the token along with the request on mitmproxy.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#74

Earlier quoted context omitted.

Well the problem is when someone scrapes ALL the good listings then pre-purchases them for resale at double the cost.

How is it different than paying 50+ low-wage remote workers to "scrape" the phonebook for you and then using the information acquired for profit?

One difference is that Feist v. Rural Telephone says that the data in a phonebook can't be copyrighted.

https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#76

Earlier quoted context omitted.

How is it different than paying 50+ low-wage remote workers to "scrape" the phonebook for you and then using the information acquired for profit?

One difference is that Feist v. Rural Telephone says that the data in a phonebook can't be copyrighted. https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R... .

What about using those employees to "crawl" the web for you then?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#77

Earlier quoted context omitted.

Yes, but see QVC v. Resultly , where the robots.txt was considered binding, not the human-readable TOS. We are getting small wins, but it's going to be slow going until we can get Congress to adjust both the CFAA and the Copyright Act, or until we can get SCOTUS to seriously alter the way these acts have been interpreted with reference to internet access.

Not quite. https://www.law360.com/articles/757906/qvc-website-crash-sui... Basically, all damage claims are null because QVC & Resultly never entered into a mutual agreement. You can write whatever the fuck you want in your ToS but it's not law binding. > Judge Beetlestone also rejected QVC’s claims that Resultly violated the Computer Fraud and Abuse Act by knowingly and intentionally harming the retailer when Result…

>Basically, all damage claims are null because QVC & Resultly never entered into a mutual agreement. You can write whatever the fuck you want in your ToS but it's not law binding.

I'm pretty sure that's what I said re: the ToS? That's only one element of the case (breach of contract). You are correct that in this case, browsewrap was not considered applicable. There have been a few other cases where it wasn't too, as in Nguyen v. Barnes & Noble, Inc., but there have been cases where it was, as in Hubbert v. Dell Corp.. Also note that most cases re: browsewrap do not challenge the viability of automatically entering an agreement by clicking around the site, but rather argue that the notification was simply not prominent enough. It could be worked around by moving the notice into a more prominent location on the page.

The other element is CFAA, and referring to the wide-open robots.txt helped Resultly establish that they were attempting to act in good faith and were not maliciously damaging QVC's systems and not exceeding authorized access to the computer system.

>Please do not cite legal resources and grossly twist realities to spread FUD. If you don't want to be web scraped, simply do not put it online.

I'm not trying to spread FUD, I'm just trying to make it clear that the legal situation is precarious. Google has clearly shown that if you are able to build your coffers and reputation faster than you can incur lawsuits, you can win on this. In fact, lots of big companies begin that way, and become big companies merely because they were lucky enough to get big enough to stand up for themselves before the legal threats started coming in the door.

It's understood that you'll be scraped if you put it online. That doesn't mean scraping is legal.

You may be confused here -- I'm not a publisher trying to stop people from doing this. I'm an entrepreneur whose business depended on scraping data from a specific source. That business got destroyed when they chose to dispatch their law firm against us.

The point of repeatedly discussing this on HN is to make the legal situation clear so that people work to change it, and to make sure people who are going into similar ventures are informed about the legal risks associated with them.

As I said on another post, I am not a lawyer, and this is according to my layman's understanding. No one should misinterpret my posts as legal advice. I'm not going to copy and paste this disclaimer into every post I make because it should be implicit, and a few IANAL disclaimers is plenty.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#78
post #35

Scrapy is indeed excellent. One feature that I really like is Scrapy Shell [1]. It allows to run and debug the scraping code without running the spider, right from the CLI. I use it extensively to test that my selectors (both CSS and XPATH) are returning the proper data on a test URL. [1] https://doc.scrapy.org/en/latest/topics/shell.html

A few things turn me off about Scrapy is that it feels over engineered for what it does. Why do I need an entire framework? I'm taking on technical debt to access data I don't have programmatic access to. CSS/Xpath are very fragile. You most likely will be changing them in the future.

> CSS/Xpath are very fragile. You most likely will be changing them in the future.

Genuinely curious what the alternative is

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#79

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Disclaimer: I'm a co-founder of Apifier [1].

It's not an open source, but free up to 10k pages per month. And it can handle modern JS web applications (your code runs in a context of crawled page). You can for example scrape API key at first and then use internal AJAX calls.

There's also a community page [2] where you can find and use crawlers made by other users.

[1] https://www.apifier.com [2] https://www.apifier.com/community/crawlers

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#80
post #68
post #63

Earlier quoted context omitted.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…

> All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.

This is totally true, but the original premise was to do it just with a MITM. I was being generous and assuming most apps do dynamic generation of their keys. I'm probably wrong now that I think about it.

Post reply on HN