I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas.
Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.
Web Scraping in 2016
11–20 of 402 posts
Re: Web Scraping in 2016
#12> AngelList even detects PhamtomJS (have not seen other sites do this).
I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and will likely retire that bot soon.
It kinda sucks that I'm just grabbing publicly-available content in a very low-bandwidth way, but I really can't convince myself that what I'm doing is very ethical.
My to-do list includes making my crawler into a more well-behaved bot and that will have to go.
Re: Web Scraping in 2016
#13Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will.
Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
Re: Web Scraping in 2016
#14Currently getting 502 Gateway. Guessing this post is also trending on reddit and we hugged it to death :(.
[deleted]
Re: Web Scraping in 2016
#15This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie programmers. It used to be you could get by with an assumption that what you see in the browser is what you get when you download the raw HTML...but that's increasingly less common the case. So now you have to teach how to debug via the console and network panel, on top of basic HTTP concepts (such as query parameters).
(Even more problematic is that college kids today seem to have a decaying understanding of what a URL is, given how much web navigation we do through the omnibar or apps, particularly on mobile, but that's another issue).
I've been archiving a few government sites to preserve them for web scraping exercises [0] (the Texas death penalty site is a classic, for both being relatively simple at first, and being incredibly convoluted depending on what level of detail you want to scrape [1])). But I imagine even government sites will move more toward AJAX/app-like sites, if the trend at the federal level means anything.
That said, I think the analytics.usa.gov site is a great place to demonstrate the difference between server-generated HTML and client-rendered HTML.
But as someone who just likes doing web-scraping, I feel the tools have mostly kept up with the changes to the web. It's been relatively easy, for example, to run Selenium through Python to mimic user action [2]. Same with PhantomJS through node, which has vastly improved how accurately it renders pages for screenshots compared to what I remember a few years back
[0] https://github.com/wgetsnaps
[1] https://github.com/wgetsnaps/tdcj-state-tx-us--death_row
Re: Web Scraping in 2016
#16Sorry guys, hit by traffic - just scaling my EC2 at the moment.
ok on m4.4xlarge now :)
Re: Web Scraping in 2016
#17Sorry guys, hit by traffic - just scaling my EC2 at the moment.
No worries, we had your page scraped just in case ;)
Google Cache link: http://webcache.googleusercontent.com/search?q=cache:https:/...
Archive.is link: http://archive.is/DQccs
Re: Web Scraping in 2016
#18Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
Not really.
Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info.
If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them.
In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine.
tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.
Re: Web Scraping in 2016
#19Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails ", this just shows a lot of immaturity.
Re: Web Scraping in 2016
#20Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
Or you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.
Scraping being illegal is as dumb as saying it's illegal to take photos in public. You aren't affecting anyone if you do it respectfully.