Web Scraping in 2016 • Hacker News Board

Re: Web Scraping in 2016

#41

Post by niftich » Tue, Aug 23, 2016, 6:47 PM UTC

danso wrote: post #15

This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie programmers. It used to be you could get by with an assumption that what you see in the browser is what you get when you download the raw HTML...but that's increasingly less common the case. So now you have to teach how to debug via the console and network panel,…

It's unfortunate that nearly every webpage these days is a Javascript State Machine which you have to execute in a sandbox and inspect its internal state to get stuff out of.

On a blog post by Paul Kinlan ('Open Web Advocate' at Google and Chromium) [1], I lamented that we ended up here instead of the semantic web because the semantic web was hard to execute. Instead, every web page is a black-box, only navigable by an intelligent and/or sufficiently persuadable human.

But this is also why I don't buy ethical arguments against scraping. Sure, legally any company can unilaterally set any TOS prohibition against behavior they don't want, and these terms may be tested in court. But navigating a page in an automated manner that's designed to resemble interactions of humans (ie. through Selenium) is in my opinion ethical, because it merely time-shifts a user's activity.

[1] https://news.ycombinator.com/item?id=12206846

Re: Web Scraping in 2016

#42

Post by elmigranto » Tue, Aug 23, 2016, 6:48 PM UTC

madamelic wrote: post #20

kuschku wrote: post #8

Earlier quoted context omitted.

Or you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.

Scraping being illegal is as dumb as saying it's illegal to take photos in public. You aren't affecting anyone if you do it respectfully.

Being dumb doesn't prevent laws from existing (we all know those "funny US law stuff" in the realms of "no kissing toy camels on the cheek (but the mouths are okay)".

Also, since this is somewhat untouched territory, don't be so sure that you'll get a judge who is as well-versed in web scarping and infrastructure as you, or shares your opinions on the subject. (And given that precedents are so important in US laws, you better hope someone else before you didn't get such a judge.)

Re: Web Scraping in 2016

#43

Post by franciskim » Tue, Aug 23, 2016, 6:48 PM UTC

slig wrote: post #11

I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas. Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.

I think I might have some trouble with some reCAPTCHA stuff, but there must be ways around it. I agree with you on your point about AWS.

Re: Web Scraping in 2016

#44

Post by Nicksil » Tue, Aug 23, 2016, 6:48 PM UTC

minimaxir wrote: post #25

madamelic wrote: post #18

Earlier quoted context omitted.

>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with…

> tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business. Analyzing data that you're not allowed to access gives you/your company a competitive advantage, which is affecting their service/business even if it's not posted/distributed publically.

I don't follow your argument. How does one get their scraper access to data they would otherwise not be able to access through 'normal' browsing techniques?

Re: Web Scraping in 2016

#45

Post by franciskim » Tue, Aug 23, 2016, 6:50 PM UTC

Jake232 wrote: post #7

Not wanting to thread hijack, but just going to post an article I wrote a few years back as it covers a few other things that are still relevant; and often still gets referenced. May it'll help some people out in combination with OP's post. http://jakeaustwick.me/python-web-scraping-resource/

Nice post Jake!

Re: Web Scraping in 2016

#46

Post by franciskim » Tue, Aug 23, 2016, 6:51 PM UTC

jgmmo wrote: post #3

Good stuff. I do a good bit of scraping, and made RubyRetriever[1] to make my life easier but it seems like I'm getting roadblocked on occasion, probably due to some of the things you mention in your article. Is there any way for a site to verify that only their JS and CSS files are linked? Like preventing injection? [1]: https://github.com/joenorton/rubyretriever

I got the 100th star on the repo! What do you mean by the verifying part?

Re: Web Scraping in 2016

#47

Post by state » Tue, Aug 23, 2016, 6:52 PM UTC

franciskim wrote: post #24

XCSme wrote: post #19

Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…

Thanks for your feedback, I do appreciate it.

As a counterpoint: I think this article is fantastic. API restrictions are incredibly annoying when they pertain to what I consider to be my data. Should data and interface be so tightly joined? Of course not.

Re: Web Scraping in 2016

#48

Post by detaro » Tue, Aug 23, 2016, 6:55 PM UTC

cookiecaper wrote: post #30

kuschku wrote: post #8

Earlier quoted context omitted.

Or you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.

I was actually searching for such a jurisdiction as my startup was shut down by a company that invoked the CFAA late last year. What do you suggest? The EU is even worse than the US when it comes to data freedom and tech access. The law on the books in many former British colonies provides marginally more protection (the "Telecommunications Act"), but it'd probably still be disputable, and you'd be shut down anyway u…

It depends on what you are doing. The CFAA is very far-reaching, but of course many aspects were the US answer is "CFAA" are covered by other laws. [EDIT: removed outdated information superseded by european decisions, which make the situation a lot less clear]

anti-scraping: If somebody were to offer a telephone book database online and you created a copy of that to sell on your own, you'd almost certainly loose in the EU (since unlike in the US, databases as pure collections of facts have their own copyright protections)

The legally safest locations probably are outside the western world if you are targeting western sites.

Re: Web Scraping in 2016

#49

Post by fake-name » Tue, Aug 23, 2016, 6:55 PM UTC

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls.

I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt.

At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for message passing middleware, SaltStack for automated VM provisioning, and python everywhere for everything else. Using some randomization, and a list of the top n user agents, I can randomly generate about ~800K unique but valid-looking UAs. Selenium+PhantomJS gets you through non-capcha cloudflare. Backing storage is Postgres.

Database triggers do row versioning, and I wind up with what is basically a mini internet-archive of my own, with periodic snapshots of a site over time. Additionally, I have a readability-like processing layer that re-writes the page content in hopes of making the resulting layout actually pleasant to read on, with pluggable rulesets that determine page element decomposition.

At this point, I have a system that is, as far as I can tell, definitionally a botnet. The only things is I actually pay for the hosts.

---

Scaling something like this up to high volume is really an interesting challenge. My hosts are physically distributed, and just maintaining the RabbitMQ socket links is hard. I've actually had to do some hacking on the RabbitMQ library to let it handle the various ways I've seen a socket get wedged, and I still have some reliability issues in the SaltStack-DigitalOcean interface where VM creation gets stuck in a infinite loop, leading to me bleeding all my hosts. I also had to implement my own message fragmentation on top of RabbitMQ, because literally no AMQP library I found could reliably handle large (>100K) messages without eventually wedging.

There are other fun problems too, like the fact that I have a postgres database that's ~700 GB in size, which means you have to spend time considering your DB design and doing query optimization too. I apparently have big data problems in my bedroom (My home servers are in my bedroom closet).

---

It's all on github, FWIW:

Manager: https://github.com/fake-name/ReadableWebProxy

Agent and salt scheduler: https://github.com/fake-name/AutoTriever

Re: Web Scraping in 2016

#50

Post by cookiecaper » Tue, Aug 23, 2016, 6:56 PM UTC

muglug wrote: post #23

madamelic wrote: post #18

Earlier quoted context omitted.

>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with…

> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript.

I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too.

Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser.

Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect.

Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser.

It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.