All of this could be accomplished with buildabot studio buildabot is.com
Web Scraping in 2016
101–110 of 402 posts
Re: Web Scraping in 2016
#102All of this could be accomplished with buildabot studio buildabotnow.com
Re: Web Scraping in 2016
#103Earlier quoted context omitted.
>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with…
> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so after N very reasonably spaced queries the site would reproducibly start throwing an uncaught MAX_DB_CONNECTIONS_EXCEEDED and just be down for everybody everywhere who might've wanted to use it.
Re: Web Scraping in 2016
#104Re: Web Scraping in 2016
#105Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…
I had a different take -- if one hasn't been keeping up with the "arms race" of modern web scraping (and the countermeasures some sites are adopting these days), or with the JS scraping ecosystem generally, then it seems like this article could make for a decent introduction.
Re: Web Scraping in 2016
#106Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…
As someone who has written a scraping framework, this article is useful AF.
> but all it explained was how to make a few API calls in order to solve a very specific problem.
Yeah, the very specific problems everyone runs into time after time. He presents specific solutions, and reasonable context. If I was googling for one of these problems, I'd be very happy to run into this page.
> Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails "
Your arrogance is my matter-of-fact.
Re: Web Scraping in 2016
#107Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
> A website is public property.
This is a gross misunderstanding of how the internet works.
Re: Web Scraping in 2016
#108Earlier quoted context omitted.
> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…
> you've thrown out a pretty absurd hypothetical
Not even remotely absurd. Where is the data your scraper consuming coming from? It's almost always served from some sort of data repository (SQL or otherwise). That data costs far more per MB to serve up quickly than JS/CSS/images.
Suppose, for example, you host a blogging platform that has one very popular user. Most accounts on your site don't get a ton of visitors, and that one very popular user's post are all stored in cache.
Then along comes a scraper. He thinks, "Hey, this site is serving up a million page impressions a day. It can definitely handle me scraping the site".
But when he runs the scraper, he fills up the cache with a ton of data that it doesn't need, causing cache evictions and general performance degradation for everyone else.
Re: Web Scraping in 2016
#109Earlier quoted context omitted.
How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
>How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. This is called "clickwrap". There is usually a notice in the footer of each page that says something like "By using this site, you agree to our Terms of Service." Typically, this kind of notice has been held enforceable. More recently, judges have been demand…
"Clickwrap" refers to situations where you have to click through before using the service, hence the name. Agreements which are simply a passive notice in a footer somewhere are called "browse-wrap", and are much less likely to be considered enforceable:
Re: Web Scraping in 2016
#110Earlier quoted context omitted.
That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.
Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it. The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be o…
>As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database.
This would be copyright infringement, since the content of the page is a substantive unique work that is automatically copyrighted by its author. A site that doesn't want you scraping its content is not going to want you posting dumps of its pages. Much like BitTorrent, they'd get into the protocol and send subpoenas to the ISPs behind the IPs that serve their pages, and use that info to sue the customer.
When my company was shut down by a legal threat related to scraping, I did suggest to my lawyer that we create something like a browser extension that would grab the data we needed out of normal client-side browsing sessions. This wouldn't be as nice as controlling the flow of information ourselves but it would've worked OK. My lawyer strongly suggested avoiding that as it could've been construed as conspiratorial conduct that would've made criminal prosecution more likely.