Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

151–160 of 402 posts

Re: Web Scraping in 2016

#151

Post by antisthenes »

muglug wrote: post #23
madamelic wrote: post #18

Earlier quoted context omitted.

>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with…

> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.

You're saying the site should pay him a consulting fee for the free load-testing service he provides?

Re: Web Scraping in 2016

#152

Post by baddox »

chucksmash wrote: post #103
muglug wrote: post #23

Earlier quoted context omitted.

> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.

Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so aft…

It seems like you could easily hit those scaling issues by manually browsing the website. While I agree that it sucks to take down a site by scraping, in that specific case it sounds like the performance issues are their fault and not yours. That said, once I realized the effect my scraping had, I would (hopefully) cease my scraping.

Re: Web Scraping in 2016

#153

Post by cookiecaper »

chucksmash wrote: post #115
chucksmash wrote: post #103

Earlier quoted context omitted.

Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so aft…

Now that I think about it a bit more, I think my hypothesis was that DB connections were allocated at the session level and that without cookies enabled each request initiated a new session. I'd consider that a bug not a feature but I still think it's incumbent on me, the guy scraping the website, not to trigger it.

You can avoid triggering this in your scraper by activating a cookie jar. Pretty simple most of the time. Even commandline cURL and wget support it. I'm sure you figured that out already, but just for anyone who's wondering. ;)

That said, while obviously you want to avoid triggering the bug since it offlines your data source, this is definitely in the site's court to fix and could easily be triggered by normal usage. Some people browse with cookies disabled, especially since the EU passed its "cookie law", requiring sites to get consent before storing a cookie on visitors' machines. If you've started to notice more sites talking about cookies over the last year, that's why. [0]

[0] http://ec.europa.eu/ipg/basics/legal/cookies/index_en.htm

Re: Web Scraping in 2016

#154

Post by stupidcar »

ralusek wrote: post #140
stupidcar wrote: post #120

I wrote a fairly complex spidering and scraping script in Node a few months ago. I found downcache[1] to be absolutely invaluable, particularly as I was debugging my parsing scripts, a I was able to rerun them relatively quickly over the cached responses. However, when the network was no longer a bottleneck, I found that the speed and single-threaded nature of Node became one. It wasn't really that slow, relatively s…

Threading in node is very easy, just use clusters. Alternatively, take any of the CPU intensive activity, like parsing the HTML and formatting as JSON, and just put that on an AWS lambda. You can invoke as many lambdas from your application as you want in parallel and you're not going to be bottlenecked by your CPU :)

Clustering in Node creates isolated child processes, not threads. I needed to have shared queues, in-memory caches, and hashes to coordinate workers and avoid them doing duplicate work.

I'm did consider using clustering and having some master process coordinate everything, and using some shared-memory caching library. But it would not be "easy" to set up, especially compared to something like Java where you get thread pools and synchronized thread-safe collections out of the box.

And Lambda would have been totally impractical. As I said, I had hundred of gigs of data to process. If I'd been uploading this over my puny ADSL upstream every time, I'd still be waiting for a single run to complete.

I'm not trashing Node. I like it. There's a reason I used in the first place, after all. But for this particular use-case, I didn't find it was very good fit.

Re: Web Scraping in 2016

#155

Post by wtracy »

cookiecaper wrote: post #119
niftich wrote: post #113

Earlier quoted context omitted.

Not the discount the validity of your experience, but the usual counterpoint to this is Google, who (like mentioned elsewhere in the thread) has been continuously scraping since the very beginning and in fact built their entire business model on doing so. They are also responsible for advancing the state-of-the-art of scraping (albeit mostly internally), through the development of V8 and headless Chromium so that the…

Yeah, Google violates the CFAA and infringes on copyright as a matter of course. Their service would be impossible if they weren't doing so. The main difference when Google was small was that Google was not dependent on any data source in particular, so even if someone denied their robot or sued them, they could cease and desist without affecting the overall value of their offering. This is different if you are getti…

> they'll sick an army of $1,000/hr lawyers on you

They don't even need to do that. They just cheerfully agree to not scrape you, and wait for you to come back and beg to be re-instated when your search traffic plummets.

Re: Web Scraping in 2016

#156

Post by baddox »

minimaxir wrote: post #82
twa927 wrote: post #73

Earlier quoted context omitted.

How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.

That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.

Regardless of whether that would be reasonable, is it actually true? I know that the United States has specific rules for "public accommodations," which are private properties that are generally accessible to the public, like retail businesses. Property owners in this case don't have complete control over who enters their property. The obvious example is refusal of service due to membership of a protected class like race or religion.

So I'm not so sure that police will escort you out of a Walmart because they caught you taking a picture of the parking lot with your smartphone.

Re: Web Scraping in 2016

#157

Post by siegecraft »

minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

I think the lessons learned from those lawsuits was to always have some sort of 3rd-party intermediary scraping consultancy firm you engage that is totally not just your business under another name.

Re: Web Scraping in 2016

#158

Post by baddox »

ysavir wrote: post #130
minimaxir wrote: post #82

Earlier quoted context omitted.

That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.

Let's go with a more apt analogy: If you're entering a country, do its laws not apply to you until you've seen a copy of them? "Oh, sorry, no one told me theft is illegal here. Where does it say that? Oh, I see. Okay. I'll stop now. Thanks for letting me know." If you cross the border without necessary documents, does that country have no right to detain you, simply because you haven't checked the laws? Just because…

Country's laws are a bit different, simply because a country has virtually absolute legal power over its territory. Countries can and do punish people for breaking laws that one cannot feasibly know they were breaking. Does any human know all the laws in the United States? Would that even be physically possible?

Re: Web Scraping in 2016

#159

Post by lossolo »

jurgenwerk wrote: post #143
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

There are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?

They probably doing it from country where it's legal. In most countries there is no law that would be applicable in this case.

Re: Web Scraping in 2016

#160

Post by lossolo »

cookiecaper wrote: post #149
jurgenwerk wrote: post #143

Earlier quoted context omitted.

There are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?

They aren't, or at least, they won't be if Google decides it doesn't like them anymore and decides to bring the matter to court. The CFAA says it's a crime to exceed "authorized access". Authorized access is whatever the server's owner says it is. If they change their mind, you must cease and desist or risk both civil and criminal penalties. A contract defining the length and nature of your authorization from the ser…

You forgot to add: In USA.
Post reply on HN