Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

141–150 of 402 posts

Re: Web Scraping in 2016

#141

Post by charlesdm »

cookiecaper wrote: post #30
kuschku wrote: post #8

Earlier quoted context omitted.

Or you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.

I was actually searching for such a jurisdiction as my startup was shut down by a company that invoked the CFAA late last year. What do you suggest? The EU is even worse than the US when it comes to data freedom and tech access. The law on the books in many former British colonies provides marginally more protection (the "Telecommunications Act"), but it'd probably still be disputable, and you'd be shut down anyway u…

Dubai?

Re: Web Scraping in 2016

#142

Post by charlesdm »

cookiecaper wrote: post #60
detaro wrote: post #48

Earlier quoted context omitted.

It depends on what you are doing. The CFAA is very far-reaching, but of course many aspects were the US answer is "CFAA" are covered by other laws. [EDIT: removed outdated information superseded by european decisions, which make the situation a lot less clear] anti-scraping: If somebody were to offer a telephone book database online and you created a copy of that to sell on your own, you'd almost certainly loose in t…

>Pro-scraping: Last big case I remember here was a flight-search site that did flight search and booking(!) via a scraper and Ryanair lost when they tried to sue them for that, since they couldn't argue convincingly how that was damaging them. Every case I've seen wrt Ryanair (they sue a lot of people) has resulted in a win for Ryanair. Do you have details on the case you're describing? >anti-scraping: [...] Scraping…

Which site were you running?

Re: Web Scraping in 2016

#143

Post by jurgenwerk »

minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

There are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?

Re: Web Scraping in 2016

#144

Post by niftich »

cookiecaper wrote: post #30
kuschku wrote: post #8

Earlier quoted context omitted.

Or you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.

I was actually searching for such a jurisdiction as my startup was shut down by a company that invoked the CFAA late last year. What do you suggest? The EU is even worse than the US when it comes to data freedom and tech access. The law on the books in many former British colonies provides marginally more protection (the "Telecommunications Act"), but it'd probably still be disputable, and you'd be shut down anyway u…

Antigua and Barbuda? See https://en.wikipedia.org/wiki/Antigua#Internet_hosting_and_g...

Re: Web Scraping in 2016

#145

Post by ramblenode »

franciskim wrote: post #63
spikej wrote: post #53

Earlier quoted context omitted.

Serious question: What do you gain from having an extra layer like docker?

Selenium grid runs in docker, so it's easy to have multiple instances running. Better control.

What are the advantages of this versus a thread pool of web drivers? I'm not really familiar with Selenium Grid.

Re: Web Scraping in 2016

#146

Post by SilasX »

cookiecaper wrote: post #110
mindslight wrote: post #88

Earlier quoted context omitted.

Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it. The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be o…

>As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database. This would be copyright infringement, since the content of the page is a substantive unique work that is automatically copyrighted by its author. A site that doesn't want you scraping its content is not going to want you posting dumps of its pages. Much l…

Isn't that what webarchive/wayback machine do? I think they use a "Fair Use" defense.

Re: Web Scraping in 2016

#147

Post by Fiahil »

hobs wrote: post #13
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

Sometimes, scraping a website is the only way at your disposal to fetch relevant information, you've paid for (or not). It could be simply the opening schedule of your local administration or the status of different pieces of public infrastructure.

If your administration don't have the resources (and it's often the case) to maintain a proper JSON API for you to fetch with a fancy python lib, then, it's not "super bad netizen stuff" to scrap a few HTML/PDF/XLS, parse them and display them for convenient public consumption on your personal website (and paying for the bandwidth).

It's 2016. State-companies holding a third party responsible for their own outages and poor planning is _bad faith_[1]. ETL? Never heard of it?

[1]: https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-d... (french)

Re: Web Scraping in 2016

#148

Post by duaneb »

twa927 wrote: post #73
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.

> A website is a public property.

This isn't even true metaphorically. It's like a shop front: there may be public access, but it is NOT public property.

Re: Web Scraping in 2016

#149

Post by cookiecaper »

jurgenwerk wrote: post #143
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

There are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?

They aren't, or at least, they won't be if Google decides it doesn't like them anymore and decides to bring the matter to court.

The CFAA says it's a crime to exceed "authorized access". Authorized access is whatever the server's owner says it is. If they change their mind, you must cease and desist or risk both civil and criminal penalties. A contract defining the length and nature of your authorization from the server's owner would go a long way to establishing your rights to access, but no one is going to give that to a small player.

Re: Web Scraping in 2016

#150

Post by ralusek »

mosburger wrote: post #12

> AngelList even detects PhamtomJS (have not seen other sites do this). I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and wil…

Mind giving a quick description of getting PhantomJS to work?
Post reply on HN