Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

311–320 of 402 posts

Re: Web Scraping in 2016

#311

Post by lerpa »

louprado wrote: post #304
bigtunacan wrote: post #301

Earlier quoted context omitted.

As the user of large websites I don't care. I'm not going to read the TOS and I will continue to scrape what I like since it makes my life more convenient. Like OP when blocked I'll just drive my scraping through a web browser which is the same as I've done for years on various sites that never provided APIs.

"As the user of large websites I don't care". Are you sure ? Do you want your OK Cupid or LinkedIn profile to be crossposted on another website without your knowledge.

If you don't want your data public, then don't make it public in the first place. That's a good rule of thumb.

Re: Web Scraping in 2016

#312

Post by lerpa »

nmstoker wrote: post #214
Analemma_ wrote: post #210

Earlier quoted context omitted.

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.

It's their site they can block whatever they want. The problem is the stupid far reaching conclusion that this is trespassing.

Even normal trespassing laws are way too overreaching (see how it is handled in the UK for a saner example) but now you have the amazing possibility of remote trespassing.

The fun part is that it's just a matter of someone hiding something that says you cannot access the site in a place that you have to access the site in order to read -- the ToS. Suing people over this is idiotic.

The real problem is the involvement of Govt, and this kind of absurdity regarding ToS, EULAs and so on, is something that has been going on for decades. If you have the money you can make Govt your personal watch dogs.

Re: Web Scraping in 2016

#313

Post by dspillett »

mack73 wrote: post #302

Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court. They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume perso…

> Corporations will abuse your personal integrity whenever they get a chance, while abiding the law.

Call my cynical if you will, but I'd leave "while abiding the law" out of that, or at least replace it with "while hoping they aren't breaking the law". Due diligence on these matters is often sadly lacking. They'll take the information first and only consider any such implications when/if they come up later.

Large organisations like Google probably will make the up-front effort to remain legal, because they are in the public eye enough for lack of doing so to attract a lot of unwanted press, but you don't have to get a lot smaller than that to start finding companies who are a lot less careful (or in some cases wilfully negligent).

Re: Web Scraping in 2016

#314

Post by niftich »

dspillett wrote: post #313
mack73 wrote: post #302

Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court. They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume perso…

> Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Call my cynical if you will, but I'd leave "while abiding the law" out of that, or at least replace it with "while hoping they aren't breaking the law". Due diligence on these matters is often sadly lacking. They'll take the information first and only consider any such implications when/if they come up later. Large or…

Often, it's indeed cheaper to pay a government-mandated fine than lose market opportunities afforded by behavior that later runs afoul of some law or regulation.

Re: Web Scraping in 2016

#315

Post by bigiain »

muglug wrote: post #23
madamelic wrote: post #18

Earlier quoted context omitted.

>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with…

> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.

I mostly agree with the post's author on the "I'm just automating something I'd otherwise be doing manually". If the local weather service publishes, say, barometric charts on their site - but has a TOS that prohibits me from scraping, and my alternative was to just hit their site every day and right-click and save-as on the chart - I feel absolutely no compunction in automating that. You need to be careful of the slippery slope though, when it's easy to grab every day's local barometric chart, it become too easy to think "Hey, I just need to stick that in a loop and I can grab 1000 different charts every day!". I'd personally _not_ do that. If it's something I'm likely to do "by hand" but would occasionally miss a day or three, I'll automate it no matter what the TOS says.

Re: Web Scraping in 2016

#316

Post by lerpa »

collinmanderson wrote: post #261
tangue wrote: post #247

Earlier quoted context omitted.

Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.

Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)

Exactly, market scrappers as search engines.

And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively.

The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case.

I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.

Re: Web Scraping in 2016

#317

Post by mthoms »

cookiecaper wrote: post #175
mthoms wrote: post #167

Earlier quoted context omitted.

Google's crawling and caching has been largely found to be fair use and thus is not considered to be infringing copyrights. https://en.wikipedia.org/wiki/Field_v._Google,_Inc . There are similar rulings for thumbnail images: https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com... . And of course books: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,... .

Incidentally, this only further proves my point. If you're a big company that's retained massive law firms, you can successfully raise a fair use and implied license defense. If you're not, you can neither mount a strong offense against that defense nor mount a strong defense against Google's hypocritical offense if you find yourself on the other side. Google's primary out here is its reputation (not guarantee) for o…

You claimed "[Google] infringes on copyright as a matter of course" despite the many real civil cases (previously cited) which have found these very activities to be non-infringing. And then, strangely you claimed:

>Fair use is a case-by-case basis, so you can't say that Google's infringing conduct is generally accepted to be fair use.

There is so much wrong with this statement. For one, how can you call something infringing at the same time you point out that nothing has been proven? That simply defies all common logic.

Secondly, in general terms, the activities in question have been found to be non-infringing by the courts. Sure fair-use is case-by-case but if you're operating within similar parameters as a previously litigated case, then the legal risk is immensely reduced.

I don't disagree with your assertion that the legal system greatly favours the well monied/connected (I don't think anyone would). But you can't claim it to be fact that Google Search is infringing anything with little to no evidence or rulings to cite. Unless you're just stating an opinion in which case you should clearly indicate that.

Re: Web Scraping in 2016

#318

Post by cm2187 »

dspillett wrote: post #313
mack73 wrote: post #302

Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court. They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume perso…

> Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Call my cynical if you will, but I'd leave "while abiding the law" out of that, or at least replace it with "while hoping they aren't breaking the law". Due diligence on these matters is often sadly lacking. They'll take the information first and only consider any such implications when/if they come up later. Large or…

I would use Microsoft as a precedent. Sure they will attempt to stay legal but by pushing it as far as they can.

For instance the browser choice script that came with Windows imposed by the EU never worked. It was a "bug". Somehow they must have omitted to test the feature...

Until last year Microsoft started playing nice, and I think Google and Facebook have become the new corporate villains. But recently the Windows team seems to be minded to challenge them in that position.

Re: Web Scraping in 2016

#319

Post by Bootvis »

nerdponx wrote: post #310
mack73 wrote: post #302

Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court. They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume perso…

The difference is that Google didn't agree to not scrape your data. You, as per their TOS, agreed not to scrape theirs, as part of the condition of using their service.

Which TOS?

I might have accepted terms when I created a Google Account but in no way do I agree to a TOS by visiting a URL.

Re: Web Scraping in 2016

#320

Post by someuser18 »

fake-name wrote: post #165
kough wrote: post #124

Earlier quoted context omitted.

Yet another incredible technical achievement due to someone's quest for more porn ( https://github.com/fake-name/AutoTriever/blob/master/setting... ).

That's a separate project: - https://github.com/fake-name/ExHentai-Archival - https://github.com/fake-name/PatreonArchiver - https://github.com/fake-name/xA-Scraper - https://github.com/fake-name/DanbooruScraper Or... well, 4 separate projects. Whoops? At one point, a friend and I were looking at trying to basically replicate the google deep-dream neural net thing, only with a training set of porn. It turns out getti…

At least Ex supports torrents and also has some custom p2p software which you can run (serves content) from which data can be siphoned off.

And what is served through their website is resized. So web-scraping is an inferior approach.

Post reply on HN