Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

271–280 of 402 posts

Re: Web Scraping in 2016

#271

Post by elmigranto »

elchief wrote: post #212
comboy wrote: post #133

Earlier quoted context omitted.

I don't understand, why only 20%? If the traffic is from known scrapers why can't you just render "scrap off" ie easily get rid of them? And traffic from good scrapers is of course pretty much impossible to measure so you don't know how big percentage of scrapers you got rid of in total.

It's easy for someone viewing some logs to say "ok this is very likely automated scraping", though it can be harder to automate detecting this. In the same way that porn is obvious to a human, but not a computer.

But it is not magic, though, and surely can be automated.

In fact, I would argue that your time might be better spent on this, instead of randomizing CSS classes. If you end up building something worthwile, that could be a great product too! (Look at all those CDNs / Anti-DDoS platforms, sounds like they could've been started this exact way.)

Re: Web Scraping in 2016

#272

Post by comex »

cookiecaper wrote: post #111
comex wrote: post #109

Earlier quoted context omitted.

"Clickwrap" refers to situations where you have to click through before using the service, hence the name. Agreements which are simply a passive notice in a footer somewhere are called "browse-wrap", and are much less likely to be considered enforceable: https://en.m.wikipedia.org/wiki/Browse_wrap

The line is blurred between clickwrap and browsewrap -- those are colloquial terms to describe ToS notices, not legal terms. Is it still browsewrap if you say "By clicking any of the links on this site, you agree to the ToS"? How far away from the clickable buttons must the statement be to be browsewrap instead of clickwrap? The distinction is really only a technicality in the wording, not anything substantive. In pr…

Neither of us has explicitly mentioned a jurisdiction, but assuming you, like me, are referring to the United States...

I suppose I can do no better than quote from the Wikipedia page I linked:

> The Second Circuit then noted that an essential ingredient to contract formation is the mutual manifestation of assent. The court found that "a consumer's clicking on a download button does not communicate assent to contractual terms if the offer did not make clear to the consumer that clicking on the download button would signify assent to those terms."

The same page cites a number of cases where a "browsewrap" agreement was found unenforceable and only one where one was found enforceable - and the latter, for what it's worth, involved a sale taking place through the website rather than anything resembling passive browsing. Of course there exist other cases not listed; and there are situations that muddle the distinction between clickwrap and browsewrap. But at least, the very common pattern of, as you said, burying "a notice in the footer of each page" without anything vaguely resembling active consent, as practiced by probably the majority of commercial websites on the internet, seems to pretty clearly fall on the unenforceable side of the line based on those precedents.

Re: Web Scraping in 2016

#273

Post by cookiecaper »

ccvannorman wrote: post #269
franciskim wrote: post #244

Earlier quoted context omitted.

On the ethics side, I don't scrape large amounts of data - eg. giving clients lead gen (x leads for y dollars) - in fact, I have never done a scraping job and don't intend to do those jobs for profit. For me it's purely for personal use and my little side projects. I don't even like the word scraping because it comes loaded with so many negative connotations (which sparked this whole comment thread) - and for a good…

This is exactly my response to "you can't legally scrape my site because of TOS." I don't think anyone has a legal right to tell me HOW I use their service. Making "browsing my website using a script you wrote yourself" illegal is akin to "You cannot use the tab key to tab between fields on my website, you must only use the touchpad to move the cursor over each field individually." It's baloney.

Under the CFAA, they do have the right to determine what constitutes authorized access. If they say you're unauthorized for using the wrong buttons on your mouse, then you're unauthorized. It's treated very similarly to trespass on private land.

You can try telling the judge it's baloney, but if he's going by current precedent, he probably won't agree with you.

Re: Web Scraping in 2016

#274

Post by Guest98123 »

encoderer wrote: post #263
Guest98123 wrote: post #262

Earlier quoted context omitted.

I was curious, so I just checked the stats through webmaster tools. For the last 90 days, the low is 450,000 daily crawled pages, average is 650,000, and yesterday was the high of 1,130,000 (780 per minute). Ouch.

Have you seen correlation between rankings and crawl rate?

This particular site is top 5,000 Alexa. The content changes every minute, and Google is fast at picking up those changes. The last cache of the homepage was 7 minutes ago from Google.

There's definitely a correlation between my sites' Google rankings, their organic traffic, and their crawl rate. The other sites I run are Alexa top 30,000 and top 100,000. They all feature dynamically changing content, but Google is definitely using a higher crawl rate on my higher ranking sites. This isn't a surprise though, Google has limited resources like everyone, and they'll focus those resources in a way that provides the most benefit.

Edit: If you're talking about the correlation between daily ranking and daily crawl rate for an individual site, then no, I'm not aware of any patterns. For example, the graph is flat for organic traffic and total indexed pages, but the crawl rate jumps up and down as mentioned, and it doesn't appear to relate on a daily basis.

Re: Web Scraping in 2016

#276

Post by cookiecaper »

devbug wrote: post #251
greglindahl wrote: post #209

Earlier quoted context omitted.

Actually, such database laws are rare. The US and Canada don't have one. See Feist v. Rural Telephone for an example of databases getting scraped & the scraper winning in court.

Exactly. You can't copyright facts.

You can't copyright facts in the US. You effectively can in the EU, as the grandparent discussed, as long as you demonstrate that it took significant investment to arrange the compendium of facts from which they were drawn.

Re: Web Scraping in 2016

#277

Post by cookiecaper »

greglindahl wrote: post #209
buro9 wrote: post #177

Earlier quoted context omitted.

The UK has a database law: https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright. You may access said database (via the web), but as soon as…

Actually, such database laws are rare. The US and Canada don't have one. See Feist v. Rural Telephone for an example of databases getting scraped & the scraper winning in court.

To be clear, this wasn't a scraper in the networked computer sense. It's actually a perfect example of how meatspace safeguards don't translate because law is not equipped to handle the nature of cyberspace.

Re: Web Scraping in 2016

#278

Post by cookiecaper »

flukus wrote: post #235
buro9 wrote: post #177

Earlier quoted context omitted.

The UK has a database law: https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright. You may access said database (via the web), but as soon as…

What is the definition of "reconstituting a database"? Aren't googles indexes doing that?

Yes. Google is violating practically every law of this type. They're allowed to do it because they have a lot of money.

Re: Web Scraping in 2016

#279

Post by esac »

pault wrote: post #217
prashnts wrote: post #10

A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr: $.ajaxSetup({ dataFilter: function (data, type) { if (this.url === 'some url that you want to watch!') { // Do anything with the data here awesomeMethod(this.data) } return data } }) I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page do…

You can also build a chrome extension if you need to navigate to multiple pages and use a long-running scraping process. I've done this several times and it's really easy to get one up and running if you use an extension boilerplate (30 minutes tops).

do you have something? i was going to write the very same extension (but distributed so i could add it to my pc and my friends) but never did that

Re: Web Scraping in 2016

#280

Post by ccvannorman »

cookiecaper wrote: post #273
ccvannorman wrote: post #269

Earlier quoted context omitted.

This is exactly my response to "you can't legally scrape my site because of TOS." I don't think anyone has a legal right to tell me HOW I use their service. Making "browsing my website using a script you wrote yourself" illegal is akin to "You cannot use the tab key to tab between fields on my website, you must only use the touchpad to move the cursor over each field individually." It's baloney.

Under the CFAA, they do have the right to determine what constitutes authorized access. If they say you're unauthorized for using the wrong buttons on your mouse, then you're unauthorized. It's treated very similarly to trespass on private land. You can try telling the judge it's baloney, but if he's going by current precedent, he probably won't agree with you.

It is not the same flavor of violation as trespassing on land, more like riding a bicycle on a sidewalk. Fortunately, I'm not in the business of scraping sites, but I still find this legal precedent abhorable, and I hope it gets struck down in court when push comes to shove. I would certainly vote that way if given the chance.
Post reply on HN