Earlier quoted context omitted.
There are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?
Isn't Google search based off of Google "scraping" the web?
Bingo
Web Scraping in 2016
211–220 of 402 posts
Re: Web Scraping in 2016
#212Earlier quoted context omitted.
Except traffic from known scrapers (or what appear to be) is down 20% Sure, xpath and css selector experts can figure it out, but that's not everyone
I don't understand, why only 20%? If the traffic is from known scrapers why can't you just render "scrap off" ie easily get rid of them? And traffic from good scrapers is of course pretty much impossible to measure so you don't know how big percentage of scrapers you got rid of in total.
It's easy for someone viewing some logs to say "ok this is very likely automated scraping", though it can be harder to automate detecting this. In the same way that porn is obvious to a human, but not a computer.
Re: Web Scraping in 2016
#213Earlier quoted context omitted.
That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.
Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it. The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be o…
The protocol is also that you send a valid, non-faked User-Agent:
"The User-Agent request-header field contains information about the user agent originating the request. This is for [...] the tracing of protocol violations [...]. User agents SHOULD include this field with requests"
Many scrapers disregard this part of the protocol. Of course, whether a headless browser should send a different UA is an interesting question.
Re: Web Scraping in 2016
#214Earlier quoted context omitted.
It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.
Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.
With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.
Re: Web Scraping in 2016
#215Earlier quoted context omitted.
How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
> A website is a public property. This isn't even true metaphorically. It's like a shop front: there may be public access, but it is NOT public property.
"public property" may not be the correct metaphor. But neither is "shop front" correct.
Taking the store metaphor further, it would be more like you knocking on the front door of a clothing store and the store owners open the door and throw every possible piece of clothing at you, shirts, shorts, underwear, including coupons to "partner" stores, when all you wanted was a pair of pants.
Upon knocking, if the store owner hands you instructions on how to enter their store and interact with their products in a personalized shopping experience, that would be one thing. But when the clothing owner throws everything at you at once, what they flung at you is for all practical purposes public property.
Re: Web Scraping in 2016
#216Earlier quoted context omitted.
There are already 6-8 major scrapers that do this constantly, across the whole internet, called search engines. You can't handle that? What if you get a normal user who says "Hey, I wanna see some of the lesser known authors on this platform" and opens up a hundred tabs with rarely-read blogs? What if you get 10 users who decide to do that on the same day? Is it reasonable to sue them? Should there be a legal protect…
Search engines respect robots.txt – not sure many scrapers do.
We do, and we also use our own user-agent string: "SiteTruth.com site rating system". A growing number of sites reject connections based on USER-AGENT string. Try "redfin.com", for example. (We list those as "blocked"). Some sites won't let us read the "robots.txt" file. In some cases, the site's USER-AGENT test forbids things the "robots.txt" allows.
Another issue is finding the site's preferred home page. We look at "example.com" and "www.example.com", both with HTTP and HTTPS, trying to find the entry point. This just looks for redirects; it doesn't even read the content. Some sites have redirects from one of those four options to another one. In some cases, the less favored entry point has a "disallow all" robots.txt file. In some cases, the robots.txt file itself is redirected. This is like having doors with various combinations of "Keep Out" and "Please use other door" signs. In that phase, we ignore "robots.txt" but don't read any content beyond the HTTP header.
Some sites treat the four reads to find the home page as a denial of service attack and refuse connections for about a minute.
Then there's Wix. Wix sometimes serves a completely different page if it thinks you're a bot.
Re: Web Scraping in 2016
#217A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr: $.ajaxSetup({ dataFilter: function (data, type) { if (this.url === 'some url that you want to watch!') { // Do anything with the data here awesomeMethod(this.data) } return data } }) I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page do…
You can also build a chrome extension if you need to navigate to multiple pages and use a long-running scraping process. I've done this several times and it's really easy to get one up and running if you use an extension boilerplate (30 minutes tops).
Re: Web Scraping in 2016
#218Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
I actually wonder if that constraining by the ToS has any legal validity in EU. Since in here, typical that kind of stuff cannot be enforced legally.
Re: Web Scraping in 2016
#219Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.
Double standard? The difference is that Google Bot is built on being unobtrusive. I can easily built a scraper that will quickly ddos a site. Linkedin for example...if they allow 10,000 people to send 100 scraping requests per second everyday then that is stolen bandwidth that Linkedin has to pay for and the scrapers get free data. The difference is that Google has standards in which site's unusually benefit from, not to mention that they allow for you to disallow their bot. It just doesn't work the same way with some random developer building a scraper.
Re: Web Scraping in 2016
#220Earlier quoted context omitted.
Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.
With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.
There isn't: the function for this standard includes expected benefit as an input. Every standard has inputs, so that certainly isn't the quality for making something a double standard. The only remaining quality is how unfair it feels, so it would probably be better to just address that, since it is obviously the only thing you disagree about.