Be careful. I got banned from Google for scraping. I did a few hundred thousand searches one day, and that night, they banned my office IP address for a week. This was in 2001, so I estimate I cost them a few hundred dollars, which is now impossible to repay. :(
I Don’t Need No Stinking API: Web Scraping For Fun and Profit
171–176 of 176 posts
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#172Earlier quoted context omitted.
They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against. If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping. There would be lots of oth…
Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.
Wikipedia explicitly allows that kind of thing with CC-BY-SA licenses, and indeed gets substantial funding from companies like answers.com that do it. (Incidentally, answers.com was the only way to see TeX equations on Wikipedia on my Android phone last time I checked, so it's not like they're adding no value.)
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#173Earlier quoted context omitted.
Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.
Yeah, Wikipedia is creative commons so that should be okay? You are right though I wonder if they have the rights to sports results and weather that they are pulling. They have even convinced us all to go mark up our page to help them pull stuff like ratings and reviews out.
Sports results are facts and are statutorily not subject to copyright in the US.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#174From a site owner's perspective: if you have a LOT of data then scraping can be very disruptive. I've had someone scraping my site for literally months, using hundreds of different open proxies, plus multiple faked user-agents, in order to defeat scraping detection. At one point they were accessing my site over 300,000 times per day (3.5/sec), which exceeded the level of the next busiest (and welcome) agent... Googlebot. In total I estimate this person has made more than 30 million fetch attempts over the past few months. I eventually figured out a unique signature for their bot and blocked 95%+ of their attempts, but they still kept trying. I managed to find a contact for their network administrator and the constant door-knocking finally stopped today.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#175There is just one major trouble with not needing stinking API. You can not POST as a possible client without requiring them to give their password to you, which actually would give you full access to their account instead of limited access with API.
You seem to be talking about a specific site? Which one?
Any social network where you can post messages for the user in the users message stream.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#176I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.
I've written some in-browser JS to download all my tweets without needing to resort to the API and oAuth nightmare. It is indeed possible to write a client, but not recommended at all...
how did you send cross domain request?