This needs our love & support : commonsearch.org
Prev discussion (few weeks back) https://news.ycombinator.com/item?id=11281700
Credit : @sylvinus
Google Deactivates Web Search API
91–100 of 132 posts
Re: Google Deactivates Web Search API
#92Earlier quoted context omitted.
Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…
> https://datamarket.azure.com/dataset/bing/search . I want to clarify that the page includes the old Bing Search APIs > that are still in use, but will be deprecated in the future. > I just migrated to the Bing API when Yahoo BOSS closed 31 March :-( Will there be a new image search api similar to the one you offers now? Can you share any timeframe for the deprecation?
https://www.microsoft.com/cognitive-services/en-us/pricing does seem to indicate that there will be a Bing Image Search API under the new umbrella.
Re: Google Deactivates Web Search API
#93Earlier quoted context omitted.
Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…
Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.
Agreed -- 1k/mo is not worth the initial time investment.
Re: Google Deactivates Web Search API
#94Earlier quoted context omitted.
This is actually the second time they have reduced their Web API for search results. Google used to offer a very full featured REST API, but then took it offline and replaced it with a weak AJAX one: http://googlesystem.blogspot.ca/2008/04/google-search-rest-a... You could actually go cool things with the original API. I used it to power a predecessor to https://friskr.com , but we had to switch web search engines af…
That's interesting, I had no idea. It's a shame that the trend at Google has been towards closing down APIs like these in favor of specialized/pay-as-you-go endpoints.
I feel like that's a larger trend. APIs are introduced to drive platform growth and then pulled or cut back once that growth has been achieved. Twitter is another example of that phenomenon.
Re: Google Deactivates Web Search API
#95I believe Google will replace it — the Google Search Appliance is going away too. Something will pick up those customers. An announcement at I/O, which is pretty soon?
It has been deprecated for years, disallowing new customers.
No sign of any replacement
Re: Google Deactivates Web Search API
#96Earlier quoted context omitted.
I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha…
This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more. Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?
We had a pool of IPs we were leasing on our own, proxy services get abused and are poisoned.
We didn't rate limit, we would just increase the size of the cookie pool if a captcha was hit, which was rare because we would scrape n-pages till a threshold was met to prevent that session from being captcha'ed so we wouldn't have to captcha solve it. We had two pools, the primary pool and the "chilling" pool, cookies near their captcha life would cool off for a few hours before returning to the active pool which behaves just like any other resource pool, every page scraped would "borrow" a cookie out of the pool, customize the encrypted location key, and make the request with a common user agent string.
Scaling it was difficult but once we had it figured out, Erlang was invaluable to us and our dependence on IPs dropped once we figured out the cookie methodology.
Solving captchas is cheaper than renting IPs.
Re: Google Deactivates Web Search API
#97Earlier quoted context omitted.
I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha…
This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more. Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?
Oh and I just used a generic HTTP request client in Erlang and xpath / HTML parser to extract what we needed.
Re: Google Deactivates Web Search API
#98Earlier quoted context omitted.
I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha…
God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.
It was indeed pretty rough it wouldn't surprise me if Google moves to js generated dom elements to combat rank trackers, at the time it was fine because they want to service non-js browsers but that might change.
Parsing it wasn't hard but it wasn't fun...
Re: Google Deactivates Web Search API
#99Since I can't post both a title and text: What's interesting about this is that the API has been officially deprecated since 2010 (and "offline" since September 2014), but today is the first day that it's actually become unavailable. Edit: Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, e…
> Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, etc) were using this API because of its simplicity (and absence of registration/authentication). Yep. My harmless little IRC bot can no longer Google search anymore :(
Why not? There's a custom search API.
https://developers.google.com/custom-search/json-api/v1/over...
Re: Google Deactivates Web Search API
#100Earlier quoted context omitted.
God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.
If you have an HTML parser, it's just a matter of selecting the right DOM nodes.
Yes, but if your markup is spaghetti, "selecting the right DOM nodes" becomes a lot more difficult.