Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Deactivates Web Search API

ajax.googleapis.com

Post reply on HN
Print view View original

101–110 of 132 posts

Re: Google Deactivates Web Search API

#101

Post by dchuk »

Ixiaus wrote: post #96
dchuk wrote: post #61

Earlier quoted context omitted.

This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more. Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?

We had a pool of IPs we were leasing on our own, proxy services get abused and are poisoned. We didn't rate limit, we would just increase the size of the cookie pool if a captcha was hit, which was rare because we would scrape n-pages till a threshold was met to prevent that session from being captcha'ed so we wouldn't have to captcha solve it. We had two pools, the primary pool and the "chilling" pool, cookies near…

Thank you for the reply! Sorry but one last question: can you share anymore about this statement "customize the encrypted location key"?

Re: Google Deactivates Web Search API

#102

Post by slig »

runarb wrote: post #81
jpsaroudakis wrote: post #65

Earlier quoted context omitted.

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

> https://datamarket.azure.com/dataset/bing/search . I want to clarify that the page includes the old Bing Search APIs > that are still in use, but will be deprecated in the future. > I just migrated to the Bing API when Yahoo BOSS closed 31 March :-( Will there be a new image search api similar to the one you offers now? Can you share any timeframe for the deprecation?

There's one https://www.microsoft.com/cognitive-services/en-us/bing-imag...

Re: Google Deactivates Web Search API

#103

Post by throwanem »

fixermark wrote: post #99
tokenizerrr wrote: post #82

Earlier quoted context omitted.

> Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, etc) were using this API because of its simplicity (and absence of registration/authentication). Yep. My harmless little IRC bot can no longer Google search anymore :(

Why not? There's a custom search API. https://developers.google.com/custom-search/json-api/v1/over...

Yes, and it's scoped by either domain or (rarely used) types from the schema.org ontology. You can't use it to just search the web.

Re: Google Deactivates Web Search API

#104

Post by slig »

Ixiaus wrote: post #98
oneeyedpigeon wrote: post #71

Earlier quoted context omitted.

God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.

It was indeed pretty rough it wouldn't surprise me if Google moves to js generated dom elements to combat rank trackers, at the time it was fine because they want to service non-js browsers but that might change. Parsing it wasn't hard but it wasn't fun...

Wouldn't it be easier if they generated the dom elements via JS? That would imply that they're getting a JSON or something like, parsing it and creating the DOM.

Re: Google Deactivates Web Search API

#105

Post by DonHopkins »

boyter wrote: post #39
SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

Yandex Bing Gigablast Those are the only web scale indexes that have an API I am aware of these days.

Hoolie's search API hasn't been returning any articles critical of Gavin Belson any more, but it sure is faster now that they're using that new ranking system!

Re: Google Deactivates Web Search API

#107

Post by xbryanx »

woodruffw wrote: post #2

Since I can't post both a title and text: What's interesting about this is that the API has been officially deprecated since 2010 (and "offline" since September 2014), but today is the first day that it's actually become unavailable. Edit: Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, e…

Could you provide an example or two of the sites that will be impacted, and how it will effect an average user? I'm not that familiar with this area.

Re: Google Deactivates Web Search API

#108

Post by fixermark »

throwanem wrote: post #103
fixermark wrote: post #99

Earlier quoted context omitted.

Why not? There's a custom search API. https://developers.google.com/custom-search/json-api/v1/over...

Yes, and it's scoped by either domain or (rarely used) types from the schema.org ontology. You can't use it to just search the web.

You can use it to search the web; you don't get the exact same experience as the old API.

https://support.google.com/customsearch/answer/2631040?hl=en

Re: Google Deactivates Web Search API

#110

Post by skrowl »

dajohnson89 wrote: post #106

What is the rationale for Google doing this? How much money was this service costing them?

I believe the problem was that if you did /google or !google on your bots for various chat platforms, you were able to find the top hit or two without ever looking at Google's Ads.

Most of their revenue is ads.

Post reply on HN