Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Deactivates Web Search API

ajax.googleapis.com

Post reply on HN
Print view View original

71–80 of 132 posts

Re: Google Deactivates Web Search API

#71

Post by oneeyedpigeon »

Ixiaus wrote: post #27
martinald wrote: post #5

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha…

God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.

Re: Google Deactivates Web Search API

#72

Post by sdrinf »

visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Writing an open, and distributed web crawler / indexer is a nice programming exercise.

Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in:

* a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicating this in a distributed manner will be played politically

* Assuming any significant usage, the second you submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage

* Executing custom ranking function on other people's computer not only introduces security risks, but will have scammers setting up honeypots for collecting other people's ranking signals, and playing accordingly.

Re: Google Deactivates Web Search API

#73

Post by tempestn »

jpsaroudakis wrote: post #65
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

Is that 1000 calls per API key, or per IP address? IE if we make client-side calls can we do higher volume?

Re: Google Deactivates Web Search API

#74

Post by masswerk »

Phew! -- Just managed to reactivate some applications[1][2] by providing cached search results.

The Google REST API was a great way to provide audiences with a known tasks in order to connect them to a historic setup or interface. RIP.

[1] http://www.masswerk.at/googleBBS/ [2] http://www.masswerk.at/google60/

Re: Google Deactivates Web Search API

#75

Post by nightcracker »

sdrinf wrote: post #72
visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in: * a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicat…

Obligatory XKCD: https://xkcd.com/810/

Re: Google Deactivates Web Search API

#77

Post by agopaul »

Jake232 wrote: post #9
martinald wrote: post #5

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of…

That is probably why some rank trackers give different results. Some probably use the API and others scrape Google's result page.

Re: Google Deactivates Web Search API

#78

Post by sylvinus »

sourcd wrote: post #56
visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Perhaps something built on top of http://commoncrawl.org/ ? I hope "OpenSearch" becomes a thing, like OpenAI.

Something like http://commonsearch.org ?

Re: Google Deactivates Web Search API

#80

Post by nerdy »

woodruffw wrote: post #21
SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

Google Custom Search[1] is still available, albeit limited to 100 free queries a day (and 10k max, even when paying). That being said, I'm still looking for alternatives as well. This broke my IRC bot and a few scripts rather unexpectedly. [1]: http://www.google.com/cse/

Doesn't this only allow you to search your own page, as mentioned at the URL you linked?

"With Google Custom Search, add a search box to your homepage to help people find what they need on your website." (emphasis mine)

Their video also seems to indicate the same thing: https://youtu.be/Qd9z48Bo8ZA

They say you can search "one website or even a specific topic across multiple websites". This is specifying multiple domains, not a full-index search.

Post reply on HN