Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Deactivates Web Search API

ajax.googleapis.com

Post reply on HN
Print view View original

81–90 of 132 posts

Re: Google Deactivates Web Search API

#81

Post by runarb »

jpsaroudakis wrote: post #65
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

> https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs > that are still in use, but will be deprecated in the future. >

I just migrated to the Bing API when Yahoo BOSS closed 31 March :-(

Will there be a new image search api similar to the one you offers now? Can you share any timeframe for the deprecation?

Re: Google Deactivates Web Search API

#82

Post by tokenizerrr »

woodruffw wrote: post #2

Since I can't post both a title and text: What's interesting about this is that the API has been officially deprecated since 2010 (and "offline" since September 2014), but today is the first day that it's actually become unavailable. Edit: Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, e…

> Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, etc) were using this API because of its simplicity (and absence of registration/authentication).

Yep. My harmless little IRC bot can no longer Google search anymore :(

Re: Google Deactivates Web Search API

#83

Post by miduil »

We had some trouble when the Youtube API was disabled for non-registered users. We were using it for some shared & local web-playlist, which is based on youtube-dl/mpv.

Sadly we couldn't use DDG because their API won't let you filter for video results only (Source: Asked on IRC, got a reply from DDG staff + haven't found any documentation mention this either).

We ended up using searx [0] which takes little more time for search results (about 1-3 seconds) but we gain more video-sources (such as vimeo or dailymotion).

I'm aware that youtube-dl got a search functionality build in as well, but the requirement that the client needs to get results without being prepared by our embedded computer wouldn't fit.

[0]: https://github.com/asciimoo/searx

Re: Google Deactivates Web Search API

#84

Post by userbinator »

oneeyedpigeon wrote: post #71
Ixiaus wrote: post #27

Earlier quoted context omitted.

I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha…

God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.

If you have an HTML parser, it's just a matter of selecting the right DOM nodes.

Re: Google Deactivates Web Search API

#85

Post by givan »

With more people placing value on privacy, the api was used by disconnect search along with tor and other privacy tools, google has just planted the seed that will become their biggest competitor.

Re: Google Deactivates Web Search API

#86

Post by Jake232 »

visarga wrote: post #47
Jake232 wrote: post #9

Earlier quoted context omitted.

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of…

> with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily Try putting a site: in the search and it becomes suspicious fast.

You're correct. However for Rank Tracking these types of queries aren't needed.

Re: Google Deactivates Web Search API

#87

Post by RubyPinch »

sdrinf wrote: post #72
visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in: * a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicat…

could embrace it in a weird way

only open source the framework for the server and client

then companies / communities / etc, can make their own algos, and buy their own servers. The reward for proving your servers / crawlers is more people use your algo (higher chance of hitting your nodes)

and then allow the client to have configurable automatic node filtering, along with manual node filtering, so if a person feels that a specific node set is just full of bs, they can filter them out (and also prefer certain node sets in turn, to which they can donate if they are consistently happy with the results)

its like, choose your own filter bubble.

Re: Google Deactivates Web Search API

#88

Post by soared »

ocdtrekkie wrote: post #64
JustUhThought wrote: post #3

#DuckDuckGoLooksUnprofessional It seems pety, but I really feel the duck look is holding it back.

You mean like the original Google home page? With it's clashing bright colored logo and terrible font use? http://www.jaygeiger.com/index.php/2007/01/21/original-googl...

No, Google had a nonsense name with a logo, like Bing. DDG is a silly name with a cartoon duck.

Re: Google Deactivates Web Search API

#89

Post by sourcd »

sdrinf wrote: post #72
visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in: * a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicat…

> submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage

Just a thought : Ranking code could itself learn & adapt to each individual user (the learned "weights" could be sync'd online across your devices). Weighted signals from users can be fed back to the mother ranking algorithm (un-customized one). Basically millions of distributed deep minds[1], instead of a single one.

I can imagine there are a lot of holes in my theory, but we can't simply accept that open sourcing the algorithm implies that it can't be done.

[1] : https://deepmind.com/

Re: Google Deactivates Web Search API

#90

Post by heyalexej »

visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Isn't this how https://www.majestic12.co.uk started?
Post reply on HN