Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Deactivates Web Search API

ajax.googleapis.com

Post reply on HN
Print view View original

41–50 of 132 posts

Re: Google Deactivates Web Search API

#41

Post by boyter »

pajop wrote: post #38
SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

the first thing that came to mind was DuckDuckGo :) https://duckduckgo.com/api

Its not a web index. They are unable to offer that as a lot of their results come from external services such as Yandex and Bing.

The results are their zeroclick info snippets and it is very useful for pulling an answer from wikipedia since they have already parsed it.

Re: Google Deactivates Web Search API

#42

Post by jrochkind1 »

pajop wrote: post #38
SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

the first thing that came to mind was DuckDuckGo :) https://duckduckgo.com/api

It has no web search api.

Re: Google Deactivates Web Search API

#43

Post by SwellJoe »

bake wrote: post #20
SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

I'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?

Re: Google Deactivates Web Search API

#44

Post by jrochkind1 »

bake wrote: post #20
SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search.

Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.

Re: Google Deactivates Web Search API

#45

Post by woodruffw »

jrochkind1 wrote: post #44
bake wrote: post #20

Earlier quoted context omitted.

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

>Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search.

You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209

Re: Google Deactivates Web Search API

#46

Post by dwynings »

SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

http://gigablast.com/searchfeed.html

Re: Google Deactivates Web Search API

#47

Post by visarga »

Jake232 wrote: post #9
martinald wrote: post #5

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of…

> with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily

Try putting a site: in the search and it becomes suspicious fast.

Re: Google Deactivates Web Search API

#48

Post by visarga »

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor.

A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2000, we have access to scientific papers, cheap cloud servers and a huge interest in freeing search, so I think the open source community do it.

Re: Google Deactivates Web Search API

#49

Post by genop »

The YouTube API used to be open too - no regstration or authentication. That ended years ago. I don't really need an "API" to do searches and transform the results into a more useful format than HTML. These API's make it easier and are addictive, for sure. It's foolish to rely on them however.

Re: Google Deactivates Web Search API

#50

Post by jrochkind1 »

woodruffw wrote: post #45
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

>Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209

Thanks. One of the comments on that SO answer from Aug 2015 says:

> WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites"

I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophole that google might close without warning.

But if someone has actually done this succesfully, with a paid account for more than 100/queries a day, I would definitely be interested in hearing about it!

Post reply on HN