Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Deactivates Web Search API

ajax.googleapis.com

Post reply on HN
Print view View original

61–70 of 132 posts

Re: Google Deactivates Web Search API

#61

Post by dchuk »

Ixiaus wrote: post #27
martinald wrote: post #5

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve. It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha…

This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more.

Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?

Re: Google Deactivates Web Search API

#62

Post by bake »

jrochkind1 wrote: post #44
bake wrote: post #20

Earlier quoted context omitted.

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Here's the older link to the Bing Web Search API -- while the new page doesn't list actual prices, this one does in nice detail:

https://datamarket.azure.com/dataset/bing/searchweb#terms

Re: Google Deactivates Web Search API

#63

Post by bake »

phatrice wrote: post #54
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

You can check out the new home of Bing Search API here: https://www.microsoft.com/cognitive-services/en-us/bing-web-... You can get key right from there and an API console.

Here's the old home, with pricing laid out (missing from the new one): https://datamarket.azure.com/dataset/bing/searchweb#terms

Re: Google Deactivates Web Search API

#64

Post by ocdtrekkie »

JustUhThought wrote: post #3

#DuckDuckGoLooksUnprofessional It seems pety, but I really feel the duck look is holding it back.

You mean like the original Google home page? With it's clashing bright colored logo and terrible font use? http://www.jaygeiger.com/index.php/2007/01/21/original-googl...

Re: Google Deactivates Web Search API

#65

Post by jpsaroudakis »

jrochkind1 wrote: post #44
bake wrote: post #20

Earlier quoted context omitted.

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services. All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pricing plan if you want to extend the monthly quota.

Some folks mentioned this page: https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs that are still in use, but will be deprecated in the future.

You can send feedback or questions here: https://cognitive.uservoice.com/

[Edit] Here is also a blog post with useful resources: https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/re...

Re: Google Deactivates Web Search API

#66

Post by hantusk »

jrochkind1 wrote: post #50
woodruffw wrote: post #45

Earlier quoted context omitted.

>Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209

Thanks. One of the comments on that SO answer from Aug 2015 says: > WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophol…

I am doing this successfully with 1,000 queries in a day, and the limit it 10k queries per day (pay per 1,000 queries)

The billing system is confusing. You just have to sign up for billing, and you can raise the limit.

The upgrade in the google system, is only if you want site search with no ads.

Re: Google Deactivates Web Search API

#67

Post by SwellJoe »

jpsaroudakis wrote: post #65
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

I keep checking out the Bing API every couple months, hoping for sort-by options (I need last changed date, most recent first). Any hope of ever seeing sort-by-date from Bing? Google has it, I don't think anyone else does.

Re: Google Deactivates Web Search API

#68

Post by tokensimian »

jpsaroudakis wrote: post #65
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause.

I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.

Post reply on HN