Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Deactivates Web Search API

ajax.googleapis.com

Post reply on HN
Print view View original

121–130 of 132 posts

Re: Google Deactivates Web Search API

#121

Post by [deleted] »

tokensimian wrote: post #68
jpsaroudakis wrote: post #65

Earlier quoted context omitted.

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.

[deleted]

Re: Google Deactivates Web Search API

#122

Post by jpsaroudakis »

runarb wrote: post #81
jpsaroudakis wrote: post #65

Earlier quoted context omitted.

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

> https://datamarket.azure.com/dataset/bing/search . I want to clarify that the page includes the old Bing Search APIs > that are still in use, but will be deprecated in the future. > I just migrated to the Bing API when Yahoo BOSS closed 31 March :-( Will there be a new image search api similar to the one you offers now? Can you share any timeframe for the deprecation?

As others pointed out, you can find information about the new Bing Image API here: https://www.microsoft.com/cognitive-services/en-us/bing-imag.... I cannot provide an official deprecation date yet, but here's a rough timeline: we will first announce the pricing plan for the new APIs in early-to-mid summer. We will then provide a notice to existing users of the old APIs to switch to the newer version within 6-12 months - note again that the dates are not final.

Re: Google Deactivates Web Search API

#123

Post by jpsaroudakis »

tokensimian wrote: post #68
jpsaroudakis wrote: post #65

Earlier quoted context omitted.

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.

Thanks for the feedback. We do not have any immediate plans to change the call limit for the preview offering. We've seen from our data that 1K calls / month are sufficient for most developers who want to get started using the APIs. We we will certainly consider revising this in the future if there's strong demand. I encourage you to file this as a request here: https://cognitive.uservoice.com. We review user ideas and feedback there on a regular basis to improve our services. Thanks again!

Re: Google Deactivates Web Search API

#124

Post by jrochkind1 »

jpsaroudakis wrote: post #65
jrochkind1 wrote: post #44

Earlier quoted context omitted.

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous…

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

This is very interesting functionality to me, that I would use if it works well. Thanks for trying to clarify what's going on with it.

But when between one fairly poorly documented API that you've told me will soon be deprecated, and another fairly poorly documented API that doesn't have pricing beyond 1K calls a month....

...my response is still "Okay, that could be good in the future, I guess I'll sit back some more and wait for pricing and better docs to show up before paying more attention to it, I hope it does soon!"

I need to know pricing before deciding to use something. And I need good (and google-able! Sorry, even if I'm using Bing API, I'm using Google to look for info on it) docs, along with of course a well-designed API that works well, but I need the first two things before I even get to evaluating the next.

Re: Google Deactivates Web Search API

#125

Post by Razengan »

sdrinf wrote: post #72
visarga wrote: post #48

We need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2…

Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in: * a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicat…

> Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure.

Why not just take humanity out of that picture, then?

With the current AI/deep learning hype everywhere, why not start developing an AI-driven search system?

I think for it to produce the most relevant results, it will need access to your browser (or be a browser) or better yet, work at the OS level, so it can have a better idea of the current context you're working in, and learn from your habits and preferences. Say I'm coding and have an IDE and a bunch of dev-related websites already open, so the AI gives more weight to development-related results. If I've been playing a certain game a lot then it should assume that I'll be looking for stuff related to that game. And so on.

So, the index would be globally accessible to all computers, but the ranking will be unique to each individual user.

Something like this could very well be the actual beginning of a true A.I. "butler," more so than Siri and whatnot.

Re: Google Deactivates Web Search API

#126

Post by jpsaroudakis »

tempestn wrote: post #73
jpsaroudakis wrote: post #65

Earlier quoted context omitted.

Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services . All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pri…

Is that 1000 calls per API key, or per IP address? IE if we make client-side calls can we do higher volume?

per API key

Re: Google Deactivates Web Search API

#127

Post by Ixiaus »

slig wrote: post #104
Ixiaus wrote: post #98

Earlier quoted context omitted.

It was indeed pretty rough it wouldn't surprise me if Google moves to js generated dom elements to combat rank trackers, at the time it was fine because they want to service non-js browsers but that might change. Parsing it wasn't hard but it wasn't fun...

Wouldn't it be easier if they generated the dom elements via JS? That would imply that they're getting a JSON or something like, parsing it and creating the DOM.

No because then you'd have to use a headless browser that can execute js. That increases time and cost when scraping, though it wouldn't surprise me if it ends up going that way.

Re: Google Deactivates Web Search API

#128

Post by Ixiaus »

dchuk wrote: post #101
Ixiaus wrote: post #96

Earlier quoted context omitted.

We had a pool of IPs we were leasing on our own, proxy services get abused and are poisoned. We didn't rate limit, we would just increase the size of the cookie pool if a captcha was hit, which was rare because we would scrape n-pages till a threshold was met to prevent that session from being captcha'ed so we wouldn't have to captcha solve it. We had two pools, the primary pool and the "chilling" pool, cookies near…

Thank you for the reply! Sorry but one last question: can you share anymore about this statement "customize the encrypted location key"?

When you set the location in Google it customizes the cookie with a named field that is a capital L I believe. That field is encoded or encrypted and I could never figure it out so I just constructed a rainbow table by using phantomjs to set a location and scrape the cookie out, pairing the known location value with the encrypted value so that we could customize "the location of the search".

Re: Google Deactivates Web Search API

#129

Post by slig »

Ixiaus wrote: post #127
slig wrote: post #104

Earlier quoted context omitted.

Wouldn't it be easier if they generated the dom elements via JS? That would imply that they're getting a JSON or something like, parsing it and creating the DOM.

No because then you'd have to use a headless browser that can execute js. That increases time and cost when scraping, though it wouldn't surprise me if it ends up going that way.

Why execute the JS? Just parse the JSON (I'm just guessing here). No need to mess with DOM or JS.

Re: Google Deactivates Web Search API

#130

Post by Ixiaus »

slig wrote: post #129
Ixiaus wrote: post #127

Earlier quoted context omitted.

No because then you'd have to use a headless browser that can execute js. That increases time and cost when scraping, though it wouldn't surprise me if it ends up going that way.

Why execute the JS? Just parse the JSON (I'm just guessing here). No need to mess with DOM or JS.

From my last and admittedly cursory look it wasn't that simple. IIRC JSON wasn't even used, I think it was a binary protocol.
Post reply on HN