Google Deactivates Web Search API • Hacker News Board

Re: Google Deactivates Web Search API

#21

Post by woodruffw » Wed, May 04, 2016, 3:19 AM UTC

SwellJoe wrote: post #16

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

Google Custom Search[1] is still available, albeit limited to 100 free queries a day (and 10k max, even when paying).

That being said, I'm still looking for alternatives as well. This broke my IRC bot and a few scripts rather unexpectedly.

[1]: http://www.google.com/cse/

Re: Google Deactivates Web Search API

#22

Post by jonlucc » Wed, May 04, 2016, 3:19 AM UTC

mozumder wrote: post #10

Why do API developers add a HTTP status code in their JSON response, when the HTTP status code is already in the header?

I think it's to simplify as much as possible for clients. If a dev wants to use the API, it is in the interest of the API maintainer to make it easy, including pitting all the useful info in a single place. I was just watching an old talk about this a week ago from StormPath (iirc). They also recommend at least offering a trigger for including other redundant data like the path for subsequent sets of paginated data and an additional method for adding data via an URL flag instead of using PUT in case it isn't available.

Re: Google Deactivates Web Search API

#23

Post by jsmthrowaway » Wed, May 04, 2016, 3:19 AM UTC

woodruffw wrote: post #14

mozumder wrote: post #10

Why do API developers add a HTTP status code in their JSON response, when the HTTP status code is already in the header?

Probably because HTTP status codes are perceived to be at a different layer of the response than the body. A lot of developers (myself included, for trivial things) treat HTTP bodies as strings with no associated metadata.

It's debatable, but that's probably antithetical to how HTTP is supposed to be perceived when you start thinking about what a body is.

Re: Google Deactivates Web Search API

#24

Post by hn_username » Wed, May 04, 2016, 3:20 AM UTC

Jake232 wrote: post #9

martinald wrote: post #5

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of…

Any additional tricks you want to share and/or cheap proxy services you'd recommend? Do you make use of selenium and/or phantomjs?

Re: Google Deactivates Web Search API

#25

Post by reefoctopus » Wed, May 04, 2016, 3:22 AM UTC

bake wrote: post #17

Does anyone know of an API that returns pages related to any given URL? I'm looking for an alternative to Google's "related:..." search, which was only reliably accessible through Google Web Search (Google Custom Search is very expensive and has a limit of 10K queries per day). For example: https://www.google.com/#q=related:http:%2F%2Fagiliq.com%2Fbl... It returns pages related to any given page, and works at the spe…

I ran into this problem several years ago while writing SEO software. There are a bunch of companies who provide google search results, but there's usually at least a couple minute delay between when the data is requested and when it arrives at your callback page.

I had success with Authority Labs. You might want to check them out. They'll raise the query limit if you ask them.

Re: Google Deactivates Web Search API

#26

Post by woodruffw » Wed, May 04, 2016, 3:22 AM UTC

jsmthrowaway wrote: post #23

woodruffw wrote: post #14

Earlier quoted context omitted.

Probably because HTTP status codes are perceived to be at a different layer of the response than the body. A lot of developers (myself included, for trivial things) treat HTTP bodies as strings with no associated metadata.

It's debatable, but that's probably antithetical to how HTTP is supposed to be perceived when you start thinking about what a body is .

I would tend to agree. Speaking beyond convenience, this isn't a practice that's easy to justify ;)

Re: Google Deactivates Web Search API

#27

Post by Ixiaus » Wed, May 04, 2016, 3:22 AM UTC

martinald wrote: post #5

Slightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.

I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve.

It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha solving. Then when you go to actually scrape google you can balance your pool of IPs and cookies, chilling them before they trigger another captcha, to handle spiking demand. Contant, sustained demand was very easy to plan for.

Anyway, we stopped using the API after the third day and realized it was not only not accurate and you couldn't turn many knobs (want to scrape results as they look for a different geographical region?)

Re: Google Deactivates Web Search API

#28

Post by partiallypro » Wed, May 04, 2016, 3:23 AM UTC

bake wrote: post #17

Does anyone know of an API that returns pages related to any given URL? I'm looking for an alternative to Google's "related:..." search, which was only reliably accessible through Google Web Search (Google Custom Search is very expensive and has a limit of 10K queries per day). For example: https://www.google.com/#q=related:http:%2F%2Fagiliq.com%2Fbl... It returns pages related to any given page, and works at the spe…

While you're looking for this feature in the meantime, ping the Bing team and ask for it as an added function of their API. Microsoft is a lot more open to feedback these days.

Re: Google Deactivates Web Search API

#29

Post by nfriedly » Wed, May 04, 2016, 3:27 AM UTC

I wrote a PageRank library a while back and just got a report yesterday that it also no longer works.

(PageRank never actually had an official API, but it was exposed for the Google toolbar. They had announced that this would go away a while back but just dropped the axe in the last day or two.)

Re: Google Deactivates Web Search API

#30

Post by woodruffw » Wed, May 04, 2016, 3:28 AM UTC

nfriedly wrote: post #29

I wrote a PageRank library a while back and just got a report yesterday that it also no longer works. (PageRank never actually had an official API, but it was exposed for the Google toolbar. They had announced that this would go away a while back but just dropped the axe in the last day or two.)

I guess they're cleaning house.