Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Infrastructure at IFTTT

engineering.ifttt.com

Post reply on HN
Print view View original

1–10 of 25 posts

Re: Data Infrastructure at IFTTT

#3

Post by lemevi »

> Lastly, in order to help monitor the behavior of the hundreds of partner APIs that IFTTT connects to, we collect information about the API requests that our workers make when running Recipes. This includes metrics such as response time and HTTP status codes, and it all gets funneled into our Kafka cluster.

> This way if you query Elasticsearch to find all API errors in the last hour, it can find the answer by looking at a single index, increasing efficiency.

This is a really good way to know if third party APIs are having problems. Staying up to date with all those APIs they support must take up significant amount of engineering effort. Many APIs are just second-class citizens for their product owners. Bugs are introduced, changes are made without announcements, and even if there are announcements when you're dealing with so many different APIs it's hard work keeping track of them all and making changes in your app to keep it running especially when APIs are turned off, or schema changes are happening. This seems to be the hard problem IFTTT is solving, integrating into APIs.

I'd shy away from starting a project that involves so many other companies APIs just because of how hard of a problem that is to manage, but IFTTT is doing a great job here.

Re: Data Infrastructure at IFTTT

#5

Post by danso »

jreed91 wrote: post #4
SEJeff wrote: post #2

Original link: http://engineering.ifttt.com/data/2015/10/14/data-infrastruc... Can a moderator such as dang fix this please?

He is probably just syndicating this to medium with the new api's introduced.

To hijack your tongue-in-cheek comment...I genuinely would appreciate it if someone, at some point, looked at analytics before and after syndicating to Medium to not just compare number of visitors between the original blog and Medium, but whether PageRank for the original blog dropped due to duplication penalties applied by Google's algorithm.

Re: Data Infrastructure at IFTTT

#6

Post by minimaxir »

danso wrote: post #5
jreed91 wrote: post #4

Earlier quoted context omitted.

He is probably just syndicating this to medium with the new api's introduced.

To hijack your tongue-in-cheek comment...I genuinely would appreciate it if someone, at some point, looked at analytics before and after syndicating to Medium to not just compare number of visitors between the original blog and Medium, but whether PageRank for the original blog dropped due to duplication penalties applied by Google's algorithm.

It's worth nothing that Medium described this syndication workflow as an expected use case during the announcement.

I don't think they would have done that if there was a knowing SEO downside. It would hurt everyone.

Re: Data Infrastructure at IFTTT

#9

Post by danso »

minimaxir wrote: post #6
danso wrote: post #5

Earlier quoted context omitted.

To hijack your tongue-in-cheek comment...I genuinely would appreciate it if someone, at some point, looked at analytics before and after syndicating to Medium to not just compare number of visitors between the original blog and Medium, but whether PageRank for the original blog dropped due to duplication penalties applied by Google's algorithm.

It's worth nothing that Medium described this syndication workflow as an expected use case during the announcement. I don't think they would have done that if there was a knowing SEO downside. It would hurt everyone.

I want to take them at their word...but it's more up to Google, isn't it? And doesn't Google still rely (in part) on the canonical meta tag to resolve duplicates? Currently, the posted Medium post has this:
It would be relatively easy for Medium to appropriately set that tag -- I mean, if "original URL" is captured somewhere in the API, or in the Medium post-create-admin interface (I don't know, I haven't logged into my own Medium for awhile)...that would explicitly resolve the ambiguity, though at an obvious cost to their own SEO.
Post reply on HN