Data Infrastructure at IFTTT • Hacker News Board

Re: Data Infrastructure at IFTTT

#11

Post by goyalanuj » Thu, Oct 15, 2015, 6:18 PM UTC

jonathankoren wrote: post #10

matsur wrote: post #8

To the author (or anyone else with experience): Any insight on why you guys chose Kafka + Secor over Kinesis?

I worked with Anuj at LinkedIn, so I'm thinking the most pedestrian answer is that Kafka works, and it's a familiar tool.

Thank you Jonathan!

As Jonathan mentioned, we made this decision around 9 months back and at that time Kinesis wasn't as mature and had less flexibility around retention period etc.

Kafka is very reliable (as I had seen it handling billions of events a day at LinkedIn) and has a huge open-source community around it. At IFTTT, we always prefer to use and contribute to open source ( http://engineering.ifttt.com/oss/2015/07/23/open-source/ ).

Re: Data Infrastructure at IFTTT

#12

Post by cheetos » Thu, Oct 15, 2015, 6:53 PM UTC

Anyone have an idea why they would use MySQL over PostgreSQL?

Re: Data Infrastructure at IFTTT

#13

Post by toomuchtodo » Thu, Oct 15, 2015, 7:21 PM UTC

cheetos wrote: post #12

Anyone have an idea why they would use MySQL over PostgreSQL?

MySQL works just as well for most use cases.

Re: Data Infrastructure at IFTTT

#14

Post by Splendor » Thu, Oct 15, 2015, 7:39 PM UTC

matsur wrote: post #8

To the author (or anyone else with experience): Any insight on why you guys chose Kafka + Secor over Kinesis?

Things might be different now that Kinesis Firehose has been announced. But Kinesis alone is not easier to manage than Kafka.

Re: Data Infrastructure at IFTTT

#15

Post by lgas » Thu, Oct 15, 2015, 7:46 PM UTC

cheetos wrote: post #12

Anyone have an idea why they would use MySQL over PostgreSQL?

The usual reason is having more experience with MySQL than PostgreSQL.

Re: Data Infrastructure at IFTTT

#16

Post by ronald_dregan » Thu, Oct 15, 2015, 8:10 PM UTC

ifttt's ios app location triggers are fucking awful and insanely frustrating

Re: Data Infrastructure at IFTTT

#17

Post by johns » Thu, Oct 15, 2015, 8:11 PM UTC

lemevi wrote: post #3

> Lastly, in order to help monitor the behavior of the hundreds of partner APIs that IFTTT connects to, we collect information about the API requests that our workers make when running Recipes. This includes metrics such as response time and HTTP status codes, and it all gets funneled into our Kafka cluster. > This way if you query Elasticsearch to find all API errors in the last hour, it can find the answer by looki…

I used to work at IFTTT and this exact set of problems is why I left to start a company. Pretty much everything I've done in the intervening years has stemmed from exactly what you describe. It's great to see the IFTTT team tackle this stuff head on.

Re: Data Infrastructure at IFTTT

#18

Post by ddollar » Thu, Oct 15, 2015, 8:15 PM UTC

If you're using ECS to manage your clusters on AWS take a look at Convox. We're building an open source platform that makes building, versioning, and deploying code to AWS via ECS incredibly easy.

https://convox.com

Re: Data Infrastructure at IFTTT

#19

Post by dang » Thu, Oct 15, 2015, 8:33 PM UTC

SEJeff wrote: post #2

Original link: http://engineering.ifttt.com/data/2015/10/14/data-infrastruc... Can a moderator such as dang fix this please?

Yes. Url changed to that from https://medium.com/engineering-at-ifttt/data-infrastructure-....

Medium's republishing API has already become a problem for original sources on HN.

p.s. Comments in the threads are an unreliable way to reach us. There are too many for us to see them all. The reliable way is to email hn@ycombinator.com.

Re: Data Infrastructure at IFTTT

#20

Post by dekayed » Thu, Oct 15, 2015, 9:45 PM UTC

goyalanuj wrote: post #11

jonathankoren wrote: post #10

Earlier quoted context omitted.

I worked with Anuj at LinkedIn, so I'm thinking the most pedestrian answer is that Kafka works, and it's a familiar tool.

Thank you Jonathan! As Jonathan mentioned, we made this decision around 9 months back and at that time Kinesis wasn't as mature and had less flexibility around retention period etc. Kafka is very reliable (as I had seen it handling billions of events a day at LinkedIn) and has a huge open-source community around it. At IFTTT, we always prefer to use and contribute to open source ( http://engineering.ifttt.com/oss/201…

I'm assuming that you run Kafka within AWS. Much of the hardware requirements/suggestions I've seen for Kafka are all for non-virtualized environments. If you can get into it, could you share some details...

- What is the size of your Kafka cluster

- What instances types do you use?

- Do you use EBS or use ephemeral storage?

- How much do you over-provision to deal with instance loss?

- Any other gotchas/considerations?

Thanks!