Data Infrastructure at IFTTT • Hacker News Board

Re: Data Infrastructure at IFTTT

#21

Post by ambicapter » Thu, Oct 15, 2015, 10:12 PM UTC

> In order to fully trust your data, it is important to have few automatic data verification steps in the flow.

Is this a typo? Did they mean 'a few'?

Re: Data Infrastructure at IFTTT

#22

Post by goyalanuj » Thu, Oct 15, 2015, 11:09 PM UTC

ambicapter wrote: post #21

> In order to fully trust your data, it is important to have few automatic data verification steps in the flow. Is this a typo? Did they mean 'a few'?

We have fixed it! Thank you!

Re: Data Infrastructure at IFTTT

#23

Post by BradRuderman » Thu, Oct 15, 2015, 11:18 PM UTC

How do you feel about the data being delayed, sometimes 1 day? Why not stream the data in realtime to the kafka cluster?

Re: Data Infrastructure at IFTTT

#24

Post by goyalanuj » Thu, Oct 15, 2015, 11:26 PM UTC

BradRuderman wrote: post #23

How do you feel about the data being delayed, sometimes 1 day? Why not stream the data in realtime to the kafka cluster?

Whatever data we need in realtime, we do stream it to the Kafka cluster.

We don't do it for the production database because we don't need it in realtime.

Re: Data Infrastructure at IFTTT

#25

Post by lenguyenthedat » Fri, Oct 16, 2015, 4:11 AM UTC

matsur wrote: post #8

To the author (or anyone else with experience): Any insight on why you guys chose Kafka + Secor over Kinesis?

Amazon Kinesis is a great tool, the main problem is that it's relatively new while Kafka have been around for a few years already.