The Kinesis consumer API is somewhat equivalent to the Simple Consumer API in Kafka. You'll have to manage the consumed sequence number yourself. There's no higher level consumer API to keep track of the consumed sequence numbers.
Amazon Kinesis
21–30 of 42 posts
Re: Amazon Kinesis
#22Re: Amazon Kinesis
#23Earlier quoted context omitted.
> Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware. I think it's unfair to call it vaporware - Amazon doesn't tend to release vaporware. You can also be fairly confident this has been in private beta for some time, so we'll probably see a few blog posts about it from some of their privileged (big spending) clients - typically someone…
The client handling it is ideal from a systems perspective, because the app won't forget to be fault tolerant on its connection to the server. Its less ideal from a maintenance perspective, because there will need to be feature-rich clients in Java and C (with dynamic language bindings). Applications will be running many many versions of the clients. Also, for coordination, the clients will need to communicate, so th…
It's not exactly the first time, but close - the Simple Workflow Service has client helper libraries for both Java and Ruby.
Re: Amazon Kinesis
#24Seems like a useful reworking of SQS, but all the hard work is being done in the client: "client library automatically handle complex issues like adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance." Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vapo…
The currently available docs reveal the client-nodes coordinate through a DynamoDB table. Processing with the library yields "at least once" semantics.
http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record...
Re: Amazon Kinesis
#25Re: Amazon Kinesis
#26[deleted]
Re: Amazon Kinesis
#27Can someone with enough knowledge give a high level comparison to Kinesis compared with something like Storm or Kafka?
Re: Amazon Kinesis
#28As a team member helped built the service, I would like to offer some of my personal understanding. I am not with Amazon now, and all my views are based on public information on the website.
Like all AWS offerings, Kinesis is a platform. It looks like kafka + storm, with fully integrated ecosystem with other AWS services. From the very beginning, the reliability, real-time processing, and transparent elasticity are built in. That's all I can say.
Re: Amazon Kinesis
#29What's going on with Amazon recently? We're seeing a torrent of new technologies and platform offerings. Are we finally catching a glimpse of Bezos's grand scheme?
Bezos said recently that he thinks AWS could be Amazon's biggest business. http://techcrunch.com/2013/11/13/jeff-bezos-believes-aws-cou...
From the press conference reported in the link: "Jeff is very excited about the AWS business and he believes - like the rest of the leadership team does – that in the fullness of time- it is very possible that AWS could be the biggest business at Amazon."
Re: Amazon Kinesis
#30This is essentially a hosted Kafka ( http://kafka.apache.org/ ). Given the complexity of operating a distributed persistent queue, this could be a compelling alternative for AWS-centric environments. (We run a large Kafka cluster on AWS, and it is one of our highest-maintenance services.)
We are about to deploy Kafka in our ecosystem and I am curious what maintenance you have? Can you explain or write a blog post? Is it on 0.8 beta? We are choosing Kafka over other solutions like RabbitMQ because we like the persistent txn-log-style messages and how cheap consumers are.
We're running 0.7 and most of our problems have been around partition rebalancing. I'm not the primary engineer on this, but here's my understanding:
If we add nodes to an existing Kafka cluster, those nodes own no partitions and therefore send/receive no traffic. A rebalancing event must occur for these servers to become active. Bouncing Kafka on one of the active nodes is one way to trigger such an event.
Fortunately, cluster resizing is infrequent. Unfortunately, network interruptions are not (at least on EC2).
When ZooKeeper detects a node failure (however brief), the node is removed from the active pool and the partitions are rebalanced. This is desirable. But when the node comes back online, no rebalancing takes place. The server remains inactive (as if it were a new node) until we trigger a rebalancing event.
As a result, we have to bounce Kafka on an active server every few weeks in response to network blips. 0.8 alleges to handle this better, but we'll see.
Handle-jiggling aside, I'm a fan of Kafka and the types of systems you can build around it. Happy to put you in touch with our Kafka guy, just email me (mike.babineau@rumblegames.com). Loggly's also running Kafka on AWS - would be interesting to hear their take on this.