Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Kinesis

aws.typepad.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Amazon Kinesis

#31

Post by idunno246 »

mikebabineau wrote: post #30
kodablah wrote: post #18

Earlier quoted context omitted.

We are about to deploy Kafka in our ecosystem and I am curious what maintenance you have? Can you explain or write a blog post? Is it on 0.8 beta? We are choosing Kafka over other solutions like RabbitMQ because we like the persistent txn-log-style messages and how cheap consumers are.

We're running 0.7 and most of our problems have been around partition rebalancing. I'm not the primary engineer on this, but here's my understanding: If we add nodes to an existing Kafka cluster, those nodes own no partitions and therefore send/receive no traffic. A rebalancing event must occur for these servers to become active. Bouncing Kafka on one of the active nodes is one way to trigger such an event. Fortunate…

Pushing a couple terabytes a day through kafka 0.7. We don't use zookeeper on the producing side and it alleviates this a lot. It's a little more brittle pushing host/partition configs around, but we accepted loss of data in this system and its worth the simplicity of it. Also played with the idea of putting an elb in front.

I'm having way more trouble with the consumer being dumb with the way it distributes topics and partitions. End up with lots of idle consumers, while others are way above max.

Re: Amazon Kinesis

#32

Post by ewoodrich »

rbanffy wrote: post #7
carterschonwald wrote: post #2

Before I clicked the link I was hoping Amazon was releasing a clone of the kinesis keyboard. Anyone else have that initial hope? :-)

I wondered why would Amazon enter the keyboard market...

They already have:

http://www.amazon.com/AmazonBasics-KU-0833-Wired-Keyboard-Bl...

Re: Amazon Kinesis

#33

Post by kylequest »

kylequest wrote: post #21

The Kinesis consumer API is somewhat equivalent to the Simple Consumer API in Kafka. You'll have to manage the consumed sequence number yourself. There's no higher level consumer API to keep track of the consumed sequence numbers.

Looks like AWS decide to put this capability in their Kinesis Client Library, which keeps track of the checkpoints in DynamoDB.

Re: Amazon Kinesis

#35

Post by cypher543 »

ewoodrich wrote: post #32
rbanffy wrote: post #7

Earlier quoted context omitted.

I wondered why would Amazon enter the keyboard market...

They already have: http://www.amazon.com/AmazonBasics-KU-0833-Wired-Keyboard-Bl...

I could be wrong, but I don't think Amazon actually designs or manufactures anything under the AmazonBasics brand. It's like buying a "white box" PC from a company like MSI and reselling it under your own brand name.

Re: Amazon Kinesis

#36

Post by rohitn »

rbanffy wrote: post #7
carterschonwald wrote: post #2

Before I clicked the link I was hoping Amazon was releasing a clone of the kinesis keyboard. Anyone else have that initial hope? :-)

I wondered why would Amazon enter the keyboard market...

Rest assured, Amazon has nothing to do with the AmazonBasics brand.

Re: Amazon Kinesis

#38

Post by mikebabineau »

idunno246 wrote: post #31
mikebabineau wrote: post #30

Earlier quoted context omitted.

We're running 0.7 and most of our problems have been around partition rebalancing. I'm not the primary engineer on this, but here's my understanding: If we add nodes to an existing Kafka cluster, those nodes own no partitions and therefore send/receive no traffic. A rebalancing event must occur for these servers to become active. Bouncing Kafka on one of the active nodes is one way to trigger such an event. Fortunate…

Pushing a couple terabytes a day through kafka 0.7. We don't use zookeeper on the producing side and it alleviates this a lot. It's a little more brittle pushing host/partition configs around, but we accepted loss of data in this system and its worth the simplicity of it. Also played with the idea of putting an elb in front. I'm having way more trouble with the consumer being dumb with the way it distributes topics a…

Thanks for the note, we'll have to take a look at that sort of configuration.

Your consumer problems sounds similar to one we had. Root cause was that the number of consumers exceeded the number of active partitions. The tricky part was that the topic was only distributed across part of the cluster (because of the issue described in my parent post), so we had fewer partitions than we thought.

Re: Amazon Kinesis

#39

Post by fizx »

javajammer wrote: post #24
fizx wrote: post #6

Seems like a useful reworking of SQS, but all the hard work is being done in the client: "client library automatically handle complex issues like adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance." Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vapo…

The currently available docs reveal the client-nodes coordinate through a DynamoDB table. Processing with the library yields "at least once" semantics. http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record...

thanks!
Post reply on HN