Amazon Kinesis
aws.typepad.com
Amazon Kinesis
1–10 of 42 posts
Re: Amazon Kinesis
#2Before I clicked the link I was hoping Amazon was releasing a clone of the kinesis keyboard. Anyone else have that initial hope? :-)
Re: Amazon Kinesis
#3I'm really excited about this - data streaming has been a crucial missing piece for building large-scale apps on AWS.
If the performance and pricing are right it's going to relieve a lot of headaches in terms of infrastructure management.
Re: Amazon Kinesis
#4it is possible that the MD5 hash of your partition keys isn't evenly distributed
how? i mean, apart from poisson stats / shot noise, obviously (and which is noise, so you can't predict it anyway).
thinking some more, i guess this (splitting and merging partitions in a non-generic way) is to handle when a consumer is slow for some reason. perhaps that partition is backing up because the consumer crashed.
but then why not say that, instead of postulating the people are going to have uneven hashes?
[edit:] maybe they allow duplicates?
Re: Amazon Kinesis
#5it is possible that the MD5 hash of your partition keys isn't evenly distributed how? i mean, apart from poisson stats / shot noise, obviously (and which is noise, so you can't predict it anyway). thinking some more, i guess this (splitting and merging partitions in a non-generic way) is to handle when a consumer is slow for some reason. perhaps that partition is backing up because the consumer crashed. but then why…
Yes, duplicates, I think. Looks like the partition key can be set to whatever you want, so maybe you log, I dunno, hits sharded by page, and your homepage gets a ton. I'd lean towards sharding randomly to avoid that, but, eh, they're just giving you enough rope to mess up your logging pipe with.
Re: Amazon Kinesis
#6Seems like a useful reworking of SQS, but all the hard work is being done in the client: "client library automatically handle complex issues like adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance."
Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware.
Re: Amazon Kinesis
#7Before I clicked the link I was hoping Amazon was releasing a clone of the kinesis keyboard. Anyone else have that initial hope? :-)
I wondered why would Amazon enter the keyboard market...
Re: Amazon Kinesis
#8Seems like a useful reworking of SQS, but all the hard work is being done in the client: "client library automatically handle complex issues like adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance." Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vapo…
> Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware.
I think it's unfair to call it vaporware - Amazon doesn't tend to release vaporware. You can also be fairly confident this has been in private beta for some time, so we'll probably see a few blog posts about it from some of their privileged (big spending) clients - typically someone like Netflix or AirBnB. But I agree it would be nice to get some more information on the details.
As for the client library handling load-balancing, fault tolerance, etc - that might not be ideal, but as long as I don't have to do it myself then it might be okay.
Re: Amazon Kinesis
#9What's going on with Amazon recently? We're seeing a torrent of new technologies and platform offerings. Are we finally catching a glimpse of Bezos's grand scheme?
Re: Amazon Kinesis
#10What's going on with Amazon recently? We're seeing a torrent of new technologies and platform offerings. Are we finally catching a glimpse of Bezos's grand scheme?
Well , the new plethora of updates are centered on AWS since AWS:re-invent is going on at present. Historically, that is when Amazon likes to release new AWS services.