21–30 of 33 posts
Hmmmm, I certainly hope dang or another admin consolidate all these stories into a single "Amazon" thread. Can't have multiple stories from a single company eating up space on the front page.
I have a question: if I were to send say 10000 event objects into a Amazon Kinesis Firehose stream it's clear that I should expect them to show up in an S3 bucket of my choosing, but should I also expect that my account will not incur any S3 HTTP POST API request fees ? Is dodging those HTTP POST fees the value-add over simply using the S3 HTTP API yourself ?
Unless I am reading it wrong, it sounds like you need to pay for the requests as well. From https://aws.amazon.com/kinesis/firehose/pricing/ : > Storage > You will be billed separately for charges associated with Amazon S3 and Amazon Redshift usage including storage and read/write requests. However, you will not be billed for data transfer charges for the data that Amazon Kinesis Firehose loads into Amazon S3 and Ama…
But to me it wouldn't make much sense to use Kinesis Firehose unless the fees were cheaper than what it would cost to utilize AWS Lamdba for the same work. I mean it can't be all that many lines of nodejs code to pop events off a stream, flush them into a tempfile in batches, HTTP POST those batches to S3 with appropriate error handling/retry logic.
Admittedly I haven't crunched the math but just by eyeballing their pricing I suspect it may be more expensive than using your own lamdba. I wonder if Kinesis Firehose is implemented internally as an AWS Lambda, it wouldn't surprise me.
I suppose that eventually the pricing on this service will drop once someone open sources such a lamdba, especially since installing a lambda is so remarkably simple.
Hmmmm, I certainly hope dang or another admin consolidate all these stories into a single "Amazon" thread. Can't have multiple stories from a single company eating up space on the front page.
Someone complains about this every time there's an AWS announcement day. Each product gets substantially different conversations. It'd be very frustrating to have to skim through dozens of top-level comments to find one on the service you're interested in.
Earlier quoted context omitted.
Someone complains about this every time there's an AWS announcement day. Each product gets substantially different conversations. It'd be very frustrating to have to skim through dozens of top-level comments to find one on the service you're interested in.
I hope you detected my sarcasm. I'm 100% with you and was upset when they consolidated all the MSFT hardware announcement threads the other day.
I want to be able both reading at different offsets in the stream AND backup it to S3 or ingest into Redshift.
With current offering I need to duplicate data into two different services with different APIs.
What are some typical use cases for Amazon Kinesis streams, on the web?
Analytics event or IoT event ingestion, for example. Then, you can hook up AWS Lambda to the Kinesis stream to actually do any processing/aggregation. Or, if you're using the new Firehose product, you could also use Lambda and attach it to the S3 event source using the bucket that the Firehose dumps into to perform batch processing on these records.
I'm still a bit ignorant as to how kinesis works, can someone explain why this would be preferable to uploading directly to S3?
In other words, you can make small writes to Kinesis and then read out in larger amounts and write larger files to S3. This is a huge optimization for any job that runs across the data in S3. Many small files can really undermine performance in something like Hadoop MapReduce because of the additional request overhead.
At Asana, we've been beta testing Kinesis Firehose. It's been quite convenient not having to manage much, and having the data end up in S3. We're also using Kinesis streams, and have a simple KCL app to pull from streams and write to Firehose. We're looking forward to when streams can be as easy to manage, or when we KCL apps can read from Firehose.
I'm confused.. why would you need a KCL app with Firehose (for reading or writing)?
I don't understand why Kinesis Firehose isn't more tightly integrated with Kinesis Streams and have unified API. I want to be able both reading at different offsets in the stream AND backup it to S3 or ingest into Redshift. With current offering I need to duplicate data into two different services with different APIs.
If you're already doing all the Kinesis shard management/KCL willy nilly why not just dump it to S3 yourself? Firehose seems to be targeting users who don't want to deal with sharding.