Amazon Kinesis Firehose – Simple and Scalable Data Ingestion • Hacker News Board

Re: Amazon Kinesis Firehose – Simple and Scalable Data Ingestion

#31

Post by jhurliman » Thu, Oct 08, 2015, 6:01 AM UTC

solofounder1 wrote: post #22

mkobit wrote: post #18

Earlier quoted context omitted.

Unless I am reading it wrong, it sounds like you need to pay for the requests as well. From https://aws.amazon.com/kinesis/firehose/pricing/ : > Storage > You will be billed separately for charges associated with Amazon S3 and Amazon Redshift usage including storage and read/write requests. However, you will not be billed for data transfer charges for the data that Amazon Kinesis Firehose loads into Amazon S3 and Ama…

To me this sounds like a way to avoid the hassle of creating a lambda to buffer streaming data into chunks before HTTP POSTing them to the S3 API. But to me it wouldn't make much sense to use Kinesis Firehose unless the fees were cheaper than what it would cost to utilize AWS Lamdba for the same work. I mean it can't be all that many lines of nodejs code to pop events off a stream, flush them into a tempfile in batch…

The only complication with this approach right now is that Lambda scripts can't communicate with anything inside a VPC (such as most Redshift instances). I imagine they will fix this issue in the future though.

Re: Amazon Kinesis Firehose – Simple and Scalable Data Ingestion

#32

Post by nivertech » Thu, Oct 08, 2015, 9:45 AM UTC

cherioo wrote: post #30

nivertech wrote: post #26

I don't understand why Kinesis Firehose isn't more tightly integrated with Kinesis Streams and have unified API. I want to be able both reading at different offsets in the stream AND backup it to S3 or ingest into Redshift. With current offering I need to duplicate data into two different services with different APIs.

To avoid interference between user application and theirs? Kinesis has pretty delicate throttle per shard and having user application that is out of their control working off the same shard will probably make the Firehose part much more fragile. If you're already doing all the Kinesis shard management/KCL willy nilly why not just dump it to S3 yourself? Firehose seems to be targeting users who don't want to deal with…

Yes, I figured out that Firehose is more higher-level product, than Kinesis. Maybe in the future it will be extended with API to read from a delivery stream.

There are competing products, like Google Cloud Pub/Sub where there is no need to manage shards manually or run your own workers, like KCL.

Re: Amazon Kinesis Firehose – Simple and Scalable Data Ingestion

#33

Post by nivertech » Thu, Oct 08, 2015, 11:54 AM UTC

    $ sudo pip install awscli --upgrade
    ...
    $ aws firehose help

    FIREHOSE()                                                          FIREHOSE()



    NAME
           firehose -

    DESCRIPTION
           Amazon  Kinesis  Firehose  is  a  fully-managed  service  that delivers
           real-time streaming data to destinations such as Amazon S3  and  Amazon
           Redshift.

    AVAILABLE COMMANDS
           o create-delivery-stream

           o delete-delivery-stream

           o describe-delivery-stream

           o help

           o list-delivery-streams

           o put-record

           o put-record-batch

           o update-destination



    	                                                            FIREHOSE()