Commanding infinite streaming storage with Apache Kafka and Pyrostore
1–10 of 29 posts
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#2I like it. Personally, one of my biggest problems with Kafka is its operational complexity. I’ve just had one too many instances of Kafka brokers getting stuck while doing an upgrade and things like that.
Additionally, I would really, really like to be able to use it as an Event Store, easily accessible by anyone in the org with infinite data retention. I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice.
This appears to be a solution to this problem. Will be interesting to see whether it gains traction.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#3> tradeoffs in our operation of Kafka have lossy effects on stream-ability. Balancing costs and operational feasibility, we ask Kafka to forget older data through retention policies.
What does ' lossy effects on stream-ability. ' mean here. Stream slows down, data loss or something else?
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#4I like it. Personally, one of my biggest problems with Kafka is its operational complexity. I’ve just had one too many instances of Kafka brokers getting stuck while doing an upgrade and things like that. Additionally, I would really, really like to be able to use it as an Event Store, easily accessible by anyone in the org with infinite data retention. I know Kafka kind-of sort-of provides this functionality, but it…
> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice.
How so?
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#5> tradeoffs in our operation of Kafka have lossy effects on stream-ability. Balancing costs and operational feasibility, we ask Kafka to forget older data through retention policies. What does ' lossy effects on stream-ability. ' mean here. Stream slows down, data loss or something else?
[deleted]
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#6> tradeoffs in our operation of Kafka have lossy effects on stream-ability. Balancing costs and operational feasibility, we ask Kafka to forget older data through retention policies. What does ' lossy effects on stream-ability. ' mean here. Stream slows down, data loss or something else?
Pyrostore co-founder here. When practitioners archive their data from
Kafka to other storage products (S3, SQL database, etc) today, they
are giving up on the log ordered structure of the data their ability
to consume their data in its original ordering, with its original
offsets and timestamps. Pyrostore structures and indexes your data in
S3 in order to provide a consumer that implements the Kafka consumer
interfaces, ensuring you are always able to stream from hot and cold
storage alike.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#7I like it. Personally, one of my biggest problems with Kafka is its operational complexity. I’ve just had one too many instances of Kafka brokers getting stuck while doing an upgrade and things like that. Additionally, I would really, really like to be able to use it as an Event Store, easily accessible by anyone in the org with infinite data retention. I know Kafka kind-of sort-of provides this functionality, but it…
> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?
It’s difficult to search through, query, run projections. Also the API assumes you want to stream realtime data, rather than query historical data.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#8Earlier quoted context omitted.
> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?
It’s difficult to search through, query, run projections. Also the API assumes you want to stream realtime data, rather than query historical data.
As a side note, we provide tooling to maintain Amazon Athena/Hive,Presto support over your Pyrostore archived data in S3, while maintaining its ability to be streamed in-order.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#9I like it. Personally, one of my biggest problems with Kafka is its operational complexity. I’ve just had one too many instances of Kafka brokers getting stuck while doing an upgrade and things like that. Additionally, I would really, really like to be able to use it as an Event Store, easily accessible by anyone in the org with infinite data retention. I know Kafka kind-of sort-of provides this functionality, but it…
> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?
One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data.
An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something like “topic-partition-offset” where offset is the offset of the first message contained within that file. You can then read those forward starting from offset zero and go until you reach the end, then start reading from Kafka for recent data.
The drawback is this isn’t integrated with Kafka so you’re now maintaining what is effectively two different systems for the same data. It also means the key-based compaction won’t work either and you’d have to re-implement that on top of the files in S3 as well.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#10Everything Distributed Masonry does is very interesting. Wish I had more excuses to use your stuff at work.
Storing all data forever in a single source of truth is awesome until regulation like GDPR comes along. Do you have plans to support excision or is your guidance on personal data to avoid putting it into a system like Kafka/Pyrostore?