Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Commanding infinite streaming storage with Apache Kafka and Pyrostore

pyrostore.io

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#11

Post by lbradstreet »

tomconnors wrote: post #10

Everything Distributed Masonry does is very interesting. Wish I had more excuses to use your stuff at work. Storing all data forever in a single source of truth is awesome until regulation like GDPR comes along. Do you have plans to support excision or is your guidance on personal data to avoid putting it into a system like Kafka/Pyrostore?

We will be launching support for native excision and data anonymization soon, as these are extremely important to storing streaming data for the long term.

Workarounds for excision in Kafka, such as key compaction, are often not possible to use as they depend on the key scheme used.

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#12

Post by insensible »

tomconnors wrote: post #10

Everything Distributed Masonry does is very interesting. Wish I had more excuses to use your stuff at work. Storing all data forever in a single source of truth is awesome until regulation like GDPR comes along. Do you have plans to support excision or is your guidance on personal data to avoid putting it into a system like Kafka/Pyrostore?

You might enjoy reading Greg Young's https://leanpub.com/esversioning, which covers this topic.

It covers several strategies, three of which are:

* Encrypt it and then throw away the key to forget it

* Store private data outside the event with the event just pointing to it

* Delete events (on systems that support this)

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#13

Post by linkmotif »

ryanworl wrote: post #9
linkmotif wrote: post #4

Earlier quoted context omitted.

> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?

One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data. An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something…

> or deal with a repartition of all data.

Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#15

Post by linkmotif »

stingraycharles wrote: post #7
linkmotif wrote: post #4

Earlier quoted context omitted.

> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?

It’s difficult to search through, query, run projections. Also the API assumes you want to stream realtime data, rather than query historical data.

I think you can use kafka streams to create whatever projections you want. Kafka can handle millions of small records per second. You can use the Kafka streams api to process all your data either into local rocks db stores or write the data to whatever database you want. This requires some work but then you have a good thing going if it works.

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#17

Post by ryanworl »

linkmotif wrote: post #13
ryanworl wrote: post #9

Earlier quoted context omitted.

One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data. An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something…

> or deal with a repartition of all data. Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?

That sounds possible, but it would both involve downtime or potentially ordering and data duplication issues if you mess it up. Dynamic expansion and contraction of partition count should be a feature that doesn’t require recreating the entire cluster like essentially every other data product in the world.

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#18

Post by rad_gruchalski »

ah- wrote: post #14

I wonder if this would ever be integrated into Kafka proper. Shipping out historical chunks onto infinite storage seems like a generally sensible thing. This would be even better if it didn't need a modified client.

I did suggest a potential solution a while ago: https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32... Relevant JIRA ticket: https://issues.apache.org/jira/plugins/servlet/mobile#issue/...

Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore

#20

Post by jbs40 »

ryanworl wrote: post #17
linkmotif wrote: post #13

Earlier quoted context omitted.

> or deal with a repartition of all data. Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?

That sounds possible, but it would both involve downtime or potentially ordering and data duplication issues if you mess it up. Dynamic expansion and contraction of partition count should be a feature that doesn’t require recreating the entire cluster like essentially every other data product in the world.

The architectural pendulum is starting to swing away from co-location of storage and compute (the trend of the last 10+ years) to decoupling of storage and processing to avoid exactly these issues, but legacy architectures hang on for a while.

In the streaming and messaging space, Apache Pulsar (pulsar.apache.org) is a more recent solution that has an architecture that decouples processing and storage. That gives you nice properties like independent scaling of storage and processing, infinite data retention, dynamic resizing and others.

Post reply on HN