This is what Apache Pulsar (https://pulsar.incubator.apache.org/) already provides - infinite streaming storage, with simple/flexible messaging streaming API and kafka compatible
Commanding infinite streaming storage with Apache Kafka and Pyrostore
21–29 of 29 posts
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#22Earlier quoted context omitted.
> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?
It’s difficult to search through, query, run projections. Also the API assumes you want to stream realtime data, rather than query historical data.
Use the correct tool for the job: hook an analytics DB up to the Kafka pipe and store the data for future queries. Kafka was never intended to support your use case.
If your inbound data that you'd like to put to Kafka isn't large, by the way, just write straight to the DB. It's irritating to see Kafka used where it's not necessary. It adds complexity to an infrastructure and the cost for doing so has to be justifiable.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#23Earlier quoted context omitted.
> or deal with a repartition of all data. Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?
That sounds possible, but it would both involve downtime or potentially ordering and data duplication issues if you mess it up. Dynamic expansion and contraction of partition count should be a feature that doesn’t require recreating the entire cluster like essentially every other data product in the world.
There's no downtime in mirroring a cluster with MirrorMaker, and it's become common to isolate producers from consumers to separate clusters, see https://medium.com/netflix-techblog/kafka-inside-keystone-pi...
Ordering of messages is completely unaffected, it's the routing of future events that's affected when you increase partition count. This is critical for some use cases (windowing of data for analytics purposes, for example) but irrelevant to others.
Data duplication issues? This sounds like FUD also, but common guidance is to design your events to be idempotent, or utilize Kafka's new exactly-once delivery.
You can expand partition counts for a topic dynamically.
You can't currently decrease partition counts, because given the current design that could orphan both data and consumers.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#24Earlier quoted context omitted.
That sounds possible, but it would both involve downtime or potentially ordering and data duplication issues if you mess it up. Dynamic expansion and contraction of partition count should be a feature that doesn’t require recreating the entire cluster like essentially every other data product in the world.
The architectural pendulum is starting to swing away from co-location of storage and compute (the trend of the last 10+ years) to decoupling of storage and processing to avoid exactly these issues, but legacy architectures hang on for a while. In the streaming and messaging space, Apache Pulsar (pulsar.apache.org) is a more recent solution that has an architecture that decouples processing and storage. That gives you…
What pendulum do you see? From here, architectural patterns are clearly converging on a "distributed mainframe" model between containerization and lambda/kappa architectures...
I think Joyent's Manta was ahead of its time in colocating compute and storage and I suspect we'll see more along this vein with the recent open sourcing of FoundationDB.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#25Integration with Azure Managed Disks :
Due to the ingestion heavy nature, the disks attached to the nodes on the cluster often result as the bottleneck. Traditionally, to scale this bottleneck, more nodes need to be added. Azure Managed Disks is a technology that provides cheaper, scalable disks that are a fraction of the cost of a node. HDInsight Kafka has integrated with these disks to provide upto 16 TB/node instead of the traditional 1 TB. This results in an exponentially higher scale, while reducing costs in the inverse, exponential manner.
https://azure.microsoft.com/en-us/services/hdinsight/apache-...
Is this same approach as pyro ?
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#26Integration with Azure Managed Disks : Due to the ingestion heavy nature, the disks attached to the nodes on the cluster often result as the bottleneck. Traditionally, to scale this bottleneck, more nodes need to be added. Azure Managed Disks is a technology that provides cheaper, scalable disks that are a fraction of the cost of a node. HDInsight Kafka has integrated with these disks to provide upto 16 TB/node inste…
Our approach archives topics to cheap, highly durable and available object stores, while keeping the data available for blending between warehoused and live data sets.
This reduces operational complexity significantly vs scaling nodes up, dealing with rebalancing, under replicated partitions, etc.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#27Earlier quoted context omitted.
> I know Kafka kind-of sort-of provides this functionality, but it doesn’t work in practice. How so?
One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data. An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something…
I've used LVM in AWS to present a single volume to Kafka >16TB which is their max EBS size. They report you can attach up to 40 EBS volumes.
Growing LVM with XFSs has worked well, 0 downtime and around 60 seconds.
Allows you to over provision just enough you do not have to babysit the drives or pay $$$ for unused disc.
If you stripe the volumes you'll also distribute your IOPs in AWS.
Outside AWS LVM still applies. Kafka's JBOD is useless without easy / auto rebalancing.
This week onsite at a client's I discovered ScaleIO which can present up to a 1PB volume and does clever sharding/replication in the background.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#28Earlier quoted context omitted.
One potential problem is a Kafka partition’s size is limited to the size of the smallest machine in the replica set. This means if you want infinite retention you have to potentially over-partition so they never get too big, keep buying bigger machines and disks, or deal with a repartition of all data. An simple way to get around this problem is dumping messages into a file and putting that file in S3 named something…
> or deal with a repartition of all data. Why is this difficult? Is mirroring clusters operationally problematic? If one cluster gets too small, in theory can’t you spin up another cluster, and mirror the first onto the second. Then when they are in sync direct writes to the new cluster?
You can but even in TB's of data never mind PB's it can take weeks to sync across.
Also running two clusters to handle large volumes of data, it's big money. Even a small modest cluster with around 20TB+ of data was north of 30K a month on AWS. That's a full app cluster though with consumers/producers aswell as brokers.
Re: Commanding infinite streaming storage with Apache Kafka and Pyrostore
#29Earlier quoted context omitted.
The architectural pendulum is starting to swing away from co-location of storage and compute (the trend of the last 10+ years) to decoupling of storage and processing to avoid exactly these issues, but legacy architectures hang on for a while. In the streaming and messaging space, Apache Pulsar (pulsar.apache.org) is a more recent solution that has an architecture that decouples processing and storage. That gives you…
What pendulum do you see? From here, architectural patterns are clearly converging on a "distributed mainframe" model between containerization and lambda/kappa architectures... I think Joyent's Manta was ahead of its time in colocating compute and storage and I suspect we'll see more along this vein with the recent open sourcing of FoundationDB.
I was thinking more specifically of the internal architectures of data processing platforms, especially the categorizations that emerged from the MPP database world. The "shared nothing" architecture has been dominant in databases (and is also the core architecture of Hadoop), designed around "co-locating data and compute". Kafka largely follows that architecture as well, using local disk on the compute nodes as its persistent storage layer.
A lot of new data processing platforms, from Snowflake in the data warehouse world to AWS Athena to Apache Pulsar in the broader data processing world, have moved to decoupled architectures.
Containerization and container management frameworks (e.g. Kubernetes) certainly do change the meaning of "local" storage, will be interesting to see how that plays out.