Designing Data Gateway for Cassandra in Open Source
1–10 of 14 posts
Re: Designing Data Gateway for Cassandra in Open Source
#2Any thoughts on whether the internal API would use streaming or unary calls?
Re: Designing Data Gateway for Cassandra in Open Source
#3A well written and concise article describing design tradeoffs and options for Stargate!
Re: Designing Data Gateway for Cassandra in Open Source
#4Any thoughts on whether the internal API would use streaming or unary calls?
Initially we'll be using unary calls but I can see use cases where streaming would be useful. One might be for keeping a schema representation in sync between the services and coordinator (for those services that need to be schema aware). Another potential could be future implementations of reactive APIs and subscription or push models. Although in the near term we can leverage streaming for performance improvements by allowing the service to begin performing transformations before the entire response has been received.
Re: Designing Data Gateway for Cassandra in Open Source
#5A well written and concise article describing design tradeoffs and options for Stargate!
Thanks! The team put a lot of work into writing this and exploring many different options before arriving at the proposed architecture.
Re: Designing Data Gateway for Cassandra in Open Source
#6Hey everyone! I wrote the words for this article but our team put in a ton of work to put this together. Just a plug that we'd love to get feedback on our design and have started a Github discussion here https://github.com/stargate/stargate/discussions/1381. Thanks!
Re: Designing Data Gateway for Cassandra in Open Source
#7Great read. As someone not very familiar with Stargate, is the internal communication via gRPC (e.g. from most services to bridge) totally new to v2? Or does Stargate v1 use gRPC internally as well?
If it is new, what drove the decision to use gRPC? Performance?
Re: Designing Data Gateway for Cassandra in Open Source
#8This was an awesome read! I appreciate how you were able to cover a lot of ground that went deep technically while still being succinct.
I'm very curious on how you solicited feedback and collaborated on the article and design decisions up to now. Can you speak to what worked well or not so well as you hashed out the details?
Re: Designing Data Gateway for Cassandra in Open Source
#9Great read. As someone not very familiar with Stargate, is the internal communication via gRPC (e.g. from most services to bridge) totally new to v2? Or does Stargate v1 use gRPC internally as well? If it is new, what drove the decision to use gRPC? Performance?
Stargate v1 is a monolith where everything runs within the same JVM which means service to persistence communication is just simple method calls. One of our main goals of Stargate v2 was to break up this monolith which meant that we could no longer communicate with just direct method calls and would instead need to bridge the gap between service and persistence with some other communication mechanism. We explored a few different options such as REST or CQL via drivers but ultimately landed on gRPC for a few reasons:
1. REST could be a viable option and has been used many times when splitting up a monolith but we didn't believe it could give us the performance we would need due to opening and closing connections
2. With REST we would essentially have to define our own schema definition language to describe the transport of messages between service and persistence which would also add the overhead of serializing and deserializing json payloads
3. CQL is a proven transport mechanism but requires the use of a driver which means whatever language used to implement the service must also have a workable driver (flexibility is a driving design principle for v2)
4. The session based nature of drivers would cause issues in a multi-user or multi-tenant environment
5. Although CQL fits most of our use cases it doesn't quite cover everything
So, yes performance was a factor in choosing gRPC but there was also the flexibility it gave us while being able to reuse CQL thus avoiding reinventing the wheel.
Re: Designing Data Gateway for Cassandra in Open Source
#10This was an awesome read! I appreciate how you were able to cover a lot of ground that went deep technically while still being succinct. I'm very curious on how you solicited feedback and collaborated on the article and design decisions up to now. Can you speak to what worked well or not so well as you hashed out the details?
Thanks, glad you enjoyed it!
Up until now we've primarily used our [Discord server](https://discord.gg/33mKDHHFUE) and Github issues for feedback. Starting with v2 we plan to start using Github Discussions as another avenue. Although we like Discord for free form discussion and quick questions it's not great for retention or searching for past solutions. Issues felt a bit constrained and tend to be directly tied to pull requests. Our hope is that with Discussions we'll be able to have more long form discussions that can be retained and made us of by others in the community. Also with Discussions we would like to start making use of an RFC process where anyone can propose designs, discuss, and then proceed with implementation.