191–200 of 249 posts
Earlier quoted context omitted.
For what its worth I have a java microservice running on 13mb of ram.
Without knowing what that service does, how many users/request it serves per second/day, I can only assume it just an http listener up on some port that returns "Hello {username}" when someone sends a GET request.
This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations? For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains…
One of the challenges with layering SQL on top of a KV store is query performance. The most obvious way to model a secondary index on top of a pure KV store is to map indexed values to keys. For example, given the (rowID, name) tuples (123, "Bob"), (345, "Jane"), (234, "Zack"), you can store these as keys: name:Bob:123 name:Jane:345 name:Zack:234 At this point you don't need or even want values, so this is effectivel…
As for a generic relational layer over K/V stores, I think it’s a superficially appealing idea that would be impossible to optimize adequately in practice. Honestly, if you want to implement a distributed relational database, I would recommend starting with Postgres as your local storage engine and pushing down as much relational logic as possible to that layer. I have worked on such a system in the past and it produced very good results for minimal development effort.
Earlier quoted context omitted.
Did you check this? https://docs.nats.io/nats-streaming-concepts/intro "..Message/event persistence - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations." and "At-least-once-delivery - NATS Streaming offers message acknowledgements between publisher and server (for…
Also check out Liftbridge ( https://liftbridge.io ), which is a Kafka-like API on top of NATS. Disclaimer: I'm the author and former core contributor of NATS and NATS Streaming.
Earlier quoted context omitted.
One of the challenges with layering SQL on top of a KV store is query performance. The most obvious way to model a secondary index on top of a pure KV store is to map indexed values to keys. For example, given the (rowID, name) tuples (123, "Bob"), (345, "Jane"), (234, "Zack"), you can store these as keys: name:Bob:123 name:Jane:345 name:Zack:234 At this point you don't need or even want values, so this is effectivel…
I disagree that an index is a set rather than a map: it is by definition a map from keys to row ids. As for a generic relational layer over K/V stores, I think it’s a superficially appealing idea that would be impossible to optimize adequately in practice. Honestly, if you want to implement a distributed relational database, I would recommend starting with Postgres as your local storage engine and pushing down as muc…
The row ID can be encoded into the key. From my example, a basic mapping might be:
name:Bob:123 =>
For example, if I want to search for "name = 'Bob'", then I simply start at the key "name:Bob:" and pluck the row ID from each key, scanning sequentially until I reach the end of my range.
This works great for multiple values. For example:
name:Bob:123 =>
name:Bob:124 =>
name:Bob:125 =>
If you store row IDs in the value, you'll risk read/write contention on the value. Let's say there are multiple rows with "Bob", you'll end up storing something like:
name:Bob => [123, 124, 125]
This also puts a constraint on the number of row IDs you can fit in a single value. KV stores typically co-locate value data with keys, so now you might not be able to efficiently scan a large range without also loading that data. You can do tricks like introducing an indirection, where you don't store "row IDs", but "row page IDs", where each page is sharded maximum of N row IDs, allowing you to sidestep the size limit on values. But that comes with other costs.
I'm not counting in-memory stores like Redis here. Implementing in-memory indexes is a completely different ball game to something that needs to live on disk.
As for "superficially appealing idea that would be impossible to optimize adequately in practice", your assertion is demonstrably false: TiDB and CockroachDB both implement performant relational databases on top of general-purpose KV stores.
I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
I understand why you chose Pulsar over RabbitMQ, but wouldn't have Kafka been a good choice as well?
Splunk just acquired streamlio and most of the core devs got sucked up. While pulsar is a great product - are you not concerned that these guys are getting paid $$ bank to do something else now?
I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
That's amazing, thank you for sharing. I understand why you chose Pulsar over RabbitMQ, but wouldn't have Kafka been a good choice as well?
I think Pulse should really focus on the integration with the rest of the Apache stack if they want to gain traction.
I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about 5 months for our implementation with a chunk of that work mostly about figuring out how to integrate our internal auth as well as a using hashicorp vault as a clean automated way to ge…
Alrighty, a few questions: - what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch? - have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly? - how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery - for the…