Building With Workers KV, a Fast Distributed Key-Value Store
blog.cloudflare.com
Building With Workers KV, a Fast Distributed Key-Value Store
1–10 of 13 posts
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#2Fantastic development. Cloudflare is making some great progress recently with workers.
Listing keys in a namespace and/or TTLs would be nice with KV.
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#3So... A hosted Consul?
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#4> Reading values from Workers KV is designed to have the same reliability as reading static files, making it much less likely to become unavailable than a traditional database. It’s designed to have the same performance as reading a file cached within our network, close to your users, giving it the speed of serving a static file as well.
I wonder how this kind of speed is achieved with an API that has to go over the network? Even if the round trip is short such as between two AWS services, there's always at least some latency.
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#5> Reading values from Workers KV is designed to have the same reliability as reading static files, making it much less likely to become unavailable than a traditional database. It’s designed to have the same performance as reading a file cached within our network, close to your users, giving it the speed of serving a static file as well. I wonder how this kind of speed is achieved with an API that has to go over the…
Reading cached files goes over the network too. CDN servers share the cache load and have just 1 or a few copies per datacenter that are served by every machine there.
It's likely that this KV store is built on top of the existing cache storage layer, which would also explain the eventual consistency and high-reads with low-writes.
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#6> Reading values from Workers KV is designed to have the same reliability as reading static files, making it much less likely to become unavailable than a traditional database. It’s designed to have the same performance as reading a file cached within our network, close to your users, giving it the speed of serving a static file as well. I wonder how this kind of speed is achieved with an API that has to go over the…
The trick likely has to do with their write semantics. The post states that writes in WorkerKV are ratelimited to 1 a second per key. From there, we can interpret a likely scenario: if at each data center requests for a given client usually route to the same server nodes, and that information is discoverable on write, then it is likely possible for commands to those nodes to be issued during the write. The commands could be as simple as "pull latest value from upstream store". All commands could easily be issued in under a second. Given that the blog states global consistency is reached in around 10 seconds, it seems reasonable to me that all nodes operating on behalf of a client would have finished pulling the updated value by then.
That said, the scenario above is just speculation on my part.
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#7I love it and want to use it, but two announcements of a new cloud hosted distributed DB and no mention of how consistency works or how conflicts are handled worries me a bit.
Maybe one of the cloudflare people that read HN can shed some light on this? What happens when I write to the same key simultaneously from multiple workers and/or multiple edge nodes?
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#8Going a bit off-topic, I can't access this CloudFlare blog using a Tor browser (using the "standard" security level) without a CAPTCHA.
Didn't we discuss just last week an article[0] by CloudFlare where they said they'd solved this problem and Tor users wouldn't need to solve CAPTCHAs for CloudFlare-fronted services anymore?
What happened to that? I've actually seen way more CloudFlare CAPTCHAs through Tor since that article came out.
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#9I love it and want to use it, but two announcements of a new cloud hosted distributed DB and no mention of how consistency works or how conflicts are handled worries me a bit. Maybe one of the cloudflare people that read HN can shed some light on this? What happens when I write to the same key simultaneously from multiple workers and/or multiple edge nodes?
> What happens when I write to the same key simultaneously from multiple workers and/or multiple edge nodes?
It's eventually-consistent. Eventually, one of the writes "wins".
If you do a read/modify/write in the context of processing a single request from a Worker, we'll automatically apply optimistic concurrency so that the write fails if another write has occurred on the same key concurrently.
But otherwise, yes, the ability to handle transactions is limited. For now, Workers KV can be used for a variety of use cases but is not a full replacement for a database.
Workers KV is only the first of multiple storage-related projects we're working on. Stay tuned. :)
Re: Building With Workers KV, a Fast Distributed Key-Value Store
#10I love it and want to use it, but two announcements of a new cloud hosted distributed DB and no mention of how consistency works or how conflicts are handled worries me a bit. Maybe one of the cloudflare people that read HN can shed some light on this? What happens when I write to the same key simultaneously from multiple workers and/or multiple edge nodes?
Hi skrebbel, storage is my project at Cloudflare.
Last write wins, for some definition of last (https://en.wikipedia.org/wiki/Eventual_consistency).
As we see how initial developers make use of the storage, we are planning to add additional synchronization and consistency functionality. For example the ETags model using If-None-Match and If-Match would be appealing to some.