Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Options for handling state at the edge?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Ask HN: Options for handling state at the edge?

#31

Post by adam_arthur »

Depends on your product, but I'm able to do everything via Cloudflare workers, KV, DurableObjects, and use JSON files stored in Cloudflare CDN as source of truth (hosted for free btw)

Cloudflare KV can store most of what you need in JSON form, while DurableObjects let you model updates with transactional guarantees.

My app is particularly read heavy though, and backing data is mostly static (but gets updated daily).

Honestly after using Cloudflare feel like they will easily become the go to cloud for building small/quick apps. Everything is integrated much better than AWS and way more user friendly from docs and dev experience perspective. Also their dev velocity on new features is pretty insane.

Honestly didn't think that much of them until I started digging into these things.

Edit: And just today their S3 competitor entered open beta https://blog.cloudflare.com/r2-open-beta/

Re: Ask HN: Options for handling state at the edge?

#33

Post by marzoeva »

We're tackling this problem at ReadySet. TLDR, ReadySet connects to your existing relational database and caches SQL query results at the edge based on the actual traffic patterns in those regions. Our goal is to augment rather than replace the database as the source of truth. You can sign up for our early access list here: https://readyset.io

Re: Ask HN: Options for handling state at the edge?

#34

Post by rektide »

weego wrote: post #21
rektide wrote: post #10

I don't have a whole lot to say on this right now (very WIP), but I have a strong belief that git is a core tool we should be using for data. Most data-formats are thick-formats, pack data into a single file. Part of the effort in switching to git would be a shift to trying to unpack our data, to really make use of the file system to store fine grained pieces of data. It's been around for a while, but Irmin[1] (writt…

I have a strong belief that git is a core tool we should be using for data It isn't, we shouldn't, and you're not the first and won't be the last person to put time into this. It's neither a compelling solution nor even a particularly good one.

Gee thanks great critique much appreciated. +1

Re: Ask HN: Options for handling state at the edge?

#35

Post by Joel_Mckay »

Depends what your problem scope entails, but in general it comes down to a few key design choices and trade-offs. 1. Application layer load-balancing which allows for high-latency out-of-band idempotent transaction state consolidation between peers (rabbitMQ or Kafka often used to handle backlogs). This is hard to do, as other teams will break what they don't understand. 2. A non-polyglot solution like Erlang/Elixir which has “peer-state-awareness” and channels (functionally act like microservices) baked into the distributed OTP. i.e. the state is shared among peers through a simple abstraction, and only the assigned roles differ in non-persistent “edge” instances. 3. If you think in terms of classic data-center AWS partitions or sequentially indexed databases... you are likely approaching this wrong... just give up an go with Hyperledger Fabric.

YMMV, I just discovered my favorite game on my phone was intended for cats. ;-)

Cheers, J

Re: Ask HN: Options for handling state at the edge?

#37

Post by nojvek »

powersurge360 wrote: post #15

I haven't done this, but I've been thinking about it lately. Fly.IO has had some very interesting ideas on this if you want to use a relational database. There was an article about litestream that would allow you to replicate your SQLite database to an arbitrary number of nodes, which means that every application server would have a SQLite file sitting on it for read queries, and then you can capture write queries an…

This goes into CAP theorem. Do you want async/inconsistent read replicas or consistent replicas with guaranteed “read new info after write everywhere” semantics.

To get consistent reads, every write needs to wait to ensure every node has acknowledged the write and then return.

So yes, you can have single digit ms sql queries, but you’d pay the price of writes being bottlenecked by speed of all nodes acknowledging the writes.

Re: Ask HN: Options for handling state at the edge?

#39

Post by jka »

rektide wrote: post #14
richardwhiuk wrote: post #12

Earlier quoted context omitted.

You really want to use CRDTs, not data types subject to human resolved merge conflicts.

I feel like crdts are sold as a panacea. I can esily imagine users making conflicting changes, so I dont really see or understand what the real value or weaknesses of CRDTs are. Im also used to seeing them used for online synchronization, & far less examples of distributed crdts, which is, to me, highly important. Git by contrast has straightforward & good merge strategies. At this point, I feel like the problems are…

Whether conflict-resolution can be performed automatically or may require manual input is important at scale.

Human editors may cause the content within a CRDT datastructure to become inconsistent in the sense of "is this document understandable by another person", but they can't cause conflicts that block the editing process on-disk.

On the other hand, git merges can -- and frequently do -- involve conflict resolution that isn't effectively handled -- especially in a distributed system -- by automated measures.

Re: Ask HN: Options for handling state at the edge?

#40

Post by KajMagnus »

don-code wrote: post #3

I recently had an opportunity to build an application on top of Lambda@Edge (AWS's equivalent of Cloudflare workers). The prevailing wisdom there was to make use of regional services, like S3 and DynamoDB, from the edge. That, of course, makes my edge application depend on calls to a larger, further away point of presence. While it's possible to distribute state to many AWS regions and select the closest one, I ended…

> I just redeploy the application with the new state.

How long does that take? From clicking Deploy until the app got updated? ... and until it's restarted and the new state is available?

Could this depend a bit on how far away a data center is? Maybe 200ms longer for some?

Post reply on HN