Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Corrosion

fly.io

Post reply on HN
Print view View original

61–70 of 109 posts

Re: Corrosion

#61

Post by otoolep »

nodesocket wrote: post #40

Anybody used rqlite[1] in production? I'm exploring how to make my application fault-tolerant using multiple app vm instances. The problem of course is the SQLite database on disk. Using a network file system like NFS is a no-go with SQLite (this includes Amazon Elastic File System (EFS)). I was thinking I'll just have to bite the bullet and migrate to PostgreSQL, but perhaps rqlite can work. [1] https://rqlite.io

rqlite creator here. Right there on the rqlite homepage[1] are listed two production users: replicated.com[2] and textgroove.com are both using it.

[1] https://rqlite.io/

[2] https://www.replicated.com/blog/app-manager-with-rqlite

Re: Corrosion

#62

Post by __turbobrew__ »

tptacek wrote: post #48
__turbobrew__ wrote: post #45

Earlier quoted context omitted.

I haven’t personally worked on envoy xds, but it is what I have seen several BigCo’s use for routing from the edge to internal applications. > Running consensus transcontinentally is very painful You don’t necessarily have to do that, you can keep your quorum nodes (lets assume we are talking about etcd) far enough apart to be in separate failure domains (fires, power loss, natural disasters) but close enough that ne…

I'm thrilled to have people digging into this, because I think it's a super interesting problem, but: no, keeping quorum nodes close-enough-but-not-too-close doesn't solve our problem, because we support a unified customer namespace that runs from Tokyo to Sydney to São Paulo to Northern Virginia to London to Frankfurt to Johannesburg. Two other details that are super important here: This is a public cloud. There is…

> you still need reliable realtime update of instances going down

The way I have seen this implemented is through a cluster of service watcher that ping all services once every X seconds and deregister the service when the pings fail.

Additionally you can use grpc with keepalives which will detect on the client side when a service goes down and automatically remove it from the subset. Grpc also has client side outlier detection so the clients can also automatically remove slow servers from the subset as well. This only works for grpc though, so not generally useful if you are creating a cloud for HTTP servers…

Re: Corrosion

#63

Post by tptacek »

__turbobrew__ wrote: post #62
tptacek wrote: post #48

Earlier quoted context omitted.

I'm thrilled to have people digging into this, because I think it's a super interesting problem, but: no, keeping quorum nodes close-enough-but-not-too-close doesn't solve our problem, because we support a unified customer namespace that runs from Tokyo to Sydney to São Paulo to Northern Virginia to London to Frankfurt to Johannesburg. Two other details that are super important here: This is a public cloud. There is…

> you still need reliable realtime update of instances going down The way I have seen this implemented is through a cluster of service watcher that ping all services once every X seconds and deregister the service when the pings fail. Additionally you can use grpc with keepalives which will detect on the client side when a service goes down and automatically remove it from the subset. Grpc also has client side outlie…

Detecting that the service went down is easy. Notifying every proxy in the fleet that it's down is not. Every proxy in the fleet cannot directly probe every application on the platform.

Re: Corrosion

#64

Post by akerl_ »

mosura wrote: post #57
tptacek wrote: post #55

Earlier quoted context omitted.

What a weird thing to say. I wrote my first OSPF implementation in 1999. The point is that we noticed the solution we'd settled on owes more to protocols like OSPF than to distributed consensus databases, which are the mainstream solution to this problem. It's not "OMG we just discovered this neat protocol called OSPF". We don't actually run OSPF. We don't even do a graph->tree reduction. We're routing HTTP requests,…

Look at one of the other comments: > in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" This is what people saw as the key takeaway. If that takeaway is news to you then I don’t know what you are doing writing distributed systems. While this message may not be what was intended it was what was broadcast.

It seems weird to take an inaccurate paraphrase from a commenter and then use it to paint the authors with your desired brush.

Re: Corrosion

#65

Post by DAlperin »

tucnak wrote: post #50
tptacek wrote: post #44

Earlier quoted context omitted.

What part of this problem do you think FPGAs would help with? In what sense do you think we need specialty routers ? How would you deploy Postgres to address these problems?

[flagged]

(I used to work at fly on networking)

Fly has a lot of interesting networking issues but I don't know that like, the actual routing of packets is the big one? And even in the places where there is bottlenecks in the overlay mesh I'm not sure that custom FPGAs are going to be the solution for now.

But also this blog post isn't about routing packets, it's about state tracking so we know _where_ to even send our packets in the first place.

Re: Corrosion

#66

Post by justinparus »

__turbobrew__ wrote: post #45
tptacek wrote: post #38

Earlier quoted context omitted.

Nope. Talk a little about how how Envoy's service discovery would scale to millions of apps in a global network? There's no way we found the only possible point in the solution space. Do they do something clever here? What we (think we) know won't work is a topologically centralized database that uses distributed consensus algorithms to synchronize. Running consensus transcontinentally is very painful, and keep the s…

I haven’t personally worked on envoy xds, but it is what I have seen several BigCo’s use for routing from the edge to internal applications. > Running consensus transcontinentally is very painful You don’t necessarily have to do that, you can keep your quorum nodes (lets assume we are talking about etcd) far enough apart to be in separate failure domains (fires, power loss, natural disasters) but close enough that ne…

The solutions across different BigCorp Clouds varies depending on the SLA from their underlying network. Doing this on top the public internet is very different than on redundant subsea fiber with dedicated BigCorp bandwidth!

Re: Corrosion

#67

Post by mosura »

akerl_ wrote: post #64
mosura wrote: post #57

Earlier quoted context omitted.

Look at one of the other comments: > in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" This is what people saw as the key takeaway. If that takeaway is news to you then I don’t know what you are doing writing distributed systems. While this message may not be what was intended it was what was broadcast.

It seems weird to take an inaccurate paraphrase from a commenter and then use it to paint the authors with your desired brush.

Not sure the replies to that comment help the cause at all.

Re: Corrosion

#68

Post by natebrennand »

> Finally, let’s revisit that global state problem. After the contagious deadlock bug, we concluded we need to evolve past a single cluster. So we took on a project we call “regionalization”, which creates a two-level database scheme. Each region we operate in runs a Corrosion cluster with fine-grained data about every Fly Machine in the region. The global cluster then maps applications to regions, which is sufficient to make forwarding decisions at our edge proxies.

This tier approach makes a lot of sense to mitigate the scaling limit per corrosion node. Can you share how much data you wind up tracking in each tier in practice?

How concise is the entry for each application -> [regions] table? Does the constraint of running this on every node mean that this creates a global limit for number of applications? It also seems like the region level database would have a regional limit for the number of Fly machines too?

Re: Corrosion

#70

Post by chucky_z »

jimmyl02 wrote: post #47

always wondered at what scale gossip / SWIM breaks down and you need a hierarchy / partitioning. fly's use of corrosion seems to imply it's good enough for a single region which is pretty surprising because iirc Uber's ringpop was said to face problems at around 3K nodes. it would be super cool to learn more about how the world's largest gossip systems work :)

Back of napkin math I’ve done previously, it breaks down around 2 million members with Hashicorps defaults. The defaults are quite aggressive though and if you can tolerate seconds of latency (called out in the article) you could reach billions without a lot of trouble.
Post reply on HN