Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Corrosion

fly.io

Post reply on HN
Print view View original

51–60 of 109 posts

Re: Corrosion

#51

Post by hedgehog »

tptacek wrote: post #46
hedgehog wrote: post #42

Earlier quoted context omitted.

Is it actually necessary to run transcontinental consensus? Apps in a given location are not movable so it would seem for a given app it's known which part of the network writes can come from. That would require partitioning the namespace but, given that apps are not movable, does that matter? It feel like there are other areas like docs and tooling that would benefit from relatively higher prioritization.

Apps in a given location are extremely movable! That's the point of the service!

We unfortunately lost our location with not a whole lot of notice and the migration to a new one was not seamless, on top of things like the GitHub actions being out of date (only supporting the deprecated Postgres service, not the new one).

Re: Corrosion

#52

Post by vlovich123 »

tptacek wrote: post #21
bananapub wrote: post #6

in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" - you can formal method and Rust and test and watchdog yourself as much as you want, but you simply have to stop doing that or the unknown unknowns will just keep taking you down.

I mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down…

> Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait.

But they have to. Physically no solution will be instantaneous because that’s not how the speed of light nor relativity works - even two events next to each other cannot find out about each other instantaneously. So then the question is “how long can I wait for this information”. And that’s the part that I feel isn’t answered - eg if the app dies, the TCP connections die and in theory that information travels as quickly as anything else you send. It’s not reliably detectable but conceivably you could have an eBPF program monitoring death and notifying the proxies. Thats the part that’s really not explained in the article which is why you need to maintain an eventually consistent view of the connectivity. I get maybe why that could be useful but noticing app connectivity death seems wrong considering I believe you’re more tracking machine and cluster health right? Ie not noticing an app instance goes down but noticing all app instances on a given machine are gone and consensus deciding globally where the new app instance will be as quickly as possible?

Re: Corrosion

#53

Post by tptacek »

tucnak wrote: post #50
tptacek wrote: post #44

Earlier quoted context omitted.

What part of this problem do you think FPGAs would help with? In what sense do you think we need specialty routers ? How would you deploy Postgres to address these problems?

[flagged]

I'm not sure you understand our problem space.

Re: Corrosion

#54

Post by tptacek »

vlovich123 wrote: post #52
tptacek wrote: post #21

Earlier quoted context omitted.

I mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down…

> Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait. But they have to. Physically no solution will be instantaneous because that’s not how the speed of light nor relativity works - even two events next to each other cannot find out about each other instantaneously. So then the question is “how long can I wait for this information”.…

A request routed to a dead instance doesn't fall into a black hole: our proxies reroute it. But that's very slow; to deliver acceptable service quality you need to minimize the number of times that happens. So you can't accept a solution that leaves large windows of time within which every instance that has gone down has a stale entry. Remember: instances coming up and down happens all the time on this platform! It's part of the point.

Re: Corrosion

#55

Post by tptacek »

tucnak wrote: post #41
mosura wrote: post #39

Someone needs to read about ant colony optimization. https://en.wikipedia.org/wiki/Ant_colony_optimization_algori... This blog is not impressive for an infra company.

I respect Fly, and it does sound like a nice place to work, but honestly, you're onto something. You would expect ostensibly Public Cloud provider to have a more solid grasp on networking. Instead, we're discovering how they're learning about things like OSPF! Makes you think that's all.

What a weird thing to say. I wrote my first OSPF implementation in 1999. The point is that we noticed the solution we'd settled on owes more to protocols like OSPF than to distributed consensus databases, which are the mainstream solution to this problem. It's not "OMG we just discovered this neat protocol called OSPF". We don't actually run OSPF. We don't even do a graph->tree reduction. We're routing HTTP requests, not packets.

Re: Corrosion

#56

Post by x0x0 »

tptacek wrote: post #36
blinkingled wrote: post #35

Earlier quoted context omitted.

I was referring to the "HTTP request in Tokyo to find the nearest instance in Sydney" part which felt to me like a differentiating feature- no other cloud provider seems to have bidding or HTTP request level cross regional lookup or whatever. The "decision that long predates Corrosion" is precisely the point I was trying to make - was it made too soon before understanding the ramifications and/or having a validated t…

That's literally the premise of the service and always has been.

fwiw, I'm happily running a company and some contract work on fly literally as aws, but what if it weren't the most massively complex pile of shit you've ever seen.

I have a couple reasonably sized, understandable toml files and another 100 lines of ruby that runs long-running rake tasks as individual fly machines. The whole thing works really nicely.

Re: Corrosion

#57

Post by mosura »

tptacek wrote: post #55
tucnak wrote: post #41

Earlier quoted context omitted.

I respect Fly, and it does sound like a nice place to work, but honestly, you're onto something. You would expect ostensibly Public Cloud provider to have a more solid grasp on networking. Instead, we're discovering how they're learning about things like OSPF! Makes you think that's all.

What a weird thing to say. I wrote my first OSPF implementation in 1999. The point is that we noticed the solution we'd settled on owes more to protocols like OSPF than to distributed consensus databases, which are the mainstream solution to this problem. It's not "OMG we just discovered this neat protocol called OSPF". We don't actually run OSPF. We don't even do a graph->tree reduction. We're routing HTTP requests,…

Look at one of the other comments:

> in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution"

This is what people saw as the key takeaway. If that takeaway is news to you then I don’t know what you are doing writing distributed systems.

While this message may not be what was intended it was what was broadcast.

Re: Corrosion

#59

Post by [deleted] »

jimmyl02 wrote: post #47

always wondered at what scale gossip / SWIM breaks down and you need a hierarchy / partitioning. fly's use of corrosion seems to imply it's good enough for a single region which is pretty surprising because iirc Uber's ringpop was said to face problems at around 3K nodes. it would be super cool to learn more about how the world's largest gossip systems work :)

[deleted]

Re: Corrosion

#60

Post by __turbobrew__ »

tptacek wrote: post #48
__turbobrew__ wrote: post #45

Earlier quoted context omitted.

I haven’t personally worked on envoy xds, but it is what I have seen several BigCo’s use for routing from the edge to internal applications. > Running consensus transcontinentally is very painful You don’t necessarily have to do that, you can keep your quorum nodes (lets assume we are talking about etcd) far enough apart to be in separate failure domains (fires, power loss, natural disasters) but close enough that ne…

I'm thrilled to have people digging into this, because I think it's a super interesting problem, but: no, keeping quorum nodes close-enough-but-not-too-close doesn't solve our problem, because we support a unified customer namespace that runs from Tokyo to Sydney to São Paulo to Northern Virginia to London to Frankfurt to Johannesburg. Two other details that are super important here: This is a public cloud. There is…

> I'm thrilled to have people digging into this, because I think it's a super interesting problem

Yes, somehow this is a problem all the big companies have, but it seems like there is no standard solution and nobody has open sourced their stuff (except you)!

Taking a step back, and thinking about the AWS outage last week which was caused by a buggy bespoke system built on top of DNS, it seems like we need an IETF standard for service discovery. DNS++ if you will. I have seen lots of (ab)use of DNS for dynamic service discovery and it seems like we need a better solution which is either push based or gossip based to more quickly disseminate service discovery updates.

Post reply on HN