Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Corrosion

fly.io

Post reply on HN
Print view View original

81–90 of 109 posts

Re: Corrosion

#81

Post by otterley »

tptacek wrote: post #80
__turbobrew__ wrote: post #76

Earlier quoted context omitted.

I believe it is possible within envoy to detect a bad backend and automatically remove it from the load balancing pool, so why can the proxy not determine that certain backend instances are unavailable and remove them from the pool? No coordination needed and it also handles other cases where the backend is bad such as overload or deadlock? It also seems like part of your pain point is that there is an any-to-any rel…

Ok, again: this isn't a cluster of load balancers in front of a discrete collection of app servers in a data center. It's thousands of load balancers handling millions of applications scattered all over the world, with instances going up and down constantly. The interesting part of this problem isn't noticing that an instance is down . Any load balancer can do that. The interesting problem is noticing than and then i…

Out of curiosity, what’s your upper bound latency SLO for propagating this state? (I assume this actually conforms to a percentile histogram and isn’t a single value.)

Re: Corrosion

#82

Post by JoachimSchipper »

DAlperin wrote: post #77
JoachimSchipper wrote: post #73

Earlier quoted context omitted.

(Hopping in here because the discussion is interesting... feel very free to ignore.) Thanks for writing this up! It was a very interesting read about a part of networking that I don't get to seriously touch. That said: I'm sure you guys have thought about this a lot and that I'm just missing something, but "why can't every proxy probe every [worker, not application]?" was exactly one of the questions I had while read…

(I used to work at Fly, specifically on the proxy so my info may be slightly out of date, but I've spent a lot of time thinking about this stuff.) > why can't every proxy probe every [worker, not application]? There are several divergent issues with this approach (though it can have it's place). First, you still need _some_ service discovery to tell you where the nodes are, though it's easy to assume this can be solv…

Thanks a lot for your response!

Re: Corrosion

#83

Post by __turbobrew__ »

tptacek wrote: post #80
__turbobrew__ wrote: post #76

Earlier quoted context omitted.

I believe it is possible within envoy to detect a bad backend and automatically remove it from the load balancing pool, so why can the proxy not determine that certain backend instances are unavailable and remove them from the pool? No coordination needed and it also handles other cases where the backend is bad such as overload or deadlock? It also seems like part of your pain point is that there is an any-to-any rel…

Ok, again: this isn't a cluster of load balancers in front of a discrete collection of app servers in a data center. It's thousands of load balancers handling millions of applications scattered all over the world, with instances going up and down constantly. The interesting part of this problem isn't noticing that an instance is down . Any load balancer can do that. The interesting problem is noticing than and then i…

> The interesting problem is noticing than and then informing every proxy in the world.

Yes and that is why I suggested why your any-to-any relationship of proxy to application is a decision you have made which is part of the painpoint that caused you to come up with this solution. The fact that any proxy box can proxy to any backend is a choice which was made which created the structure and mental model you are working within. You could batch your proxies into say 1024 cells and then assign a customer app to say 4/1024 cells using shuffle sharding. Then that decomposes the problem into maintaining state within a cell instead of globally.

Im not saying what you did was wrong or dumb, I am saying you are working within a framework that maybe you are not even consciously aware of.

Re: Corrosion

#84

Post by tptacek »

__turbobrew__ wrote: post #83
tptacek wrote: post #80

Earlier quoted context omitted.

Ok, again: this isn't a cluster of load balancers in front of a discrete collection of app servers in a data center. It's thousands of load balancers handling millions of applications scattered all over the world, with instances going up and down constantly. The interesting part of this problem isn't noticing that an instance is down . Any load balancer can do that. The interesting problem is noticing than and then i…

> The interesting problem is noticing than and then informing every proxy in the world. Yes and that is why I suggested why your any-to-any relationship of proxy to application is a decision you have made which is part of the painpoint that caused you to come up with this solution. The fact that any proxy box can proxy to any backend is a choice which was made which created the structure and mental model you are work…

Again: it's the premise of the platform. If you're saying "you picked a hard problem to work on", I guess I agree.

We cannot in fact assign our customers apps to 0.3% of our proxies! When you deploy an app in Chicago on Fly.io, it has to work from a Sydney edge. I mean, that's part of the DX; there are deeper reasons why it would have to work that way (due to BGP4), but we don't even get there before becoming a different platform.

Re: Corrosion

#85

Post by tptacek »

otterley wrote: post #79
__turbobrew__ wrote: post #60

Earlier quoted context omitted.

> I'm thrilled to have people digging into this, because I think it's a super interesting problem Yes, somehow this is a problem all the big companies have, but it seems like there is no standard solution and nobody has open sourced their stuff (except you)! Taking a step back, and thinking about the AWS outage last week which was caused by a buggy bespoke system built on top of DNS, it seems like we need an IETF sta…

I work for AWS; opinions are my own and I’m not affiliated with the service team in question. That a DNS record was deleted is tangential to the proximate cause of the incident. It was a latent bug in the control plane that updated the records, not the data plane. If the discovery protocol were DNS++ or /etc/hosts files, the same problem could have happened. DNS has a lot of advantages: it’s a dirt cheap protocol to…

I'm nodding my head to this but have to call out that DNS with "interesting" RRs is extensively filtered by middleboxes --- just none of the middleboxes AWS would deploy or allow to be deployed anywhere it peers.

Re: Corrosion

#87

Post by __turbobrew__ »

tptacek wrote: post #84
__turbobrew__ wrote: post #83

Earlier quoted context omitted.

> The interesting problem is noticing than and then informing every proxy in the world. Yes and that is why I suggested why your any-to-any relationship of proxy to application is a decision you have made which is part of the painpoint that caused you to come up with this solution. The fact that any proxy box can proxy to any backend is a choice which was made which created the structure and mental model you are work…

Again: it's the premise of the platform. If you're saying "you picked a hard problem to work on", I guess I agree. We cannot in fact assign our customers apps to 0.3% of our proxies! When you deploy an app in Chicago on Fly.io, it has to work from a Sydney edge. I mean, that's part of the DX; there are deeper reasons why it would have to work that way (due to BGP4), but we don't even get there before becoming a diffe…

I think the impedance mismatch here is I am assuming we are talking about a hyperscaler cloud where it would be reasonable to have say 1024 proxies per region. Each app would be assigned to 4/1024 proxies in each region.

I have no idea how big of a compute footprint fly.io is, and maybe due to that the design I am suggesting makes no sense for you.

Re: Corrosion

#88

Post by tptacek »

__turbobrew__ wrote: post #87
tptacek wrote: post #84

Earlier quoted context omitted.

Again: it's the premise of the platform. If you're saying "you picked a hard problem to work on", I guess I agree. We cannot in fact assign our customers apps to 0.3% of our proxies! When you deploy an app in Chicago on Fly.io, it has to work from a Sydney edge. I mean, that's part of the DX; there are deeper reasons why it would have to work that way (due to BGP4), but we don't even get there before becoming a diffe…

I think the impedance mismatch here is I am assuming we are talking about a hyperscaler cloud where it would be reasonable to have say 1024 proxies per region. Each app would be assigned to 4/1024 proxies in each region. I have no idea how big of a compute footprint fly.io is, and maybe due to that the design I am suggesting makes no sense for you.

The design you are suggesting makes no sense for us. That's OK! It's an interesting conversation. But no, you can't fix the problem we're trying to solve with shuffle shard.

Re: Corrosion

#89

Post by __turbobrew__ »

otterley wrote: post #79
__turbobrew__ wrote: post #60

Earlier quoted context omitted.

> I'm thrilled to have people digging into this, because I think it's a super interesting problem Yes, somehow this is a problem all the big companies have, but it seems like there is no standard solution and nobody has open sourced their stuff (except you)! Taking a step back, and thinking about the AWS outage last week which was caused by a buggy bespoke system built on top of DNS, it seems like we need an IETF sta…

I work for AWS; opinions are my own and I’m not affiliated with the service team in question. That a DNS record was deleted is tangential to the proximate cause of the incident. It was a latent bug in the control plane that updated the records, not the data plane. If the discovery protocol were DNS++ or /etc/hosts files, the same problem could have happened. DNS has a lot of advantages: it’s a dirt cheap protocol to…

> It was a latent bug in the control plane that updated the records, not the data plane

Yes, I know that. But part of the issue is that the control plane exists in the first place to smooth the impedance mismatch between DNS and how dynamic service discovery works in practice. If we had a protocol which better handled dynamic service discovery, the control plane would be much less complex and less prone to bugs.

As far as I have seen, most cloud providers internally use their own service discovery systems and then layer dns on top of that system for third party clients to access. For example, DynamoDB is registered inside of AWS internal service discovery systems, and then the control plane is responsible for reconciling the service discovery state into DNS (the part which had a bug). If instead we have a standard protocol for service discovery, you can drop that in place of the AWS internal service discovery system and then clients (both internal and external) can directly resolve the DynamoDB backends without needing a DNS intermediary.

I don’t know how AWS or DynamoDB works in practice, but I have worked at other hyperscalers where a similar setup exists (DNS is layered on top of some internal service discovery system).

> If you’re doing to replace DNS, you’re going to have a steep hill to climb.

Yes, no doubt. But as we have seen with wireguard, if there is a good idea that has merit it can be quickly adopted into a wide range of operating systems and libraries.

Re: Corrosion

#90

Post by otterley »

__turbobrew__ wrote: post #89
otterley wrote: post #79

Earlier quoted context omitted.

I work for AWS; opinions are my own and I’m not affiliated with the service team in question. That a DNS record was deleted is tangential to the proximate cause of the incident. It was a latent bug in the control plane that updated the records, not the data plane. If the discovery protocol were DNS++ or /etc/hosts files, the same problem could have happened. DNS has a lot of advantages: it’s a dirt cheap protocol to…

> It was a latent bug in the control plane that updated the records, not the data plane Yes, I know that. But part of the issue is that the control plane exists in the first place to smooth the impedance mismatch between DNS and how dynamic service discovery works in practice. If we had a protocol which better handled dynamic service discovery, the control plane would be much less complex and less prone to bugs. As f…

> If instead we have a standard protocol for service discovery, you can drop [reconciliation] in place of the AWS internal service discovery system and then clients (both internal and external) can directly resolve the DynamoDB backends without needing a DNS intermediary.

DNS is a service discovery protocol! And a rather robust one, too. Don’t forget that.

AWS doesn’t want to expose to the customer all the dirty details of how internal routing is done. They want to publish a single regional service endpoint, put a SLO on it, and handle all the complexity themselves. Saving unnecessary complexity from customers is, after all, one of the key value propositions of a managed service. It also allows the service provider the flexibility to change the underlying implementation without impacting customer clients.

I’m not sure the best response to “the reconciler had a bug, and other reconcilers might, too” is to replace it with an entirely new and untested service discovery protocol. A proposed compensating control to this bug might be as simple as “if the result would be to delete the zone or empty it of all RRs, halt and page the on-call.” Fail open, as it were.

Also, anyone proposing a new protocol in response to a problem—especially one that had nothing to do with the protocol itself—should probably be burdened with defining and implementing its replacement. ;)

Post reply on HN