Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Corrosion

fly.io

Post reply on HN
Print view View original

31–40 of 109 posts

Re: Corrosion

#31

Post by blinkingled »

> The bidding model is elegant, but it’s insufficient to route network requests. To allow an HTTP request in Tokyo to find the nearest instance in Sydney, we really do need some kind of global map of every app we host.

So is this a case of wanting to deliver a differentiating feature before the technical maturity is there and validated? It's an acceptable strategy if you are building a lesser product but if you are selling Public Cloud maybe having a better strategy than waiting for problems to crop up makes more sense? Consul, missing watchdogs, certificate expiry, CRDT back filling nullable columns - sure in a normal case these are not very unexpected or to-be-ashamed-of problems but for a product that claims to be Public Cloud you want to think of these things and address them before day 1. Cert expiry for example - you should be giving your users tools to never have a cert expire - not fixing it for your stuff after the fact! (Most CAs offer API to automate all this - no excuse for it.)

I don't mean to be dismissive or disrespectful, the problem is challenging and the work is great - merely thinking of loss of customer trust - people are never going to trust a new comer that has issues like this and for that reason move fast break things and fix when you find isn't a good fit for this kind of a product.

Re: Corrosion

#32

Post by tptacek »

kg wrote: post #29
tptacek wrote: post #24

Earlier quoted context omitted.

The design hasn't changed in years. If someone has a screenshot and a browser version we can try to figure out why it's coming out fucky for you.

Looking at the css, there's a .text-gray-600 CSS style that would cause this, and it's overridden by some other style in order to achieve the actual desired appearance. Maybe the override style isn't loading - perhaps the GP has javascript disabled?

Thanks! Relayed.

Re: Corrosion

#33

Post by tptacek »

blinkingled wrote: post #31

> The bidding model is elegant, but it’s insufficient to route network requests. To allow an HTTP request in Tokyo to find the nearest instance in Sydney, we really do need some kind of global map of every app we host. So is this a case of wanting to deliver a differentiating feature before the technical maturity is there and validated? It's an acceptable strategy if you are building a lesser product but if you are s…

It's not a "differentiating feature"; it eliminated a scaling bottleneck. It's also a decision that long predates Corrosion.

Re: Corrosion

#34

Post by conradev » 

  To ensure every instance arrives at the same “working set” picture, we use cr-sqlite, the CRDT SQLite extension.
Cool to see cr-sqlite used in production!

Re: Corrosion

#35

Post by blinkingled »

tptacek wrote: post #33
blinkingled wrote: post #31

> The bidding model is elegant, but it’s insufficient to route network requests. To allow an HTTP request in Tokyo to find the nearest instance in Sydney, we really do need some kind of global map of every app we host. So is this a case of wanting to deliver a differentiating feature before the technical maturity is there and validated? It's an acceptable strategy if you are building a lesser product but if you are s…

It's not a "differentiating feature"; it eliminated a scaling bottleneck. It's also a decision that long predates Corrosion.

I was referring to the "HTTP request in Tokyo to find the nearest instance in Sydney" part which felt to me like a differentiating feature- no other cloud provider seems to have bidding or HTTP request level cross regional lookup or whatever.

The "decision that long predates Corrosion" is precisely the point I was trying to make - was it made too soon before understanding the ramifications and/or having a validated technical solution ready? IOW maybe the feature requiring the problem solution could have come later? (I don't know much about fly.io and its features, so apologies if some of this is unclear/wrongly assumes things.)

Re: Corrosion

#36

Post by tptacek »

blinkingled wrote: post #35
tptacek wrote: post #33

Earlier quoted context omitted.

It's not a "differentiating feature"; it eliminated a scaling bottleneck. It's also a decision that long predates Corrosion.

I was referring to the "HTTP request in Tokyo to find the nearest instance in Sydney" part which felt to me like a differentiating feature- no other cloud provider seems to have bidding or HTTP request level cross regional lookup or whatever. The "decision that long predates Corrosion" is precisely the point I was trying to make - was it made too soon before understanding the ramifications and/or having a validated t…

That's literally the premise of the service and always has been.

Re: Corrosion

#37

Post by __turbobrew__ »

tptacek wrote: post #21
bananapub wrote: post #6

in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" - you can formal method and Rust and test and watchdog yourself as much as you want, but you simply have to stop doing that or the unknown unknowns will just keep taking you down.

I mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down…

> Proxies in Asia need to know about that, right away, and this time you can't afford to wait.

Did you ever consider envoy xDS?

There are a lot of really cool things in envoy like outlier detection, circuit breakers, load shedding, etc…

Re: Corrosion

#38

Post by tptacek »

__turbobrew__ wrote: post #37
tptacek wrote: post #21

Earlier quoted context omitted.

I mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down…

> Proxies in Asia need to know about that, right away, and this time you can't afford to wait. Did you ever consider envoy xDS? There are a lot of really cool things in envoy like outlier detection, circuit breakers, load shedding, etc…

Nope. Talk a little about how how Envoy's service discovery would scale to millions of apps in a global network? There's no way we found the only possible point in the solution space. Do they do something clever here?

What we (think we) know won't work is a topologically centralized database that uses distributed consensus algorithms to synchronize. Running consensus transcontinentally is very painful, and keep the servers central, so that update proposals are local and the protocol can run quickly, subjects large portions of the network to partition risk. The natural response (what I think a lot of people do, in fact) is just to run multiple consensus clusters, but our UX includes a global namespace for customer workloads.

Re: Corrosion

#40

Post by nodesocket »

Anybody used rqlite[1] in production? I'm exploring how to make my application fault-tolerant using multiple app vm instances. The problem of course is the SQLite database on disk. Using a network file system like NFS is a no-go with SQLite (this includes Amazon Elastic File System (EFS)).

I was thinking I'll just have to bite the bullet and migrate to PostgreSQL, but perhaps rqlite can work.

[1] https://rqlite.io

Post reply on HN