Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

github.com

Post reply on HN
Print view View original

41–50 of 133 posts

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#41

Post by diggan »

wejick wrote: post #31
diggan wrote: post #17

Earlier quoted context omitted.

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…

This is very good explanation, no judgment and simply educational. Appreciated Though I'm still surprised that some people run DB alteration on application start up. Never saw one in real life.

> Though I'm still surprised that some people run DB alteration on application start up

I think I've seen it more commonly in the Golang ecosystem, for some reason. Also not sure how common it is nowadays, but seen lots of deployments (contained in Ansible scripts, Makefiles, Bash scripts or whatever) where the migration+deploy is run directly in sequence automatically for each deploy, rather than as discrete steps.

Edit: Maybe it's more of an educational problem than something else, where learning resources either don't specify when to actually run migrations or straight up recommend people to run migrations on application startup (one example: https://articles.wesionary.team/integrating-migration-tool-i...)

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#42

Post by stephenr »

diggan wrote: post #38
stephenr wrote: post #22

Earlier quoted context omitted.

Others have described the how part if you do need truly zero downtime deployments, but I think it's worth pointing out that for most organisations, and most migrations, the amount of downtime due to a db migration is virtually indistinguishable from zero, particularly if you have a regional audience, and can aim for "quiet" hours to perform deployments.

> the amount of downtime due to a db migration is virtually indistinguishable from zero Besides, once you've run a service for a while that has acquired enough data for migrations to take a while, you realize that there are in fact two different types of migrations. "Schema migrations" which are generally fast and "Data migrations" that depending on the amount of data can take seconds or days. Or you can do the "data…

Very much so, we handle these very differently for $client.

Schema migrations are versioned in git with the app, with up/down (or forward/reverse) migration scripts and are applied automatically during deployment of the associated code change to a given environment.

SQL Data migrations are stored in git so we have a record but are never applied automatically, always manually.

The other thing we've used along these lines, is having one or more low priority job(s) added to a queue, to apply some kind of change to records. These are essentially still data migrations, but they're written as part of the application code base (as a Job) rather than in SQL.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#43

Post by jgalt212 »

ahoka wrote: post #35
jgalt212 wrote: post #18

Earlier quoted context omitted.

You still need some warm-up routine to run for the newly online server before the hand-off occurs. I'm not a k8s expert, but the above described events can be easily handled by a bash or fab script.

What events do you mean? If the app needs a warm up, then it can use its readiness probe to ask for some delay until it gets request routed to it.

GET requests to pages that fill caches or those that make apache start up more than n processes.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#44

Post by nchmy »

ksec wrote: post #36
olieidel wrote: post #6

Earlier quoted context omitted.

> I feel like in the span between them there are far more companies for which Kamal is not enough. I feel like this is a bias in the HN bubble: In the real world, 99% of companies with any sort of web servers (cloud or otherwise) are running very boring, constant, non-Uber workloads.

Not just HN but overall the whole internet. Because all the news and article, tech achievements are pumped out from Uber and other big tech companies. I am pretty sure Uber belongs to the 1% of the internet companies in terms of scale. 37Signals isn't exactly small either. They spend $3M a year on infrastructure in 2019. Likely a lot higher now. The whole Tech cycle needs to stop having a top down approach where ever…

They spend considerably less on infra now - this was the entire point of moving off cloud. DHH has written and spoken lots about it, providing real numbers. They bought their own servers and the savings paid for it all in like 6 months. Now its just money in the bank til they replace the hardware in 5 years.

Cloud is a scam for the vast majority of companies.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#45

Post by miki123211 »

wejick wrote: post #31
diggan wrote: post #17

Earlier quoted context omitted.

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…

This is very good explanation, no judgment and simply educational. Appreciated Though I'm still surprised that some people run DB alteration on application start up. Never saw one in real life.

It makes things somewhat easier if your app is smallish and your workflow is something like e.g. Github Actions automatically deploying all commits on main to Fly or Render.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#47

Post by miki123211 »

martinald wrote: post #9
oDot wrote: post #2

DHH mentioned they built it to move from the cloud to bare metal. He glorifies the simplicity but I can't help thinking they are a special use case of predictable, non-huge load. Uber, for example, moved to the cloud. I feel like in the span between them there are far more companies for which Kamal is not enough. I hope I'm wrong, though. It'll be nice for many companies to be have the choice of exiting the cloud.

I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune. The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. Slow single thread performance because of using the most parallel CPUs p…

You can always buy some servers to handle your base load, and then get extra cloud instances when needed.

If you're running an ecommerce store for example, you could buy some extra capacity from AWS for Christmas and Black Friday, and rely on your own servers exclusively for the rest of the year.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#48

Post by vorticalbox »

irundebian wrote: post #7
000ooo000 wrote: post #3

Strange choice of language for the actions: >To route traffic through the proxy to a web application, you *deploy* instances of the application to the proxy. *Deploying* an instance makes it available to the proxy, and replaces the instance it was using before (if any). >e.g. `kamal-proxy deploy service1 --target web-1:3000` 'Deploy' is a fairly overloaded term already. Fun conversations ahead. Is the app deployed? Y…

Deployed = running and registered to the proxy.

Then wouldn’t “register” be a better term?

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#49

Post by miki123211 »

toomuchtodo wrote: post #39
jsheard wrote: post #37

Earlier quoted context omitted.

It's sad that what should have been a huge efficiency win, amortizing hardware costs across many customers, ended up often being more expensive than just buying big servers and letting them idle most of the time. Not to say the efficiency isn't there, but the cloud providers are pocketing the savings.

If you want a compute co-op, build a co-op (think VCs building their own GPU compute clusters for portfolio companies). Public cloud was always about using marketing and the illusion of need for dev velocity (which is real, hypergrowth startups and such, just not nearly as prevalent as the zeitgeist would have you believe) to justify the eye watering profit margin. Most businesses have fairly predictable interactive…

Wikipedia is often brought up in these discussions, but it's a really bad example.

To a vast majority of Wikipedia users who are not logged in, all it needs to do is show (potentially pre-rendered) article pages with no dynamic, per-user content. Those pages are easy to cache or even offload to a CDN. FOr all the users care, it could be a giant key-value store, mapping article slugs to HTML pages.

This simplicity allows them to keep costs down, and the low costs mean that they don't have to be a business and care about time-on-page, personalized article recommendations or advertising.

Other kinds of apps (like social media or messaging) have very different usage patterns and can't use this kind of structure.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#50

Post by toomuchtodo »

miki123211 wrote: post #49
toomuchtodo wrote: post #39

Earlier quoted context omitted.

If you want a compute co-op, build a co-op (think VCs building their own GPU compute clusters for portfolio companies). Public cloud was always about using marketing and the illusion of need for dev velocity (which is real, hypergrowth startups and such, just not nearly as prevalent as the zeitgeist would have you believe) to justify the eye watering profit margin. Most businesses have fairly predictable interactive…

Wikipedia is often brought up in these discussions, but it's a really bad example. To a vast majority of Wikipedia users who are not logged in, all it needs to do is show (potentially pre-rendered) article pages with no dynamic, per-user content. Those pages are easy to cache or even offload to a CDN. FOr all the users care, it could be a giant key-value store, mapping article slugs to HTML pages. This simplicity all…

> Other kinds of apps (like social media or messaging) have very different usage patterns and can't use this kind of structure.

Reddit can’t turn a profit, Signal is in financial peril. Meta runs their own data centers. WhatsApp could handle ~3M open TCP connections per server, running the operation with under 300 servers [1] and serving ~200M users. StackOverflow was running their Q&A platform off of 9 on prem servers as of 2022 [2]. Can you make a profitable business out of the expensive complex machine? That is rare, based on the evidence. If you’re not a business, you’re better off on Hetzner (or some other dedicated server provider) boxes with backups. If you’re down you’re down, you’ll be back up shortly. Downtime is cheaper than five 9s or whatever.

I’m not saying “cloud bad,” I’m saying cloud where it makes sense. And those use cases are the exception, not the rule. If you're not scaling to an event where you can dump these cloud costs on someone else (acquisition event), or pay for them yourself (either donations, profitability, or wealthy benefactor), then it's pointless. It's techno performance art or fancy make work, depending on your perspective.

[1] https://news.ycombinator.com/item?id=33710911

[2] https://www.datacenterdynamics.com/en/news/stack-overflow-st...

Post reply on HN