Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How Long Does a Deployment Take at Your Company

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Ask HN: How Long Does a Deployment Take at Your Company

#31

Post by rvdginste »

shoo wrote: post #9

I worked on a little web service in a much larger system, that was deployed to the company's on-prem datacentres. To deploy the entire system, maybe it'd take a few hours: there'd be 1-2 dozen services, a few databases, probably 20+ external integrations. Usually we'd only deploy the services or components that were changing. Deploying a single service would only take a few minutes, even with human in loop manually t…

How do you handle database/data store upgrades? It seems there is a window where both the old and new system write into the same data store.

Re: Ask HN: How Long Does a Deployment Take at Your Company

#32

Post by shoo »

rvdginste wrote: post #31
shoo wrote: post #9

I worked on a little web service in a much larger system, that was deployed to the company's on-prem datacentres. To deploy the entire system, maybe it'd take a few hours: there'd be 1-2 dozen services, a few databases, probably 20+ external integrations. Usually we'd only deploy the services or components that were changing. Deploying a single service would only take a few minutes, even with human in loop manually t…

How do you handle database/data store upgrades? It seems there is a window where both the old and new system write into the same data store.

Great question. I don't know the details of how database schema changes were deployed, I only worked on the easy stuff - stateless web services.

Exactly as you say, there is a time window where both the old and new system write to the same data store. Both old and new systems, and the details of the deployment, need to be designed to tolerate this. Even if there is no change to the database schema, you need to think through what will happen if the old version of a component reads data written to the database by a newer version of that same component, or vice versa. Similar considerations if you need to roll back to the old version after the new version has run in production for a few hours, but the newly written data is still there. This can all be planned out and tested in staging.

I don't think this is unique to the blue / green deployment pattern. If you did a rolling deployment to upgrade app servers in a pool behind some customer-traffic facing load balancer, there would be a time window when both old and new versions of your app servers are all attached to your database. Same fundamental problem.

Re: Ask HN: How Long Does a Deployment Take at Your Company

#33

Post by gfarah »

As other have mentioned depending on what is being deployed.

Lambda/Cloud Functions code: testing 1m-5m, deployment Containers: testing 5m-1h (depending on build time and type of tests), deployment 3m-10m.

Migrations: anywhere from 1m up to 1h depending on the tables/migration type and the number of affected PostgreSQL instances.

Infraestructure: anywhere from 2m up to 8h depending on what's being changed.

Re: Ask HN: How Long Does a Deployment Take at Your Company

#34

Post by pitsnatch »

throwaway118899 wrote: post #10

Throwaway for obvious reason. But… 1-2 days. Multiple senior engineers cherry picking commits into a release branch with even more seniors doing atteststion. It’s a company-wide effort that happens every sprint. We have “staff” SREs who can’t figure out automated releases.

That sounds hilarious

Re: Ask HN: How Long Does a Deployment Take at Your Company

#36

Post by rozenmd »

OnlineOrNot's frontend deploys in about 60 seconds if the cache is warm, two and a half minutes otherwise.

The backend services take about two minutes.

(It's my own company, so I took the time to make deployments as fast as possible since I value short feedback loops)

Re: Ask HN: How Long Does a Deployment Take at Your Company

#37

Post by rvdginste »

shoo wrote: post #32
rvdginste wrote: post #31

Earlier quoted context omitted.

How do you handle database/data store upgrades? It seems there is a window where both the old and new system write into the same data store.

Great question. I don't know the details of how database schema changes were deployed, I only worked on the easy stuff - stateless web services. Exactly as you say, there is a time window where both the old and new system write to the same data store. Both old and new systems, and the details of the deployment, need to be designed to tolerate this. Even if there is no change to the database schema, you need to think…

We intend to do something similar. The users get a pop-up in the front end when a new version is available and are asked to load the new version, but they can keep on using the old version for a while. The length of the time window does not really matter that much: whether it is 5 minutes or several hours, the issue is the same. Regarding the database upgrades, I believe that if you really want to, you can split up a not-backwards compatible upgrade into multiple upgrades that are each one on one backwards compatible. But that is extra work and like you said you really have to think those things through for every change you roll out. So I think the extra effort is not always worth it and also I wonder how other people are approaching this.

Re: Ask HN: How Long Does a Deployment Take at Your Company

#38

Post by ktrnka »

If I remember right, my team's pipelines took about 40-60 min on average. We deployed machine learning models and code to lambda via cdk. The things that made it slow were having to deploy dev staging and prod with no resource sharing between them, docker builds and uploads, and end to end tests.

Re: Ask HN: How Long Does a Deployment Take at Your Company

#39

Post by Rev359a »

It depends on country and companies.

If you are in the company which has consultancy, consultants and freelancers and permanent employees - it can take ages because consultancy, consultants and freelancers are all competing each other while sucking the money from the companies. Every deployment in this case will result in a disaster because they never work with each other - they are just sucking the money and the permanent employee suffers and they start leaving for better options

If you are in the company which has talented permanent employees and good visionary leaders who work with the spirit of team building and team work I can assure you it will be as easy as pie - they work with each other and they make good results and take responsibility.

Piece of advice look for silos in your company and make them work as a team and it will take less time.

Development time can be more but deployment should always be easy if you have the right people. Try to hire right people for the right job

Deployment now a days is all cloud managed with in house or service providers like aws, google or azure and mostly offers serverless solutions as most of the databases and services are available as a service and a good DevOps would do it easily without taking much time provided if your team has people who do not like to work in silos and compete with each other

Re: Ask HN: How Long Does a Deployment Take at Your Company

#40

Post by iends »

I work on a large enterprise SaaS. It depends on how long a services test take, but our deploys to production require approval by the security team, which all happens via JIRA approvals. So less than a day. If it's break the glass urgent in under an hour, mostly constrained by test automation. If it was something insane, I could probably do it in under 10 minutes for a single amazon region by turning off the test automation, but never had to do this. Feature toggles only take about 5 minutes to propagate and much less work than doing a deploy.

We have Dev, Test, & 10+ prod regions. The service I work on takes about an hour to run tests in each region, but that involves almost 10 years of test automation, building a custom AMI on EC2, and deploying. There is also cross region AMI copying which slows things down.

To bootstrap a new region takes about 2 months for the entire product with developers kinda working in the background. My service takes about a week worth of work, but lots of external dependencies and issues pop up. We do this about once a year so its' almost not worth optimizing for.

Post reply on HN