21–30 of 42 posts
Throwaway for obvious reason. But… 1-2 days. Multiple senior engineers cherry picking commits into a release branch with even more seniors doing atteststion. It’s a company-wide effort that happens every sprint. We have “staff” SREs who can’t figure out automated releases.
...right?
Depending on /what/ is being deployed, somewhere between "seconds" and "it's literally impossible so let's hope we don't have to".
We can see in some of the other answers that comments assume wildly different meanings for this. From when code is pushed to it being available in production to how long it takes to start the service.
My experience include a lot of the whole spectrum. Back at a previous place, if you were unlucky, i.e. this is the start of the quarter and you made a small change that took you 5 minutes to make, you'd have to wait a quarter of a year minus 5 minutes until your change would be deployed to production during a weekend night in which the application would be taken offline (actually not fully offline, but read-only state w/ queues for write operations not delivering messages but taking them). This was for a large telecom provider's backend (ordering) systems.
To nowadays where the smallest service for our SaaS starts up in a second or two, depending on what you count. Does the cluster have room for the new pod? Yeah it's seconds. Does k8s think it needs to add a new node? Well you're gonna wait a bit. And yes some services take minutes to initialize. But no matter what, customers won't notice. Even if there's a database update that's included in the changeset that gets deployed and that runs hours per tenant (times thousands of tenants), the services will be available during that time. It has to be coded that way and be deployed in stages.
- CI steps to build and package containers usually take 1-10 minutes, depending on whether caches are used
- running unit tests can take 1-5 minutes, depending on the system and infrastructure
- running integration tests can take 5-30 minutes, depending on the system and infrastructure
- scanning the build artifacts can take around 5 minutes (e.g. Trivy)
- uploading them to a container registry will usually take 1-5 minutes, depending on the network speed
- launching new containers will probably take 1-10 minutes, depending on whether there's DB migrations etc.
Things that are especially useful on a technical level: a package cache (e.g. Maven ".m2" folder) or a self-hosted package repository (like Sonatype Nexus), maybe both; some sort of a build cache, which you largely get out of the box when working with containers, especially if you do multi-stage builds with an optimized build order/layers (e.g. first dependencies that change infrequently, then the code); a setup where you parallelize lots of the build steps and can add new runner/follower servers for actually doing the steps
The human aspects:
- the people delivering the software might not be the same ones running it, registering a release with instructions might take 30-60 minutes
- the people who will run the new version might need to change all of the necessary configuration for the new version, which might take another 30 or so minutes
- the people who will demand that a new version be launched on any given infrastructure might need to be made aware of the new release, which might take around 30 minutes
- before anything goes into prod or moves across different environments, testing and further fixes might be necessary, which can take from a day to a few weeks
Personally, I've been in scenarios where I've deployed new versions to prod in minutes, and I've seen cases where new releases of software haven't been deployed to prod in months, despite technically being delivered. Everything from fully automated pipelines, to shipping manually built binaries (thankfully this was years ago, for nothing important; I promptly setup proper CI/CD regardless).
Then again, the kinds of software that people work on might differ a lot. Here's an interesting post from a while ago: https://news.ycombinator.com/item?id=18442941
It would be good to distinguish build-and-deploy which Vercel does by default to other pipelines where you might build the artefacts, test them, then the deploy is more of a copying of those artefacts into production. In the latter case the “deploy” would be faster.
I would probably condense my definition to how quickly can you go from new code added to someone seeing that code running (in web dev terms essentially available on some URL). So even if you had different layers of pipeline, the build artifact stage would still be included, because you can't deploy without it.
Throwaway for obvious reason. But… 1-2 days. Multiple senior engineers cherry picking commits into a release branch with even more seniors doing atteststion. It’s a company-wide effort that happens every sprint. We have “staff” SREs who can’t figure out automated releases.
Attestation makes it sound like a social, not technical blocker to automated releases.
Sometimes there could also be attestation for things like "does the design and implementation of new service's architecture comply with security policy - does it have the approval of the security representative". some of these things can be automated away with tooling -- e.g. install a tool to check for glaring container security vulnerabilities that runs at code review time, and block changes from being merged or being deployed into dev or staging environments if there are high or critical sev security issues, with some manual process to override the inevitable false positives when the tooling makes a silly decision.
All that said, it isn't necessarily practical or cost effective to attempt to automate everything. Skilled human expert review of proposed designs and implementations for security flaws will likely do a much, much more effective job than relying on tools alone. So it's quite reasonable for designs and implementation to require independent security review and approval (or "attestation") in contexts where security is critical. But if things like security review of implementation (let alone design) is being done while cherry-picking commits onto a release branch, that sounds far too late in the lifecycle.
Another familiar example of attestation is things like code review, in projects that require all changes to be approved by one (or perhaps more) reviewers. By approving a PR, the reviewer "attests" that the change looks good to them, that it will implement the stated requirements and doesn't have any obvious flaws, etc. High quality code review is incredibly valuable, requires manual effort, but can be structured into sensible release and deployment processes, particularly by taking advantage of github / gitlab etc.
This reviewing and "attesting" activity can happen much earlier in the lifecycle than deployment or activation.
Depends if there's a database migration or not. If no database migration, than maybe a minute or two (via GitHub actions). If there is a database migration it might take a while depending on how much data needs to get moved around. Adding columns usually takes no time. Adding indexes to a big table takes a few hours.