Moving Fast and Securing Things • Hacker News Board

Re: Moving Fast and Securing Things

#21

Post by toomuchtodo » Fri, May 04, 2018, 3:22 PM UTC

wgerard wrote: post #16

toomuchtodo wrote: post #8

Earlier quoted context omitted.

It’s usually a Sisyphean task. Everyone wants to look and act cutting edge (“but Netflix!”) but nobody wants to make the necessary investments in the tooling, org structure, and management ability/support that is required to support that sort of deployment cadence (if your org focuses on who broke something instead of the process, and management doesn’t want to change that culture, all hope is already lost [based on…

> There are some legitimate needs for continuous deployment, the rest of it is cargo culting. Maybe, but I wouldn't go that far. Small companies already often do CD, because there's rarely a rigid deploy schedule. It's a practice people understand and feel the benefits of immediately. If you ask someone who moved from a small startup to a huge company what their biggest complaints are, I bet "longer/stricter deploy p…

For every developer that complains about the longer/stricter deploy process, I'd offer up for consideration deployments that went out through the CD pipeline where production data was mangled with no rollback possible. As with everything, its determining your appetite for risk.

Re: Moving Fast and Securing Things

#22

Post by kendallpark » Fri, May 04, 2018, 3:24 PM UTC

jimktrains2 wrote: post #19

vasilakisfil wrote: post #15

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

They were/are helpful in making medicine safer too. https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.ncbi.nlm.nih.gov/pubmed/24116973

Same with aviation.

http://qualitysafety.bmj.com/content/24/7/428

Re: Moving Fast and Securing Things

#23

Post by wgerard » Fri, May 04, 2018, 3:28 PM UTC

toomuchtodo wrote: post #21

wgerard wrote: post #16

Earlier quoted context omitted.

> There are some legitimate needs for continuous deployment, the rest of it is cargo culting. Maybe, but I wouldn't go that far. Small companies already often do CD, because there's rarely a rigid deploy schedule. It's a practice people understand and feel the benefits of immediately. If you ask someone who moved from a small startup to a huge company what their biggest complaints are, I bet "longer/stricter deploy p…

For every developer that complains about the longer/stricter deploy process, I'd offer up for consideration deployments that went out through the CD pipeline where production data was mangled with no rollback possible. As with everything, its determining your appetite for risk.

Hmm, I don't see how that changes with longer/stricter deploy processes - unless you have some of the tooling around that makes CD very possible in the first place (automated checks/etc.)

I've certainly worked in places with very long and strict deploy processes that managed to mangle production data frequently. Even worse, because the deploy process was so strict and long the bad code managed to stay on production for much longer than 10 minutes (the deploy time mentioned in the article).

There's some vague notion out there that long deploy process == safe, but there's very little evidence to suggest that's the case. If anything, it seems much more dangerous because larger changesets are going out all at once.

Re: Moving Fast and Securing Things

#24

Post by lightedman » Fri, May 04, 2018, 3:31 PM UTC

ejcx wrote: post #12

I love this! If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate. It seems like some simple checklist app but having a non Jira process that takes only a few minutes is so valuable, and "security reviews" and "threat models" as part of your SDLC take insane amounts of time and honestly aren't super helpful.

"If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate."

Except we just saw automation fail with two big companies - Twitter and GitHub.

Re: Moving Fast and Securing Things

#25

Post by toomuchtodo » Fri, May 04, 2018, 3:32 PM UTC

wgerard wrote: post #23

toomuchtodo wrote: post #21

Earlier quoted context omitted.

For every developer that complains about the longer/stricter deploy process, I'd offer up for consideration deployments that went out through the CD pipeline where production data was mangled with no rollback possible. As with everything, its determining your appetite for risk.

Hmm, I don't see how that changes with longer/stricter deploy processes - unless you have some of the tooling around that makes CD very possible in the first place (automated checks/etc.) I've certainly worked in places with very long and strict deploy processes that managed to mangle production data frequently. Even worse, because the deploy process was so strict and long the bad code managed to stay on production f…

It goes back to my original comment above; if you have the proper tooling (tests executed that must pass prior to deploy given a green light, green/blue deploys, canaries, automated datastore snapshots/point in time recovery, granular control of the deployment process), I think continuous deployment provides a great deal of value above what you've invested into the process. But that investment is critical if you've bought into CD. Otherwise, it's "deploy and pray".

Re: Moving Fast and Securing Things

#26

Post by wgerard » Fri, May 04, 2018, 3:36 PM UTC

toomuchtodo wrote: post #25

wgerard wrote: post #23

Earlier quoted context omitted.

Hmm, I don't see how that changes with longer/stricter deploy processes - unless you have some of the tooling around that makes CD very possible in the first place (automated checks/etc.) I've certainly worked in places with very long and strict deploy processes that managed to mangle production data frequently. Even worse, because the deploy process was so strict and long the bad code managed to stay on production f…

It goes back to my original comment above; if you have the proper tooling (tests executed that must pass prior to deploy given a green light, green/blue deploys, canaries, automated datastore snapshots/point in time recovery, granular control of the deployment process), I think continuous deployment provides a great deal of value above what you've invested into the process. But that investment is critical if you've b…

Sure, and I guess my point is: If you haven't invested in those things, waterfall-esque deploy processes are just as bad and perhaps even worse because there's more chance for confounding changes to cause a nasty error.

The only reason waterfall-esque deploy processes work without those things is because companies often waste tons of people-hours on testing things out in the staging environment (which requires time, obviously).

Re: Moving Fast and Securing Things

#27

Post by ggregoire » Fri, May 04, 2018, 3:48 PM UTC

vasilakisfil wrote: post #15

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Re: Moving Fast and Securing Things

#28

Post by auntad » Fri, May 04, 2018, 3:50 PM UTC

vasilakisfil wrote: post #15

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

This. There are some great bits about checklists and their (underutilized) role in preventing human error in Don Norman's Design of Everyday Things (chapter 5).

Re: Moving Fast and Securing Things

#29

Post by insensible » Fri, May 04, 2018, 3:56 PM UTC

ejcx wrote: post #12

I love this! If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate. It seems like some simple checklist app but having a non Jira process that takes only a few minutes is so valuable, and "security reviews" and "threat models" as part of your SDLC take insane amounts of time and honestly aren't super helpful.

What I think is brilliant here is that that sort of work can take place separately and provide feedback to these checklists when problems or deficiencies are found. Basically the security checklists are a deliverable that can be iterated on independently while they still benefit from the existing one.

Re: Moving Fast and Securing Things

#30

Post by perpetualcrayon » Fri, May 04, 2018, 3:58 PM UTC

wgerard wrote: post #7

boffinism wrote: post #2

> The process of deploying code to production is very simple, and takes about ten minutes total. This results in a life cycle in which we deploy code to production approximately 100 times per day. What? They spend 1000 minutes out of every 1440 deploying to production? The deployment process is occurring over 16 hours out of every 24? Am I the only one who is nonplussed by this? EDIT: Ok I get it, I get it. I guess I…

Not a slack employee but worked at a company with similar CD views: (Likely) various groups of people are deploying to production throughout the day. Out of those 100 deploys, an individual is probably only involved in 1 or 2 a day. As soon as you're ready to deploy your code, you queue up and see it all the way through to production along with probably a few other people doing the same thing. The actual "change the…

I once worked on a project for one of the largest corporations in the world. First part of my career I only ever worked on smaller teams.

I was excited for the move to a large corporation where there would be amazing room for growth and learning.

I have to say that almost a year into my work on this project, I was absolutely stunned how inept this company was at coordinating a technology project.

Something a small team could accomplish in a matter of months was taking 100's of developers and 100's more in supporting / operational roles years to accomplish. My guess is the developers on this project would gladly trade places with Sisyphus.