Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving Fast and Securing Things

slack.engineering

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Moving Fast and Securing Things

#41

Post by peterwwillis »

ggregoire wrote: post #27
vasilakisfil wrote: post #15

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Exactly. Checklists are great for non-critical tasks. Hospital triage uses the same checklist for pretty much every new patient. Triage is boring, but the fact that it is checklisted is what helps drive efficiency in the ER.

Similar boring processes in a tech business can be checklisted to increase efficiency, and the checklist itself can be iterated over. Everything from designing a new feature, to troubleshooting an error, to addressing a customer support ticket, to getting access to a new resource should use checklists.

Re: Moving Fast and Securing Things

#42

Post by icebraining »

ggregoire wrote: post #27
vasilakisfil wrote: post #15

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

I've been convinced that checklists are great (by theory and by practice), yet I still write way fewer than I should.

I strongly dislike repetitive mental work, and writing a checklist is essentially resigning myself that such work will be necessary. Until I write it, I can still convince myself I'll be able to automate the process.

Re: Moving Fast and Securing Things

#44

Post by electrum »

xyzzy_plugh wrote: post #6
boffinism wrote: post #2

> The process of deploying code to production is very simple, and takes about ten minutes total. This results in a life cycle in which we deploy code to production approximately 100 times per day. What? They spend 1000 minutes out of every 1440 deploying to production? The deployment process is occurring over 16 hours out of every 24? Am I the only one who is nonplussed by this? EDIT: Ok I get it, I get it. I guess I…

I think you're assuming that each deploy is somehow disruptive or blocking, as if systems enter some "deploy mode". This may be true of a fleet overall, but is not when you take an instance into account. Instead imagine each deploy as an edge, and imagine them to be near instantaneous. With respect to an instance, and a user, and the observable effects of a deploy, this paints a more accurate picture. 100 times a day…

Deployment at Facebook: https://code.facebook.com/posts/270314900139291/rapid-releas...

Re: Moving Fast and Securing Things

#46

Post by Domenic_S »

peterwwillis wrote: post #41
ggregoire wrote: post #27

Earlier quoted context omitted.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Exactly. Checklists are great for non-critical tasks. Hospital triage uses the same checklist for pretty much every new patient. Triage is boring, but the fact that it is checklisted is what helps drive efficiency in the ER. Similar boring processes in a tech business can be checklisted to increase efficiency, and the checklist itself can be iterated over. Everything from designing a new feature, to troubleshooting a…

Checklists are great for critical tasks too. Every phase of a flight has an associated checklist, for example.

Re: Moving Fast and Securing Things

#47

Post by wpietri »

vasilakisfil wrote: post #15

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

I think they work much better (and are less boring) when the checklists are living artifacts created by the people who use them.

Re: Moving Fast and Securing Things

#49

Post by vlovich123 »

wgerard wrote: post #26
toomuchtodo wrote: post #25

Earlier quoted context omitted.

It goes back to my original comment above; if you have the proper tooling (tests executed that must pass prior to deploy given a green light, green/blue deploys, canaries, automated datastore snapshots/point in time recovery, granular control of the deployment process), I think continuous deployment provides a great deal of value above what you've invested into the process. But that investment is critical if you've b…

Sure, and I guess my point is: If you haven't invested in those things, waterfall-esque deploy processes are just as bad and perhaps even worse because there's more chance for confounding changes to cause a nasty error. The only reason waterfall-esque deploy processes work without those things is because companies often waste tons of people-hours on testing things out in the staging environment (which requires time,…

The thing you're missing is that you're amortizing the cost. Yeah, it's typically prohibitive to run the manual testing on every CL. However, if you have any manual testing you need to run, then at some point you have to batch the changes & test them out together anyway. Automated tests don't necessarily solve this problem either. 1) Some automated tests can be time-consuming & so require batching of CLs to run too 2) it's impossible to predict if you are going to catch all issues via automated testing 3) there's always things it's easier to test for manually.

When it comes to data integrity, I would think you need a structured mechanism (at least for larger teams that have a high cost for failure) for rolling back any given CL either by tracking writes, making sure to have a plan in place to recover from any given CL (e.g. nuking the data doesn't break things), being able to undo the bad writes, or just reverting to a snapshot. Without being careful here CD-style development feels like lighting up a cigarette beside an O2 tank. Now for web development this is fine since it's not touching any databases directly. More generally it feels like a trickier thing to attempt everywhere across the industry.

Re: Moving Fast and Securing Things

#50

Post by MaulingMonkey »

icebraining wrote: post #42
ggregoire wrote: post #27

Earlier quoted context omitted.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

I've been convinced that checklists are great (by theory and by practice), yet I still write way fewer than I should. I strongly dislike repetitive mental work, and writing a checklist is essentially resigning myself that such work will be necessary. Until I write it, I can still convince myself I'll be able to automate the process.

I use checklists to turn mental work ("uhh, let me think, what all do I need to do...") into straightforward physical work ("check, next step..."). I also use checklists as a first step towards automation and a stopgap until automation is complete, since I live in a constant state of infinite backlog.

If I run through the checklist a couple times and it seems to:

    [ ] Cover everything
    [ ] Not require complicated decision making or value judgments
    [ ] Has few edge cases in need of handling
    [ ] Doesn't require automation-opaque tooling
    [ ] Not change more frequently than I execute the checklist
Then I know I have a prime candidate for automation, and already have great documentation of exactly what to automate.
Post reply on HN