Live data from Hacker News

Moving Fast and Securing Things

slack.engineering

51–60 of 65 posts

Re: Moving Fast and Securing Things

#51

Earlier quoted context omitted.

They were/are helpful in making medicine safer too. https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.ncbi.nlm.nih.gov/pubmed/24116973

The New Yorker article was later expanded into an entire book of the same name.

The Checklist Manifesto

Re: Moving Fast and Securing Things

#52
post #26

Earlier quoted context omitted.

Sure, and I guess my point is: If you haven't invested in those things, waterfall-esque deploy processes are just as bad and perhaps even worse because there's more chance for confounding changes to cause a nasty error. The only reason waterfall-esque deploy processes work without those things is because companies often waste tons of people-hours on testing things out in the staging environment (which requires time,…

The thing you're missing is that you're amortizing the cost. Yeah, it's typically prohibitive to run the manual testing on every CL. However, if you have any manual testing you need to run, then at some point you have to batch the changes & test them out together anyway. Automated tests don't necessarily solve this problem either. 1) Some automated tests can be time-consuming & so require batching of CLs to run too 2…

> However, if you have any manual testing you need to run, then at some point you have to batch the changes & test them out together anyway.

Wait, why is that? Manual testing should be reserved for workflows that can't be automatically tested (or at least, aren't yet).

I'm not sure I see why doing any amount of manual testing would necessitate manually testing everything.

> Some automated tests can be time-consuming & so require batching of CLs to run too

I'm not sure I see why this is a problem, and CD certainly doesn't require that only one changeset go live at a time.

> it's impossible to predict if you are going to catch all issues via automated testing

This is also true of manual testing.

> there's always things it's easier to test for manually.

I'd go further and say it's almost always easier to test manually, but the cost of an automated test is definitely amortized and you come out ahead at some point. That point is usually quicker than you think.

> I would think you need a structured mechanism...

This paragraph is entirely true of traditional deploys with long cadences as well. The need (or lack thereof) for very formal and structured mechanisms for rolling back deploys doesn't really have much to do with the frequency that you deploy.

> Now for web development this is fine since it's not touching any databases directly.

Maybe we're speaking about different things here, but the trope about web development is that it's basically a thin CRUD wrapper around a database, so I'm not sure this is true.

Re: Moving Fast and Securing Things

#53

Earlier quoted context omitted.

I've been convinced that checklists are great (by theory and by practice), yet I still write way fewer than I should. I strongly dislike repetitive mental work, and writing a checklist is essentially resigning myself that such work will be necessary. Until I write it, I can still convince myself I'll be able to automate the process.

I use checklists to turn mental work ("uhh, let me think, what all do I need to do...") into straightforward physical work ("check, next step..."). I also use checklists as a first step towards automation and a stopgap until automation is complete, since I live in a constant state of infinite backlog. If I run through the checklist a couple times and it seems to: [ ] Cover everything [ ] Not require complicated decis…

Awesome, a meta-checklist.

Re: Moving Fast and Securing Things

#55

Earlier quoted context omitted.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

I've been convinced that checklists are great (by theory and by practice), yet I still write way fewer than I should. I strongly dislike repetitive mental work, and writing a checklist is essentially resigning myself that such work will be necessary. Until I write it, I can still convince myself I'll be able to automate the process.

Isn't a checklist something you'd end up writing anyway in the process of determining how to automate process?

At least that's what I keep telling myself. I also write much fewer checklists than I should.

Re: Moving Fast and Securing Things

#56

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Most don't have decent tooling set up for such a thing. Org-mode has a great feature where you can quickly capture todo items along with the place you were when you made them etc. (file/project, for example). I could be writing a function and get a thought about some upstream calling code, press a key, make a note, forget about it and carry on coding, knowing that it made it on to my todo list for investigation later.

If I had to switch to some other note taking platform it would probably break my flow enough that I wouldn't do it.

Re: Moving Fast and Securing Things

#57
post #14

One of the things I think about when analyzing organizational behavior is where something falls on the supportive vs controlling spectrum. It's really impressive how much they're on the supportive end here. When organizations scale up, and especially when they're dealing with risks, it's easy for them to shift toward the controlling end of things. This is especially true when internally people can score points by ass…

Yes, I did some translation work for a global engineering company whose approach to achieving "zero fatal accidents" was exactly what you suggest. Instead of placing the blame on people for not following the rules, they identify the ultimate cause and fix the problems with their systems that made the accident possible in the first place.

Re: Moving Fast and Securing Things

#59
post #34
post #14

One of the things I think about when analyzing organizational behavior is where something falls on the supportive vs controlling spectrum. It's really impressive how much they're on the supportive end here. When organizations scale up, and especially when they're dealing with risks, it's easy for them to shift toward the controlling end of things. This is especially true when internally people can score points by ass…

Having recently escaped from a "control and blame" environment, this is also horrible for releases as left unchecked, more energy is expended trying to double-down on architecting for perfection in fault tolerance. Risk aversion goes through the roof cripples decision making, and before you know it your entire team of developers have become full time maintenance coders, you stop innovating and spend cycles creating i…

For those for whom this excellent description has resonance, I strongly recommend the book, "Why Does He Do That? Inside the Minds of Angry and Controlling Men". [1] It's nominally written about domestic abuse, but its descriptions of abuser psychology and its taxonomy of abuser behaviors have been really helpful to me in a work context.

[1] https://www.amazon.com/Why-Does-He-That-Controlling-ebook/dp...

Re: Moving Fast and Securing Things

#60
So glad they finally published this, saw the OWASP AppSec talk, was eagerly awaiting it.

However - I would want to caution: I think this model works because Slack has a self-described "culture of developer trust". I tend to think, they hire bright engineers and ensure they are equipped to do the right thing. I believe the vast majority of organizations are NOT ready for this. I direly want them to be, but simple fact is there are too many mediocre developers, and they can't be trusted without guardrails (and some straight up need babysitters).

Post reply on HN