Live data from Hacker News

Moving Fast and Securing Things

slack.engineering

61–65 of 65 posts

Re: Moving Fast and Securing Things

#61

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Checklists are useful, but they have their limits.

People too often get used to the routine and will end up skipping bits from checklists, or even outright missing stuff. It's strange, given whole idea of there being something to actually check would hopefully mitigate that.

Generally I do my utmost to avoid having checklists, so that they're left for things where they can't be avoided (e.g. where automation makes no sense, or potentially makes things worse)

Re: Moving Fast and Securing Things

#62

Earlier quoted context omitted.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Exactly. Checklists are great for non-critical tasks. Hospital triage uses the same checklist for pretty much every new patient. Triage is boring, but the fact that it is checklisted is what helps drive efficiency in the ER. Similar boring processes in a tech business can be checklisted to increase efficiency, and the checklist itself can be iterated over. Everything from designing a new feature, to troubleshooting a…

How exactly do you classify hospital triage non-critical? It’s pretty darn critical to the patients, I can assure you of that.

Re: Moving Fast and Securing Things

#64
post #61

Earlier quoted context omitted.

I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.

Checklists are useful, but they have their limits. People too often get used to the routine and will end up skipping bits from checklists, or even outright missing stuff. It's strange, given whole idea of there being something to actually check would hopefully mitigate that. Generally I do my utmost to avoid having checklists, so that they're left for things where they can't be avoided (e.g. where automation makes no…

> People too often get used to the routine and will end up skipping bits from checklists, or even outright missing stuff.

This depends on the person and their approach.

If I understand the need for each checklist step as a means of avoiding future hassle/work or outright danger of some sort, and I'm buying into the concept that I actually need to reference the checklist (perhaps even going so far as to print out a copy and actually physically check things off), and being properly conservative (no "I think I did that...", only "I know I did that" or "I double checked and yes I did that"), checklists stay very effective.

In other words, it's not enough to have a checklist, you need checklist discipline, the kind of discipline that can only be accomplished through buy-in to the concept. The same checklist approached with two different mindsets can range from a damn-near error free way to accomplish something, to a piece of scrap paper that would be worth more blank.

This does sometimes make checklist handoff problematic.

And your discipline might depend on the checklist. I'm way more disciplined about a release checklist than I am about a weekend chores checklist - and this is perfectly fine.

Re: Moving Fast and Securing Things

#65
post #52

Earlier quoted context omitted.

The thing you're missing is that you're amortizing the cost. Yeah, it's typically prohibitive to run the manual testing on every CL. However, if you have any manual testing you need to run, then at some point you have to batch the changes & test them out together anyway. Automated tests don't necessarily solve this problem either. 1) Some automated tests can be time-consuming & so require batching of CLs to run too 2…

> However, if you have any manual testing you need to run, then at some point you have to batch the changes & test them out together anyway. Wait, why is that? Manual testing should be reserved for workflows that can't be automatically tested (or at least, aren't yet). I'm not sure I see why doing any amount of manual testing would necessitate manually testing everything. > Some automated tests can be time-consuming…

> Wait, why is that? Manual testing should be reserved for workflows that can't be automatically tested (or at least, aren't yet). I'm not sure I see why doing any amount of manual testing would necessitate manually testing everything.

I never said you need to manually test everything. This is about continuous deployment where typically a push to master is the last step anyone does before the system at some point deploys it live shortly thereafter. In the general case, however, how do you know in an automatic fashion if a given CL may or may not need manual testing? If you have any manual testing then you can't just continuously deploy.

> This is also true of manual testing.

I never opined that there should only be one or the other exclusively so I don't know why you're building this strawman & arguing it throughout. A mix of automatic & manual testing is typically going to be more cost effective for a given quality bar (or vice-versa - for a given cost you're going to have a higher quality) because (good) manual testing involves humans that can spot problems that weren't considered in automation (you then obviously improve automation if you can) or things automation can't give you feedback on (e.g. UX issues like colors not being friendly to color-blind users).

> The need (or lack thereof) for very formal and structured mechanisms for rolling back deploys doesn't really have much to do with the frequency that you deploy.

That just isn't true. If you're thorough with your automatic & manual testing you may establish a much greater degree of confidence that things won't go catastrophically wrong. You deploy a few times a year & you're done. Now of course you should always do continuous delivery so that to the best of your ability you ensure in an automated fashion an extremely high quality bar for tip of tree at all times so that you're able to kick off a release. Whether that translates into also deploying tip of tree frequently is a different question. Just to make clear what the thesis of my post is, I was saying continuous deployment is not something that's generally applicable to every domain (continuous delivery is). If you want an example, consider a FW release for some IoT device. If you deployed FW updates all the time you're putting yourself in a risky scenario where a bug bricks your units (e.g. OTA protocol breaks) & causes a giant monetary cost to your business (RMA, potential lawsuits, etc). By having a formal manual release process where you perform manual validation to catch any bugs/oversights in your automation you're paying some extra cost as insurance against a bad release.

> Maybe we're speaking about different things here, but the trope about web development is that it's basically a thin CRUD wrapper around a database, so I'm not sure this is true.

The frontend code itself doesn't talk do the DB directly (& if it does, you're just asking for huge security problems). The middle/backend code takes front-end requests, validates permissions, sanitizes DB queries, talks to other micro-services etc. Sometimes there are tightly coupled dependencies but I think that's rarer if you structure things correctly. Even FB which can be seen as the prototypical example of moving in this direction no longer pushes everything live. Things get pushed weekly likely to give their QA teams time to manually validate changes for the week, do staged rollouts across their population to catch issues, etc.

In general I think as you scale up continuous deployment degrades to continuous delivery because the risks in the system are higher; more users, more revenue, more employees & more SW complexity means the cost of a problem go up as does the probability of a catastrophic problem occurring. When I worked at a startup continuous deployment was fine. When I worked at big companies I always push for continuous delivery but continuous deployment would just be the wrong choice & irresponsible to our customers.

Post reply on HN