How release canaries can save your bacon • Hacker News Board

Re: How release canaries can save your bacon

#51

Post by Touche » Tue, Apr 04, 2017, 8:17 PM UTC

gefh wrote: post #50

Touche wrote: post #47

Earlier quoted context omitted.

If one.js changes but two.js doesn't, then two.js should come from the cache. Only one.js should be fetched from network. Sticking all assets in an /assets/v2 folder invalidates everything .

If one.js and two.js are really separate components, they should each get a version. If they're closely coupled, they should be compiled together into one unit, to take advantage of deduplication, inlining, dead code elimination, fewer requests, better compression, etc etc.

Versioning assets separately by sticking them in new subfolders is just re-inventing ETags in a bad way. Just use ETags.

The point of the folder versioning scheme the article proposes is to make rollbacks easier. You can easily rollback a /assets/v2 folder to /assets/v1 by updating your server template, but if you have a dozen separate version folders (with different latest version numbers) for each resource then it's no longer easy to roll those all back.

Re: How release canaries can save your bacon

#52

Post by qqg3 » Wed, Apr 05, 2017, 11:14 AM UTC

d4mi3n wrote: post #30

nurettin wrote: post #28

Earlier quoted context omitted.

You are a CTO sitting atop very expensive hardware and software. Would you start removing deployment and runtime safety guards (such as a consumer-facing staging environment) because you want to "discipline coders and devops"?

A post-mortem should never be about placing blame on individuals, it should be about identifying flaws in a system or a process. There are places where post-mortems can turn into blame games, but in my experience such things are counter-productive to actually solving problems. Luckily, there are plenty of engineering organizations that do not make this mistake! :)

The easiest way to avoid that is to have well structured post-mortem process, and post-mortem everything. Successful and unsuccessful releases.

Re: How release canaries can save your bacon

#53

Post by ojilles » Wed, Apr 05, 2017, 12:59 PM UTC

qqg3 wrote: post #52

d4mi3n wrote: post #30

Earlier quoted context omitted.

A post-mortem should never be about placing blame on individuals, it should be about identifying flaws in a system or a process. There are places where post-mortems can turn into blame games, but in my experience such things are counter-productive to actually solving problems. Luckily, there are plenty of engineering organizations that do not make this mistake! :)

The easiest way to avoid that is to have well structured post-mortem process, and post-mortem everything. Successful and unsuccessful releases.

We need to go from postmortem to postpartum!

Re: How release canaries can save your bacon

#54

Post by mrj » Wed, Apr 05, 2017, 3:35 PM UTC

jpatokal wrote: post #33

mrj wrote: post #19

Earlier quoted context omitted.

Host a high-traffic site on Google's infrastructure. Since you can see the version number of the platform, it's obvious when they're rolling out changes. This has cause many partial outages until the change was (I assume) automatically rolled back. It's a little hard to take this advice from Google after being the victim of so many bad rollouts. Because we use a lot of services, we are far more likely to have problem…

But without canaries, you would have complete outages instead of partial ones.

Sure, but I'm pointing out that this strategy relies on real customers encountering an error. I caution people to not forget that is a failure for us trying to ensure reliable websites.