Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When deployments are easy, code becomes simpler

bitbytebit.substack.com

Post reply on HN
Print view View original

91–100 of 124 posts

Re: When deployments are easy, code becomes simpler

#91

Post by ljm »

merb wrote: post #79
steffres wrote: post #76

Earlier quoted context omitted.

I see. Thanks for the clarifications. And these DB migrations, did your team keep a history of them? If so, did you manage them yourselves, or did you use some tools like flyway? I'm asking because I'm starting a project where we will manage the persistence SQL layer without any ORM (always did it so far with Django's migrations), but might consider some third party tools for DB migrations.

btw. it's also bad to drop a column if you have multiple people in a team when they switch between branches. it's always a headache, so the best thing is to delay dropping/deleting. renaming stuff with that gets a little bit tricky, but you can workaround that with database triggers if you really need to rename things.

The problem I've seen a lot, particularly with Rails, is when migrations generate a schema dump after running them, which can get really messy if people blindly commit the diff or if you rely on running several years of migrations in your local environment from scratch (many of which may have been edited or removed if they directly referenced application code). Given the migrations are executed through a DSL and the dump is just a structural snapshot of the DB at the end of the run, they're not quite as reproducible as plain SQL migrations.

You just end up with weird, unpredictable DB states that don't reflect production at all. Especially when you're dealing with old versions of MySQL and the character encoding and collation are all over the place.

Re: When deployments are easy, code becomes simpler

#92

Post by chiefalchemist »

vrosas wrote: post #45
chiefalchemist wrote: post #15

Earlier quoted context omitted.

> just well-intentioned eager people with more knowledge than wisdom. Yup. And that by definition is a junior colleague ;)

I like to describe my colleagues as the delta between how smart they are, and how smart they _think_ they are. “Junior” devs almost always fall in the “wide gap” spectrum but it is by no means restricted to any age group.

Dunning Kruger as a spectrum within the context of engineering?

That's genius. I'm going to borrow (read: steal) that.

Re: When deployments are easy, code becomes simpler

#93

Post by wiredfool »

robertlagrant wrote: post #82
mrweasel wrote: post #80

Earlier quoted context omitted.

The way I've seen it work is hand written SQL for the migrations, numbered and tracked in Git. There shouldn't be any reason that you can't do it with Flyway, but I would be concerned about fighting Flyway a bit. I use Django a fair bit and I honestly don't see a good way to make this approach work for Django, not suggesting that you can't, but you would be fighting Django a fair bit, it's not really how it's designe…

Django has migrations; why would this be harder with Django?

Django migrations have painful edge cases when you deal with data migrations.

Re: When deployments are easy, code becomes simpler

#94

Post by OJFord »

perrygeo wrote: post #87
mcny wrote: post #27

Earlier quoted context omitted.

I always say this - there is no such thing as a rollback. Anyone who pretends you can simply "roll back" a stateful application is out of their mind. If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.

The system can only move forwards. Just like a `git revert`, it's a new commit to an immutable history. This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.

In what sense is 'downgrade' not just that prepared ahead of time? Or do you just not like the name?

Of course usually it would be followed by something that fixes it 'properly' - applying the original 'upgrade' again but with some correction, but typically that'll take longer than reverting to the previous version, and given it's gone wrong you'll probably want to take longer with it, test more, restore confidence.

Re: When deployments are easy, code becomes simpler

#95

Post by rjmunro »

chaxor wrote: post #31
ljm wrote: post #24

Earlier quoted context omitted.

The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.

If a git commit hash was directly tied to a data hash at that state (IPFS), that would be trivial.

Only if there were no changes to the data.

Re: When deployments are easy, code becomes simpler

#96

Post by bullen »

With my SaaS/PaaS you hot-deploy async. to all global nodes with complete (type last code character to ready) turnaround of ~1-2 seconds on live (just enough time to switch to your client/browser):

http://github.com/tinspin/rupy

It also allows for session persistence, so you can test the dev/bug feature without logging in again.

And yes code becomes more manageable if you work like this because it encourages modularity.

Re: When deployments are easy, code becomes simpler

#97

Post by wrsh07 »

drekipus wrote: post #10
smokel wrote: post #8

I have recently seen someone wrap a single function in about 10 classes of Java to turn it into a standalone application, slap on some Docker Compose magic, add some build scripts, and then continue to look proud at the feature being wholly configurable at deployment time. Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the p…

Just raise it up early, often, and simply. "Isn't this just a Boolean flag? If X else y?" I've saved a great many man months by doing just this. Sometimes work just vanishes. And it's not just the juniors that do it, sometimes the senior / tech lead types just miss a crucial point to make the solution trivial. But juniors are often used to wanting to make work, rather than solve a problem.

Just yesterday I was helping with an incident and we had outlined two possible solutions. The first was bad (do nothing, leave things in a somewhat bad state), the second was a ton of work (including restoring from snapshot). I was able to describe these solutions as a spectrum.

1. Do nothing, bad state remains 2. Surgically restore to world before bad state 3. Remove all bad state (even if it wasn't caused by the incident)

I'm omitting some details, but I think this strategy can be effective.

How configurable should X be? We have one customer who wants it to be configured like this. All other customers want it configured like that.

We can make X maximally configurable which would allow future customers to have a custom setup - how likely is that?

If it's not likely, here's an alternative...

Re: When deployments are easy, code becomes simpler

#98

Post by wrsh07 »

> Much of the time I’m using flags so I can commit unfinished code

There are two interpretations of this, and I find one of them horrifying

1. I'm working on a feature, I need to update multiple components, so I make small changes for each component and feature flag those (this is fine! Totally normal)

2. I commit code that doesn't work because I'm unwilling to use or understand git

The second of these scares the crap out of me. Why? Doesn't it feel like a bit of a footgun to have broken code deployed?

However, I like most of the other comments of the essay - deploys should be easy! That makes rollbacks safe and easy. Using bool flags is fine too - imo feature flagging systems are best at de risking expensive deploys.

If you have an expensive deploy (thinking at least 20m to deploy prod) and you have staging which mirrors prod (probably also 15-20m to deploy staging), then any commit you want to deploy to fix or revert a change in prod is going to cost you a lot of time or hassle. Feature flags give you a mechanism to instantly rollback.

But it sounds like you already have that - it's called your normal deploy.

Re: When deployments are easy, code becomes simpler

#99

Post by wrsh07 »

teeray wrote: post #3

Anyone have advice for teams without control over when their software is deployed by the people actually running it? Say you have a product released to a third party and they may or may not take releases and may be arbitrarily delayed in deploying those they choose to take.

If you're talking about an app store you can bundle two binaries in a single binary and then have an external feature flag that selects which binary to use

Dealing with the long tail of customers waiting to upgrade will be a persistent challenge though

Re: When deployments are easy, code becomes simpler

#100

Post by tianzhou »

perrygeo wrote: post #87
mcny wrote: post #27

Earlier quoted context omitted.

I always say this - there is no such thing as a rollback. Anyone who pretends you can simply "roll back" a stateful application is out of their mind. If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.

The system can only move forwards. Just like a `git revert`, it's a new commit to an immutable history. This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.

>> This is why I've never used or understood the value of the "downgrade" feature in some database migration tools

It's technically a roll forward, while to most users it's more nature to say rollback/revert

Post reply on HN