Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When deployments are easy, code becomes simpler

bitbytebit.substack.com

Post reply on HN
Print view View original

71–80 of 124 posts

Re: When deployments are easy, code becomes simpler

#71

Post by bratbag »

hsn915 wrote: post #51
daigoba66 wrote: post #32

As an aside, those “feature flags as a service” tools have some neat features, but are just way too expensive for what most apps probably need: simple binary flags that can change at runtime. Example: just use a database table. Query and cache in memory for 60s. If you want, build a simple internal web page or tool to toggle flags. This works, and scales (from experience). Do many apps really need A/B testing or segm…

Is "feature flag as a service" the new trend in backend development? Why would anyone replace a literal byte in memory with a full program? Why?

Because feature flags often intersect with segmentation and AB testing.

So it's not just a byte in memory, but often also correlating the status of said byte with a users identity and then tracking and summarising user behaviour based on that relationship.

It's become fairly standardised and requires engineer time to setup and maintain the services behind all that, so it's valid to go third party for less than the cost of said engineer time, if all you want is standard.

Edit:it's also hard to always predict when a standard flags going to become part of a test, so just integrating for every flag and making that a standard process for your teams becomes the simplest approach.

Re: When deployments are easy, code becomes simpler

#72

Post by clintonb »

brigadier132 wrote: post #58
ketchupdebugger wrote: post #46

>Much of the time I’m using flags so I can commit unfinished code because I don’t want to create a separate branch and have one source of truth This seems like an incident in the making. If each dev on a team commits unfinished code into prod behind flags, the whole project is going to be littered with flags and unfinished code. Some intern is going to delete a flag or an if check and then everything is going to brea…

> This seems like an incident in the making This is how every major tech company works today.

Not quite. I never used flags to hide unfinished code. I left that on a branch. Feature flags are used to rollout production-ready features.

This just seems like taking trunk-based development to an unnecessary extreme.

Re: When deployments are easy, code becomes simpler

#73

Post by steffres »

mrweasel wrote: post #49
ljm wrote: post #24

Earlier quoted context omitted.

The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.

One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a…

That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?

Re: When deployments are easy, code becomes simpler

#74

Post by wofo »

smokel wrote: post #68
matsemann wrote: post #64

Earlier quoted context omitted.

Are there no seniors on the team? Why aren't anyone helping and guiding your team to make a better product? Seniority isn't about technical skill, it's about being a multiplier and paving the road for others. If you're not, and you're letting the team in your eyes waste time, the whole team acts just as junior.

Touché! Both luckily and unfortunately, this incident was not on my team. Even then, I find it increasingly hard to argue people out of the nonsense they pick up on blogs and conferences. Especially for newer techniques that are not yet proven to be inefficient, such as using micro services in the wrong context, one has to resort to arguments by authority. Some junior and senior devs are not very susceptible to that.…

> the nonsense they pick up on blogs and conferences

Thanks for this bit. As I gain more experience, I grow less and less patient with many blogs and conference talks. Often authors are just very excited about a topic, and my general impression is that they tend to propose ideas that are not seriously battle-tested in production.

Re: When deployments are easy, code becomes simpler

#75

Post by jaimebuelta »

steffres wrote: post #73
mrweasel wrote: post #49

Earlier quoted context omitted.

One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a…

That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?

You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column.

The typical example is the renaming of a column, which needs to be done in several steps:

1. Create the new column, copying the data from the old column (DB migration) both columns exist, but only the old one is used

2. Deploy new code that works with both columns, reading from old and writing new and old

3. Deploy data migration (DB migration) that ensures old and new columns has the same values (to ensure data consistency). At this point, there are no “old column only” writes by the code deployed in previous step

4. Deploy new code using only new column. Old column is deprecated

5. Delete old column

At any given point, code versions N (current) and N-1 (previous) are compatible with the DB. Any change on the DB is done in advance in a backwards compatible way.

Re: When deployments are easy, code becomes simpler

#76

Post by steffres »

jaimebuelta wrote: post #75
steffres wrote: post #73

Earlier quoted context omitted.

That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?

You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) bot…

I see. Thanks for the clarifications.

And these DB migrations, did your team keep a history of them? If so, did you manage them yourselves, or did you use some tools like flyway?

I'm asking because I'm starting a project where we will manage the persistence SQL layer without any ORM (always did it so far with Django's migrations), but might consider some third party tools for DB migrations.

Re: When deployments are easy, code becomes simpler

#77

Post by onetimeuse92304 »

mrweasel wrote: post #49
ljm wrote: post #24

Earlier quoted context omitted.

The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.

One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a…

Not saying it is a bad idea, but the way it works is it ensures certain things happen that you would normally want to happen, namely:

* test of a rollback procedure,

* developers thinking about backwards compatibility and rollback procedure.

The main issue I see with this approach is that the test of the rollback is only partial. Just because the schema is usable by the previous version of the application does not mean the new data that is going to be put by the application will perform the same way.

Another issue I am seeing is that it is not a separate testing event but essentially happening live when the application is turned on on production. Not nice.

On the pros, this is very useful when you want to have more than one version of the application code to coexist at the same time. But I would not rely on learning about incompatibility when I start deploying the application, I would want to know that well before then.

Re: When deployments are easy, code becomes simpler

#78

Post by Palmik »

At Google, there are no branches, so typically development indeed happens under some experiment flag. Binary, configs and experiment flags are on separate deployment schedules, and often you need to coordinate your release across different systems (with their own binary, config and experiment flag deployment schedules). This requires you to think about backward & forward compatibility of all your changes.

Re: When deployments are easy, code becomes simpler

#79

Post by merb »

steffres wrote: post #76
jaimebuelta wrote: post #75

Earlier quoted context omitted.

You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) bot…

I see. Thanks for the clarifications. And these DB migrations, did your team keep a history of them? If so, did you manage them yourselves, or did you use some tools like flyway? I'm asking because I'm starting a project where we will manage the persistence SQL layer without any ORM (always did it so far with Django's migrations), but might consider some third party tools for DB migrations.

btw. it's also bad to drop a column if you have multiple people in a team when they switch between branches. it's always a headache, so the best thing is to delay dropping/deleting.

renaming stuff with that gets a little bit tricky, but you can workaround that with database triggers if you really need to rename things.

Re: When deployments are easy, code becomes simpler

#80

Post by mrweasel »

steffres wrote: post #76
jaimebuelta wrote: post #75

Earlier quoted context omitted.

You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) bot…

I see. Thanks for the clarifications. And these DB migrations, did your team keep a history of them? If so, did you manage them yourselves, or did you use some tools like flyway? I'm asking because I'm starting a project where we will manage the persistence SQL layer without any ORM (always did it so far with Django's migrations), but might consider some third party tools for DB migrations.

The way I've seen it work is hand written SQL for the migrations, numbered and tracked in Git.

There shouldn't be any reason that you can't do it with Flyway, but I would be concerned about fighting Flyway a bit. I use Django a fair bit and I honestly don't see a good way to make this approach work for Django, not suggesting that you can't, but you would be fighting Django a fair bit, it's not really how it's designed to work.

If you don't have en ORM, then this is actually much much easier to do right. I'd design the initial schema, either by hand or using some pgAdmin, TOAD or whatever you database has. For there on everything is just hand written migrations.

Post reply on HN