Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When deployments are easy, code becomes simpler

bitbytebit.substack.com

Post reply on HN
Print view View original

61–70 of 124 posts

Re: When deployments are easy, code becomes simpler

#61

Post by JimDabell »

ajkjk wrote: post #28
digging wrote: post #9

Earlier quoted context omitted.

> My cynicism is probably not the best approach to change the world for the better, so any hint on how to teach younger colleagues to stop snacking micro service candy is much appreciated. I could be way off base here, but I'd bet a lot of junior devs are just trying to stand out. They want respect, raises, promotions, and new job offers. Invisible good solutions don't bring those.

Strongly disagree this is a junior dev thing. Most of the terribly complex abstractions are, in my experience creates by people who have enough seniority that others can't easily question them. There are good devs and bad devs, and there are junior and senior devs, but it's the first axis that determines code quality.

I’ve seen entire orgs paralysed by a single architecture astronaut who has been there since the beginning who nobody has found a way to route around. Simplicity as a cultural value is underrated, and fetishising complexity can kill a business.

Re: When deployments are easy, code becomes simpler

#62

Post by JimDabell »

perlgeek wrote: post #7

Feature flags give you orthogonality. When you have an `if False:` or equivalent in code, you have to deploy to enable a new feature, so if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature. Another feature of feature flags is that a non-coder can toggle them. If you need neither of these, su…

> if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature.

This is a symptom of long release cycles. The more work you “save up” to deploy all at once, the more difficult deployments become. Everything becomes much easier when you minimise unreleased work.

Re: When deployments are easy, code becomes simpler

#63

Post by sverhagen »

hsn915 wrote: post #51
daigoba66 wrote: post #32

As an aside, those “feature flags as a service” tools have some neat features, but are just way too expensive for what most apps probably need: simple binary flags that can change at runtime. Example: just use a database table. Query and cache in memory for 60s. If you want, build a simple internal web page or tool to toggle flags. This works, and scales (from experience). Do many apps really need A/B testing or segm…

Is "feature flag as a service" the new trend in backend development? Why would anyone replace a literal byte in memory with a full program? Why?

I am not a fan nor a user of backend feature flags, so don't ding me for relaying this: they tout features such as central management of these settings, bringing them in the hands of the product team rather than being tugged away in code, allowing for canary deployments where you fade in a feature based on performance metrics, a/b testing, and so on. In their own interest, these folks take an as broad as possible view on what is a feature flag, often including things that you'd otherwise call system configuration or entitlements or permission toggles. It's not a new trend, LaunchDarkly is probably the best known commercial party here, and they've been around for about ten years, and I don't think they were first.

Re: When deployments are easy, code becomes simpler

#64

Post by matsemann »

smokel wrote: post #8

I have recently seen someone wrap a single function in about 10 classes of Java to turn it into a standalone application, slap on some Docker Compose magic, add some build scripts, and then continue to look proud at the feature being wholly configurable at deployment time. Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the p…

Are there no seniors on the team? Why aren't anyone helping and guiding your team to make a better product? Seniority isn't about technical skill, it's about being a multiplier and paving the road for others. If you're not, and you're letting the team in your eyes waste time, the whole team acts just as junior.

Re: When deployments are easy, code becomes simpler

#65

Post by sverhagen »

hsn915 wrote: post #50
ketchupdebugger wrote: post #46

>Much of the time I’m using flags so I can commit unfinished code because I don’t want to create a separate branch and have one source of truth This seems like an incident in the making. If each dev on a team commits unfinished code into prod behind flags, the whole project is going to be littered with flags and unfinished code. Some intern is going to delete a flag or an if check and then everything is going to brea…

There's nothing special about flags that makes them more likely for an intern to delete by mistake. It's just code. If the team is so bad that an intern can mess things up, they will, and the mess will have nothing to do with feature flags.

Aren't there common patterns of good "just code" and bad "just code"? I've been told for a long time that global variables are a bad pattern. Maybe feature flags are a bad pattern too.

One concern about feature flags is testing, and the added permutations of testing needed to include all the feature flags in testing. You tested with flag A on and off, you tested with flag B on and off, but did you ever test with them both on and both off? Without feature flags, a big change that could have been represented by a feature flag would hopefully have to make its way past some quality gates. With feature flags, the exact permutation that you're going to cause later today by flipping on some feature flags may well not have been tested. Not that forgetting to test is something you can't protect yourself against with tools and processes, but testing all the permutations may be expensive.

You may not have to test all the permutations, if you can predict which permutations are relevant for your flipping feature flags later today. But a lot of organizations have poor discipline in cleaning up old feature flags, so it may not be so predictable. Maybe that's not a feature flag problem but an organizational problem, but the feature flags are gonna get blamed at some point, nonetheless.

Re: When deployments are easy, code becomes simpler

#66

Post by matsemann »

totallywrong wrote: post #38

TIL there's an industry around feature flags. I can see how it can be useful, but imo if you're doing the modern approach with k8s, gitops, etc. deployments are too easy to bother with this.

Often it's not about only turning something on/off (a simple bool). But doing a slow ramp up, only enable for certain cohorts to test etc. Things still doable in code, of course, but it's nice with a dashboard to control it.

You quickly end up with a NIH thing. You make a boolean toggle in the db. Then make an internal dashboard. Then some more advanced toggles. And suddenly you've spent a few weeks of coding time to essentially save yourself from spinning up an Unleash container.

Re: When deployments are easy, code becomes simpler

#67

Post by dgb23 »

smokel wrote: post #8

I have recently seen someone wrap a single function in about 10 classes of Java to turn it into a standalone application, slap on some Docker Compose magic, add some build scripts, and then continue to look proud at the feature being wholly configurable at deployment time. Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the p…

If you’re one of these people who are oozing with sarcasm then just be yourself. It’s a fun way to convey when something actually is ridiculous. Nothing wrong with that.

Just also make sure from time to time that you’re on the same side. Don’t hesitate to laugh at your own mistakes openly. Sprinkle in some sympathetic, honest remarks. Target the thing, the mistakes, never the person.

Bottom line: laugh together at the expense of human fallibility and not a particular person.

Re: When deployments are easy, code becomes simpler

#68

Post by smokel »

matsemann wrote: post #64
smokel wrote: post #8

I have recently seen someone wrap a single function in about 10 classes of Java to turn it into a standalone application, slap on some Docker Compose magic, add some build scripts, and then continue to look proud at the feature being wholly configurable at deployment time. Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the p…

Are there no seniors on the team? Why aren't anyone helping and guiding your team to make a better product? Seniority isn't about technical skill, it's about being a multiplier and paving the road for others. If you're not, and you're letting the team in your eyes waste time, the whole team acts just as junior.

Touché! Both luckily and unfortunately, this incident was not on my team.

Even then, I find it increasingly hard to argue people out of the nonsense they pick up on blogs and conferences.

Especially for newer techniques that are not yet proven to be inefficient, such as using micro services in the wrong context, one has to resort to arguments by authority. Some junior and senior devs are not very susceptible to that.

For areas such as web development for user interfaces, or taking in enormous dependency trees with package managers, the problem is even worse. Here, an entire industry had standardised on suboptimal methods. One can argue that this is wrong, but there is no viable alternative.

In the latter case, guiding only one junior developer does not improve things. One has to educate the entire industry.

Re: When deployments are easy, code becomes simpler

#69

Post by once_inc »

> I did an inventory of my recent feature flags and realized that about 80% of them aren’t there to roll things out to specific populations, or do any sort of A/B testing, but to hide unfinished code.

Which means that for 20% you actually do need the feature flags. So you can't do away with the product, and as the author and many of the respondents here have mentioned, there is also a number of benefits to using feature flags.

In the trivial case of a feature for a single user still in development; yeah, a boolean is enough. But that is not the intended use case for feature flags, nor is it what the libraries promote you to do.

Re: When deployments are easy, code becomes simpler

#70

Post by vinay_ys »

Pushing config to production is also deployment. Pushing config update to a dynamic config serving service from which all other services pull the config is also deployment.

You need safety guardrails for any deployment of code or config – a clearly defined unit of deployment that is version controlled, changes going through a repeatable automated flow in which lints/tests can be added, changes can be rolled out atomically all at once, or rolled out with careful control gradually, can be rolled back to a known safe version instantly when needed etc. All of this applies to config and code (binary).

Within code, having feature flags, dynamic config variables, experiments that can be ramped up slowly and ramped down automatically upon regression detection etc – these are mechanisms to make your life easier when you have unknown-unknowns manifest to bite you. And they always do in any non-trivial real-world large-scale distributed architecture application that needs to evolve continuously.

And these unknown-unknowns are not just code bugs, they are mix shifts in usage patterns, data etc causing unanticipated behaviors too. They are old stale stored state interacting with new code that you didn't test for. They are changes in code outside of your control – like remote services or OS/language libraries – that you assume to be mostly stable and don't exhaustively test every single time – that you don't even know when they changed - that change behavior without changing interface etc. These are real-world software systems problems.

If your software is built like an appliance – frozen in time and doesn't change ever after, is stateless, doesn't interact with outside world etc – then you don't need any of those guardrails and you can keep your code simple.

Post reply on HN