Live data from Hacker News

Mistakes engineers make in large established codebases

seangoedecke.com

51–60 of 384 posts

Re: Mistakes engineers make in large established codebases

#51
post #3

The "The cardinal mistake is inconsistency" is 100% true. We used to call the guiding philosophy of working in these codebases "When in Rome".

"When is Rome" is good, might use that.

My old boss used to say: "Be a chameleon. I don't want to know that I didn't write this."

Re: Mistakes engineers make in large established codebases

#52
post #22

One thing I did was implement a code formatter, and enforce it in CI. "dotnet format" can do wonders, and solved most serious inconsistency issues.

Honestly Go's approach to code formatting and it being taken over by other parties has saved so much trivial debates. I remember spending stupid amounts of review comments on stupid shit like formatting, trailing commas, semicolons, spacing, etc that should have been spent on more important things. How come automatic, standardized, non-ide bound automatic formatting has only been a thing in the past decade or so? I do recall Checkstyle for Java ages ago but I forgot if it did any formatting of its own.

Re: Mistakes engineers make in large established codebases

#53
post #30
post #3

The "The cardinal mistake is inconsistency" is 100% true. We used to call the guiding philosophy of working in these codebases "When in Rome".

How do you tackle the case where the codebase is consistent in a bad way, like pervasive use of antipatterns that make code difficult to change or to reason about? If you want to improve that, you have to start somewhere. Of course, Chesterton’s Fence applies.

Or when people keep the old pattern because there's a "higher priority".

I've worked on a project with absolutely terrible duplication of deserialisers of models, each one slightly different even those most properties were named the same and should've been handled the same. But we can't touch anything because important things are happening in business and we can't risk anything. The ignored part was that this led to bugs and extreme confusion from new people. They were even too worried to accept a repo-wide whitespace normalisation.

Re: Mistakes engineers make in large established codebases

#54
post #31
post #3

The "The cardinal mistake is inconsistency" is 100% true. We used to call the guiding philosophy of working in these codebases "When in Rome".

I don't like this philosophy as it often leads to stagnation in patterns and ways of working that seep into newer systems. "That's not how we do things here" becomes a common criticism, resulting in systems and services that share the same flaws and trade-offs, making progress difficult. Engineers often adhere too rigidly to these principles rather than taking a pragmatic approach that balances existing practices wit…

What's bad is code exhibiting multiple fragmentary inconsistencies, and no plan or effort exists to bring older code up to match the new patterns. An example I was closely involved with: A java programmer who wrote a plethora of new code in a pure functional paradigm, scattering Vavr library uses all over it. The existing code was built on Dropwizard and any experienced Java programmer could rapidly get comfortable with it. The difference between the existing code and the new was jarring (sorry for the pun) to say the least, and I wonder if, later, the company ever managed to recruit anyone who understood both well enough to maintain the system.

ETA: upon reflection I'd consider that programmer a canonical example of the kinds of mistakes the author covers in the article.

Re: Mistakes engineers make in large established codebases

#56
I'm now working on a codebase which is quite large (13 micro-services required to run the main product); all containerized to run on Kubernetes. The learning curve was quite steep but luckily, I was familiar with most of the tech so that made it easier (I guess that's why they hired me). The project has been around for over 10 years so it has a lot of legacy code and different repos have different code styles, engine versions and compatibility requirements.

The biggest challenge is that it used to be maintained by a large team and now there are just 2 developers. Also, the dev environment isn't fully automated so it takes like 20 minutes just to launch all the services locally for development. The pace of work means that automating this hasn't been a priority.

It's a weird experience working on such project because I know for a fact that it would be possible to create the entire project from scratch using only 1 to 3 services max and we would get much better performance, reliability, maintainability etc... But the company wouldn't be willing to foot the cost of a refactor so we have to move at steady snail's pace. The slow pace is because of the point mentioned in the article; the systems are all intertwined and you need to understand how they integrate with one another in order to make any change.

It's very common that something works locally but doesn't work when deployed to staging because things are complicated on the infrastructure side with firewall rules, integration with third-party services, build process, etc... Also, because there are so many repos with different coding styles and build requirements, it's hard to keep track of everything because some bug fixes or features I implement touch on like 4 different repos at the same time and because deployment isn't fully automated, it creates a lot of room for error... Common issues include forgetting to push one's changes or forgetting to make a PR on one of the repos. Or sometimes the PR for one of the repos was merged but not deployed... Or there was a config or build issue with one of the repos that was missed because it contained some code which did not meet the compatibility requirements of that repo...

Re: Mistakes engineers make in large established codebases

#57
"Coding defensively" is perhaps the understatement of the year. Good software architecture is, in my opinion, the single most powerful tool you have to keep things from becoming an unmanageable mess.

If I could give one "defensive coding" tip, it would be for seniors doing the design to put in road blocks and make examples that prevent components from falling for common traps (interdependency, highly variant state, complex conditions, backwards-incompatibility, tight coupling, large scope, inter-dependent models, etc) so that humans don't have to remember to avoid those things. Make a list of things your team should never do and make them have a conversation with a senior if they want to do it anyway. Road blocks are good when they're blocking the way to the bad.

Starting with good design leads to continuing to follow good design. Not starting with good design leads to years of pain. Spend a lot more time on the design than you think you should.

Re: Mistakes engineers make in large established codebases

#58
post #31

Earlier quoted context omitted.

I don't like this philosophy as it often leads to stagnation in patterns and ways of working that seep into newer systems. "That's not how we do things here" becomes a common criticism, resulting in systems and services that share the same flaws and trade-offs, making progress difficult. Engineers often adhere too rigidly to these principles rather than taking a pragmatic approach that balances existing practices wit…

And that's a fair criticism, however, if you change a pattern without changing it everywhere, you now have two patterns to maintain (the article mentions this). And if multiple people come up with multiple patterns, that maintenance debt multiplies. Progress and improvement is fine, great even, but consistency is more important. If you change a pattern, change it everywhere.

Change it at once everywhere on an existing large codebase? That's going to be one huge PR no one will want to review properly, let alone approve.

Document the old pattern, document the new pattern, discuss, and come up with a piece by piece plan that is easy to ship and easy to revert if you do screw things up.

Unless the old pattern is insecure or burns your servers, that is.

Re: Mistakes engineers make in large established codebases

#59
post #17

> The other reason is that you cannot split up a large established codebase without first understanding it. I have seen large codebases successfully split up, but I have never seen that done by a team that wasn’t already fluent at shipping features inside the large codebase. You simply cannot redesign any non-trivial project (i.e. a project that makes real money) from first-principles. This resonates. At one former c…

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolit…

> Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features.

Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.

Re: Mistakes engineers make in large established codebases

#60
post #10

I really liked this: "as a general rule, large established codebases produce 90% of the value." People see the ugliness -- because solving real problems, especially if business practices are involved, is often very messy -- but that's where the value is.

I also find amusing that “legacy” more often than not gets used in negative conotation. I hear “legacy” and I think “bunch of people wrote some AWESOME shit that lasted so long that now other people get to view it as ‘legacy’”

Measuring value by what stays the longer is tempting, but sometimes it's just that the mess is such that no one can touch it :)
Post reply on HN