Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistakes engineers make in large established codebases

seangoedecke.com

Post reply on HN
Print view View original

301–310 of 384 posts

Re: Mistakes engineers make in large established codebases

#301

Post by potatolicious »

bluGill wrote: post #264
danjac wrote: post #245

Earlier quoted context omitted.

Often the problem with companies running the Feature Factory production treadmill too long is you have code supporting unused features and business logic, but nobody knows any more which features can be dropped or simplified (particularly after lots of employee churn and lack of documentation). So the problem is not so much technical debt, but product debt. You can refactor, but you're also wasting time optimizing co…

I was involved with a big rewrite. Our manager had on his desk the old system with a sign "[managers name]'s product owner". Nearly every time someone wanted to know how to do something the answer was load that old thing up and figure out what it did. Eventually we did retire the old system - while the new code base is much cleaner I'm convinced it would have been cheaper to just clean that code up in place. It still…

> "I'm convinced it would have been cheaper to just clean that code up in place"

Generally agreed. I'm generally very bearish on large-scale rewrites for this reason + political/managerial reasons.

The trick with any organization that wants to remain employed is demonstrating progress. "Go away for 3 years while we completely overhaul this." is a recipe for getting shut down halfway through and reassigned... or worse.

A rewrite, however necessarily, must always be structured as multiple individual replacements, each one delivering a tangible benefit to the company. The only way to stay alive in a long-term project is to get on a cadence of delivering visible benefit.

Importantly doing this also improves your odds of the rewrite going well - forcing yourself to productionize parts of the rewrite at a a time validates that you're on the right track.

Re: Mistakes engineers make in large established codebases

#302

Post by staunton »

hinkley wrote: post #300
staunton wrote: post #200

Earlier quoted context omitted.

> A bridge that is missing support probably wouldn't have been opened to the public in the first place. That's not always been the case and came to be because people have died... Is anyone going to die if your codebase is an unmaintainable mess?

Companies die because nobody is willing to work on the code anymore. If VCs ever came to expect less than 90% of their investments to essentially go to zero, maybe that would change. But they make enough money off of dumb luck not leading to fatal irreversible decisions often enough to keep them fat and happy.

That doesn't sound nearly as bad or serious as people dying.

Re: Mistakes engineers make in large established codebases

#303

Post by rmk »

ReflectedImage wrote: post #219

"as a general rule, large established codebases produce 90% of the value." This is only until your new upstart competitor comes along, rewrites your codebase from scratch and runs you out of the market with higher development velocity (more features).

> rewrites your codebase from scratch

This almost never happens. It takes a long time and huge amounts of money to come up to parity, and in the meantime, the legacy org is earning money on the thing you're trying to rewrite.

It's more often the case that the technology landscape shifts dramatically helping a niche player (who has successfully saturated the niche) become mainstream or more feasible. Take, for example, Intel. Their CISC designs and higher power consumption is now being challenged by relatively simpler RISC and lower power designs. Or Nvidia with its GPUs. In both cases, it's the major shifts that have hurt Intel. No one can outcompete Intel in making server CPUs of old, if they are starting from scratch.

Take another example, this time, of a successful competitor (of sorts). Oracle vs Postgres. Same deal, except that Postgres is the successor of Ingres (which doesn't exist anymore), and was developed at Berkeley and was open-source (i.e., it relied upon the free contributions of a large number of developers). I doubt that another proprietary database has successfully challenged Oracle. Ask any Oracle DB user, and you will likely get the answer that other databases are a joke compared to what it offers.

Re: Mistakes engineers make in large established codebases

#304

Post by hinkley »

citizenpaul wrote: post #106
digging wrote: post #67

Earlier quoted context omitted.

> If it's actually better, others will start following your lead. Not really my experience in teams that create inconsistent, undocumented codebases... but you might get 1 or 2 converts.

>Not really my experience in teams that create inconsistent, undocumented codebases... but you might get 1 or 2 converts. This has also been my experience. Usually there is a "Top" sticky/unhelpful/reticent person. They are not really a director or exec but they often act like it and seem immune from any repercussions from the actual higher ups. This person tends to attract "followers" that know they will keep their…

There are however some people who think they are sticky but aren’t really. Some but not all of them use Impostor Syndrome to keep their followers in line. You can recruit most easily from people they’ve left twisting in the wind when their suggestions and ideas turned out to not work, but only if you always deal with the poor consequences of your own decisions. People will follow ideas they don’t quite understand if they know they won’t be working alone at 7 pm on a Thursday fixing it.

These sort of people will vote for you publicly. However some lot them will still take the path of least resistance when you aren’t looking.

It was sort of a nasty surprise when I figured out one day that there are people in this industry that will agree with high minded sentiments in public but not lift a finger to get there. I ended up in a group that had two or three of them. And one day due to a requirements process fuckup we had a couple weeks with nothing to do. They just did the Hands Are Tied thing I’d been seeing for over a year (yes we should do X but we have to do Y for reasons) and I saw red. Luckily I was on a conference call instead of sitting in front of them at that moment. But I’m sure they heard the anger lines over the phone.

If the boss doesn’t give you an assignment, you work on tech debt they haven’t previously insisted that you work on. Simple as that. At most places if my boss disappeared, I could keep busy for at least three months without any direction. And keep several other people busy as well. If you don’t know what to work on then I don’t know what’s wrong with you.

Re: Mistakes engineers make in large established codebases

#305

Post by cess11 »

zwnow wrote: post #197
bcrosby95 wrote: post #187

Earlier quoted context omitted.

New features are the right time to refactor. If you can't make the code not complete shit you don't have time to add the feature. Never refactor code to make it prettier or whatever, refactor it when it becomes not-fit-for-purpose for what you need to do. There's obviously exceptions (both ways) but those are exceptions not rules. At least, that's what I teach our devs.

My company didn't even have time to keep the dependencies up to date so now we are stuck with Laravel 5 and Vue 2. Refactoring/Updating can be an incredible workload. Personally I'd say rewriting the whole thing would be more efficient but that's not my choice to make. If you have plenty of time for a task, I fully agree with you.

I was in an organisation that made decent money on a system built on Laravel 3, I think. The framework was written in an only static classes style, which they over ten years had run with while building the business so everything was static classes. Once you have a couple of million lines of that, rewrite is practically impossible because you need to take the team of two OK devs and a junior off firefighting and feature development for years and that will hurt reputation and cashflow badly.

My compromise was to start editing Laravel and implementing optimisations and caching, cutting half a second on every request within a month of starting, and then rewriting crude DIY arithmetic on UNIX epoch into standard library date/time/period functions and similar adjustments. I very openly pushed that we should delete at least two hundred thousand lines over a year which was received pretty poorly by management. When I left in anger due to a googler on the board fucking up the organisation with their annoying vision where this monster was to become "Cloud Native" on GCP credits he had, a plan as bad as a full rewrite, it only took a few months until someone finally convinced them to go through with deletions and cut LoC in half in about six months.

I don't think they do containers or automatic tests yet, probably never will, but as of yet the business survives.

Re: Mistakes engineers make in large established codebases

#306

Post by cess11 »

LouisSayers wrote: post #255
peterldowns wrote: post #34

I agree that consistency is important — but what about when the existing codebase is already inconsistent? Even worse, what if the existing codebase is both inconsistent and the "right way to do things" is undocumented? That's much closer to what I've experienced when joining companies with lots of existing code. In this scenario, I've found that the only productive way forward is to do the best job you can, in your…

> but what about when the existing codebase is already inconsistent? Then you get people together to agree what consistent looks like. I find the easiest way to do this is to borrow someone else's publicly documented coding conventions e.g. Company ABC. Then anyone disagreeing isn't disagreeing with you, they're disagreeing with Company ABC, and they (and you) just have to suck it up. From there on in, you add lintin…

If there's resistance to picking a style guide, autoformatting might be a viable start and will probably do quite a bit for shallow consistency at the price of large PR:s once per file. Once one has worked with a forced style for a while it starts to feel weird to see breaches of it, and I think that might help softening people to adapting a style guide regarding more subtle things like error handling or attitude to standardised protocols like HTTP.

Re: Mistakes engineers make in large established codebases

#307

Post by cratermoon »

simoncion wrote: post #261
harrall wrote: post #77

Earlier quoted context omitted.

Also an issue is that the director attempted a full rewrite as a separate project. You can do successful rewrites but your rewrite has to be usable in production within like a month. If you don’t know how to achieve that, don’t even try. The quiet developer was able to get their own rewrite done because they understood that. Looks like the director of engineering showed some classic inexperience. You can tell when so…

> You can do successful rewrites but your rewrite has to be usable in production within like a month. I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0] The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work. [0] I've worke…

If the schedule is three years, and in the meantime the product being rewritten isn't getting maintenance, the company might as well go ahead and fold and save everyone pain and disappointment. https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Re: Mistakes engineers make in large established codebases

#308

Post by zzbzq »

Wrong, wrong. Opposite of everything he said. All his examples are backwards. The article is basically inversing the Single Responsibility Principle.

First of all, consistency does not matter at all, ever. THat's his main thesis so it's already wrong. Furthermore, all his examples are backwards. If you didn't know the existence of "bot" users, you probably don't want your new auth mechanism to support them. Otherwise, the "nasty surprise" is the inverse of what he said: not that you find you don't support bot users, but you find out that you do.

Build stuff that does exactly what you want it to do, nothing more. This means doing the opposite of what he said. Do not re-use legacy code with overloaded meanings.

Re: Mistakes engineers make in large established codebases

#309

Post by rusk »

tomohawk wrote: post #296
cwalv wrote: post #287

Earlier quoted context omitted.

> I'd love to work in a fantasy company that allows for fixing legacy code You're not supposed to ask. It's like a structural engineer asking if it's okay to spend time doing a geological survey; it's not optional. Or a CFO asking if it's okay to pay down high interest debt. If you're the 'engineer', you decide the extent it's necessary

It's why Scotty was always giving longer estimates that Kirk wanted, but Kirk was also able to require an emergency fix to save the ship. The estimate was building in the time to get it done without breaking too much other stuff. For emergency things, Scotty would be dealing with that after the emergency. If your captain is always requiring everything be done as an emergency with no recovery time, you've got bigger p…

Scotty manages his tech debt

Re: Mistakes engineers make in large established codebases

#310

Post by sfink »

StellarScience wrote: post #168
sfink wrote: post #138

Earlier quoted context omitted.

Hopefully, you have a monorepo or something with similar effects, and a lack of fiefdoms. In that case, if the current way is undocumented and/or inconsistent, you make it better before or while adding in your new approach. If there are 4 ways to do the same thing and you really want to do it a different way, then replace one of those ways with your new one in the process of adding it. For extra credit, get to the po…

Great points, I'd just add: > A lot of inconsistency is the result of unwillingness to fix other people's stuff Agree, so we find it best to practice "no code ownership" or better yet "shared code ownership." So we try to think of it all as "our stuff" rather than "other people's stuff." Maybe you just joined the project, and are working around code that hasn't been touched in 5 years, but we're all responsible for i…

> That requires a high trust environment; I don't know if it could work for Firefox where you may have some very part-time contributors.

Ironically, that's why it works for Firefox. Contributors follow a power law. There are a lot of one-shot contributors. They'll be doing mostly spot fixes or improvements, and their code speaks for itself. Very little trust is needed. We aren't going to be accepting binary test blobs from them. There are relatively few external contributors who make frequent contributions, and they've built up trust over time -- not by reporting to the right manager or being a friend of the CTO, but through their contributions and discussions. Code reviews implicitly factor in the level of trust in the contributor. All in all, the open nature of Firefox causes it to be fundamentally built on trust, to a larger extent than seems possible in most proprietary software companies. (There, people are less likely to be malicious, but for large scale refactoring it's about trusting someone's technical direction. Having a culture where trust derives from contribution not position means it's reasonable to assume that trusted people have earned that trust for reasons relevant to the code you're looking at.)

There are people who, out of the blue, submit large changes with good code. We usually won't accept them. We [the pool of other contributors, paid or not] aren't someone's personal code maintenance team. Code is a liability.

> But having documented standards, plus clang-format and clang-tidy to automate some of the simpler things, also goes a long way.

100% agree. It's totally worth it even if you disagree with the specific formatting decisions made.

Post reply on HN