Mistakes engineers make in large established codebases • Hacker News Board

Re: Mistakes engineers make in large established codebases

#101

Post by Supermancho » Wed, Jan 08, 2025, 1:07 AM UTC

dasil003 wrote: post #98

peterldowns wrote: post #44

Earlier quoted context omitted.

And it's fine, right? Honestly I think people need to realize that part of being a good engineer is being able to deal with inconsistency. Maybe submodule A and submodule B do network requests slightly differently but if both ways are reasonable, working, and making the company money, it's probably not worth delaying product improvements in order to make things "more consistent." On the other hand if no one in your c…

Yeah 100%. Honestly style / technique / language consistency are implementation details, it helps with engineer fungibility and ramp up, but it also works against engineers applying local judgement. This is something to briefly consider when starting new services/features, but definitely not something to optimize for in an existing system. On the other hand, data and logic consistency can be really important, but you…

A common experience (mostly in the Pacific North West) I have had is to implement a feature in a straightforward manner that works with minimal code, for some backlog issue. Then I'm told the PR will be looked at.

A couple days later I am told this is not the way to do X. You must do it Y? Why Y? Because of historical battles won and lost why, not because of a specific characteristic. My PR doesn't work with Y and it would be more complicated...like who knows what multiplier of code to make it work. Well that makes it a harder task than your estimate, which is why nobody ever took it up before and was really excited about your low estimate.

How does Y work? Well it works specifically to prevent features like X. How am I supposed to know how to modify Y in a way that satisfies the invisible soft requirements? Someone more senior takes over my ticket, while I'm assigned unit tests. They end up writing a few hundred lines of code for Y2.0 then implement X with a copy paste of a few lines.

I must not be "a good fit". Welcome to the next 6-12 months of not caring about this job at all, while I find another job without my resume starting to look like patchwork.

Challenging people's egos by providing a simpler implementation for something someone says is very hard, has been effective at getting old stagnant issues completed. Unnaturally effective. Of course, those new "right way" features are just as ugly as any existing feature, ensuring the perpetuation of the code complexity. Continually writing themselves into corners they don't want to mess with.

Re: Mistakes engineers make in large established codebases

#102

Post by peterldowns » Wed, Jan 08, 2025, 1:14 AM UTC

agentultra wrote: post #88

peterldowns wrote: post #34

I agree that consistency is important — but what about when the existing codebase is already inconsistent? Even worse, what if the existing codebase is both inconsistent and the "right way to do things" is undocumented? That's much closer to what I've experienced when joining companies with lots of existing code. In this scenario, I've found that the only productive way forward is to do the best job you can, in your…

> In this scenario, I've found that the only productive way forward is to do the best job you can, in your own isolated code, and share loudly and frequently why you're doing things your new different way. Now you have N+1 ways. It can work if you manage to get a majority of a team to support your efforts, create good interfaces into the legacy code paths, and most importantly: write meaningful and useful integration…

I’ll check out that book, thanks for the reference.

Re: Mistakes engineers make in large established codebases

#103

Post by lmm » Wed, Jan 08, 2025, 1:14 AM UTC

I've worked in codebases like this and disagree. Consistency isn't the most important, making your little corner of the codebase nicer than the rest of it is fine, actually, and dependencies are great - especially as they're the easiest way to delete code (the article is right about the importance of that). What's sometimes called the "lava layer anti-pattern" is actually a perfectly good way of working, that tends to result in better systems than trying to maintain consistency. As Wall says, the three cardinal virtues of a programmer are laziness, impatience, and hubris; if you don't believe you can make this system better then why would you even be working on it?

Also if the system was actually capable of maintaining consistency then it would never have got that large in the first place. No-one's actual business problem takes 5M lines of code to describe, those 5M lines are mostly copy-paste "patterns" and repeated attempts to reimplement the same thing.

Re: Mistakes engineers make in large established codebases

#104

Post by galimaufry » Wed, Jan 08, 2025, 1:15 AM UTC

bdangubic wrote: post #75

forty wrote: post #60

Earlier quoted context omitted.

Measuring value by what stays the longer is tempting, but sometimes it's just that the mess is such that no one can touch it :)

100% but mess or not - it works - otherwise you’d have no option but to touch it

Sometimes it doesn't work and no one can tell.

Re: Mistakes engineers make in large established codebases

#105

Post by francisofascii » Wed, Jan 08, 2025, 1:18 AM UTC

PaulDavisThe1st wrote: post #78

> You can’t practice it beforehand (no, open source does not give you the same experience). This is ridiculous. Even if you want to ignore the kernel, there are plenty of "large established codebases" in the open source world that are at least 20 years old. Firefox, various *office projects, hell, even my own cross-platform DAW Ardour is now 25 years old and is represented by 1.3M lines of code at this point in time.…

Agreed. If you didn’t work on a codebase of 1000 engineers, how else would you practice.

Re: Mistakes engineers make in large established codebases

#106

Post by citizenpaul » Wed, Jan 08, 2025, 1:23 AM UTC

digging wrote: post #67

peterldowns wrote: post #34

I agree that consistency is important — but what about when the existing codebase is already inconsistent? Even worse, what if the existing codebase is both inconsistent and the "right way to do things" is undocumented? That's much closer to what I've experienced when joining companies with lots of existing code. In this scenario, I've found that the only productive way forward is to do the best job you can, in your…

> If it's actually better, others will start following your lead. Not really my experience in teams that create inconsistent, undocumented codebases... but you might get 1 or 2 converts.

>Not really my experience in teams that create inconsistent, undocumented codebases... but you might get 1 or 2 converts.

This has also been my experience. Usually there is a "Top" sticky/unhelpful/reticent person. They are not really a director or exec but they often act like it and seem immune from any repercussions from the actual higher ups. This person tends to attract "followers" that know they will keep their jobs if they follow the sticky person for job security. There usually are a few up and coming people that want better that will kinda go along with you for their own skill building benefit but its all very shaky and you can't count on them supporting you if resistance happens.

I've literally had the "I was here before you and will be after" speech from one of the "sticky's" before.

All these HN how to do better write ups seem to universally ignore the issues of power and politics dynamics and give "in a vacuum" advice. Recognizing a rock and a hard place and saving your sanity by not caring is a perfectly rational decision.

Re: Mistakes engineers make in large established codebases

#107

Post by mjr00 » Wed, Jan 08, 2025, 1:24 AM UTC

photonthug wrote: post #97

cratermoon wrote: post #36

Earlier quoted context omitted.

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolit…

Even if you’re generally suspicious of so called best practice/ design patterns / Martin Fowlerisms.. this is a time for the strangler approach. (Parent and siblings are already talking about the same idea without naming it.) Rewrites from scratch never work with sufficiently large systems, and anyone that’s been involved with these things should be savvy enough to recognize this. The only question is around the exac…

A similar, more concrete approach is parallel implementations, as written about by John Carmack[0]. I suppose the main difference is that parallel implementation has you very explicitly and intentionally leave the "old stuff" around until you're ready to flip the switch. I've used this approach in large-scale refactorings very successfully.

One of the benefits is that you get to compare the new vs old implementation quickly and easily, and it lets you raise questions about the old implementation; every time I've done this I've found real, production-impacting bugs because the old system was exhibiting behaviors that didn't match the new system, and it turned out they weren't intentional!

[0] http://sevangelatos.com/john-carmack-on-parallel-implementat...

Re: Mistakes engineers make in large established codebases

#108

Post by dasil003 » Wed, Jan 08, 2025, 1:29 AM UTC

Supermancho wrote: post #101

dasil003 wrote: post #98

Earlier quoted context omitted.

Yeah 100%. Honestly style / technique / language consistency are implementation details, it helps with engineer fungibility and ramp up, but it also works against engineers applying local judgement. This is something to briefly consider when starting new services/features, but definitely not something to optimize for in an existing system. On the other hand, data and logic consistency can be really important, but you…

A common experience (mostly in the Pacific North West) I have had is to implement a feature in a straightforward manner that works with minimal code, for some backlog issue. Then I'm told the PR will be looked at. A couple days later I am told this is not the way to do X. You must do it Y? Why Y? Because of historical battles won and lost why, not because of a specific characteristic. My PR doesn't work with Y and it…

Hard for me to comment definitively here since I don't have the other side of the story, but I will say that I have seen teams operating based on all kinds of assumed constraints where we lose sight of the ultimate objective of building systems that serve human needs. I've definintely seen cases where the true cost of technical debt is over-represented due to a lack of trust between between business stakeholders and engineering, and those kind of scenarios could definintely lead to this kind of rote engineering policy detached from reality. Without knowledge of your specific company and team I can't offer any more specific advice other than to say that I think your viewpoint of the big picture sounds reasonable and would resonate in a healthy software company with competent leadership. Your current company may not be that, but rest assured that such companies do exist! Never lose that common sense grounding, as that way madness lies. Good luck in finding a place where your experience and pragmatism is valued and recognized!

Re: Mistakes engineers make in large established codebases

#109

Post by citizenpaul » Wed, Jan 08, 2025, 1:31 AM UTC

peterldowns wrote: post #44

mgfist wrote: post #42

Earlier quoted context omitted.

I rarely see large 10m+ LOC codebases with any sort of strong consistency. There are always flavors of implementations and patterns all over the place. Hell, it's common to see some functionality implemented multiple times in different places

And it's fine, right? Honestly I think people need to realize that part of being a good engineer is being able to deal with inconsistency. Maybe submodule A and submodule B do network requests slightly differently but if both ways are reasonable, working, and making the company money, it's probably not worth delaying product improvements in order to make things "more consistent." On the other hand if no one in your c…

>and it's fine, right?

The hard part of being an engineer is realizing that sometimes even when something is horribly wrong people may not actually want it fixed. I've seen systems where actual monetary loss was happening but no one wanted it brought to light because "who gets blamed"

Re: Mistakes engineers make in large established codebases

#110

Post by magicalhippo » Wed, Jan 08, 2025, 1:33 AM UTC

notyourwork wrote: post #26

gwbas1c wrote: post #25

Unit tests, exhaustive regression tests, and automated tests are the best way to prevent regressions. Time spent writing good unit tests today allows you to make riskier changes tomorrow; good unit tests de-risk refactors.

Unit tests cover the single functionality but ignore the system as a whole. Most regressions I've seen in industry are because of a lack of understanding how the system components interact with one another. Therefore, I see unit tests as one pillar but also suspect that without good quality integration or end-to-end testing you won't be able to realize the riskier re-factors you describe. Perhaps you consider these p…

The way I like to view it is that Lego bricks might pass unit tests aka QC with zero issue, but you can still easily build an unreliable mess with them.