Mistakes engineers make in large established codebases • Hacker News Board

Re: Mistakes engineers make in large established codebases

#261

Post by simoncion » Wed, Jan 08, 2025, 1:57 PM UTC

harrall wrote: post #77

cratermoon wrote: post #36

Earlier quoted context omitted.

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolit…

Also an issue is that the director attempted a full rewrite as a separate project. You can do successful rewrites but your rewrite has to be usable in production within like a month. If you don’t know how to achieve that, don’t even try. The quiet developer was able to get their own rewrite done because they understood that. Looks like the director of engineering showed some classic inexperience. You can tell when so…

> You can do successful rewrites but your rewrite has to be usable in production within like a month.

I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0]

The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work.

[0] I've worked for an Agile shop for a long time, so please don't tell me that I'm "Doing Agile Wrong" or that I misunderstand Agile. "No True Scotsman" conversations about Agile are pretty boring and pointless, given Agile's nebulous definition.

Re: Mistakes engineers make in large established codebases

#262

Post by ctxc » Wed, Jan 08, 2025, 2:00 PM UTC

dav wrote: post #140

I have three maxims that basically power all my decisions as an engineer: 1. The three C’s: Clarity always, Consistency with determination, Concision when prudent. 2. Keep the pain in the right place. 3. Fight entropy! So in the context of the main example in this article, I would say you can try to improve clarity by e.g. wrapping the existing auth code in something that looks nicer in the context of your new endpoi…

It's got a nice ring to it :)

Re: Mistakes engineers make in large established codebases

#263

Post by dotancohen » Wed, Jan 08, 2025, 2:26 PM UTC

peterldowns wrote: post #34

I agree that consistency is important — but what about when the existing codebase is already inconsistent? Even worse, what if the existing codebase is both inconsistent and the "right way to do things" is undocumented? That's much closer to what I've experienced when joining companies with lots of existing code. In this scenario, I've found that the only productive way forward is to do the best job you can, in your…

  > your work is an experiment and you will later revise it

I advise against this if you have not been allocated the time or budget to revise the code. For one thing, you're lying. For another thing, were you hired to be a part of the contributing team or hired to be part of a research team doing experiments in the contributing team's codebase and possibly deploying your experiment on their production systems?

I would immediately push back on any new guy who says this, no matter how confident he seems that his way is the right way.

Re: Mistakes engineers make in large established codebases

#264

Post by bluGill » Wed, Jan 08, 2025, 2:41 PM UTC

danjac wrote: post #245

huijzer wrote: post #181

Earlier quoted context omitted.

And then at some point the codebase becomes so unusable that new features take too long and out of frustration management decides to hire 500 extra programmers to fix the situation, which makes the situation even more slow. As I understand, there is a balance between refactoring and adding new features. It’s up to the engineers to find a way to do both. Isn’t it also fair if engineers push sometimes back on managemen…

Often the problem with companies running the Feature Factory production treadmill too long is you have code supporting unused features and business logic, but nobody knows any more which features can be dropped or simplified (particularly after lots of employee churn and lack of documentation). So the problem is not so much technical debt, but product debt. You can refactor, but you're also wasting time optimizing co…

I was involved with a big rewrite. Our manager had on his desk the old system with a sign "[managers name]'s product owner". Nearly every time someone wanted to know how to do something the answer was load that old thing up and figure out what it did.

Eventually we did retire the old system - while the new code base is much cleaner I'm convinced it would have been cheaper to just clean that code up in place. It still wouldn't be as clean as the current is - but the current as been around long enough to get some cruft of its own. Much of the old cruft was in places nobody really touched anymore anyway so there was no reason to care.

Re: Mistakes engineers make in large established codebases

#265

Post by mihaaly » Wed, Jan 08, 2025, 2:43 PM UTC

dotancohen wrote: post #263

peterldowns wrote: post #34

I agree that consistency is important — but what about when the existing codebase is already inconsistent? Even worse, what if the existing codebase is both inconsistent and the "right way to do things" is undocumented? That's much closer to what I've experienced when joining companies with lots of existing code. In this scenario, I've found that the only productive way forward is to do the best job you can, in your…

> your work is an experiment and you will later revise it I advise against this if you have not been allocated the time or budget to revise the code. For one thing, you're lying. For another thing, were you hired to be a part of the contributing team or hired to be part of a research team doing experiments in the contributing team's codebase and possibly deploying your experiment on their production systems? I would…

Counter-thought:

We are making brand new things here and not being in an assembly line coming up with the very same thing dozens to million times. We are paid to make new products never existed, having novelty elements in it desired to be a bigger extent than not!

Those pretending knowing exactly what they are doing are lying!

Of course we are speculating here about the size of novelty content to a differing extent, which is never 0% and never 100%, but something inbetween. But those pushing back on those at least trying to revise the work - putting emphasis on it -, deserve no-one coming to them to be pushed back (at least for the inability of allocating resources for this essential activity of development. Development!).

(tried to mimic the atmosphere of the message, sorry if failed)

Re: Mistakes engineers make in large established codebases

#266

Post by lmm » Wed, Jan 08, 2025, 2:43 PM UTC

edanm wrote: post #251

lmm wrote: post #103

I've worked in codebases like this and disagree. Consistency isn't the most important, making your little corner of the codebase nicer than the rest of it is fine, actually, and dependencies are great - especially as they're the easiest way to delete code (the article is right about the importance of that). What's sometimes called the "lava layer anti-pattern" is actually a perfectly good way of working, that tends t…

> No-one's actual business problem takes 5M lines of code to describe, those 5M lines are mostly copy-paste "patterns" and repeated attempts to reimplement the same thing. I'm pretty sure this is trivially untrue. Any OS is probably more than 5M lines (Linux - 27.8 lines according to a random Google Search). Facebook is probably more lines of code. Etc.

> Any OS is probably more than 5M lines (Linux - 27.8 lines according to a random Google Search).

Linux is notoriously fragmented/duplicative, and an OS isn't the solution to anyone's actual business problem. A well-factored solution to a specific problem would be much smaller, compare e.g. QNX.

> Facebook is probably more lines of code.

IIRC Facebook is the last non-monorepo holdout among the giants, they genuinely split up their codebase and have parts that operate independently.

Does Facebook have more than 5M lines of code now? I'm sure they do. Does that actually result in a better product than when it was less than 5M lines of code? Ehhhh. Certainly if we're talking about where the revenue is being generated, as the article wants to, then I suspect at least 80% of it is generated by the So I mean yeah, on some level solving the business problem can take as many lines as you want it to, because it's always possible to add some special case enhancement for some edge case that takes more lines. But if you just keep growing the codebase until it's unprofitable then that's not actually particularly valuable code and it's not very nice to work on either.

Re: Mistakes engineers make in large established codebases

#267

Post by billy99k » Wed, Jan 08, 2025, 2:48 PM UTC

citizenpaul wrote: post #106

digging wrote: post #67

Earlier quoted context omitted.

> If it's actually better, others will start following your lead. Not really my experience in teams that create inconsistent, undocumented codebases... but you might get 1 or 2 converts.

>Not really my experience in teams that create inconsistent, undocumented codebases... but you might get 1 or 2 converts. This has also been my experience. Usually there is a "Top" sticky/unhelpful/reticent person. They are not really a director or exec but they often act like it and seem immune from any repercussions from the actual higher ups. This person tends to attract "followers" that know they will keep their…

This exactly. I worked at a place one time with a terrible code base. They based it on open source and slapped on additions with no style or documentation.

My first day, I couldn't even stand the code base up on my local dev environment, because there were so many hard-coded paths throughout the application, it broke (they were unwilling to fix this or have me fix it).

I tried to accept their way of coding and be part of the team, but it got too much for me. They were staunch SVN supporters. This isn't much of a problem, but we had constant branching problems that Git would have resolved.

As I got assigned work, I noticed I would have to fix more bugs and bad coding, before I could even start the new addition/feature. It was riddled with completely obvious security vulnerabilities that were never fixed. Keep in mind that this was the new product of the entire company with paying customers and real data.

The team lead was also very insecure. I couldn't even nicely mention or suggest fixes in code that he had written. The interesting thing is that he didn't even really have a professional coding background. He went straight from tech support to this job.

I lasted about a year. I got let go due to 'money issues'. Shortly before this, they wanted me to merge my code into my branch with the Jr. developer's code right before my vacation (literally the day before).

I merged it and pushed it up to the repo (as instructed) and the team lead sent me nasty emails throughout my vacation about how various parts of my code 'didn't work'. Not only were these parts the Jrs code, it wasn't ready for production.

The other thing to know about the team lead is that he was extremely passive aggressive and would never give me important project details unless I asked (I'm not talking details, just high-level, what needs to be completed).

We had a call where he told me I 'wasn't a senior developer'. I wanted to tell him to fuck off, but I needed the job. The company went out of business 2 months later.

I found out their entire business model relied only on Facebook Ads, and they got banned for violating their rules.

Re: Mistakes engineers make in large established codebases

#268

Post by z3t4 » Wed, Jan 08, 2025, 2:53 PM UTC

If you have no idea what the usage is you are already doomed. However tests helps, whenever there is a regression you should write a test so that the same thing wont regress again.

Writing code is not like in real life where herd mentally usually saves your life. Go ahead and improve the code, what helps is tests... but also at least logging errors, and throwing errors. Tests and errors go hand in hand. Errors are not your enemy, errors helps you improve the program.

Re: Mistakes engineers make in large established codebases

#269

Post by tsarchitect » Wed, Jan 08, 2025, 3:21 PM UTC

Animats wrote: post #169

> Single-digit million lines of code (~5M, let’s say) > Somewhere between 100 and 1000 engineers working on the same codebase > The first working version of the codebase is at least ten years old That's 5,000 to 50,000 lines of code per engineer. Not understaffed. A worse problem is when you have that much code, but fewer people. Too few people for there to be someone who understands each part, and the original autho…

LOC is 'not a good' metric to 'you should be able to understand a codebase'. In either scenario, too many people or too few people, or (my favorite) 'not enough' (whatever that means). Mythical Man-Month comes to mind. What I think you're trying to get at is you need skill to reverse engineer software. And even if you have that skill it takes time (how much?). We work in a multifaceted industry and companies need to build today. At any given project, the probabilities are small that there is a dev who has the skill. We all know 'they can do it/they can learn on the job/they'll figure it out'. And then OP's observation comes into fruition.

Re: Mistakes engineers make in large established codebases

#270

Post by Kon5ole » Wed, Jan 08, 2025, 3:23 PM UTC

Sometimes the right approach is to keep the consistency. Other times, that approach is either impossible or catastrophic.

IMO software development is so diverse and complex that universal truths are very very rare.

But to us programmers, anything that promises to simplify the neverending complexity is very tempting. We want to believe!

So we're often the equivalent of Mike Tyson reading a book by Tiger Woods as we look down a half-pipe for the first time. We've won before and read books by other winners, now we're surely ready for anything!

Which leads to relational data stored in couchDB, datalayers reimplemented as microservices, simple static sites hosted in kubernetes clusters, spending more time rewriting tests than new features, and so on.

IMO, most advice in software development should be presented as "here's a thing that might work sometimes".