Live data from Hacker News

Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

news.ycombinator.com

451–460 of 704 posts

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#451
Agree on lots of the points folks have shared. Testing, fix small chunks at a time, more testing.

I'll share some less technical thoughts that I think hold true regardless of the approach taken (rewrite or not).

My experience with changes like this is that you need to be as transparent as possible to both parties (the devs, and your execs). This means consistent comms around goals, achievements, and crucially the challenges preventing the first two.

With any team, you are not going to win much by implying that their work sucks or the thing they have built is broken. While they might know it, a third party is just not going to get a good reception with that mentality. It will be important for the devs to understand why the change is needed from a business perspective (e.g. time to market is too slow to remain competitive, changing regs, etc.). The intent here is to focus the devs on what the hope is for the future as opposed to shitting over the thing they have poured their blood, sweat, and tears into.

With the execs, they need to understand just how bad of a shape things are in so they give you and the team the space they need to make a significant enough change that isn't just going to revert to the same mess as before. If you're dealing with tech background execs it might be a bit of a simpler set of convos. But if not, then you are going to have to illustrate for them how bad things are. One way I've done this is to first get an idea of what the execs want as the final state of the team / codebase / product (e.g. time to market is < 4 wks) and then draw them a picture/flowchart of what it takes to get that thing done in the current state. Could use some form of value stream map to do this as it combines players, states, activities, and also timelines.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#452

Thank you for so many suggestions. The main issue is productivity within a context where the company is trying to reinvent itself in terms of marketing and business model. This has for consequence that many new big features are being requested and promised by management to headquarters. But in the last years, all bug evolutions have been failures. That's why I've been asked to intervene. I love the idea of the strang…

What is a bug evolution? Do you mean a bug fix?

"big evolution"

"u" is next to "i" on many keyboards.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#453

Some of the things the poster lists is just jaw dropping, but this is the most painful one. > - it has been developed for 12 years directly on production with no source control ( hello index-new_2021-test-john_v2.php ) First step would be to get that into source control. > - the routing is managed exclusively as rewrites in NGInX ( the NGInX config is around 10,000 lines ) This might be a benefit actually. I'd just s…

Didn't you hear "Twitter" develops on production too, with no staging env.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#454
post #30

Seeing a lot of people say that a rewrite is a terrible idea, but (as someone who doesn’t understand why) I’d love to hear a more fleshed out explanation re: why exactly that’d be a bad idea.

https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#455

Earlier quoted context omitted.

"Full rewrite" is a description of the end state, not the process. The best way to do a full rewrite is incrementally, with test support and consideration for natural separation of internal subsystems.

The best way to do a rewrite may be incremental, but the terminology of "full rewrite" doesn't usually refer to an incremental rewrite, it refers to starting from scratch.

I don't think that's true -- a "full" rewrite is used in contrast to a "partial" rewrite, where only part of the system is replaced. It's called a "full" rewrite because the goal from the start is to fully replace the system with new code.

Consider that if this were not true, then there would be no way to describe an incremental full rewrite, nor any way to describe a from-scratch replacement of a subsystem.

I've written on this topic before, for example https://increment.com/software-architecture/exit-the-haunted...

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#456
I would wrap this in some kind of thin candy shell so that you can inspect/record every single thing the application is doing, from a behavioral standpoint. For instance, put API gateways in front of it so that all requests are logged, the routes, parameters, etc. eBPF comes to mind as a tool that might help here. Same would go for instrumenting the database. The goal is to reverse engineer 80% of how it works by just looking at external IO. This is one area where containers can be very powerful, since they naturally encapsulate the system and allow for these sorts of bulk measurements 'for free'

My thinking is that you can essentially plug a thousand probes into this frankenstein monster and start to learn the true shape and surface area of it without needing to step through the mess of the code. Then the code might make more sense, or at least a clearer path forward as to what a new architecture needs to look could appear.

Static analyzers might also be helpful, since the full featured ones tend to provide gui tools or outputs of things like call graphs or dependency chains. That can be useful in learning the 'true' surface area of the app, too.

20 mil a year is no joke, so use that to your advantage. It sounds like this has been stretched so thin that at this point it is a huge disaster/liability waiting to happen, so I would try and leverage some of that cash to use whatever paid/advanced tooling might be necessary to help here. Old PHP apps are a security nightmare waiting to happen, particularly as the world has moved on to higher and higher TLS levels.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#457
I think you've received lots of helpful feedback here. I've read quite some of it and can follow some thoughts. What I wonder is:

- Is anything broken after all? Yes, there are annoyances and risks, but in the greater schenem, everything seems to work. Is fixing really necessary or would you just feel better after that?

- What does the "aggressive roadmap" look like? Build another product? Double or triple the revenue from this product? I think this helps/defines how to handle the situation.

- Your job as middle level management (at least that's how I understand it, being in that position myself) is to shield your teams from direct hits with piles of shit, while getting them running to evade the stuff by themselves at some point. Seems like your team already did great things in building the product, now help them get better, one small step at a time. I think they can see the benefits in things like using Git but probably you need to help them make some room to learn it without fearing that upper level management thinks they are lazy and not doing anything...

- Leaving the company: Maybe that's a viable option, too. You can't save them all. And if you feel overwhelmed by the task and see no way forward, you should leave. That's not about being weak, it's about protecting yourself from an unmanagable task.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#460
First thing you gotta do is set up VCS of some kind, ideally git. Next, like others have said, get some sort of proxy/logging layer in front of EVERYTHING. Work from there to document all the different ways customers interact with the service and how that maps to various parts of the codebase. Once you have a shit ton of documentation/logs to reference, you/team should start rewriting things piece by piece and using the aforementioned proxy layer to duplicate requests as a test and then divert traffic over. This is one of those things where every move has to be precisely calculated, but there's still a way out. Good luck.
Post reply on HN