Live data from Hacker News

Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

news.ycombinator.com

71–80 of 704 posts

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#71
First of all start by preserving the application state to the version control.

Then start thinking about replicating deployment of the application. At start it can be a script that compresses and extracts the files to the production environment. This will benefit you to build similar environments or other more experimental development environments.

Once you have a flow of the application state management and deployment under control, you can start building on top of it.

The most valuable work would be to build a separate test suite that documents the most mission critical parts of code or application.

Only after this I would try to reason the changes to the application. The great part is that you have Nginx configuration as an abstraction layer. From there you dissect the application to smaller pieces and replace the application one redirection call at time.

If the application has an expected lifetime of over 2 years. Then these changes will pay themselves back as faster development cycles and maintainability of workers and codebase. This can be a selling point to management for roadmap or recruitment.

Good luck.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#72
Wow hard one to untangle and answer in a HN reply!

I sort of think... if you have to ask this here you might be in the wrong job? Was this a job that seemed like something else then became this? This sounds like a job for an experienced VP Engineering. It is a tough order. Wouldn't know how to do it myself. Lots of technical challenges, people challenges, growth challenges, and managing up and down.

The resistance to change is something you need to get to the bottom of. People are naturally resistant to change if they are comfortable, and we've all been through 'crappy' changes before at companies and been burned.

The solution might be to get them to state the problems and get them to suggest solutions. You are acting more like a facilitator than an architect or a boss. If one of them suggests using SVN or Git because they are pissed off their changes got lost last week, then it was their idea. No need to sell it.

This assumes the team feels like a unit. If the 3 are individualistic, then that should be sorted first. E.g. if Frank thinks it is a problem but no one else does, and they can't agree amongst themselves, then the idea is not sold yet.

Once you know more about what your team think the problems are and add in a pinch of your own intuitions you might be able to formulate confidently the problems, so you can manage their expectations.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#73
post #30

Seeing a lot of people say that a rewrite is a terrible idea, but (as someone who doesn’t understand why) I’d love to hear a more fleshed out explanation re: why exactly that’d be a bad idea.

having seen several rewrite attempts in my career, none of which were fully successful, here are some thoughts:

- The ultimate reason: it will take too long and be over budget. The business will (rightfully) ask why should they invest x amount of capital just to get essentially the same feature set back. Businesses do not care about whats under the hood.

And here is why: - the rewriter team usually does not fully understand edge/corner cases that the current mess handles, but obscures it.

- the rewrite inevitably ends up following the same patters that the original did leading to unusual/weird cases

- rewrite teams get too ambitious and attempt to over abstract and over engineer, eventually creating another mess understood by only them

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#74
I've been in quite a similar situation, in which we tried to redo everything without proper knowledge. That was a catastrophe. Please, don't promise a panacea to the managers - better code is not more profitable, necessarily. New mistakes will be made (inevitably) and old stuff may become unstable. Also, it's not uncommon that developers bail out due to pressure. Have your team prepared for baby-steps. * Before doing any actual work, I'd suggest everybody reading Clean Code and Clean Architecture. You'll have a better understanding of SOLID principles by then. * Start by adding version control and a separate environment for development / testing. * Try refactoring the least important things first. If they crash, it won't be so critical. The most complex modules will end up with more quality.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#75
post #66

Earlier quoted context omitted.

> It doesn't work. That's simply not true. I've inherited something just as bad as this. We did a full rewrite and it was quite successful and the company went on to triple the revenue. > get some testing in place Writing tests for something that is already not functional, will be a waste of time. How do you fix the things that the test prove are broken? It is better to spend the time figuring out what all the featur…

The problem with people new to the company starting a rewrite from scratch is that they often are poorly informed on why things were the way they were before. If you start big, you can have bad outcomes where the new system might be objectively worse than the old one... but you are stuck trying to get the new thing out for the next 5 years because too many people sunk too much political capital into it. As an example…

We (my good friend and I who both have 20+ years of experience) were brought in specifically do to the rewrite. We were new to the company. We actually had to rebuild the entire IT department while we were at it as well.

> new tech team immediately committed to rewriting the code base into ruby micro-services

well... sigh.

> These refactors inevitably failed to produce a meaningful impact to revenue

It sounds like less about the refactor itself and more about the skills of the team doing the refactor. You certainly can't expect a refactor to go well if the team makes poor decisions to begin with.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#77
Let me rearrange some of your points:

> - it runs on PHP

> - it doesn't use composer or any dependency management. It's all require_once.

Great --- explicit dependencies are better than magic. Personally, I'm a fan of require rather than require_once, because of some history, but require_once is mostly fine.

> - it doesn't use any framework

> - no MVC pattern of course, or whatever pattern. No templating library. It's PHP 2003 style.

This is the proper way to run PHP. Can you imagine if they used frameworks? It'd be a slow mess, with about 70 different frameworks. At least this is likely a bare metal, fast mess.

> - this code generates more than 20 million dollars a year of revenue

> - team is 3 people, quite junior. One backend, one front, one iOS/android. Resistance to change is huge.

So you've got 3 junior people managing 20M of revenue

> - productivity is abysmal which is understandable. The mess is just too huge to be able to build anything.

> I have to find a strategy to fix this development team without managing them directly.

> This business unit has a pretty aggressive roadmap as management and HQ has no real understanding of these blockers. And post COVID, budget is really tight.

HQ doesn't understand the process, can't even budget a manger, because apparently it's not your job to manage them. I'd bet their requirements are unclear and poorly communicated too.

> - the routing is managed exclusively as rewrites in NGInX ( the NGInX config is around 10,000 lines )

Great, the routing is one place!

> - no caching ( but there is memcached but only used for sessions ...)

Do you actually need caching? You didn't say anything about the performance, so I'm guessing not.

> - In many places I see controllers like files making curl requests to its own rest API (via domain name, not localhost) doing oauth authorizations, etc... Just to get the menu items or list of products...

Curl to the same server port is a bad pattern; yeah. Localhost or domain name doesn't make it better or worse. Figure out how to make those a call to a backend service maybe? Are you also saying this is running on a single machine (I think you are, but you didn't mention it)

> - it has been developed for 12 years directly on production with no source control ( hello index-new_2021-test-john_v2.php )

Ok, check in what you have, and make a deployment procedure that doesn't suck, and set things up so you have to use the deployment procedure.

> - no code has ever been deleted. Things are just added . I gather the reason for that is because it was developed on production directly and deleting things is too risky.

If you can, run profiling on the production site to see what code appears to be dead code, and run down the list.

> - the database structure is the same mess, no migrations, etc... When adding a column, because of the volume of data, they add a new table with a join.

Depending on the size and volume of the database and the operational requirements, this is kind of what you need to do. Do you have anyone with operational database experience who could help them consolidate tables, if that's what's really required? Is the database a bottleneck? You didn't say that, you just said you didn't like it. There's ways to add columns and migrate data, but it requires either downtime or a flexible replication system and some know-how. Consolidating the tables without at least write downtime is going to be a lot more challenging than if they had the opportunity to add columns at the right time... of course, sometimes having tables with a join is the right thing to do anyway.

Is there budget for a staging system, complete with enough database instances to test a data migration and time to do it? Maybe focus on developing a plan for future column additions rather than trying to clean up the current mess.

> - JS and CSS is the same. Multiple versions of jQuery fighting each other depending on which page you are or even on the same page.

jQuery is pretty compatible right? You can make a list of all the pages and all the versions and maybe make time to test updating the pages with the oldest versions to newer versions, etc. Again, a staging system would help with testing. Developing a testing plan and running the tests is something that doesn't require much from the three overworked developers, but could be offloaded to a manager.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#79

Thank you for so many suggestions. The main issue is productivity within a context where the company is trying to reinvent itself in terms of marketing and business model. This has for consequence that many new big features are being requested and promised by management to headquarters. But in the last years, all bug evolutions have been failures. That's why I've been asked to intervene. I love the idea of the strang…

> This has for consequence that many new big features are being requested and promised by management to headquarters. But in the last years, all bug evolutions have been failures. That's why I've been asked to intervene. I love the idea of the strangler pattern associated with big unit testing coverage.

The first thing that needs to be strangled is unachievable management promises. Figure out how to get local management to not write checks the tiny team can't cash. A team of 3 juniors will likely overestimate their ability to deliver, so you've probably got to teach them to say no to things they can't do also.

Post reply on HN