Live data from Hacker News

Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

news.ycombinator.com

311–320 of 704 posts

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#311
Upper management need to understand the problem and the options and need to buy-in on whatever you want to do.

Practically: cut the bleeding, get the current team at least using version control and working with a CI environment. That will be a lot of effort (been there before with a similar .Net product but much better team).

Then you're going to need significant resources to re-build on a modern architecture. I would simply go with releasing another product if that's at all possible. You clearly have some market and channel to sell into.

Just beware: this sounds like a problem which will take 3-5 years to solve and whose chance of success is dependant on organisational buy-in. So you need to ask yourself if you're willing to commit to that. If not, quit early.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#312

Earlier quoted context omitted.

That only works if the new pieces correspond to old pieces. If there's no good structure to build on, the units to be replaced will constrain the architecture of the new ship. At some point you end up trying to change a pumpkin boat into an aircraft carrier, and there's no obvious way you can do that one piece at a time.

> If there's no good structure to build on, the units to be replaced will constrain the architecture of the new ship. Which is why you do it in stages: add scaffolding until local rewrites are possible, then rewrite the business logic, then tear the scaffolding down.

That's a good analogy actually. Scaffolding is a kind of temporary test structure that you can use to maintain function while you figure out something better.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#313
post #284
post #66

Earlier quoted context omitted.

The problem with people new to the company starting a rewrite from scratch is that they often are poorly informed on why things were the way they were before. If you start big, you can have bad outcomes where the new system might be objectively worse than the old one... but you are stuck trying to get the new thing out for the next 5 years because too many people sunk too much political capital into it. As an example…

> why does our tracking pixel return a purple image? Now I'm really curious, is there some exciting non-obvious reason for a tracking pixel to be purple? Was it #FF00FF or more like #6600DD?

Obviously !! The anti-doppler shift trick /s :)

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#314
I’ve been in this exact situation, and honestly - just move on.

I mean, if you see this a fantastic opportunity to grow or whatever then fine, have at it.

However, you’re going to be fighting a two-front battle, both against the devs and against management, for widely different reasons. It’s going to take a toll on you.

Ask yourself if you really want to spend the next few years doing work you probably won’t see any recognition for.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#315
post #281

Earlier quoted context omitted.

I think the thing is that the codebase after 5 years of development was already capable of making 19M/year, and the past 7 extra years have just added 1M/year. The next year of development will not add any more because the thing is collapsing under it’s own weight.

so it will just be 20 mil a year with 3 jr devs sounds ok?

Only in absolute terms. Kind a feels like a missed chance if you actually have a 100M market.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#316
To be honest. I would run off. This is the kind of hell where nobody wants to work. Where only "experts" know how things work and how to expand or fix things. It will secure their job but will make yours hell.

You personally will gain no knowledge there, just that your codebase is hell.

You cat try to convince the management of creating a new gen implementation. Not a rewrite. New software, that can fulfill customer needs better. Compete better, is safer and better to extend to do all this in the future.

One thing you can do though is to immediately set up modern practices. SCM, Code Review, CI, Tests (most of the code might not be unit testable in this state, but some tests at least) - This way you can see what others do when they add if fix something and learn better (SCM, Reviews), make changes and know that you did not break the whole thing (Tests) and have CI to at least ensure the tests run and everything works and it will glue all together.

Good luck

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#317
post #64

> this code generates more than 20 million dollars a year of revenue From a business perspective, nothing is broken. In fact, they laid a golden goose. > team is 3 people, quite junior. One backend, one front, one iOS/android. Resistance to change is huge. My mistake, they didn't lay a golden goose--they built a money printer. The ROI here is insane. > productivity is abysmal which is understandable. The mess is just…

This is very insightful. I learned it kind of the hard way. The business world is a mess. Requirements are a mess and always changing. This leads to messy code that requires a lot of time to clean up. You don't have time for that as long as there is always more customer wishes and projects coming in. As long as the business keeps working there's always something of top priority coming in. The pain starts growing but the steaming pile of code just doesn't collapse. It just kind of keeps on working while you are adding more and more code. Sure, the pain is big and progress is quite slow but what's the alternative?

My advice would be to listen to the developers, to understand them and the business. To understand what they really need. What a viable path forward would be. A complete rewrite, a second system for new developments, many more developers or something. Or maybe it is the optimum solution right now because the whole company is so messy and your job is not to change the company structure. Then you maybe you could support them by slowly enhancing their skill set and accept what you can't change. Doesn't sound like fun? Then leave soon, staying won't do you any good.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#318

Sounds like you're working with a previous client of mine. The best solution - for me - ended up dropping them as a client. There was zero interest in change from both developers and management (no matter how senior). We parted ways and I wished them good luck. Occasionally I wonder what happened to the application containing 50,000 procedural PHP files. Yes, 50k. And no source control or off-server backup.

Yes. It's a pointless uphill battle to try to change people who don't want to change. The employees have a lot of leverage by not documenting the mess. If they leave you will take the blame.

Get another job ASAP. Let natural selection do its magic.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#319
Unpopular opinion: this goes to show that you don't need no fancy microservices, distributed, asynchronous, highly available architecture to build a product that "generates more than 20 million dollars a year of revenue". No unikernels. No Kubernetes. Non of that cloud native mumbo-jumbo.

Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

#320
So, I do a have a war story about something like this, possibly at a worst state. And possibly with somewhat higher stakes (around 400m$/year) at the time. I came in as a consultant with my own "parallel implementation team". In my case I was somewhat lucky because most of the system was composed of batch jobs. They did have "frameworks" with "ORMs" but they had 4 or 5 of them, with many files being pinned to some older version. Which meant that actually there were dozens.

There were thousands and thousands of business rules no one knew why they were there and if they were still relevant. I remember one fondly. If product=="kettle" and color=="blue" and volume=="1l" then volume=1.5l... This rule like many others would run on the millions of product lines they would import daily. And the cutest thing in the system was that if any single exception happened during a batch run... the whole run would fail. And every run would take close to 15 hours (sometimes more).

Not going into details ... But they couldn't afford the run going over 24 hours... And every day they were inching closer.

Similar to OP they extensively used EAV + "detail tables" to be able to add "things" to the database.

The web application itself was similar but less of a time-bomb. It was using some proprietary search engine that was responsible for structuring much of the interaction (a lot of it was drill-down in categories).

Any change on the system had to happen live with no downtime. Every minute of downtime was $1,000 in lost revenue.

The assumptions we had were: 1. At some point the system will catastrophically fail so 100% of the revenu will be lost for a long time. 2. Even if it were possible to rewrite the system to the same specs (which it wasn't because no one knew what the system actually did) such a rewrite would probably be delivered after the catastrophe.

The approach we used was to 1. Instrument the code - see what was used what wasn't. We set some thresholds - and we explained to the stakeholders they were going to be potentially be losing revenue/functionality. And we started NoOping PHP files like crazy. Remember, whatever they did the worse thing they could do is raise 2. Transform all batch jobs to async workers (we initially kept the logic the same) - but this allowed us with 1# to group things by frequency. 3. Rewrite the most frequent jobs in a different language (we chose Ruby) to make sure no debt could be carried over. NoOp the old code. 4. Proxy all http traffic and group coherent things together with front controllers that actually had 4 layers "unclean external" - whatever mess we got from the outside. "clean internal" which was the new implementation. "clean external" and "unclean internal" which would do whatever hacks needed to recreate side effects that were actually necessary. The simple mandate was that whenever someone did any change to frontend code they needed to move the implementation to "clean external". 5. We ported over the most crucial, structuring parts to Ruby as independent services (not really micto-services just reasonable well structured chunks that were sufficiently self-contained). If I remember correctly this was something of the size of "User" and "Catalog browser" the other things stayed as PHP scripts. 6. And with savagery any time we got the usage levels of anything low enough.. we'd NoOp them.

Around a year in there was still a huge mess of PHP around but most of it was no longer doing any critical business functions. Most of the traffic was going through the new clean interfaces that had unit tests, documentation etc. I think that 100% of the "write path" was ported over to Ruby. A lot of reports (all of them?) and some pages were still in PHP.

I don't think anyone ever noticed all the functionality that went away. We had time to replace the search engine with Elastic Search. It wasn't clean by any means but it was sturdy enough not to have catastrophes.

The company was bought by some corp around that time... and they transitioned the whole thing to a SaaS solution. I was no longer involved for quite awhile so I only heard about it later. But we bought them that extra year or more.

So .. as far as recommendations go: 1. Instrument the code (backfire.io !) 2. Find bang for the buck and some reasonable layer separation and do it chunk by chunk. 3. Don't try to reproduce everything you have. Go for major use-cases 4. Communicate clearly that this is coming with functionality loss. 5. Be emotionally ready for this being a long long journey.

Post reply on HN