Earlier quoted context omitted.
In my opinion OP should seriously consider this advice. I really mean nothing patronizing here, but I suspect OP does not have the corporate experience to handle this situation. This is a corporate equivalent of a double-black diamond downhill route. OP was hired by people who have little understanding of tech and already came in with guns blazing. I might almost wonder if OP's a sacrificial lamb. But, the tech advic…
> OP does not have the corporate experience to handle this situation Even before that, s/he doesn't even seem to understand any measure of business. $20m/year with 3 people is BIG. Any disturbance to whatever makes that happen will hurt the business greatly. When a full rewrite shakes the boat and causes a few millions of revenue loss or the loss of potential market share or opportunities, they will rightly fire him.…
Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
541–550 of 704 posts
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#542Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#543One tip — don't complain to management about how "awful the codebase is" or "how you need to start over" (100% agree this is usually a terrible idea). Managers have been hearing this over their entire career as a technical manager (over time they can lose empathy being out of the weeds). It becomes an overused trope and management will start to see you as being problem-oriented.
I'm not saying don't surface the issues — management absolutely should have an accurate understanding. Instead, try and balance the good with the bad (and there will always be some good). Don't catastrophize — approach is as a manageable problem with quantified risk e.g. responding to estimation with "typically this is a straightforward problem to solve, but I've explored this area of the codebase and there are some challenges we'll need to overcome — the estimate will be larger and less precise then we want to see, and we'll benefit from prototyping/research/spikes to reduce risk of introducing serious bugs and come to a more accurate estimate".
You'll build trust by consistently delivering on the expectations you set around concrete features/task (including the negative expectations) then management will reach the conclusion themselves and will trust your assessment with any new project. Plus, management will ultimately see you as an incredible asset to help bridge the gap of the technical black box and their purvey.
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#544> this code generates more than 20 million dollars a year of revenue From a business perspective, nothing is broken. In fact, they laid a golden goose. > team is 3 people, quite junior. One backend, one front, one iOS/android. Resistance to change is huge. My mistake, they didn't lay a golden goose--they built a money printer. The ROI here is insane. > productivity is abysmal which is understandable. The mess is just…
1. You focus on the advice here, adding tests (which IMHO is actually pretty difficult on a legacy poorly architected/documented system), source control, refactoring, etc.... with any time remaining in the schedule devoted to adding features.
2. Your doppelgänger chugs along and releases several more features than you do, giving them a market edge.
What happens next I suppose depends upon what difference those extra features make. If the delta is small, you may be able to pull out ahead, in the long run. But if it's large, then your company may start to lose revenue / customers. Then, the screws will likely tighten even harder, and you'll be forced to sideline refactoring efforts and double down on delivering features. And then those refactoring/cleanup efforts will bit rot and you'll find yourself back roughly where you started, except now you're behind your competitor.
There's a quote I can't seem to find atm that summarizes this. It was something along the lines of "With php our product is already up and gaining market share meanwhile our competitor is still futzing with their rails configurations." (If you know the actual quote I'd love to see it.))
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#545It would be incredibly unlikely to convince management to stop the roadmap for a full rewrite unless you can really give some solid evidence and numbers to show the rewrite costs less than the effort needed to get new functionality added reliably into those parts of the system with issues. For a large system that would be basically impossible. If not able to pause the roadmap, trying to continue development on new features and making sure the new code base is kept synchronized will just be a nightmare.
Like many others have said, the most likely strategy that will get a successful outcome would be to:
- Get some automated testing for key business flows in place. These act as documentation and contracts for the basic business functionality that guarantees that revenue. These then act as safety net for when refactoring is taking place.
- Do targeted refactoring either as part of a 20% tech debt reduction budget your work into your roadmap planning, and/or factored into new feature estimates (fix as you are in there changing something)
- Get the basic structure and processes in place early as those will likely be possible to set up without a big, or at least minimal, impact to production (source control, branch management, PR process, coding standards, CI, deployment process)
It will take time to get through the whole source code, but you would be seeing incremental improvements over time at least. Plus, you can at least still manage to continue with the roadmap with adjusted expectations a bit more easily.
I have gone through a few different projects where it was either a full rewrite with new features only going into the new code base, full rewrite being kept synchronized with existing codebase receiving updates, and the incremental rewrite and the incremental generally will make the most sense.
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#546Earlier quoted context omitted.
Best answer. If the money is ok, and the environment not too toxic/stressful, you might just see it as a challenge to secretly improve a codebase without anybody noticing, while still delivering what the higher-ups want to see. Or maybe just scratch the first part and try to see how much further you can push that turd with every coding crime imaginable. One-up the juniors in ugly hack Olympics. Ship a feature and put…
> a challenge to secretly improve a codebase without anybody noticing That is how it should be done in any case anyway. Improvements should get slowly rolled out without disturbing the users and the business.
Not exactly. IT management should be always telling people stuff like "did you notice that the integration with XYZ that never worked well stopped failing?" or "did you notice that we delivered those few last features at record time?" and explaining why.
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#547There's no 20m/yr 3 jr dev team, it's just to get the scene set for asking questions about "what if there was a bad code base that was making money, how would you bring it up to spec"
This community is great at offering advice and telling people how to do things the "better" way.
Posted on the weekend too so people that are having downtime on Sunday have enough time to reply. Sorry guys, if you don't get contacted, you haven't passed the first tech interview.
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#548Earlier quoted context omitted.
We (my good friend and I who both have 20+ years of experience) were brought in specifically do to the rewrite. We were new to the company. We actually had to rebuild the entire IT department while we were at it as well. > new tech team immediately committed to rewriting the code base into ruby micro-services well... sigh. > These refactors inevitably failed to produce a meaningful impact to revenue It sounds like le…
You have a great experience and did a great job indeed. My only question is how does one get 20 years of such experience without horrific flashbacks of “let’s just rewrite it” decisions. Do you do rewrites/redesigns often? What’s your success rate?
First one was the above example. It was for the largest hardcore porn company on the planet. Myself and my good friend Jeff rebuilt an already successful business IT department from the ground up and made it even more successful. Ever heard of 'the armory in sf'?
Second was that jeff and I were hired as contractors by Rob @ Pivotal Labs (ceo) to help the CloudFoundry team rewrite itself after he had bought the team and trimmed it down to only the good people. That one was a huge mess. We spent a lot of time deep in shitty ruby code using print statements trying to figure out what 'type' of an object something was and, of course, backfilling tests. It was a fun project and both Jeff and I learned the Pivotal way, which was probably the most enlightening thing I had ever learned about how to develop software correctly from a PM perspective. If you want to improve your skills beyond just slinging code, spend some time figuring their methodology out. Much of it is documented in Pivotal Tracker help documentation and blog posts.
Third one was not really a rewrite, but the original two founders, who were not technical, had tried to hire a guy and got burned because the guy couldn't finish the job. Sadly, they had already paid the person a ton of money and got really nothing functional out of it. We (jeff and I again!) just started over. We did a MVP in 3 months (to the exact date, because we both know how to write stories using pivotal tracker and do proper estimates) and ended up doing $80m in revenue, in our first year with an initial team of 5 people.
Fourth one was three guys (who were also not technical) I kind of randomly met after I moved to Vietnam. They were deploying litecoin asic miners into a giant Vietnamese military telco (technically, they are all military). They had hired another guy to do the IT work and he was messing it all up. They invited me out to help install machines, I came out, rebuilt their networking layout and then proceeded to eventually fix their machines because the software 'firmware' that was on them was horrible. I also added monitoring with prometheus so that I could 'see' issues with all these machines. That first day on the job, they fired the other guy and made me CTO. We ended up deploying in another datacenter as well. It was a really wild experience with a ton more stories.
Life has been, um, interesting. Thanks for reading this far.
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#549To make allies of senior management, you need metrics. You need to show, concretely, how current operations put revenue at risk and make the incremental investment necessary for their roadmap items prohibitive. If you can swing a penetration test, they'll probably find plenty on a stack like this. Then you have a security justification. If not, get the best monitoring stack you can. Demonstrate reliability and performance issues. (As well as reliability and performance improvements.)
From there... I'll say the #1 tool I've used in situations like this is Fastly. VCL is way more flexible than your 10k line nginx rewrite file (I've been there, too). And the edge caching will paper over minor outages. Rollbacks are easy. Rebuild your site piece by piece and stitch it all together with a reverse proxy at the edge.
Advice: propose a "canary" portion of the site to rebuild, and make it the lowest revenue / highest complexity thing you can. Once you stabilize the cash cow, getting the buy-in to finish the job and deprecate the old code base will be tough.
I'd also advocate for adding 1 incremental engineer to your team. Make it a senior dev and interview specifically for people who have done this sort of thing before. Your team needs a hands-on mentor in the trenches with them.
Best of luck. It isn't easy, but it's rewarding.
Re: Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
#550This is bad.
> It's all require_once. > it doesn't use any framework
This is not necessarily bad.
> no code has ever been deleted
This is bad
> Multiple versions of jQuery
This is bad
> a full rewrite is necessary, but how to balance it?
You never need to fully rewrite something. You can always take an incremental approach. If the code and 3 people generate 20 million dollars of revenue (on their own? or with massive sales support? what is the cost of goods?) then it's got to be doing something right.
I'd start with the source control. Just check everything in, so it's easy to go back. Do that on the server they develop on, even. (But have a script that pushes the checked-in code to offsite.)
Second, make it possible to spin up a second instance of the same application, in some automated fashion, out of bare source control. This may mean dumping schemas and checking them in, and probably figuring out what data in the database are "necessary configuration" versus "user payload data."
Then, you can initiate integration testing on top of the second cluster. You can also turn this into some kind of local sandbox development setup.
Once that is done, you may be able to change the code quicker, because you can do and test it locally, and perhaps have some acceptance tests on top of the mudball. At that point, you can switch over to doing development locally and deploying (and, ideally, having the ability to un-deploy.)
After that, starting to clean up should at least be possible with less risk, because you can test it in isolation. You can then start pulling on threads in the code, such as standardizing library versions, detecting and deleting unused code, putting like modules together, and so on.
You don't need fancy tools for managing this, shell scripts and command-line git are probably plenty enough. Resist the temptation to spend six months engineering the build system of the future!
Of course a lot will depend on details, but from your brief description, this sounds like the path forward -- focus on making it possible and safe and cheap to iterate, and then you can get on with actual iteration. Don't waste time on big rewrites; instead do things incrementally. Don't believe that any one tool will save the day, because it won't.