Genuinely would like to know how anyone has managed to do both of:
> write as many end-to-end and integration tests as you can
and
> make sure your tests run fast enough to run the full set of tests after every commit
How to Improve a Legacy Codebase
291–300 of 300 posts
Re: How to Improve a Legacy Codebase
#292Earlier quoted context omitted.
Depending on how convoluted the case is, you don't know what the end result would save in costs. "we'll be saving __ hours/days of effort per build/year" is a complete unknown.
No, the time/effort to make a build with a working CI system is zero, plus any activities that remain manual by design (e.g. installations that require physical access to the server). The uncertainty is only about the feasibility and cost of implementing CI: this is one of the rare cases in which the benefits of software can be measured objectively, easily and in advance.
I would disagree, you still have the build time. And even if engineer doesn't do anything during that time, they are still occupied. No one would switch tasks during 10 minute build, there's just nothing you can do during such a short timeframe. In that case, in terms of business costs, it doesn't matter if engineer is busy or not during that period: they are still not gonna be doing more.
Re: How to Improve a Legacy Codebase
#293Earlier quoted context omitted.
That's the thing, the research and cost assesment itself would take days of work. So not many would care, and one might argue why /should/ you care to begin with. That's how it ends up being stalled at the idea stage.
I can conjure a scientific wild ass guess on the spot. "I wasted around 10 hours last week thanks to inadvertently pulling broken builds because we don't have a CI server. I spent 4 hours manually deploying things because we don't have a CI server." "When can I move the new CI server I already setup on my workstation - because fuck wasting half my week to that nonsense, and I had nothing better to do while the devs w…
Thank you for insight. If the deployment indeed takes hours and there are high chances of pulling wrong builds, then indeed the time costs are very tangible. But if manual deployment takes at most 10 to 20 minutes and relatively straightforward, then it might be less of a case. Depending on how often you deploy also impacts this greatly. I guess in certain cases ROI is just not extremely high and that greatly reduces the appeal of such investment.
Re: How to Improve a Legacy Codebase
#294Earlier quoted context omitted.
In my personal experience, the codebase is not the problem, but the people and the culture who made the codebase. We sent people to the moon with the computing power of a calculator, and with enough good people and effort, and version control, you can rewrite any legacy codebase to meet rigorous standards and meet the performance needs of the users. Out of all the things humanity is trying and has accomplished, this…
Your comment reflects what I heard last week from a friend. He's a PE (Professional Engineer) working with a large power utility company, and his company uses software that, in his words, "really sucks". He was telling me this because he got a job offer from a startup, where they wanted him to be the subject matter expert for an application they were developing for power utilities. It's a good offer, and he's tempted…
Interesting.
I am getting the same kind of offers. Theres alot of subsidies and VC investment in "clean" energy and "smart grid" so developers left and right have a new market to apply their software development skills to.
In my experience, I am not being used as a reference book, and my long term investment in coding/Minor in Compsci and personal projects make me able to translate enegineering speak to software speak. I have not had the experience that the developers think software is the answer and that power is trivial.
I think the real clash of cultures is the culture of software development realizing how much beuracracy surrounds the power grid, and how much resistance there is to change within the industry, because the industry really does not understand how 95% of their day manually editing excel workbooks that output fortran run files from the 80s could be deleted and improved, and of course, most of them don't want to improve since there is a 30yr generational gap in the power industry.
Half of the people I work with don't believe in climate change and think Elon Musk is taking their jobs. This industry is ripe for disruption and I don't think the mentality that software/smart grid stuff can improve it is incorrect, but I do think the naivete that everyone in every industry is as openminded and continually putting in effort to learn and produce working products, which is a mindset required by successful growing software firms, is proving to be a big barrier and wake up call to software companies trying to come in an help.
I honestly blame our industry over software developers, but yeh, theres also a goldrush in "smartgrid" "clean energy" stuff and everyone want's to be apart of it.
Definitely as a Power Engineer it leaves you getting lots of opportunities and having to sort through who is willing to invest in understanding the complexities of innovating on the power grid, and who is going to cop out once they realize you can't whip up an app and make money off users the way snapchat will.
Theres also, and for good reason, lots of cybersecurity policies surrounding software running on the power grid, because hacking the grid has detrimental effects that can quickly translate to coast wide blackouts etc. That also means the newest github release of that multiplatform coffeescript spinoff is not going to be allowed to be used in alot of the grid side applications, and there is more work involveded with vetting development.
All of these things really quickly weed out the devs who are looking for quick stardom and the next easiest cashcow, and landing on the latest buzzwords related to clean energy. It can be frustrating weeding out the companies who try to hire you from that perspective.
It doesn't mean the grid doesnt need better software, it just means people, even developers who want to cash in on hot finance markets are going to take the path of least resistance, and the powergrid is not the path of least resistance (no pun intended) when it comes to quick cash and unicorn apps.
You have to actually CARE about innovating on the grid, and not just to pretend to care because theres billions sitting around in funding waiting to fund good smart energy innovation. Regardless, because this money is there now, and there's the backing of social reinforcement of being able to herald your startup as bleeding edge world saving technology thats going to stop societies impending doom from global warming, is a very compelling emotional appeal that makes marketing and justifying your product easy.
It's hot right now and in then ext ten years we will see who is around to grab quick cash and feel good about being the poster child for saving the climate and who is willing to invest in truly renovating the powergrid and enabling clean energy as a sutainable long term solution that is eocnomically viable without startup subsidies to cover the cost of initial investments.
Eventually these companies have to show a profit...
It is frustrating to be in the industry as an Engineer under the age of 35, and also have exposure to Compsci and friends working at Amazon and Google, and having to explain a hyperlink to a coworker 3 levels above you.
It's also frustrating to take graduate level classes and do research and R&D and have spent years designing power grid and actually being on the power grid and seeing construction and putting in hard work to learn Electrical Power before it was "cool" and then have an insurmountable rush of developers who want you to help them change the world. They are the CEO, you are the reference book engineer.
I get that, but its important to look past that and see there is true benefit to the innovation. And for the most part they are right, this industry has been sitting in static mode riding comfortably for a long time in many ways when it comes to trying to stay technologicially relevant, and maintain sustainable infrastructure that allows for growth. So it is frustrating, but ultimately the software/tech community is in the right. It's time for a change and this industry needs to admit it kind of sucked at changing itself for decades.
It also helps that I have 8 years of coding experience I put in on my own time to help ease this barrier but I am an exception to the rule as I have been told by recruiters doing smart grid dev.
Re: How to Improve a Legacy Codebase
#295Earlier quoted context omitted.
I can conjure a scientific wild ass guess on the spot. "I wasted around 10 hours last week thanks to inadvertently pulling broken builds because we don't have a CI server. I spent 4 hours manually deploying things because we don't have a CI server." "When can I move the new CI server I already setup on my workstation - because fuck wasting half my week to that nonsense, and I had nothing better to do while the devs w…
Thank you for insight. If the deployment indeed takes hours and there are high chances of pulling wrong builds, then indeed the time costs are very tangible. But if manual deployment takes at most 10 to 20 minutes and relatively straightforward, then it might be less of a case. Depending on how often you deploy also impacts this greatly. I guess in certain cases ROI is just not extremely high and that greatly reduces…
No problem, hope it's useful :). You're right about deploy frequency - that 10-20 minute manual deploy done 3-6x a day already adds up to my 4 hours a week deploying things. 10% of the work week right there!
On the other end of the spectrum, a lot of my personal projects don't warrant even the small effort of configuring an existing build server. I'm the only contributor, nobody else will be breaking anything or blocking me, builds are super fast... even if "it will pay off in the long run", there are other higher impact things I could do that will pay off even better in the long run.
In the middle, I've put off automating some build stuff for our occasional (~monthly) "package" builds for our publisher - especially the brittle, easy to fix by hand, rightly suspected to be hard to automate stuff. I was generally asked to summarize VCS history for QA/PMs anyways - can't automate that.
When we started doing daily package builds near a release, however, it ate up enough of my time that non-technical management actually noticed and prodded me before I thought to prod them. Started by offloading my checklist to our internal QA (an interesting alternative to fully automating things) and eventually automated all the parts QA would forget, not know how to handle, or waste too much attention on.
Even then, some steps remained manual - e.g. uploading to our publisher's FTP server. Tended to run out of quota, occasionally full despite having quota available, sometimes unreachable, or uploading too slow thanks to internet issues - at which point someone would have to transfer by sneakernet instead anyways. Not much of a point trying to make the CI server handle all that.
Re: How to Improve a Legacy Codebase
#296Earlier quoted context omitted.
No, the time/effort to make a build with a working CI system is zero, plus any activities that remain manual by design (e.g. installations that require physical access to the server). The uncertainty is only about the feasibility and cost of implementing CI: this is one of the rare cases in which the benefits of software can be measured objectively, easily and in advance.
I would disagree, you still have the build time. And even if engineer doesn't do anything during that time, they are still occupied. No one would switch tasks during 10 minute build, there's just nothing you can do during such a short timeframe. In that case, in terms of business costs, it doesn't matter if engineer is busy or not during that period: they are still not gonna be doing more.
The alternative to spending X minutes waiting for a CI build is spending slightly more than X minutes executing a manual build with a nonzero chance of time-wasting mistakes, not doing nothing.
Not waiting for a build means not testing it, in which case a manual non-build of an uninteresting software configuration seems attractively elegant but a CI system still provides value by recording that a certain configuration compiles and by making the built application available for later use in case it's needed.
Re: How to Improve a Legacy Codebase
#297Earlier quoted context omitted.
I would disagree, you still have the build time. And even if engineer doesn't do anything during that time, they are still occupied. No one would switch tasks during 10 minute build, there's just nothing you can do during such a short timeframe. In that case, in terms of business costs, it doesn't matter if engineer is busy or not during that period: they are still not gonna be doing more.
The alternative to spending X minutes waiting for a CI build is spending slightly more than X minutes executing a manual build with a nonzero chance of time-wasting mistakes, not doing nothing. Not waiting for a build means not testing it, in which case a manual non-build of an uninteresting software configuration seems attractively elegant but a CI system still provides value by recording that a certain configuratio…
That is indeed true, CI is way less error prone, thanks. Having a build ready to go is quite handy too.
What do you mean by "not testing it" "seems attractively elegant"? Testing a build is still a must, although that ends up to be manual testing usually (unit tests don't assure much, integration take a lot of engineering effort for setting up and writing, especially if they were not taken care of from the start).
Re: How to Improve a Legacy Codebase
#298Earlier quoted context omitted.
I think its pretty common- and I think you're lucky. I was surprised to see the article say "It happens at least once in the lifetime of every programmer,". I think if you work on greenfield projects your whole career you're likely the one who's creating these 'steaming piles of manure'. By working on bad legacy projects you learn an awful lot of things about what works and what is a problem to maintain - it will mak…
I think you're setting up a false dichotomy. There are codebases other than just legacy and greenfield projects: high-quality, well-structured and well-maintained code. I would agree that if all you work on is greenfield you're probably making the messes others are cleaning up, but I don't think that means developers are bound to either make messes or clean them up. There are plenty of good, long-lived projects out t…
This is what I've been wondering about. I don't care if the stack isn't the newest, or the tech is the shiniest. I'm just more interested in working on code that was _engineered_. That is code that was designed and then built. That's the problem I have with most of the code I'm working in.
At my current place of work, we're not even using xmlhttprequest. We're using an antiquated xml library that's been hand rolled (xajax + major changes) to emulate our ajax requests. It's insanity to me that we're still in this mode.
Re: How to Improve a Legacy Codebase
#299How does one get better if they only ever work in code bases that are steaming piles of manure? So far I've worked at two places and the code bases have been in this state to an extreme. I feel like I've been in this mode since the very beginning of my career and am worried that my skill growth has been negatively impacted by this. I work on my own side projects, read lots of other people's code on github and am alwa…
I've learned an enormous amount from fixing terrible bugs in terrible code. One tip is that when you've finally found the actual line(s) with the bug, always try to understand why the programmer made that mistake. This has taught me much about what constructs are error prone.
This is an interesting one. I'll have to think about this while I'm taking notes about bugs.
Re: How to Improve a Legacy Codebase
#300I mostly agree with this - bite-sized chunks is really the main ingredient to success with complex code base reformations. FWIW, if you want to have a look at a reasonably complex code base being broken up into maintainable modules of modernized code, I rewrote Knockout.js with a view to creating version 4.0 with modern tooling. It is now in alpha, maintained as a monorepo of ES6 packages at https://github.com/knocko…
Can I still install and use via a nuget package? It looks like it's integrated with all those crazy npm tools now but I'm not sure if that's just for development nor usage.
Sorry, @flukus, the alpha is not yet on nuget.