Ask HN: How’d you go from a noodler to writing large well structured programs?

> How do you learn to structure a large program?

You can try to organize you stuff by: - layers (time) - functional units (what it does) - subsystems (parts assigned to people)

Over that, you can write a state machine to sequence the jobs, add tracing and isolate parameters.

The idea is to group, organize and classify iteratively.

Your project looks interesting.

Feel free to reach me @gmail.

Read Clean Code and Clean Architecture

Sounds like you and I are in similar situations. I work in finance and am somewhat of a "DevOps" type of person, although much less so than I used to be in my previous company. That being said, I wrote 2-3 monsters in the past few years - all were projects that I worked on solely from scratch - no outside involvement. The key for me personally was twofold:

1) Owning the project solely, in its entirety, from start to finish. this gives you that sense of responsibility that motivates you when you hit a road block or discover a major hurdle/roadblock/bug in your code

2) Creating large blocks of time to work on it. I found that when I tried to look at the code for 30-60 minutes at a time and get something done, that was the worst work I did and frequently had bugs or needed to be restructured entirely. The only way this didn't happen was when I was working frequently on the project. If I only had, for example, 5 hours in a given week to work on it, it was better to just not work on it unless those 5 hours could be sequential. It is hard to jump in to something, then think back to how it might affect something you wrote 2 weeks ago and then have to refer to that code. You'll spend more time just getting back up to speed than actually getting anything done.

For #1, this may or may not be possible for you. Not sure what the rules are about oversight and how well defined the roles are. But I found it extraordinarily helpful to involve absolutely no one. That also gave me a bird's eye view of everything and when you can recall how you structured a function/method/library that is being called, it's easy to see the pitfalls of how you will encounter a bug or hit your error capture section of your code. If you can't do #1 fully, then section off whatever you can. Clearly delineate what you are responsible for and what someone else is responsible for. For me, I hit limits with other systems that I needed to interact with within the company. So the maintainers of those systems were responsible for their system and only their system and I did my end separately.

For #2, this is usually very difficult to do. For the extremely large hurdles I needed absolute silence and huge blocks of time. This meant there were several weekends that I spent in the office, from 9 AM to 10 PM doing nothing but cranking out code with only food breaks. It worked for me, though. But it sucks big time for obvious reasons. Avoid that scenario at all costs if you can. Try to get those dedicated blocks at the office for complex coding tasks.

I would add a third critical element - that being to plan everything out in advance so that you can make sure what you have in mind actually works well. When you step through everything logically and see it all at once, you can identify problems, unforeseen situations, edge cases, and bottlenecks that will make you pull your hair out later on. Since this code already exists, I would instead modify this to be that you should plan out whatever solution you are going to pursue in full before you write even a line of code. Wait until you have a full game plan.

A lot of commenters have recommended reading books that lay out clean code structure and the such, but if the problem is as bad as you say it is, I would recommend reading the somewhat satirical "How to write unmaintainable code". You should be able to find out what issues you do need to fix with that, and the remedies you'll be able to find in the recommended clean code books.

It's not technically difficult to structure a project. But it can be very difficult to well-structure a project. And it has a cost: to set up, to maintain, and an on-going cognitive cost (hopefully offset by the gains).

One way is to avoid duplication: e.g. if the same code appears in two different places, make it into a separate function, and call it from those places.

For general organization, unless you have enough experience tonknow how your code is likely to evolve in future, your organizational strategy may quickly become unsuitable. For the details of Quant Analysis, this may be particularly true. You may need a rougher, broader organization scheme (such as display as opposed to analysis), or a framework that provides the common support code needed by all analysis routines, so the analysis become less cluttered. Apply this even when the support code is only used by one analysis - so that its parts appear in different places.

But here there's a cognitive cost: that analysis may be harder to understand with its parts in different places.

The version control and testing seems simpler to start. You could start using git right away; but it is possible to muck things up - usually can be sorted oit, but there's a learning curve. But for such key code, it's probably another reason to hire an experienced dev, as others have said.