Ask HN: Are we overcomplicating software development?

From what you are describing, it seems like there is more a problem associated with your team than with anything else. Maybe you don't have the right set of expertise in the team and people tend to work better with what they are comfortable with. Microservices/redundancy support/CI at a fundamental level increase the complexity of how you go about things, but they do have benefits. They require a way of thinking and developing that should be a cultural fit for the team for it not to feel like you are constantly fighting the system. One way to get there is to incrementally add these after the primary project is done. When tackling one thing at a time, things end up being simpler and and the need for these things gets into the working habits of everyone better and they are no longer fighting the system.

Is everything else between these two groups completely equal? I seriously doubt it is, in which case I don't think it's fair to make any conclusions that hold weight.

This is one of the problems I have with these sorts of things. My company went Agile about two years ago, and lots of people like to rant about how much better everything is now and how much more productive we all are because of it. Except we actually have no way of knowing whether it made any difference at all.

- 1) Choose languages that developers are familiar with, not the best tool for the job

95% of the time, a language that your developers are familiar with is the correct tool for the job simply for that reason! There are cases where it is not the case but those involve special case languages and special case systems. If you don't know what special case means then you're situation is almost certainly on that list.

- 2) Avoid microservices where possible, the operational cost considering devops is just immense

"If your data fits on one machine then you don't need hadoop ..." Same thing applies here. Microservices have place and putting them in the wrong one will bite you bad.

- 3) Advanced reliability / redundancy even in critical systems ironically seems to causes more downtime than it prevents due to the introduction of complexity to dev & devops.

Then there's probably something wrong or limited with the deployment that needs to be reviewed (2 node when you need a 3 node cluster, bad networking environments, etc.) If you have a reasonable setup with solid tech under it, deployed per specs then this should not be true. If, on the other hand, something is out of whack (say running a 2 node cluster with Linux HA and only a single communication path between them) you're going to have problems and the only way to fix them is to get it done right.

- 4) Continuous integration seems to be a plaster on the problem of complex devops introduced by microservices.

I'm not sure about this but, if your deployment system requires CI you have a problem. An individual, given hardware and assets/code, should be able to spin up a complete system on a fresh box cleanly and in a reasonable timeframe. (Fresh data restores can take longer of course but the system should be runnable barring that.) If this requires (i.e. it can't reasonably be done manually) something like an CI script or ansible/chef/etc. script then you're deployment process is probably too complex and needs to be re-evaluated.

- 5) Agile "methodology" when used as anything but a tool to solve specific, discrete, communications issues is really problematic

Agile is commonly used to gloss over a complete lack of structured process or a broken. Even with Agile there should be some clean process and design work that goes into things or you're hosed.

Yes, we are over complicating software development.

I am working in huge non-IT company as a software developer. I guess that is what gives me a totally different point of view on your lessons:

1) Without a unified technology stack and a common framework we would not be able to build and maintain our applications. We decided on C# as it works best for us. Currently we are 5 developers. Not a single one of us has ever written a line of C# code before entering the company - learning the language from ground up enables us to pick up patterns that our colleagues who joined the company earlier found to be best practices.

2) If you are not introducing a whole new stack with every micro service that you develop the devops costs are quite low.

3) I agree with you on that - I think redundancy always introduces more complexity. However there are systems that handle that job quite well (e.g. SQL Server). For application servers we use hot-spares and a load balancer that only routes traffic to them, when the main servers are not reachable. This works for us, as all our applications are low traffic applications.

4) Continuous integration works brilliant for our unified stack. In the last two years we went down from 1d setup + 20min deploy to 10min setup + 20s deploy.

5) We use agile methodology whenever possible and it works like a charm. However we had a lot of learnings. Most recent example: Always have at least one person from all your target groups in any meeting where you try to create user-stories.

Planning our software architecture has been a key element in my teams success and I do not see a point where we are going to cut it.