My approach to building large technical projects

My work is nowhere near as impressive as Terraform, Consul and Packer. Hashicorp products show insight and vision. They get things done. So take my thoughts as I haven't done something as impressive as that.

I generally work on small technical side projects that are achievable and do something interesting or useful. Then I start another one in a different subject or problem but also useful by itself with the desire of plugging/joining them together. Because they're all interrelated. They're technical demonstrations, not products.

I play in IPython Notebook or replit with no goal of shipping something then I when I have something satisfying I move it to its own repository on GitHub, then I add features increasingly until I am satisfied I understand what I intended. This is not product thinking, it's building to understand something that you think is interesting or useful.

A few months ago on HN I wrote a comment talking about my dream to create a JIT Compiler - similar but much less serious than V8, JVM or the CLR. I also talked about a multithreaded runtime that can schedule lightweight threads similar to Golang or Erlang. I want to understand asynchrony, parallelism, coroutines, multithreading and locks enough to abstract them away from developers. My JIT compiler is incomplete: it can JIT compile MOV and ADD instructions and printf without parameter passing being done but not much else. When I feel inspired to work on the next piece of work on it, I'll keep working on it. At the moment I'm more interested in parallelism and asynchrony runtimes so my attention is there.

I've noticed that there is a task that we put off with the desire to come up with an elegant solution so we don't have to do it. Such as a refactoring or yak shaving to support something we need to support to do what we want to do.

I've been learning about delimited continuation passing and algebraic affects.

With some of the core counts on AWS you could process many requests in parallel and yield on IO so you could get some extremely impressive requests rates were the software well designed. I think 600,000 requests per second is top of the line for a single c++ server on the fortune TechEmpower benchmarks.

I want to make my own things and understand them well. I also want to learn other project's codebases and understand them well enough to work and think in them.

If you have a side project, I encourage you to work on it. I am always interested in people's writeups of their side projects.

There is so much work to do, it's not just 1 thing you need to learn and understand and think in (the programming language) there's a 1000 other tasks that need to be done to get anything shipped or useful for other people to use generally. The "getting past the recognising you have 1000 tasks to do for the first time" is really difficult in just working and forging ahead and getting stuff done. If only I could tackle all those 1000 tasks the same way I do the things I am good at, rather than not experienced or effective at.

> At their best user stories represent problems to solve for the end user, but rarely have I been able trace back a user story to something that an end user actually wanted because nobody bothered to ask them.

This is such a universal problem in the industry. It's also the source of the endless reinvention cycles we go through.

Everyone hates user stories now because no one actually does the other work involved - talking to the users to get the stories in the first place. So it becomes an unproductive chore and people grow to hate it. Someday, someone will discover talking to the user and they'll have their own favorite little way of writing down the conversation and they'll come up with their own title for it. It will start getting hyped and eventually we'll all be doing it - except we still won't be talking to the users and the next generation will hate it. And it will repeat.

It's the same with User Stories, OOP, and every other concept in computer science since the 50s.

Sounds nice!

Definitely true. That's something my co-founder talks a lot about.

If you just stick at it for a few months, projects get huge. It's incredible how much code gets written when you look 6-months back. If you had to look forward and try to guess, you'd never imagine this much would have happened.