Architecture.md (2021)

The two usual ways I've seen it is either by following the code/control-flow (calls or inverted as called by) or by following the data flow. You can select any code function and see the call (or called) graph (shown as a tree), similarly for any data element and see the data elements that use (or is used by) graph and pruning cycles.

One particular aspect of project architecture I often see people do wrong: a failure to have clear dependency structure between directories (and, often, too much stuff within a directory). This is particularly common if there is a directory named something like "common", "util", or "misc" (I am not actually arguing against having such directories, only noting they are prone to confusion).

I developed the following rules, which can be automatically enforced if you explicitly write the ranks (don't try implicit ranking; that means you won't get sane errors when you violate the rules):

0. Every directory has an implicit dependency on all its contained children.

1. A directory that does not depend on any other directories has rank 0. It suffices to only use rank between sibling directories, and it's probably simplest to maintain, though global ranking does work.

2. A directory that depends on others has a rank of 1 + the highest rank among its dependencies. To ease refactoring you could loosen this to "has a rank that is strictly greater than the highest rank among its dependencies".

3. Thus, circular dependencies between directories are forbidden; refactor (preferably, by splitting directories; most projects are too merged already) until you have a DAG. (circular dependencies between files in a single directory are allowed, subject to language-specific caution)

4. Thus, it is forbidden for a subdirectory to depend on a parent (or ancestor). If you encounter this, move the relevant files to a new subdirectory (since depending on a sibling or uncle is okay; if rank only applies between siblings this means the parent has to add a dependency on the uncle so that it gets the correct rank).

5. Depending on a cousin (or nephew) directory should be treated as a dependency on that cousin's parent, though (depending on what amount of directory structure your language forces you to use) it may be a hint you're doing something wrong.

6. Each directory can produce at most one library (shared and static count as the same library) or executable (at least, user-facing ones; code generators and tests might not count). Note that for other reasons it's generally inadvisable to ship multiple shared libraries or multiple static libraries, though you might use them during development.

7. (YMMV) If a directory contains any generated files (or their inputs), it should not contain any other files. Note that there are at least 3 major workflows for generated files, so the details will vary, but isolating them is useful regardless.

Note again: this is both coarser and finer than build dependencies - we treat directories as units, but add conceptual dependencies. As a general rule, I find it useful to define that the client (which calls `connect`) depends on the server (which calls `listen` and `accept`), and/or the data consumer depends on the data producer. Admittedly I have not deeply considered the case of servers that are worker-like, but note that it is often still possible to satisfy both by splitting directories further.

Note that additionally defining a "weight" (1 + weight of dependencies), although possible, is not particularly useful at the directory level. Long chains are easier to understand than tangled messes, but have a higher weight, and we don't want to discourage splitting a directory into a chain.

=====

In my experience, a project of about 100 kLoC organized itself about 20 immediate subdirectories of src/ with a max rank of about 10. This was roughly follows:

0-1: core support for the language, compiler, and replacements for the standard library; these almost never change. If somehow you have other kinds of code that doesn't depend on these, consider artificially inflating their rank to at least 2. About 1 each.

2-3: most fairly-project-agnostic (but not polyfill-like) "common"-like directories. About 2-3 each, probably.

4-6: semi-project-specific common stuff, most executables/libraries. IME generated files tend to belong here, and may be responsible for splitting a directory into a chain of length 3. There are a lot of these, usually with 1-2 dependencies of the previous rank and several more of the rank before that; likely there is nothing that depends on the entirety of any previous rank.

7-9: dependencies of the most complicated executable. About 1 each, a single chain at this point, though they might still have low-rank dependencies. At this point there's a decent chance that a directory might depend on the entirety of rank 2.

10: the most complicated executable (at least for me, this was the executable that had a conceptual dependency on other executables, even if it didn't have a source dependency. If you have a simple executable that conceptually depends on a complicated one, the simple one would be the highest rank instead). Note that only one of its dependencies is a 9; the rest are in the 3-6 range.

An alternate structure with deeper nesting, which I considered but never bothered to implement:

  src/base/ - the directories ranked 0-1 above
  src/lib/ - everything that either is part of a shared library, or is used by multiple executables (all directories ranked 2-3, and some ranked 4-6)
  src/each-executable/various/ - dependencies of a particular executable (including all directories ranked 7-9 above)
  src/each-executable/main/ or just src/each-executable/ - the file containing `main` and as little else as possible (otherwise we're likely to confuse "code closely related to `main`" and "code that is executable-specific but not involved in many dependencies")

At top level, I never bothered to formalize it, but it was basically a single chain (though exactly what a "dependency" is not quite as clear here):

0: scripts/ - executable scripts used during the build that don't have to be built, or that might be installed with no more than a shebang update.

1: src/ - all source code, including that for tests and tool/

2: tool/ - built executables needed for later parts of the build. Refactoring to split this out from shebang scripts is nice for your sanity, but may be noisy, especially if your build system's dependencies are sloppy. In contrast to build/ these are never cross-compiled.

3: build/ (all other output (data and potentially-cross-compiled code); contains bin/, lib/, and share/ at least)

Hopefully, projects that get much bigger than this can be split into further subdirectories so that the dependency ranking only need be done separately within them. I'm not sure if any big project is actually that nice in whole, but you should at least be able to create sanity in part without too much distress.

That said, keep in mind that this is just one approach to this particular problem, and it is just one architecture-related problem. Particularly, if your tooling makes it difficult to split directories (or files for that matter), fix your tooling first (related: "recursive make considered harmful")

I love the idea here, but i loath the example. TMI. Give the core idea, any communication points (sockets, apis etc), maybe an abstract diagram.

I've always found this to be a very useful practice. Many projects have a few core files (or packages / modules / whatever) where most of the changes happen. Being able to familiarize new contributors (or old returning ones) with those quickly really helps the startup time on a project.

I've added architecture files to projects at multiple jobs now [0], [1] and they've been well received. They're not perfect, but they're better than nothing.

[0]: https://github.com/zapier/zapier-platform/pull/324

[1]: https://github.com/stripe/stripe-cli/blob/master/ARCHITECTUR...

I’m 100% on board with this. Even if it’s just a see docs within the repo. I should be able to understand the intent of things without having to go read your website (should you’d remember to write one).

ASCII diagrams of components or your draw.io diagrams would go here. Knowing what you have is half the battle.