Ask HN: What are some architectural decisions that improved your codebase?

Honestly, tests and by extension testable code. The amount of enterprises processing tens to hundreds of millions of dollars (either business value or actual revenue) without tests of vital parts of their software is something which is mind blowing. You can sometimes not fathom how they are comfortable with changing a line without having tests to back them up. They f5 a page or recompile the server software, redeploy click through it and "yup it works let's ship" and then a few days later find out it broke a csv import of the external warehouse inventory system which runs once a week because they removed a dash between sku and title for better SEO in the online catalog. Oops, good luck finding out where the problem is because you have zero integration tests. A few million down the drain because import division couldn't possibly know what to forecast on due to no stock data. And this is not an exception to dumb bugs and malfunctions occurring because developers don't write tests.

You can start an entire business in consulting on test automation and you would never run out of work.

I don't know of any. Maybe the people who know this are too busy building them. I might be in trouble for writing as much as I did. :-)

YAGNI, KISS

Choose Boring Technology http://boringtechnology.club/

Build your system to be level-triggered as much as possible. Its default mode should be reconciliation: examining its current state and transforming that into the desired state, especially if the current state is "something went wrong". Build in dumb reconciliation before worrying about making it more real-time.

The fewer moving parts, the better. Don't go multi-service architecture until you absolutely have to (see YAGNI, KISS).

Keep your business logic contained, separated from everything else, in ONE place. If I open up your business logic code, I shouldn't see anything about persistence, the network, etc. Similarly, I shouldn't find any business logic in your other concerns. The business logic interacts with other concerns via abstractions.

Be unforgiving when it comes to correctness guarantees. Use the type system as much as possible to make errors impossible.

This is great! These are some concrete and non-trivial architectural techniques for "Systems Programming" :-)

I have had opportunity to work on/with some of these techniques on Fast Network Protocol/Security Appliances and so have some familiarity with them. However some of your hints(breadcrumbs?) are not known to me and hence i have something to research and study. Thank you.

PS: Can you add some more details on the above techniques? Like System/Library/API calls to look into, books/papers/articles to read etc?

Another hint.

Speaking of breadcrumbs, if the ring buffer has fixed-size entries, a reader can come in later and start reading old entries first, say halfway back. This is helpful if you want to start a new reader and then kill an old one, and not skip any entries.

It helps if the ring has power-of-two size, and the head pointer/index is 64 bits and increases monotonically. Then the high bits are easily masked off on each use, so that arithmetic on pairs of positions is simpler.

For variable-sized entries, an array of N "breadcrumbs", past positions near 1/Nth indices, allows jumping in at earlier positions. If traffic is low enough, you might be able to buffer a whole day's traffic, and get random access starting from breadcrumbs; otherwise, you can log old entries to a sequential file, and also log the breadcrumbs, translated to file offsets, as a global index.

Downstream processes can each sequentially log an individual field of each record, with a breadcrumb index to enable full records to be reconstructed. Often these column logs can be compressed, with enormous efficiency, between breadcrumbs: 98% compression may be easy to achieve for slowly-changing or limited-alphabet values.

Lz4 and Zstd are excellent compression engines. Lz4 really shines for fast decompression. There is no excuse for zlib/gz compression anymore.