Ask HN: For a startup, when should I begin writing tests?

How many engineers do you have and how fast are they likely to write code? It depends more on the product that you're building rather than the stack - if the tech is the product, spend more time on making sure it works as advertised. If the tech enables the product, spend less. Any competent engineer can pick up thousands of lines of technical debt/spaghetti code and spend a few afternoons swearing at it and figure out how to modify it. Technical debt is a problem, but more of a problem when you hit millions of lines, hundreds of engineers, and you need to be sure you can change stuff without breaking it.

Debt is useful - as long as you can take it on in a calculated way, don't stress about it too much. You might need to pivot and rewrite everything anyway, or at least have your fundamental assumptions changed in such a way that requires a partial rewrite. Tests won't help you much in those scenarios.

Pros

* Established the proper culture from day one. If you end up being successful, this is extremely valuable. Culture is very, very hard to change.

* Eliminated a lot of bugs and regressions.

* Gave us confidence working in a regulated space.

Cons

* Adds a lot of time to initial development

* Product complexity doesn't really require testing in the early stages.

* Slows down ability to pivot

-----

If I were to do it again. I'd write fewer tests, but establish milestones for requiring changing/additions to be tested.

* My rule of thumb would be: If you can't easily test it in the UI, write a test. Or, if it's business critical, write a test.

* Verify authorization and authentication. It's very important.

* Test anything that makes/loses money (for example, Strip, Recurly, etc).

* Jest snapshots are quick and easy. Use them.

* Write tests for utility functions

Don't test:

* Database migrations

* UI interactions (unless critical). They take a lot of time to write and UI changes frequently.

* Vendor/3rd party API integrations beyond surface level. In my experience, quick implementations always involve a bunch of mocking/stubbing - making these types of tests less useful.

Testing the product with end-to-end tests will stop you from shipping a completely broken product to your customer, where the most damage can occur. Add the test script at the end of the build and alert to a dashboard. “Alert” can mean email, and “dashboard” can mean “PC showing inbox for testalerts@mycompany”. You can get that running immediately. Alerts won’t tell you what broke of course, though you can use version control and bisecting to isolate breakages to commits, which is almost as good.

When you find bugs, then each bug is an opportunity to put some scaffolding around smaller subsystems as smaller tests to show the actual bug, as well as the fact that the bug has been fixed. Avoid committing the test without committing the fix. It’s a bad habit. You can show how the test used to fail in your commit message.

You’ll get an idea as to what are the most problematic pieces of your app. If one part is much less reliable than the others, then you can select that subsystem for verification with bottom up unit tests. Don’t spend too much time on each one though — it’s highly likely that in exercising your APIs through unit testing, you’re going to see ways in which they need to be refactored to make more sense and, importantly, become more reliable.

You don’t have to write thousands of unit tests. But you should absolutely have a basic test suite with integration tests that give you confidence in a new release. And it will make it easier to hire developers in the future.

A good test suite doesn’t slow you down at all, in fact, it speeds you up by helping you identify problems rapidly so you can fix code before it becomes a real problem.

I wrote these tests after launching the product publicly and just before support requests / bug fixing period started with the simple aim of avoiding regressions as I work through the improvements.

If I were to give general suggestions on the topic,

- In most cases having high level sanity tests is low effort high value decision.

- To write tests or not shouldn't be a binary decision. E.g. If you think a specific aspect is going to need refinement work then tests for those modules can help you immensely.

- Going for high % coverage very early on for the sake of mental satisfaction or bragging rights is usually not a good time investment

My only regret is not having any automated tests for frontend. Investing in some basic selenium tests after launch would have been worth it.