[deleted]
Ask HN: For a startup, when should I begin writing tests?
31–40 of 40 posts
Re: Ask HN: For a startup, when should I begin writing tests?
#32If you have hardly any experience, how can you be confident in your belief that "product agility is a huge asset early on"?
Re: Ask HN: For a startup, when should I begin writing tests?
#33Rather than skipping it as you are currently considering doing I’d add them as soon as possible for at least some end-to-end coverage of important features and authentication. If you have two or more developers I believe regression can take place shockingly quickly and it’s kind of sad to be made aware of a major problem by a user, I know first hand.
One thing that has been touched upon is that you are setting the culture, if you say you’re hesitant to sacrifice velocity for tests already that will probably become the long term culture and one day there will be no tests and everyone will be afraid to make any changes. Tests do take up front investment but they run multiple times a day and thousands or even tens of thousands of times over the lifetime of a product and pay it back.
I think striking a balance where you’re not overly concerned about coverage but have key end-to-end flows covered can make things more efficient while saving your skin and help avoid unhappy situations. I say limited UI front-end tests (these are the greatest time sink) of key flows, but then as much coverage of apis as you can do.
Also I’m founder at Tesults (https://www.tesults.com). Check it out. There’s a free plan but I’ll go beyond that and give everyone who posted in this thread one full target free if you email me. So you can report your test results nicely too and keep track of failures without cost. Any direct feedback is appreciated.
Re: Ask HN: For a startup, when should I begin writing tests?
#34If the accepted wisdom was true, that you can trade quality for speed, then you shouldn't write tests.
I do not believe that to be the case. I find the biggest value of tests is being able to change the code more quickly. Running tests is MUCH faster than a regression script.
There is a scale where you'll never have to change the code enough for the tests to pay off. Personally, I think that scale is quite small, maybe a week of development work.
However, it takes some experience to get good enough at the tests that you feel that benefit. If you're not there, the answer is probably different. However, if your startup scale is non trivial, I'd bet that the payoff if learning that skill and building some testing framework from the get go would be worth it.
Re: Ask HN: For a startup, when should I begin writing tests?
#35If you have no tests at all I would start with simple E2E tests to ensure your critical parts of app are working like sign up page and billing. You can add later other important sections of your app to ensure happy paths for your features are working fine.
When I started my own product for Ruby & JS developers https://knapsackpro.com I was doing mostly unit testing and later relayed on E2E tests for testing user dashboard to ensure happy paths are covered.
It's very easy to introduce weird bugs and it's much faster to run CI build to test your app than manually verify if it works correctly in your browser when you make some changes.
Re: Ask HN: For a startup, when should I begin writing tests?
#36We have automated tests for some parts of our app, depending on how much value tests add.
For example, code that parses data generally has lots of tests. It's usually easy to write unit tests for parsers, and it's hard to write parsing code correctly. These tests are invaluable and catch lots of errors, especially when you add features to the parser, or refactor the code, or optimize it. I would not touch any parsing code that doesn't have tests, because no matter how smart you think you are, you are going to break something everytime you make a change.
We also have tests for code that interfaces with 3rd party components. When those components change behavior, we can detect problems early.
We don't test most UI code, since tests are often difficult to write, need to be updated every time we make changes to the UI, and they catch few bugs.
Re: Ask HN: For a startup, when should I begin writing tests?
#37Working inwards from the edges of your product is a quick strategy that could provide immediate business value (compared to working from the bottom up watching metrics like coverage) and that means end-to-end testing . Testing the product with end-to-end tests will stop you from shipping a completely broken product to your customer, where the most damage can occur. Add the test script at the end of the build and aler…
End to end tests also tend to be more fragile and require a more complex test environment, this means you might end up spending too much time fixing broken stuff (tests, environment) and getting a lot of false failures.
On the other hand if you start early you can take testability into the design and make at least some of the above problems go away, for example by having a modular architecture that allows easy mocking.
I do recommend to minimize the use of UI tests (Selenium for web for example) as they tend to be even more fragile, focus on the most obvious sunny day scenarios and never ignore a broken test as it get rotten over time
Re: Ask HN: For a startup, when should I begin writing tests?
#38Adding a bit to the great answers here.
Are you doing Exploratory Testing [0] ? if not then you probably should.
You will probably find out that you need a bunch of tools to help you exploring the product, so why don't you spend some time turning those tools into crude test automation ?
Remember that the best test is the test that actually runs, an unwritten test doesn't really help you.
Re: Ask HN: For a startup, when should I begin writing tests?
#39I think it depends on the nature of your application. In my experience, in a startup situation the product design can be iterating quickly so both the tests and the code itself can be out of date very quickly. Personally would wait until I was close to product/market fit before writing tests.
Obviously, if you have a clear understanding of the finished product like in many enterprise applications then testing makes sense.
I would however, strongly recommend it if you are doing nuclear reactor software ;-)
Re: Ask HN: For a startup, when should I begin writing tests?
#40Try to use tests to save time? A good compromise might be that if something doesn't work the first time you try it, you can write a test to make debugging faster. If it turns out you made a bad design decision down the road, you can always delete irrelevant tests and reapply the rule about testing to save time. Unfortunately, inexperience is going to make it take longer to write tests. That probably won't be what kil…
I'm curious - what kind of organisational issues relating to hiring new developers would be alleviated by having some tests? Are you referring to incoming developers being put off by the perception of a legacy codebase which is horrible to work with? Just curious!
Not perception, here's what happens in projects I've been on with poor test quality (low quality tests, too few tests, sometimes essentially no tests, etc.):
Experienced developers (the ones who made it) are generally fast, quick to identify the source of an issue or where to make a change. They have better than even odds of not mucking it up (discovered by users later because testing in-house is too poor to catch it).
New developers are either paralyzed by uncertainty (will this really do what I want, does it have an unintended effect elsewhere that I can't detect?) or make changes and have better than even odds (much better) of mucking it up and creating a mess. Hopefully it's discovered before release, but most likely it's discovered after.
One of the critical things for new developers to a project is the opportunity to make changes and see what happens, with good feedback. If you permit them to make the changes, but lack the ability to provide good feedback on the effect of those changes, then they cannot learn the system from the perspective of a developer. Along with exploratory testing, it should be possible for developers to make compilable, but arbitrary, changes and see tests fail. This lets them know the impact of a change and understand how that section of code relates to the system as a whole. That holistic view is impossible on larger projects without good testing, or more experience than you want to wait for.
The importance of this will vary based on the importance of the system and its size and scope. Smaller size and scope, then they can probably learn the system well enough (say, less than 100k lines of code, depending on language). Beyond that size, the system will become a rotten mess and your customers will leave you if they have any choice.