Ask HN: Why don't you write functional tests?

I'm curious why some developers don't write functional* tests for their apps. My theory is that really smart & small teams just starting out probably don't feel the need. But when they get bigger, and have real revenue, users, and reputation on the line, they get serious about end-to-end testing.

But am I wrong?

Why don't you write end-to-end tests?

* I'm making a distinction between unit tests (e.g. no db/disk access, no I/O) and functional tests (e.g. full end-to-end test including the full server and web browser or mobile app)

(disclosure: I created a functional testing tool and started a testing-focused company.)

Thanks, Jacob. :-) Follow-up question: Do you do monitoring in production? Does it have the same issues in that it tells you that /something/ is broken, but doesn't tell you what's broken?

Most of my "unit tests" are functional by your criteria, at least as far as things like file I/O and database access are concerned. Maintain a reasonable separation between UI and the rest of your code, and this remains fairly straightforward even without any tool more fancy than xUnit and a collection of SQL scripts to automatically set up your test database. I've also found it easier in many cases to maintain a single parameterized test loop along with a library of sample data and corresponding "expected results" than lots of little tests in code. This is especially true in situations where you're changing code (and adding tests) directly in response to specific inputs your system failed to handle. And you can frequently re-use the same test loop on larger collections of sample data for other sorts of testing.

More comprehensive testing including multiple servers, client user interfaces, and so on, tends to be labor intensive to set up and maintain even with good tool support, so here I suspect most of the answers boil down to time and money.

I'm a big fan of unit & functional testing. One reason for not writing functional tests though is that they are way harder to write than unit tests (assuming that your unit tests are relatively straightforward because your code is well factored)

I always find starting functional tests, especially in a new area of your program, to be a tremendous exercise. Maybe I'm doing it wrong. For example, last week we moved on to writing code that hits an sftp server - so our functional test spins up an sftp server locally & overrides the product code settings so that it connects to that. Getting this set up and working took me a couple of days, and this is such a tiny aspect of the code under test that I'm afraid I'm exhausting my teammates patience for "test first functional tests".

In other areas e.g. hobbyist opengl code, I think this initial barrier to getting started is so high (you'd have to write a sort of machine vision thing to analyse the actual output, rather than just what opengl calls were made) that truly end-to-end tests are a complete nonstarter. You have to give in and settle for integration tests instead.

I suspect many people are in that same place with regard to functional testing of web apps. Yes there are tools like selenium, but there are issues of test data & making sure product code connects to test databases. There are solveable, but people don't see that the value is greater than the costs of wrestling with that.

"a sort of machine vision thing" -- You could use Sikuli or SimpleCV. I'm using OpenCV in my video game playing robot to find elements on the screen. (Of course, it's a bit of a Frankenstein to create and run.) At least it's good to know it's possible, though. But, yes, I can see why others would not want to go through this much work -- it is hard.