100,000 e2e selenium tests? Sounds like a nightmare

Oh the weekly HN unit tests suck/unit tests are design tool discussion.

I feel we may be getting into bike-shedding territory here. When I'm refactoring, as part of the TDD Red/Green/Refactor cycle, I'm changing how code is implemented, usually to improve the design and readability, such that the functionality stays the same e.g. all of my tests pass without touching them. Anything I gain from the refactoring is bonus on top.

Refactoring without introducing bugs is the #1 reason people do TDD vs Test After. But if you don't trust the test suite to catch bugs introduced when refactoring then I'd argue the test suite itself probably isn't that useful and needs work.

On a side note I believe the term "refactoring" has been way overused these days. You can change code without it being "refactoring". When I've seen people use the term "refactoring" (and I've caught myself multiple times on this) they mean rewriting, for in most cases this rewriting means changing the code and changing the tests. If you have to change the tests because of changes to the code, that's not refactoring, that's just changing code.

I'd actually be extremely interested in hearing more about your experiences with HealthCare.gov. I'm working on implementing test infrastructure for my current employer and would love to learn from folks more experienced than myself about the process!

I think the difference between an Airbus Airplane and a Software System is that once Airbus chooses a material that it meets it characteristics it does not go back an change that material. And the material does not change by itself. So the airbus test pyramid works because the bottom is stable.

In my personal experience in software systems is that the bottom is not stable you never end up selecting a material and having component that never changes after you build it. All layers of an application tend to change as the application changes and user needs change. Therefore you can't really say that oh we tested the bottom of the pyramid and know for sure that it works.

Writing end to end tests for application is quite hard work and requires a lot of though to design an application that can tested both at the unit level, the component level, and the system level.

Even though end to end testing is very hard it's value is massive as an industry we should be focused on lowering the cost of end to end testing rather than saying that unit is good enough.

Refactoring code without tests is definitely scary. But those tests don't need to be unit tests.

(caveat: i understand "unit test" to mean a test which tests a single class, or sometimes a very small number of classes, with any collaborators replaced with stubs or mocks)

In my experience, bigger tests - what my colleagues call integration tests, which might involve 3-30 classes and often the database - give me a lot more confidence in my changes.

Tests are only a useful safety net for changes which are entirely confined to the thing they test. If you're changing the way some particular method is implemented, say from looping over a collection to mapping a lambda over it, then the change is confined to that class, and a unit test can help. But if you're changing the method's contract, say from taking a collection of objects to taking one at a time, then the change takes in that class and its clients, and the unit test is useless - it tests that the method does something that it should no longer do! Yes, you can rewrite the unit test, but that does nothing to reassure you that the overall behaviour hasn't changed. You're starting from scratch.

The thing is, in my experience, valuable refactorings tend to take in more than one class. Sometimes much more than one class. I want to be able to make those refactorings with a safety net. Unit tests can't give me that.