Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superior Testing: Stop Stopping

arturdryomov.online

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Superior Testing: Stop Stopping

#11

Post by tokyodude »

lordnacho wrote: post #8

Once you have something nearing a steady state design, you need to have tests. I can understand if something is in a very early stage it might not have tests yet. At my current work we have loads of automatic tests. Every time something is changed, there's pipelines that test the following: - Simple unit tests: Start with state 0, make and action, is state 1 what you expect? There's literally hundreds of these, and t…

I am not convinced this is always true. The first 25 years of my career was spent making games. Never had tests. We had testers but no tests. I then worked at a big company with tests. My first experience with them. Working on something important I loved them but they easily cut my velocity by 70% compared to what I used to get when working on games. The games were AAA and shipped to millions of customers on CDs and Cartridges so no bug fixes allowed after the fact. Being a big company and big project there were 10-30 engineers who's maintained testing infrastructure, continuous integration, etc. That's larger than the entire engineering team on most games I worked on.

I'm not arguing against tests and having had the experience of testing I'd use them when appropriate I'm just not sure the exact cutoff. If I was working on a game engine for multiple teams I'd be writing tests for sure. If I was working on a small 5-20 person team with custom tech I'm not sure I'd start adding tests where I didn't before. Maybe if there was IAP or multi-player online or some other server component and metrics. Or maybe if it was easier to get started and maintained.

Theres a bunch of big differences with games vs much other software. Often games are not maintained after shipping. Of course that's less true today than it was in the past with longer term online games but it's still also true that many games area pretty much done the moment they ship. Another big difference is the teams are 70-90% non-engineers making tons and tons of data.

I mostly bring this up as that often software engineers talk past each other since no all software engineering is the same.

https://www.joelonsoftware.com/2002/05/06/five-worlds/

Re: Superior Testing: Stop Stopping

#12

Post by airfreak »

I like the concept of error budgets. Start off by knowing what kind of quality and resiliency a system requires and design your test strategy around that. Means talking to the client about that.

I'm not going to invest a load of time in various types of automated test for an internal site with a form over a database that 2 users use for low priority work. The idea of 80%-100% code coverage for basic work like that seems like waste to me.

But for critical path of the eCommerce shopping experience I'm going to going to write all kinds of automated tests at multiple layers of the stack, right up to chaos/stress testing it, so that we know when black friday comes we can handle it.

I don't like dogma and TDD seems too dogmatic for me. I am very pro testing, having been both a QA, Developer and Ops engineer. I want the freedom to exercise my own expert judgement. The problem with dogma is that it makes Thinking take a back seat. Suddenly we have 80% code coverage enforced on a page that loads a grid from a table, going through a three layered monstrosity of code.

Re: Superior Testing: Stop Stopping

#13

Post by millstone »

kstenerud wrote: post #6

Regardless of all the arguments for and against tests, it's important to remember your purpose as an IT professional: Your job is to solve the customer's problem. That's it. HOW you solve their problems is entirely up to you, and some solutions will work better than others. The only thing about customers guaranteed to be consistent is that they will request changes. Some will be good changes. Some will be bad changes…

There's a case for ugly entangled tests too. If you, say, isolate your tests from the filesystem (mocks, etc), your tests can become fast, hermetic, etc. But they lose all sensitivity to differences between filesystems, and encode any assumptions about how filesystems behave.

I like to have tests that exercise the real stuff. The tests are flakier but have better fidelity to the actual use.

Re: Superior Testing: Stop Stopping

#14

Post by millstone »

How do people here test their UIs? As an extreme example, the link here contains hundreds of lines of declarative code; a single typo (unclosed bracket, misspelled CSS property...) could break the site. So how is it tested?

Probably it's not tested in any automated way, because manual testing (open the site and look) is much easier. Maybe lots of software is like that, to various degrees.

Re: Superior Testing: Stop Stopping

#15

Post by 2T1Qka0rEiPr »

millstone wrote: post #14

How do people here test their UIs? As an extreme example, the link here contains hundreds of lines of declarative code; a single typo (unclosed bracket, misspelled CSS property...) could break the site. So how is it tested? Probably it's not tested in any automated way, because manual testing (open the site and look) is much easier. Maybe lots of software is like that, to various degrees.

You can implement a combination of end-to-end browser driven tests as well as applying visual regression analysis etc. I wrote something which would actually parse two sites and return the differences in the DOM/CSSOM many years ago, but it worked nicely :)

Re: Superior Testing: Stop Stopping

#16

Post by pmontra »

> Customers ask providers for a quality product

It's not even the "quality" thing. It's that tests help at delivering the product.

Of course if they tell me, "I need this tomorrow" because of some emergency there is hardly enough time to code for the happy path and manually check that it looks OK. No tests. I make sure they understand it will be full of bugs and we'll fix them later on. This happens rarely but it happens.

Re: Superior Testing: Stop Stopping

#17

Post by leethargo »

millstone wrote: post #14

How do people here test their UIs? As an extreme example, the link here contains hundreds of lines of declarative code; a single typo (unclosed bracket, misspelled CSS property...) could break the site. So how is it tested? Probably it's not tested in any automated way, because manual testing (open the site and look) is much easier. Maybe lots of software is like that, to various degrees.

I always thought that websites are tested with Selenium and similar (https://www.seleniumhq.org/).

For other kind of GUIs, you can also emulate mouse + keyboard interaction with tools like Sikuli (http://www.sikulix.com/). This is also scriptable, e.g. in Python, but actually quite painful in my experience: When you GUI needs significant time for loading and rendering, the Sikuli program might fail to find a button that did not appear yet. So you need to play with timeouts and trial-and-error.

Re: Superior Testing: Stop Stopping

#19

Post by djd20 »

tokyodude wrote: post #11
lordnacho wrote: post #8

Once you have something nearing a steady state design, you need to have tests. I can understand if something is in a very early stage it might not have tests yet. At my current work we have loads of automatic tests. Every time something is changed, there's pipelines that test the following: - Simple unit tests: Start with state 0, make and action, is state 1 what you expect? There's literally hundreds of these, and t…

I am not convinced this is always true. The first 25 years of my career was spent making games. Never had tests. We had testers but no tests. I then worked at a big company with tests. My first experience with them. Working on something important I loved them but they easily cut my velocity by 70% compared to what I used to get when working on games. The games were AAA and shipped to millions of customers on CDs and…

I think the key distinction there is - you'd ship the games, and never change them again. What the tests do is let you continually evolve the software over a long period of time. What you need tests for is to give you confidence that chaning system a doesn't impact system b negatively, allowing the software to continue evolving at a much faster pace once you have a good test suite in place.

Re: Superior Testing: Stop Stopping

#20

Post by heavenlyblue »

tokyodude wrote: post #11
lordnacho wrote: post #8

Once you have something nearing a steady state design, you need to have tests. I can understand if something is in a very early stage it might not have tests yet. At my current work we have loads of automatic tests. Every time something is changed, there's pipelines that test the following: - Simple unit tests: Start with state 0, make and action, is state 1 what you expect? There's literally hundreds of these, and t…

I am not convinced this is always true. The first 25 years of my career was spent making games. Never had tests. We had testers but no tests. I then worked at a big company with tests. My first experience with them. Working on something important I loved them but they easily cut my velocity by 70% compared to what I used to get when working on games. The games were AAA and shipped to millions of customers on CDs and…

Well, I mainly work in a team where developers ship separate products for the internal usage. So it's not really working in a team. Products are generally quite technical - it's not your usual CRUD shop.

I do write tests, but only post-factum, when I know for sure this component will be re-used in other components. I write them in cases where I know I will forget about the certain edge-cases in some time, so changing the code will most certainly introduce bugs.

When you have components that are reused over the course of several years (and obviously being optimised if the requirement is there), the chance of regressions is severely increased.

So I see that "writing tests for everything" is more of a political stance, rather than entirely practical.

Post reply on HN