Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I wish I knew when I became CTO

medium.com

Post reply on HN
Print view View original

171–180 of 258 posts

Re: What I wish I knew when I became CTO

#171

Post by crdoconnor »

forgotpw1123 wrote: post #146
latch wrote: post #65

> I’ve found it a real struggle to get our team to adopt writing tests. If you're struggling to judge the engineering culture of a company that you're considering joining, consider this indicative of a poor one. It isn't definitive, but it's something you should ask about and probe further. Ask to see their CI dashboard and PR comments over the last few days. When they talk about Agile, ask what _engineering_ techniq…

PR comments I agree with, but after believing in unit tests for years I'm drifting slowly into the "waste of time" camp. I'm convinced that unit tests don't usually find bugs. IMO, most bugs are edge cases that were an oversight in the design. If the dev didn't handle the case in code they're not going to know to test for it. Fuzzing is a much better approach. At my current position I have the opportunity to work wit…

I think unit tests will die one day, and that day is probably not too far away.

These days I follow three "good practice" rules, all of which are violated when you follow common unit testing practise:

* Only put tests on the edge of a project. If you feel like you need lower level test than that then either a) you don't or b) architecturally, you need to break that code off into a different project.

* Test as realistically as possible. That means if your app uses a database, test against the real database - same version as in prod, data that's as close to prod as possible. Where speed conflicts with realism, bias toward realism.

* Isolate and mock everything in your tests that has caused test indeterminism - e.g. date/time, calls to external servers that go down, etc.

Re: What I wish I knew when I became CTO

#172

Post by thejosh »

merb wrote: post #157
fergie wrote: post #150

Earlier quoted context omitted.

To be fair, testing is to some extent an unsolved problem. The joys of testing were being extolled long before test frameworks were actually usable. Now that they are, and you can glue Travis, GitHub and the test lib of your choice together pretty easily you have solved about 30% of what needs to be tested. If, say, you are developing an Office add-in on a Mac, and you want to test it on Word 2013 on Windows 7, there…

well it happens a lot of techniques are still not understood. I mean we develop database heavy code. Should we never test the code running with the database? would be a poor choice since we would loose a lot of coverage. What we did instead were transactional tests. Which means that in PostgreSQL sense that we actually use SAVEPOINTS to actually wrap our tests inside a savepoint and than rollback to the sane state an…

Our tests take about 10 seconds to run (mainly the tewts which need to test a lot of endpoints, our domain is around 1.4s), with the compile being the slow part, which brings CI to around 1min30s-1min50s on average.

We use elixir, so we get nice features like ecto sandbox with Async tests out of the box.

Re: What I wish I knew when I became CTO

#173

Post by njs12345 »

ereyes01 wrote: post #99
Rapzid wrote: post #73

Earlier quoted context omitted.

How many tests does a PR need? One? Five? When would you not write tests for something(that something not necessarily being a PR, but maybe a unit or feature)? Tests, like any process, should be serving you and your goals.. You shouldn't be serving your processes or testing practices. This sort of un-nuanced thinking isn't indicative of a high performing startup or CTO IMHO. Perhaps your policies are not directly ind…

I guess it might sometimes be fine to be a relativist and write off the need for tests as a result of "nuanced thinking," but I think you have to accept that you are running a risk by shipping untested code into your product. As others have said, line coverage is a misleading metric. Ideally, your tests would fully cover all _program states_, and even 100% line coverage doesn't guarantee full state coverage. If you h…

If you have any concurrency in your system then you aren't going to cover all the states using unit tests. You'll need some sort of formal model for that.

Re: What I wish I knew when I became CTO

#174

Post by icebraining »

yread wrote: post #162
apocalyptic0n3 wrote: post #23

Earlier quoted context omitted.

I'm not a CTO but I do lead the dev team at our agency (was previously 16 devs, but we've slimmed down to 7 currently). I want to preface this by saying that at an agency, your biggest enemy is always time; sales teams have to sell projects for the absolute minimum in order to get a contract, so you can't waste time on non-essentials for most projects. That said, the biggest resistance I have found is "this feature i…

I always find code coverage such a useless metric: if you have two independent ifs next to each other and one test goes in one if and another test in the other you have 100% coverage. Congratulations. But you've never tested what happens when you go in both

That's just the basic coverage metrics, there's more than that: https://en.wikipedia.org/wiki/Code_coverage

Re: What I wish I knew when I became CTO

#175

Post by tejasmanohar »

wolco wrote: post #107
latch wrote: post #92

Earlier quoted context omitted.

Tests don't cost time. Not true 100% of the time, but it's the right "default" mindset, because it's true the majority of the time.

From a pure number of keystrokes tests can add time. If you know exactly what to write because you have done this 100s of times before tdd will slow you down. If you are unsure of what the outcome of what you write tdd will give you training wheels and help guide you. That may make some quicker for a little while.

>If you know exactly what to write because you have done this 100s of times before tdd will slow you down.

I once had this attitude. Then, I worked with other people. It makes all the difference. My perspective shifted when I was bitten by something small when making a small patch to a foreign system because someone else didn't leave good test cases behind.

Re: What I wish I knew when I became CTO

#176

Post by arekkas »

forgotpw1123 wrote: post #146
latch wrote: post #65

> I’ve found it a real struggle to get our team to adopt writing tests. If you're struggling to judge the engineering culture of a company that you're considering joining, consider this indicative of a poor one. It isn't definitive, but it's something you should ask about and probe further. Ask to see their CI dashboard and PR comments over the last few days. When they talk about Agile, ask what _engineering_ techniq…

PR comments I agree with, but after believing in unit tests for years I'm drifting slowly into the "waste of time" camp. I'm convinced that unit tests don't usually find bugs. IMO, most bugs are edge cases that were an oversight in the design. If the dev didn't handle the case in code they're not going to know to test for it. Fuzzing is a much better approach. At my current position I have the opportunity to work wit…

Let's break this down.

> I'm convinced that unit tests don't usually find bugs.

They don't, they test whether or not the API contract the developer had in mind is still valid or not.

> IMO, most bugs are edge cases that were an oversight in the design. If the dev didn't handle the case in code they're not going to know to test for it.

You don't write test to find bugs (in 98% of cases), but you can write tests for bugs found.

> Fuzzing is a much better approach.

If you're writing an I/O intense thing, such as a JSON parser, then yes. For 80% which is CRUD, probably not.

> The project with high code coverage is basically shit and has so many bugs that we regularly cull anything marked less than "medium" severity as "not worth fixing". This project was written by a team that loves "patterns", so you can find all sorts of gems like Aspect Based Programming, CQRS, N-tier, etc. well mixed into a featureless grey Java EE goo. We get so many bug reports that its someones job to go through them.

You are blaming tests for bad design choices. With the patterns raised unit tests only get you so far, integration tests are what help you prevent bad deployments.

> The other project with no tests is a dream to work on. Not a single file over a few hundreds lines, everything linted and well documented. Almost no methods that don't fit on the screen, no recursion. No bullshit "layering" or "patterns". I can't remember the last time we had a bug report, since our monitoring picks up any exception client and server side. Every bug I've worked on was identified by our monitoring and fixed before anyone noticed.

So how many exceptions were raised due to bad deploys? Core review only gets you so far.

> If your developers are great then tests would hardly fail and be fairly useless, and if they're terrible tests don't save you.

Failing tests don't have to do with devs being "great" or not. Developers must have the capability of quickly testing the system without manual work, in order to be more effective and ship new features faster. If the tests are one-sided (only unit tests, only integration tests), then this will get you only so far, but it still get's you that far.

Don't abandon good development practices only because you saw a terrible Java EE application.

Re: What I wish I knew when I became CTO

#177

Post by chatmasta »

siliconc0w wrote: post #97

So hiring is pretty hard but I kinda disagree with most of the points there.. * only hire when desperate Strong talent is so hard to get you should probably always be hiring. If you're hiring too many people your bar is probably too low. * only hire to keep up with growth You need to be at least a little preemptive. The hiring process itself can take months, plus the time to train even good new hires is at least a fe…

> Don’t hire someone to do something you’ve not yet figured out

I think this is not an indictment of hiring for something you do not know how to do, so much as it is of hiring someone before you have a defined job for them to do.

When you’re hiring an engineer, presumably you’ll be placing them onto a team that is responsible for some well-defined part of the stack. So you should know what skills you’re looking for when you’re interviewing. This should make interviewing easier; if you know what capabilities you need a new hire to have, then you know exactly what to test for in new candidates.

(This is yet another reason why generic whiteboard interviews make no sense. They’re optimizing for solving problems that could be wholly unrelated to the problems your company faces on a daily basis. I’m surprised more companies do not give interviews that focus more specifically on their relevant problem domains.)

If you don’t know what the new hire is going to do when he or she starts work, then you have no idea what skills to measure in the interview, and end up settling for the “least common denominator” of whiteboard coding ability.

Re: What I wish I knew when I became CTO

#178

Post by adrianN »

yread wrote: post #162
apocalyptic0n3 wrote: post #23

Earlier quoted context omitted.

I'm not a CTO but I do lead the dev team at our agency (was previously 16 devs, but we've slimmed down to 7 currently). I want to preface this by saying that at an agency, your biggest enemy is always time; sales teams have to sell projects for the absolute minimum in order to get a contract, so you can't waste time on non-essentials for most projects. That said, the biggest resistance I have found is "this feature i…

I always find code coverage such a useless metric: if you have two independent ifs next to each other and one test goes in one if and another test in the other you have 100% coverage. Congratulations. But you've never tested what happens when you go in both

There are different kinds of coverage metrics. If line coverage is not enough for your liking you can always go for full path coverage. You'd have to write an exponential number of tests though.

Re: What I wish I knew when I became CTO

#179

Post by mnm1 »

latch wrote: post #65

> I’ve found it a real struggle to get our team to adopt writing tests. If you're struggling to judge the engineering culture of a company that you're considering joining, consider this indicative of a poor one. It isn't definitive, but it's something you should ask about and probe further. Ask to see their CI dashboard and PR comments over the last few days. When they talk about Agile, ask what _engineering_ techniq…

Tests or unit tests? When people refer to writing tests, they usually mean unit tests. In the last four years, our unit tests have caught maybe one or two bugs. The time and resources spent on fixing those bugs after they were in production would have been a fraction of the time and resources spent on writing unit tests. Unit tests simply don't catch bugs and spending time on them is time wasted. Judging a company as having a bad engineering culture because they don't do pointless, unnecessary, and superfluous work that doesn't benefit them seems to be more a reflection of your ideas than the company itself. I'd say that reflects quite well on the company and its engineering culture. If you're talking about other automated tests, they may or may not make sense depending on your team size and product.

Re: What I wish I knew when I became CTO

#180

Post by crdoconnor »

latch wrote: post #151
forgotpw1123 wrote: post #146

Earlier quoted context omitted.

PR comments I agree with, but after believing in unit tests for years I'm drifting slowly into the "waste of time" camp. I'm convinced that unit tests don't usually find bugs. IMO, most bugs are edge cases that were an oversight in the design. If the dev didn't handle the case in code they're not going to know to test for it. Fuzzing is a much better approach. At my current position I have the opportunity to work wit…

Thanks. I also have a history with tests and I continue to struggle to find the right balance not just between coverage, but also unit vs integration (and within unit, between behavioural and implementation). I think this uncertainty based on experience is in a whole other class than "I can't get my employees to write tests." Two quick points. 1 - I've added fuzz testing to my arsenal and find it a good value, especi…

>That said, testing is primarily a design tool (aka, identifying tight coupling). The more you do it, the more you learn from it, the less value you get from it because you inherently start to program better

Unit tests "identify" tight coupling because they are themselves a form of tight coupling.

Post reply on HN