Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

github.com

Post reply on HN
Print view View original

1–10 of 18 posts

Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#1

Post by _cfl0 »

Hi, we are working on a tool for speeding up test runs, by skipping tests unaffected by code changes.

Effectivly, Saving 80-95% of the time, by skipping 80-95% of tests.

We started a few months ago, and have managed to get into a few production CI systems. All our prospects and users are on holiday right now. So we decided to repackage and open-source for local test running. available here (https://github.com/nabaz-io/nabaz) under MIT license.

One line change: pytest -v -> nabaz test --cmdline "pytest -v"

Stalk us on GitHub, or just Star us. Ask questions, we'll answer in under 30 seconds. we have auto refresh on.

Show HN: We built a tool for fast-forwarding 95% of tests (MIT)
github.com

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#3

Post by _cfl0 »

How it works: First time you run nabaz:

1. Collects code coverage for every tests that runs, seperatly.

2. Saves code coverage in local sqlite db, along with commit ID as key.

Second to n runs later:

1. Traverse git tree and queries db via commit ID -> per test code coverage

2.a git diff between new code (code change) and the code coverage from the old commit, test case by test case.

2.b. If an intersection between new code and old code coverage is found, test is rerun. else, it will be skipped.

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#5

Post by _cfl0 »

metadat wrote: post #2

Cool idea.. Has this strategy already been tried before?

Meta and Google employ AI based methods for skipping tests (See):

they are pretty much flawed in a way, because they require you to wait weeks/months to gain statistical sagnificance. And even then they are too costly to train to make sense in smaller scales.

There's an ex-Google employee who came out with something similar in 2005 for JUnit java tests, since erased it was called google-testar.

We talked to him and ulitmatly he sunsetted the project because:

1) static analysis back then was basically copying language parser code from the Language distro and retro fitting it which was HARD to maintain.

2) Misses when resource files were changed, he was a one man show to my understanding.

Our View: Static analysis has gone a long way since... and yet no one has made something for system tests and integration tests, there's a couple unit tests solution out there.

See: https://static.googleusercontent.com/media/research.google.c...

https://research.facebook.com/publications/predictive-test-s...

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#6

Post by _cfl0 »

maxilevi wrote: post #4

How do you deal with tests that interact with something external to the codebase? The result of those could potentially change without a code change.

That's a great point!

Depends on the what you define by "externality".

I like to look at it this way, what's being tested? I see three groups:

  - testing the software you wrote.
  - testing integration with other software you wrote (microservice interaction maybe?)
  - testing integration to third party software.
I'm telling you our current plan, feel free to poke holes:

In the first case the only externalities that seem to really matter are mostly resource and configuration files. We could detect I/O with such files and rerun the test if the underlying resource/config changed.

In the second case were planning on collecting coverage for all services being tested and then employ a more sophisticated diff heurastic.

For the third case mark those tests as always run, hopefully you are testing your codebase and not just thirdparty integrations.

We have a poc of a syscall tracer that gives us an I/O map of the test being run or the software being tested. this includes reading/writing to files, pipes, network etc...

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#7

Post by metadat »

_cfl0 wrote: post #6
maxilevi wrote: post #4

How do you deal with tests that interact with something external to the codebase? The result of those could potentially change without a code change.

That's a great point! Depends on the what you define by "externality". I like to look at it this way, what's being tested? I see three groups: - testing the software you wrote. - testing integration with other software you wrote (microservice interaction maybe?) - testing integration to third party software. I'm telling you our current plan, feel free to poke holes: In the first case the only externalities that seem…

This sounds like the best you could possibly do!

What are the odds this baby will be made compatible with Java?

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#8

Post by _cfl0 »

metadat wrote: post #7
_cfl0 wrote: post #6

Earlier quoted context omitted.

That's a great point! Depends on the what you define by "externality". I like to look at it this way, what's being tested? I see three groups: - testing the software you wrote. - testing integration with other software you wrote (microservice interaction maybe?) - testing integration to third party software. I'm telling you our current plan, feel free to poke holes: In the first case the only externalities that seem…

This sounds like the best you could possibly do! What are the odds this baby will be made compatible with Java?

java is scheduled next, would absolutely love to hear about your use-case?

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#9

Post by metadat »

_cfl0 wrote: post #8
metadat wrote: post #7

Earlier quoted context omitted.

This sounds like the best you could possibly do! What are the odds this baby will be made compatible with Java?

java is scheduled next, would absolutely love to hear about your use-case?

I have a pretty large Java code base (40+ micro services) and the integration test suite takes hours, many hours, to run.

However they are usually testing components from end-to-end. This seems not very amenable to benefitting this approach, since "side effects" are an inherent part of nearly every integration test.

Re: Show HN: We built a tool for fast-forwarding 95% of tests (MIT)

#10

Post by _cfl0 »

metadat wrote: post #9
_cfl0 wrote: post #8

Earlier quoted context omitted.

java is scheduled next, would absolutely love to hear about your use-case?

I have a pretty large Java code base (40+ micro services) and the integration test suite takes hours, many hours, to run. However they are usually testing components from end-to-end. This seems not very amenable to benefitting this approach, since "side effects" are an inherent part of nearly every integration test.

Wow! Seems like a challenge to test, would love to see what we can do.

Regarding side effects, completely fine as long as they're mapped out. Would love to hear more, nothing pitchy or salesy.

Post reply on HN