Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

nexte.st

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#41

Post by cogman10 »

sunshowers wrote: post #6

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

First, I like how all this works, it's great.

Can you run tests serially in the (horrible) case when tests need to build on one another?

How are you querying for the tests? Is that just built into rust's test stuff?

Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#42

Post by 27183 »

landr0id wrote: post #10
sunshowers wrote: post #6

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

Big fan of nextest and this is my first time seeing this site. I'll be real I feel a bit ridiculous commenting this but you might want to consider rephrasing this: >Treat tests as cattle, not pets. Detect and terminate slow tests. Not sure saying, "hey, treat your tests as an animal you can kill at will" paints the right image.

"Treat tests as annuals not perennials"

"Treat tests as workers not management"

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#43

Post by sunshowers »

cogman10 wrote: post #41
sunshowers wrote: post #6

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

First, I like how all this works, it's great. Can you run tests serially in the (horrible) case when tests need to build on one another? How are you querying for the tests? Is that just built into rust's test stuff? Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the te…

> Can you run tests serially in the (horrible) case when tests need to build on one another?

Yes: https://nexte.st/docs/configuration/test-groups/ (edit: though tests that build on each other is a bit harder — test groups are meant for when tests need access to a shared resource)

> How are you querying for the tests? Is that just built into rust's test stuff?

Just running --list against test binaries.

> Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.

This is possible in principle, but nextest doesn't really inject itself into tests like that (injection can cause reliability issues in practice, and a big focus of nextest is reliability). Forking is also not possible in multithreaded programs.

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#44

Post by sunshowers »

mohsen1 wrote: post #28
sunshowers wrote: post #26

Earlier quoted context omitted.

Thanks! BTW you might enjoy setting CARGO_TERM_COLOR=always in your environment :) dtolnay/rust-toolchain does this automatically but it looks like you aren't using that action.

Thanks! Any pro tips for sharding? I landed on single job because couldn't get cache to work properly for shards to be fast enough to worth it

I think in practice you're right that GitHub Actions' caching slowness can end up dominating performance. I don't have any great tips other than to measure.

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#45

Post by sunshowers »

jstrong wrote: post #27

seems like `cargo nextest run` just runs `--lib` tests by default? however, `cargo test` is not so slow if you do `cargo test --lib`. how do I get nextest to execute the doc tests, too?

The benchmarks are against non-doctest cargo test runs. (But you should measure against your own project, of course!) Unfortunately Cargo doesn't provide enough information for nextest to run doctests reliably with.

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#46

Post by evntdrvn »

sunshowers wrote: post #44
mohsen1 wrote: post #28

Earlier quoted context omitted.

Thanks! Any pro tips for sharding? I landed on single job because couldn't get cache to work properly for shards to be fast enough to worth it

I think in practice you're right that GitHub Actions' caching slowness can end up dominating performance. I don't have any great tips other than to measure.

I've basically abandoned GA's caching for most things because measurements ended up showing things were slower with caching than without :woozy:

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#47

Post by dpc_01234 »

sunshowers wrote: post #6

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

Thank you! It's great.

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#48

Post by sunshowers »

coding-wizard wrote: post #34
sunshowers wrote: post #6

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

Hey, I love nextest. But, perhaps because of the one-process per test approach, endpoint security solutions like CrowdStrike Falcon or Palo Alto Cortex tend to make computers hang whenever tests kick-in. I would love if you were able to introduce a workaround, because none of those companies will fix their stuff. I am guessing a possible mitigation would be to have stagger the first invocation of any large test binar…

Nextest does run tests in lexicographic (binary, test name) order by default, so first executions tend to be staggered. Unfortunately this is a known problem that is much larger than nextest, and it is why Windows introduced Dev Drive with a mode to disable AV/EDR on those drives entirely.

I wish I had a better answer here — it is frustrating how poorly behaved the EDR stuff tends to be. Maybe I should try and network a bit to get in touch with the people working on this stuff. (Planning to be at DEF CON this year, so come find me if you work on this stuff and will be there!)

Re: Cargo-nextest: 3x faster than cargo test, per-test isolation, first-class CI

#50

Post by moi2388 »

landr0id wrote: post #10
sunshowers wrote: post #6

Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.

Big fan of nextest and this is my first time seeing this site. I'll be real I feel a bit ridiculous commenting this but you might want to consider rephrasing this: >Treat tests as cattle, not pets. Detect and terminate slow tests. Not sure saying, "hey, treat your tests as an animal you can kill at will" paints the right image.

Jesus Christ do we seriously need to limit our speech and get rid of perfectly good analogies for every special little snowflake?
Post reply on HN