Earlier quoted context omitted.
If you're not programming in Rust and Julia, are you really programming? /s
I thought HN crowd was more into Nim or Zig?
Nah, it's all about MUMPS.
Our Rails test suite runs in 1 minute on Buildkite
11–20 of 29 posts
Re: Our Rails test suite runs in 1 minute on Buildkite
#12One minute? That's terrible. What kinds of tests and how many are you running? > We never run all of our application’s tests in local development. It’s not a good use of time and will never be as fast as running them on CI. WTF this is bad practice. Developers should always get in the habit of running tests locally first. No test should require running in the cloud. And, even more, it's actually rare that CI is more…
How can you tell that if you don't know the complexity of their projects?
This is because RoR projects do not do unit testing, they do integration testing with a real DB.
This approach is terrible and causes the poor performance.
There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API.
If you're writing unit tests, write unit tests with mocked dependencies.
Re: Our Rails test suite runs in 1 minute on Buildkite
#13Earlier quoted context omitted.
How can you tell that if you don't know the complexity of their projects?
This is because RoR projects do not do unit testing, they do integration testing with a real DB. This approach is terrible and causes the poor performance. There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API. If you're writing unit tests, write unit tests with mocked dependencies.
It's hard to really test business logic without hitting the database or other components, no matter the language and the framework. You really have to trust your mocks and sometimes you discover only in production that one of them didn't do what it should do. It happened to me.
The teams I work with tend to write unit tests to demonstrate that a class or module or whatever work, then integration tests to demonstrate that that part of the application works from the frontend code down to the database.
Re: Our Rails test suite runs in 1 minute on Buildkite
#14Earlier quoted context omitted.
How can you tell that if you don't know the complexity of their projects?
This is because RoR projects do not do unit testing, they do integration testing with a real DB. This approach is terrible and causes the poor performance. There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API. If you're writing unit tests, write unit tests with mocked dependencies.
> This is because RoR projects do not do unit testing, they do integration testing with a real DB.
This is not correct.
Rails projects use unit testing, integration testing, and functional testing, as appropriate for the case.
Re: Our Rails test suite runs in 1 minute on Buildkite
#15Would be nice if they shared how many tests they have :)
In the grand scheme of things it doesn't matter that much. Once you starts parallelizing, CI time is: setup_time + (test_run_time / workers), so assuming money isn't a problem you can add more workers as you keep adding more tests. What really matter is how fast you can setup your test workers and how slow individual tests are.
I think your last statement captures something that not often emphasized, which is that startup time of workers and app are variable costs which scale with the amount of parallelization and so if you have a very large test suite there may be a limit where setup eats up so much time that parallelizing more is going to be pretty "wasteful" and I guess that's where the money comes in.
Secondly if you're actually trying to startup 100+ test workers per build and so on there's going to be some time distribution for how long it takes for each worker to startup and that adds a bit more time for all workers to complete. This distribution probably isn't _that_ wide timewise but if you really start to push your test suite runtime down it may pop up. If you're running things in docker sometimes a node doesn't have the image in it's docker cache...
Unsure if CI services like buildkite have really made this that much faster but it seems like they are using a single box with 64 cores.
Re: Our Rails test suite runs in 1 minute on Buildkite
#16Earlier quoted context omitted.
How can you tell that if you don't know the complexity of their projects?
This is because RoR projects do not do unit testing, they do integration testing with a real DB. This approach is terrible and causes the poor performance. There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API. If you're writing unit tests, write unit tests with mocked dependencies.
> This is because RoR projects do not do unit testing, they do integration testing with a real DB.
Lots of RoR projects do unit testing without the DB involved, I did it for ~10years and you can write small unit tests until your eyes bleed, but eventually you'll want to test the controllers.
> There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API.
You can just use the built in minitest tools and some fixtures to hit the API, and even back it with sqlite so it's fast and light. Most people don't because thing like Rspec, FaktoryBot, and Cuprite are convenient and nice to work with if a bit slow.
>If you're writing unit tests, write unit tests with mocked dependencies.
There were like 3 talks at Rails Conf about this, people do it all the time in Rails land.
Re: Our Rails test suite runs in 1 minute on Buildkite
#17Nice simple article describing a technique to parallelize tests and a quick fix to an issue blocking it.
It is interesting to observe the time that things take in various environments. I worked in ad tech previously with lots of offline big-data jobs. Test completion took a few minutes even fully parallelized. Now I work in HFT and our test suite completes on an M1 Max in 21 s and it's got higher coverage.
The situations aren't really comparable. But it's interesting to see the different baselines.
Re: Our Rails test suite runs in 1 minute on Buildkite
#18Re: Our Rails test suite runs in 1 minute on Buildkite
#19Earlier quoted context omitted.
How can you tell that if you don't know the complexity of their projects?
This is because RoR projects do not do unit testing, they do integration testing with a real DB. This approach is terrible and causes the poor performance. There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API. If you're writing unit tests, write unit tests with mocked dependencies.
If you write all yours tests without spinning up once a database, I'm not vouching for the effectiveness of said tests...
Especially that we're talking about a web framework, the db is a core part of your stack.
Yes there's going to be strictly unit tests but then at some point there are also queries and models.