Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automating Git Bisect with Ephemeral Environments

qckfx.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Automating Git Bisect with Ephemeral Environments

#11

Post by chw9e »

cortesoft wrote: post #5

The tricky thing with git bisect run is that you usually have to have your test script not be part of your git checkout for the process to work. This is because of the particular situation that bisect is valuable in; you find a regression that doesn’t have a test, and you aren’t sure when it was committed. First, it has to be a regression; if it is just a bug, then there was no previous version that didn’t have the b…

Another good case is for rolling back a single bad commit from a batch that got merged into main at the same time.

Doing batch merges with a merge queue can speed up things if you have a ton of longer running end to end and integration tests. But then if a test fails you need to identify which commit out of the batch is causing it so you don’t reject the entire batch.

Re: Automating Git Bisect with Ephemeral Environments

#13

Post by arccy »

cortesoft wrote: post #5

The tricky thing with git bisect run is that you usually have to have your test script not be part of your git checkout for the process to work. This is because of the particular situation that bisect is valuable in; you find a regression that doesn’t have a test, and you aren’t sure when it was committed. First, it has to be a regression; if it is just a bug, then there was no previous version that didn’t have the b…

git bisect run can take a script, you can easily script adding a test case and running the test, either to an existing file or a new file.

Re: Automating Git Bisect with Ephemeral Environments

#14

Post by hughdbrown »

jakub_g wrote: post #8
cortesoft wrote: post #5

The tricky thing with git bisect run is that you usually have to have your test script not be part of your git checkout for the process to work. This is because of the particular situation that bisect is valuable in; you find a regression that doesn’t have a test, and you aren’t sure when it was committed. First, it has to be a regression; if it is just a bug, then there was no previous version that didn’t have the b…

As long as your test suite just finds and runs all files in a given folder (without needing to explicitly "enable" them in some index file), this should work: - create a NEW test file in `some/path/to/test.ext` (and back it up outside repo just in case) - do NOT commit it in the repo - `git bisect` That way, bisect would check out different commits, but without touching `some/path/to/test.ext` because it's not tracke…

This is exactly what I do: I put my test file outside the git repo's tree. This does not introduce any complications to testing and is minimally annoying.

Re: Automating Git Bisect with Ephemeral Environments

#15

Post by nlunbeck »

> Because ephemeral environments are reproducible on demand (via Docker images, Kubernetes pods, or a cloud VM), you can guarantee that each bisect step sees the same conditions. This drastically reduces "works on my machine" fiascos.

Agree on this pattern for all code changes. Hard to understate the amount of time we've saved by testing against the full prod-like environment right away. An ephemeral env implementation makes this easy and low stakes, so diving right into E2E testing a copy of your real infra isn't wildly unreasonable. However, I work for Shipyard (https://shipyard.build) so I'm a bit biased on these processes.

Re: Automating Git Bisect with Ephemeral Environments

#16

Post by LegionMammal978 »

andmarios wrote: post #6
LegionMammal978 wrote: post #4

I've always wished for a "git trisect", or a "git n -sect" in general, that can try multiple commits in parallel. The use case would be for testing changes in software that has a long, single-threaded component in the build process (e.g., a heavily overloaded configure script). For long-running projects where each bisect takes over a dozen steps, those components lead to lots of thumb-twiddling.

The magic of bisect is that you rule out half of your remaining commits every time you run it. So even if you have 1000 commits, it takes at most 10 runs. An n-bisect wouldn't be that much faster, it could be slower because you will not always be able to rule out half your commits.

The idea is, suppose I did a trisect, splitting the range [start,end) into [start,A), [A,B), and [B,end). At each step, I test commits A and B in parallel. If both A and B are bad, I continue with [start,A). If A is good and B is bad, I continue with [A,B). If both A and B are good, I continue with [B,end).

This lets me rule out two thirds of the commits, in the same time that an ordinary bisect would have ruled out half. (I'm assuming that the tests don't benefit from having additional cores available.) In general, for an n-sect, you'd test n - 1 commits in parallel, and divide the number of remaining commits by n each time.

Re: Automating Git Bisect with Ephemeral Environments

#17

Post by LegionMammal978 »

flir wrote: post #7
LegionMammal978 wrote: post #4

I've always wished for a "git trisect", or a "git n -sect" in general, that can try multiple commits in parallel. The use case would be for testing changes in software that has a long, single-threaded component in the build process (e.g., a heavily overloaded configure script). For long-running projects where each bisect takes over a dozen steps, those components lead to lots of thumb-twiddling.

I think you'd have to hack it together on a per-project basis, but could you do it with containers? You'd have to identify the point where the build process diverges, make n copies of the container... (If I'm understanding you correctly). But there's not much that's faster than a binary search.

The problem isn't actually running the builds, so much as selecting the commits. Ordinary bisect gives you one commit to test at each step, and one result to report. But I want to be given n - 1 commits at each step, evenly spread out along the search range, and I want to report all n - 1 results at once, to cut the range by a factor of 1/n. If the testing process doesn't utilize all available cores, this will be faster than just testing commits one at a time.

Re: Automating Git Bisect with Ephemeral Environments

#18

Post by agumonkey »

LegionMammal978 wrote: post #16
andmarios wrote: post #6

Earlier quoted context omitted.

The magic of bisect is that you rule out half of your remaining commits every time you run it. So even if you have 1000 commits, it takes at most 10 runs. An n-bisect wouldn't be that much faster, it could be slower because you will not always be able to rule out half your commits.

The idea is, suppose I did a trisect, splitting the range [start,end) into [start,A), [A,B), and [B,end). At each step, I test commits A and B in parallel. If both A and B are bad, I continue with [start,A). If A is good and B is bad, I continue with [A,B). If both A and B are good, I continue with [B,end). This lets me rule out two thirds of the commits, in the same time that an ordinary bisect would have ruled out…

have you ever seen this implemented somewhere ? interesting idea

Re: Automating Git Bisect with Ephemeral Environments

#19

Post by rcxdude »

andmarios wrote: post #6
LegionMammal978 wrote: post #4

I've always wished for a "git trisect", or a "git n -sect" in general, that can try multiple commits in parallel. The use case would be for testing changes in software that has a long, single-threaded component in the build process (e.g., a heavily overloaded configure script). For long-running projects where each bisect takes over a dozen steps, those components lead to lots of thumb-twiddling.

The magic of bisect is that you rule out half of your remaining commits every time you run it. So even if you have 1000 commits, it takes at most 10 runs. An n-bisect wouldn't be that much faster, it could be slower because you will not always be able to rule out half your commits.

Yes, you'd need 4x parallelism for a 2x speedup (16x for 4x, etc). But there's plenty of situations where that would be practical and worthwhile (think a build and test cycle that takes ~1 hour each and can't be meaningfully parallelised further).

Re: Automating Git Bisect with Ephemeral Environments

#20

Post by LegionMammal978 »

agumonkey wrote: post #18
LegionMammal978 wrote: post #16

Earlier quoted context omitted.

The idea is, suppose I did a trisect, splitting the range [start,end) into [start,A), [A,B), and [B,end). At each step, I test commits A and B in parallel. If both A and B are bad, I continue with [start,A). If A is good and B is bad, I continue with [A,B). If both A and B are good, I continue with [B,end). This lets me rule out two thirds of the commits, in the same time that an ordinary bisect would have ruled out…

have you ever seen this implemented somewhere ? interesting idea

No, unfortunately not. If your history is strictly linear, you could probably hack together something relatively simple on top of git rev-list. But git bisect does all sorts of magic to deal with merge commits and other funny situations, and generalizing that to an n-sect would take a fair bit of work.
Post reply on HN