[flagged]
Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
41–50 of 56 posts
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#42Earlier quoted context omitted.
Yes, bayesect accuracy increases with more iterations. The comparison was at a fixed budget(300 test runs) when I was running. Sorry should have clarified more on that.
Yep, you can run bayesect to an arbitrary confidence level. This script in the repo https://github.com/hauntsaninja/git_bayesect/blob/main/scrip... will show you that a) the confidence level is calibrated, b) how quickly you get to that confidence level (on average, p50 and p95) For the failure rates you describe, calibration.py shows that you should see much higher accuracy at 300 tests
You're right, at 300 tests bayesect converges to ~97-100% across the board. I reran with calibration.py and confirmed.
Went a step further and tested graph-weighted priors (per-commit weight proportional to transitive dependents, Pareto-distributed). The prior helps in the budget-constrained regime:
128 commits, 500 trials:
Budget=50, 70/30: uniform 22% → graph 33% Budget=50, 80/20: uniform 71% → graph 77% Budget=100, 70/30: uniform 56% → graph 65% At 300 tests the gap disappears since there's enough data to converge anyway. The prior is worth a few bits, which matters when bits are scarce.
Script: https://gist.github.com/rs545837/b3266ecf22e12726f0d55c56466...
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#43Super cool! A related situation I was in recently was where I was trying to bisect a perf regression, but the benchmarks themselves were quite noisy, making it hard to tell whether I was looking at a "good" vs "bad" commit without repeated trials (in practice I just did repeats). I could pick a threshold and use bayesect as described, but that involves throwing away information. How hard would it be to generalize thi…
I have this same issue a lot. I vibe up a lot of really simple casual games, which should have very minimal demands, and the LLM-agent introduces bad things a lot that don't present right away. Either it takes multiple bad things to notice, or it doesn't really affect anything on a dev machine but is horrible on wasm+mobile builds, or I just don't notice right away. This is all really hard to track down, there's nois…
This is a real pain point. One thing that helps: when an LLM agent makes changes across multiple commits, look at what it actually touched structurally. Often the agent adds a feature in commit 5 but subtly breaks something in commit 3 by changing a shared function it didn't fully understand.
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#44I hope this comment is not out of place, but I am wondering what the application for all this is? How can this help us or what does it teach us or help us prove? I am asking out of genuine curiosity as I barely understand it but I believe it has something to do with probability. edit: thanks for the responses! I was not even familiar with `git bisect` before this, so I've got some new things to learn.
Opening the discussion to include properties of nondeterministic bugs. Often these bugs depend on timing, caused by unpredictable thread scheduling, CPU load, disk and networking timing, etc. Git commits can affect app timing and change the likelihood of the bug occurring, but in many cases these changes aren't related to the underlying bug. That's distinct from a regular git bisect to find a deterministic bug. One c…
[dead]
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#45Does this tool assume it takes the same amount of time to test two commits once as it does to test one commit twice? Maybe true for interpreted languages, but if you're waiting 15 minutes to compile LLVM you're probably going to want to run your 1 second flaky test more than once. Probably pretty trivial to fix this though? Great idea anyway!
One cheap optimization for the compile overhead case: skip commits that only touch files unrelated to the failing test. If you know the test's dependency chain, any commit that doesn't touch that chain gets prior weight zero. Equivalent to git bisect skip but automatic. Cuts the search space before you compile anything.
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#46git bisect works great for tracking down regressions, but relies on the bug presenting deterministically. But what if the bug is non-deterministic? Or worse, your behaviour was always non-deterministic, but something has changed, e.g. your tests went from somewhat flaky to very flaky. In addition to the repo linked in the title, I also wrote up a little bit of the math behind it here: https://hauntsaninja.github.io/g…
This is really cool! Is there an alternative way of thinking about it involving a hidden markov model, looking for a change in value of an unknown latent P(fail)? Or does your approach end up being similar to whatever the appropriate Bayesian approach to the HMM would be?
[flagged]
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#47Super cool! A related situation I was in recently was where I was trying to bisect a perf regression, but the benchmarks themselves were quite noisy, making it hard to tell whether I was looking at a "good" vs "bad" commit without repeated trials (in practice I just did repeats). I could pick a threshold and use bayesect as described, but that involves throwing away information. How hard would it be to generalize thi…
I don't yet know a better way to do this than using a threshold!
I think if you assume perf is normally distributed, you can still get some of the math to work out. But I will need to think more about this... if I ever choose this adventure, I'll post an update on https://github.com/hauntsaninja/git_bayesect/issues/25
(I really enjoy how many generalisations there are of this problem :-) )
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#48I hope this comment is not out of place, but I am wondering what the application for all this is? How can this help us or what does it teach us or help us prove? I am asking out of genuine curiosity as I barely understand it but I believe it has something to do with probability. edit: thanks for the responses! I was not even familiar with `git bisect` before this, so I've got some new things to learn.
Opening the discussion to include properties of nondeterministic bugs. Often these bugs depend on timing, caused by unpredictable thread scheduling, CPU load, disk and networking timing, etc. Git commits can affect app timing and change the likelihood of the bug occurring, but in many cases these changes aren't related to the underlying bug. That's distinct from a regular git bisect to find a deterministic bug. One c…
[flagged]
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#49What I love about this is you applied Bayesian methods to a problem I never would have thought to use it on!
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#50> our entropy calculation will now have to use the posterior means
Now hang on for a bit, you can't just plug in averages.
At least that's what I initially thought, but in this particular instance it works out correctly because you're calculating an expected value of the entropy from the two possible outcomes and there the posterior mean is indeed the correct probability to use.
You do have to take the prior into account when calculating the posterior distributions for B, but that formula is in the article.