[dead]
Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
51–56 of 56 posts
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#52Okay this is really fun and mathematically satisfying. Could even be useful for tough bugs that are technically deterministic, but you might not have precise reproduction steps. Does it support running a test multiple times to get a probability for a single commit instead of just pass/fail? I guess you’d also need to take into account the number of trials to update the Beta properly.
I think if you make your test script compile and then run the tests up to N times, failing on first fail, then when you run bayesect, it just "sees" a test that is "N times more" deterministic, so will behave appropriately.
I'm not sure how to choose an optimal value of N. My first hunch is make it so that it takes at least as long to run all the tests as it takes to setup (checkout, compile link etc.), but it may make sense to go a lot more than that. I'd have to do some thinking about the maths.
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#53Earlier quoted context omitted.
This is really cool! Is there an alternative way of thinking about it involving a hidden markov model, looking for a change in value of an unknown latent P(fail)? Or does your approach end up being similar to whatever the appropriate Bayesian approach to the HMM would be?
[flagged]
Please stop spamming HN wih AI slop.
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#54git bisect works great for tracking down regressions, but relies on the bug presenting deterministically. But what if the bug is non-deterministic? Or worse, your behaviour was always non-deterministic, but something has changed, e.g. your tests went from somewhat flaky to very flaky. In addition to the repo linked in the title, I also wrote up a little bit of the math behind it here: https://hauntsaninja.github.io/g…
Neat! I work on a system with some very similar math, but a slightly different model. I really like how in bayesect making the error rates asymmetric via independent Beta priors on bidirectional errors allows the computations to be nice and symmetric.
I haven't worked these all the way through, but I'm slightly skeptical or at least confused by a few details:
1. Another way to frame P(D|B=b) would be to have the old vs new side draws be beta-binomial distributed, in which case we should then have binomial coefficients for each of the draw side probabilities for the number of possible orderings of the observations. Do they end up cancelling out somewhere? [ed: Oh yes, of course -- D includes that in each case we observe exactly one of the C(n,k) orderings.]
2. I think your expected conditional entropy code is treating the imputed new observations as independent from the past observations, though even if that's the case it may not affect it much in this model. If it does though, it might be worth explicitly unit-testing the naive vs efficient calculations to ensure they match.
Anyway, thanks for sharing!
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#55Super cool! A related situation I was in recently was where I was trying to bisect a perf regression, but the benchmarks themselves were quite noisy, making it hard to tell whether I was looking at a "good" vs "bad" commit without repeated trials (in practice I just did repeats). I could pick a threshold and use bayesect as described, but that involves throwing away information. How hard would it be to generalize thi…
I don't yet know a better way to do this than using a threshold! I think if you assume perf is normally distributed, you can still get some of the math to work out. But I will need to think more about this... if I ever choose this adventure, I'll post an update on https://github.com/hauntsaninja/git_bayesect/issues/25 (I really enjoy how many generalisations there are of this problem :-) )
Perf is not normally distributed. Or rather, it is very very common for it to not be. Where I work, we often see multimodal (usually mostly bimodal) distributions, and we'll get performance alerts that when you look at them are purely a result of more samples happening to shift from one mode to another.
It's easy to construct ways that this could happen. Maybe you're running a benchmark that does a garbage collection or three. It's easy for a GC to be a little earlier or a little later, and sneak in or out of the timed portion of the test.
Warm starts vs cold starts can also do this. If you don't tear everything down and flush before beginning a test, you might have some amount of stuff cached.
The law of large numbers says you can still make it normal by running enough times and adding them up (or running each iteration long enough), but (1) that takes much longer and (2) smushed together data is often less actionable. You kind of want to know about fast paths and slow paths and that you're falling off the fast path more often than intended.
As usual you can probably cover your eyes, stick your fingers in your ears, and proceed as if everything were Gaussian. It'll probably work well enough!
Re: Show HN: Git bayesect – Bayesian Git bisection for non-deterministic bugs
#56git bisect works great for tracking down regressions, but relies on the bug presenting deterministically. But what if the bug is non-deterministic? Or worse, your behaviour was always non-deterministic, but something has changed, e.g. your tests went from somewhat flaky to very flaky. In addition to the repo linked in the title, I also wrote up a little bit of the math behind it here: https://hauntsaninja.github.io/g…
My team doesn’t always have cleanly bisectable branches being merged to main —- it’s not uncommon to see “fix syntax error” types of commits. But, to merge we need to have all tests pass. (If tests flakily pass then we get new flakey tests, yay!) I know git-bisect doesn’t support this: but could git-bayesect have an option to only consider merge commits? Being able to track a flake change back to an individual PR wou…
> I know git-bisect doesn’t support this
It does.