Hypothesis, Antithesis, synthesis • Hacker News Board

Re: Hypothesis, Antithesis, synthesis

#31

Post by DRMacIver » Tue, Mar 24, 2026, 5:11 PM UTC

ngruhn wrote: post #26

DRMacIver wrote: post #19

Earlier quoted context omitted.

> But the problem remains verifying that the tests actually test what they're supposed to. Definitely. It's a lot harder to fake this with PBT than with example-based testing, but you can still write bad property-based tests and agents are pretty good at doing so. I have generally found that agents with property-based tests are much better at not lying to themselves about it than agents with just example-based testin…

> I have generally found that agents with property-based tests are much better at not lying to themselves I also observed the cheating to increase. I recently tried to do a specific optimization on a big complex function. Wrote a PBT that checks that the original function returns the same values as the optimized function on all inputs. I also tracked the runtime to confirm that performance improved. Then I let Claude…

Ouch. Classic Claude. It does tend to cheat when it gets stuck, and I've had some success with stricter harnesses, reflection prompts and getting it to redo work when it notices it's cheated, but it's definitely not a solved problem.

My guess is that you wouldn't have had a better time without PBT here and it would still have either cheated or claimed victory incorrectly, but definitely agreed that PBT can't fully fix the problem, especially if it's PBT that the agent is allowed to modify. I've still anecdotally found that the results are better than without it because even if agents will often cheat when problems are pointed out, they'll definitely cheat if problems aren't pointed out.

Re: Hypothesis, Antithesis, synthesis

#32

Post by js8 » Tue, Mar 24, 2026, 5:12 PM UTC

pron wrote: post #8

> property-based testing is going to be a huge part of how we make AI-agent-based software development not go terribly. There's no doubt, I think, testing will remain important and possibly become more important with more AI use, and so better testing is helpful, PBT included. But the problem remains verifying that the tests actually test what they're supposed to. Mutation tests can allow agents to get good coverage…

> There's no doubt, I think, testing will remain important and possibly become more important with more AI use, and so better testing is helpful, PBT included.

Given Curry-Howard isomorphism, couldn't we ask AI to directly prove the property of the binary executable under the assumption of the HW model, instead of running PBTs?

By no means I want to dismiss PBTs - but it seems that this could be both faster and more reliable.

Re: Hypothesis, Antithesis, synthesis

#33

Post by cmrdporcupine » Tue, Mar 24, 2026, 5:18 PM UTC

DRMacIver wrote: post #5

hugeBirb wrote: post #4

Not that it matters at this point but the hegelian dialectic is not thesis, antithesis and synthesis. Usually attributed to Hegel but as I understand it he actually pushed back on this mechanical view of it all and his views on these transitory states was much more nuanced.

Conversation with Will (Antithesis CEO) a couple months ago, heavily paraphrased: Will: "Apparently Hegel actually hated the whole Hegelian dialectic and it's falsely attributed to him." Me: "Oh, hm. But the name is funny and I'm attached to it now. How much of a problem is that?" Will: "Well someone will definitely complain about it on hacker news." Me: "That's true. Is that a problem?" Will: "No, probably not." (Wh…

I think it's more that Hegel was fine with "dialectics" but that the antithesis/synthesis stuff is not actually what's going on in his dialectic. It's a bit of a popular misconception about the role of negation and "movement" in Hegel.

I believe (unless my memory is broken) they get into this a bunch in Ep 15 of my favourite podcast "What's Left Of Philosophy": https://podcasts.apple.com/gb/podcast/15-what-is-dialectics-...

Also if you're not being complained about on HN, are you even really nerd-ing?

Re: Hypothesis, Antithesis, synthesis

#34

Post by skybrian » Tue, Mar 24, 2026, 5:21 PM UTC

It isn't used by anyone besides me, but I wrote a property-testing library for Deno [1] that has a form of "sometimes" assertions (inspired by Antithesis) and uses "internal shrinking" (inspired by Hypothesis).

But it's still a "blind" fuzzer and it would be nice to write one that gets feedback from code coverage somehow. Instead, you have to run code coverage yourself and figure out how to change test data generation to improve it.

[1] https://jsr.io/@skybrian/repeat-test

Re: Hypothesis, Antithesis, synthesis

#35

Post by zero0529 » Tue, Mar 24, 2026, 5:24 PM UTC

hugeBirb wrote: post #4

Not that it matters at this point but the hegelian dialectic is not thesis, antithesis and synthesis. Usually attributed to Hegel but as I understand it he actually pushed back on this mechanical view of it all and his views on these transitory states was much more nuanced.

I remember first learning about Hegel when playing Fallout NV. Caesar made it seem so simple.

Re: Hypothesis, Antithesis, synthesis

#36

Post by sigbottle » Tue, Mar 24, 2026, 5:28 PM UTC

viccis wrote: post #30

sigbottle wrote: post #25

Earlier quoted context omitted.

From what I understand, it's a proof technique (other techniques include Kant's Transcendental Deduction or Descartes's pure doubt) that requires generating new conceptual thoughts via internal contradiction and showing necessarily that you lead from one category to the next. The necessity thing is the big thing - why unfold in this way and not some other way. Because the premises in which you set up your argument ca…

>other techniques include Kant's Transcendental Deduction or Descartes's pure doubt This is not quite accurate. Kant says very explicitly in the (rarely studied) Transcendental Doctrine of Method (Ch 1 Section 4, A789/B817) that this kind of proof method (he calls it "apagogic") is unsuitable to transcendental proofs. You might be thinking of the much more well studied Antinomies of Pure Reason, in which he uses this…

Sorry I meant "proof method" as more like "this was this guy's angle of attack", not that they would've thought each others angles were valid at all or that they're commensurable with say, 20th century formal proof logic (or Aristotelian logic for example). Descartes and Leibniz were squarely the rationalists that Kant wanted to abolish, and Hegel rejected Kants distinction between noumena and phenomena entirely, so they're already starting from very different places.

I guess it would be more accurate to state Kants actual premises here as making the distinction between appearance and thing-in-itself rather than the deduction, but the deduction technique itself was fascinating when I first learned it so that's what I associate most with Kant lol.

I guess I have not thought critically why we couldn't use a Transcendental argument to support Descartes. I just treated it as a vague category error (to be fair I don't actually know Descartes philosophy that well, even less than I know Kants lol). Could be a fun exercise when I have time.

Re: Hypothesis, Antithesis, synthesis

#37

Post by ux266478 » Tue, Mar 24, 2026, 5:28 PM UTC

AndrewKemendo wrote: post #23

sigbottle wrote: post #20

Earlier quoted context omitted.

I thought the dialectic was just a proof methodology, and especially the modern political angles you might year from say a Youtube video essay on Hegel, was because of a very careful narrative from some french dude (and I guess Marx with his dialectical materialism). I mean, I agree with many perspectives from 20th century continental philosophy, but it has to be agreed that they refactored Hegel for their own purpos…

Oh the amount of branching and forking and remixing of Hegel is more or less infinite I think it’s worth again pointing out that Hegel was at the height of contemporary philosophy at the time but he wasn’t a mathematician and this is the key distinction. Hagel lives in the pre-mathematical economics world. The continental philosophy world of words with Kant etc… and never crossed into the mathematical world. So I lik…

> The continental philosophy world of words with Kant

Interestingly, a lot of arguments and formulations Kant had were lifted from Leibniz and reframed with a less mathematical flavor. I remember in particular his argument against infinite regress was pretty much pound for pound just reciting some conjecture from Leibniz (without attribution)

Re: Hypothesis, Antithesis, synthesis

#38

Post by AndrewKemendo » Tue, Mar 24, 2026, 5:31 PM UTC

sigbottle wrote: post #27

AndrewKemendo wrote: post #23

Earlier quoted context omitted.

Oh the amount of branching and forking and remixing of Hegel is more or less infinite I think it’s worth again pointing out that Hegel was at the height of contemporary philosophy at the time but he wasn’t a mathematician and this is the key distinction. Hagel lives in the pre-mathematical economics world. The continental philosophy world of words with Kant etc… and never crossed into the mathematical world. So I lik…

I mean I don't know about Hegel, but Kant certainly dipped into mathematics. One of the reasons why he even wrote CPR was to unify in his mind, the rationalists (had Leibniz) versus the empiricists (had Newton). 20th century analytic philosophy was heavily informed by Kantian distinctions (Logical Positivism uses very similar terminology, and Carnap himself was a Neo-Kantian originally, though funnily enough Heidegge…

My claim is that genuinely all of those previous analytical forms are absolutely useless if you have the capacity to utilize a more mathematical framework

The problem is, those more mathematically challenging frameworks are inaccessible to the majority of the people

so they don’t actually take off because there’s no mechanism to translate more rigor in social studies and social sciences in large part because humans reject the concept of being measured and experimeted with, which is understandable if not optimal

So as a function, applications of mathematics trended towards things that were not human focused and they were machine focused and financial focused

So the big transition happened after TV and Internet (really just low cost high reach advertising) became pervasive and social scientists began utilizing statistical methods across consumer and attention action as social science experimentation platforms

Social science moved from the squishy into the precise precisely to give companies a market advantage in capturing market share through manipulating human behavior

ultimately that was the wet dream of political philosophers since pahotep

Hegel is irrelevant in the age of measurement

Re: Hypothesis, Antithesis, synthesis

#39

Post by DRMacIver » Tue, Mar 24, 2026, 5:32 PM UTC

lwhsiao wrote: post #28

DRMacIver, can you comment on how this fits into the existing property-based testing ecosystems for various languages? E.g., if I use proptest in Rust, why would/should I switch to Hegel?

The short answer to how it fits into existing ecosystems is... in competition I suppose. We've got a lot of respect for the people working on these libraries, but we think the Hypothesis-based approach is better than the various approaches people have adopted. I don't love that the natural languages for us to start with are ones where there are already pretty good property-based testing libraries whose toes we're stepping on, but it ended up being the right choice because those are the languages people care about writing correct software in, and also the ones we most want the tools in ourselves!

I think right now if you're a happy proptest user it's probably not clear that you should switch to Hegel. I'd love to hear about people trying, but I can't hand on my heart say that it's clearly the correct thing for you to do given its early state, even though I believe it will eventually be.

But roughly the things that I think are clearly better about the Hegel approach and why it might be worth trying Hegel if you're starting greenfield are:

* Much better generator language than proptest (I really dislike proptest's choices here. This is partly personal aesthetic preferences, but I do think the explicitly constructed generators work better as an approach and I think this has been borne out in Hypothesis). Hegel has a lot of flexible tooling for generating the data you want.

* Hegel gets you great shrinking out of the box which always respects the validity requirements of your data. If you've written a generator to always ensure something is true, that should also be true of your shrunk data. This is... only kindof true in proptest at best. It's not got quite as many footguns in this space as original quickcheck and its purely type-based shrinking, but you will often end up having to make a choice between shrinking that produces good results and shrinking that you're sure will give you valid data.

* Hegel's test replay is much better than seed saving. If you have a failing test and you rerun it, it will almost immediately fail again in exactly the same way. With approaches that don't use the Hypothesis model, the best you can hope for is to save a random seed, then rerun shrinking from that failing example, which is a lot slower.

There are probably a bunch of other quality of life improvements, but these are the things that have stood out to me when I've used proptest, and are in general the big contrast between the Hypothesis model and the more classic QuickCheck-derived ones.

Re: Hypothesis, Antithesis, synthesis

#40

Post by mullr » Tue, Mar 24, 2026, 5:32 PM UTC

DRMacIver wrote: post #2

Post author here btw, happy to take questions, whether they're about Hegel in particular, property-based testing in general, or some variant on "WTF do you mean you wrote rust bindings to a python library?"

Why would I use this over the existing Proptest library in Rust?