Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bombadil: Property-based testing for web UIs

github.com

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Bombadil: Property-based testing for web UIs

#81

Post by Mr_RxBabu »

danbruc wrote: post #12
IanCal wrote: post #4

I'm a huge fan of property based testing, I've built some runners before, and I think it can be great for UI things too so very happy to see this coming around more. Something I couldn't see was how those examples actually work, there are no actions specified. Do they watch a user, default to randomly hitting the keyboard, neither and you need to specify some actions to take? What about rerunning things? Is there shr…

How effective is property based testing in practice? I would assume it has no trouble uncovering things like missing null checks or an inverted condition because you can cover edge cases like null, -1, 0, 1, 2^n - 1 with relatively few test cases and exhaustively test booleans. But beyond that, if I have a handful of integers, dates, or strings, then the state space is just enormous and it seems all but impossible to…

[dead]

Re: Bombadil: Property-based testing for web UIs

#82

Post by inaseer »

Bombadil takes a fresh approach to UI testing I haven't seen before: online monitoring through LTL formulas. Unlike model-checking (say by TLC), LTL formulas over here unfold in lock-step with the UI and allow users to express interesting temporal properties during testing.

The other intriguing aspect was how state is modeled (or rather, maybe not explicitly modeled?). A lot of the examples show the state extracted from the DOM and temporal properties indicating what the next (or eventual) state _should be_. If we want to look at the existing state (according to the model/spec) when predicting the next state (similar to how you can use s when specifying s' in TLA+), there seems to be no direct way to do that. It should of course be possible to capture the state at an earlier time in a closure and use it in a thunk at a later point, so it should be possible to work around this but that can be a little awkward, maybe. I'm working on a project in this space (primarily geared towards backend API model-based testing) and the state of the _real_ system isn't globally inspectable unlike a web page so took a different route over there. Having said that, this is a very interesting design choice that's very intriguing (in a good way).

Re: Bombadil: Property-based testing for web UIs

#83

Post by owickstrom »

Darmani wrote: post #79

Timely! I spent a good chunk of yesterday investigating Bombadil. I'm a huge fan of PBT and a huge fan of temporal logic (though not linear temporal logic) and we're currently trying to level up our QA, so it seemed like a good match. Unfortunately, I concluded that Bombadil is a toy . Not as in "Does some very nice things, but missing the features for enterprise adoption." I mean that in a very strong sense, as in:…

Hey, yeah, so actions are indeed independent and you can't express fixed sequences of them (e.g. fill out thus form by doing X, then Y, then Z). If you want to enforce certain sequentiality you'd need to precondition your generators. This is admittedly limited right now. I do want to add QoL things like `filter` on the generators to make that more ergonomic.

Also, as you note, you can't implement custom actions. And that's just something that I haven't gotten to yet. It'd be quite straightforward to add a plain function (toStringed) as one of the variants of the Action type and send that to the browser for eval. (Btw, we're taking contributions!)

Weighting is also important right now. There's no smart exploration in Bombadil yet, only blind random, so manual weighting becomes pretty crucial to have more effective tests (i.e. unlikely to run in circles). I'd like to both make the Bombadil "fuzzer" better at this, but eventually you might want to run Bombadil inside Antithesis instead to get a much better exploration, even in a single campaign.

The Until operator is also one of those things that I haven't gotten to yet. I actually didn't expect someone to hit this as a major limitation of the tool so quickly, which is why I've focused on other things. Surprising!

To add some more context, Bombadil is an OSS project with one developer (but we're hiring!) and it's 4 months old and marked experimental. I'm sorry you were disappointed by its current state, but it's very early days, so expect a lot of these things to improve. And your feedback will be taken into account. Thanks!

Re: Bombadil: Property-based testing for web UIs

#84

Post by owickstrom »

inaseer wrote: post #82

Bombadil takes a fresh approach to UI testing I haven't seen before: online monitoring through LTL formulas. Unlike model-checking (say by TLC), LTL formulas over here unfold in lock-step with the UI and allow users to express interesting temporal properties during testing. The other intriguing aspect was how state is modeled (or rather, maybe not explicitly modeled?). A lot of the examples show the state extracted f…

Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and LLMs) would have to learn, and to have to write tools for. Now you can write Bombadil specs with a good LSP and be able to import packages off of npm or whathaveyou. Most web devs will probably be comfy with JS or TS, so that's why I chose that style.

I hope that makes sense?

Thanks for the nice feedback!

Re: Bombadil: Property-based testing for web UIs

#85

Post by magicalhippo »

NoraCodes wrote: post #16

My kingdom for a way to stop this godforsaken industry from stripping Tolkien's fiction for parts.

Indeed. Why not follow Douglas Adams' example and use obscure place names instead[1].

[1]: https://en.wikipedia.org/wiki/The_Meaning_of_Liff

Re: Bombadil: Property-based testing for web UIs

#86

Post by maxwg »

owickstrom wrote: post #65
sequoia wrote: post #13

Struggling to understand what this is or how it works.

Did you have a look at the intro in the manual? https://antithesishq.github.io/bombadil/1-introduction.html If that's not clear, please let me know how we can improve it!

> Bombadil itself decides what is an interesting event and when to capture state.

It would be very nice to have a little more details on how this works. Though I guess we can figure it out via code/trialing it out.

I'll give it a go later today!

Re: Bombadil: Property-based testing for web UIs

#87

Post by owickstrom »

maxwg wrote: post #86
owickstrom wrote: post #65

Earlier quoted context omitted.

Did you have a look at the intro in the manual? https://antithesishq.github.io/bombadil/1-introduction.html If that's not clear, please let me know how we can improve it!

> Bombadil itself decides what is an interesting event and when to capture state. It would be very nice to have a little more details on how this works. Though I guess we can figure it out via code/trialing it out. I'll give it a go later today!

Go for it! I've been meaning to do an "architecture of Bombadil" blog post that'd likely answer this question. It's not super advanced by any means, but it's a mindset shift to how you might think about browser testing coming from the mainstream frameworks like Playwright.

Re: Bombadil: Property-based testing for web UIs

#88

Post by inaseer »

owickstrom wrote: post #84
inaseer wrote: post #82

Bombadil takes a fresh approach to UI testing I haven't seen before: online monitoring through LTL formulas. Unlike model-checking (say by TLC), LTL formulas over here unfold in lock-step with the UI and allow users to express interesting temporal properties during testing. The other intriguing aspect was how state is modeled (or rather, maybe not explicitly modeled?). A lot of the examples show the state extracted f…

Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and…

Yes, I understand why you made these design decisions. And I also agree that sticking to JS/TS keeps things simple (for humans, and LLMs). I generally default to the s and s' way of specifying things (in a C# property-based testing framework I'm working on) but looking at how you approached things here gives me another angle to think about.

Good work!

Re: Bombadil: Property-based testing for web UIs

#89

Post by owickstrom »

inaseer wrote: post #88
owickstrom wrote: post #84

Earlier quoted context omitted.

Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and…

Yes, I understand why you made these design decisions. And I also agree that sticking to JS/TS keeps things simple (for humans, and LLMs). I generally default to the s and s' way of specifying things (in a C# property-based testing framework I'm working on) but looking at how you approached things here gives me another angle to think about. Good work!

Cool. Is this publicly available?

Re: Bombadil: Property-based testing for web UIs

#90

Post by inaseer »

owickstrom wrote: post #89
inaseer wrote: post #88

Earlier quoted context omitted.

Yes, I understand why you made these design decisions. And I also agree that sticking to JS/TS keeps things simple (for humans, and LLMs). I generally default to the s and s' way of specifying things (in a C# property-based testing framework I'm working on) but looking at how you approached things here gives me another angle to think about. Good work!

Cool. Is this publicly available?

Soon!
Post reply on HN