Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bombadil: Property-based testing for web UIs

github.com

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Bombadil: Property-based testing for web UIs

#101

Post by owickstrom »

squid_ca wrote: post #98

I probably missed it in the documentation, but how do I automate the browser with Bombadil? How do I get it to login to my app so I can test authenticated pages? Edit: as others have said, I am having trouble figuring out how to make this work and understanding how to use it, more practical examples of that would be helpful. Looking forward to using this. Thank you!

One smartass way to answer that for the general case is: you don't. And I mean that in the same sense as "you don't write example based tests using a PBT framework". You supply independent actions through generators and have it apply them "chaotically", not strictly in sensible sequences.

That said, for things like auth, you need to do some fixed steps to get anywhere at all. I haven't added support for this yet, but as I'm envisioning it, you'd either:

a) authenticate the browser using headers or cookies that you inject into the Bombadil browser, or b) use a "custom action" (essentially a stringifed JS function) that'd perform your particular steps

Or what you could do now: add preconditions to your actions so that they'd necessarily fill out the form before anything else could be done.

Another thing we might need in Bombadil is secrets, but for now I think it's fine to inject test credentials that are plain visible text.

Please let me know if you try it out and if you need any help!

Re: Bombadil: Property-based testing for web UIs

#102

Post by owickstrom »

javierhonduco wrote: post #69
owickstrom wrote: post #2

Author here, happy to answer questions about Bombadil! :)

Super cool project! Curious on how this compares to Meticulous in terms of functionality and approach. Thanks!

Thanks! I don't know much about it, but from what I can tell from their site, it does differential testing based on screenshots and behavior, and mocks out network IO, focusing on the client side. I'm not sure what the AI part is, if it's for extracting information from the page, for generating actions, for smarter diffing, or some combination of those.

Bombadil doesn't mock the network (although I'm working with a contributor to add something like Playwright's Route system for manipulating responses). It runs on your full stack (if you want) and if you need determinism for perfect bug reproducibility, you should run it all in Antithesis.

On the AI front, my view is that Bombadil should be optimized for using LLMs to generate and refine specs over time (properties and action generators), but that it should be an offline process. It's not like a "browser testing agent". It's analogous to having the LLM generate a program, rather than having the LLM itself perform actions. For some problems, the latter makes more sense, but for many problems the former is better, and I believe that PBT (and things like proof systems, but that's not my domain) are suited to the former strategy.

I have some sketches on how the refinement-with-human-in-the-loop thing should work with Bombadil and LLMs, but it's only a roadmap item for now.

Re: Bombadil: Property-based testing for web UIs

#103

Post by owickstrom »

ameliaquining wrote: post #95
owickstrom wrote: post #2

Author here, happy to answer questions about Bombadil! :)

Is there any particular reason why this is its own bespoke test runner, instead of a library that plugs into existing ones?

Do you mean test runners like JUnit, pytest, etc? Or browser test runners specifically?

Re: Bombadil: Property-based testing for web UIs

#104

Post by nz »

bertylicious wrote: post #97
nz wrote: post #48

Earlier quoted context omitted.

And sometimes you find errors in code that absolutely should never have errors: I found an (as of yet not-root-caused) error in sqlite (no crash or coredump, just returns the wrong data, and only when using sqlite in ram-only-mode). Had to move to postgres for that reason alone. This is part of the reason why I have a strong anti-library bias (and I sound like a lunatic to most colleagues because they "have never had…

Did you report that SQLite bug? What's the company's name?

Didn't get around to reporting it (huge backlog of tasks). Luckily I am working on a project that _has_ to support SQLite, so if I run into the bug again, I'll report it.

I don't believe that I can tell you the name of the company (they made me sign some NDAs, before the interview, and I have no clue how enforceable those are). Also, this was in 2019, so I would be shocked if they did not fix the problem by now -- especially after I interviewed there (plus I can't be the only one to have noticed this, since).

That said, you have a few data-points if you want to try to triangulate them yourself: physical vm-hosting and storage product, existed since at least 2019, used linux kernel as hypervisor, custom FS, international customers across 2 continents. All of those data-points are my recollection from 2019.

Re: Bombadil: Property-based testing for web UIs

#106

Post by squid_ca »

owickstrom wrote: post #101
squid_ca wrote: post #98

I probably missed it in the documentation, but how do I automate the browser with Bombadil? How do I get it to login to my app so I can test authenticated pages? Edit: as others have said, I am having trouble figuring out how to make this work and understanding how to use it, more practical examples of that would be helpful. Looking forward to using this. Thank you!

One smartass way to answer that for the general case is: you don't. And I mean that in the same sense as "you don't write example based tests using a PBT framework". You supply independent actions through generators and have it apply them "chaotically", not strictly in sensible sequences. That said, for things like auth, you need to do some fixed steps to get anywhere at all. I haven't added support for this yet, but…

Thank you for replying, I appreciate it! I’ll reach out when I get in trouble ;)

Re: Bombadil: Property-based testing for web UIs

#107

Post by ameliaquining »

owickstrom wrote: post #103
ameliaquining wrote: post #95

Earlier quoted context omitted.

Is there any particular reason why this is its own bespoke test runner, instead of a library that plugs into existing ones?

Do you mean test runners like JUnit, pytest, etc? Or browser test runners specifically?

The latter. For example, the concept here seems like it ought to work with the Playwright test runner for Node.js (and then you'd have compatibility with the Playwright ecosystem).

Re: Bombadil: Property-based testing for web UIs

#108

Post by owickstrom »

ameliaquining wrote: post #107
owickstrom wrote: post #103

Earlier quoted context omitted.

Do you mean test runners like JUnit, pytest, etc? Or browser test runners specifically?

The latter. For example, the concept here seems like it ought to work with the Playwright test runner for Node.js (and then you'd have compatibility with the Playwright ecosystem).

OK. Yeah so I made the decision early to go one level below Playwright and target CDP directly, because I believe I need the tight connection with Chrome/Chromium to make it as fast as possible. Also I really value the distribution aspects of writing this in Rust and compiling to a single executable with everything in it.

Re: Bombadil: Property-based testing for web UIs

#109

Post by dsq »

owickstrom wrote: post #100
dsq wrote: post #99

I’m curious regarding your choice of name. What in your company or product brought you to adopt that name, or whether you just liked it (So do I!).

I'll quote my lobste.rs reply to this question: > I think his jolly chaotic good energy, along with how he's uncorruptible and sort of outside the normal world, fits a fuzzer in a spiritual sense. But it might all just be a good excuse to use one of my favorite names from my favorite book. It's also my favorite part of the book, and definitely not something I'd skip. I know some people would object, but I think of it…

Excellent section of a sublime book/s

Re: Bombadil: Property-based testing for web UIs

#110

Post by wittjeff »

terpimost wrote: post #78
wittjeff wrote: post #74

From your title my immediate thought was "cool, maybe this will move us a bit closer to making components (or the testing thereof) cover accessibility thoroughly by default". See, the idea with the semantic web, and the ARIA markup equivalents, is that things should have names, roles, values, and states. Devs frequently mess up role/keyboard interaction agreement (example: role=menu means a list will drop on Enter ke…

Hey, sorry for antithesis.com. Thank you for noticing this! The bug is subtle: we do have styles for that, but in some change which I probably did we use css variable, which isn't there... We will fix that soon.

I'm a big proponent of use of design systems for efficient remediation and maintenance, so I'm also aware of how a small oops bug can have wide repercussions.

Let me know if you'd like to chat about use of your system to help enforce proper use of semantics and ARIA in web UI. It looks good.

Post reply on HN