Hey Oskar ~ great project and looks promising. I would be curious to hear what is still work-in-progress for Bombadil. It's helpful to know what the tool maintainers see as upcoming or incomplete work. It also saves a consultant like me a lot of time to evaluate new tools for clients if I also know the limitations before diving in. Maybe a section in the manual for "What Bombadil can't do". Great work!
Bombadil: Property-based testing for web UIs
41–50 of 112 posts
Re: Bombadil: Property-based testing for web UIs
#42Re: Bombadil: Property-based testing for web UIs
#43My kingdom for a way to stop this godforsaken industry from stripping Tolkien's fiction for parts.
Re: Bombadil: Property-based testing for web UIs
#44Author here, happy to answer questions about Bombadil! :)
From a project management perspective, the 5 examples don't help me understand how/why I might switch from Playwright/Cypress to this framework. It seems like Bombadil is a much lower-level test framework focusing on DOM properties but in the "Why Bombadil?" introduction you say "maintaining suites of Playwright or Cypress tests takes a lot of work" ... I'd like if there was an example showing how this is true, perha…
Re: Bombadil: Property-based testing for web UIs
#45The major breakthrough here is that they managed to write a project in Rust and not mention it on the headline! Jokes aside, great project and documentation (manual)! Getting started was really simple, and I quickly understood what it can and cannot do.
Re: Bombadil: Property-based testing for web UIs
#46very cool! does this work? can you describe the kinds of real bugs you've caught with this?
Re: Bombadil: Property-based testing for web UIs
#47Earlier quoted context omitted.
I use quicktheories (Java) and generate a determistic random test scenario, then I generate input values and run the tests. This way I can create tests that should fail or succeed, but differ in the steps executed and in the order with "random input".
OK. What kind of problems do you hit from third-party libraries with that?
Re: Bombadil: Property-based testing for web UIs
#48I'm doing propety-based test since years for frontend stuff. The hardest part is, that there is so much between the test inputs and the application under test, that I find 50% of the time problems with the frontend test frameworks/libs and not in our code.
Sometimes, only thing you can do is let the plague spread, and hope that the people who survive start showering and washing their hands.
[0]: I once interviewed at a company that sold a kind of on-prem VM hosting and storage product. They were shipping a physical machine with Linux and a custom filesystem (so not ZFS), and they bragged about how their filesystem was very fast, much faster than ZFS or Btrfs on SSDs. I asked them, if they were allowed to tell me how they achieved such consistent numbers. They listed a few things, one of which was: "we disabled block-level check-summing". I asked: "how do you prevent corruption?". They said: "we only enable check-summing during the nightly tests". So, a little unsettled, I asked: "you do not do _any_ check-summing at any point in production"? They replied: "Exactly. It's not necessary". So, throwing caution to the wind (at this point I did not care to get the job), I asked: "And you've never had data corruption in production"? They said: "Never. None". To which I replied: "But how do you _know_"? My no-longer-future-coworker thought for a few seconds, and realization flashed across his face. This was a company that had actual customers on 2 continents, and was pulling in at least millions per year. They were probably silently corrupting customer data, while promising that they were the solution -- a hospital selling snake-oil, while thinking it really is medicine.
Re: Bombadil: Property-based testing for web UIs
#49I'm a huge fan of property based testing, I've built some runners before, and I think it can be great for UI things too so very happy to see this coming around more. Something I couldn't see was how those examples actually work, there are no actions specified. Do they watch a user, default to randomly hitting the keyboard, neither and you need to specify some actions to take? What about rerunning things? Is there shr…
Hey, yeah the default specification includes a set of action generators that are picked from randomly. If you write a custom spec you can define your own action generators and their weights. Rerunning things: nothing built for that yet, but I do have some design ideas. Repros are notoriously shaky in testing like this (unless run against a deterministic app, or inside Antithesis), but I think Bombadil should offer be…
Should be pretty easy to make it deterministic if you follow that precondition.
(How I had my review apps wired up was I dumped the staging DB nightly and containerized it, I believe Neon etc make it easy to do this kind of thing.)
Ages ago I wired up something much more basic than this for a Python API using hypothesis, and made the state machine explicit as part of the action generator (with the transitions library), what do you think about modeling state machines in your tests? (I suppose one risk is you don’t want to copy the state machine implementation from inside the app, but a nice fluent builder for simple state machines in tests could be a win.)
Re: Bombadil: Property-based testing for web UIs
#50Earlier quoted context omitted.
Hey, yeah the default specification includes a set of action generators that are picked from randomly. If you write a custom spec you can define your own action generators and their weights. Rerunning things: nothing built for that yet, but I do have some design ideas. Repros are notoriously shaky in testing like this (unless run against a deterministic app, or inside Antithesis), but I think Bombadil should offer be…
For re-running, I assume you want to do this all on a review app with a snapshot of the DB, so you start with a clean app state. Should be pretty easy to make it deterministic if you follow that precondition. (How I had my review apps wired up was I dumped the staging DB nightly and containerized it, I believe Neon etc make it easy to do this kind of thing.) Ages ago I wired up something much more basic than this for…
Regarding state machines: yeah, it can often become an as-complex mirror of the system your testing, if the system has a large complicated surface. If on the other hand the API is simple and encapsulates a lot of complexity (like Ousterhout's "Deep Modules") state machine specs and model-based testing make more sense. Testing a key-value store is a great example of this.
If you're curious about it, here's a very detailed spec for TodoMVC in Bombadil: https://github.com/owickstrom/bombadil-playground/blob/maste... It's still work-in-progress but pretty close to the original Quickstrom-flavored spec.