This project resonates with me a lot. Call me old-fashioned, but I still appreciate a nice ole' deterministic program that I can fully understand and operate reliably.
With that said, there is undoubtedly still room to innovate on the long-tail of RPA. In the healthcare domain, for example, there are 1000s of sites that might need to be scraped occasionally, somewhat transactionally as e.g. a new patient comes in. However, there are other sites that need regular attention and even the smallest of errors can be catastrophic.
The combination of browser-use & workflow-use seems like a really natural fit for such use cases. Nice work!
We've also experimented with the self-healing ideas you are playing with here. In our case, we wrote a chrome extension that connects to an LLM of your choice as well as a process running locally on your machine. You write a description of the job to be done, click around the browser, and then click "go". The extension grabs all the context, asks the LLM to write a typescript program, sends that typescript program to the local process where it is compiled & type-checked against our internal workflow harness, and then immediately allows you to execute the program against your existing, open browser context.
We've found that even this basic loop is outrageously productive. If the script doesn't do what you expect, there is a big "FIX IT" button that lets you tweak and try again. For the record, we're not a competitor and have no intention of trying to sell/offer this extension externally.
I suspect one of the harder parts about this whole ordeal will be how to integrate with the rest of the workflow stack. For us, we've really appreciated the fact that our extension outputs typescript that seamlessly fits into our stack and that is more easily verifiable than JSON. The TS target also allows us to do nice things like tell the self-healing bot which libraries will be available so that e.g. it can use `date-fns` instead of `Date`. We've also thought about adopting more traditional workflow tools like Temporal to manage the core workflow logic, vending out the browser connectivity remotely. Curious how you guys are thinking about this?
Rooting for you guys, we will be sure to keep an eye on your progress and consider adopting the technology as it matures!
PS. If you like things like this, want to work at a growing health-tech startup, and live in Boston, we're hiring! Reach out here: https://wgwx7h7be0p.typeform.com/to/LV0t8OjI