Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stagehand – an open source browser automation framework powered by AI

github.com

Post reply on HN
Print view View original

21–30 of 89 posts

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#21

Post by hackgician »

jsdalton wrote: post #15

Does it operate by translating your higher level AI methods into lower level Playwright methods, and if so is it possible to debug the actual methods those methods were translated to? Also is there some level of deterministic behavior here or might every test run result in a different underlying command if your wording isn’t precise enough?

It's a little hacky, but we have a method in the act() handler called performPlaywrightMethod that takes in a playwright method + xpath and executes the playwright method on the xpath. There's definitely a lot of room for improvement here, and we're working on making observe() fill those gaps. I think observe() aims to be like GitHub Copilot's gray suggested text that you can then confirm in a secondary step; whereas act() takes on a more agentic workflow that you let the underlying agent loop make decisions on your behalf

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#22

Post by hackgician »

kevmo314 wrote: post #20
hackgician wrote: post #8

Earlier quoted context omitted.

Playwright codegen is incredibly powerful, but still pretty brittle. Its DOM selectors are still hardcoded, so you run the risk of Playwright selecting an unsustainable DOM selector. With Stagehand, the code is self-healing since it's dynamically generating Playwright every time, making it much more resilient to minor DOM changes

How do you avoid this becoming horrendously expensive per run? Are the results cached if the DOM doesn't change?

The purpose of using Playwright is to basically write deterministic workflows in deterministic automation code. We have basic prompt caching right now that works if the DOM doesn't change (as you mention), but also the best way to reduce token cost is to reduce reliance on AI itself. You have the most control over how much you want to rely on AI v. how much you want to write repeatable Playwright code.

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#23

Post by hackgician »

pryelluw wrote: post #14

Can it be adapted to use ollama? Seems like a good tool to setup locally as a navigation tool.

Yes, you can certainly use Ollama! However, we strongly recommend using a more beefed up model to get sustainable results. Check out our external_client.ts file in examples/ that shows you how to setup a custom LLMClient: https://github.com/browserbase/stagehand/blob/main/examples/...>

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#25

Post by kevmo314 »

hackgician wrote: post #22
kevmo314 wrote: post #20

Earlier quoted context omitted.

How do you avoid this becoming horrendously expensive per run? Are the results cached if the DOM doesn't change?

The purpose of using Playwright is to basically write deterministic workflows in deterministic automation code. We have basic prompt caching right now that works if the DOM doesn't change (as you mention), but also the best way to reduce token cost is to reduce reliance on AI itself. You have the most control over how much you want to rely on AI v. how much you want to write repeatable Playwright code.

That seems like a pretty tough sell over bare playwright. Unless the UI is constantly changing, the cost of verifying tests are still successful seems like it would eclipse the cost of an engineer maintaining the test pretty quickly.

Some minimal model that could be run locally and specifically tuned for this purpose might be pretty fruitful here compared to delegating out to expensive APIs.

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#26

Post by dchuk »

This looks awesome.

What I would love to see either as something leveraging this, or built in to this, is if you prompt stagehand to extract data from a page, it also returns the xpath elements you'd use to re-scrape the page without having to use an LLM to do that second scraping.

So basically, you can scrape new pages never before seen with the non-deterministic LLM tool, and then when you need to rescrape the page again to update content for example, you can use the cheaper old-school scraping method.

Not sure how brittle this would be both going from LLM version to xcode version reliably, or how to fallback to the LLM version if your xcode script fails, but overall conceptually, being able to scrape using the smart tools but then building up basically a library of dumb scraping scripts over time would be killer.

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#27

Post by hackgician »

dchuk wrote: post #26

This looks awesome. What I would love to see either as something leveraging this, or built in to this, is if you prompt stagehand to extract data from a page, it also returns the xpath elements you'd use to re-scrape the page without having to use an LLM to do that second scraping. So basically, you can scrape new pages never before seen with the non-deterministic LLM tool, and then when you need to rescrape the page…

Yeah, I think someone opened a similar issue on GitHub: https://github.com/browserbase/stagehand/issues/389

Repeatability of extract() is definitely super interesting and something we're looking into

Re: Show HN: Stagehand – an open source browser automation framework powered by AI

#29

Post by hackgician »

kevmo314 wrote: post #25
hackgician wrote: post #22

Earlier quoted context omitted.

The purpose of using Playwright is to basically write deterministic workflows in deterministic automation code. We have basic prompt caching right now that works if the DOM doesn't change (as you mention), but also the best way to reduce token cost is to reduce reliance on AI itself. You have the most control over how much you want to rely on AI v. how much you want to write repeatable Playwright code.

That seems like a pretty tough sell over bare playwright. Unless the UI is constantly changing, the cost of verifying tests are still successful seems like it would eclipse the cost of an engineer maintaining the test pretty quickly. Some minimal model that could be run locally and specifically tuned for this purpose might be pretty fruitful here compared to delegating out to expensive APIs.

Definitely a very interesting problem we're trying to dig deep into. We'd welcome any PRs here as well from the community :)
Post reply on HN