Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Magnitude – Open-source AI browser automation framework

github.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Show HN: Magnitude – Open-source AI browser automation framework

#21

Post by TheTaytay »

anerli wrote: post #14
rozap wrote: post #12

There are a number of these out there, and this one has a super easy setup and appears to Just Work, so nice job on that. I had it going and producing plausible results within a minute or so. One thing I'm wondering is if there's anyone doing this at scale? The issue I see is that with complex workflows which take several dozen steps and have complex control flow, the probability of reaching the end falls off pretty…

Glad you were able to get it set up quickly! We currently are optimizing for reliability and quality, which is why we suggest Claude - but it can get expensive in some cases. Using Qwen 2.5-VL-72B will be significantly cheaper, though may not be always reliable. Most of our usage right now is for running test cases, and people seem to often prefer qwen for that use case - since typically test cases are clearer how to…

I am desperately waiting for someone to write exactly this! Use the LLM to write the repeatable, robust script. If the script fails, THEN fall back to an LLM to recover and fix the script.

Re: Show HN: Magnitude – Open-source AI browser automation framework

#23

Post by adenta »

anerli wrote: post #16
plufz wrote: post #15

Earlier quoted context omitted.

But we can use an LLM to write that script though and give that agent access to a browser to find DOM selectors etc. And than we have a stable script where we, if needed, manually can fix any LLM bugs just once…? I’m sure there are use cases with messy selectors as you say, but for me it feels like most cases are better covered by generating scripts.

Yeah we've though about this approach a lot - but the problem is if your final program is a brittle script, you're gonna need a way to fix it again often - and then you're still depending on recurrently using LLMs/agents. So we think its better to have the program itself be resilient to change instead of you/your LLM assistant having to constantly ensure the program is working.

I wonder if a nice middle ground would be: - recording the playwright behind the scenes and storing - trying that as a “happy path” first attempt to see if it passes - if it doesn’t pass, rebuilding it with the AI and vision models

Best of both worlds. The playwright is more of a cache than a test

Re: Show HN: Magnitude – Open-source AI browser automation framework

#24

Post by anerli »

TheTaytay wrote: post #21
anerli wrote: post #14

Earlier quoted context omitted.

Glad you were able to get it set up quickly! We currently are optimizing for reliability and quality, which is why we suggest Claude - but it can get expensive in some cases. Using Qwen 2.5-VL-72B will be significantly cheaper, though may not be always reliable. Most of our usage right now is for running test cases, and people seem to often prefer qwen for that use case - since typically test cases are clearer how to…

I am desperately waiting for someone to write exactly this! Use the LLM to write the repeatable, robust script. If the script fails, THEN fall back to an LLM to recover and fix the script.

Yeah, I think its a little tricky to do this well + automatically but is essentially our goal - not necessarily literally writing a script but storing the actions taken by the LLM and being able to repeat them, and adapt only when needed

Re: Show HN: Magnitude – Open-source AI browser automation framework

#25

Post by dataviz1000 »

Hey guys, I got a question.

I've been working on a Chrome extension with a side panel. Think about it like the side panel copilot in VSCode, Cursor, or Windsurf. Currently it is automating workflows but those are hard coded. I've started working on a more generalized automation using langchain. Looking at your code is helpful because I can in only a few hundred lines of code recreate a huge portion Playwright's capabilities in a Chrome extension side panel so I should be able to port it to the Chrome extension. That is, I'm creating a tools like mouse click, type, mouse move, open tab, navigate, wait for element, ect..

Looking at your code, I'm thinking about pulling anything that isn't coupled to node while mapping all the Playwright capabilities to the equivalent in a Chrome extension. It's busy work.

If I do that why would I prefer using .baml over the equivalent langchain? What's the differnce? Am I'm comparing apples to oranges? I'm not worried about using langgraph because I should be able to get most of the functionality with xstate v5 [0] plus serialized portable JSON state graphs so I can store custom graphs on a remote server that can be queried by API.

That is my question. I don't see langchain in the dependencies which is cool, but why .baml? Also, what am I'm missing going down this thought path?

[0] https://chatgpt.com/share/685dfc60-106c-8004-bbd0-1ba3a33aba...

Re: Show HN: Magnitude – Open-source AI browser automation framework

#27

Post by anerli »

adenta wrote: post #23
anerli wrote: post #16

Earlier quoted context omitted.

Yeah we've though about this approach a lot - but the problem is if your final program is a brittle script, you're gonna need a way to fix it again often - and then you're still depending on recurrently using LLMs/agents. So we think its better to have the program itself be resilient to change instead of you/your LLM assistant having to constantly ensure the program is working.

I wonder if a nice middle ground would be: - recording the playwright behind the scenes and storing - trying that as a “happy path” first attempt to see if it passes - if it doesn’t pass, rebuilding it with the AI and vision models Best of both worlds. The playwright is more of a cache than a test

I think the difficulty with this approach is (1) you want a good "lookup" mechanism - given a task, how do you know what cache should be loaded? you can do a simple string lookup based on the task content, but when the task might include parameters or data, or be a part of a bigger workflow, it gets trickier. (2) you need a good way to detect when to adapt / fall back to the LLM. When the cache is only a playwright script, it can be difficult to know when it falls out of the existing trajectory. You can check for selector timeouts and things, but you might be missing a lot of false negatives.

Re: Show HN: Magnitude – Open-source AI browser automation framework

#29

Post by anerli »

dataviz1000 wrote: post #25

Hey guys, I got a question. I've been working on a Chrome extension with a side panel. Think about it like the side panel copilot in VSCode, Cursor, or Windsurf. Currently it is automating workflows but those are hard coded. I've started working on a more generalized automation using langchain. Looking at your code is helpful because I can in only a few hundred lines of code recreate a huge portion Playwright's capab…

Hey, curious about your use cases for a chrome extension, care to share more?

To answer your question - BAML is as DSL that helps to define prompts, organize context, and to get better performance on structured output from the LLM. In theory you should be able to map over similar logic to other clients.

Re: Show HN: Magnitude – Open-source AI browser automation framework

#30

Post by pzo »

anerli wrote: post #29
dataviz1000 wrote: post #25

Hey guys, I got a question. I've been working on a Chrome extension with a side panel. Think about it like the side panel copilot in VSCode, Cursor, or Windsurf. Currently it is automating workflows but those are hard coded. I've started working on a more generalized automation using langchain. Looking at your code is helpful because I can in only a few hundred lines of code recreate a huge portion Playwright's capab…

Hey, curious about your use cases for a chrome extension, care to share more? To answer your question - BAML is as DSL that helps to define prompts, organize context, and to get better performance on structured output from the LLM. In theory you should be able to map over similar logic to other clients.

Chrome extension has advantage of user friendly distribution - so that non tech savy users can also do automation. I'm also looking for automation for mobile devices (app webview or safari mobile) and because of platform limitation also this doesn't seem can by anytime extended to mobile devices
Post reply on HN