Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Magnitude – open-source, AI-native test framework for web apps

github.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#11

Post by tough »

anerli wrote: post #9
tough wrote: post #5

Earlier quoted context omitted.

You might like https://pypi.org/project/llm-predictive-router/

Oh this is interesting. In our case we are being very specific about which types of prompts go where, so the planner essentially creates prompts that will be executed by Moondream, instead of trying to route prompts generally to the appropriate model. The types of requests that our planner agent vs Moondream can handle are fundamentally different for our use case.

interesting, will check out yours i'm mostly interested on these dynamic routers so I can mix local and API based depending on needs, i cannot run some models locally but most of the tasks don't even require such power (on building ai agentic systems)

there's also https://github.com/lm-sys/RouteLLM

and other similar

I guess your system is not as open-ended task oriented so you can just build workflows deciding which model to use at each step, these routing mechanisms are more useful for open-ended tasks that dont fit on a workflow so well (maybe?)

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#12

Post by anerli »

tobr wrote: post #8

Interesting! My first concern is - isn’t this the ultimate non-deterministic test? In practice, does it seem flaky?

So the architecture is built with determinism in mind. The plan-caching system is still a work in progress, but especially once fully implemented it should be very consistent. As long as your interface doesn't change (or changes in trivial ways), Moondream alone can execute the same exact web actions as previous test runs without relying on any DOM selectors. When the interface does eventually change, that's where it becomes non-deterministic again by necessity, since the planner will need to generatively update the test and continue building the new cache from there. However once it's been adapted, it can once again be executed that way every time until the interface changes again.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#13

Post by anerli »

NitpickLawyer wrote: post #7

> The idea is the planner builds up a general plan which the executor runs. We can save this plan and re-run it with only the executor for quick, cheap, and consistent runs. When something goes wrong, it can kick back out to the planner agent and re-adjust the test. I've been recently thinking about testing/qa w/ VLMs + LLMs, one area that I haven't seen explored (but should 100% be feasible) is to have the first run…

So this is a path that we definitely considered. However we think its a half-measure to generate actual Playwright code and just run that. Because if you do that, you still have a brittle test at the end of the day, and once it breaks you would need to pull in some LLM to try and adapt it anyway.

Instead of caching actual code, we cache a "plan" of specific web actions that are still described in natural language.

For example, a cached "typing" action might look like: { variant: 'type'; target: string; content: string; }

The target is a natural language description. The content is what to type. Moondream's job is simply to find the target, and then we will click into that target and type whatever content. This means it can be full vision and not rely on DOM at all, while still being very consistent. Moondream is also trivially cheap to run since it's only a 2B model. If it can't find the target or it's confidence changed significantly (using token probabilities), it's an indication that the action/plan requires adjustment, and we can dynamically swap in the planner LLM to decide how to adjust the test from there.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#14

Post by dimal »

> Pure vision instead of error prone "set-of-marks" system (the colorful boxes you see in browser-use for example)

One benefit not using pure vision is that it's a strong signal to developers to make pages accessible. This would let them off the hook.

Perhaps testing both paths separately would be more appropriate. I could imagine a different AI agent attempting to navigate the page through accessibility landmarks. Or even different agents that simulate different types of disabilities.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#15

Post by badmonster »

anerli wrote: post #10
badmonster wrote: post #6

How does Magnitude differentiate between the planner and executor LLM roles, and how customizable are these components for specific test flows?

So the prompts that are sent to the planner vs executor are completely distinct. We allow complete customization of the planner LLM with all major providers (Anthropic, OpenAI, Google AI Studio, Google Vertex AI, AWS Bedrock, OpenAI compatible). The executor LLM on the other hand has to fit very specific criteria, so we only support the Moondream model right now. For a model to act as the executor it needs to be able…

does Moondream handle multi-step UI tasks reliably (like opening a menu, waiting for render, then clicking), or do you have to scaffold that logic separately in the planner?

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#16

Post by daxfohl »

anerli wrote: post #12
tobr wrote: post #8

Interesting! My first concern is - isn’t this the ultimate non-deterministic test? In practice, does it seem flaky?

So the architecture is built with determinism in mind. The plan-caching system is still a work in progress, but especially once fully implemented it should be very consistent. As long as your interface doesn't change (or changes in trivial ways), Moondream alone can execute the same exact web actions as previous test runs without relying on any DOM selectors. When the interface does eventually change, that's where it…

In a way, nondeterminism could be an advantage. Instead of using these as unit tests, use them as usability tests. Especially if you want your site to be accessible by AI agents, it would be good to have a way of knowing what tweaks increase the success rate.

Of course that would be even more valuable for testing your MCP or A2A services, but could be useful for UI as well. Or it could be useless. It would be interesting to see if the same UI changes affect both human and AI success rate in the same way.

And if not, could an AI be trained to correlate more closely to human behavior. That could be a good selling point if possible.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#17

Post by anerli »

dimal wrote: post #14

> Pure vision instead of error prone "set-of-marks" system (the colorful boxes you see in browser-use for example) One benefit not using pure vision is that it's a strong signal to developers to make pages accessible. This would let them off the hook. Perhaps testing both paths separately would be more appropriate. I could imagine a different AI agent attempting to navigate the page through accessibility landmarks. O…

Yeah good criticism for sure. We definitely want to keep this in mind as we continue to build. Some kind of accessibility tests which run in parallel with each visual test that are only allowed to use the accessibility tree could make it much easier for developers to identify how to address different accessibility concerns.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#18

Post by anerli »

daxfohl wrote: post #16
anerli wrote: post #12

Earlier quoted context omitted.

So the architecture is built with determinism in mind. The plan-caching system is still a work in progress, but especially once fully implemented it should be very consistent. As long as your interface doesn't change (or changes in trivial ways), Moondream alone can execute the same exact web actions as previous test runs without relying on any DOM selectors. When the interface does eventually change, that's where it…

In a way, nondeterminism could be an advantage. Instead of using these as unit tests, use them as usability tests. Especially if you want your site to be accessible by AI agents, it would be good to have a way of knowing what tweaks increase the success rate. Of course that would be even more valuable for testing your MCP or A2A services, but could be useful for UI as well. Or it could be useless. It would be interes…

Originally we were actually thinking about doing exactly this and building agents for usability testing. However, we think that LLMs are much better suited for tackling well defined tasks rather than trying to emulate human nuance, so we pivoted to end-to-end testing and figuring out how to make LLM browser agents act deterministically.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#19

Post by anerli »

badmonster wrote: post #15
anerli wrote: post #10

Earlier quoted context omitted.

So the prompts that are sent to the planner vs executor are completely distinct. We allow complete customization of the planner LLM with all major providers (Anthropic, OpenAI, Google AI Studio, Google Vertex AI, AWS Bedrock, OpenAI compatible). The executor LLM on the other hand has to fit very specific criteria, so we only support the Moondream model right now. For a model to act as the executor it needs to be able…

does Moondream handle multi-step UI tasks reliably (like opening a menu, waiting for render, then clicking), or do you have to scaffold that logic separately in the planner?

The planner can plan out multiple web actions at once, which Moondream can then execute in sequence on its own. So Moondream is never deciding how to execute more than one web action in a single prompt.

What this really means for developers writing the tests is you don't really have to worry about it. A "step" in Magnitude can map to any number of web actions dynamically based on the description, and the agents will figure out how to do it repeatably.

Post reply on HN