Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Magnitude – open-source, AI-native test framework for web apps

github.com

Post reply on HN
Print view View original

1–10 of 45 posts

Show HN: Magnitude – open-source, AI-native test framework for web apps

#1

Post by anerli »

Hey HN, Anders and Tom here - we’ve been building an end-to-end testing framework powered by visual LLM agents to replace traditional web testing.

We know there's a lot of noise about different browser agents. If you've tried any of them, you know they're slow, expensive, and inconsistent. That's why we built an agent specifically for running test cases and optimized it just for that:

- Pure vision instead of error prone "set-of-marks" system (the colorful boxes you see in browser-use for example)

- Use tiny VLM (Moondream) instead of OpenAI/Anthropic computer use for dramatically faster and cheaper execution

- Use two agents: one for planning and adapting test cases and one for executing them quickly and consistently.

The idea is the planner builds up a general plan which the executor runs. We can save this plan and re-run it with only the executor for quick, cheap, and consistent runs. When something goes wrong, it can kick back out to the planner agent and re-adjust the test.

It’s completely open source. Would love to have more people try it out and tell us how we can make it great.

Repo: https://github.com/magnitudedev/magnitude

Show HN: Magnitude – open-source, AI-native test framework for web apps
github.com

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#2

Post by grbsh »

I know moondream is cheap / fast and can run locally, but is it good enough? In my experience testing things like Computer Use, anything but the large LLMs has been so unreliable as to be unworkable. But maybe you guys are doing something special to make it work well in concert?

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#3

Post by anerli »

grbsh wrote: post #2

I know moondream is cheap / fast and can run locally, but is it good enough? In my experience testing things like Computer Use, anything but the large LLMs has been so unreliable as to be unworkable. But maybe you guys are doing something special to make it work well in concert?

So it's key to still have a big model that is devising the overall strategy for executing the test case. Moondream on its own is pretty limited and can't handle complex queries. The planner gives very specific instructions to Moondream, which is just responsible for locating different targets on the screen. It's basically just the layer between the big LLM doing the actual "thinking" and grounding that to specific UI interactions.

Where it gets interesting, is that we can save the execution plan that the big model comes up with and run with ONLY Moondream if the plan is specific enough. Then switch back out to the big model if some action path requires adjustment. This means we can run repeated tests much more efficiently and consistently.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#4

Post by grbsh »

anerli wrote: post #3
grbsh wrote: post #2

I know moondream is cheap / fast and can run locally, but is it good enough? In my experience testing things like Computer Use, anything but the large LLMs has been so unreliable as to be unworkable. But maybe you guys are doing something special to make it work well in concert?

So it's key to still have a big model that is devising the overall strategy for executing the test case. Moondream on its own is pretty limited and can't handle complex queries. The planner gives very specific instructions to Moondream, which is just responsible for locating different targets on the screen. It's basically just the layer between the big LLM doing the actual "thinking" and grounding that to specific UI…

Ooh, I really like the idea about deciding whether to use the big or small model based on task specificity.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#5

Post by tough »

grbsh wrote: post #4
anerli wrote: post #3

Earlier quoted context omitted.

So it's key to still have a big model that is devising the overall strategy for executing the test case. Moondream on its own is pretty limited and can't handle complex queries. The planner gives very specific instructions to Moondream, which is just responsible for locating different targets on the screen. It's basically just the layer between the big LLM doing the actual "thinking" and grounding that to specific UI…

Ooh, I really like the idea about deciding whether to use the big or small model based on task specificity.

You might like https://pypi.org/project/llm-predictive-router/

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#7

Post by NitpickLawyer »

> The idea is the planner builds up a general plan which the executor runs. We can save this plan and re-run it with only the executor for quick, cheap, and consistent runs. When something goes wrong, it can kick back out to the planner agent and re-adjust the test.

I've been recently thinking about testing/qa w/ VLMs + LLMs, one area that I haven't seen explored (but should 100% be feasible) is to have the first run be LLM + VLM, and then have the LLM(s?) write repeatable "cheap" tests w/ traditional libraries (playwright, puppeteer, etc). On every run you do the "cheap" traditional checks, if any fail go with the LLM + VLM again and see what broke, only fail the test if both fail. Makes sense?

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#9

Post by anerli »

tough wrote: post #5
grbsh wrote: post #4

Earlier quoted context omitted.

Ooh, I really like the idea about deciding whether to use the big or small model based on task specificity.

You might like https://pypi.org/project/llm-predictive-router/

Oh this is interesting. In our case we are being very specific about which types of prompts go where, so the planner essentially creates prompts that will be executed by Moondream, instead of trying to route prompts generally to the appropriate model. The types of requests that our planner agent vs Moondream can handle are fundamentally different for our use case.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#10

Post by anerli »

badmonster wrote: post #6

How does Magnitude differentiate between the planner and executor LLM roles, and how customizable are these components for specific test flows?

So the prompts that are sent to the planner vs executor are completely distinct. We allow complete customization of the planner LLM with all major providers (Anthropic, OpenAI, Google AI Studio, Google Vertex AI, AWS Bedrock, OpenAI compatible). The executor LLM on the other hand has to fit very specific criteria, so we only support the Moondream model right now. For a model to act as the executor it needs to be able to specific specific pixel coordinates (only a few models support this, for example OpenAI/Anthropic computer use, Molmo, Moondream, and some others). We like Moondream because its super tiny and fast (2B). This means as long as we still have a "smart" planner LLM we can have very fast/cheap execution and precise UI interaction.
Post reply on HN