Show HN: Magnitude – open-source, AI-native test framework for web apps • Hacker News Board

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#41

Post by engfan » Sat, Apr 26, 2025, 1:23 PM UTC

anerli wrote: post #12

tobr wrote: post #8

Interesting! My first concern is - isn’t this the ultimate non-deterministic test? In practice, does it seem flaky?

So the architecture is built with determinism in mind. The plan-caching system is still a work in progress, but especially once fully implemented it should be very consistent. As long as your interface doesn't change (or changes in trivial ways), Moondream alone can execute the same exact web actions as previous test runs without relying on any DOM selectors. When the interface does eventually change, that's where it…

Anerli wrote: “When the interface does eventually change, that's where it becomes non-deterministic again by necessity, since the planner will need to generatively update the test and continue building the new cache from there.”

But what determines that the UI has changed for a specific URL? Your software independent of the planner LLM or do you require the visual LLM to make a determination of change?

You should also stop saying 100% open source when test plan generation and execution depend on non-open source AI components. It just doesn’t make sense.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#42

Post by anerli » Sat, Apr 26, 2025, 3:34 PM UTC

arendtio wrote: post #38

It looks pretty cool. One thing that has bothered me a bit with Playwright is audio input. With modern AI applications, speech recognition is often integrated, but with Playwright, using voice as an input does not seem straightforward. Given that Magnitude has an AI focus, adding a feature like that would be great: test('can log in and see correct settings') .step('log in to the app') .say('my username is user@exampl…

Huh that’s an interesting use case. Yeah using an AI driven system definitely opens up some cool possibilities that aren’t possible with playwright alone. Would be curious to hear more about what you’re trying to test this way with audio.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#43

Post by anerli » Sat, Apr 26, 2025, 3:36 PM UTC

retreatguru wrote: post #40

Any advice about using ai to write test cases? For example recording a video while using an app and converting that to test cases. Seems like it should work.

This is definitely top of mind for us! A lot of ways to potentially approach it. We want to make sure the test case execution works really well so our focus is there but also want to think about test case generation going forward. Recording a video especially with small VLMs that can tokenize videos would be super neat.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#44

Post by anerli » Sat, Apr 26, 2025, 3:41 PM UTC

engfan wrote: post #41

anerli wrote: post #12

Earlier quoted context omitted.

So the architecture is built with determinism in mind. The plan-caching system is still a work in progress, but especially once fully implemented it should be very consistent. As long as your interface doesn't change (or changes in trivial ways), Moondream alone can execute the same exact web actions as previous test runs without relying on any DOM selectors. When the interface does eventually change, that's where it…

Anerli wrote: “When the interface does eventually change, that's where it becomes non-deterministic again by necessity, since the planner will need to generatively update the test and continue building the new cache from there.” But what determines that the UI has changed for a specific URL? Your software independent of the planner LLM or do you require the visual LLM to make a determination of change? You should als…

The small VLM (Moondream) decides when interface changes / its actions no longer line up.

We say 100% open source because all of our code (test runner and AI agents) is completely open source. It’s also completely possible to run an entire OSS stack because you can configure with an open source planner LLM, and Moondream is open source. You could run it all locally even if you have solid hardware.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#45

Post by arendtio » Sun, Apr 27, 2025, 7:05 AM UTC

anerli wrote: post #42

arendtio wrote: post #38

It looks pretty cool. One thing that has bothered me a bit with Playwright is audio input. With modern AI applications, speech recognition is often integrated, but with Playwright, using voice as an input does not seem straightforward. Given that Magnitude has an AI focus, adding a feature like that would be great: test('can log in and see correct settings') .step('log in to the app') .say('my username is user@exampl…

Huh that’s an interesting use case. Yeah using an AI driven system definitely opens up some cool possibilities that aren’t possible with playwright alone. Would be curious to hear more about what you’re trying to test this way with audio.

I have this project planning prototype [1], where you can use audio to plan a project, like' Please create a project plan for an example project and make assumptions' or 'During the last two weeks of the year, nobody is available due to the holidays, please adjust the resource availability'.

To test this, you need an openai api key and add it in the settings (it will be stored in your browser's localstorage). After that, you can use the microphone icon in the ribbon menu (press it once to start the recording, press it again to stop the recording, and the processing begins).

You can also test most things via text input in this app, but for example, I have another app for kids that supports audio input only. There, the kid can say 'I want to learn about apple trees' and the system creates apple tree content ;-) However, it also has some content filters to allow only content suited for certain age levels. That is something you might want to include in automated tests.

[1]: https://critically.app