Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Magnitude – open-source, AI-native test framework for web apps

github.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#31

Post by anerli »

ekzy wrote: post #24
anerli wrote: post #13

Earlier quoted context omitted.

So this is a path that we definitely considered. However we think its a half-measure to generate actual Playwright code and just run that. Because if you do that, you still have a brittle test at the end of the day, and once it breaks you would need to pull in some LLM to try and adapt it anyway. Instead of caching actual code, we cache a "plan" of specific web actions that are still described in natural language. Fo…

Did you consider also caching the coordinates returned by moondream? I understand that it is cheap, but it could be useful to detect if an element has changed position as it may be a regression

So the problem is if we cache the coordinates and click blindly at the saved positions, there's no way to tell if the interface changes or if we are actually clicking the wring things (unless we try and do something hacky like listen for events on the DOM). Detecting whether elements have changed position though would definitely be feasible if re-running a test with Moondream, could compared against the coordinates of the last run.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#32

Post by tomatohs »

NitpickLawyer wrote: post #7

> The idea is the planner builds up a general plan which the executor runs. We can save this plan and re-run it with only the executor for quick, cheap, and consistent runs. When something goes wrong, it can kick back out to the planner agent and re-adjust the test. I've been recently thinking about testing/qa w/ VLMs + LLMs, one area that I haven't seen explored (but should 100% be feasible) is to have the first run…

This is exactly our workflow, though we defined our own YAML spec [1] for reasons mentioned in previous comments.

We have multiple fallbacks to prevent flakes; The "cheap" command, a description of the intended step, and the original prompt.

If any step fails, we fall back to the next source.

1. https://docs.testdriver.ai/reference/test-steps

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#34

Post by chrisweekly »

anerli wrote: post #31
ekzy wrote: post #24

Earlier quoted context omitted.

Did you consider also caching the coordinates returned by moondream? I understand that it is cheap, but it could be useful to detect if an element has changed position as it may be a regression

So the problem is if we cache the coordinates and click blindly at the saved positions, there's no way to tell if the interface changes or if we are actually clicking the wring things (unless we try and do something hacky like listen for events on the DOM). Detecting whether elements have changed position though would definitely be feasible if re-running a test with Moondream, could compared against the coordinates o…

sounds a lot like snapshot testing

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#35

Post by chrisweekly »

This looks pretty cool, at least at first glance. I think "traditional web testing" means different things to different people. Last year, the Netflix engineering team published "SafeTest"[1] an interesting hybrid / superset of unit and e2e testing. Have you guys (Magnitude devs) considered incorporating any of their ideas?

1. https://netflixtechblog.com/introducing-safetest-a-novel-app...

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#36

Post by anerli »

aoeusnth1 wrote: post #33

Why not make the strong model compile a non-ai-driven test execution plan using selectors / events? Is Moondream that good?

Definitely a good question. Using an actual LLM as the execution layer allows us to more easily swap to the planner agent in the case that the test needs to be adapted. We don’t want to store just a selector based test because it’s difficult to determine when it requires adaptation, and is inherently more brittle to subtle UI changes. We think using a tiny model like Moondream makes this cheap enough that these benefits outweigh an approach where we cache actual playwright code.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#37

Post by anerli »

chrisweekly wrote: post #35

This looks pretty cool, at least at first glance. I think "traditional web testing" means different things to different people. Last year, the Netflix engineering team published "SafeTest"[1] an interesting hybrid / superset of unit and e2e testing. Have you guys (Magnitude devs) considered incorporating any of their ideas? 1. https://netflixtechblog.com/introducing-safetest-a-novel-app...

Looks cool! Thanks for sharing! The idea of having a hybrid framework for component unit testing + end to end testing is neat. Will definitely consider how this might be applicable to magnitude.

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#38

Post by arendtio »

It looks pretty cool. One thing that has bothered me a bit with Playwright is audio input. With modern AI applications, speech recognition is often integrated, but with Playwright, using voice as an input does not seem straightforward. Given that Magnitude has an AI focus, adding a feature like that would be great:

  test('can log in and see correct settings')
    .step('log in to the app')
      .say('my username is user@example.com')

Re: Show HN: Magnitude – open-source, AI-native test framework for web apps

#39

Post by o1o1o1 »

Thanks for sharing, this looks interesting.

However, I do not see a big advantage over Cypress tests.

The article mentions shortcomings of Cypress (and Playwright):

> They start a dev server with bootstrapping code to load the component and/or setup code you want, which limits their ability to handle complex enterprise applications that might have OAuth or a complex build pipeline.

The simple solution is to containerise the whole application (including whatever OAuth provider is used), which then allows you to simply launch the whole thing and then run the tests. Most apps (especially in enterprise) should already be containerised anyway, so most of the times we can just go ahead and run any tests against them.

How is SafeTest better than that when my goal is to test my application in a real world scenario?

Post reply on HN