Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Adept Experiments – use AI workflows to delegate repetitive tasks

adept.ai

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Introducing Adept Experiments – use AI workflows to delegate repetitive tasks

#11

Post by jaggederest »

tmcneal wrote: post #9
koreth1 wrote: post #8

Earlier quoted context omitted.

That's one aspect of reliability, but the one I was more curious about was determinism. If I repeatedly run the same test suite on the same code base and the same data and configuration, am I guaranteed to get the same test results every time, or is it possible for ai() to change its mind about what actions to take?

Ah got it. So GPT is non-deterministic, but we somewhat handle that by having a caching layer in our AI. Basically if you make an ai() call, and we see that the page state is identical to a previous invocation of that exact AI prompt, then we will not consult the AI and install return you the cached result. We did this mainly to reduce costs and speed up execution of the 2nd-to-nth run of the same test, but it does m…

That makes a lot of sense, thank you for the explanation, I will have to explore this the next time I am building page tests. Have considered doing it myself but much happier using a relatively inexpensive product than maintaining the creaky homebuild version.

Re: Introducing Adept Experiments – use AI workflows to delegate repetitive tasks

#13

Post by jasonjmcghee »

tmcneal wrote: post #3

For anyone looking to try this in an E2E testing context, we just released a library for Playwright called ZeroStep ( https://zerostep.com/ ) that lets you script AI based actions, assertions, and extractions. This is a working example that tests the core "book a meeting" workflow in Calendly: import { test, expect } from '@playwright/test' import { ai } from '@zerostep/playwright' test.describe('Calendly', () => { t…

It would be much easier to consider this as solution if it would _output_ the generated test steps, and/or cache them and only modify them if needed.

Your example above - 7 function calls in one test. let's say usually closer to 5, we have hundreds of tests. Every single PR runs E2E tests. We open a handful of PRs a day. Let's call it 5. We're already looking at thousands of invocations a day. Based on your pricing, that would be incredibly expensive.

This is with 3 eng.

Re: Introducing Adept Experiments – use AI workflows to delegate repetitive tasks

#14

Post by tmcneal »

msoad wrote: post #7
tmcneal wrote: post #3

For anyone looking to try this in an E2E testing context, we just released a library for Playwright called ZeroStep ( https://zerostep.com/ ) that lets you script AI based actions, assertions, and extractions. This is a working example that tests the core "book a meeting" workflow in Calendly: import { test, expect } from '@playwright/test' import { ai } from '@zerostep/playwright' test.describe('Calendly', () => { t…

Nice! I'm going to try this out! Nit: For me, it would be nicer if `ai` was a fixture itself. test.describe('Calendly', ({ ai }) => {

Done! We added the ability to use it as a fixture. Documented here: https://github.com/zerostep-ai/zerostep#playwright-fixture

Re: Introducing Adept Experiments – use AI workflows to delegate repetitive tasks

#15

Post by ushakov »

tmcneal wrote: post #3

For anyone looking to try this in an E2E testing context, we just released a library for Playwright called ZeroStep ( https://zerostep.com/ ) that lets you script AI based actions, assertions, and extractions. This is a working example that tests the core "book a meeting" workflow in Calendly: import { test, expect } from '@playwright/test' import { ai } from '@zerostep/playwright' test.describe('Calendly', () => { t…

Does it send the webpage contents to ZeroStep?

Cool demo btw.

Post reply on HN