What is the point of asking LLM to do manual testing? IMHO it would be much better to make it write automated tests. So you can just rerun them?
A new era for software testing
51–60 of 67 posts
Re: A new era for software testing
#52Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
I heard people say this before. I'm wondering, how do you instruct the LLM to generate the tests? Do you tell it the scenarios that would be covered, or do you just tell it to write tests for the code?
>cover all edge cases
That’s probably the extent of the prompt.
Re: A new era for software testing
#53If you are working with a web application, playwright + frontier LLM is incredibly capable. They added some recent features to make this sort of use case go a lot smoother:
https://playwright.dev/docs/release-notes#version-159
If you set this up correctly, you can have a main agent issue natural language testing instructions to this playwright agent which returns a natural language summary of what it experiences. This is the sort of thing where I begin to get interested in the idea of agents working while I sleep.
Re: A new era for software testing
#54Earlier quoted context omitted.
Agents read the docs. People don't. That's the underlying reason.
> People don't. People falling all over themselves to write docs for their pile-of-linear-algebra-with-a-smiley-face-painted-on-it [0] don't read the docs, no. People who give a shit about writing solid software that doesn't get them paged at three in the damn morning do. [0] The face is there to provide social-trustworthiness signals to engage the human pack-bonding instinct, natch.
Your sarcasm is unwarranted, because what I said is true and reflects the experience of a lot of people.
A decade ago I left a job and spent the last week thoroughly documenting every flow and code section of an app that I worked with, which was the core value proposition of the company. A couple years later I ask around and nobody even took a look at that.
People just don't read, and there are actually good reasons for that, one of them being that documentation is outdated in most orgs and the effort to keep it up to date is greater than reading the code.
Re: A new era for software testing
#55Earlier quoted context omitted.
Rust's zero cost abstractions mainly stem from its affine ownership model managing memory lifetimes safely and correctly with zero cost - as that is the killer feature... That's what I do. When people think of "zero cost" they don't think about std::optional. They think about not having to manage memory lifetimes AND NOT having to pay for a Garbage Collector to do it for you. That was always the trade you made until…
> affine ownership huh? you can rotate and scale the ownership?
Affine as in substructural linear types. They correspond to linear logic [0], and affine logic is named such because the way it's defined corresponds to affine functions. You don't literally need to scale your pointers though.
Re: A new era for software testing
#56[dead]
Re: A new era for software testing
#57Earlier quoted context omitted.
> People don't. People falling all over themselves to write docs for their pile-of-linear-algebra-with-a-smiley-face-painted-on-it [0] don't read the docs, no. People who give a shit about writing solid software that doesn't get them paged at three in the damn morning do. [0] The face is there to provide social-trustworthiness signals to engage the human pack-bonding instinct, natch.
Your sarcasm is unwarranted, because what I said is true and reflects the experience of a lot of people. A decade ago I left a job and spent the last week thoroughly documenting every flow and code section of an app that I worked with, which was the core value proposition of the company. A couple years later I ask around and nobody even took a look at that. People just don't read, and there are actually good reasons…
> ...what I said is true and reflects the experience of a lot of people.
Wow. What I said is true and reflects the experience of a lot of people. Amazing!
Re: A new era for software testing
#58Scenario testing is the new word for it and I think this is a game changer. Two of the reasons I never liked writing tests is - they didn’t seem to usually assert much internal logic - they would have to be maintained along with the original code I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot. So imagine I’m making an…
>Scenario testing is the new word How is scenario different from a behavior (as in Behavior-Driven Development)? Gherkin and things like Cucumber are not something new, are they?
Well, scenario contains the letters s, c and n, while behavior doesn't.
Re: A new era for software testing
#59Earlier quoted context omitted.
Your sarcasm is unwarranted, because what I said is true and reflects the experience of a lot of people. A decade ago I left a job and spent the last week thoroughly documenting every flow and code section of an app that I worked with, which was the core value proposition of the company. A couple years later I ask around and nobody even took a look at that. People just don't read, and there are actually good reasons…
> ...what I said is true and reflects the experience of a lot of people. Wow. What I said is true and reflects the experience of a lot of people. Amazing!
That's nice, but if you write docs and people don't read them then that's clearly not a winning strategy in many orgs.
Re: A new era for software testing
#60Earlier quoted context omitted.
I heard people say this before. I'm wondering, how do you instruct the LLM to generate the tests? Do you tell it the scenarios that would be covered, or do you just tell it to write tests for the code?
>cover all edge cases That’s probably the extent of the prompt.
I have only one thing to say....
what