Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?

#12

Post by harikshore »

Answering your questions from my knowledge level on AI:

Managing AI agents is like running a knowledge business. If you are not an expert enough to understand what agents are doing, there is still large room for errors. Its important to set design decisions early in the development process.

Your questions:

3. I think this is absurd. Agent can generate the test cases, but you still have to verify the tests are actually valid.

4. The architecture document should be the source of truth. SO when a engineer reviews they know what to look for & flag/fix. The comprehensive the document is the better. Version it based on what the prototype carries. — Data models, API design, Services, Roles, etc. You can ask this to AI — What all should a high/low level SDD carry?

5. Just make sure the architecture doc is solid, development is as per the document, ask AI to generate/perform regression test cases and document them as well (pass/fail).

So the engineer knows where things exactly stand.

Thats the best can do

Appreciate thoughts on this.

Re: Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?

#13

Post by thegreatkahuna »

taleodor wrote: post #10

If you're planning to put it into production anything you do outside of bringing somebody qualified at this point is likely a waste of time. Highly recommend this recent talk by Kelsey Hightower on the subject - https://www.youtube.com/watch?v=A7WFt2JQ5sg

Thanks for the link! You could very well be right that there is not much more that could be done before the review. My reading of what Kelsey said is that agents can produce useful outcomes, it is just that humans need to validate and own them. In other words, I couldn't conclude that this somehow disproves that at least some of the functionalities that were generated in my project, couldn't be (close to) production-grade.

Re: Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?

#14

Post by taleodor »

thegreatkahuna wrote: post #13
taleodor wrote: post #10

If you're planning to put it into production anything you do outside of bringing somebody qualified at this point is likely a waste of time. Highly recommend this recent talk by Kelsey Hightower on the subject - https://www.youtube.com/watch?v=A7WFt2JQ5sg

Thanks for the link! You could very well be right that there is not much more that could be done before the review. My reading of what Kelsey said is that agents can produce useful outcomes, it is just that humans need to validate and own them. In other words, I couldn't conclude that this somehow disproves that at least some of the functionalities that were generated in my project, couldn't be (close to) production-…

This is more about understanding the big picture and architecture of what's going on (the part about the diagrams / blueprints). One practical thing actually that may be useful to you - try thinking and testing in terms of "what can go wrong" rather than "is this prod-ready".

Re: Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?

#15

Post by thegreatkahuna »

harikshore wrote: post #12

Answering your questions from my knowledge level on AI: Managing AI agents is like running a knowledge business. If you are not an expert enough to understand what agents are doing, there is still large room for errors. Its important to set design decisions early in the development process. Your questions: 3. I think this is absurd. Agent can generate the test cases, but you still have to verify the tests are actuall…

I agree that in perfect world I could have done this exercise with competent engineers and a UX desginer. I know about software development processes and applied that to e.g. testing and hardening, but it is not a substitute for deep technical knowledge. THe idea behind having a separate agent create the tests is to that then the coding agent couldn't get functionality to work or fix bugs by weakening the test. I believe Bob Martin has talked about this. Noted about the architecture document - will have to ensure it is as solid as possible. I already run the full CI pipeline that we run for our app + the new ones that the agents developed - would it be good to have additional regression tests?

Re: Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?

#18

Post by cyanydeez »

I'd do two things before any code changes:

1. Ask the AI to document the code base; either in a single repo or in a sidebyside like .md; then scale up with larger documents for functionality/submodules etc. This will get you and your team to properly contemplate the monstrosity.

2. Once you have documentation, then you write tests; unit tests, e2e, etc. Make sure everything is passing the tests.

After that, you'll hopefully both gained proper insight and solidified functionality. Then you can port whatever you like.

Post reply on HN