I think the main issue with this approach is that your solution space eventually narrows. You go from formulating unique solutions to flagging things that look wrong and then just picking an alternative LLM generated one. I’ve noticed this recently as all my side project work is being done by hand. If I do ask an LLM another question about something small it will offer solutions but doesn’t offer the solution I think…
I've noticed that Claude with the current model can act most of the time like a trusted senior engineer sounding board for me. However on occasions we disagree. Recently we had a stand up argument about a proposed solution to a potential security risk in a system. I realized Claude had taken on a kind of "corporate culture" thinking on the problem (which was to do with use of a vault/broker to hold user credential data) that would be appropriate in some contexts, but wasn't in ours. The experience was totally like arguing with someone on the corporate Slack, but in the end I control what gets deployed so I won. But I'd say 90% of the time I get as much value out of Claude on these discussions as I would if I knew one competent person for each of the subjects we discuss, and had called that person for advice. Of course even though my network is...vast...I just don't know a competent person in every single field.