Earlier quoted context omitted.
You also must be very confident in your own ability if you don’t think that at least some of the things you’re doing that you’d classify as “skill at using the tool” aren’t just superstitions à la Skinner’s pigeons.
I'm sure a lot of them are superstitions! I've written about that before: https://simonwillison.net/2023/Aug/27/wordcamp-llms/#superst... One of the more "engineering" like skills in using this stuff is methodically figuring out what's a superstition and what actually works.
> One of the more "engineering" like skills in using this stuff is methodically figuring out what's a superstition and what actually works.
The problem is there are so many variables and the system is so chaotic that this is a nearly impossible task for things that don’t have an absolutely enormous effect size.
For most things you’re testing, you need to run the experiment many many times to get any kind is statistically significant result, which rules out manual review.
And since we have tried and failed to develop objective code quality metrics, you’re left with metrics like “does this pass the automated test or not!”, but that doesn’t tell you whether the code is any good, or whether it is overfitting the test suite. Then when a new model comes out, you have to scrap your results and run your experiments all over. This is engineering of the laws of physics were constantly changing, and I lived in that universe, I think I’d take my ball and go home.
There's always been a bit of magic to being a programmer, and if you look at the cover of SICP people like to imagine that they are wizards or alchemists. But "vibe engineering" moves that to a whole new level. You're a wizard mixing up gunpowder and sacrificing chickens to fire spirits before you light it. It's not engineering because unless the models fundamentally change you'll never be able to really sort the science from the superstition. Software engineering already had too much superstition for my taste, but we're at a whole new level now.