For my specific niche (medical imaging) all current models still suck. The amount of expert knowledge required to understand the data and display it in the right way - probably never was in the training set. We have this one performance-critical 3D reconstruction engine part, that just just has to go FAST through billions of voxels. From time to time we try to improve it, by just a bit. I have probably wasted at leas…
Did you feed back the results of the tests / benchmark to the model? I’m presuming you have a very robust test framework / benchmark setup etc? I’m presuming you fed the model the baseline results of that setup as a starting point ?
I have had put the agents in a small feedback loop - but limited to ~5 iterations, since I do not want to burn through my credits. Honestly, I think putting it in a loop for a case like this does not show the "intelligence" part of AI, it makes it more of a more-sophisticated random walk.