Thanks for your reply, beijbom.
I believe it is a very common mistake for intelligent people to assume that others will behave at least reasonable. But in my experience, when people do AI without understanding it, all bets are off.
"Sure, if they throw in a picture from anywhere else, they are toast, but why would they?" Since you list a Barcodeless Scanner as an example, the manufacturer of strawberries might run a promotion for blueberries on their box. For a non-expert user, it is unimaginable that a model trained on 3D blueberries might be triggered by a 2D photo of blueberries.
Also, I'm going to go with your legume example. As soon as each new truck arrives, the intern runs out and takes photos of the legumes in their boxes for the AI training. He uploads the images to your website and trains a model. TADA! The model is deployed to production and starts causing issues. But the people working alongside the fancy new celebrated machine don't want to lose their job, so they silently fix what's going wrong. You've just reduced productivity by introducing a costly machine.
Turns out, the different suppliers arrive at different times of day, so the lighting is different. And different suppliers use different box types. But without expert domain knowledge, you wouldn't even consider that this might be a problem. Also, why do you assume the customer will verify their model on independently sampled production data? To someone lacking the domain knowledge, using the exact same set of photos for training and for verification seems just fine. Actually, it's a lot less work that way.
That's what I tried to get at with my blind driver analogy. An untrained person will do things that seem absurdly unreasonable to us. But to them, it's the logical choice. They lack the knowledge to properly understand why what they are doing might be problematic.
Based on your description, however, it sounds like you (and your team of experts) are actively working with this customer and giving them feedback on what to do and how to do it. Have you considered making that part of your offering?
"Use Nyckel to integrate state of the art machine learning into your application. Anyone can curate their data set with our ML platform. A quick chat with an experienced AI engineer helps identify the best model and training procedure for your use case. It only takes minutes to finish your first model. Once created, your functions can be invoked in real-time using our API."
I'm pretty sure any serious business user would be happy to spend $100 for a 15 minute chat with someone that checks that their data is OK and their approach is reasonable. And it's also a nice way to segment out those that'll never become paid users anyway.