Earlier quoted context omitted.
No you don’t. I can feed images of crazy driving scenarios into LLaVA and get reasonable responses. That’s a general purpose LLM with $500 worth of fine tuning running locally on my PC. You should look into what can be done with the current state of the art LLMs. Your intuition for what’s possible is out of date. If I can do that with open source LLaMA variants, I can only imagine what’s possible if you have an actua…
You getting six nines of accuracy on that with good latency? Did you watch the “how our large driving model deals with stop signs” from Tesla AI department? Given the multiplicative effect of driving decisions and the weird real world out there, it has be extremely reliable and robust to be a good driver as the miles mount up.
LLMs with vision work completely differently. You’re leveraging the world model, built from a terabyte of text data, to aid your classification. The classic example of an image they handle well is a man ironing clothes on the back of a taxi. Where traditional image classifiers wouldn’t have a hope of handling that, vision LLMs describe it with ease.