Do you feel that you may be being a bit provocative by calling LLM's 'bullshit machines'? I understand the frustration as I've been bullshitted by these models just as much as the next programmer, but surely with recent advancements in RAG and reasoning, they're not just 'bullshit machines' at this point, are they?
It is based on the assumption LLMs make mistakes (they will!) but be confident about it. Use a 1B model and ask it about a city for fun mistakes in an otherwise impressive array of facts. Then you will see how bullshitty LLMs are.
Anyway the idea is to constrain the LLM and a language understanding and to choose a constrained response.
For example give it the capability to read out something from my calendar. Using the logits it gets to choose what that calendar item is. But then regular old code does the lookup and writes a canned sentence saying "At 6pm today you have a meeting titled $title".
This way, my meeting schedule won't make my LLM talk like a pirate.
This massively culls what the LLM can do but what it does do, is like a search and so just like Google (before gen AI!) you get a result but you can judge it as a human.