Sampling and structured outputs in LLMs
parthsareen.com
Sampling and structured outputs in LLMs
1–10 of 99 posts
Re: Sampling and structured outputs in LLMs
#2This post dives into that "black magic" layer, especially in the context of emerging thinking models and tools like Ollama or GPT-OSS. It’s a thoughtful look at why sampling, formatting, and standardization are not just implementation details, but core to the future of working with LLMs.
Re: Sampling and structured outputs in LLMs
#3Sounds like brute force to me.
Re: Sampling and structured outputs in LLMs
#4Hmm, so if structured output affects the quality of the response maybe it's better to convert the output to a structured format as a post-processing step?
Re: Sampling and structured outputs in LLMs
#5It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
Re: Sampling and structured outputs in LLMs
#6When doing structured sampling, why is the token sampled, checked against the grammar, and resampled if it's wrong by applying the mask ?
Why wouldn't we apply the mask immediately for the first sampling? Is this an optimization somehow, is masking expensive?
Re: Sampling and structured outputs in LLMs
#7This constrains the output of the LLM to some grammar.
However, why not use a grammar that does not have invalid sentences, and from there convert to any grammar that you want?
Re: Sampling and structured outputs in LLMs
#8It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
Wouldn't that have implications for inference batching, since you would have to track state and apply a different mask for each sequence in the batch? If so, I think it would directly affect utilisation and hence costs. But I could be talking out of my ass here.
Re: Sampling and structured outputs in LLMs
#9Just wait till people realize that if you have agents speak in structured output rather than chatting with you, your observability and ability to finely program your agent goes through the roof.
Re: Sampling and structured outputs in LLMs
#10Hmm, so if structured output affects the quality of the response maybe it's better to convert the output to a structured format as a post-processing step?
It's a tradeoff between getting "good enough" performance w/ guided/constrained generation and using 2x calls to do the same task. Sometimes it works, sometimes it's better to have a separate model. One good case of 2 calls is the "code merging" thing, where you "chat" with a model giving it a source file + some instruction, and if it replies with something like ... //unchanged code here ... some new code ... //the rest stays the same, then you can use a code merging model to apply the changes. But that's become somewhat obsolete by the new "agentic" capabilities where models learn how to diff files directly.