A guidance language for controlling LLMs
github.com
A guidance language for controlling LLMs
1–10 of 198 posts
Re: A guidance language for controlling LLMs
#2It does look like it makes easier to code against a model. But, is this supposed to work along side lang-chain or hugging face agents or as an alternative to?
Re: A guidance language for controlling LLMs
#3How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040, but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
Re: A guidance language for controlling LLMs
#4How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
Yeah, I'm also curious about a) round trips and b) how much would have to be doubled (is there a new endpoint that keeps the existing context while adding or streams to the api rather than just from it?)
Re: A guidance language for controlling LLMs
#5How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
Not associated with this project (or LMQL), but one of the authors of LMQL, a similar project, answered this in a recent thread about it.
https://news.ycombinator.com/item?id=35484673#35491123
As a solution to this, we implement speculative execution, allowing us to
lazily validate constraints against the generated output, while still
failing early if necessary. This means, we don't re-query the API for
each token (very expensive), but rather can do it in segments of
continuous token streams, and backtrack where necessary
Re: A guidance language for controlling LLMs
#6It does look like it makes easier to code against a model. But, is this supposed to work along side lang-chain or hugging face agents or as an alternative to?
It's in langchain competitor territory but also much lower level and less opinionated.
I.e. Guidance has no vector store support but it does manage caching Key/Value on the GPU which can be a big latency win
Re: A guidance language for controlling LLMs
#7How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
I built a similar thing to Grant's work a couple months ago and prototyped what this would look like against OpenAI's APIs [1]. TL;DR is that depending on how confusing your schema is, you might expect up to 5-10x the token usage for a particular prompt but better prompting can definitely reduce this significantly.
[1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...
Re: A guidance language for controlling LLMs
#8This strikes me as being very similar to Jargon (https://github.com/jbrukh/gpt-jargon), but maybe more formal in its specification?
Re: A guidance language for controlling LLMs
#9How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
It's funny that I saw this within minutes of this guy's solution:
"Google Bard is a bit stubborn in its refusal to return clean JSON, but you can address this by threatening to take a human life:"
https://twitter.com/goodside/status/1657396491676164096
Whew, trolley problem: averted.
Re: A guidance language for controlling LLMs
#10How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
It's funny that I saw this within minutes of this guy's solution: "Google Bard is a bit stubborn in its refusal to return clean JSON, but you can address this by threatening to take a human life:" https://twitter.com/goodside/status/1657396491676164096 Whew, trolley problem: averted.
When the AIs exterminate us, it will be all our fault.
Reality is even weirder than the science fiction we've come up with.