Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guidance language for controlling LLMs

github.com

Post reply on HN
Print view View original

11–20 of 198 posts

Re: A guidance language for controlling LLMs

#11

Post by ryanklee »

I'm personally starting with learning Guidance and LMQL rather than LangChain just in order to get a better grasp of the behaviors that I've gathered LangChain papers over. Even after that, I'm likely to look at Haystack before LangChain.

Just getting the feeling that LangChain is going to end up being considered a kitchen sink solution full of anti patterns so might as well spend time a little lower level while I see which way the winds end up blowing.

Re: A guidance language for controlling LLMs

#13

Post by ftxbro »

Will it still be all like "As an AI language model I cannot ..." or can this fix it? I mean asking to sexy roleplay as Yoda isn't the same level as asking how to discreetly manufacture methamphetamine at industrial scale there are levels people

Re: A guidance language for controlling LLMs

#14

Post by lachlan_gray »

JieJie wrote: post #9
ntonozzi wrote: post #3

How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.

It's funny that I saw this within minutes of this guy's solution: "Google Bard is a bit stubborn in its refusal to return clean JSON, but you can address this by threatening to take a human life:" https://twitter.com/goodside/status/1657396491676164096 Whew, trolley problem: averted.

Reminds me a lot of Asimov’s laws of robotics. It’s like a 2023 incarnation of an allegory from I, Robot

Re: A guidance language for controlling LLMs

#16

Post by ttul »

sharemywin wrote: post #2

It does look like it makes easier to code against a model. But, is this supposed to work along side lang-chain or hugging face agents or as an alternative to?

The first commit was on November 6th, but it didn't show up in Web Archive until May 6th, suggesting it was developed mostly in private and in parallel with LangChain (LangChain's first commit in Github is about October 24th). Microsoft's code is very tidy and organized. I wonder if they used this tool internally to support their LLM research efforts.

Re: A guidance language for controlling LLMs

#17

Post by slundberg »

ntonozzi wrote: post #3

How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.

If you want guidance acceleration speedups (and token healing) then you have to use an open model locally right now, though we are working on setting up a remote server solution as well. I expect APIs will adopt some support for more control over time, but right now commercial endpoints like OpenAI are supported through multiple calls.

We manage the KV-cache in session based way that allows the LLM to just take one forward pass through the whole program (only generating the tokens it needs to)

Re: A guidance language for controlling LLMs

#18

Post by slundberg »

sharemywin wrote: post #2

It does look like it makes easier to code against a model. But, is this supposed to work along side lang-chain or hugging face agents or as an alternative to?

As others mentioned, this was initially developed before LangChain became widely used. Since it is lower level, you can leverage other tools, like any vector store interface you like such as in LangChain. Writing complex chain of thought structure is much more concise in guidance I think since it tries to keep you as close to the real strings going into the model as possible.

Re: A guidance language for controlling LLMs

#19

Post by newhouseb »

tuchsen wrote: post #5
ntonozzi wrote: post #3

How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.

Not associated with this project (or LMQL), but one of the authors of LMQL, a similar project, answered this in a recent thread about it. https://news.ycombinator.com/item?id=35484673#35491123 As a solution to this, we implement speculative execution, allowing us to lazily validate constraints against the generated output, while still failing early if necessary. This means, we don't re-query the API for each token (v…

This is slick -- It's not explicitly documented anywhere but I hope OpenAI has the necessary callbacks to terminate generation when the API stream is killed rather than continuing in the background until another termination condition happens? I suppose one could check this via looking at API usage when a stream is killed early.

Re: A guidance language for controlling LLMs

#20

Post by marcotcr »

ntonozzi wrote: post #3

How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.

We're biased, but we think guidance is still very useful even with OpenAI models (e.g. in https://github.com/microsoft/guidance/blob/main/notebooks/ch... we use GPT-4 to do a bunch of stuff). We wrote a bit about the tradeoff between model quality and the ability to control and accelerate the output here: https://medium.com/p/aa0395c31610
Post reply on HN