Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A guidance language for controlling LLMs

github.com

Post reply on HN
Print view View original

41–50 of 198 posts

Re: A guidance language for controlling LLMs

#41

Post by tuchsen »

newhouseb wrote: post #19
tuchsen wrote: post #5

Earlier quoted context omitted.

Not associated with this project (or LMQL), but one of the authors of LMQL, a similar project, answered this in a recent thread about it. https://news.ycombinator.com/item?id=35484673#35491123 As a solution to this, we implement speculative execution, allowing us to lazily validate constraints against the generated output, while still failing early if necessary. This means, we don't re-query the API for each token (v…

This is slick -- It's not explicitly documented anywhere but I hope OpenAI has the necessary callbacks to terminate generation when the API stream is killed rather than continuing in the background until another termination condition happens? I suppose one could check this via looking at API usage when a stream is killed early.

Yeah I did a CLI tool for talking to ChatGPT. I'm pretty sure they stop generating when you kill the SSE stream, based on my anecdotal experience of keeping ChatGPT4 costs down by killing it as soon as i get the answer I'm looking for. You're right that it's undocumented behavior though, on a whole the API docs they give you are as thin as the API itself.

Re: A guidance language for controlling LLMs

#42

Post by leroy-is-here »

ryanklee wrote: post #39
leroy-is-here wrote: post #25

Earlier quoted context omitted.

If this comment performative comedy? Are these real technologies ?

Not quite sure what the spirit of your comment is. But, yes, they are real technologies. Very confused as to why you would even find that dubious.

Not dubious, I just read your comment and it felt like I was reading satire. Even the cadence of your words felt funny.

Anyway, I’m not surprised. It’s a new market, everyone’s in on it.

Re: A guidance language for controlling LLMs

#43

Post by jameshart »

ryanklee wrote: post #39
leroy-is-here wrote: post #25

Earlier quoted context omitted.

If this comment performative comedy? Are these real technologies ?

Not quite sure what the spirit of your comment is. But, yes, they are real technologies. Very confused as to why you would even find that dubious.

Consider how similar your comment reads, for an outsider, to this explanation of AWS InfiniDash: https://twitter.com/TartanLlama/status/1410959645238308866

Re: A guidance language for controlling LLMs

#45

Post by killthebuddha »

tuchsen wrote: post #41
newhouseb wrote: post #19

Earlier quoted context omitted.

This is slick -- It's not explicitly documented anywhere but I hope OpenAI has the necessary callbacks to terminate generation when the API stream is killed rather than continuing in the background until another termination condition happens? I suppose one could check this via looking at API usage when a stream is killed early.

Yeah I did a CLI tool for talking to ChatGPT. I'm pretty sure they stop generating when you kill the SSE stream, based on my anecdotal experience of keeping ChatGPT4 costs down by killing it as soon as i get the answer I'm looking for. You're right that it's undocumented behavior though, on a whole the API docs they give you are as thin as the API itself.

I'm skeptical that the streaming API would really save that much cost. In my experience the vast majority of all tokens used are input tokens rather than completed tokens.

Re: A guidance language for controlling LLMs

#46

Post by indus »

This reminds me of the time when I wrote a cgi script.

Basically instructing the templating engine (a very crude regex) to replace session variables, database lookups to the merge fields:

Hello {{firstname}}!

1996 and 2023 smells alike.

Re: A guidance language for controlling LLMs

#47

Post by Spivak »

I think it's cool that a company like Microsoft is willing to base a real-boy product on pybars3 which is its author's side-project instead of something like Jinja2. If this catches on I can imagine MS essentially adopting the pybars3 project and turning it into a mature thing.

Re: A guidance language for controlling LLMs

#48

Post by killthebuddha »

rain1 wrote: post #35
Der_Einzige wrote: post #28

There has been a huge explosion of awesome tooling which utilizes constrained text generation. Awhile ago, I tried my own hand at constraining the output of LLMs. I'm actively working on this to make it better, especially with the lessons learned from repos like this and from guidance https://github.com/hellisotherpeople/constrained-text-genera...

This looks incredible. Wow.

I agree, it looks great. A couple similar projects you might find interesting:

- https://github.com/newhouseb/clownfish

- https://github.com/r2d4/rellm

The first one is JSON only and the second one uses regular expressions, but they both take the same "logit masking" approach as the project GP linked to.

Re: A guidance language for controlling LLMs

#49

Post by killthebuddha »

ryanklee wrote: post #39
leroy-is-here wrote: post #25

Earlier quoted context omitted.

If this comment performative comedy? Are these real technologies ?

Not quite sure what the spirit of your comment is. But, yes, they are real technologies. Very confused as to why you would even find that dubious.

I'm not an outsider but I also don't understand the reaction. I'm going to randomly think of 5 names for technologies and see how they sound:

React, Supabase, Next, Kafka, Redis

I mean, IMO "LangChain" is kind of a silly name but I feel like there's nothing to see here.

Re: A guidance language for controlling LLMs

#50

Post by m3kw9 »

I’m not understanding how Guidence Accelerating works. It says “ This cuts this prompt's runtime in half vs. a standard generation approach.” and it gives an example of it asking LLM to generate json. I don’t see anywhere how it accelerates anything because it’s a simple json completion call. How can you accelerate that?
Post reply on HN