Live data from Hacker News

A guidance language for controlling LLMs

github.com

71–80 of 198 posts

Re: A guidance language for controlling LLMs

#71
post #38

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

It won't necessarily turn into some that is fundamentally the same as a current programming language. Rather than a "VM" or "interpreter" or "compiler" we have this "LLM". Even if it requires a lot of domain knowledge to program using an "LLM-interpreted" language, the means of specification (in terms of how the software code is interpreted) may be different enough that it enables easier-to-write, more robust, (more…

This is a hopeful evolutionary path. My concern is that I can literally feel Conway's law emanating from current LLM approaches as they switch between the actual LLM and the governing code around it that layers a buch of conditionals of the form:

if (unspeakable_things): return negatory_good_buddy

I see this happen a few times per day where the UI triggers a cancel even on its own fake typing mode and overwrites a user response that has at least half-rendered the trigger-warning-inducing response.

It's pretty clear from a design perspective that this is intended to be proxy to facial expressions while being worthy of an MVP postmortem discussion about what viability means in a product that's somewhere on a spectrum of unintended consequences that only arise at runtime.

Re: A guidance language for controlling LLMs

#72
post #61

I'm personally starting with learning Guidance and LMQL rather than LangChain just in order to get a better grasp of the behaviors that I've gathered LangChain papers over. Even after that, I'm likely to look at Haystack before LangChain. Just getting the feeling that LangChain is going to end up being considered a kitchen sink solution full of anti patterns so might as well spend time a little lower level while I se…

What do you think about Haystack vs LangChain?

I haven't had the chance to dig in yet, but my impression is that it's less opinionated than LangChain. I'd love to know if that's true or not, since I'm really trying to prioritize my time around learning this stuff in a way that let's me (1) understand prompt dynamics a bit more clearly and (2) not sacrifice practicality too much.

If only there were a clear syllabus for this stuff! There's such an incredible amount to keep up with. The pace is bonkers.

Re: A guidance language for controlling LLMs

#73

Earlier quoted context omitted.

Not quite sure what the spirit of your comment is. But, yes, they are real technologies. Very confused as to why you would even find that dubious.

I'm not an outsider but I also don't understand the reaction. I'm going to randomly think of 5 names for technologies and see how they sound: React, Supabase, Next, Kafka, Redis I mean, IMO "LangChain" is kind of a silly name but I feel like there's nothing to see here.

It's not that silly IMO, Chain LLMs together like composing functions, however I guess 'Chain' has a certain connotation in 2023 after the last few years of crypto.

Re: A guidance language for controlling LLMs

#74

I'm personally starting with learning Guidance and LMQL rather than LangChain just in order to get a better grasp of the behaviors that I've gathered LangChain papers over. Even after that, I'm likely to look at Haystack before LangChain. Just getting the feeling that LangChain is going to end up being considered a kitchen sink solution full of anti patterns so might as well spend time a little lower level while I se…

If this comment performative comedy? Are these real technologies ?

Is it Pokemon or Big Data?

http://pixelastic.github.io/pokemonorbigdata/

Re: A guidance language for controlling LLMs

#75
post #38

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

Of course. Laws are the same way; defined norms for future behavior.

The professional managerial class must maintain appropriate distinctions between their rights and ours. Their belief in exclusive right to profit from our agency is at risk if AI can generate too much noise.

Re: A guidance language for controlling LLMs

#76
post #38

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

A number of years ago we were designing a way to specify insurance claim adjudication rules in natural language, so that "the business" could write their own rules. The "natural" language we ended up with was not so natural after all. We would have had to teach users this specific English dialect and grammar (formal and precise syntax, as you said). So, in the end, we abandoned that project and years later just rewro…

> but it wasn't something the stakeholders outside of engineering even wanted

Ha this reminds me of the craze for BDD/Cucumber type testing. Don’t think I ever once saw a product owner take interest in a human readable test case haha

Re: A guidance language for controlling LLMs

#78
post #38

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

A number of years ago we were designing a way to specify insurance claim adjudication rules in natural language, so that "the business" could write their own rules. The "natural" language we ended up with was not so natural after all. We would have had to teach users this specific English dialect and grammar (formal and precise syntax, as you said). So, in the end, we abandoned that project and years later just rewro…

Just saw this on HN a couple days ago, sounds like just what was needed!

https://en.wikipedia.org/wiki/Attempto_Controlled_English?wp...

https://news.ycombinator.com/item?id=35936396

Re: A guidance language for controlling LLMs

#79
post #38

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

I think LLMs can transform between precise and imprecise languages.

So it's useful to have a library that helps and the input or output be precise, when that is what the task involves.

Re: A guidance language for controlling LLMs

#80
post #38

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

Well to be fair, yes we do need to integrate programming languages with large neural nets in more advanced ways. I don’t think it’s really reinventing it so much as learning how to integrate these two different computing concepts.
Post reply on HN