101–110 of 198 posts
This reminds me of the time when I wrote a cgi script. Basically instructing the templating engine (a very crude regex) to replace session variables, database lookups to the merge fields: Hello {{firstname}}! 1996 and 2023 smells alike.
RegEx didn't hallucinate though.
Earlier quoted context omitted.
What do you think about Haystack vs LangChain?
I haven't had the chance to dig in yet, but my impression is that it's less opinionated than LangChain. I'd love to know if that's true or not, since I'm really trying to prioritize my time around learning this stuff in a way that let's me (1) understand prompt dynamics a bit more clearly and (2) not sacrifice practicality too much. If only there were a clear syllabus for this stuff! There's such an incredible amount…
Earlier quoted context omitted.
A number of years ago we were designing a way to specify insurance claim adjudication rules in natural language, so that "the business" could write their own rules. The "natural" language we ended up with was not so natural after all. We would have had to teach users this specific English dialect and grammar (formal and precise syntax, as you said). So, in the end, we abandoned that project and years later just rewro…
SQL looks the way it does (rather than some much more succinct relational algebra notation) because it was intended to be used by non-technical management/executive personnel so they could create whatever reports they needed without somebody having to translate business-ese to relalg. That, uh, didn't quite happen.
This is pretty fascinating, but I'm not sure I understand the benefit of using a Handlebars-like DSL here. For example, given this code from https://github.com/microsoft/guidance/blob/main/notebooks/ch... create_plan = guidance('''{{#system~}} You are a helpful assistant. {{~/system}} {{#block hidden=True}} {{#user~}} I want to {{goal}}. {{~! generate potential options ~}} Can you please generate one option for how t…
We could write a python package that could? A codegen tool that generates codegen that will then generate code?
Earlier quoted context omitted.
A number of years ago we were designing a way to specify insurance claim adjudication rules in natural language, so that "the business" could write their own rules. The "natural" language we ended up with was not so natural after all. We would have had to teach users this specific English dialect and grammar (formal and precise syntax, as you said). So, in the end, we abandoned that project and years later just rewro…
> but it wasn't something the stakeholders outside of engineering even wanted Ha this reminds me of the craze for BDD/Cucumber type testing. Don’t think I ever once saw a product owner take interest in a human readable test case haha
They had some previous negative experiences with uncertainty about what "was working" in releases, and a pretty slapdash process before I came on board, so it was an important trust building tool.
I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.
They talk about improving tokenization but I don't believe that's the fundamental problem of controlling LLMs. The problem with LLMs is all the data comes in as (tokenized) language and the result is nothing but in-context predicted output. That's where all the "prompt-injection" exploits come from - as well as the hallucinations, "temper tantrums" and so-forth.
I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees. I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.
You need to be able to define all of the possible edge cases so there isn’t any Undefined Behavior: that’s the formal part
Humans can use LLMs to manipulate these languages to achieve specific goals. I can imagine designing formal languages intended for LLMs to manipulate or generate, but I can’t imagine the need for the languages themselves going away.
I think it's cool that a company like Microsoft is willing to base a real-boy product on pybars3 which is its author's side-project instead of something like Jinja2. If this catches on I can imagine MS essentially adopting the pybars3 project and turning it into a mature thing.
How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040 , but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.
I'm getting valid JSON out of gpt-3.5-turbo without trouble. I supply an example via the assistant context, and tell it to output JSON with specific fields I name. It does fail roughly 1/10th of the time, but it does work.
What production use case, you ask? You could do zero-shot entity extraction using ChatGPT if it were more reliable. Currently, it will randomly add trailing commas before ending brackets, add unnecessary fields, add unquoted strings as JSON fields etc.
Earlier quoted context omitted.
> but it wasn't something the stakeholders outside of engineering even wanted Ha this reminds me of the craze for BDD/Cucumber type testing. Don’t think I ever once saw a product owner take interest in a human readable test case haha
I've used Cucumber on a few consulting projects I've done and had management / C-level interested and involved. It's a pretty narrow niche, but they were definitely enthusiastic for the idea that we had a defined list of features that we could print out (!!) as green or red for the current release. They had some previous negative experiences with uncertainty about what "was working" in releases, and a pretty slapdash…
This is so often completely missed in these conversations about these tools.
Great point.