291–300 of 315 posts
Earlier quoted context omitted.
Given how this works, I don’t think that is possible unless OpenAI implements it themselves.
really? the docs seem to promise something like that "can work with any model"
I'm not sure how this is different than: https://github.com/1rgs/jsonformer or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or https://github.com/ggerganov/llama.cpp/pull/1773 or https://github.com/Shopify/torch-grammar Overall there are a ton of these logit based guidance systems, the reason they don't get tons of traction is the SOTA models are behind REST APIs that don't enable this f…
Thanks for bringing clownfish and relm to my attention! afaik other libraries loop over the entire vocabulary at every step of the generation. We on the other hand build an index at initialization by looping once over the vocabulary. Then generation is just as fast as standard generation.
Earlier quoted context omitted.
I wasn't talking about deterministic Vs nondeterministic. > If you are willing to spend the effort to write a grammar, you can do that without an LLM. How are you taking, for example, a request to make a "fun but not over the top character from the middle ages, with relevant weapons and a backstory. Game theme is a world populated by anthropomorphic vegetables." And get back a character for the game in a specific JSO…
As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). At which point, again I have to ask: what do you need the LLM for? You've already done all the hard work by hand…
You wouldn't need to, that's the point here. You let the LLM work on generating semantically valid responses and use a tool like this to restrict it to syntactically correct ones.
Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). Let the LLM deal with coming up with a name and backstory that work, making sure the description and type of the weapon make sense (gpt4 suggested a close range carrot dagger for example), and let this work as your type structure.
{
"type": "object",
"title": "character",
"properties": {
"backstory": {
"type": "string"
},
"weapons": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"description": {
"type": "string"
},
"weapon_type": {
"type": "string",
"enum": ["ranged", "close", "magic"]
},
"range": {
"minimum": 0,
"maximum": 150
},
"damage": {
"type": "number"
}
},
"required": [
"name",
"description",
"range",
"damage"
]
}
},
"name": {
"type": "string"
}
},
"required": [
"backstory",
"weapons",
"name"
]
}
Nonsense. Like "Colorless green ideas sleep furiously" the famous sentence that's grammatically correct but utter nonsense.
> Plus, if you already have the grammar that can cover the anthropomorphic vegetable world it's only a bit more work to use it to parse such natural language requests, anyway.
I really do not think this is the case. Parsing and understanding arbitrary requests about something like this?
GPT2 doesn't seem to take instruction well. I've tried llama gpt-medium etc etc.
They all either pick up a different language, or freeze.
EDIT: I see tons of activity and work in the github issues, so ignore this for now.
Super excited when I'll be able to have this working for myself!
I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…
That re-prompting on error trick is what this new Microsoft library does, too: https://github.com/microsoft/TypeChat Here's their prompt for that: https://github.com/microsoft/TypeChat/blob/c45460f4030938da3... I think the approach using grammars (seen here, but also in things like https://github.com/ggerganov/llama.cpp/pull/1773 ) is a much more elegant solution.
Earlier quoted context omitted.
I wasn't talking about deterministic Vs nondeterministic. > If you are willing to spend the effort to write a grammar, you can do that without an LLM. How are you taking, for example, a request to make a "fun but not over the top character from the middle ages, with relevant weapons and a backstory. Game theme is a world populated by anthropomorphic vegetables." And get back a character for the game in a specific JSO…
As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). At which point, again I have to ask: what do you need the LLM for? You've already done all the hard work by hand…
This is a rewritten history of natural language processing tech. Years of fine-tuned theory-heavy grammar coding for parsing and generating human language got the field basically nowhere.
Earlier quoted context omitted.
As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). At which point, again I have to ask: what do you need the LLM for? You've already done all the hard work by hand…
> As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). You wouldn't need to, that's the point here. You let the LLM work on generating semantically valid responses a…
That'd be nice, but it's not how this tool works. If you look at the repo, there's an example of following a json schema or pydantic model. It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand:
class Weapon(str, Enum):
sword = "sword"
axe = "axe"
mace = "mace"
spear = "spear"
bow = "bow"
crossbow = "crossbow"
>> I really do not think this is the case. Parsing and understanding arbitrary requests about something like this?
Not arbitrary. See my casting-to-type analogy. The point I'm trying really hard to get across is that generating free-form text is all nice and cool, but if you want to give it structure, you need to have the entire structure defined before-hand, otherwise the text that can't be made to conform to it simply won't.
So if you haven't got anthropomorphic vegetables in your json schema, your LLM may generate them, they'll never end up in your json.
Earlier quoted context omitted.
> As far as I can tell you won't be able to use the approach proposed here to create a character matching your above description unless every element of it is encoded in the guiding grammar (including the possibility for the character to have middle ages-relevant weapons, and the anthropomorphic vegetables). You wouldn't need to, that's the point here. You let the LLM work on generating semantically valid responses a…
>> Here's an example jsonschema (a bit handwritten so maybe some errors but it should be clear enough). That'd be nice, but it's not how this tool works. If you look at the repo, there's an example of following a json schema or pydantic model. It's clear that if you wanted a "carrot dagger" in your json, you'd need to define it beforehand: class Weapon(str, Enum): sword = "sword" axe = "axe" mace = "mace" spear = "sp…
Given the stateful nature of tokenizers, I decided that trying to preprocess the individual token ids was a losing battle. Even in the simple case of whitespace - tokenizer merges can really screw up generating a static mask, e.g. we expect a space next, but a token decodes to 'foo', but is actually a '_foo' and would've decoded with a whitespace if it were following a valid pair. When I go to construct the static vocab mask, it would then end up matching against 'foo' instead of ' foo'.
How did you work around this for the FSM approach? Does it somehow include information about merges / whitespace / tokenizer statefulness?
Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…
I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs. I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.