11–20 of 174 posts
This is a cute idea and it looks like it should work, but I could see this getting expensive with larger models and input prompts. Probably not a fix for all scenarios.
I swear I think of something and Anders Hejlsberg builds it. Structured requests and responses are 100% the next evolution of LLMs. People are already getting tired of chatbots. Being able to plug in any backend without worrying about text parsing and prompts will be amazing.
Yup, a general desire of mine is to locally run an LLM which has actionable interfaces that i provide. Things like "check time", "check calendar", "send message to user" and etc.
TypeChat seems to be in the right area. I can imagine an extra layer of "fit this JSON input to a possible action, if any" and etc.
I see a neat hybrid future where a bot (LLM/etc) works to glue layers of real code together. Sometimes part of ingestion, tagging, etc - sometimes part of responding to input, etc.
All around this is a super interesting area to me but frankly, everything is moving so fast i haven't concerned myself with diving too deep in it yet. Lots of smart people are working on it so i feel the need to let the dust settle a bit. But i think we're already there to have my "dream home interface" working.
Looks like it just runs the LLM in a loop until it spits out something that type checks, prompting with the error message. This is a cute idea and it looks like it should work, but I could see this getting expensive with larger models and input prompts. Probably not a fix for all scenarios.
It's not clear to me how they ensure the responses will be valid JSON, are they just asking for it, then parsing the result with error checking?
Looks like it just runs the LLM in a loop until it spits out something that type checks, prompting with the error message. This is a cute idea and it looks like it should work, but I could see this getting expensive with larger models and input prompts. Probably not a fix for all scenarios.
> It's unfortunately easy to get a response that includes { "name": "grande latte" } type Item = { name: string; ... size?: string; I'm not really following how this would avoid `name: "grande latte"`? But then the example response: "size": 16 > This is pretty great! Is it? It's not even returning the type being asked for? I'm guessing this is more of a typo in the example, because otherwise this seems cool.
> It's unfortunately easy to get a response that includes { "name": "grande latte" } type Item = { name: string; ... size?: string; I'm not really following how this would avoid `name: "grande latte"`? But then the example response: "size": 16 > This is pretty great! Is it? It's not even returning the type being asked for? I'm guessing this is more of a typo in the example, because otherwise this seems cool.
{
name: “the brown one”,
size: “the espresso cup”,
… }
But now why would you want that to be locked into the type syntax? You probably want something more like Zod where you can use some runtime data to build up those union types.
You also want restrictions on the types too, like quantity should be a positive, non-fractional integer. Of course you can just validate the JSON values afterwards, but now the user gets two kinds of errors. One from the LLM which is fluent and human sounding, and the other which is a weird technical “oops! You provided a value that is too large for quantity” error.
The type syntax seems like the wrong place to describe this stuff.
Someone should just get this working on Llama 2 instead of O̶p̶e̶n̶AI.com [0] All this is it's just talking to a AI model sitting on someone else's server. [0] https://github.com/microsoft/TypeChat/blob/main/src/model.ts...
I'm totally happy to be able to receive structured queries, but I'm also not 100% sure TypeScript is the right tool, it seems to be an overkill. I mean obviously you don't need the power of TS with all its enums, generics, etc.
Plus given that it will run multiple queries in loop, it might end up very expensive for it abide by your custom-mage complex type
Looks like it just runs the LLM in a loop until it spits out something that type checks, prompting with the error message. This is a cute idea and it looks like it should work, but I could see this getting expensive with larger models and input prompts. Probably not a fix for all scenarios.
At least with OpenAI, wouldn't it be better if under the hood it was using the new function call feature?
I imagine closing the loop (using the TS compiler to restrict token output weights) is in the works, though it's probably not totally trivial. You'd need:
* An incremental TS compiler that could report "valid" or "valid prefix" (ie, valid as long as the next token is not EOF)
* The ability to backtrack the model
Idk how hard either one piece is.