Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TypeChat

microsoft.github.io

Post reply on HN
Print view View original

161–170 of 174 posts

Re: TypeChat

#161

Post by waffletower »

geysersam wrote: post #138
rolisz wrote: post #136

Earlier quoted context omitted.

> when you can guarantee a valid JSON syntax by only sampling tokens that are valid? Instead of greedily picking the highest-scoring token every time, you select the highest-scoring token that conforms to the requested format. Yes, you can guarantee a syntactically correct JSON that way, but will it be a semantically correct? If the model really really really wanted to put another token there, but you are forcing it…

Well, if the output doesn't conform to the format it's useless. If the model can't produce good and correct output then it's simply not up to the task.

In my experience, LLM responses result in a fair distribution of outputs that do have semantically useful outputs but do not precisely adhere to the requested format. If I chose to use a strongly typed language for LLM parsing, perhaps I would be tempted to eliminate complexity and simply throw structural outliers away, and explain to the suits that a certain percentage of our queries/expenses are unusable. Instead, more sophisticated coercion techniques could be applied instead to increase output utilization.

Re: TypeChat

#162

Post by ploppyploppy »

BoorishBears wrote: post #71
bwestergard wrote: post #40

Earlier quoted context omitted.

The value is in: 1. Running the typescript type checker against what is returned by the LLM. 2. If there are type errors, combining those into a "repair prompt" that will (it is assumed) have a higher likelihood of eliciting an LLM output that type checks. 3. Gracefully handling the cases where the heuristic in #2 fails. https://github.com/microsoft/TypeChat/blob/main/src/typechat... In my experience experimenting wi…

Here's a project that does that better imo: https://github.com/dzhng/zod-gpt And by better I mean doesn't tie you to OpenAI for no good reason

Anything like this but for Python?

Re: TypeChat

#163

Post by ploppyploppy »

LordDragonfang wrote: post #74
BoorishBears wrote: post #71

Earlier quoted context omitted.

Here's a project that does that better imo: https://github.com/dzhng/zod-gpt And by better I mean doesn't tie you to OpenAI for no good reason

I don't know where all you people work that your employer would prefer a random git repo (that has no support and no guarantee of updates) over a solution from Microsoft . (Alternatively: that you have so much free time that you'd prefer to fiddle with your own validation code instead of writing your actual app) Open source solutions are great (which this still is, btw), but having a first-party solution is also a go…

Look at Guidance - that's being ignored by Microsoft yet it's an official repo

Re: TypeChat

#164

Post by ploppyploppy »

dvt wrote: post #72

This is my hot take: we're slowly entering the "tooling" phase of AI, where people realize there's no real value generation here, but people are so heavily invested in AI, that money is still being pumped into building stuff (and of course, it's one of the best way to guarantee your academic paper gets published). I mean, LangChain is kind of a joke and they raised $10M seed lol. DeFi/crypto went through this phase 2…

Your analysis is based on what's possible now. This is the worst it'll ever be.

Re: TypeChat

#165

Post by verdverm »

ofslidingfeet wrote: post #96
verdverm wrote: post #89

Earlier quoted context omitted.

There are multiple ways to get structured output, and what this library is doing is not really that interesting. The concept is interesting and has had multiple implementations already, the code (and abstraction) here is not interesting and creates more issues than it solves

Tell me how to get reliably structured output. I'm all ears.

here is one of our early examples: https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/pro...

Re: TypeChat

#166

Post by verdverm »

parentheses wrote: post #95
verdverm wrote: post #27

I don't see the value add here. Here's the core of the message sent to the LLM: https://github.com/microsoft/TypeChat/blob/main/src/typechat... You are basically getting a fixed prompt to return structured data with a small amount of automation and vendor lockin. All these LLM libraries are just crappy APIs to the underlying API. It is trivial to write a script that does the same and will be much more flexible as mod…

The value is turn unstructured data into structured data and ensure it satisfies schema constraints. For example: you have 1000 free-text survey responses about your product, building a schema and for-each `TypeChat`ing them would get you a dataset for that free-text. It's mind-bogglingly useful.

yes, turning unstructured data into structured data is one of the most useful ways to use an LLM right now. It has been done before with using schemas and could be done without all the extra cruft.

There was a similar example a few months back using XML instead, but I haven't heard much about it since, because again, the library did not add value on top of doing these things in a more open or scripted setting.

MSFT has another project in similar vain, guardrails, interesting idea, but made worse by wrapping it in a library. Most of these LLM ideas are better as a function than a library, make them transform the i/o rather than every library needing to write wrappers around the LLM APIs as well

There are several more making use of OpenAPI / JSONSchema rather than TS.

We use a subset of CUE, essentially JSON without as many quotes or commas. The LLMs are quite flexible with few-shot learning. They can be made more reliable with fine-tuning. They can be made faster and cheaper with distillation.

Re: TypeChat

#167

Post by bottlepalm »

throwaway290 wrote: post #134
bottlepalm wrote: post #116

Earlier quoted context omitted.

It's July, they just needed to put a voice interface on ChatGPT, it'd easily help them sell more pro licenses as well. I'm not a conspiracy person, but this just seems so obvious it feels like there's something else going on here.

No big company wants their appliance to accidentally talk customer's child into suicide or spouse into a divorce. Bad for image.

It's not like ChatGPT can't do that already..

Re: TypeChat

#169

Post by sdwr »

9dev wrote: post #37
sdwr wrote: post #29

Earlier quoted context omitted.

I was thinking about this yesterday. ChatGPT really is good enough to act as a proper virtual assistant / home manager, with enough toggles exposed.

ChatGPT isn’t the limiting factor here, a good way to expose the toggles is. I recently tried to expose our company CRM to employees by means of a Teams bot they could ask for stuff in natural language (like „send an invite link to newlead@example.org“ or „how many MAUs did customer Foo have in June“), but while I almost got there, communicating an ever-growing set of actionable commands (with an arbitrary number of…

Interacting with APIs is the old style. The magic of ChatGPT is the same magic as google had back in the day - you ask it in plain english and it has an answer.

I'm guessing the solution looks like a model trained to take actions on the internet. Kinda sucks for those of us on the outside, because whatever we make is going to be the same, brittle, chewing-gum and duct tape approach as usual. Best to wait for the bleeding edge, like what that MinecraftGPT project was aiming at.

Re: TypeChat

#170

Post by Scaevolus »

andy_xor_andrew wrote: post #121

Here's one thing I don't get. Why all the rigamarole of hoping you get a valid response, adding last-mile validators to detect invalid responses, trying to beg the model to pretty please give me the syntax I'm asking for... ...when you can guarantee a valid JSON syntax by only sampling tokens that are valid? Instead of greedily picking the highest-scoring token every time, you select the highest-scoring token that co…

Llama.cpp recently added grammar based sampling, which constraints token selection to follow a rigid format like you describe.

https://github.com/ggerganov/llama.cpp/pull/1773

Post reply on HN