Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TypeChat

microsoft.github.io

Post reply on HN
Print view View original

121–130 of 174 posts

Re: TypeChat

#121

Post by andy_xor_andrew »

Here's one thing I don't get.

Why all the rigamarole of hoping you get a valid response, adding last-mile validators to detect invalid responses, trying to beg the model to pretty please give me the syntax I'm asking for...

...when you can guarantee a valid JSON syntax by only sampling tokens that are valid? Instead of greedily picking the highest-scoring token every time, you select the highest-scoring token that conforms to the requested format.

This is what Guidance does already, also from Microsoft: https://github.com/microsoft/guidance

But OpenAI apparently does not expose the full scores of all tokens, it only exposes the highest-scoring token. Which is so odd, because if you run models locally, using Guidance is trivial, and you can guarantee your json is correct every time. It's faster to generate, too!

Re: TypeChat

#122

Post by jameshart »

verdverm wrote: post #87
whimsicalism wrote: post #51

Earlier quoted context omitted.

Yes as the abstractions gets better it becomes easier to code useful things.

the debate is about how valuable the abstraction here is to warrant a library, and the fact that it predefines the prompt and api call flow, so you cannot prompt engineer or use something like CoT/ToT

This amounts to saying ‘how dare someone publish some code that they wrote!’

Is it your impression that this is being pitched as some grand solution?

That this was published as a way to shut out other people from doing the same thing in other ways?

Can’t we just look at a cool thing someone did, and released for other people to play with, and say ‘huh! That’s neat!’ And get inspired?

Re: TypeChat

#123

Post by zarzavat »

andy_xor_andrew wrote: post #121

Here's one thing I don't get. Why all the rigamarole of hoping you get a valid response, adding last-mile validators to detect invalid responses, trying to beg the model to pretty please give me the syntax I'm asking for... ...when you can guarantee a valid JSON syntax by only sampling tokens that are valid? Instead of greedily picking the highest-scoring token every time, you select the highest-scoring token that co…

It’s like the story of the brown M&Ms[0]. If the model is returning semantically correct data, you would hope that it can at least get the syntax correct. And if it can’t then you ought to throw the response away anyway.

Also I believe that such a method cannot capture the full complexity of TypeScript types.

[0] https://www.snopes.com/fact-check/brown-out/

Re: TypeChat

#124

Post by rezonant »

akavi wrote: post #80
rezonant wrote: post #55

Earlier quoted context omitted.

For the TS compiler: If you took each generation step, closed any partial JSON objects (ie close any open `{`), checked that it was valid JSON and then validated it using a deep version of Partial , that should do the trick.

Not for even the simplest schemas. Eg, given even the type: {"aLongerKey": "value"} The generation prefix: {"a would by your algorithm produce the following invalid output: {"a}

That's why I mentioned you check the JSON validity first. You'd obviously need to continue letting it generate tokens until you can parse the JSON to check if the type is partial. You could of course close even the quotes but then you'd get "not valid" signals from TS when the AI is like "just let me finish!" :-)

Re: TypeChat

#125

Post by perryizgr8 »

bottlepalm wrote: post #22

How does no voice assistant (Apple, Google, Amazon, Microsoft) integrate LLMs into their service yet, and how has OpenAI not released their own voice assistant? Also like RSS, if there were some standard URL a websites exposed for AI interaction, using this TypeChat to expose the interfaces, we'd be well on our way here.

Talking to Alexa is laughable now, after having interacted with ChatGPT and Bing. It's so frustrating to see capable hardware being let down by crappy software for years upon years.

Re: TypeChat

#126

Post by sdwr »

J_Shelby_J wrote: post #62
9dev wrote: post #37

Earlier quoted context omitted.

ChatGPT isn’t the limiting factor here, a good way to expose the toggles is. I recently tried to expose our company CRM to employees by means of a Teams bot they could ask for stuff in natural language (like „send an invite link to newlead@example.org“ or „how many MAUs did customer Foo have in June“), but while I almost got there, communicating an ever-growing set of actionable commands (with an arbitrary number of…

https://github.com/ShelbyJenkins/LLM-OpenAPI-minifier I have a working solution to exposing the toggles. I’m integrating it into the bot I have in the other repo. Goal is you point to an openapi spec and then GPT can run choose and run functions. Basically Siri but with access to any API.

Good shit!

Re: TypeChat

#127

Post by CGamesPlay »

andy_xor_andrew wrote: post #121

Here's one thing I don't get. Why all the rigamarole of hoping you get a valid response, adding last-mile validators to detect invalid responses, trying to beg the model to pretty please give me the syntax I'm asking for... ...when you can guarantee a valid JSON syntax by only sampling tokens that are valid? Instead of greedily picking the highest-scoring token every time, you select the highest-scoring token that co…

OpenAI doesn’t expose this information because it makes it vastly easier to train your model off theirs.

Re: TypeChat

#128

Post by TeeWEE »

mahalex wrote: post #39

So, it's a thing that appends "please format your response as the following JSON" to the prompt", then validates the actual response against the schema, all in a "while (true)" loop (literally) until it succeeds. This unbelievable achievement is a work of seven people (authors of the blog post). Honestly, this is getting beyond embarrassing. How is this the world we live in?

Yeah it’s basically a retry loop. I’m curious about the average response time and the worst case amount of iterations.

At best, all these “retry until successfully” are just hacks to bridge the formal world with the stochastic. It’s just useless without some stats on how it performs.

And even if it conforms. Your not sure the data makes sense. Probably .. but exactly that probably

I would not recommend using this in production.

Re: TypeChat

#130

Post by csomar »

andy_xor_andrew wrote: post #121

Here's one thing I don't get. Why all the rigamarole of hoping you get a valid response, adding last-mile validators to detect invalid responses, trying to beg the model to pretty please give me the syntax I'm asking for... ...when you can guarantee a valid JSON syntax by only sampling tokens that are valid? Instead of greedily picking the highest-scoring token every time, you select the highest-scoring token that co…

The LLM will be able to handle more complex scenarios. I could imagine a use-case: If you are ordering from a self-vending machine, instead of having to go through the whole process you just say your order out loud. You can say, for example, a couple chocolate bars and the LLM tries to guess from inventory.

Of course, if you are on the web, it makes no sense. It is much easier to use the mouse to click on a couple of items.

Post reply on HN