OpenAI integration is going to be a goldmine for criminals in the future.
Everyone and their momma is gonna start passing poorly validated/sanitized client input to shared sessions of a non-deterministic function.
I love the future!
Native JSON Output from GPT-4
81–90 of 258 posts
Re: Native JSON Output from GPT-4
#82i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…
It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…
I could not get GPT-4 to reliably not give some sort of text response, even if was just a simple "Sure" followed by the JSON.
Re: Native JSON Output from GPT-4
#83Glad we didn't get to far into adopting something like Guardrails. This sort of kills it's main value prop for OpenAI. https://shreyar.github.io/guardrails/
i mean only at the most superficial level. she has a ton of other validators that arent superceded (eg SQL is validated by branching the database - we discussed on our pod https://www.latent.space/p/guaranteed-quality-and-structure )
yeah, listened to the pod (that's how I found out about guardrails!).
fair point, I should have said: "value prop for our use case"... the thing I was most interested in was how well Guardrails structured output.
Re: Native JSON Output from GPT-4
#84Earlier quoted context omitted.
New techniques like semantic caching will help. This is the modern era's version of building a performant social graph.
What's semantic caching?
With LLMs, the inputs are highly variable so exact match caching is generally less useful. Semantic caching groups similar inputs and returns relevant results accordingly. So {"dish":"spaghetti bolognese"} and {"dish":"spaghetti with meat sauce"} could return the same cached result.
Re: Native JSON Output from GPT-4
#85Earlier quoted context omitted.
It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…
even with gpt 4, it hallucinates enough that it’s not reliable, forgetting to open/close brackets and quotes. This sounds like it’d be a big improvement.
Not that it matters now but just doing something like this works 99% of the time or more with 4 and 90% with 3.5.
It is VERY IMPORTANT that you respond in valid JSON ONLY. Nothing before or after. Make sure to escape all strings. Use this format:
{“some_variable”: [describe the variable purpose]}
Re: Native JSON Output from GPT-4
#86This is useful, but for me at least, GPT-4 is unusable because it sometimes takes 30 seconds + to reply to even basic queries.
Re: Native JSON Output from GPT-4
#87This is useful, but for me at least, GPT-4 is unusable because it sometimes takes 30 seconds + to reply to even basic queries.
Also the rate limit is pretty bad if you want to release any type of app
Re: Native JSON Output from GPT-4
#88Earlier quoted context omitted.
"Trivial" is misleading. From OpenAI's docs and demos, the full ReAct workflow is an order of magnitude more difficult than typical ChatGPT API usage with a new set of constaints (e.g. schema definitions) Even OpenAI's notebook demo has error handling workflows which was actually necessary since ChatGPT returned incorrect formatted output.
Maybe trivial isn't the right word, but it's still very straight-forward to get something basic, yet really powerful... ReAct Setup Prompt (goal + available actions) -> Agent "ReAction" -> Parse & Execute Action -> Send Action Response (success or error) -> Agent "ReAction" -> repeat As long as each action has proper validation and returns meaningful error messages, you don't need to even change the control flow. The…
[deleted]
Re: Native JSON Output from GPT-4
#89Earlier quoted context omitted.
What's semantic caching?
With LLMs, the inputs are highly variable so exact match caching is generally less useful. Semantic caching groups similar inputs and returns relevant results accordingly. So {"dish":"spaghetti bolognese"} and {"dish":"spaghetti with meat sauce"} could return the same cached result.
Or store as sentence embedding and calculate the vector distance, but creates many edge cases
Re: Native JSON Output from GPT-4
#90Earlier quoted context omitted.
What's the implication of this new change for Microsoft Guidance, LMQL, Langchain, etc.? It looks like much of their functionality (controlling model output) just became obsolete. Am I missing something?
If anything this removes a major roadblock for libraries/languages that want to employ LLM calls as a primitive, no? Although, I fear the vendor lock-in intensifies here, also given how restrictive and specific the Chat API. Either way, as part of the LMQL team, I am actually pretty excited about this, also with respect to what we want to build going forward. This makes language model programming much easier.
`Although, I fear the vendor lock-in intensifies here, also given how restrictive and specific the Chat API.`
Eh, would be pretty easy to write a wrapper that takes a functions-like JSON Schema object and interpolates it into a traditional "You MUST return ONLY JSON in the following format:" prompt snippet.