Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Function calling and other API updates

openai.com

Post reply on HN
Print view View original

11–20 of 174 posts

Re: Function calling and other API updates

#12

Post by com2kid »

GPT3.5 has been undergoing constant improvements, this price decrease (and context length increase) is great news!

The main problem I see with people using GPT3.5 is they try and ask it to "write a short story about aliens" and then they get back a crap boring response that sounds like it was written by an AI that was asleep at the wheel.

Good creative prompts are long and detailed, and to get the best results you really need to be able to tune temperature / top_p. Even small changes to a 3 paragraph prompt can result in a dramatic changes in the output, and unless people are willing to play around with prompting, they won't get good results.

None of the prompt guides I've seen really cover pushing GPT3.5 to its limit, I've published one of my more complicated prompts[1] but getting GPT3.5 to output good responses in just this limited sense has taken a lot of work.

As for the longer context, output length is different than following instructions, especially for a lot of use cases, pushing more input tokens is of as much interest as having more output tokens.

From what I have explored, even at 4k context length, with a detailed prompt earlier instructions in the prompt are "forgotten" (or maybe just ignored). The blog post calls out better understanding of input text, but again, I hope that isn't orthogonal to following instructions!

Finally in regards to function outputs, I wonder if it is a second layer they are running on top of the initial model output. I have always had a challenge getting the model to output parsable responses, there is a definite trade off between written creativity and well formatted responses, and to some extent having a creative AI extend out the format I specify has been really nice because it has allowed me to add features I did not think of myself!

[1] https://github.com/devlinb/arcadia/blob/main/backend/src/rou...

Re: Function calling and other API updates

#13

Post by civilitty »

Finally! I’ve been getting the shakes waiting for next OpenAI release.

16k context with 3.5-turbo is huge. It’ll make all those dime a dozen document driven assistants a lot more useful.

I’m curious to see if people will figure out ways to hack functions to get more reliable structured JSON data out out of GPT without tons of examples, giving lots more context room to play with

Re: Function calling and other API updates

#17

Post by newhouseb »

misterdata wrote: post #10
minimaxir wrote: post #7

The big feature here is the function calls, as this is effectively a replacement for the "Tools" feature of Agents popularized by LangChain, except in theory much more efficient since it may not require an extra call to the API. In the case of LangChain which selects Tools and their functional outputs through JSON Markdown shennanigans (which often fails and causes ParsingErrors), this variant of ChatGPT appears to b…

As they are accepting a JSON schema for the function calls, it is likely they are using token biasing based on the schema (using some kind of state machine that follows along with the tokens and only allows the next token to be a valid one given the grammar/schema). I have successfully implemented this for JSON Schema (limited subset) on llama.cpp. See also e.g. this implementation: https://github.com/1rgs/jsonformer

As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation:

  The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters).
So sadly, it is just fine tuning. There's no hard biasing applied :(. You were so close, but so far OpenAI!

[1] https://github.com/newhouseb/clownfish

[2] https://platform.openai.com/docs/guides/gpt/function-calling

Re: Function calling and other API updates

#18

Post by sashank_1509 »

bioemerl wrote: post #11

I hate seeing these guys succeed because everyone of their successes is a new day that AI becomes less accessible to the average person and more locked behind their APIs.

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference, and even then it will be a slow, unpolished experience.

On the other hand OpenAI’s API makes things blazingly fast and dirt cheap to the average consumer. It honestly does feel like they have enabled the power of AI to be accessible to anyone with a laptop. If that requires fending off competition from Behemoths like Google, Meta by not releasing model weights then so be it. This critique would be more apt to Nvidia who are artificially increasing datacenter GPU prices thus pricing out the average consumer. OpenAI is doing the opposite.

Re: Function calling and other API updates

#20

Post by hackernewds »

com2kid wrote: post #12

GPT3.5 has been undergoing constant improvements, this price decrease (and context length increase) is great news! The main problem I see with people using GPT3.5 is they try and ask it to "write a short story about aliens" and then they get back a crap boring response that sounds like it was written by an AI that was asleep at the wheel. Good creative prompts are long and detailed, and to get the best results you re…

Would you mind sharing what are then good prompt structures? Seems you have a grasp
Post reply on HN