Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Function calling and other API updates

openai.com

Post reply on HN
Print view View original

21–30 of 174 posts

Re: Function calling and other API updates

#21

Post by hackernewds »

sashank_1509 wrote: post #18
bioemerl wrote: post #11

I hate seeing these guys succeed because everyone of their successes is a new day that AI becomes less accessible to the average person and more locked behind their APIs.

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even i…

An "average" person is not someone who knows how to call an API. Perhaps only on HN

Re: Function calling and other API updates

#23

Post by civilitty »

newhouseb wrote: post #17
misterdata wrote: post #10

Earlier quoted context omitted.

As they are accepting a JSON schema for the function calls, it is likely they are using token biasing based on the schema (using some kind of state machine that follows along with the tokens and only allows the next token to be a valid one given the grammar/schema). I have successfully implemented this for JSON Schema (limited subset) on llama.cpp. See also e.g. this implementation: https://github.com/1rgs/jsonformer

As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation: The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters). So sadly, it is just fine tuning. There's no hard biasing applied :(. You were…

Or there’s a trade off between more complex schemas and logit bias going off the rails since there’s probably little to no backtracking.

Re: Function calling and other API updates

#24

Post by braindead_in »

> With these updates, we’ll be inviting many more people from the waitlist to try GPT-4 over the coming weeks, with the intent to remove the waitlist entirely with this model. Thank you to everyone who has been patiently waiting, we are excited to see what you build with GPT-4!

What about the rate limits? The docs say that it's 200 RPM and "We are unable to accommodate requests for rate limit increases due to capacity constraints."

Re: Function calling and other API updates

#25

Post by sergiotapia »

What are you thoughts here regarding functions:

I have some data I can pass in CSV format to the context and ask a question against that data. "Who are my best customers?" and pass in a CSV of the top 100 customers.

vs

I create a function that returns my best customers and call a ChatGPT function.

When would I use one or the other? The function call seems like it would be more accurate with better guardrails, but it does require me to know what questions my users will make beforehand.

Maybe that's the point, use functions when you know what kind of questions your users will make.

Re: Function calling and other API updates

#26

Post by IMTDb »

hackernewds wrote: post #21
sashank_1509 wrote: post #18

Earlier quoted context omitted.

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even i…

An "average" person is not someone who knows how to call an API. Perhaps only on HN

Then an "average" person is certainly not someone who is able to download and run an LLM on their device.

Re: Function calling and other API updates

#28

Post by wellthisisgreat »

sashank_1509 wrote: post #18
bioemerl wrote: post #11

I hate seeing these guys succeed because everyone of their successes is a new day that AI becomes less accessible to the average person and more locked behind their APIs.

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even i…

> LLAMA 65B model at a very slow pace

How does it compare to GPT 3.5, or 4? I mean if you ask the same questions. Is it usable at all?

I tried the models that work with 4090 and they were completely useless for anything practical (code questions, etc.). Curiosities sure, but on Eliza level.

Re: Function calling and other API updates

#29

Post by rolisz »

hackernewds wrote: post #21
sashank_1509 wrote: post #18

Earlier quoted context omitted.

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even i…

An "average" person is not someone who knows how to call an API. Perhaps only on HN

If they don't know how to call an API, they won't know how to run local models (at the moment it's quite a pain to set-up all the dependencies)

Re: Function calling and other API updates

#30

Post by jumploops »

newhouseb wrote: post #17
misterdata wrote: post #10

Earlier quoted context omitted.

As they are accepting a JSON schema for the function calls, it is likely they are using token biasing based on the schema (using some kind of state machine that follows along with the tokens and only allows the next token to be a valid one given the grammar/schema). I have successfully implemented this for JSON Schema (limited subset) on llama.cpp. See also e.g. this implementation: https://github.com/1rgs/jsonformer

As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation: The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters). So sadly, it is just fine tuning. There's no hard biasing applied :(. You were…

They may have just fine-tuned 3.5 to respond with valid JSON more times than not.

Building magic functions[0] I ran into many examples where JSONSchema broke for gpt-3.5-turbo but worked well for gpt-4.

[0] https://github.com/jumploops/magic

Post reply on HN