Has tool use accuracy improved?
OpenAI: Streaming is now available in the Assistants API
61–70 of 89 posts
Re: OpenAI: Streaming is now available in the Assistants API
#62they should do streaming for voice inputs on the chatgpt app. right now it's very slow. Voice interfaces need to be streaming
Re: OpenAI: Streaming is now available in the Assistants API
#63For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator,…
Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.
Hi there, I help design the OpenAI APIs. Would you be able to share more?
You can reply here or email me at atty@openai.com.
(Please don't hold back; we would love to hear the pain points so we can fix them.)
Re: OpenAI: Streaming is now available in the Assistants API
#64Earlier quoted context omitted.
Elaborate?
"YET ANOTHER shiny new toy to distract me. Can't help myself even though I think it's mostly a waste of time" Am I just projecting? Relatable, in any case :)
I immediately implemented streaming into my rocketchat gpt bot, was definitely a distraction but my colleagues liked it. No more waiting until the complete response is sent.
Re: OpenAI: Streaming is now available in the Assistants API
#65Has anyone put out a voice-to-text interface for OpenAI? Or anything in the Ollama-verse?
The mobile app is pretty good Horrendous in non english languages though, the accents are extremely American
Is there a way to use the mobile app on PCs?
I tried with Windows Subsystem for Android but the app refused to work.
Re: OpenAI: Streaming is now available in the Assistants API
#66Assistant API is too much of a beta still.
I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response.
I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad and after a few back and forth messages the cost ballooned to over 10K tokens per message.
When I looked closely at the Assistant API and Chat API, I noticed that Assistant API is just a wrapper over Chat API and acts as a web service that stores the previous messages(so slow response problem was probably due to the web server which keeps track of the context). So I went ahead and implemented my own Assistant API which has more control. For example, I set max token cost per message and if the context balloons over that, I make a request with the context and ask OpenAI to create a summary with all the facts so far, add that summary as a system prompt and my context gets compressed back into reasonable territory.
Re: OpenAI: Streaming is now available in the Assistants API
#67Earlier quoted context omitted.
Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.
Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)
Hey, random question.
Is there a technical reason why log probs aren't available when using function calling? It's not a problem, I've already found a workaround. I was just curious haha.
In general I feel like the function calling/tool use is a bit cumbersome and restrictive so I prefer to write the typescript in the functions namespace myself and just use json_mode.
Re: OpenAI: Streaming is now available in the Assistants API
#68Finally! I've been using the assistants api in building an ai mock interviewer ( https://comp.lol ) but the responses were painfully slow when using the latest iterations of the gpt-4 model. This will make things so much more responsive
I'd still want to see the entire response all at once. Having it stream in while I read it would be very distracting and make it difficult for me to read.
It's a request the front-end developer should be confronted with, not OpenAI.
The website could as well buffer the incoming stream until the used clicks an area to request the display of the next block of the response, once he has finished reading the initial sentences.
Re: OpenAI: Streaming is now available in the Assistants API
#69Earlier quoted context omitted.
yes, it like surfing porn in the early internet year using a dialup modem. One line a the time until you finally can see enough of the picture (reply) to realize that is was not the reply you were looking for. LLM streaming must be a cost saving feature to prevent you from overloading the servers by asking to many questions with in a short time frame. Annoying feature IMHO
How is hiding it behind a loading spinner any better? You still can't spam it with questions since you need to wait for it to finish. With streaming you can at least hit the stop button if it looks incorrect, so you actually spam it more with it enabled.
For me, the constant visual changes of new parts being streamed in are annoying, and straining on the eyes. Ideally, web frontends would honor `prefers-reduced-motion` and buffer the response when set.
Re: OpenAI: Streaming is now available in the Assistants API
#70Earlier quoted context omitted.
I guess it's time for Claude 3 (I imagine you were using it for the LLMs).
My Anthropic account was suspended for suspicious activity, even though I never used it. I had forgotten I had signed up, and tried to sign up using a new email with the same phone number. Locked out forever.
please contact support: https://support.anthropic.com/en/ . we'll get it fixed. sorry!