Earlier quoted context omitted.
Elaborate?
"YET ANOTHER shiny new toy to distract me. Can't help myself even though I think it's mostly a waste of time" Am I just projecting? Relatable, in any case :)
Yep, you captured the moment ^_^
OpenAI: Streaming is now available in the Assistants API
21–30 of 89 posts
Re: OpenAI: Streaming is now available in the Assistants API
#22Earlier quoted context omitted.
Edit/remove/retry is just including the whole conversation over again (IIUC this is even how the app works.) It's part of why the API is so expensive
The Assistants API doesn't let you recreate the conversation (with edits or not) because you can't (re)create messages with role=assistant.
Not true
Re: OpenAI: Streaming is now available in the Assistants API
#23Any way to have a consistent system prompt across queries without sending it (and using tokens) for each completion?
The assistant has its own "instructions" (replacement for system prompt)
and then on each run, you have the option to add more guidance to the run explicitly, without modifying the assistant instructions (system prompt)
It's a little bit different but kind of the same
Re: OpenAI: Streaming is now available in the Assistants API
#24Throwing a feature request in here just in case someone from OpenAI sees it.
I'd really like it if the streaming versions of their APIs could return a token usage count at the end.
The non-streaming APIs do this right now:
curl https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" -d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "A short fun fact about pigeons"
}
]
}'
{
"id": "chatcmpl-92UiIWQaf442wq7Eyp7kF8ge0e3fE",
"object": "chat.completion",
"created": 1710381746,
"model": "gpt-3.5-turbo-0125",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Pigeons are one of the few bird species that can drink water by sucking it up through their beaks, rather than tilting their heads back to swallow."
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 14,
"completion_tokens": 33,
"total_tokens": 47
},
"system_fingerprint": "fp_4f0b692a78"
}
But if I add "stream": true I get back an SSE stream that looks like this:
...
data: {"id":"chatcmpl-92Uk81oNjrcUJQnPX8fSNqFINLfSI","object":"chat.completion.chunk","created":1710381860,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_4f0b692a78","choices":[{"index":0,"delta":{"content":"."},"logprobs":null,"finish_reason":null}]}
data: {"id":"chatcmpl-92Uk81oNjrcUJQnPX8fSNqFINLfSI","object":"chat.completion.chunk","created":1710381860,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_4f0b692a78","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}
data: [DONE]
I noticed the other day that the Claude streaming API returns a "usage" block with the last message. I'd love it if OpenAI's API did the same thing.
I need this right now because I'm starting to build features for end users of my own software, and I want to be able to give them X,000 tokens "free" before starting to charge them for extras. Counting those tokens myself (probably using tiktoken) is code I'd rather not have to write - especially since features like tools/functions or images make counting tokens a lot less obvious.
Re: OpenAI: Streaming is now available in the Assistants API
#25This website is now like 30% about this probability based autocomplete nonsense. Feels like all those bitcoin hypes and "running everything on blockchain" fad of few years ago. Now it's running everything through "large autocomplete" model.
I really hope this will fade and focus will turn back to highlighting some broader actual human ingenuity in IT, rather than constant stream of "we used autocomplete for this new thing" or "we build this new API for this glorified autocomplete".
Boring.
Re: OpenAI: Streaming is now available in the Assistants API
#26Throwing a feature request in here just in case someone from OpenAI sees it. I'd really like it if the streaming versions of their APIs could return a token usage count at the end. The non-streaming APIs do this right now: curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "c…
This and/or being able to fetch the responses with their token usage by id. What is that ID for without a way to retrieve the completions with it?
Re: OpenAI: Streaming is now available in the Assistants API
#27Throwing a feature request in here just in case someone from OpenAI sees it. I'd really like it if the streaming versions of their APIs could return a token usage count at the end. The non-streaming APIs do this right now: curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "c…
We do the token counting on our end literally just running tiktoken on the content chunks (although I think usually its one token per chunk). Its a bit annoying and I too expected they'd have the usage block but its one line of code if you already have tiktoken available. I've found the accounting on my side lines up well with what we see on our usage dashboard.
Re: OpenAI: Streaming is now available in the Assistants API
#28This was one of the limitations of the Assistants API that made me entirely ignore it up until now. I am curious if the Assistants API lets you edit/remove/retry messages yet. I don't see anything implying this has changed. It's annoying that the Assistants API doesn't give you enough control to support basic things that the ChatGPT app does.
Like the other commenter said, edit/remove/retry messages can be implemented by the API client already. The API doesn't maintain state so every new message in a "conversation" includes previous messages as context. To edit a message you would re-submit the conversation history with the desired changes. I get what you're asking for though. It would be nice if this was easier. But that would require OpenAI changing the…
That is what "assistant api" is, you create a thread and add new user message to the thread. The messages are stored on the server.
There is an API to modify messages, though I am not sure of its constraints.
Re: OpenAI: Streaming is now available in the Assistants API
#29Openai banned my account for suspicious payment activities, and I never was able to talk to a real person. Just several layers of chat bots posing as people. I literally want to give them my money and can't. Every few weeks for shirts and giggles i send an email to them saying, "any update on this?"
I guess it's time for Claude 3 (I imagine you were using it for the LLMs).
Re: OpenAI: Streaming is now available in the Assistants API
#30For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language.
2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator, or better - an asynchronous iterator, since this is streaming over the network. We could be destructing instances of named tuples with pattern matching, or even just doing `"".join(delta.text for delta in prompt (...)`. But no here subclass this instead, tells me the wrapper around a web API.