Earlier quoted context omitted.
How is hiding it behind a loading spinner any better? You still can't spam it with questions since you need to wait for it to finish. With streaming you can at least hit the stop button if it looks incorrect, so you actually spam it more with it enabled.
For me, the constant visual changes of new parts being streamed in are annoying, and straining on the eyes. Ideally, web frontends would honor `prefers-reduced-motion` and buffer the response when set.
OpenAI: Streaming is now available in the Assistants API
71–80 of 89 posts
Re: OpenAI: Streaming is now available in the Assistants API
#72Earlier quoted context omitted.
Edit/remove/retry is just including the whole conversation over again (IIUC this is even how the app works.) It's part of why the API is so expensive
The Assistants API doesn't let you recreate the conversation (with edits or not) because you can't (re)create messages with role=assistant.
At the time I implemented a work-around in Langroid[1]: since you can only insert a “user” role message, prepend the content with ASSISTANT: whenever you want it to be treated as an assistant role. This actually works as expected and I was able to do caching. I explained it in this forum:
https://community.openai.com/t/add-custom-roles-to-messages-...
[1] the Langroid code that adds a message with a given role, using this above “assistant spoofing trick”:
https://github.com/langroid/langroid/blob/main/langroid/agen...
Re: OpenAI: Streaming is now available in the Assistants API
#73Assistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response. I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad an…
Re: OpenAI: Streaming is now available in the Assistants API
#74Assistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response. I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad an…
I still don't even know what the Assistant API is supposed to afford me.
Re: OpenAI: Streaming is now available in the Assistants API
#75Earlier quoted context omitted.
I still don't even know what the Assistant API is supposed to afford me.
It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.
Re: OpenAI: Streaming is now available in the Assistants API
#76Earlier quoted context omitted.
It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.
Is the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.
Re: OpenAI: Streaming is now available in the Assistants API
#77Earlier quoted context omitted.
It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.
Is the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.
However I never tried fine tuning, I rely on RAG and the Assistant API does provide you some tools to make this a bit easier. What tools? They provide an "editor interface" where you can set function calls, upload some files and access the code interpreter.
So if you are making a chatbot for Company Y, you can create an assistant which has information about Company Y in the system prompt and also can access up to date information about the company through function calls you define and the files you upload.
If you use only Chat API, you will have to handle these stuff yourself. Actually, though I'm using Chat API I do use the Assistant Editor UI to manage the functions and the system prompts. What I do is, I retrieve the assistant info from the OpenAI Assistant API and then I use this on Chat API. This way I don't have to bother with creating my own UI or fiddle with text files or the code.
As Assistant API is just a wrapper, most the data structures I receive from Assistant API directly work in Chat API.
Re: OpenAI: Streaming is now available in the Assistants API
#78Earlier quoted context omitted.
Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.
Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)
Web developer/designer for 24 years so I have a lot of ideas
Re: OpenAI: Streaming is now available in the Assistants API
#79Earlier quoted context omitted.
I still don't even know what the Assistant API is supposed to afford me.
It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.
Re: OpenAI: Streaming is now available in the Assistants API
#80Earlier quoted context omitted.
Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.
Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)
if you got 3-5 developers to try and use one of the sdks to build something, i bet you'd see common trends.
e.g. we recently had to update an assistant with new data everyday and get 1 response, and this is what the engineer came up with. probably it could be improved, but this is really ugly
``` const file = await openai.files.create({ file: fs.createReadStream(fileName), purpose: 'assistants', }) await openai.beta.assistants.update(assistantId, { file_ids: [file.id], })
const { id: threadId } = await openai.beta.threads.create({
messages: [
{
role: 'user',
content:
'Create PostSuggestions from the file. Remember to keep the style fun and engaging, not just regurgitating the headlines. Read the WHOLE article.',
},
],
})
const getSuggestions = async (runIdArg: string) => {
return new Promise(resolve => {
const checkStatus = async () => {
const { status, last_error, required_action } = await openai.beta.threads.runs.retrieve(threadId, runIdArg)
console.log({ status })
if (status === 'requires_action') {
if (required_action?.type === 'submit_tool_outputs') {
required_action?.submit_tool_outputs?.tool_calls?.forEach(async toolOutput => {
const parsed = PostSuggestions.safeParse(JSON.parse(toolOutput.function.arguments))
if (parsed.success) {
await openai.beta.threads.runs.cancel(threadId, runIdArg)
resolve(parsed.data)
} else {
console.error(`failed to parse args from openai to my type (errors=${parsed.error.errors}`)
}
})
} else {
console.error(`requires_action, but not submit_tool_outputs (type=${required_action?.type})`)
}
} else if (status === 'completed') {
throw new Error(`status is completed, but no data. supposed to go to requires_action`)
} else if (status === 'failed') {
throw new Error(`message=${last_error?.message}, code=${last_error?.code}`)
} else {
setTimeout(checkStatus, 500)
}
}
checkStatus()
})
}
const { id: runId } = await openai.beta.threads.runs.create(threadId, {
assistant_id: assistantId,
})
console.time('openai create thread')
const newsSuggestions = await getSuggestions(runId)
console.timeEnd('openai create thread')
```