Live data from Hacker News

OpenAI: Streaming is now available in the Assistants API

platform.openai.com

71–80 of 89 posts

Re: OpenAI: Streaming is now available in the Assistants API

#71
post #57

Earlier quoted context omitted.

How is hiding it behind a loading spinner any better? You still can't spam it with questions since you need to wait for it to finish. With streaming you can at least hit the stop button if it looks incorrect, so you actually spam it more with it enabled.

For me, the constant visual changes of new parts being streamed in are annoying, and straining on the eyes. Ideally, web frontends would honor `prefers-reduced-motion` and buffer the response when set.

Personally, I've fallen in love with that visual effect of streaming text you're talking about. It's a bit pavlovian, but I think in my head it signifies that I'm reading something high signal (even though it isn't always).

Re: OpenAI: Streaming is now available in the Assistants API

#72
post #19
post #14

Earlier quoted context omitted.

Edit/remove/retry is just including the whole conversation over again (IIUC this is even how the app works.) It's part of why the API is so expensive

The Assistants API doesn't let you recreate the conversation (with edits or not) because you can't (re)create messages with role=assistant.

This was indeed true in the beginning, and I don’t know if this has changed. Inserting messages with Assistant role is crucial for many reasons, such as if you want to implement caching, or otherwise edit/compress a previous assistant response for cost or other reason.

At the time I implemented a work-around in Langroid[1]: since you can only insert a “user” role message, prepend the content with ASSISTANT: whenever you want it to be treated as an assistant role. This actually works as expected and I was able to do caching. I explained it in this forum:

https://community.openai.com/t/add-custom-roles-to-messages-...

[1] the Langroid code that adds a message with a given role, using this above “assistant spoofing trick”:

https://github.com/langroid/langroid/blob/main/langroid/agen...

Re: OpenAI: Streaming is now available in the Assistants API

#73
post #66

Assistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response. I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad an…

I still don't even know what the Assistant API is supposed to afford me.

Re: OpenAI: Streaming is now available in the Assistants API

#74
post #73
post #66

Assistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response. I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad an…

I still don't even know what the Assistant API is supposed to afford me.

It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.

Re: OpenAI: Streaming is now available in the Assistants API

#75
post #74
post #73

Earlier quoted context omitted.

I still don't even know what the Assistant API is supposed to afford me.

It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.

Is the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.

Re: OpenAI: Streaming is now available in the Assistants API

#76
post #74

Earlier quoted context omitted.

It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.

Is the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.

What training? Beyond supplying context, I don't think assistants has any fine tuning involved.

Re: OpenAI: Streaming is now available in the Assistants API

#77
post #74

Earlier quoted context omitted.

It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.

Is the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.

So, the Assistant API in OpenAI is just a wrapper over the Chat API. They let you choose which model you would like to use, so as a result of you fine tune a model you should be able to use it.

However I never tried fine tuning, I rely on RAG and the Assistant API does provide you some tools to make this a bit easier. What tools? They provide an "editor interface" where you can set function calls, upload some files and access the code interpreter.

So if you are making a chatbot for Company Y, you can create an assistant which has information about Company Y in the system prompt and also can access up to date information about the company through function calls you define and the files you upload.

If you use only Chat API, you will have to handle these stuff yourself. Actually, though I'm using Chat API I do use the Assistant Editor UI to manage the functions and the system prompts. What I do is, I retrieve the assistant info from the OpenAI Assistant API and then I use this on Chat API. This way I don't have to bother with creating my own UI or fiddle with text files or the code.

As Assistant API is just a wrapper, most the data structures I receive from Assistant API directly work in Chat API.

Re: OpenAI: Streaming is now available in the Assistants API

#78

Earlier quoted context omitted.

Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.

Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)

Who can I reach out to for feedback on the web UI? Specifically, the chat.openai.com interface.

Web developer/designer for 24 years so I have a lot of ideas

Re: OpenAI: Streaming is now available in the Assistants API

#79
post #74
post #73

Earlier quoted context omitted.

I still don't even know what the Assistant API is supposed to afford me.

It's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.

Yeah that was kind of my idea, it does not serve much if any purpose and only limits the capability.

Re: OpenAI: Streaming is now available in the Assistants API

#80

Earlier quoted context omitted.

Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.

Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)

does your team do usability tests on the apis before launching them?

if you got 3-5 developers to try and use one of the sdks to build something, i bet you'd see common trends.

e.g. we recently had to update an assistant with new data everyday and get 1 response, and this is what the engineer came up with. probably it could be improved, but this is really ugly

``` const file = await openai.files.create({ file: fs.createReadStream(fileName), purpose: 'assistants', }) await openai.beta.assistants.update(assistantId, { file_ids: [file.id], })

  const { id: threadId } = await openai.beta.threads.create({
   messages: [
    {
     role: 'user',
     content:
      'Create PostSuggestions from the file. Remember to keep the style fun and engaging, not just regurgitating the headlines. Read the WHOLE article.',
    },
   ],
  })
  const getSuggestions = async (runIdArg: string) => {
   return new Promise(resolve => {
    const checkStatus = async () => {
     const { status, last_error, required_action } = await openai.beta.threads.runs.retrieve(threadId, runIdArg)

     console.log({ status })
     if (status === 'requires_action') {
      if (required_action?.type === 'submit_tool_outputs') {
       required_action?.submit_tool_outputs?.tool_calls?.forEach(async toolOutput => {
        const parsed = PostSuggestions.safeParse(JSON.parse(toolOutput.function.arguments))
        if (parsed.success) {
         await openai.beta.threads.runs.cancel(threadId, runIdArg)
         resolve(parsed.data)
        } else {
         console.error(`failed to parse args from openai to my type (errors=${parsed.error.errors}`)
        }
       })
      } else {
       console.error(`requires_action, but not submit_tool_outputs (type=${required_action?.type})`)
      }
     } else if (status === 'completed') {
      throw new Error(`status is completed, but no data. supposed to go to requires_action`)
     } else if (status === 'failed') {
      throw new Error(`message=${last_error?.message}, code=${last_error?.code}`)
     } else {
      setTimeout(checkStatus, 500)
     }
    }

    checkStatus()
   })
  }
  const { id: runId } = await openai.beta.threads.runs.create(threadId, {
   assistant_id: assistantId,
  })
  console.time('openai create thread')
  const newsSuggestions = await getSuggestions(runId)
  console.timeEnd('openai create thread')
```
Post reply on HN