Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI: Streaming is now available in the Assistants API

platform.openai.com

Post reply on HN
Print view View original

81–89 of 89 posts

Re: OpenAI: Streaming is now available in the Assistants API

#81

Post by willsmith72 »

willsmith72 wrote: post #80
athyuttamre wrote: post #63

Earlier quoted context omitted.

Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)

does your team do usability tests on the apis before launching them? if you got 3-5 developers to try and use one of the sdks to build something, i bet you'd see common trends. e.g. we recently had to update an assistant with new data everyday and get 1 response, and this is what the engineer came up with. probably it could be improved, but this is really ugly ``` const file = await openai.files.create({ file: fs.cre…

just to add to this, it's not helped by the docs. either they don't exist, or the seo isn't working right.

e.g. search term for me "openai assistant service function call node". The first 2 results are community forums, not what i'm looking for. The 3rd is seemingly the official one but doesn't actually answer the question (how to use the assistance service with node and function calling) with an example. The 4th is in python.

https://community.openai.com/t/how-does-function-calling-act...

https://community.openai.com/t/how-assistant-api-function-ca...

https://platform.openai.com/docs/guides/function-calling

https://learn.microsoft.com/en-us/azure/ai-services/openai/h...

Re: OpenAI: Streaming is now available in the Assistants API

#82

Post by SoulAuctioneer »

TowerTall wrote: post #15
cosmotic wrote: post #7

Earlier quoted context omitted.

I'd still want to see the entire response all at once. Having it stream in while I read it would be very distracting and make it difficult for me to read.

yes, it like surfing porn in the early internet year using a dialup modem. One line a the time until you finally can see enough of the picture (reply) to realize that is was not the reply you were looking for. LLM streaming must be a cost saving feature to prevent you from overloading the servers by asking to many questions with in a short time frame. Annoying feature IMHO

It's more about UX, to reduce the perceived delay. LLMs inherently stream their responses, but if you wait until the LLM has finished inference, the user is sitting around twiddling their thumbs.

Re: OpenAI: Streaming is now available in the Assistants API

#83

Post by SoulAuctioneer »

mrtksn wrote: post #66

Assistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response. I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad an…

It does considerably more than (poorly) managing the context window. It also (poorly) enables persistent document storage, knowledge retrieval, function calling and code execution.

Re: OpenAI: Streaming is now available in the Assistants API

#84

Post by rattray »

willsmith72 wrote: post #80
athyuttamre wrote: post #63

Earlier quoted context omitted.

Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)

does your team do usability tests on the apis before launching them? if you got 3-5 developers to try and use one of the sdks to build something, i bet you'd see common trends. e.g. we recently had to update an assistant with new data everyday and get 1 response, and this is what the engineer came up with. probably it could be improved, but this is really ugly ``` const file = await openai.files.create({ file: fs.cre…

I'm sorry for your experience, and thanks very much for sharing the code snippet - that's helpful!

We did indeed code up some sample apps and highlighted this exact concern. We have some helpers planned to make it smoother, which we hope to launch before Assistants GA. For streaming beta, we were focused just on the streaming part of these helpers.

Re: OpenAI: Streaming is now available in the Assistants API

#85

Post by rattray »

msp26 wrote: post #67
athyuttamre wrote: post #63

Earlier quoted context omitted.

Hi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)

Hey, random question. Is there a technical reason why log probs aren't available when using function calling? It's not a problem, I've already found a workaround. I was just curious haha. In general I feel like the function calling/tool use is a bit cumbersome and restrictive so I prefer to write the typescript in the functions namespace myself and just use json_mode.

Have you seen/tried the `.runTools()` helper?

Docs: https://github.com/openai/openai-node?tab=readme-ov-file#aut...

Example: https://github.com/openai/openai-node/blob/bb4bce30ff1bfb06d...

(if what you're fundamentally trying to do is really just get JSON out, then I can see how json_mode is still easier).

Re: OpenAI: Streaming is now available in the Assistants API

#87

Post by arthurcolle »

baobabKoodaa wrote: post #60
arthurcolle wrote: post #23

Earlier quoted context omitted.

The assistant has its own "instructions" (replacement for system prompt) and then on each run, you have the option to add more guidance to the run explicitly, without modifying the assistant instructions (system prompt) It's a little bit different but kind of the same

No, adding run instructions will replace existing instructions for that run

That's what they say ;)

Re: OpenAI: Streaming is now available in the Assistants API

#88

Post by megous »

XCSme wrote: post #31
megous wrote: post #25

This website is now like 30% about this probability based autocomplete nonsense. Feels like all those bitcoin hypes and "running everything on blockchain" fad of few years ago. Now it's running everything through "large autocomplete" model. I really hope this will fade and focus will turn back to highlighting some broader actual human ingenuity in IT, rather than constant stream of "we used autocomplete for this new…

I just added this "autocomplete" in my app, and customers emailed to say they actually love it: https://docs.uxwizz.com/guides/ask-ai-new

Yes, customers will love anything that helps them. You can get customers to love you by adding any kind of automation for stuff they had to do by hand up to that point. Does this mean there should be 10 articles per day shared about "I added XLSX import to my app, so my customers don't have to do data entry via dialogs"?

My point is about repetitiveness of LLM topics. Not about usefullness of LLM itself. And LLMs are glorified autocomplete. Their internals are maybe interesting, but that's often not what's being discussed here or even written about in the shared articles.

Post reply on HN