Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI: Streaming is now available in the Assistants API

platform.openai.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: OpenAI: Streaming is now available in the Assistants API

#41

Post by doctorpangloss »

pedrovhb wrote: post #30

For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator,…

My experience is their official Python library was easy to use, no surprises, everything is typed and generated from the OpenAPI spec in a thoughtful way.

The tools are great because they don't invent their own DSL, they "just" use JSON schemas.

Maybe they ought to contribute changes to OpenAPI to support streaming APIs better.

In contrast so many startups make their own annotation-driven DSLs for Python with their branding slapped over everything. It gives desperate-for-lock-in vibes. The last people OpenAI should be taking advice from for their API design is this forum.

Re: OpenAI: Streaming is now available in the Assistants API

#42

Post by thejohnconway »

GaggiX wrote: post #29
potsandpans wrote: post #20

Openai banned my account for suspicious payment activities, and I never was able to talk to a real person. Just several layers of chat bots posing as people. I literally want to give them my money and can't. Every few weeks for shirts and giggles i send an email to them saying, "any update on this?"

I guess it's time for Claude 3 (I imagine you were using it for the LLMs).

My Anthropic account was suspended for suspicious activity, even though I never used it. I had forgotten I had signed up, and tried to sign up using a new email with the same phone number. Locked out forever.

Re: OpenAI: Streaming is now available in the Assistants API

#43

Post by ukuina »

potsandpans wrote: post #20

Openai banned my account for suspicious payment activities, and I never was able to talk to a real person. Just several layers of chat bots posing as people. I literally want to give them my money and can't. Every few weeks for shirts and giggles i send an email to them saying, "any update on this?"

I suspected as much when one of their support "personnel" used the phrase "I apologize for the earlier confusion..." (there was no confusion, I was simply contradicting what they were saying)

Re: OpenAI: Streaming is now available in the Assistants API

#44

Post by romanhn »

chaxor wrote: post #38
megous wrote: post #25

This website is now like 30% about this probability based autocomplete nonsense. Feels like all those bitcoin hypes and "running everything on blockchain" fad of few years ago. Now it's running everything through "large autocomplete" model. I really hope this will fade and focus will turn back to highlighting some broader actual human ingenuity in IT, rather than constant stream of "we used autocomplete for this new…

"old man yells at cloud" Seriously though, it's not going away no matter how much anyone hates it. Emails and blogs will continue to be written with it, letters of recommendation will be/are written with it, Presidential speeches will be written with it, academic articles will be / are written with it (almost all ml and cs research is), news is written with it... It's not going to stop, but it will _probably_/_very l…

It occurs to me that over time, reading comprehension will become significantly more important than the ability to write. Anyone will be able to write something smart-sounding with AI's help, but it'll take real skill to make sure the output is correct and appropriate.

Re: OpenAI: Streaming is now available in the Assistants API

#45

Post by xcv123 »

megous wrote: post #25

This website is now like 30% about this probability based autocomplete nonsense. Feels like all those bitcoin hypes and "running everything on blockchain" fad of few years ago. Now it's running everything through "large autocomplete" model. I really hope this will fade and focus will turn back to highlighting some broader actual human ingenuity in IT, rather than constant stream of "we used autocomplete for this new…

If deep learning algorithms are "autocomplete" then so is the human mind when it strings words together. No, that's not how it works.

Re: OpenAI: Streaming is now available in the Assistants API

#46

Post by mvkel »

willsmith72 wrote: post #35
pedrovhb wrote: post #30

For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator,…

Everything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.

...except for all the others.

Use Claude in Safari and the browser completely locks up after a single response.

Re: OpenAI: Streaming is now available in the Assistants API

#47

Post by rattray »

pedrovhb wrote: post #30

For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator,…

Hey there, I helped design the Python library.

The `stream` context manager actually does expose an async iterator (in the async client), so you could instead do this for the simple case:

    with client.beta.threads.runs.create_and_stream(…) as stream:
      async for text in stream.text_deltas:
        print(text, end="", flush=True)
which I think is roughly what you want.

Perhaps the docs should be updated to highlight this simple case earlier.

We are also considering expanding this design, and perhaps replacing the callbacks, like so:

    with client.beta.threads.runs.create_and_stream(…) as stream:
      async for event in stream.all_events:
        if event.type == 'text_delta':
          print(event.delta.value, end='')
        elif event.type == 'run_step_delta':
          event.snapshot.id
          event.delta.step_details...
which I think is also more in line with what you expect. (you could also `match event: case TextDelta: …`).

Note that the context manager is required because otherwise there's no way to tell if you `break` out of the loop (or otherwise stop listening to the stream) which means we can't close the request (and you both keep burning tokens and leak resources in your app).

Re: OpenAI: Streaming is now available in the Assistants API

#48

Post by pedrovhb »

doctorpangloss wrote: post #41
pedrovhb wrote: post #30

For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator,…

My experience is their official Python library was easy to use, no surprises, everything is typed and generated from the OpenAPI spec in a thoughtful way. The tools are great because they don't invent their own DSL, they "just" use JSON schemas. Maybe they ought to contribute changes to OpenAPI to support streaming APIs better. In contrast so many startups make their own annotation-driven DSLs for Python with their b…

How is suggesting the use of iterators and named tuples related to creating domain specific languages? If anything I'd say they're a much more generic and universally recognizable approach than having users subclass `AssistantEventHandler` to be passed to `client.beta.threads.runs.create_and_stream`, the context manager. This is very much a long way past just using JSON schemas but that part is ok - there's a REST API, and there's a library. If you're keen on the simplicity of JSON schema then by all means use the API with `requests` or your preferred http client library. Since that's always an option, it stands to reason that the point of having a dedicated library is to provide thoughtful abstractions that make it easier to use the service.

What I'm arguing is precisely that the abstractions in the library (such as the `AssistantEventHandler` shown in the article) are ineffective in making things simpler. They force you to over-engineer solutions and distribute state unnecessarily and be aware of that specific class interface when it could've just been something you use in a `for x in y` loop like everyone would know to do without spending an afternoon looking over docs and figuring out how the underlying implicit FSM works.

Re: OpenAI: Streaming is now available in the Assistants API

#49

Post by dns_snek »

xcv123 wrote: post #45
megous wrote: post #25

This website is now like 30% about this probability based autocomplete nonsense. Feels like all those bitcoin hypes and "running everything on blockchain" fad of few years ago. Now it's running everything through "large autocomplete" model. I really hope this will fade and focus will turn back to highlighting some broader actual human ingenuity in IT, rather than constant stream of "we used autocomplete for this new…

If deep learning algorithms are "autocomplete" then so is the human mind when it strings words together. No, that's not how it works.

[citation needed]

Just because that makes for a nice narrative in the copyright infringement argument, doesn't make it so.

We know next to nothing about how the human brain works.

Re: OpenAI: Streaming is now available in the Assistants API

#50

Post by ametrau »

rattray wrote: post #47
pedrovhb wrote: post #30

For all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator,…

Hey there, I helped design the Python library. The `stream` context manager actually does expose an async iterator (in the async client), so you could instead do this for the simple case: with client.beta.threads.runs.create_and_stream(…) as stream: async for text in stream.text_deltas: print(text, end="", flush=True) which I think is roughly what you want. Perhaps the docs should be updated to highlight this simple…

Context managers are a great abstraction.
Post reply on HN