OpenAI DevDay 2024 live blog
simonwillison.net
OpenAI DevDay 2024 live blog
1–10 of 99 posts
Re: OpenAI DevDay 2024 live blog
#2Seems mostly standard items so far.
Re: OpenAI DevDay 2024 live blog
#3Blog updates:
- Introducing the Realtime API: https://openai.com/index/introducing-the-realtime-api/
- Introducing vision to the fine-tuning API: https://openai.com/index/introducing-vision-to-the-fine-tuni...
- Prompt Caching in the API: https://openai.com/index/api-prompt-caching/
- Model Distillation in the API: https://openai.com/index/api-model-distillation/
Docs updates:
- Realtime API: https://platform.openai.com/docs/guides/realtime
- Vision fine-tuning: https://platform.openai.com/docs/guides/fine-tuning/vision
- Prompt Caching: https://platform.openai.com/docs/guides/prompt-caching
- Model Distillation: https://platform.openai.com/docs/guides/distillation
- Evaluating model performance: https://platform.openai.com/docs/guides/evals
Additional updates from @OpenAIDevs: https://x.com/OpenAIDevs/status/1841175537060102396
- New prompt generator on https://playground.openai.com
- Access to the o1 model is expanded to developers on usage tier 3, and rate limits are increased (to the same limits as GPT-4o)
Additional updates from @OpenAI: https://x.com/OpenAI/status/1841179938642411582
- Advanced Voice is rolling out globally to ChatGPT Enterprise, Edu, and Team users. Free users will get a sneak peak of it (except EU).
Re: OpenAI DevDay 2024 live blog
#4Any word on increased weekly caps on o1 usage?
Re: OpenAI DevDay 2024 live blog
#5> The first big announcement: a realtime API, providing the ability to use WebSockets to implement voice input and output against their models.
I guess this is using their "old" turn-based voice system?
Re: OpenAI DevDay 2024 live blog
#6> The Realtime API improves this by streaming audio inputs and outputs directly, enabling more natural conversational experiences. It can also handle interruptions automatically, much like Advanced Voice Mode in ChatGPT.
> Under the hood, the Realtime API lets you create a persistent WebSocket connection to exchange messages with GPT-4o. The API supports function calling(opens in a new window), which makes it possible for voice assistants to respond to user requests by triggering actions or pulling in new context.
-
This sounds really interesting, and I see a great use cases for it. However, I'm wondering if the API provides a text transcription of both the input and output so that I can store the data directly in a database without needing to transcribe the audio separately.
-
Edit: Apparently it does.
It sends `conversation.item.input_audio_transcription.completed` [0] events when the input transcription is done (I guess a couple of them in real-time)
and `response.done` [1] with the response text.
[0] https://platform.openai.com/docs/api-reference/realtime-serv...
[1] https://platform.openai.com/docs/api-reference/realtime-serv...
Re: OpenAI DevDay 2024 live blog
#7> The first big announcement: a realtime API, providing the ability to use WebSockets to implement voice input and output against their models. I guess this is using their "old" turn-based voice system?
No, it's the same thing as ChatGPT advanced voice. Full speech-to-speech model.
Re: OpenAI DevDay 2024 live blog
#8Blog updates: - Introducing the Realtime API: https://openai.com/index/introducing-the-realtime-api/ - Introducing vision to the fine-tuning API: https://openai.com/index/introducing-vision-to-the-fine-tuni... - Prompt Caching in the API: https://openai.com/index/api-prompt-caching/ - Model Distillation in the API: https://openai.com/index/api-model-distillation/ Docs updates: - Realtime API: https://platform.openai.…
> Advanced Voice is rolling out globally to ChatGPT Enterprise, Edu, and Team users. Free users will get a sneak peak of it.
So regular paying users from EU are still left out in the cold.
Re: OpenAI DevDay 2024 live blog
#9Blog updates: - Introducing the Realtime API: https://openai.com/index/introducing-the-realtime-api/ - Introducing vision to the fine-tuning API: https://openai.com/index/introducing-vision-to-the-fine-tuni... - Prompt Caching in the API: https://openai.com/index/api-prompt-caching/ - Model Distillation in the API: https://openai.com/index/api-model-distillation/ Docs updates: - Realtime API: https://platform.openai.…
> Advanced Voice is rolling out globally to ChatGPT Enterprise, Edu, and Team users. Free users will get a sneak peak of it. So regular paying users from EU are still left out in the cold.
It's probably stuck in legal limbo in the EU. The recently passed EU AI Act prohibits "AI systems aiming to identify or infer emotions", and Advanced Voice does definitely infer the user's emotions.
(There is an exemption for "AI systems placed on the market strictly for medical or safety reasons, such as systems intended for therapeutical use", but Advanced Voice probably doesn't benefit from that exemption.)
Re: OpenAI DevDay 2024 live blog
#10The eval platform is a game changer.
It's nice to have have a solution from OpenAI given how much they use a variant of this internally. I've tried like 5 YC startups and I don't think anyone's really solved this.
There's the very real risk of vendor lock-in but quickly scanning the docs seems like it's a pretty portable implementation.