Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Omni: Native Omni AI model for text, image and video

github.com

Post reply on HN
Print view View original

11–20 of 152 posts

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#11

Post by adastra22 »

simonw wrote: post #8
adastra22 wrote: post #5

Earlier quoted context omitted.

Isn’t that how all LLMs work?

The existing vision LLMs all work like this, which is most of the major models these days. Multi-modal audio models are a lot less common. GPT-4o was meant to be able to do this natively from the start but they ended up shipping separate custom models based on it for their audio features. As far as I can tell GPT-5 doesn't have audio input/output at all - the OpenAI features for that still use GPT-4o-audio. I don't k…

What I mean is that all processing in an LLM occurs in state space. The next-token prediction is the very last step.

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#12

Post by a_e_k »

simonw wrote: post #6

The model weights are 70GB (Hugging Face recently added a file size indicator - see https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct/tree... ) so this one is reasonably accessible to run locally. I wonder if we'll see a macOS port soon - currently it very much needs an NVIDIA GPU as far as I can tell.

That's at BF16, so it should fit fairly well on 24GB GPUs after quantization to Q4, I'd think. (Much like the other 30B-A3B models in the family.)

I'm pretty happy about that - I was worried it'd be another 200B+.

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#13

Post by OJFord »

neilmovva wrote: post #4

The multilingual example in the launch graphic has Qwen3 producing the text: > "Bonjour, pourriez-vous me dire comment se rendreà la place Tian'anmen?" translation: "Hello, could you tell me how to get to Tiananmen Square?" a bold choice!

Not really, it's a significant place which is why the protest (and hence massacre) was there, so especially for Chinese people (I expect) merely referencing it doesn't so immediately refer to the massacre, they have plenty of other connotations for it.

e.g. if something similar happened in Trafalgar Square, I expect it would still be primarily a major square in London to me, not oh my god they must be referring to that awful event. (In fact I think it was targeted in the 7/7 bombings for example.)

Or a better example to go with your translation - you can refer to the Bastille without 'boldly' invoking the histoire of its storming in the French Revolution.

No doubt the US media has referred to the Capitol without boldness many times since 6 Jan '21.

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#14

Post by uniqueuid »

adastra22 wrote: post #11
simonw wrote: post #8

Earlier quoted context omitted.

The existing vision LLMs all work like this, which is most of the major models these days. Multi-modal audio models are a lot less common. GPT-4o was meant to be able to do this natively from the start but they ended up shipping separate custom models based on it for their audio features. As far as I can tell GPT-5 doesn't have audio input/output at all - the OpenAI features for that still use GPT-4o-audio. I don't k…

What I mean is that all processing in an LLM occurs in state space. The next-token prediction is the very last step.

There are many more weird and complex architectures in models for video understanding.

For example, beyond video->text->llm and video->embedding in llm, you can also have an llm controlling/guiding a separate video extractor.

See this paper for a pretty thorough overview.

Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Liu, P., Feng, M., Zheng, F., Zhang, J., Luo, P., Luo, J., & Xu, C. (2025). Video Understanding with Large Language Models: A Survey (No. arXiv:2312.17432). arXiv. https://doi.org/10.48550/arXiv.2312.17432

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#15

Post by adastra22 »

uniqueuid wrote: post #14
adastra22 wrote: post #11

Earlier quoted context omitted.

What I mean is that all processing in an LLM occurs in state space. The next-token prediction is the very last step.

There are many more weird and complex architectures in models for video understanding. For example, beyond video->text->llm and video->embedding in llm, you can also have an llm controlling/guiding a separate video extractor. See this paper for a pretty thorough overview. Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Liu, P., Feng, M.…

Sure but all of these find some way of mapping inputs (any medium) to state space concepts. That's the core of the transformer architecture.

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#16

Post by em500 »

OJFord wrote: post #13
neilmovva wrote: post #4

The multilingual example in the launch graphic has Qwen3 producing the text: > "Bonjour, pourriez-vous me dire comment se rendreà la place Tian'anmen?" translation: "Hello, could you tell me how to get to Tiananmen Square?" a bold choice!

Not really, it's a significant place which is why the protest (and hence massacre) was there, so especially for Chinese people (I expect) merely referencing it doesn't so immediately refer to the massacre, they have plenty of other connotations for it. e.g. if something similar happened in Trafalgar Square, I expect it would still be primarily a major square in London to me, not oh my god they must be referring to th…

Not to mention, Tiananmen Square is one of the major tourist destinations in Beijing (similar to National Mall in Washington DC), for both domestic and foreign visitors.

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#17

Post by nmitchko »

Next steps for AI in general:

  - additional modalities
  - Faster FPS (inferences per second)
  - Reaction time tuning (latency vs quality tradeoff) for visual and audio inputs/outputs
  - built-in planning modules in the architecture (think premotor frontal lobe)
  - time awareness during inference (towards an always inferring / always learning architecture)

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#18

Post by simonw »

You can try it out on https://chat.qwen.ai/ - sign in with Google or GitHub (signed out users can't use the voice mode) and then click on the voice icon.

It has an entertaining selection of different voices, including:

*Dylan* - A teenager who grew up in Beijing's hutongs

*Peter* - Tianjin crosstalk, professionally supporting others

*Cherry* - A sunny, positive, friendly, and natural young lady

*Ethan* - A sunny, warm, energetic, and vigorous boy

*Eric* - A Sichuan Chengdu man who stands out from the crowd

*Jada* - The fiery older sister from Shanghai

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#19

Post by hadlock »

Speech input + speech output is a big deal. In theory you can talk to it using voice, and it can respond in your language, or translate for someone else, without intermediary technologies. Right now you need wakeword, speech to text, and then text to speech, in addition to your core LLM. A couple can input speech, or output speech, but not both. It looks like they have at least 3 variants in the ~32b range.

Depending on the architecture this is something you could feasibly have in your house in a couple of years or in an expensive "ai toaster"

Re: Qwen3-Omni: Native Omni AI model for text, image and video

#20

Post by ludwigschubert »

adastra22 wrote: post #15
uniqueuid wrote: post #14

Earlier quoted context omitted.

There are many more weird and complex architectures in models for video understanding. For example, beyond video->text->llm and video->embedding in llm, you can also have an llm controlling/guiding a separate video extractor. See this paper for a pretty thorough overview. Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Liu, P., Feng, M.…

Sure but all of these find some way of mapping inputs (any medium) to state space concepts. That's the core of the transformer architecture.

The user you originally replied to specifically mentioned > without going to text first
Post reply on HN