Paper summary: Qwen2.5-Omni Technical Report

Problem

One model that can see, hear, read and speak in real time needs streaming inputs, synchronised audio and video, and generation of text and speech at once without the two interfering.

Method

• Block-wise audio and visual encoders, for streaming input.
• TMRoPE (Time-aligned Multimodal RoPE): audio and video are interleaved in time order with a new position embedding, so their timestamps line up.
• Thinker–Talker: the Thinker is an LLM that generates text. The Talker is a dual-track autoregressive model that reads the Thinker’s hidden states and produces audio tokens. Both are trained and run end to end.
• A sliding-window DiT decodes audio tokens in a streaming way to reduce the delay before audio starts.

Key results

From the abstract:

• comparable to similarly sized Qwen2.5-VL, and better than Qwen2-Audio
• state of the art on Omni-Bench
• speech-instruction following comparable to text input on MMLU and GSM8K
• the streaming Talker beats most streaming and non-streaming alternatives on robustness and naturalness

Limitations

There’s no dedicated limitations section in the extracted text. Results are the developers’ own.