#multimodal

29 Oct 2026 · AI updates
Mistral Large 4 ("Le Chonk"): ~1T-parameter MoE with sparse attention
Mistral Large 4 is a multimodal MoE of about 1T total parameters with a 1M-token context, combining expert sparsity with sparse attention and a Pixtral vision encoder, released in FP8 and NVFP4 under an MIT-derived open-weight licence.
28 Oct 2026 · Paper summary
Paper summary: ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence
Training a spatial VLM on one kind of answer (e.g. left/right) doesn't teach related ones (e.g. distance), which the authors call "interface lock-in"; ChronoReplay replays mixed spatial supervision as time-ordered episodes and improves three streaming benchmarks together.
16 Oct 2026 · Paper summary
Paper summary: OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
OmniCompass is a manually annotated benchmark for video grounding and understanding, with a vision track and an audio-visual track (when, where and what); across 50 MLLMs, the best scores are only around 40%.
15 Jul 2026 · AI updates
Inkling: a ~1T-parameter open multimodal model with 1M context
Thinking Machines released Inkling on Hugging Face, a ~1T-parameter decoder-only multimodal MoE (41B active) that takes image, text and audio, has a 1M context, and was trained on 45T tokens; a smaller Inkling-Small followed.
9 Jul 2026 · Study
Generative AI · Multimodal models (CLIP)
CLIP learns a shared space for images and text by contrastive training on image–caption pairs; zero-shot classification of photos by comparing them with text prompts.
1 Jul 2026 · Paper summary
Paper summary: Orca: The World is in Your Mind
Orca learns one latent "world state" space from 125K hours of video and 160M event annotations via next-state prediction, and frozen, it supports text, image and robot-action readouts through small decoders.
16 Jun 2026 · Paper summary
Paper summary: JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
JoyAI-VL-Interaction is an open 8B vision-language model that watches a live video stream and decides each second whether to stay silent, respond, or hand off to a bigger background model.
3 Jun 2026 · AI updates
Gemma 4 12B: an encoder-free multimodal model for laptops
Google introduced Gemma 4 12B, a mid-sized open model where vision and audio inputs flow straight into the LLM backbone with no separate encoders; it runs in 16GB and ships with multi-token-prediction drafters.
2 Apr 2026 · AI updates
Gemma 4: open multimodal models under Apache 2.0
Google DeepMind's Gemma 4 family is Apache 2.0, accepts images or video plus text (and audio for the small models), and adds Per-Layer Embeddings and a shared KV cache for efficiency.
27 Mar 2026 · Paper summary
Paper summary: Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale
Intern-S1-Pro is described as the first one-trillion-parameter scientific multimodal foundation model, covering over 100 specialised science tasks while staying top-tier among open models on general ability.
13 Jan 2026 · Paper summary
Paper summary: BabyVision: Visual Reasoning Beyond Language
BabyVision tests core visual abilities that don't depend on language, which young children handle easily; the best model tested (Gemini3-Pro-Preview) scores 49.7, behind 6-year-olds and far behind adults' 94.1.
19 Dec 2025 · Paper summary
Paper summary: Kling-Omni Technical Report
Kling-Omni is one end-to-end framework for video generation, editing and reasoning-based editing from mixed inputs (text, reference images, video context), built on a dedicated data system.
7 Nov 2025 · Paper summary
Paper summary: Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
"Thinking with Video" uses a video generation model (Sora-2) as a reasoner, drawing its solution as video frames; on a new benchmark it rivals strong VLMs on visual puzzles and reaches 92% on MATH.
23 Sept 2025 · Paper summary
Paper summary: Qwen3-Omni Technical Report
Qwen3-Omni is a single model across text, image, audio and video that's reported to match same-sized single-modal Qwen models, with a Thinker–Talker MoE design and 234 ms theoretical first-packet speech latency.
26 Aug 2025 · Paper summary
Paper summary: InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
InternVL3.5 adds Cascade RL (offline then online RL), a Visual Resolution Router and decoupled vision/language deployment, reporting up to +16% reasoning and 4.05× faster inference than InternVL3.
2 Jul 2025 · Paper summary
Paper summary: GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
GLM-4.1V-Thinking and GLM-4.5V use a reasoning-centric training framework with Reinforcement Learning with Curriculum Sampling (RLCS); GLM-4.5V is reported to lead open models of its size on nearly all of 42 benchmarks.
26 Jun 2025 · AI updates
Gemma 3n: on-device multimodal models with "effective" sizes
Gemma 3n arrived across open-source tools; its E2B and E4B models have 5B and 8B real parameters but need only 2B- and 4B-class memory, and accept image, audio, video and text.
13 May 2025 · Paper summary
Paper summary: Seed1.5-VL Technical Report
ByteDance's Seed1.5-VL pairs a 532M vision encoder with a 20B-active MoE LLM and reports state of the art on 38 of 60 public benchmarks, plus strong GUI-control and gameplay agent results.
15 Apr 2025 · Paper summary
Paper summary: InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
InternVL3 trains multimodal and language abilities together in one pre-training stage, rather than adapting a text-only LLM afterwards, and reports 72.2 on MMMU for its 78B model.
8 Apr 2025 · Paper summary
Paper summary: SmolVLM: Redefining small and efficient multimodal models
SmolVLM is a family of compact vision-language models designed for on-device inference; the 256M model uses under 1GB of GPU memory and is reported to beat the 300× larger Idefics-80B.
5 Apr 2025 · AI updates
Llama 4 Scout and Maverick: Meta moves to mixture-of-experts
Meta's Llama 4 introduces two natively multimodal MoE models with 17B active parameters each: Scout (~109B total, 16 experts) and Maverick (~400B total, 128 experts), trained on up to 40T tokens.
27 Mar 2025 · Paper summary
Paper summary: Qwen2.5-Omni Technical Report
Qwen2.5-Omni takes text, images, audio and video as input and streams back both text and speech, using a Thinker (LLM) and a Talker (speech generator) that share hidden states.
17 Mar 2025 · Paper summary
Paper summary: SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion
SmolDocling is a 256M-parameter vision-language model that converts whole document pages end to end into DocTags, a markup capturing content, structure and location of every element.
12 Mar 2025 · AI updates
Gemma 3: multimodal, 128K context, 140+ languages
Google's Gemma 3 comes in 1B, 4B, 12B and 27B sizes; the 4B and up accept images, support 128K context and 140+ languages, and the 27B instruct model is reported to beat Gemini 1.5 Pro.
20 Feb 2025 · Paper summary
Paper summary: Qwen2.5-VL Technical Report
Qwen2.5-VL adds precise object localisation, robust document parsing and hour-long video understanding through a natively dynamic-resolution ViT and absolute time encoding; the 72B model is reported to match GPT-4o and Claude 3.5 Sonnet.
3 Jan 2025 · Paper summary
Paper summary: 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining
The authors turn 22,000 class hours of instructional videos into an interleaved image–text "textbook" corpus for vision-language pre-training, with denser knowledge than web-crawled interleaved data.
9 Dec 2024 · Paper summary
Paper summary: Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
InternVL 2.5 keeps InternVL 2.0's architecture but improves training, testing and data, and studies scaling of vision encoders, LLMs, data and test-time settings; it's reported to rival GPT-4o and Claude-3.5-Sonnet.
5 Dec 2024 · Paper summary
Paper summary: PaliGemma 2: A Family of Versatile VLMs for Transfer
PaliGemma 2 pairs the SigLIP-So400m vision encoder with Gemma 2 language models from 2B to 27B at three resolutions, and studies what drives transfer across a broad set of fine-tuning tasks.
27 Nov 2024 · Paper summary
Paper summary: ShowUI: One Vision-Language-Action Model for GUI Visual Agent
ShowUI is a vision-language-action model for GUI agents that sees screenshots like a person, trims redundant screen tokens with a UI graph, and interleaves vision, language and actions.
22 Nov 2024 · Paper summary
Paper summary: Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
Mixed Preference Optimization and the automatically built MMPR dataset improve chain-of-thought reasoning in multimodal LLMs (InternVL2-8B and 76B) beyond what supervised fine-tuning alone gives.
17 Nov 2024 · Paper summary
Paper summary: LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
LLaVA-CoT trains a vision-language model to reason in explicit stages (summary, visual interpretation, logical reasoning, conclusion) and adds a stage-wise retracing search at test time.
30 Oct 2024 · Paper summary
Paper summary: CLEAR: Character Unlearning in Textual and Visual Modalities
CLEAR is an open benchmark for removing information about specific people from multimodal models, with 200 fictitious individuals, 3,700 images and linked Q&A, used to compare 11 unlearning methods.
30 Sept 2024 · Paper summary
Paper summary: Emu3: Next-Token Prediction is All You Need
Emu3 tokenises images, text and video into one discrete space and trains a single Transformer from scratch with only next-token prediction, reported to beat SDXL at generation and LLaVA-1.6 at perception.
26 Sept 2024 · Paper summary
Paper summary: Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo is a family of open vision-language models trained on PixMo, new datasets collected without using any external VLMs, including detailed captions, free-form Q&A and 2D pointing data.
25 Sept 2024 · AI updates
Llama 3.2: vision models and small on-device models
Meta's Llama 3.2 added vision models (11B and 90B) and small text-only models (1B and 3B) for on-device use, plus a vision-capable Llama Guard; ten open-weight models in total.
19 Aug 2024 · Paper summary
Paper summary: xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
BLIP-3 (xGen-MM) is Salesforce's open framework for large multimodal models (datasets, recipe, architecture and models), releasing 4B and 14B base and instruction-tuned models that handle interleaved image–text input.
10 Jul 2024 · Paper summary
Paper summary: Vision language models are blind: Failing to translate detailed visual features into words
BlindTest's seven simple visual tasks, such as whether two circles overlap or how many times two lines cross, trip up strong vision-language models: four state-of-the-art VLMs averaged 58.07%.
7 Jun 2024 · Paper summary
Paper summary: ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
ShareGPT4Video builds dense, precise video captions (40K with GPT-4V, then 4.8M with its own captioner) to improve both video understanding models and text-to-video generation.
28 May 2024 · Paper summary
Paper summary: An Introduction to Vision-Language Modeling
A tutorial-style introduction to vision-language models from a large group of authors: what VLMs are, the main training families, how to train and evaluate them, and the move to video.
17 May 2024 · Paper summary
Paper summary: Chameleon: Mixed-Modal Early-Fusion Foundation Models
Chameleon is one token-based model that reads and generates images and text in any order, using early fusion, plus a training recipe designed to keep this mixed-modal setup stable.
15 Mar 2024 · Paper summary
Paper summary: MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Apple's MM1 ablates how to build multimodal LLMs and finds the data mix and image encoder (resolution, token count) matter a lot, while the vision–language connector design matters little.
2 Jan 2024 · Paper summary
Paper summary: DocLLM: A layout-aware generative language model for multimodal document understanding
DocLLM adds layout awareness to an LLM using only text bounding boxes, with no image encoder, via disentangled text/layout attention and a text-infilling pre-training objective.
3 Dec 2023 · Paper summary
Paper summary: Merlin:Empowering Multimodal LLMs with Foresight Minds
Merlin trains a multimodal LLM to predict object trajectories and then reason about future events, using two stages called Foresight Pre-Training and Foresight Instruction-Tuning.
22 Aug 2023 · AI updates
IDEFICS: an open reproduction of Flamingo
Hugging Face released IDEFICS, an open visual language model in 9B and 80B sizes that reproduces DeepMind's unreleased Flamingo using only public data and models.
4 Aug 2023 · Paper summary
Paper summary: Learning to Model the World with Language
Dynalang treats language as a signal for predicting the future: an agent learns a multimodal world model and acts from imagined rollouts.