Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks

Abstract: We present Omni-RGPT, a multimodal large language model designed to facilitate region-level comprehension for both images and videos. To achieve consistent region representation across spatio-temporal dimensions, we introduce Token Mark, a set of discretized tokens highlighting the target regions within the visual feature space. These tokens are directly embedded into spatial regions using region prompts (e.g., boxes or masks) and simultaneously incorporated into the text prompt to specify the target, establishing a direct connection between visual and text tokens. To further sup- port robust video understanding without requiring tracklets, we introduce an auxiliary task that guides Token Mark by leveraging the consistency of the tokens, enabling stable region interpretation across the video. Additionally, we introduce a large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT achieves state-of-the-art results on image and video-based commonsense reasoning benchmarks while showing strong performance in captioning and referring expression comprehension tasks.

**Figure 2: Architecture Overview.** Omni-RGPT enables region-level understanding across image and video inputs. Given region prompts (e.g. boxes or masks) in a single image or the initial frame of a video, we assign Token Mark — a set of vectors serving as spatio-temporal region indicators — to the region. These vectors are embedded into the spatial region localized by the region prompt and directly injected into both visual and text prompts to indicate the target. We further introduce Temporal Region Guide Head to achieve robust region understanding in videos without relying on tracklet prompts. Building on Token Mark's consistent representation of target objects across frames, this auxiliary task classifies the target Token Mark for visual tokens in subsequent frames.

◀️ ▶️👆 Click to Navigate Qualitative Results

We introduce a large-scale, diverse, and fine-grained **Reg**ion-level Video Instruction Dataset (**RegVID-300k**). The dataset includes 98k unique videos, with 214k regions curated from 10 public video datasets and 294k region-level instruction samples.

Acknowledgements

We would like to thank Qiushan Guo for assistance in setting up the project. We also appreciate the NVIDIA VILA team for the efforts in developing a robust framework that greatly facilitated our research.

BibTeX

@article{OmniRGPT,
  title={Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks},
  author={Heo, Miran and Chen, Min-Hung and Huang, De-An and Liu, Sifei and Radhakrishnan, Subhashree and Kim, Seon Joo and Wang, Yu-Chiang Frank and Hachiuma, Ryo},
  journal={arXiv preprint arXiv:2501.08326},
  year={2025}
}