ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System

Cet article n'a pas de fiche

Statut : exclu au tri — filtre mots-clés : aucun terme du groupe « origine ». Seuls les articles retenus au tri et dont le PDF est accessible sont lus en entier.

Résumé des auteurs

Existing deep video models are limited by specific tasks, fixed input-output spaces, and poor generalization capabilities, making it difficult to deploy them in real-world scenarios. In this paper, we present our vision for multimodal and versatile video understanding and propose a prototype system, \system. Our system is built upon a tracklet-centric paradigm, which treats tracklets as the basic video unit and employs various Video Foundation Models (ViFMs) to annotate their properties e.g., appearance, motion, \etc. All the detected tracklets are stored in a database and interact with the user through a database manager. We have conducted extensive case studies on different types of in-the-wild videos, which demonstrates the effectiveness of our method in answering various video-related problems. Our project is available at https://www.wangjunke.info/ChatVideo/