Vinci2: Providing Proactive Assistance
in Continuous Egocentric Videos

Abstract

Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries, or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. We address this from both the evaluation and modeling sides. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video, comprising over 3,000 service instances organized along 4 temporal memory horizons and 11 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to decide whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks.

Three Paradigms of Egocentric Assistance

Most assistants are reactive (answer only when asked) or semi-proactive (monitor for predefined, task-relevant events). We study a third paradigm: a proactive agent that autonomously decides when and how to intervene, reasoning over the user's history, habits, and goals.

Live Demo

EgoMemo running on a continuous egocentric stream: the live first-person feed (left) drives an evolving reasoning process (right), autonomously triggering proactive services such as Task State Guidance and Lab Safety Alert & Action Recognition as the scene unfolds.

Contributions

The EgoServe Benchmark

Built upon EgoLife, HoloAssist, and CaptainCook4D, EgoServe spans diverse daily activities and temporal scales, with proactive services organized into 4 temporal memory horizons and 11 service categories — from instant safety alerts to long-term habit coaching.

EgoMemo: A Memory-Augmented Agent

EgoMemo continuously processes incoming video, maintains structured long-term memory, and performs context-aware reasoning at each timestep — autonomously deciding when to provide helpful interventions. Both proactive and reactive modes share a unified, fully streaming architecture.

Experimental Results

EgoMemo establishes strong baselines on EgoServe and generalizes to five established egocentric benchmarks without architectural modification.

SA: Safety Alert · TU: Tool Use · NG: Next-Step Guidance · ER: Error Recovery · RR: Resource Reminder · MR: Memory Recall · TR: Task Reminder · HC: Habit Coaching · MC: Memory Link · PP: Personal Feedback · RO: Routine Optimization. EgoMemo nearly doubles GPT-4o's overall F1 (7.5 vs. 4.3), with the largest gains on long-term services where structured memory enables cross-session reasoning.

EgoMemo achieves the best results on 4 of 5 benchmarks, confirming that its streaming-first design generalizes across both streaming and offline settings without architectural modification.

BibTeX

@inproceedings{gong2026vinci2,
  title     = {Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos},
  author    = {Gong, Sitong and Yan, Tianyu and Kang, Caixin and Zheng, Bo and
               Ruan, Xiang and Lu, Huchuan and Zhang, Kaipeng and Sato, Yoichi and Huang, Yifei},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}