ZHENG FANG, LU Zhengping, Chenbo ZENG, Xiaoyu DONG, Guanlin Li, Yujie Chen, Fufang Wen, Yin Yang

Memory-augmented video world models retrieve past images to regenerate previously observed scenes during inference. Retrieval scores and final image errors alone cannot explain why an image is selected or whether it improves reconstruction. We introduce Retrieval Intervention and Paired Analysis (RIPA) to examine these two questions. RIPA changes the shared reference depth map, holds the candidate pool and read budget fixed, and compares selected histories using frozen models.

Under a shared scoring reference, depth errors in an unselected image can change other candidates’ rankings and the final selection. In paired tests with frozen WorldMem, these selection changes alter the generated images and can either increase or decrease reconstruction error. Recent observations also change which history is useful. Across six real scenes, adding the same recent image to both history selections reverses their ranking in 5 of 24 reconstruction cases. Outperforming another history selection does not guarantee a benefit over using the recent image alone. In two cases, even the better selection has higher reconstruction error than using the recent image alone. Pixel-level analysis separates error changes in newly covered regions from those in regions covered by both outputs.

These findings show that memory evaluation must account for unselected candidates and the observations already available to the model. RIPA provides reusable controls for the shared reference, comparisons against recent observations alone, and separate measurements of coverage and reconstruction error.

Paper In "CVRP"｜Click Here To Get The Full Paper