Panoptic Lifting by Additional Viewpoint Selection

We propose an incremental learning method for Panoptic Lifting, a novel view synthesis technique that leverages both RGB images and panoptic segmentation masks to represent a 3D scene. While Panoptic Lifting is valuable in applications such as VR, autonomous vehicles, and robotics, it faces a challenge when observations are limited (e.g., due to occlusions), as retraining from scratch becomes computationally expensive. Our approach incrementally updates the model to achieve high accuracy with reduced computational cost. To prevent catastrophic forgetting, a phenomenon where previous learned knowledge is lost during model updates, We introduce a viewpoint selection algorithm that solves a maximum coverage problem to identify a subset of viewpoints that maximizes the visibility of the scene. Experimental results demonstrate a 12\% reduction in computation time compared to the na\"ive approach, while effectively preventing catastrophic forgetting.