ABSTRACT: Robust 6D object pose estimation is essential for enabling intelligent systems to interact with the physical world. While existing methods have achieved notable progress, they remain constrained by two major limitations: poor scalability when new objects are introduced and severe performance degradation caused by catastrophic forgetting in sequential learning settings. To address these challenges, we propose a novel prompt-based continual learning framework for 6D pose estimation. Our approach leverages frozen foundation features for robust appearance and geometric representation, while introducing a Multi-Head pose predictor to isolate task-specific regressors and a Prompt pool that serves as a rehearsal-free knowledge repository. The proposed framework enables task-agnostic inference by retrieving relevant prompts and activating the appropriate predictor without requiring explicit task labels. We conduct comprehensive experiments on the Linemod, YCB-Video, and Occlusion Linemod datasets under various settings within continual learning. Results show that our method achieves competitive performance, while exhibiting strong robustness to task order, adaptability to arbitrary object groupings, and high scalability through lightweight predictor expansion. To the best of our knowledge, this work represents the first systematic study of continual learning for 6D pose estimation, providing a foundation toward scalable and adaptive open-world perception.
Framework
Framework. (a) During training, cropped image and masked point cloud from Task i are fed into vision and geometry foundation models to extract features, which are fused into pixel-level representations and passed to a Multi-Head Pose Predictor for 6D pose regression. In parallel, the RGB image is used for task-identity inference: a subset of prompts is selected from a prompt pool via cosine similarity, combined with patch embeddings from a pre-trained encoder, and classified to produce the task ID, which determines the corresponding pose predictor. (2) During inference, the task-identity classifier takes the RGB image to predict the task ID, while the RGB-D features are extracted and routed to the selected pose predictor to estimate the 6D pose.
Pseudo Code
Quantitative Results on 6Dof pose estimation continual learning
Performance comparison on the Split-Linemod, Split-YCB-Video, and Split-LM-O datasets, respectively. Methods with marked † represent results obtained under the task-agnostic setting where the task identity is unknown during inference, and other results are obtained under the task-incremental setting. Objects with bold name are symmetric, and values with bold name indicate the best performance except for the joint. As a result, our method achieve a state-of-the-art compared with existing representive approach. These experimental results verify the effectiveness of our approach. Further details are provided in the paper.