OpenSet 3D Neural Scene Segmentation with Pixel-wise Features and Rendered Novel Views

ICLR 2024

1ETH Zurich 2Google 3Microsoft 4TU Munich

Open-vocabulary 3D semantic segmentation on point clouds. Compared to LERF, the segmentation masks of OpenNeRF are more accurate and better localized, while achieving more fine-grained classification than OpenScene. We show zero-shot results on the Replica dataset.


Large visual-language models (VLMs), like CLIP, enable open-set image segmentation to segment arbitrary concepts from an image in a zero-shot manner. This goes beyond the traditional closed-set assumption, i.e., where models can only segment classes from a pre-defined training set. More recently, first works on open-set segmentation in 3D scenes have appeared in the literature. These methods are heavily influenced by closed-set 3D convolutional approaches that process point clouds or polygon meshes. However, these 3D scene representations do not align well with the image-based nature of the visual-language models. Indeed, point cloud and 3D meshes typically have a lower resolution than images and the reconstructed 3D scene geometry might not project well to the underlying 2D image sequences used to compute pixel-aligned CLIP features. To address these challenges, we propose OpenNeRF which naturally operates on posed images and directly encodes the VLM features within the NeRF. This is similar in spirit to LERF, however our work shows that using pixel-wise VLM features (instead of global CLIP features) results in an overall less complex architecture without the need for additional DINO regularization. Our OpenNeRF further leverages NeRF's ability to render novel views and extract open-set VLM features from areas that are not well observed in the initial posed images. For 3D point cloud segmentation on Replica, OpenNeRF outperforms recent open-vocabulary methods such as LERF and OpenScene by at least +4.9 mIoU.

Zero-Shot 3D Semantic Segmentation

Replica - Office 0

Replica - Office 1

Replica - Office 2

Replica - Office 3

Replica - Room 0

Replica - Room 1

Replica - Room 2

Open-Vocabulary 3D Search


   title={{OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views}},
   author={Engelmann, Francis and Manhardt, Fabian and Niemeyer, Michael and Tateno, Keisuke and Pollefeys, Marc and Tombari, Federico},
   booktitle={International Conference on Learning Representations},