SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences

We create a globally consistent 3D scene graph by fusing predictions of a graph neural network (GNN) from an incremental geometric segmentation created from an RGB-D sequence. Our method merges nodes on the same object instance and naturally grows and improves over time when new segments and surfaces are discovered. As a by-product, our method produces accurate panoptic segmentation of large-scale 3D scans. The nodes represent the different object segments.

Video

Paper

@InProceedings{Wu_2021_CVPR,
    author    = {Wu, Shun-Cheng and Wald, Johanna and Tateno, Keisuke and Navab, Nassir and Tombari, Federico},
    title     = {SceneGraphFusion: Incremental 3D Scene Graph Prediction From RGB-D Sequences},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {7515-7525}
}

Code

We open-sourced our code into two repositories: framework and network. The framework repository consists of the incremental segmentation system as well as the scene graph fusion. The network repository consists of the data generation pipeline, training, testing and tracing codes.

Humanoid Lola

This work has been used as the visual perception system for Humanoid Lola. For more detail about Humanoid Lola please check this page