Learning 3D Dynamic Scenes from 2D Visual Observations
3D Neural Scene Representations for Visuomotor Control
We present a 3d-aware scene representation learning model that enables visuomotor control for challenging manipulation tasks involving both rigid bodies and fluids, where the target is specified in a viewpoint different from what the robot operates on.Our model combines time contrastive learning with an auto-decoding framework, which learns viewpoint-invariant 3d-aware scenerepresentations.We show that a dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks involving both rigid bodies and fluids, where the target is specified in a viewpoint different from what the robot operates on.When coupled with an auto-decoding framework, it can even support goal specification from cameraviewpoints that are outside the training distribution.We further demonstrate the richness of the learned 3d dynamics model by performing future predictionand novel view synthesis.Finally, we provide detailed ablation studiesregarding different system designs and qualitative analysis of the learned representations.
