Hierarchical Planning for Vision-and-Language Navigation in Continuous Environments
1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)
This report presents the methods of the winning entry of the rxr-habitatcompetition in cvpr 2022. the competition addresses the problem of vision-and-language navigation in continuous environments (vln-ce), which requires an agent to follow step-by-step natural language instructions to reach a target.We present a modular plan-and-control approach for the task.Our model consists of three modules : the candidate waypoints predictor (cwp), the history enhanced planner and the tryout controller.In each decision loop,
the candidate waypoints predictor first predicts a set of candidate waypoints based on depth observations from multiple views.Then, a history-enhanced planner is adopted to select one of the candidate waypoints as the subgoal.The planner additionally encodes historical memory to track the navigation progress, which is especially effective for long-horizon navigation.Finally, we propose a non-parametric heuristic controller named tryout to execute low-level actions to reach the planned subgoal.It is based on the trial-and-error mechanism which can help the agent to avoid obstacles and escape from getting stuck.We further take several recent advances of vision-and-language navigation (vln) to improve the performance such as pretraining based on large-scale synthetic in-domain dataset, environment-level data augmentation and snapshot model ensemble.
