Hierarchical Planning for Vision-and-Language Navigation in Continuous Environments
1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)
This report presents the methods of the winning entry of the rxr-habitatcompetition in cvpr 2022. the competition addresses the problem of vision-and-language navigation in continuous environments (vln-ce), which requires an agent to follow step-by-step natural language instructions to reach a target.
We present a modular plan-and-control approach for the task.
Our model consists of three modules : the candidate waypoints predictor (cwp), the history enhanced planner and the tryout controller.
In each decision loop,
the candidate waypoints predictor first predicts a set of candidate waypoints based on depth observations from multiple views.
Then, a history-enhanced planner is adopted to select one of the candidate waypoints as the subgoal.
The planner additionally encodes historical memory to track the navigation progress, which is especially effective for long-horizon navigation.
Finally, we propose a non-parametric heuristic controller named tryout to execute low-level actions to reach the planned subgoal.
It is based on the trial-and-error mechanism which can help the agent to avoid obstacles and escape from getting stuck.
We further take several recent advances of vision-and-language navigation (vln) to improve the performance such as pretraining based on large-scale synthetic in-domain dataset, environment-level data augmentation and snapshot model ensemble.
Dong An, Zun Wang, Yangguang Li, Yi Wang, Yicong Hong, Yan Huang, Liang Wang, Jing Shao