Our understanding of the world started with a research that aimed to improve the indoor navigation for first-person navigators by fusing IMU data collected from their smartphone with the vision information concurrently obtained through the phone’s camera. We used the concept of vanishing directions in an EM framework to detect the person’s relative orientation with respect to the scene coordinates (perceived by the walls) from straight edge-lines in video frames. We incorporated the orthogonality constraint of the man-made environments into this approach to obtain a more robust estimation of the orientation. We also proposed an alternative approach that directly solves the objective function of the EM algorithm for orientation angles and therefore gives full control over orientation angles. We demonstrated that this approach gives more accurate estimations when we have some prior knowledge about the camera rotation axis. We synchronously combined the orientation angle obtained from video with the angles measured from IMU unit using Kalman filter to remove gyroscope drift in order to obtain better orientation estimation and enhance navigation. Finally, we used detected baselines and depth lines in the image sequence to extract floor planes in order to infer depth and width information of the path taken. We evaluated the performance of our video-based algorithms in relative orientation estimation on videos recorded from various indoor scenes. The results showed the capability and robustness of our approach in reliable orientation estimation from video frames in all scenes. Further, we applied our vision-inertial data fusion framework on an IMU-augmented video recorded from a rotary hallway. We showed that this fusion provided nearly drift-free estimation of orientation. In addition, we generated a 2D map of the rotary hallway using hallways’ width estimated from video and displacement measurements from accelerometer data.
Selected Publication:
“First-Person Indoor Navigation via Vision-Inertial Data Fusion,” IEEE/ION PLANS Conference, 2018.