Effective Remote Drone Control Using Augmented Virtuality

Augmented Virtuality, UAV, Drone Piloting, Virtual Scene, Navigation Elements, First Person View, Third Person View.

Since the remote drones control is mentally very demanding, supporting the pilot with both, first person view (FPV) and third person view (TPV) of the drone may help the pilot with orientation capability during the mission. Therefore, we present a system that is based on augmented virtuality technology, where real data from the drone are integrated into the virtual 3D environment model (video-stream, 3D structures, location information). In our system, the pilot is mostly piloting the drone using FPV, but can whenever switch to TPV in order to freely look around the situation of poor orientation. The proposed system also enables efficient mission planning, where the pilot can define 3D areas with different potential security risks or set navigation waypoints, which will be used during the mission to navigate in defined zones and visualize the overall situation in the virtual scene augmented by online real data.

The resulting application was developed using the Unity game engine. For creating the 3D virtual environment, the environment maps and heightmaps are loaded from external sources in order to create the terrain in Unity (maps, 3D building models, textures, heights). Next, flying informations and sensory data are transmited to the application from the drone and visualized (pose, height, video-stream, battery status, flight speed, flight mode).

The 3D virtual data are integrated from existing data sources and the coordination system is registered to actual real drone position. The GPS drone coordinates and orientation data is used for registration with virtual 3D frame. The augmentation of the virtual 3D scene is achieved by rendering the live video-stream from the drone front camera to projection plane in front of the virtual drone. 

The additional 3D navigation UI elements like mission points, direction to next point or virtual walls, are rendered into virtual scene and presented to the user also. The registration method might be improved by computer vision techniques, but this step will be considered later according to user tests, when the effect of rough GPS and compass based registration and caused video latency will be considered by professional pilots as an important issue.