Synthetic Prototyping in Mixed Reality Environments for Human-Robot Teams

The MxR Lab, in collaboration with the ACT Lab (Automatic Coordination of Teams), is leveraging mixed reality to reduce the gap between simulation and implementation by enabling the prototyping of cooperative swarming algorithms on combinations of physically real and synthetic objects, including people, drones, and obstacles. Open source Bitcraze Crazyflie micro quadcopters serve as stand-ins for fully instrumented unmanned aerial platforms as they fly inside a PhaseSpace optical tracking volume. ROS (Robot Operating System) and Unity 3D are used for control, physics simulation, and rendering to a wireless VR headset. This test bed can aid in the design of multi-modal user interfaces for human-swarm interaction, development of robotic assistants for squad-level training, performance capture and after-action review. The early synthetic prototype approach can ease the introduction of unmanned swarms embedded with small-unit ground personnel.