MIT Researchers Use VR to Train Autonomous Drones

Perfecting autonomous flight is no mean feat. Even the simplest, most obvious obstacles can be a challenge to navigate. The result in computer science labs developing autonomous systems is often regular breaks where expensive drone components need to be put back together again.

Researchers at MIT have developed a new way of doing things that could make training drones a whole lot easier. Using a virtual-reality training platform, drones can be fooled into thinking they are seeing (and learning to navigate around) genuine obstacles despite flying in a wide open empty space.

Taking on FPV pilots

The drone hallucination system is called “Flight Goggles,” and is being used by MIT to train drones to fly in complex environments, without having to pick up the pieces after each flight. In particular, the focus is on speed. Performing tasks while dodging obstacles at speed is the objective, but the team likes the idea of taking on genuine FPV pilots to prove the technology.

“We think this is a game-changer in the development of drone technology, for drones that go fast,” says Sertac Karaman, associate professor of aeronautics and astronautics at MIT. “If anything, the system can make autonomous vehicles more responsive, faster, and more efficient.”

“In the next two or three years, we want to enter a drone racing competition with an autonomous drone, and beat the best human player,” Karaman says.

Building a VR training experience for drones isn’t just about saving time and money in putting wreckage back together. It’s also useful because, in order to train drones to fly quickly through challenging environments, a new training system is required to help the drones learn to process visual information at speed.

“The moment you want to do high-throughput computing and go fast, even the slightest changes you make to its environment will cause the drone to crash,” Karaman says. “You can’t learn in that environment. If you want to push boundaries on how fast you can go and compute, you need some sort of virtual-reality environment.”

Making a drone hallucinate with Flight Goggles

The Flight Goggles virtual reality training kit features a motion capture system, an image rendering program, and electronics that enable the team to quickly process images and transmit them to the drone.

Despite how complex the environment may seem to the drone, the actual test space is essentially just a gymnasium in MIT’s new drone-testing facility. Motion-capture cameras are lined along the walls that track the orientation of the drone as it makes its way around the empty space.

Using the image-rendering system, the MIT team can feed bespoke scenes int the drone as it’s flying.

“The drone will be flying in an empty room, but will be ‘hallucinating’ a completely different environment, and will learn in that environment,” Karaman explains.

Karaman believes that eventually, the system could train drones to work alongside humans, recognize them and predict their movements well enough to avoid them with regularity. He explains how the test facility could be split in two, with a human moving around and providing a dynamic obstacle that the drone has to avoid while flying through its own empty space.

“One day, when you’re really confident, you can do it in reality, and have a drone flying around a person as they’re running, in a safe way,” Karaman says. “There are a lot of mind-bending experiments you can do in this whole virtual reality thing. Over time, we will showcase all the things you can do.”