Inside the Drone-Powered ‘Magic Box’ for Emergency Management

Project CLARKE uses ‘magic box’ to assess storm damage, find missing persons

By DRONELIFE Features Editor Jim Magill

A software package that uses artificial intelligence (AI) and machine learning to help quickly analyze drone-collected images to assess property damage might seem like the stuff of science fiction, but it’s currently being deployed by a team of scientists at Texas A&M University.

Project CLARKE (Computer Vision and Learning for Analysis of Roads and Key Edifices) has been used to assess hurricane damage in Pennsylvania and Florida as well as to help search for a woman who was reported missing on a hiking trail in Japan. The system can take images gathered by drones at the scene using standard survey mapping tools and within minutes put together orthomosaics, creating large neighborhood-level views while classifying the level of damage inflicted to buildings and roads.

It can also analyze images to spot flecks of color or other indications that could lead rescuers to locate lost hikers or disaster victims, a process that could take human analysts hours or days to complete.

Representatives of the public service agencies that have taken advantage of the system often refer to CLARKE as “the magic box,” said Dr. Robin Murphy, who leads the project. This nickname helped inspire the formal name of the system. It was dubbed CLARKE, partially in honor of visionary science fiction writer Arthur C. Clarke who famously said, “Any sufficiently advanced technology is indistinguishable from magic.”

Based on the images fed into the system, CLARKE can process that data, and produce the analysis results in several formats, Murphy said.

“You can get the orthomosaic with all the polygons and the colors on the polygons showing the damage,” she said. Another output comes in a spreadsheet format, which lists all the buildings and roads in the image, describing the level of damage suffered by each.

The different data output formats help the Texas A&M team fill the needs of the various public service agencies the team works with. “One says, ‘Just give me the spreadsheet, that’s what I need.’ Another says ‘Just give me the overlays,’” Murphy said.

As the U.S. approaches the most active weeks of the 2025 hurricane season, the CLARKE team has trained about 90 people across 53 emergency agencies in several states in the use of the damage assessment system. Last year, several agencies used the system to assess the damages wrought by Hurricanes Debby and Helene.

Debby made landfall on Florida’s Gulf Coast August 5, 2024, before traveling up the Eastern Seaboard states all the way to Quebec, Canada. Less than two months later, Helene, a deadly and powerful storm, hit in the same part of Florida before bringing devastating floods to much of the U.S. Southeast, including North Carolina, Tennessee and Virginia. Murphy said the CLARKE team worked closely with two state agencies impacted by those twin storms.

“We are part of Florida UAS 1, which coordinates drone flights and drone operations for the state of Florida. Also, we are with the Pennsylvania Emergency Management Agency,” she said. “Debbie wound up ravaging a lot of their rural areas with flash floods, with all that tropical storm weather.”

Murphy said the emergency response agencies typically use the CLARKE system to respond to a disaster in two phases. The first phase focuses on the immediate response. “The first 72 hours are just trying; the drones are out there and they’re looking at the data to try to figure out what’s going on so the tactical decision-makers can get the information,” she said.

The agencies forward the drone-captured data to the Project CLARKE team, which can provide damage assessment data almost in real time.

“Instead of having to do kind of a quick glance, they can actually see, in a matter of minutes. (For example) it’s telling them most of this neighborhood’s destroyed,” she said. “Or everything’s fine except for one corner over here. Oh, and by the way, all the roads are out, so you’re going to have to send people in from the north to get in to help with the emergency response.”

In the next phase, the system provides strategic decision-makers with the damage assessment they need to determine the extent of the destruction, where and how much long-term resources are needed for rebuilding and how to plan ways to mitigate the impacts of future disasters.

“They want those building counts and road damages documented, both to get their state and federal documentation for their reimbursements, but also to be planning ahead to see how much resources to get there and to get them there quicker,” she said.

Drone Disaster Response: Search and Rescue 

In addition to its work assessing damages in the aftermath of natural disasters, Project CLARKE has shown a great deal of potential for use as a tool in wilderness search and rescue efforts.

“I think the trick to remember for wilderness search and rescue is that you’re trying to find a needle in a haystack,” saidTom Manzini, a PhD candidate in computer science and engineering who helped develop the system. “So, drones go out, they fly the area, collect a ton of imagery, and then — at least until recently — humans have been the ones who have been sort of staring at that imagery after it gets collected.”

Through machine learning, the CLARKE system is able to streamline the tedious process of analyzing images one at a time, focusing on small pops of color that don’t fit into the background and that might help identify a piece of clothing belonging to a person.

“So, the computer can say, ‘Hey, there’s something you should look at in this photo. And then the human gets to make the ultimate decision about whether or not that thing the computer found is actually important,” he said.

Unlike with damage assessments, the use of Project CLARKE in wilderness search and rescue efforts has proven to be a work in progress. Manzini said there are a lot of similar programs on the market, which like Project CLARKE use AI tools to locate missing persons, particularly in cases involving people swept up in floodwater.

“It’s a real challenging problem and one that people have been looking at for well over a decade,” he said. “It’s difficult, particularly if you think about a flooding situation where everybody’s been washed down, and is dirty, muddy and not necessarily wearing bright colors.”

System still in beta mode

The CLARKE system was made possible by a grant funding from the National Science Foundation, AI Institute for Societal Decision Making

Manzini said that although the Texas A&M team has shared the research behind Project CLARKE, the system is not yet ready to be developed commercially. “We’re publishing academically to make sure that other researchers are looking at our system,” he said.

“On the operation side, this is still a system that’s in early beta,” he said. ”If there’s going to be a deployment, that’s the sort of thing that we would want to have our hands in to make sure that the model’s behaving appropriately and making sure that the data that’s coming out of the system is something that’s actually useful for the needs of the responders.”

Both Murphy and Manzini had extensive experience in the use of drones in responding to disasters. Murphy, who has been working in the field of disaster response since 1995, led the first response using small unmanned aerial systems to Hurricane Katrina in 2005. “My basic research is in artificial intelligence for robotics. So, part and parcel, it’s been a delight to me to see the types of AI for computer vision become mature enough to actually be useful,” she said.

Manzini compared his experience is to “sort of sitting in the middle of three Venn diagrams.” His formal education is in computer science with a focus in machine learning. “But independent of that, I also have practical experience working as an EMT and as a firefighter for almost 12 years,” he said.

In addition, he has background in aviation with both a Part 107 drone pilot’s license and a Part 61 commercial pilot’s license. “I like to think I’ve got some background in the aviation side, the first response side and in the computer vision-academic side to bring to the table.”

Read more:

• Texas A&M’s RELLIS Campus Drives the Future of Autonomous Systems
• Drones in Disaster: Why Coordination Saves Lives
• AeroSense Launches Long-Range Rescue Drone for Disaster Response

Jim Magill is a Houston-based writer with almost a quarter-century of experience covering technical and economic developments in the oil and gas industry. After retiring in December 2019 as a senior editor with S&P Global Platts, Jim began writing about emerging technologies, such as artificial intelligence, robots and drones, and the ways in which they’re contributing to our society. In addition to DroneLife, Jim is a contributor to Forbes.com and his work has appeared in the Houston Chronicle, U.S. News & World Report, and Unmanned Systems, a publication of the Association for Unmanned Vehicle Systems International.