Satellites, drones, and land vehicles (autos, trucks, tractors, etc.) will generate massive amounts of spatial data over the coming decade. In his keynote address at the AUVSI Xponential trade show this May, Brian Krzanich, CEO of Intel, said: “Data is the new oil.” A single autonomous car can generate the same amount of data as 3000 people surfing the Internet, while a small drone fleet could easily collect 150 terabytes of data per day. The data rate is going to explode on us in the next few years.
Esri and Microsoft have teamed up to build new tools to process and analyze these huge data sets. The following is an interview I conducted with Lawrie Jordan, Director of Imagery and Remote Sensing for Esri, and Dr. Lucas Joppa, Microsoft’s Chief Environmental Scientist. Their bios are listed after this interview.
McNeil: Can you give us an overview of what you are working on and how the GIS community will benefit from the new technology you are developing?
Joppa: To better protect and manage our natural world – including water and other ecosystems – we first need to get a clearer picture of the current state. And then we need to quickly and easily understand how our lands are changing due to factors like urbanization and climate change. In partnership with Esri, the Chesapeake Conservancy in Maryland, and their partners in the Chesapeake Bay program, we set out to produce a land cover mapping system capable of analyzing and classifying terrain at a one-meter resolution across the United States. That’s a huge increase from the 30-meter resolution data that was previously available and a significant advantage to organizations working to map and monitor local land and water resources.
A nice early example of this work and its impact is the success the Chesapeake Conservancy has had in combining Esri’s GIS technology with Microsoft’s cloud solutions to produce the first high-resolution land cover map of the Chesapeake watershed.
More recently, we have applied new developments in artificial intelligence, specifically an approach called “deep learning” or “deep neural networks.” These are network structured algorithms that provide astonishingly accurate insights and future predictions when there’s enough data available to learn from. In this case, using the Azure GPU-powered virtual machine and a Microsoft technology called the “Cognitive Toolkit,” we were able to train deep learning algorithms fairly easily and produce a single land classification model that enabled us to not only classify land cover in the Chesapeake Bay area, but also in other places like Oakland, Michigan.
We then integrated that with Esri’s ArcGIS Pro through their raster function, creating an even more powerful mapping visualization. We showed this real-time capability recently at the Esri User Conference.
In my mind, this partnership represents a powerful technology integration point between Microsoft’s algorithms and the computational power of Azure, Esri’s world-class GIS software, and data expertise of organizations like the Chesapeake Conservancy. We’re excited to move this project beyond the Chesapeake and build a foundation for land management all over the United States. We’re not there yet, but we’re making it happen.
Jordan: As Lucas mentioned, the data used to train the model was the result of a project done by the Chesapeake Conservancy. They had used supervised classification with NAIP (National Agriculture Imagery Program) imagery and ArcGIS’ machine learning running on the Azure cloud to create an initial high resolution land classification map of the Chesapeake watershed. They had also manually refined the results where necessary. This was a perfect input to train the CNTK (Cognitive Toolkit) which could determine inferences between the imagery and the required output. ArcGIS Image Server was used to provide fast access to the NAIP imagery. This enables any of the 120TB of NAIP multispectral imagery to be instantly accessible to the applications. ArcGIS Pro, which is the desktop component of the Esri platform, was used for image and data management functions, and also as the interface to submit the imagery to the algorithms.
ArcGIS further complements such developments by providing access to massive datasets that can be used to train the algorithms and then apply them as new images become available. This technology can be used to dynamically analyze incoming data collected by drones, satellites, and ground-based mobile devices. Plus, Esri has several million knowledgeable GIS users who have their own datasets, and they will be able to take advantage of this technology.
McNeil: In addition to creating high-resolution land cover maps, what other applications will the technology focus on?
Jordan: Search and rescue as well as disaster response are other good examples of where this technology has value. Let’s say I’m an emergency responder and I am concerned about a disaster that just happened. I want to have a dataset already loaded on a drone that shows the pre-disaster terrain and buildings. Next, I want to fly the drone over the impacted area and dynamically observe what has changed, and then send down just the information about what has changed. So I want to know: Did the bridge collapse or is the hospital still intact? I need that information right away as a first responder. Therefore, it is important to have the pre-event information so this data can be compared, in near real time. It is hard to know the impact of the disaster unless you know what the area looked like just before the event.
Joppa: We are also working with the University of Southern California’s AI and Society Institute to analyze drone-collected data to help prevent animal poaching in Africa, and as part of our newly launched AI for Earth initiative, Microsoft has additional projects focused on improving environmental surveillance, precision agriculture, and biodiversity monitoring.
McNeil: Are you to the stage that this technology can be productized?
Joppa: We’re not at the stage where we can make this solution widely available yet, but we are rapidly moving in that direction. Ultimately, we are focused on getting this powerful tool into the hands of as many groups as possible, so they can increase their impact. Microsoft’s sustainability mission is to empower people and organizations everywhere to thrive in a resource-constrained world — increasing access to tools like this is a critical part of advancing that mission.
Jordan: From a scalability standpoint, some of the barriers in the past that prevented us from doing some of these things are rapidly disappearing. Now we can handle tremendous amounts of data in the Microsoft cloud. These data include thermal, infrared, multispectral, day, night, low light level, and even 3D images collected from full-spectrum sensors mounted on various mobile devices. On the Esri side, the ArcGIS software platform is designed to scale up to virtually an unlimited size dataset or number of images. All the big data handling/processing and throughput issues that used to be a barrier to us have pretty much gone away. We are now in a position to manage and process these images in near real time. So that is very exciting.
Joppa: That’s a very important point. As all the sensors that Lawrie mentioned become increasingly attached to devices like drones, it used to be that we worried about how to transmit and store all of that data in real time. Now though, we have the capacity to collect and process large amounts of data almost instantaneously. This means we don’t have to think about storing all of the raw sensor data, we can just store the bits of information that are important to our job.
McNeil: Is there any way users can access the technology before it is commercially available?
Joppa: Yes, Microsoft Research accepts applications for development grants that utilize Microsoft’s AI solutions and Esri’s GIS technology. At present we are only supporting universities and non-profit research labs, but proposals need not be limited to land cover uses. They can take any shape or form, and are of particular interest if they align with the focus areas of our AI for Earth initiative, which include agriculture, water, biodiversity, and climate change. You can learn more at https://www.microsoft.com/en-us/aiforearth.
McNeil: Is there anything else you would like to share with our readers?
Joppa: I’m very excited by this technology and Microsoft’s partnership with Esri. By integrating the traditional GIS workflow with artificial intelligence and bringing that capability to more organizations, the smarter the entire system becomes and the better we’ll be able to address some of the most critical conservation, land use, and resource management challenges of our time.
Jordan: Looking a little further out, one of the big promises of this is the explosion of data being generated by devices like satellites, micro satellites, drones, ships, and automotive and truck equipment, not to mention underground and sub-sea sensors. We’re moving in the direction of being able to map, measure, and monitor everything on (or below) the surface of the earth that moves or changes every single day at high resolution. I think with the technology that Lucas described and the various sources of dynamic data now becoming available, we can actually bring the planet to life — what I have called for a long time the living planet. We can see the surface of the earth come to life and change every day. Continually refreshed high-resolution imagery and AI technology will be able to take us there in terms of getting meaningful analytics and understanding new patterns from the earth. Now I think this is going to change everything in the way we view models of climate, forests, oceans, and human movement. We’ll see new patterns emerge or appear that we’ve never seen before. And it will be in 3D. I think AI and Microsoft’s leadership in this field coupled with the Azure cloud technology leveraging a GIS framework will change the future of geography.