Georeferenced Full Motion Video: Mitigating a Difficult ‘Big Data’ Problem

The unmanned drone RQ-4 Global Hawk in flight in 2007 (Image credit: U.S. Air Force photo by Bobbi Zapka)

By Art Kalinski, GISP

For more than a decade, the military has been struggling with cataloging and retrieving its huge libraries of full motion video (FMV). The video, captured by both manned and unmanned aircraft, rapidly reached unmanageable levels. If you have ever tried to organize vacation photos after returning home from a long trip, you know that it’s easy to lose track of where each photo was taken. Date/time stamps help, but the effort is still difficult if your vacation took you to numerous locations.

The manned U.S. Air Force Beech King Air 350 and 350ER MC-12W Project Liberty Aircraft are designed for intelligence, surveillance and reconnaissance (ISR).

Now imagine trying to retrieve several critical minutes out of thousands of hours of video of barren land or repetitive-looking villages, and you get a sense of the magnitude of the problem. Without some way to catalog the video, critical details can be lost, because finding the right video clip becomes impossible in a reasonable period of time. Everyone agreed that the solution to the problem is to index the video clips by date/time and location, preferably with an exact georeferenced footprint. This is now possible with tools from Esri, Hexagon Geospatial and others.

Several years ago, at the USGIF GEOINT Tech Days, Sarnoff (SRI) demonstrated a system that pinned aerial video to its geographic footprint and maintained that registration despite the movement of the aircraft. It was an achievement that impressed everyone in the audience. I changed jobs soon after that conference and lost track of developments in the FMV field.  At the February Esri Federal Users Conference, I was thrilled to see Jack Dangermond briefly demonstrate the same kind of FMV georeferencing capability in ArcGIS 10.2 during his opening plenary session. I learned that Esri developed the capability in 2013, and later learned that Intergraph (Hexagon Geospatial) also developed a similar capability in 2010.

MISB: The Critical Improvement

The key technical development that made this possible was NGA’s creation of “Motion Imagery Standards” and the Motion Imagery Standards Board (MISB). The MISB developed standards for a consistent way to capture and record telemetry data during the video capture as metadata that becomes part of the video stream. This “Open Standard” metadata includes information such as the accurate xyz location of the aircraft, attitude, tilt, camera angle, and camera characteristics. This information travels with the individual video frames and permits the GIS/viewing software to perform the georeferencing.

This process is very similar to the oblique imagery capture system used for years by Pictometry, which at 20 FPS was technically FMV. MISB like Pictometry requires accurate GPS and IMU data to continuously capture and record the metadata. The MISB also gets involved with video compression standards such as the newer H.264 used on Blu-Ray discs and streaming video. H.264 has, for the most part, replaced MPEG2 and older MPEG4 as the video compression standard of choice. Much of the video captured by low-end small UAVs is just a simple video stream with no MISB telemetry data. However, I’m sure that lower prices, increased capability and smaller size of sensors will fix that with time. Sorry, no one has yet figured a way to “hack in” the metadata for legacy video captured without the MISB telemetry data. The one exception is those videos that contain usable telemetry data that was burned into the video and can be read with OCR. It might be possible to insert that information as MISB-compliant data.

The ArcGIS Full Motion Video Add-In

The ArcGIS Full Motion Video 1.2 Add-In (for ArcGIS 10.1, 10.2 and 10.3) is a free tool for ArcGIS users. It permits users to play georeferenced live or previously recorded video files in the map view.  The screen capture below shows several features of this tool and is from an online video.

The re-sizeable smaller window displays the video as it runs. The map view shows a changing footprint (green trapezoid) of the video as the aircraft flies over the site.  The short green line shows the flight path of the aircraft.  Demonstrating the interconnectivity of the two data sets, the user in the demo video drew a light blue polygon on the map view.  Note that the Esri Intersect function re-projects and displays the same polygon correctly in the video view.

Frames from the video can be extracted as single georeferenced images or groups of images and stored as a mosaic dataset. Playback of time-stamped video data can be synchronized with other time-enabled data and played together on the map. Features can be digitized directly on the video player and will appear on the map and vice versa.

The extension supports playback and management of multiple simultaneous video feeds. The Add-In also allows you to record the sensor, frame center, and footprint data in a geodatabase so the Bookmark Manager can perform searches for bookmarked video scenes. For more information regarding the Esri FMV tool, visit this ArcGiS site.

Hexagon Geospatial GeoMedia Motion Video Analyst Professional

Another robust FMV system that takes advantage of MISB telemetry is GeoMedia Motion Video Analyst Professional (MVA) from Hexagon Geospatial. The Hexagon Geospatial system includes tools to catalog videos as geospatial features with attributes extracted from the metadata, and has some elegant graphic selection tools that help an operator search and retrieve needed video clips.

In the screen capture below, you can see the map view with the geo-registered color video overlaid on the black-and-white ortho base image. As the video plays, the georeferenced video footprint continuously moves to the correct location as the tracking graphics in red show the position of the aircraft. The transparency of the video can be adjusted so an operator can compare features between the base image and the video and digitize directly in the map window.

MVA also includes a full set of tools for placing clipmarks as geospatial features with attributes and linked to the cataloged video, extracting snapshots and videos clips, on-the-fly enhancements, stabilization, registration and more. The system also facilitates rapid report generation so as an operator searches and plays appropriate video clips, the same operator or a partner can rapidly generate reports as documents or PowerPoint presentations in minutes. Another feature of the system is a “de-hazing” tool that removes a surprising amount of haze or smoke.

See a very good video overview of GeoMedia Motion Video Analyst Professional on this Hexagon Geospatial page. Like the Esri video, both are far better at explaining the capabilities than I can in this short column.

Other defense contractors are taking advantage of the MISB metadata, so check with your provider. Although these systems found their first home with military analysts, the Esri and Hexagon Geospatial reps indicated that many other users are finding the capability valuable in their work. Emergency operations centers come to mind first, but more mundane uses include rail and utility property management, the news media and video used in court proceedings. So, if you shoot lots of aerial video and need to catalog and retrieve video clips quickly, consider using MISB in your capture process.