Engineers blend multiple compute components in hybrid architectures to facilitate sensor and signal processing on ISR platforms.
"Threats facing warfighters today are more unpredictable than at any time in history," driving the need for data, says Larry Schaffer, director of business development at Abaco Systems in Huntsville, Ala.
"In World War I, the first 'technological' war, forces still fought face-to-face; huge groupings thrust against each other, reminiscent of all wars before," resulting in 11 million casualties, Schaffer explains. Then war entered a new age, fought less by force-on-force conflict, and warfighters needed to know more before they faced the opposition.
"Just as WWI changed our need for information, current stateless asymmetric threats change war- fighters' information needs," Schaffer says. "What is needed today is 'situational understanding' or a content-rich, real-time picture as well as historical and predictive information in many domains." Situational awareness refers to knowing what's going on in real time - "knowing where the good guys and bad guys are, what the terrain and potential mission threats are," he explains.
This need for more information is not unique to militaries, and is driving the demand for powerful persistent surveillance systems (PSS) throughout civil and defense organizations across the globe. The warfighter relies on persistent surveillance, but so too do facilities in the electrical and information infrastructure and where large groups of people assemble, such as stadiums, entertainment venues, transportation hubs, and schools, Schaffer says.
"Intelligence, surveillance, and reconnaissance (ISR) is hot and has been hot for some time, and the plunging cost of aerial platforms has brought ISR to the masses," Schaffer adds. "It is unbelievably widespread."
Surveillance sea change
Early wide-area motion imaging (WAMI) systems developed by the U.S. military have been large and expensive to procure, operate, and maintain, and they had a very large processing, exploitation, and dissemination (PED) analyst footprint, explains John Marion, president of Logos Technologies LLC in Fairfax, Va. As a result, many concluded that WAMI is difficult and expensive, but that has changed in recent years. Logos Technologies' Redkite, for example, "is a more capable system at 20 pounds than was Constant Hawk when it deployed at over 1,000 pounds," he says.
"The community is starting to appreciate the value of WAMI," Marion says. "There was a bit of a learning curve, but there is a widening awareness of what WAMI can add to force protection, event security, disaster relief, and asset protection. These assets could range from protecting a nuclear power plant to protecting rhinos from poachers."
Marion compares global WAMI adoption to that of full-motion video (FMV). "A decade ago, FMV was in limited use, but now it's everywhere. I expect the same thing to happen with WAMI. The U.S. military has been leading on that front, but there has also been a dramatic increase in international awareness and interest in WAMI." Logos has more than a dozen systems offered for sale or under active development, and the uses include disaster relief, border and critical infrastructure protection, force protection, law enforcement, and counter-poaching surveillance. Key characteristics customers seek are low system size, weight, and power (SWaP) and improved analyst tools to reduce the PED burden.
"A key driver for the proliferation of WAMI is reducing system SWaP," Marion says. "Take the 700-pound Gorgon Stare, for example. Not many unmanned aerial vehicles (UAVs) can carry it, but if you drop the weight of the WAMI system to 20 pounds including processing, as we have done with Redkite, then you get a sensor that can go on a large number of platforms, including Group 3 UAVs."
Marion anticipates greater proliferation in mission sets, as well. "The first WAMI systems, Constant Hawk and Angel Fire, were deployed to Iraq in 2006 for ISR. In 2011, Kestrel was deployed on aerostats to protect troops operating in and around forward operating bases (FOBs) in Afghanistan. In the past few years, we've also seen the testing of WAMI for border and event security; and right now, we see it being evaluated for disaster relief supporting humanitarian aid missions as well as to protect megafauna from poachers."
Persistent surveillance, wide-area motion imaging, and wide-area aerial surveillance (WAAS) systems are as varied as the missions they fulfill, if not more so. Some commonality among systems exists, however, given the growing demand for and use of open architectures, commercial off-the-shelf (COTS) rather than proprietary parts, and adherence to industry standards. Logos, for example, "integrates these systems from a mixture of custom and COTS parts," Marion describes. "We are always tracking commercial development in three key areas: cameras, focal planes, and processing components."
More compute power
Persistent surveillance isn't just capturing data anymore; it's now much more sophisticated and encompasses "analytics, data fusion, sensor fusion, predictive analysis, including gaze recognition, intent recognition, crowd interaction dynamics, and more," Abaco's Schaffer observes. "As we get more and more sensor modalities (UV, IR, lidar, radar, sonar, spectral, visible) adding to a-priori (geomorphology, past events, weather and climate patterns), the richness of this data added to the need for processing it quickly is driving demand for higher levels of processing power."
This compelling need has brought Abaco many customers seeking high-performance embedded computing (HPEC), according to Schaffer. The company provides the computer that fuses radar, lidar, visible, infrared, and synthetic (a-priori) data into one real-time visualization for helicopter pilot situational understanding - "one single picture that encompasses multiple data streams and sensor modalities. These HPEC systems must [also] be carried on space/power-constrained platforms, giving us a 'sizable' advantage, as the HPEC unit that does this is about the size of a six-pack of beer."
Big data and bandwidth
The past decade has brought not only huge strides in wireless data transmission, to which cell phone users can testify, but also massive arrays of ever-higher-resolution sensors operating in multiple modalities on remote platforms, Schaffer says. "This need for bandwidth has far out-paced the ability of down-links to cope, so we take lots of data and squirt it out over time and, therefore, sacrifice real-time understanding.
"Less than desirable for sure, but far more than the time it takes to transmit data is the problem of aggregating, correlating, and developing an understanding of all this data - which, of course, is useless without knowing what it means," Schaffer adds. "What is increasingly important is a better kind of data compression, where only results, not raw data, are sent. This is the work to be done on the platform. How, you ask? It will be done by 'learning machines' - high-performance computers that can process massive data sets based on experience, or processing based on an understanding of what is important in a multi-domain dataset."
Onboard processing is key, says Logos Technologies' Marion, because it enables users to exploit data in real time, as it is collected, with a data link whose capacity is three orders of magnitude lower than the rate at which data is collected.
Sensor systems
The focus on sensor systems usually involves specifics of the sensor itself, but data processing and analytics are just as important - especially in exploiting the WAMI data with other intelligence sources, Marion says. For the past seven years, Logos engineers have been taking advantage of developments in commercial embedded processing to produce "much more capable WAMI processing systems in ever smaller packages," he says. "For these systems, software development needs to be done in parallel with the hardware development so analytic functions can be performed at these data rates in real time but in SWaP-miniaturized hardware."
In the design of Logos customized processing boards, engineers use a mix of digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and general-purpose processors. The processing stack for the company's Redkite UAV system performs all the processing, yet the weighs only three pounds and draws 100 watts, Marion says. "We have a fully capable WAMI system - including sensor, embedded processing performing all analytics, and data storage - housed in a 20-pound package. Our infrared system for nighttime use is fairly light too: only 50 percent heavier than the daytime WAMI."
Along with the small size of the sensors, it follows that the processing elements must be small as well, Abaco's Schaffer says. "Small size and low mass work well to support something else that is needed: ruggedness. At Abaco, we have a fully rugged, 300 MFLOP computer the size of a credit card that takes less than 10 watts, so we know how to do rugged, small, and powerful."
Ruggedness and operational availability are extremely important, Marion says, because WAMI systems are often used close to 24/7. "For example, the operational availability of our Kestrel systems in Afghanistan has been at 94 percent with over 200,000 operation hours."
"The ability to operate for a long time is a given, but computers produced today will quickly be replaced as sensors demand even higher-performance processors," Schaffer adds. "In only the past few years, our computers have reduced weight by 75 percent and consume 90 percent less power per MFLOP. This is an unrelenting trend and we are on it daily."
Abaco's computing solutions are "widespread in the processing of video, radar, sonar, and other complex datasets related to PSS/WAAS," Schaffer says. "Go small or go home. Today, it is truly about low SWaP; higher performance and lower consumption in all domains; and doing more with less."
Analyst overload
Myriad surveillance systems are capturing more data now than at any other time in history. Analysts are overwhelmed, and technology firms are helping to lighten the load.
"For years, we've been performing real-time onboard processing to stabilize and geo-locate the imagery as well as provide multiple ground users their own pan and zoom view within the imagery. That was demonstrated for a U.S. government customer nearly a decade ago," says Marion. "More recently, we've added precision geo- registration, automated tracking of all the moving targets in the scene, and other tools that help dramatically reduce the burden on analysts.
"We are now providing tools to organize the WAMI data, along with data from other intelligence sources, in ways that very significantly streamline the analytical process, allowing single analysts to solve problems much faster, and allowing them to solve much more complex problems," Marion continues.
Overcoming past limitations
"Persistent surveillance systems are in high demand, and application areas for such technologies are proliferating wildly," says Mark Littlefield, vertical product manager, defense at Kontron in Poway, Calif. "Persistent surveillance isn't new; systems like this have been deployed by the U.S. as far back as the Vietnam War. However, until relatively recently, persistent surveillance systems have been custom-crafted to meet extremely low SWaP requirements. Extremely small and low-power COTS computing hardware, along with standardized methods to forming distributed networks, are making low-cost, COTS-based solutions possible."
In the past, significant digital signal processing or image processing was limited to fairly large platforms because extremely small, low-power platforms didn't have the computing horsepower needed for these applications, Littlefield continues. Times have changed with the availability of processors in the sub- or couple-of-watt range capable of performing real-time image processing or DSP.
"When these processors are linked to sensors and ad-hoc networking hardware based on standards, they form an ideal platform for PSS," Littlefield says. The actual deployment platform can vary widely; the common element is that they sense their environment and share data and information with neighboring sensor platforms. "Each platform has its own needs and challenges which drive system design."
Kontron leverages small/low-power computing platforms, the Internet of Things (IoT), and standards-based network communications software to provide COTS- based solutions for the computing and communications portions of the PSS application - leaving the sensors used and the "special sauce" of the applications codes for processing and analyzing the data collected by the sensors to customers, Littlefield says.
Kontron engineers marry standards-based COMe Mini and SMARC small-form-factor hardware components with software tools like MQTT to create an application platform for persistent surveillance. Littlefield advises engineers against spinning their "own hardware and software frameworks. There are COTS solutions out there; engage the market. Even if there is not a product that is 100 percent suitable for your application, there may be something that could be adjusted slightly to meet your specific needs. That's the power of COTS."
Popular processing trio
Processors today - multicore chips, GPUs, FPGAs, and a combination - are very efficient in performance per unit of power and capabilities, supporting the unmanned vehicle doing the processing in the sky, explains Marc Couture, senior product manager at Curtiss-Wright Controls Defense Solutions in Littleton, Mass.
"The Achilles heel is typically the common data link to the ground, not a lot of bandwidth," Couture says. "Instead of sending whole images from an MQ-9 Reaper type of unmanned aircraft, just sending the information - that capability exists now, and not just for the biggest UAVs like Global Hawk, but also for smaller [platforms], including the Shadow, because we're able to pack a lot of processing in a small footprint."
A magic trio of processing technologies is popular now, Couture describes. "Often FPGAs are closest to the electro-optic/infrared (EO/IR) sensor, typically seated right behind it. They are input/output (I/O) chameleons that adapt to sensor output and perform image compression; they are good at doing repetitive things in parallel quickly."
Further, a multicore Intel device is the general-purpose processor of choice and well suited to identifying threats on the ground. The third device, a general-purpose graphics processing unit (GPGPU), is very good at very compute-intensive tasks, Couture says. "They provide teraflops (floating-point operations per second) of computing power for stitching together images from multi-gigapixel cameras, matching and correlating data, orthogonal rectification, intensive algorithms, and so on. When the Intel processor gets overwhelmed, work is offloaded to GPUs.
"This heterogeneous trio enables big capabilities for WAAS/PED while keeping SWaP-C low. It can scale the processing for big UAVs or smaller ones. Now it's open to all different size birds," Couture adds.
The goal is to have multiple UAVs with very high-resolution cameras and processing covering the surface area on the Planet Earth, says Couture. "A platoon leader just wants to get troops over a dune. Now he can call up and say 'I need specific imagery for this area.' The UAV will beam down an image of that area to him," Couture says. "A two-star general might want [data about] 30 square miles; the same UAV can give an overall snapshot of that. It's like having a server in the sky, capturing data in real time and in no time beaming it to the ground to have analysts look at it. They have real-time access to data in the midst of it - in-theater data.
"High-level processing on the platform is possible because of the availability of Intel Xeon D, the processor of choice in the supercomputing realm we're bringing into the mil-aero realm," Couture continues. "WAAS is one of the most compute-intensive applications there is. Customers are asking how much they should prosecute with FPGAs, processors, and GPUs. Now we're down to the chip level, and the tricky thing is how to build boards and enclosures to make use of these massive efficiencies. Infrastructure technologies and hybrid processing may change how we architect systems in the future, and enable new ways to do things."
