Electro-optical sensor payloads combine with sophisticated digital signal processing to create persistent-surveillance capabilities able to look backwards and forwards through time.
By J.R. Wilson
Maintaining constant surveillance of the airspace around important military and intelligence sites has been a priority for since World War II, when the only persistent capability was radar, still in its infancy.
The Lockheed Martin Hybrid Airship combines the upsides of an airship with the ability of a hovercraft. It can maneuver on the ground, without a traditional runway or even smoothed surfaces.
Radar, significantly improved in range and detection capability since then, remains a mainstay of persistent surveillance, but in the 21st Century is only one part of a growing number of technologies covering not only airspace but providing surveillance of surrounding land and seascapes, as well.
“The key element for [airborne] persistent surveillance is there are different sizes of UAVs [unmanned aerial vehicles] with different payloads and CONOPs, the data they collect, analyzing real-time, storing, and available cooling,” says Marc Couture, director of engineering for intelligence, surveillance, and reconnaissance (ISR) at Curtiss-Wright Defense Solutions Division in Ashburn, Va.
High-performance embedded computing (HPEC) aspects of today’s persistent-surveillance and electronic warfare (EW) systems are taking over several jobs from human intelligence analysts, Couture points out.
“With cognitive EW, you are replacing some of the things once performed by an analyst who would pick out targets, but now these cognitive systems are getting better at identifying items of interest — tanks under trees, and civilian vs. mil convoys,” he says. “It not only aids faster detection, it can identify and assess different targets, provide certain parameters and help guide decisions much, much faster as actual encounters occur. It puts a tighter loop on the kill chain, better solutions and fewer mistakes.”
Sensor operators for persistent-surveillance systems use multi-window graphics systems to keep a close eye on broad areas for suspicious activity or other movements of interest.
Of growing importance — especially given the volume of data from multiple sensors involved in persistent surveillance — are the quality, capabilities, and in-field upgradeability of the system’s onboard software.
“Depending on how you pass around data, you may have tens of thousands of lines of pre-written code by the primes, so our job is to provide deployable field replaceable units that provide choices,” Couture says. “The networking and fabricing are absolutely critical, in terms of speed, but also software reliability; there are a lot of correlated data in time and space that have to be accounted for so you don’t want to miss the right thing.”
High-performance embedded computing
Curtiss-Wright is pursuing Open HPEC, which combines multi-core Xeon general-purpose processors, general-purpose graphics processing units (GPGPUs), and in the future will tie in field-programmable gate arrays,” Couture explains. “The days of some programmer being able to optimize all this data traffic are over, so the software has to be just as good as the hardware.”
Multi-core Intel Xeons and GPGPUs are tightly associated with deep and machine learning, which are necessary for demanding applications like cognitive EW, Couture says. “You need different elements for different functions. And you have SWaP [size, weight and power] considerations, as well.”
One expert has discussed the technologies and sensors required by UAVs, manned aircraft, and spacecraft essential to activity-based persistent surveillance of tactical targets.
Those include passive detection, radar detection, antenna monopulse, active electronic scanned antennas (AESA), moving target detection (MTD) and moving target tracking (MTT), motion compensation, tactical target spectral characteristics, space/time adaptive processing (STAP), synthetic aperture radar (SAR) imaging, change detection (CCD) and synthetic monopulse, veteran design engineer and consultant David Lynch Jr., president of DL Sciences Inc. in El Segundo, Calif., wrote in his 2018 book, Tactical Persistent Surveillance Radar with Applications.
As technological evolution continues to shrink those components in size and power, while increasing range, capability, and interoperability, more elements can be packed into the tight confines of UAVs, helicopters, satellites, and other platforms — or enable relatively small aircraft to carry systems that traditionally have been outside their capabilities. That is expected to be a major asset to developing swarm procedures for tactical, operational, and strategic ISR.
The BAE Systems 2-Color Advanced Warning System (2C-AWS) will provide aircraft with missile warning and hostile-fire protection to improve survivability and mission effectiveness in contested environments.
Aerostats a preferred platform
While persistent surveillance is still seen largely as the province of UAVs, ground-based radars, and other sensors, an increasingly common solution involves aerostats — lighter-than-air, typically tethered unmanned platforms, often resembling blimps, of various sizes, payload capacities, and altitude limits.
Markets & Markets, a global research and analysis firm, projects the worldwide aerostat market will reach nearly $11 billion by 2021, up from about $6 billion today. Another market research firm — Inkwood Research — projects the U.S. and Canadian aerostat market alone will reach nearly $8.7 billion by 2026, China and India nearly $6.5 billion, and Europe $4.4 billion. The majority of new aerostats will be for military and homeland security applications, as is the case today.
Mixing cutting-edge technologies with more than 165 years of design evolution, aerostats are produced for military and homeland security persistent surveillance applications by a wide range of contractors worldwide, including potential adversaries:
- Aero Surveillance in Avignon, France;
- Aeros in Montebello, Calif.;
- Airborne Industries in Southend-On-Sea, England;
- Allsopp Helikites Ltd. in Fordingbridge, England;
- Auger RosAeroSystems in Moscow;
- Drone Aviation Corp. in Jacksonville, Fla.;
- Elbit Systems in Haifa, Israel;
- ILC Dover LP in Frederica, Del.;
- Israel Aerospace Industries (IAI) in Tel Aviv;
- Lockheed Martin Aeronautics in Fort Worth, Texas;
- Raven Aerostar in Sioux Falls, S.D.;
- Raytheon Co. in Waltham, Mass.;
- Rheinmetall Canada in Quebec City;
- RT LTA Systems Ltd. in Yavne, Israel;
- SES in Betzdorf, Luxembourg; and
- TCOM in Columbia, Md.
A variation on the theme is Aero Surveillance’s new Tethered Quad and Octocopter ASV 30 400 and 800 series, some of which can be used in tethered and free-flying modes, controlled by the same ground station. Offered in four- and eight-rotor configurations, the company says it can remain aloft for as long as 12 hours at 328 feet in tethered mode or one hour with a range of six miles in free flight. With a payload capacity to 13.23 pounds, it can carry a high-performance electro-optical sensor able to detect mobile objects from as far away as six miles.
The Kestrel wide-area electro-optical surveillance system from Logos Technologies is for deployment on aerostats and airships operating in Afghanistan at altitudes between 1,000 and 5,000 feet off the ground.
Operations planning
All approaches to persistent surveillance — especially of a non-domestic or foreign locale or asset — are subject to unexpected or largely unavoidable problems, from man-made countermeasures to natural events like earthquakes, volcanic eruptions, floods, and storms. One goal of persistent surveillance providers is a quick-response capability to reduce or even counter the impact of such events.
Experts proposed a strategy to simplify updates and reduce surveillance gaps in multi-agent, persistent surveillance missions in a January 2017 paper on “Cloud-Supported Coverage Control for Persistent Surveillance Missions.” Authors were from the University of California-Santa Barbara and one from the United Technologies Research Center in East Hartford, Conn.
The SkEye WAPS from Elbit Systems in Haifa, Israel, provides situational awareness of on-the- ground intelligence data, and enables a large number of users to receive real-time, high-resolution imagery and even go back in time.
“This approach decouples assignment from motion planning operations in a modular framework,” the paper’s abstract says. “Coverage assignments and surveillance parameters are managed on the cloud and transmitted to mobile agents via unplanned and asynchronous exchanges. These updates promote load-balancing, while also allowing effective pairing with typical path planners.
“Namely, when paired with a planner satisfying mild assumptions, the scheme ensures that (1) coverage regions remain connected and collectively cover the environment, (2) regions may go uncovered only over bounded intervals, (3) collisions (sensing overlaps) are avoided and (4) for time-invariant event likelihoods, a Pareto optimal configuration is produced in finite time.”
Pareto optimality is a state in which it is impossible to reallocate resources to make any one individual or preference criterion better off without making at least one other individual or preference criterion worse off.
Using several different sensor platforms, such as unmanned aircraft swarms or combined air-, ground-, sea-, and even space-based systems, also can improve surveillance continuity and effectiveness, although both are significantly more difficult if the surveillance target is in unfriendly territory. That is a major consideration pushing the development of new sensor and platform technologies with greater range and system defenses (passive in the short-term) with higher SWaP characteristics.
These issues are prominent in persistent surveillance schemes of the U.S. and its allies, but also those of potential adversaries or nations that may market such systems to unfriendly actors, including non-state, terrorist and even criminal enterprises.
State of the art
While much of the research and current state of the art of various surveillance systems and technologies are classified, even the Chinese have published papers investigating the problem and potential solutions. One such, published in January 2018 in the International Journal of Distributed Sensor Networks by scientists primarily from the School of Aeronautics & Astronautics at Sichuan University in Chengdu, China, looked at ways to control several UAVs over a region of interest (ROI}, obstacle and collision avoidance and group coordination on-station to reduce unobserved areas as much as possible.
“Due to the fact that the loss or damage of individual UAV does not affect the performance of the system, multiple UAVs are highly reliable and more scalable than single complicated UAV. Moreover, multiple UAVs can provide multi-angle observations, which is very important in the surveillance task. Consequently, multiple UAVs are advantageous over single UAV for executing persistent surveillance task,” they wrote in “Potential Field Method for Persistent Surveillance of Multiple UAV Sensors.”
Addressing swarms of small UAVs and “flocking” micro-UAVs, the paper concludes with a call for future studies to determine the optimal number of platforms to provide uninterrupted persistent surveillance of various types and sizes of targets and returns on investment. Such efforts already are underway at U.S. military facilities, looking into future sensor and platform development; remote control versus increased autonomy; networking multiple sensors on multiple platforms; fusing data for transmission back to surveillance command and control centers.
Wide-area motion imagery (WAMI) can monitor an area of many square miles in great detail, and uses smart algorithms to extract relevant information automatically.
That is not limited to small or micro-UAVs, however. Curtiss-Wright, for example, is working many of those same issues on large high altitude/long endurance (HALE) and medium altitude/long endurance (MALE) platforms, such as the RQ-4 Global Hawk, MQ-4C Triton, and MQ-9 Reaper. That includes sensor processing – SAR, fire control, E/O-IR focal plane arrays, searching for RF emitters associated with SIGINT, and EW.
The role of SWaP-C
“We are always asked to do a lot of processing in a small, constrained SWaP-C envelope, which usually means having a tightly coupled, high-speed interface to the sensor processor so the data being collected in real-time can be processed as quickly as possible — not just one type or processor but several,” Curtiss-Wright’s Couture explains.
“The most common is two or three FPGAs [field-programmable gate arrays], using processing technology that is right behind some form of Ethernet from the radar phased array; then behind that Xeon-D processor, which is very popular in defense, because they give a lot of processing power but don’t require a lot of power, so you can embed them.”
General Atomics Aeronautical Systems MQ-9B SkyGuardian version of its Predator UAV can provide persistent surveillance by flying in excess of 35 hours with air speeds to 210 knots at altitudes of more than 40,000 feet.
That leads to what Couture calls the heterogeneous processing technology triple play — FPGAs, Xeon-D, and GPGPUs — which he describes as a massive math accelerator, with terafloating point operations per second. As such technologies continue to grow in power and capability, so will the need for greater autonomy to produce real-time information for decision-makers.
“Deep learning and machine learning, often used synonymously, get into concepts that stem from AI theory,” Couture says. “GPGPUs have hundreds or thousands of cores and can do a lot of what a human analyst did in the past. In persistent surveillance, you have multiple waterfalls of data — SAR imaging of ground topology, emitter data, RF, microwave — you may be looking to process or even superimpose with other data and hyperspectral imaging.
Heterogeneous processing
“That takes a lot of horsepower, especially in real time, so a lot of what we build are geared toward ingesting massive amounts of data and being able to pass it between these processing elements. So it’s not just about putting powerful processing technology on printed circuit boards, but all the plumbing and infrastructure, the physical hardware, along with the middleware and fabric software and all that being able to function properly in adverse environment.”
Hyperspectral is a wider swath of the electromagnetic spectrum, while multispectral is different types of cameras, he explains.
“So you don’t just have a set of eyes on a target, but different types of eyes. They can see through things at different wavelengths that other sensors can’t. Being able to have that data plus SAR imagery and EO — the more information you can gather, the better decision you will make.”
As the long wars in Iraq and Afghanistan demonstrated, U.S. military aircraft of all types — helicopters, fighters, bombers, cargo, and transport — are airborne around the clock and, in some cases, fly to theater from U.S. bases, returning once their mission is accomplished. For nearly four decades, the U.S. has had nearly unchallenged air superiority, with little danger to the higher-flying platforms and a record high mission success rate at all levels.
If any future conflict involves Russia, China, or one of their military clients, that will no longer be the case. And while persistent surveillance always has been associated with defending the air, ground, and sea spaces around U.S. and allied installations and formations or keeping a constant eye on unfriendly sites and military movements, future conflicts will require a higher level of persistence in a moving bubble around U.S. aircraft, as well.
Influential applications
In the early days of Operation Iraqi Freedom, the Army fielded the Common Missile Warning Solution to protect helicopters from shoulder-fired missiles. Developed by BAE Systems, it later was updated to provide hostile fire protection. In December 2017, BAE Systems Information and Electronic Systems in Nashua, N.H., won the contract for the Limited Interim Missile Warning System (LIMWS) Quick Reaction Capability, designed to fill a gap between the current AN/AAR-57 common missile warning system (CMWS) and the Army’s next threat warning system program of record, the Advanced Threat Detection System (ATDS).
Designed to detect and track man-portable air defense system threats for missile warning, as well as various forms of fire to provide hostile fire indication (HFI), the LIMWS is based on BAE’s 2-Color Advanced Warning System (2C-AWS), developed in concert with Leonardo DRS (Dallas, TX), which is providing the 2-color infrared sensor — the eyes of the system.
“The CMWS was fielded across the Army fleet — more than 2000 aircraft — and saved hundreds of aircraft and lives. With the LIMWS, the Army wants to add an IR sensing capability, which we looked at as an upgrade to our CMWS legacy, making what we had learned in fielding that program part of our 2C-AWS bid,” says Chris Austin, director of advanced warning systems at BAE Systems.
Lighter-than-air aerostats are among the most promising and widely used air platforms for persistent surveillance sensor payloads.
“A combination of hardware and software improvements and advanced missile warning and hostile fire algorithms, it brings an enhanced threat protection capability to address new and emerging threats. Initially, it will be installed on the Army’s UH-60M Black Hawk helicopters. It is limited to 360-degree protection of an individual aircraft. It does have a reactive capability, with the 2C-AWS coupled with flare and directable IR countermeasures.”
The biggest difference between the CMWS and LIMWS, he added, is the new system incorporates E/O-IR sensors, where CMWS uses the ultraviolet spectrum. While the current design does not include multispectral or hyperspectral sensors, he says those are advanced capabilities being looked at for future upgrades.
The first units are scheduled for fielding in 2020. Competition for the ATDS is not expected to begin until later in the decade, giving the LIMWS about a decade to enhance persistent surveillance for Army — and possibly other military — rotorcraft.
The next generation
“We’re looking at future capabilities, at the next generation leads for aircraft survivability. This pr ogram is for the Army, but we’re looking at opportunities with the Navy and Air Force, as well, primarily rotorcraft,” Austin adds. “We’re always looking at where the Army may go, focused primarily on rotor wing aircraft and how the services plan to utilize those and making sure we stay up with their needs.
“On the Air Force side, CMWS was selected as the missile warning system for their new Combat Rescue Helicopter. There is no active solicitation for something like 2C-AWS, but the Army is fielding it on their Black Hawks and CRH is a Black Hawk variant, so it could be expanded to that platform. We have developed this system for future advanced capabilities the Army may require — future sensors and countermeasures, so it has the ability to grow and adapt to future new technologies.”
BAE also provides a form of persistent surveillance for the Navy with its Lock-on Naval Electro-optic/infrared Sensor System (LockNESS), a fully automated shipboard defense system designed to protect carriers and other surface ships and crews against current and evolving surface threats.
Homeland and border security
Persistent surveillance also has expanded into homeland security border protection capabilities.
Elbit Systems in Haifa, Israel, has developed the Border Surveillance System (BSS) to provide real-time regional surveillance, early warning, targeting and border-patrol mission management and improve terrestrial and maritime security, regional control and border protection by enabling complete real-time connectivity across the entire border-security array. According to Elbit, “these solutions enable optimum coordination of ISR efforts throughout all operational zones and along the country’s borders, supporting stationary control centers, deployed command posts and tactical patrol agents.”
In the U.S. Navy’s proposed 2019 budget, the Marine Corps is seeking to improve its ground-based persistent surveillance capabilities, citing the need to “facilitate strategic planning for resources and technology and enable the migration of discrete functional capabilities into integrated capabilities that are in alignment with the strategic vision for the Marine Corps Intelligence, Surveillance, and Reconnaissance Enterprise. Ground Based Operational Surveillance was absorbed into the new family of systems, Terrestrial Collection.
The Harris Corp. CorvusEye intelligent wide-area motion imagery (WAMI) sensor system is a down-sized SWaP-compliant offshoot of the military Gorgon Stare program for use on small aircraft.
“[Requested budget increases for] Battlespace Awareness Information Technology Services supports non-labor information technology requirements reportable under the provisions of the Clinger Cohen Act of 1996. Increase comes from combining GBOSS, MAGTF Secondary Imagery Dissemination System and Tactical Remote Sensor system to become Terrestrial Collection, [along with increases for] Distributed Common Ground System (DCGS)–Marine Corps (DCGS-MC), Technical Control and Analysis (TCAC), Communications Emitter Sensing Attack System (CESAS), [and] Counter Intel Human Intel Equipment (CIHEP).”
Other systems being offered in this arena include: Airborne Industries’ Persistent Ground Surveillance Systems (PGSS Aerostats) for wide area surveillance. According to the company, “Due to the increased security threat in many regions, persistent surveillance has become critical in most military operations. With a flight duration of 7 days, the Aerostat systems’ ability to detect potential threats and provide intelligence to decision-makers over long periods of time are unrivaled.”
Persistent-surveillance sensor payloads can blend a wide variety of electro-optical, radar, and even sound sensors to create a complex picture of activity down below.
The Harris Corp. CorvusEye intelligent wide-area motion imagery (WAMI) sensor system is a down-sized, SWaP-compliant offshoot of the Department of Defense’s Gorgon Stare program for use on smaller aircraft. According to Harris, “CorvusEye provides an overview of and automatically detects movement within its coverage area. The dual-mode electro-optic/infrared sensor means surveillance operations can continue uninterrupted from day-to-night and night-to-day.”
Logos Technologies maintains a major position in WAMI with the aerostat-mounted Kestrel, Kestrel Block II, Simera and Serenity; aircraft-mounted Redkite, Hermes and LEAPS, and maritime Skua systems. According to the company, “WAMI allows operators and analysts to detect and track multiple geographically dispersed targets of interest at once [and] uncover hidden relationships between locations and events within the sensor’s vast field of view. The newest generation of WAMI sensors are miniaturized and more affordable and are available for a range of platforms, including various sized aerostats and multiple classes of unmanned aircraft.”
Thus persistent surveillance, in one form or another, is growing as a key element for all four U.S. military services and the Department of Homeland Security in protecting critical infrastructure and assets and is likely to see increased technology advances and applications with each year going forward.
