Dealing with Big Data

Aerospace and defense applications call for innovative information storage solutions delivering increased capacity, security, and speed.

Current and future missions increasingly rely on the timely and reliable capture, processing, storage, and delivery of accurate, actionable information. This is true of myriad civil and defense missions, whether on land, at sea, or in air or space.

Modern sensor, camera, communications, and computer systems deliver a wealth and variety of mission-critical data to today's warfighter. The availability and use of this data contributes to increased situational awareness, survivability, and mission effectiveness; yet, managing-and, in the process, gleaning valuable information from-this vast volume of data is a growing challenge.

"Demands for actionable information have spiked as warfighters at every level are called upon to make well-informed decisions," says a representative of the Defense Advanced Research Projects Agency (DARPA) in Arlington, Va. "To maximize mission effectiveness and enhance national security, the [U.S.] Department of Defense (DOD) is now challenged to more efficiently fuse, analyze, and disseminate the massive volumes of data this network produces."

A majority of currently fielded electronics and software cannot easily be scaled to meet this data deluge. Rather, "the volume and characteristics of the data, and the range of applications for data analysis, require a fundamentally new approach to data science, analysis, and incorporation into mission planning on timelines consistent with operational tempo," the representative adds.

Government, industry, and academia are working together to solve the challenges of and exploit the tremendous value of "Big Data."

Curtiss-Wright Vortex Compact Network Storage products are aboard U.S. Air Force HC/MC-130J special operations aircraft.
Curtiss-Wright Vortex Compact Network Storage products are aboard U.S. Air Force HC/MC-130J special operations aircraft.

Big data deluge

The Obama Administration in March 2012 launched the Big Data Research and Development Initiative. The initiative is intended to advance state-of-the-art core technologies needed to collect, store, preserve, manage, analyze, and share huge quantities of data; harness these technologies to accelerate the pace of discovery in science and engineering, strengthen national security, and transform teaching and learning; and expand the workforce needed to develop and use Big Data technologies.

"We intend to change the game and plan to be the first to leverage Big Data across the full scope of military operations in new and unconventional ways," says Zachary Lemnios, U.S. assistant secretary of defense for research and engineering. "We are within sight of a new generation of systems that understand and interpret the real world with computer speed, computer precision, and human agility. These systems will be central to helping our commanders and analysts make sense of the huge volumes of data our military sensors collect at speeds 100X faster than today."

The project has the potential to solve key challenges and move government plans ahead for Big Data. "By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help solve some the nation's most press- ing challenges," says John P. Holdren, assistant to the president and director of the White House Office of Science and Technology Policy (OSTP). "Six Federal departments and agencies announced more than $200 million in new commitments that promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data."

The DOD will invest $250 million annually under the initiative, including $60 million dedicated to new research projects. DOD Big Data projects focus on three areas: data to decisions, autonomy, and human systems. Data-to-decisions investments will develop computational techniques and software tools for analyzing large volumes of data. Autonomy investments center on support tools that can recognize trends, adapt to real-world conditions, and succeed in complex, dynamic environments without relying on human intervention. Human-systems investments will advance human-machine interfaces to enable collaboration for operations and for training.

Banding together

Some companies already are sponsoring Big Data-related competitions, and providing funding for university research, according to Tom Kalil, deputy director for policy, White House OSTP. "Universities are beginning to create new courses and entire courses of study to prepare the next generation of 'data scientists.'"

DARPA officials, recognizing the challenges associated with large volumes of data, started the XDATA program in concert with the launch of the Big Data Initiative last spring. The goal is to develop computational techniques and software tools for processing and analyzing the vast amount of mission-oriented information for defense activities.

XDATA also aims to address the need for: scalable algorithms for processing and visualization of imperfect and incomplete data, human-computer interaction tools that could be easily customized for different missions, and open-source software toolkits to enable collaboration among the applied mathematics, computer science, and data visualization communities.

"It's a great time to leverage recent commercial and academic advances in processing large amounts of data for analysis," says Chris White, DARPA program manager. "We are calling on all technical communities with expertise in this area to help us ensure our men and women in uniform have the benefit of the best information we can provide."

The U.S. National Science Foundation (NSF), in collaboration with the National Institutes of Health (NIH), is seeking proposals under a Big Data solicitation: "Core Techniques and Technologies for Advancing Big Data Science & Engineering." It will fund research to develop and evaluate new algorithms, statistical methods, technologies, and tools for improved data collection and management, data analytics, and e-science collaboration environments.

"NSF is developing a bold and comprehensive approach for this new data-centric world, from fundamental mathematical, statistical, and computational approaches needed to understand the data, to infrastructure at a national and international level needed to support and serve our communities, to policy enabling rapid dissemination and sharing of knowledge," explains Ed Seidel, assistant director for NSF's mathematical and physical sciences directorate.

The Microsemi Corp. TRRUST-Stor secure solid-state drive, shown above, was designed for data-intensive applications that require secure, anti-tamper data storage.
The Microsemi Corp. TRRUST-Stor secure solid-state drive, shown above, was designed for data-intensive applications that require secure, anti-tamper data storage.

Defense directives

Forces operating together in deployments throughout the world require real-time access to actionable intelligence, says a spokesperson at Lockheed Martin Corp. in Bethesda, Md. "The Department of Defense's response to this need is a global enterprise where military services, national agencies, and coalition partners can access time-sensitive intelligence, surveillance, and reconnaissance (ISR) data. This collaborative enterprise is the Distributed Common Ground System (DCGS)."

DCGS draws and correlates intelligence data from several integrated ISR sensors and systems into a comprehensive view of the battlefield. At the heart of DCGS is the DCGS integration backbone (DIB), which enables users worldwide to access data.

Lockheed Martin engineers are upgrading this open-standards software infrastructure that national agencies, coalition partners, and military services use to share intelligence. The $2.6 million contract from the DCGS Multi-Service Execution Team office will help "the DCGS team achieve its vision for maximizing the value of DCGS to commanders, war- fighters, and intelligence agencies," says Jim Quinn, vice president of C4ISR Systems for the Lockheed Martin Information Systems & Global Solutions Defense segment in Rockville, Md. "The latest DIB upgrade will enable our partners to securely share intelligence faster than ever before."

Lockheed Martin engineers are enhancing the latest version of the DIB with increased security filtering capabilities, an enhanced data ingest framework, increased ingest and query capabilities, and the Distributed Data Framework, which uses free and open-source software and supports the integration of new data sources, data transformation services, and user interfaces into the DIB.

High-level U.S. Army officials, during the 2012 Association of the U.S. Army (AUSA) conference in Washington, shared the organization's vision of the future or Army 2020, of which Army Intelligence 2020 is an integral part. Major efforts under Army Intelligence 2020 include: ubiquitous access to data via a cloud- enabled, interoperable DCGS-A with advanced analytics; optimized, multifunctional, and integrated sensors and collectors; and assured communications and robust processing power in theater via Relevant Intelligence to the Edge (RITE), also called Relevant ISR to the Edge.

Location is everything

"Sensors have become prolific," says Allan Cantle, chief technology officer of Interconnect Systems Inc. (ISI) and president of Nallatech in Camarillo, Calif. The speed, resolution, and data rates of sensors have increased, while geometries continue to shrink. "Invariably, the bandwidth of the communication channels is not keeping pace with these increasing amounts of data. This challenge has created the need to either process the data locally at the sensor or store the data for retrieval and postprocessing later after a mission is complete."

More sensors acquiring more data, in higher quality and quantity, are driving the need for both processing at the edge and data storage. "Whilst raw data storage is often essential in many military systems, the ever- growing data deluge is making it impractical to simply continue capturing and storing this data," Cantle notes. "Instead, it is becoming far more important to find techniques that can capture the important data and either store this significantly reduced data or make intelligent and immediate decisions based on the real-time data."

Aerospace and defense "storage systems are typically Flash memory based because of their ruggedness and speed compared to spinning hard drives," Cantle says. "These storage systems will often include both compression and encryption techniques to maximize storage density, as well as protect the data should the unmanned aerial vehicle (UAV) get into the wrong hands.

"Often, local data processing is favored over storage because the delays involved with data storage and postprocessing can often lead to stale intelligence, which becomes worthless," Cantle continues. "In these circumstances, data processing at the sensor will normally focus on data reduction techniques where the resulting bandwidth is low enough for transmission over the communication channels."

Alternatively, says Cantle, "the computational speed and low power of today's embedded computers are facilitating complete processing systems to reside on a UAV, including sensor processing, data fusion, and situational awareness. These types of systems provide a very rapid and intelligent identification of events that can be easily communicated with extremely low bandwidth requirements or can even allow a UAV to 'close the loop' and immediately react to a hostile event."

ISI and Nallatech, specializing in the miniaturization of electronics systems, unveiled the NSC series to enable high-speed processing at the edge using field-programmable gate array (FPGA) technology. "These systems allow customers to put 'Near Sensor Computing' (NSC) right next to tiny sensors, allowing for real time computing of high-speed data streaming from the sensor, without costly and power-hungry communication systems or bulky storage systems," Cantle explains. "Arrays of sensors with these NSC engines can then be used to build powerful and efficient distributed computing within the UAV.

"This type of NSC engine has been utilized in experimental sense-and-avoid systems, where a UAV must identify potential head-on collision events with other aircraft and react extremely quickly in order to avoid a collision," Cantle says.

The Lockheed Martin HC/MC-130J network-attached storage (NAS) system onboard the HC/MC-130J aircraft employs Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products.
The Lockheed Martin HC/MC-130J network-attached storage (NAS) system onboard the HC/MC-130J aircraft employs Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products.

Certified security

The need for data processing and data storage in theater is driving the demand for greater security onboard manned and unmanned vehicles.

"Data security is a major concern for mobile vehicle system architects," says Paul Davis, director of product management at Curtiss-Wright Controls Defense Solutions in Dayton, Ohio. "Quite often the storage modules are transported from a ground station to vehicle, and back again. During that transport process, critical data-at-rest must be protected."

Security based on Federal Information Processing Standard (FIPS) 140-2 is acceptable for most commercial aerospace and some military applications, Davis explains. Many companies, including Curtiss-Wright, provide certified storage products so customers do not have to worry about the certification process. Different levels of FIPS security exist, he cautions, so buyers should inquire about certification levels.

"For more critical DOD storage applications, National Security Agency (NSA) Type 1 security is often required. NSA certification is more difficult to achieve as it requires a DOD program sponsor and takes a long time to complete the certification process," Davis says. "In other countries, FIPS 140-2 certified encryption products are widely recognized and accepted. In the U.K., Communications Electronics Security Group (CESG) certification may be required when providing data storage to the Ministry of Defense (MOD). In any application, it is always a good idea to leverage existing certified products."

Less than zero

Engineers and end users have often sought removable hard disk drives or a method by which a disk drive and the data contained therein could be destroyed instantly or erased. Now, the aerospace and defense community is growing "more comfortable with the idea of zeroizing, or purging, the encryption key," Davis says.

"An AES256-bit key can be zero-ized in a fraction of a second," Davis adds. "It renders the data unclassified and safe for transport. It is estimated that it would take more than 1,056 years to break an AES256-bit key by brute-force methods. Since any data is time-sensitive, key zeroizing is a very effective method of protection."

Zeroizing is now often performed instead of executing a military secure-erase purge algorithm, Davis explains. "These older methods require overwriting of data several times with specific data patterns. With one terabyte of solid-state memory, such purge algorithms can take up to an hour to execute. In an emergency situation, like a UAV landing, an hour may be too long to secure the critical data."

Lockheed Martin Aeronautics engineers in Marietta, Ga., adopted Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products on the HC/MC-130J network-attached storage (NAS) system deployed on U.S. Air Force Air Combat Command MC-130J special mission aircraft and Air Force Special Operations Command HC-130J personnel recovery aircraft. The Lockheed Martin HC/MC-130J Super Hercules extended-range transport aircraft is designed for special missions, such as search and rescue.

"The data is secured with the FIPS 140-2 certified FSM product. AES256-bit encryption keys are used within inline media encryptors, which protect the data prior to storage on solid-state SATA drives," Davis describes. "These devices also decrypt the data when being retrieved by network clients. The AES256-bit keys can be zeroized in any one of three convenient ways: pushbutton, command, or discreet input."

Design considerations

Data security is a common concern, says Paul Ambuehl, storage technical lead at Extreme Engineering Solutions (X-ES) in Middleton, Wis. "Many customers look for solid-state drives (SSDs) that support military erase features and 256-bit AES encryption.

"Mil-aero system designers continue to prefer SSDs, as they are built to military-grade temperature, shock, and vibration specifications, conditions where spinning media would simply not perform," Ambuehl adds. "Mil-aero designers prefer SLC (single-level cell) flash technology, known for its extremely high reliability coupled with its exceptional read/write performance."

Ambuehl advises aerospace and defense engineers to pay attention to how the storage system or solution is constructed. Does the system use off-the-shelf SSDs? If so, what kind and what connectors are internal to the system? "If the system utilizes standard 2.5- or 1.8-inch SATA SSDs, it could be using standard SATA connectors. In some harsh environments, if a system is not properly designed, implemented, and tested, standard SATA connectors could be prone to failure due to excessive vibration," he advises.

When starting any system design, start with commercial off-the-shelf (COTS) secure storage products that are on the market already, recommends Curtiss-Wright's Davis. "The DOD end-user may be willing to consider a secure data-at-rest solution that exists in order to reduce technical and schedule risk. Such an existing solution can save considerable non-recurring engineering (NRE) and yet provide a robust solution.

In the U.S., NSA Type 1 certified solutions are frequently required, Davis says. "Solutions such as the Curtiss-Wright's Compact Network Storage (CNS) system combined with General Dynamics' DAR400E encryption device provide a certified Secret and Below (SAB) solution for data at rest. The CNS/DAR provides NSA Type 1 encryption now, instead of waiting years to accomplish."

Storage capacity is another important consideration, especially in ISR applications with sensors producing continuous streams of data, Davis says. "Applications often start by considering a minimal storage capacity. Quickly they realize that more and more devices in a network-centric storage system naturally grow to save more and more critical data."

Area density, continues to be one of the biggest challenges that the mil-aero community faces, says Ambuehl at X-ES. "System designers typically do not have a lot of size, weight, and power (SWaP) to sacrifice for storage, so it requires the SSD modules to pack a lot of flash into a small area. With storage requirements of one terabyte or more and with the physical size of flash components, this can prove to be challenging."

Engineers certainly must weigh all the storage options to find the optimal fit for each individual aerospace and defense project. Technology companies continue broaden the portfolio of products from which aerospace and defense engineers can select their optimal solutions, particularly in the areas of information management, sharing, and storage.

Bet on big data

"The Department of Defense is placing a big bet on Big Data. Help us reach our visions by submitting your ideas to the solicitations," says Zachary Lemnios, U.S. assistant secretary of defense for research and engineering. Learn about the latest opportunities at hwww.defenseinnovation, a new website designed to better communicate DOD investment priorities with performers in industry and academia.


Aitech Defense Systems
Barracuda Networks
Cavium Networks
Crystal Group
Curtiss-Wright Controls Defense Solutions
DRS Technologies
Elma Electronic
Extreme Engineering Solutions (X-ES)
Falcon Electronics
Galaxy Data Storage
General Micro Systems
Germane Systems
Interconnect Systems Inc.
L-3 Communications Targa Systems
Mentor Graphics
Phoenix International Systems Inc.
RightNow Technologies
SANBlaze Technology
Themis Computer

Get All the Military Aerospace Electronics News Delivered to Your Inbox or Your Mailbox

Subscribe to Military Aerospace Electronics Magazine or email newsletter today at no cost and receive the latest information on:

  • C4ISR
  • Cyber Security
  • Embedded Computing
  • Unmanned Vehicles

Get All the Military Aerospace Electronics News Delivered to Your Inbox or Your Mailbox

Subscribe to Military Aerospace Electronics Magazine or email newsletter today at no cost and receive the latest information on:

  • C4ISR
  • Cyber Security
  • Embedded Computing
  • Unmanned Vehicles

Military & Aerospace Photos

Most Popular Articles

Related Products

XTend7103 | COM Express® Carrier for COM Express® Type 10 Mezzanine Modules

The XTend7103 is a COM Express® carrier card designed to provide a low-cost and compact platform ...

XPedite7575 | 5th Generation Intel® Core™ i7 Broadwell-H Processor-Based Conduction- or Air-Cooled 3U VPX-REDI Module

The XPedite7575 is a high-performance, 3U VPX-REDI, single board computer based on the 5th genera...

XPort6173 | 3U VPX Carrier for Two 2.5 in. Solid-State Drives (SSDs)

The XPort6173 supports two, standard, 2.5 in. Solid-State Drives (SSDs) in a single 0.8 in. or 1....

XCalibur4444 | Intel® Core™ i7 Processor-Based Conduction-Cooled 6U VPX Module

The XCalibur4444 is a high-performance, multiprocessing, 6U VPX, single board computer that is id...

XCalibur1840 | Freescale Twelve-Core T4240 Processor-Based Conduction- or Air-Cooled 6U VPX Module

The XCalibur1840 is a high-performance, 6U VPX, single board computer supporting Freescale QorIQ ...

XPm2010 |3U cPCI PICMG 2.11 Power Supply with Integrated MIL-STD-461E

The XPm2010 is a PICMG 2.11 power supply that takes in a MIL-STD-704 28 VDC input voltage and pro...

XPedite7478 | Intel® Core™ i7 Processor-Based 3U VPX-REDI SBC with Integrated 1553 and CAN Bus

The XPedite7478 is a high-performance, low-power, 3U VPX-REDI, single board computer based on the...

XCalibur1641 | Freescale Eight-Core P4080 Processor-Based Conduction- or Air-Cooled 6U VPX Module

The XCalibur1641 is a high-performance, 6U VPX, single board computer supporting Freescale QorIQ ...

XPedite7572 | 5th Gen Intel® Core™ i7 Broadwell-H Based Conduction- or Air-Cooled 3U VPX-REDI Module with SecureCOTS™

The XPedite7572 is a secure and high-performance, 3U VPX-REDI, single board computer based on the...

XPm2020 | 3U VPX Power Supply with Integrated MIL-STD-461E Filtering

The XPm2020 is a VITA 62-compliant 3U VPX power supply that takes in a MIL-STD-704 28 VDC input v...

Related Companies

Southwest Antennas

Designs and manufactures high-performance RF and Microwave antennas and accessories designed for today’s communicatio...

Curtiss-Wright Defense Solutions

About Curtiss-Wright Defense Solutions Curtiss-Wright Defense Solutions (CWDS) is a long established techno...

United Electronic Industries Inc

UEI is a leader in the PC/Ethernet data acquisition and control, Data Logger/Recorder and Programmable Automation Con...

Fabritech Inc

Fabritech was established in 1986 and specializes in the fabrication of EMI, RFI, and thermal shielded products made ...

NOVA Power Solutions Inc

Provides mission-critical rugged uninterruptable power supplies (UPS), power conditioners, and battery backup modules...

Sierra-Olympic Technologies Inc (SOTI)

Offers advanced infrared (IR) cameras (SWIR, MWIR, LWIR), thermal imaging components, MWIR/LWIR zoom optics and compl...


Spectracom supports mission-critical communications systems with precise and reliable time and frequency synchronizat...


Is a mechanical engineering consulting company headquartered in Los Angeles, CA with operations in Billerica, MA, pro...

Dspnor AS

Offers radar signal processing and distribution. The products interface to virtually any radar system in use today. T...


Pelorus Naval Systems is a specialist naval defense engineering and support services company with headquarters in Ran...


The Intel Xeon-D processor and its role in high-performance embedded computing (HPEC)

The rugged Intel Xeon-D server-class multicore microprocessor is set to revolutionize high-performance embedded computing. By itself, the processor will bring unprecedented power to embedded computing applic...

Harsh Environment Protection for Advanced Electronics and Components

This webinar will offer an opportunity to learn more about ultra-thin Parylene conformal coatings – how they are applied, applications they protect today, and the properties and benefits they offer, includin...

Press Releases


Curtiss-Wright Corporation today announced that its Defense Solutions division has received a contract from Sierra Nevada Corporation (SNC) to supply its small form factor ...

Innovative Integration Announces the FMC-Servo

Camarillo, CA June 19, 2015, Innovative Integration, a trusted supplier of signal processing and data acquisition hardware and software solutions, today announced the FMC-S...


Curtiss-Wright Corporation today announced that its Defense Solutions division has further enhanced its innovative VRD1 high definition (HD) video management system (VMS) w...

All Access Sponsors

Mil & Aero Magazine

August 2015
Volume 26, Issue 8

Download Our Apps




Follow Us On...


Military & Aerospace Electronics

Weekly newsletter covering technical content, breaking news and product information

Cyber Security

Monthly newsletter covering cyber warfare, cyber security, information warfare, and information security technologies, products, contracts, and procurement opportunities

Defense Executive

Monthly newsletter covering business news and strategic insights for executive managers

Electronic Warfare

Quarterly newsletter covering technologies and applications in electronic warfare, cyber warfare, optical warfare, and spectrum warfare.

Embedded Computing Report

Monthly newsletter covering news on embedded computing in aerospace, defense and industrial-rugged applications

Unmanned Vehicles

Monthly newsletter covering news updates for designers of unmanned vehicles