Dealing with Big Data

Aerospace and defense applications call for innovative information storage solutions delivering increased capacity, security, and speed.

Current and future missions increasingly rely on the timely and reliable capture, processing, storage, and delivery of accurate, actionable information. This is true of myriad civil and defense missions, whether on land, at sea, or in air or space.

Modern sensor, camera, communications, and computer systems deliver a wealth and variety of mission-critical data to today's warfighter. The availability and use of this data contributes to increased situational awareness, survivability, and mission effectiveness; yet, managing-and, in the process, gleaning valuable information from-this vast volume of data is a growing challenge.

"Demands for actionable information have spiked as warfighters at every level are called upon to make well-informed decisions," says a representative of the Defense Advanced Research Projects Agency (DARPA) in Arlington, Va. "To maximize mission effectiveness and enhance national security, the [U.S.] Department of Defense (DOD) is now challenged to more efficiently fuse, analyze, and disseminate the massive volumes of data this network produces."

A majority of currently fielded electronics and software cannot easily be scaled to meet this data deluge. Rather, "the volume and characteristics of the data, and the range of applications for data analysis, require a fundamentally new approach to data science, analysis, and incorporation into mission planning on timelines consistent with operational tempo," the representative adds.

Government, industry, and academia are working together to solve the challenges of and exploit the tremendous value of "Big Data."

Curtiss-Wright Vortex Compact Network Storage products are aboard U.S. Air Force HC/MC-130J special operations aircraft.
Curtiss-Wright Vortex Compact Network Storage products are aboard U.S. Air Force HC/MC-130J special operations aircraft.

Big data deluge

The Obama Administration in March 2012 launched the Big Data Research and Development Initiative. The initiative is intended to advance state-of-the-art core technologies needed to collect, store, preserve, manage, analyze, and share huge quantities of data; harness these technologies to accelerate the pace of discovery in science and engineering, strengthen national security, and transform teaching and learning; and expand the workforce needed to develop and use Big Data technologies.

"We intend to change the game and plan to be the first to leverage Big Data across the full scope of military operations in new and unconventional ways," says Zachary Lemnios, U.S. assistant secretary of defense for research and engineering. "We are within sight of a new generation of systems that understand and interpret the real world with computer speed, computer precision, and human agility. These systems will be central to helping our commanders and analysts make sense of the huge volumes of data our military sensors collect at speeds 100X faster than today."

The project has the potential to solve key challenges and move government plans ahead for Big Data. "By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help solve some the nation's most press- ing challenges," says John P. Holdren, assistant to the president and director of the White House Office of Science and Technology Policy (OSTP). "Six Federal departments and agencies announced more than $200 million in new commitments that promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data."

The DOD will invest $250 million annually under the initiative, including $60 million dedicated to new research projects. DOD Big Data projects focus on three areas: data to decisions, autonomy, and human systems. Data-to-decisions investments will develop computational techniques and software tools for analyzing large volumes of data. Autonomy investments center on support tools that can recognize trends, adapt to real-world conditions, and succeed in complex, dynamic environments without relying on human intervention. Human-systems investments will advance human-machine interfaces to enable collaboration for operations and for training.

Banding together

Some companies already are sponsoring Big Data-related competitions, and providing funding for university research, according to Tom Kalil, deputy director for policy, White House OSTP. "Universities are beginning to create new courses and entire courses of study to prepare the next generation of 'data scientists.'"

DARPA officials, recognizing the challenges associated with large volumes of data, started the XDATA program in concert with the launch of the Big Data Initiative last spring. The goal is to develop computational techniques and software tools for processing and analyzing the vast amount of mission-oriented information for defense activities.

XDATA also aims to address the need for: scalable algorithms for processing and visualization of imperfect and incomplete data, human-computer interaction tools that could be easily customized for different missions, and open-source software toolkits to enable collaboration among the applied mathematics, computer science, and data visualization communities.

"It's a great time to leverage recent commercial and academic advances in processing large amounts of data for analysis," says Chris White, DARPA program manager. "We are calling on all technical communities with expertise in this area to help us ensure our men and women in uniform have the benefit of the best information we can provide."

The U.S. National Science Foundation (NSF), in collaboration with the National Institutes of Health (NIH), is seeking proposals under a Big Data solicitation: "Core Techniques and Technologies for Advancing Big Data Science & Engineering." It will fund research to develop and evaluate new algorithms, statistical methods, technologies, and tools for improved data collection and management, data analytics, and e-science collaboration environments.

"NSF is developing a bold and comprehensive approach for this new data-centric world, from fundamental mathematical, statistical, and computational approaches needed to understand the data, to infrastructure at a national and international level needed to support and serve our communities, to policy enabling rapid dissemination and sharing of knowledge," explains Ed Seidel, assistant director for NSF's mathematical and physical sciences directorate.

The Microsemi Corp. TRRUST-Stor secure solid-state drive, shown above, was designed for data-intensive applications that require secure, anti-tamper data storage.
The Microsemi Corp. TRRUST-Stor secure solid-state drive, shown above, was designed for data-intensive applications that require secure, anti-tamper data storage.

Defense directives

Forces operating together in deployments throughout the world require real-time access to actionable intelligence, says a spokesperson at Lockheed Martin Corp. in Bethesda, Md. "The Department of Defense's response to this need is a global enterprise where military services, national agencies, and coalition partners can access time-sensitive intelligence, surveillance, and reconnaissance (ISR) data. This collaborative enterprise is the Distributed Common Ground System (DCGS)."

DCGS draws and correlates intelligence data from several integrated ISR sensors and systems into a comprehensive view of the battlefield. At the heart of DCGS is the DCGS integration backbone (DIB), which enables users worldwide to access data.

Lockheed Martin engineers are upgrading this open-standards software infrastructure that national agencies, coalition partners, and military services use to share intelligence. The $2.6 million contract from the DCGS Multi-Service Execution Team office will help "the DCGS team achieve its vision for maximizing the value of DCGS to commanders, war- fighters, and intelligence agencies," says Jim Quinn, vice president of C4ISR Systems for the Lockheed Martin Information Systems & Global Solutions Defense segment in Rockville, Md. "The latest DIB upgrade will enable our partners to securely share intelligence faster than ever before."

Lockheed Martin engineers are enhancing the latest version of the DIB with increased security filtering capabilities, an enhanced data ingest framework, increased ingest and query capabilities, and the Distributed Data Framework, which uses free and open-source software and supports the integration of new data sources, data transformation services, and user interfaces into the DIB.

High-level U.S. Army officials, during the 2012 Association of the U.S. Army (AUSA) conference in Washington, shared the organization's vision of the future or Army 2020, of which Army Intelligence 2020 is an integral part. Major efforts under Army Intelligence 2020 include: ubiquitous access to data via a cloud- enabled, interoperable DCGS-A with advanced analytics; optimized, multifunctional, and integrated sensors and collectors; and assured communications and robust processing power in theater via Relevant Intelligence to the Edge (RITE), also called Relevant ISR to the Edge.

Location is everything

"Sensors have become prolific," says Allan Cantle, chief technology officer of Interconnect Systems Inc. (ISI) and president of Nallatech in Camarillo, Calif. The speed, resolution, and data rates of sensors have increased, while geometries continue to shrink. "Invariably, the bandwidth of the communication channels is not keeping pace with these increasing amounts of data. This challenge has created the need to either process the data locally at the sensor or store the data for retrieval and postprocessing later after a mission is complete."

More sensors acquiring more data, in higher quality and quantity, are driving the need for both processing at the edge and data storage. "Whilst raw data storage is often essential in many military systems, the ever- growing data deluge is making it impractical to simply continue capturing and storing this data," Cantle notes. "Instead, it is becoming far more important to find techniques that can capture the important data and either store this significantly reduced data or make intelligent and immediate decisions based on the real-time data."

Aerospace and defense "storage systems are typically Flash memory based because of their ruggedness and speed compared to spinning hard drives," Cantle says. "These storage systems will often include both compression and encryption techniques to maximize storage density, as well as protect the data should the unmanned aerial vehicle (UAV) get into the wrong hands.

"Often, local data processing is favored over storage because the delays involved with data storage and postprocessing can often lead to stale intelligence, which becomes worthless," Cantle continues. "In these circumstances, data processing at the sensor will normally focus on data reduction techniques where the resulting bandwidth is low enough for transmission over the communication channels."

Alternatively, says Cantle, "the computational speed and low power of today's embedded computers are facilitating complete processing systems to reside on a UAV, including sensor processing, data fusion, and situational awareness. These types of systems provide a very rapid and intelligent identification of events that can be easily communicated with extremely low bandwidth requirements or can even allow a UAV to 'close the loop' and immediately react to a hostile event."

ISI and Nallatech, specializing in the miniaturization of electronics systems, unveiled the NSC series to enable high-speed processing at the edge using field-programmable gate array (FPGA) technology. "These systems allow customers to put 'Near Sensor Computing' (NSC) right next to tiny sensors, allowing for real time computing of high-speed data streaming from the sensor, without costly and power-hungry communication systems or bulky storage systems," Cantle explains. "Arrays of sensors with these NSC engines can then be used to build powerful and efficient distributed computing within the UAV.

"This type of NSC engine has been utilized in experimental sense-and-avoid systems, where a UAV must identify potential head-on collision events with other aircraft and react extremely quickly in order to avoid a collision," Cantle says.

The Lockheed Martin HC/MC-130J network-attached storage (NAS) system onboard the HC/MC-130J aircraft employs Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products.
The Lockheed Martin HC/MC-130J network-attached storage (NAS) system onboard the HC/MC-130J aircraft employs Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products.

Certified security

The need for data processing and data storage in theater is driving the demand for greater security onboard manned and unmanned vehicles.

"Data security is a major concern for mobile vehicle system architects," says Paul Davis, director of product management at Curtiss-Wright Controls Defense Solutions in Dayton, Ohio. "Quite often the storage modules are transported from a ground station to vehicle, and back again. During that transport process, critical data-at-rest must be protected."

Security based on Federal Information Processing Standard (FIPS) 140-2 is acceptable for most commercial aerospace and some military applications, Davis explains. Many companies, including Curtiss-Wright, provide certified storage products so customers do not have to worry about the certification process. Different levels of FIPS security exist, he cautions, so buyers should inquire about certification levels.

"For more critical DOD storage applications, National Security Agency (NSA) Type 1 security is often required. NSA certification is more difficult to achieve as it requires a DOD program sponsor and takes a long time to complete the certification process," Davis says. "In other countries, FIPS 140-2 certified encryption products are widely recognized and accepted. In the U.K., Communications Electronics Security Group (CESG) certification may be required when providing data storage to the Ministry of Defense (MOD). In any application, it is always a good idea to leverage existing certified products."

Less than zero

Engineers and end users have often sought removable hard disk drives or a method by which a disk drive and the data contained therein could be destroyed instantly or erased. Now, the aerospace and defense community is growing "more comfortable with the idea of zeroizing, or purging, the encryption key," Davis says.

"An AES256-bit key can be zero-ized in a fraction of a second," Davis adds. "It renders the data unclassified and safe for transport. It is estimated that it would take more than 1,056 years to break an AES256-bit key by brute-force methods. Since any data is time-sensitive, key zeroizing is a very effective method of protection."

Zeroizing is now often performed instead of executing a military secure-erase purge algorithm, Davis explains. "These older methods require overwriting of data several times with specific data patterns. With one terabyte of solid-state memory, such purge algorithms can take up to an hour to execute. In an emergency situation, like a UAV landing, an hour may be too long to secure the critical data."

Lockheed Martin Aeronautics engineers in Marietta, Ga., adopted Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products on the HC/MC-130J network-attached storage (NAS) system deployed on U.S. Air Force Air Combat Command MC-130J special mission aircraft and Air Force Special Operations Command HC-130J personnel recovery aircraft. The Lockheed Martin HC/MC-130J Super Hercules extended-range transport aircraft is designed for special missions, such as search and rescue.

"The data is secured with the FIPS 140-2 certified FSM product. AES256-bit encryption keys are used within inline media encryptors, which protect the data prior to storage on solid-state SATA drives," Davis describes. "These devices also decrypt the data when being retrieved by network clients. The AES256-bit keys can be zeroized in any one of three convenient ways: pushbutton, command, or discreet input."

Design considerations

Data security is a common concern, says Paul Ambuehl, storage technical lead at Extreme Engineering Solutions (X-ES) in Middleton, Wis. "Many customers look for solid-state drives (SSDs) that support military erase features and 256-bit AES encryption.

"Mil-aero system designers continue to prefer SSDs, as they are built to military-grade temperature, shock, and vibration specifications, conditions where spinning media would simply not perform," Ambuehl adds. "Mil-aero designers prefer SLC (single-level cell) flash technology, known for its extremely high reliability coupled with its exceptional read/write performance."

Ambuehl advises aerospace and defense engineers to pay attention to how the storage system or solution is constructed. Does the system use off-the-shelf SSDs? If so, what kind and what connectors are internal to the system? "If the system utilizes standard 2.5- or 1.8-inch SATA SSDs, it could be using standard SATA connectors. In some harsh environments, if a system is not properly designed, implemented, and tested, standard SATA connectors could be prone to failure due to excessive vibration," he advises.

When starting any system design, start with commercial off-the-shelf (COTS) secure storage products that are on the market already, recommends Curtiss-Wright's Davis. "The DOD end-user may be willing to consider a secure data-at-rest solution that exists in order to reduce technical and schedule risk. Such an existing solution can save considerable non-recurring engineering (NRE) and yet provide a robust solution.

In the U.S., NSA Type 1 certified solutions are frequently required, Davis says. "Solutions such as the Curtiss-Wright's Compact Network Storage (CNS) system combined with General Dynamics' DAR400E encryption device provide a certified Secret and Below (SAB) solution for data at rest. The CNS/DAR provides NSA Type 1 encryption now, instead of waiting years to accomplish."

Storage capacity is another important consideration, especially in ISR applications with sensors producing continuous streams of data, Davis says. "Applications often start by considering a minimal storage capacity. Quickly they realize that more and more devices in a network-centric storage system naturally grow to save more and more critical data."

Area density, continues to be one of the biggest challenges that the mil-aero community faces, says Ambuehl at X-ES. "System designers typically do not have a lot of size, weight, and power (SWaP) to sacrifice for storage, so it requires the SSD modules to pack a lot of flash into a small area. With storage requirements of one terabyte or more and with the physical size of flash components, this can prove to be challenging."

Engineers certainly must weigh all the storage options to find the optimal fit for each individual aerospace and defense project. Technology companies continue broaden the portfolio of products from which aerospace and defense engineers can select their optimal solutions, particularly in the areas of information management, sharing, and storage.

Bet on big data

"The Department of Defense is placing a big bet on Big Data. Help us reach our visions by submitting your ideas to the solicitations," says Zachary Lemnios, U.S. assistant secretary of defense for research and engineering. Learn about the latest opportunities at hwww.defenseinnovation, a new website designed to better communicate DOD investment priorities with performers in industry and academia.


Aitech Defense Systems
Barracuda Networks
Cavium Networks
Crystal Group
Curtiss-Wright Controls Defense Solutions
DRS Technologies
Elma Electronic
Extreme Engineering Solutions (X-ES)
Falcon Electronics
Galaxy Data Storage
General Micro Systems
Germane Systems
Interconnect Systems Inc.
L-3 Communications Targa Systems
Mentor Graphics
Phoenix International Systems Inc.
RightNow Technologies
SANBlaze Technology
Themis Computer

Get All the Military Aerospace Electronics News Delivered to Your Inbox or Your Mailbox

Subscribe to Military Aerospace Electronics Magazine or email newsletter today at no cost and receive the latest information on:

  • C4ISR
  • Cyber Security
  • Embedded Computing
  • Unmanned Vehicles

Get All the Military Aerospace Electronics News Delivered to Your Inbox or Your Mailbox

Subscribe to Military Aerospace Electronics Magazine or email newsletter today at no cost and receive the latest information on:

  • C4ISR
  • Cyber Security
  • Embedded Computing
  • Unmanned Vehicles

Military & Aerospace Photos

Most Popular Articles

Related Products

XPand6020 | Small Form Factor (SFF) System Featuring XPedite5205 Running Cisco IOS® and XPedite7450

The XPand6020 is a Small Form Factor (SFF) system that features an XPedite5205, which runs Cisco ...

XCalibur4401 | Intel® Core™ i7 Processor-Based Conduction- or Air-Cooled 6U CompactPCI Single Board Computer

The XCalibur4401 is a high-performance, 6U CompactPCI, multiprocessing, single board computer tha...

XPedite6100 | Freescale Quad-Core, T1042 Processor-Based, Air-Cooled XMC/PrPMC with Four Gigabit Ethernet Ports

The XPedite6100 is a high-performance, XMC/PrPMC, single board computer supporting a Freescale Qo...

XPort3200 | Freescale QorIQ P1020 Processor-Based Conduction- or Air-Cooled XMC/PMC IEEE 1588v2 Grandmaster Clock Module

The XPort3200 is a rugged, IEEE 1588v2 Precision Time Protocol (PTP) ordinary clock with grandmas...

XChange3100 | 6U VPX 10 Gigabit Ethernet Switch with Optional Layer 2 Switching and Layer 3 Routing Management Support

The XChange3100 is a conduction- or air-cooled, 6U OpenVPX™ 10 Gigabit Ethernet switch module. Th...

XPand4208 | Flight Qualified Intel® Core™ i7-Based Multiprocessor 3U OpenVPX™ COTS System

The XPand4208 is a high-performance, expandable, computing and networking platform for environmen...

XPedite7472 | Intel® Core™ i7 Processor-Based Conduction- or Air-Cooled 3U VPX-REDI SBC with SecureCOTS™

The XPedite7472 is a secure and high-performance, 3U VPX-REDI, single board computer based on the...

XCalibur4444 | Intel® Core™ i7 Processor-Based Conduction-Cooled 6U VPX Module

The XCalibur4444 is a high-performance, multiprocessing, 6U VPX, single board computer that is id...

XPedite7470 | Intel® Core™ i7 Processor-Based Conduction- or Air-Cooled 3U VPX-REDI SBC

The XPedite7470 is a high-performance, low-power, 3U VPX-REDI, single board computer based on the...

XPort5005 | XMC Form Factor PCIe Mini Card Carrier Board

The XPort5005 is an XMC module that can be quickly configured to support a platform’s specific I/...

Related Companies

General Atomics Aeronautical Systems Inc

GA-ASI is a leading manufacturer of proven, reliable Remotely Piloted Aircraft (RPA) systems, radars, and electro-opt...

DiCon Fiberoptics Inc

Offers fiber optic switches, tunable filters, and VOAs. Founded in 1986, the company is a US based, AS9100 certified,...

Curtiss-Wright Defense Solutions

About Curtiss-Wright Defense Solutions Curtiss-Wright Defense Solutions (CWDS) is a long established techno...


Offers complete solutions for embedded software developers with a focus on mission- and safety-critical applications....

Harris Corporation

Harris provides advanced, technology-based solutions that solve government and commercial customers' mission critical...

United Electronic Industries Inc

UEI is a leader in the PC/Ethernet data acquisition and control, Data Logger/Recorder and Programmable Automation Con...

Crane Aerospace & Electronics

When failure is NOT an option...rely on Crane Aerospace & Electronics. We supply high-density, high-reliability c...


Signal integrity leaders and preferred vertically integrated manufacturer of high-performance electrical and electron...

AcQ Inducom

Develops and produces non-certified and certified high-tech modular hardware- and software solutions for on-board and...

Advanced Conversion Technology Inc

ACT designs and manufactures, since 1981, an extensive range of AC-DC and DC-DC power supplies (switching, linear, ra...
Wire News provided by   

Press Releases

Model INCX-4001

The INCX-4001 consists of a high quality audio transceiver specifically designed to implement a complete fiber optic intercom.

Model PS-1210

The PS-1210 is a 1A, 12VDC stand-alone or rack mountable non-switcher (no RF noise) power supply.

Model OS-3121

Optical switches are utilized to disconnect, bypass and reroute fiber optic communications. All of these optical switches are purely optical path, there is no optical to e...


New Design Tools That Help You Develop Radar That Sees the Un-seeable and Detects the Undetectable

Xilinx EW/ISR System Architect, Luke Miller, has new tricks and he’s going to tell you all about them in a new Xilinx Webinar—for free. His Webinar will cover new ways to implement Radar functions including ...
Sponsored by:

All Access Sponsors

Mil & Aero Magazine

June 2015
Volume 26, Issue 6

Download Our Apps




Follow Us On...


Military & Aerospace Electronics

Weekly newsletter covering technical content, breaking news and product information

Cyber Security

Monthly newsletter covering cyber warfare, cyber security, information warfare, and information security technologies, products, contracts, and procurement opportunities

Defense Executive

Monthly newsletter covering business news and strategic insights for executive managers

Electronic Warfare

Quarterly newsletter covering technologies and applications in electronic warfare, cyber warfare, optical warfare, and spectrum warfare.

Embedded Computing Report

Monthly newsletter covering news on embedded computing in aerospace, defense and industrial-rugged applications

Unmanned Vehicles

Monthly newsletter covering news updates for designers of unmanned vehicles