Dealing with Big Data

Aerospace and defense applications call for innovative information storage solutions delivering increased capacity, security, and speed.

Current and future missions increasingly rely on the timely and reliable capture, processing, storage, and delivery of accurate, actionable information. This is true of myriad civil and defense missions, whether on land, at sea, or in air or space.

Modern sensor, camera, communications, and computer systems deliver a wealth and variety of mission-critical data to today's warfighter. The availability and use of this data contributes to increased situational awareness, survivability, and mission effectiveness; yet, managing-and, in the process, gleaning valuable information from-this vast volume of data is a growing challenge.

"Demands for actionable information have spiked as warfighters at every level are called upon to make well-informed decisions," says a representative of the Defense Advanced Research Projects Agency (DARPA) in Arlington, Va. "To maximize mission effectiveness and enhance national security, the [U.S.] Department of Defense (DOD) is now challenged to more efficiently fuse, analyze, and disseminate the massive volumes of data this network produces."

A majority of currently fielded electronics and software cannot easily be scaled to meet this data deluge. Rather, "the volume and characteristics of the data, and the range of applications for data analysis, require a fundamentally new approach to data science, analysis, and incorporation into mission planning on timelines consistent with operational tempo," the representative adds.

Government, industry, and academia are working together to solve the challenges of and exploit the tremendous value of "Big Data."

Curtiss-Wright Vortex Compact Network Storage products are aboard U.S. Air Force HC/MC-130J special operations aircraft.
Curtiss-Wright Vortex Compact Network Storage products are aboard U.S. Air Force HC/MC-130J special operations aircraft.

Big data deluge

The Obama Administration in March 2012 launched the Big Data Research and Development Initiative. The initiative is intended to advance state-of-the-art core technologies needed to collect, store, preserve, manage, analyze, and share huge quantities of data; harness these technologies to accelerate the pace of discovery in science and engineering, strengthen national security, and transform teaching and learning; and expand the workforce needed to develop and use Big Data technologies.

"We intend to change the game and plan to be the first to leverage Big Data across the full scope of military operations in new and unconventional ways," says Zachary Lemnios, U.S. assistant secretary of defense for research and engineering. "We are within sight of a new generation of systems that understand and interpret the real world with computer speed, computer precision, and human agility. These systems will be central to helping our commanders and analysts make sense of the huge volumes of data our military sensors collect at speeds 100X faster than today."

The project has the potential to solve key challenges and move government plans ahead for Big Data. "By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help solve some the nation's most press- ing challenges," says John P. Holdren, assistant to the president and director of the White House Office of Science and Technology Policy (OSTP). "Six Federal departments and agencies announced more than $200 million in new commitments that promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data."

The DOD will invest $250 million annually under the initiative, including $60 million dedicated to new research projects. DOD Big Data projects focus on three areas: data to decisions, autonomy, and human systems. Data-to-decisions investments will develop computational techniques and software tools for analyzing large volumes of data. Autonomy investments center on support tools that can recognize trends, adapt to real-world conditions, and succeed in complex, dynamic environments without relying on human intervention. Human-systems investments will advance human-machine interfaces to enable collaboration for operations and for training.

Banding together

Some companies already are sponsoring Big Data-related competitions, and providing funding for university research, according to Tom Kalil, deputy director for policy, White House OSTP. "Universities are beginning to create new courses and entire courses of study to prepare the next generation of 'data scientists.'"

DARPA officials, recognizing the challenges associated with large volumes of data, started the XDATA program in concert with the launch of the Big Data Initiative last spring. The goal is to develop computational techniques and software tools for processing and analyzing the vast amount of mission-oriented information for defense activities.

XDATA also aims to address the need for: scalable algorithms for processing and visualization of imperfect and incomplete data, human-computer interaction tools that could be easily customized for different missions, and open-source software toolkits to enable collaboration among the applied mathematics, computer science, and data visualization communities.

"It's a great time to leverage recent commercial and academic advances in processing large amounts of data for analysis," says Chris White, DARPA program manager. "We are calling on all technical communities with expertise in this area to help us ensure our men and women in uniform have the benefit of the best information we can provide."

The U.S. National Science Foundation (NSF), in collaboration with the National Institutes of Health (NIH), is seeking proposals under a Big Data solicitation: "Core Techniques and Technologies for Advancing Big Data Science & Engineering." It will fund research to develop and evaluate new algorithms, statistical methods, technologies, and tools for improved data collection and management, data analytics, and e-science collaboration environments.

"NSF is developing a bold and comprehensive approach for this new data-centric world, from fundamental mathematical, statistical, and computational approaches needed to understand the data, to infrastructure at a national and international level needed to support and serve our communities, to policy enabling rapid dissemination and sharing of knowledge," explains Ed Seidel, assistant director for NSF's mathematical and physical sciences directorate.

The Microsemi Corp. TRRUST-Stor secure solid-state drive, shown above, was designed for data-intensive applications that require secure, anti-tamper data storage.
The Microsemi Corp. TRRUST-Stor secure solid-state drive, shown above, was designed for data-intensive applications that require secure, anti-tamper data storage.

Defense directives

Forces operating together in deployments throughout the world require real-time access to actionable intelligence, says a spokesperson at Lockheed Martin Corp. in Bethesda, Md. "The Department of Defense's response to this need is a global enterprise where military services, national agencies, and coalition partners can access time-sensitive intelligence, surveillance, and reconnaissance (ISR) data. This collaborative enterprise is the Distributed Common Ground System (DCGS)."

DCGS draws and correlates intelligence data from several integrated ISR sensors and systems into a comprehensive view of the battlefield. At the heart of DCGS is the DCGS integration backbone (DIB), which enables users worldwide to access data.

Lockheed Martin engineers are upgrading this open-standards software infrastructure that national agencies, coalition partners, and military services use to share intelligence. The $2.6 million contract from the DCGS Multi-Service Execution Team office will help "the DCGS team achieve its vision for maximizing the value of DCGS to commanders, war- fighters, and intelligence agencies," says Jim Quinn, vice president of C4ISR Systems for the Lockheed Martin Information Systems & Global Solutions Defense segment in Rockville, Md. "The latest DIB upgrade will enable our partners to securely share intelligence faster than ever before."

Lockheed Martin engineers are enhancing the latest version of the DIB with increased security filtering capabilities, an enhanced data ingest framework, increased ingest and query capabilities, and the Distributed Data Framework, which uses free and open-source software and supports the integration of new data sources, data transformation services, and user interfaces into the DIB.

High-level U.S. Army officials, during the 2012 Association of the U.S. Army (AUSA) conference in Washington, shared the organization's vision of the future or Army 2020, of which Army Intelligence 2020 is an integral part. Major efforts under Army Intelligence 2020 include: ubiquitous access to data via a cloud- enabled, interoperable DCGS-A with advanced analytics; optimized, multifunctional, and integrated sensors and collectors; and assured communications and robust processing power in theater via Relevant Intelligence to the Edge (RITE), also called Relevant ISR to the Edge.

Location is everything

"Sensors have become prolific," says Allan Cantle, chief technology officer of Interconnect Systems Inc. (ISI) and president of Nallatech in Camarillo, Calif. The speed, resolution, and data rates of sensors have increased, while geometries continue to shrink. "Invariably, the bandwidth of the communication channels is not keeping pace with these increasing amounts of data. This challenge has created the need to either process the data locally at the sensor or store the data for retrieval and postprocessing later after a mission is complete."

More sensors acquiring more data, in higher quality and quantity, are driving the need for both processing at the edge and data storage. "Whilst raw data storage is often essential in many military systems, the ever- growing data deluge is making it impractical to simply continue capturing and storing this data," Cantle notes. "Instead, it is becoming far more important to find techniques that can capture the important data and either store this significantly reduced data or make intelligent and immediate decisions based on the real-time data."

Aerospace and defense "storage systems are typically Flash memory based because of their ruggedness and speed compared to spinning hard drives," Cantle says. "These storage systems will often include both compression and encryption techniques to maximize storage density, as well as protect the data should the unmanned aerial vehicle (UAV) get into the wrong hands.

"Often, local data processing is favored over storage because the delays involved with data storage and postprocessing can often lead to stale intelligence, which becomes worthless," Cantle continues. "In these circumstances, data processing at the sensor will normally focus on data reduction techniques where the resulting bandwidth is low enough for transmission over the communication channels."

Alternatively, says Cantle, "the computational speed and low power of today's embedded computers are facilitating complete processing systems to reside on a UAV, including sensor processing, data fusion, and situational awareness. These types of systems provide a very rapid and intelligent identification of events that can be easily communicated with extremely low bandwidth requirements or can even allow a UAV to 'close the loop' and immediately react to a hostile event."

ISI and Nallatech, specializing in the miniaturization of electronics systems, unveiled the NSC series to enable high-speed processing at the edge using field-programmable gate array (FPGA) technology. "These systems allow customers to put 'Near Sensor Computing' (NSC) right next to tiny sensors, allowing for real time computing of high-speed data streaming from the sensor, without costly and power-hungry communication systems or bulky storage systems," Cantle explains. "Arrays of sensors with these NSC engines can then be used to build powerful and efficient distributed computing within the UAV.

"This type of NSC engine has been utilized in experimental sense-and-avoid systems, where a UAV must identify potential head-on collision events with other aircraft and react extremely quickly in order to avoid a collision," Cantle says.

The Lockheed Martin HC/MC-130J network-attached storage (NAS) system onboard the HC/MC-130J aircraft employs Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products.
The Lockheed Martin HC/MC-130J network-attached storage (NAS) system onboard the HC/MC-130J aircraft employs Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products.

Certified security

The need for data processing and data storage in theater is driving the demand for greater security onboard manned and unmanned vehicles.

"Data security is a major concern for mobile vehicle system architects," says Paul Davis, director of product management at Curtiss-Wright Controls Defense Solutions in Dayton, Ohio. "Quite often the storage modules are transported from a ground station to vehicle, and back again. During that transport process, critical data-at-rest must be protected."

Security based on Federal Information Processing Standard (FIPS) 140-2 is acceptable for most commercial aerospace and some military applications, Davis explains. Many companies, including Curtiss-Wright, provide certified storage products so customers do not have to worry about the certification process. Different levels of FIPS security exist, he cautions, so buyers should inquire about certification levels.

"For more critical DOD storage applications, National Security Agency (NSA) Type 1 security is often required. NSA certification is more difficult to achieve as it requires a DOD program sponsor and takes a long time to complete the certification process," Davis says. "In other countries, FIPS 140-2 certified encryption products are widely recognized and accepted. In the U.K., Communications Electronics Security Group (CESG) certification may be required when providing data storage to the Ministry of Defense (MOD). In any application, it is always a good idea to leverage existing certified products."

Less than zero

Engineers and end users have often sought removable hard disk drives or a method by which a disk drive and the data contained therein could be destroyed instantly or erased. Now, the aerospace and defense community is growing "more comfortable with the idea of zeroizing, or purging, the encryption key," Davis says.

"An AES256-bit key can be zero-ized in a fraction of a second," Davis adds. "It renders the data unclassified and safe for transport. It is estimated that it would take more than 1,056 years to break an AES256-bit key by brute-force methods. Since any data is time-sensitive, key zeroizing is a very effective method of protection."

Zeroizing is now often performed instead of executing a military secure-erase purge algorithm, Davis explains. "These older methods require overwriting of data several times with specific data patterns. With one terabyte of solid-state memory, such purge algorithms can take up to an hour to execute. In an emergency situation, like a UAV landing, an hour may be too long to secure the critical data."

Lockheed Martin Aeronautics engineers in Marietta, Ga., adopted Curtiss-Wright's Vortex Compact Network Storage (CNS) subsystems and Flash Storage Module (FSM) products on the HC/MC-130J network-attached storage (NAS) system deployed on U.S. Air Force Air Combat Command MC-130J special mission aircraft and Air Force Special Operations Command HC-130J personnel recovery aircraft. The Lockheed Martin HC/MC-130J Super Hercules extended-range transport aircraft is designed for special missions, such as search and rescue.

"The data is secured with the FIPS 140-2 certified FSM product. AES256-bit encryption keys are used within inline media encryptors, which protect the data prior to storage on solid-state SATA drives," Davis describes. "These devices also decrypt the data when being retrieved by network clients. The AES256-bit keys can be zeroized in any one of three convenient ways: pushbutton, command, or discreet input."

Design considerations

Data security is a common concern, says Paul Ambuehl, storage technical lead at Extreme Engineering Solutions (X-ES) in Middleton, Wis. "Many customers look for solid-state drives (SSDs) that support military erase features and 256-bit AES encryption.

"Mil-aero system designers continue to prefer SSDs, as they are built to military-grade temperature, shock, and vibration specifications, conditions where spinning media would simply not perform," Ambuehl adds. "Mil-aero designers prefer SLC (single-level cell) flash technology, known for its extremely high reliability coupled with its exceptional read/write performance."

Ambuehl advises aerospace and defense engineers to pay attention to how the storage system or solution is constructed. Does the system use off-the-shelf SSDs? If so, what kind and what connectors are internal to the system? "If the system utilizes standard 2.5- or 1.8-inch SATA SSDs, it could be using standard SATA connectors. In some harsh environments, if a system is not properly designed, implemented, and tested, standard SATA connectors could be prone to failure due to excessive vibration," he advises.

When starting any system design, start with commercial off-the-shelf (COTS) secure storage products that are on the market already, recommends Curtiss-Wright's Davis. "The DOD end-user may be willing to consider a secure data-at-rest solution that exists in order to reduce technical and schedule risk. Such an existing solution can save considerable non-recurring engineering (NRE) and yet provide a robust solution.

In the U.S., NSA Type 1 certified solutions are frequently required, Davis says. "Solutions such as the Curtiss-Wright's Compact Network Storage (CNS) system combined with General Dynamics' DAR400E encryption device provide a certified Secret and Below (SAB) solution for data at rest. The CNS/DAR provides NSA Type 1 encryption now, instead of waiting years to accomplish."

Storage capacity is another important consideration, especially in ISR applications with sensors producing continuous streams of data, Davis says. "Applications often start by considering a minimal storage capacity. Quickly they realize that more and more devices in a network-centric storage system naturally grow to save more and more critical data."

Area density, continues to be one of the biggest challenges that the mil-aero community faces, says Ambuehl at X-ES. "System designers typically do not have a lot of size, weight, and power (SWaP) to sacrifice for storage, so it requires the SSD modules to pack a lot of flash into a small area. With storage requirements of one terabyte or more and with the physical size of flash components, this can prove to be challenging."

Engineers certainly must weigh all the storage options to find the optimal fit for each individual aerospace and defense project. Technology companies continue broaden the portfolio of products from which aerospace and defense engineers can select their optimal solutions, particularly in the areas of information management, sharing, and storage.

Bet on big data

"The Department of Defense is placing a big bet on Big Data. Help us reach our visions by submitting your ideas to the solicitations," says Zachary Lemnios, U.S. assistant secretary of defense for research and engineering. Learn about the latest opportunities at hwww.defenseinnovation, a new website designed to better communicate DOD investment priorities with performers in industry and academia.


Aitech Defense Systems
Barracuda Networks
Cavium Networks
Crystal Group
Curtiss-Wright Controls Defense Solutions
DRS Technologies
Elma Electronic
Extreme Engineering Solutions (X-ES)
Falcon Electronics
Galaxy Data Storage
General Micro Systems
Germane Systems
Interconnect Systems Inc.
L-3 Communications Targa Systems
Mentor Graphics
Phoenix International Systems Inc.
RightNow Technologies
SANBlaze Technology
Themis Computer

Military & Aerospace Photos

Most Popular Articles

Related Products

SBC-K7 Embedded PC for Instrumentation and Control

The SBC-K7 is an ideal platform for embedded instrumentation that combines an Atom PC running Win...

Rugged Mobile Communications Server

Advanced communications server designed to be deployed in environments where it needs to meet cer...

RR2P Removable Canister RAID System

Transportable data storage for mobile field use aboard planes, ships and ground transport. 2U, du...

P317xx Portable Computer System w/ Three 17" LCDs

The P317xx portable " lunchbox" system is unique in offering three 17" 1280x1024 resolution LCD d...

M1U20xx 1U Military-Grade Computer System

The M1U20xx Military-Grade Rack Mount System is a reliable high-performance mil-spec 1U solution ...

M4U20xx 4U Rackmount Military-Grade Computer

A 4U rugged military-grade computer offering the latest long-availability high-performance mother...

TFX-19 Trifold Rackmount LCD

The TFX1-19 is a rugged military grade high performance 2U rackmount LCD panel display offering t...

CCXR-17 Slideways Side Access Rackmount LCD Keyboard

The CCXR-17 is a unique Side Access space-efficient military-grade high-performance 1U rackmount ...

Sensors Unlimited GA1280JSX High Resolution, Mil-Rugged, Extended High Sensitivity InGaAs SWIR Camera

Manufactures a high resolution, mil-rugged, extended high-sensitivity 0.9 Mpixel InGaAs SWIR came...

High Temperature Resistant Linear Position Sensors Perform in Military and Defense Applications with High Accuracy

Macro Sensors has designed a series of high temperature and rugged LVDT linear position sensors t...

Related Companies

Innovative Integration

  Since 1988, Innovative Integration has grown to become one of the world's leading suppliers of DSP and data ac...

Elma Electronic Inc

Who we are...   About Elma Electronic Systems   The Systems division of Elma Electronic Inc. supplies the

Winchester Systems Inc

At its founding in 1981, Winchester Systems introduced its first 5 MB disk system for Intel development system users....

Extreme Engineering Solutions Inc (X-ES)

 Extreme Engineering Solutions, Inc. (X-ES) is a leader in the design, manufacture, and support of standard and ...

American Infrared Solutions (AIRS)

American Infrared Solutions

Raycom Electronics

Raycom Electronics, Inc., a subsidiary of Electro Technik Industries (ETI), was incorporated in 1965.  The compa...

GE Intelligent Platforms

GE Intelligent Platforms is a leading global provider of embedded computing systems and modules for a wide range of a...

Trenton Systems Inc

Designs and manufactures rugged rackmount computers and MIL-STD-810 systems using both standard COTS as well as custo...

Intersil Corp

Intersil is leveraging a deep portfolio of intellectual property, a history of design and process innovation and a sy...
Wire News provided by   

Press Releases

One Component, Snap Cure Epoxy Features High Strength Properties

Suitable for a variety of applications in the electronic, aerospace and OEM industries, Master Bond EP3SP5F...

One Part Epoxy Resists up to 500°F and Meets NASA Low Outgassing Specifications

Master Bond Supreme 12AOHT-LO is a one component epoxy for a variety of bonding and sealing applications in...

Low Viscosity, One Part Cyanoacrylate Is Non-Toxic and Meets ISO 10993-5 Specifications

Master Bond MB250NT is widely used for a variety of applications ranging from repair to high speed producti...

Thermally Conductive, Two Component Epoxy Passes USP Class VI Tests and ISO 10993-5 Specifications

With biocompatibility and cytotoxicity certifications, Master Bond EP21AOLV-2Med is often selected for bond...


Curtiss-Wright Corporation’s Defense Solutions division applauds Northrop Grumman Corporation (NYSE: NOC) o...

Curtiss-Wright’s New Rugged Mobile IP Router Subsystem Features an Integrated Cisco® 5915 ESR Router

Curtiss-Wright Corporation today announced that its Defense Solutions division, a Cisco® Systems Solution T...

VICTORY Shared Processing, Fire Control Computer, and Switch for Ground Vehicles Introduced by Curtiss-Wright

Curtiss-Wright Corporation today announced that its Defense Solutions division has introduced a new fully i...

GE Announces First Sub-Credit Card-Sized Multi-Function High Definition (HD) Video Tracker

HUNTSVILLE, AL.— OCTOBER 13, 2014—GE’s Intelligent Platforms business today announced at AUSA (October 13-...


Meeting the Gen3 backplane challenge with OpenVPX and COTS

Tight Pentagon budgets mean military systems must stay in the field for longer than ever before. This doesn't mean obsolete technology, however. Today's military electronics are being upgraded constantly, an...
Sponsored by:

Design Strategy Considerations for DO-178C Certified Multi-core Systems

Join Wind River to learn how system architecture and design choices can minimize your DO-178C certification challenges.

Sponsored by:

Flying, Sailing or Driving - The Rugged, Embedded Intel-based Server that goes where you need it!Flying Sailing or Driving

Leveraging the power of server-class processors is no longer relegated to the confines of data centers. Through several innovations, Mercury Systems has ruggedized Intel’s server-class chips for deployment. ...
Sponsored by:

All Access Sponsors

Mil & Aero Magazine

April 2015
Volume 26, Issue 4

Download Our Apps




Follow Us On...


Military & Aerospace Electronics

Weekly newsletter covering technical content, breaking news and product information

Cyber Security

Monthly newsletter covering cyber warfare, cyber security, information warfare, and information security technologies, products, contracts, and procurement opportunities

Defense Executive

Monthly newsletter covering business news and strategic insights for executive managers

Electronic Warfare

Quarterly newsletter covering technologies and applications in electronic warfare, cyber warfare, optical warfare, and spectrum warfare.

Embedded Computing Report

Monthly newsletter covering news on embedded computing in aerospace, defense and industrial-rugged applications

Unmanned Vehicles

Monthly newsletter covering news updates for designers of unmanned vehicles