DARPA to develop real-time intelligence processor to uncover patterns in vast data

U.S. military researchers are asking for industry's help in developing a new data processor to help intelligence analysts understand relationships in vast data streams from cameras, social media, sensor feeds, and scientific data.

Sep 15th, 2016
1609mae News Darpa

ARLINGTON, Va. - U.S. military researchers are asking for industry's help in developing a new data processor to help intelligence analysts understand relationships in vast data streams from cameras, social media, sensor feeds, and scientific data.

Officials of the U.S. Defense Advanced Research Projects Agency (DARPA) in Arlington, Va., have issued a presolicitation (DARPA-BAA- 16-52) for the Hierarchical Identify Verify Exploit (HIVE) project. The HIVE program seeks to develop a generic and scalable graph processor that specializes in processing sparse graph primitives, and achieves 1000-times improvement in processing efficiency over standard processors.

Military researchers are trying to capitalize on the best of industry's high-performance computing technologies to uncover patterns in vast amounts of data.

This capability will help intelligence analysts discover the relationships between events as they unfold in the field, rather than relying on forensic analysis in data centers, DARPA officials say.

The program will develop chip prototypes and software tools to support programming the new hardware, as well as design a system architecture to support efficient multi-node scaling.

Today large amounts of data come from sources like social media, cameras, other kinds of sensor feeds, and scientific data. Graph analytics has emerged as a way to understand the relationships between these heterogeneous types of data to enable analysts to draw conclusions from the patterns in the data and to answer previously unthinkable questions, DARPA experts explain.

Yet analysts might be able to understand a more complete picture of the problem by understanding the complex relationships between different data feeds.

Today most graph analytics happens in large data centers on large cached or static data sets. This requires massive amounts of processing power - particularly for "needle-in-the-haystack" types of problems. Moreover, the nature of the graph can be very sparse, as analysts are not clear on the number of relationships between entities.

Analysts also need to make decisions in real time. To do this, they must understand how relationships in the graph evolve over time. The graph must update at the speed of incoming data, not as an offline process, because the graph must develop and change in real time.

Trying to analyze the graph with standard processors is extremely inefficient because sparse data must be processed in real time, DARPA officials say.

Graph analytics shifts the processing workload to locating the information and moving the data; only 4 percent of processing time and power goes to the overall effort.

Such inefficiency either limits the size of the graph to what the chip can hold, or requires an extremely large cluster of computers. Instead, the HIVE program seeks to create a special processor that works on graph analytics 1000 times faster than can today's standard processors.

The program will focus on improving the efficiency of random access memory transactions to limit data movement, efficient parallelism to improve scalability, and new accelerators designed specifically for graph computation.

The HIVE program has three phases. First, an architectural phase will develop new memory controllers, new accelerators based on graph primitives, new data flow models, new data mapping tools, and new middleware to enable seamless transition of existing graph algorithms onto the new hardware.

Second, a prototyping phase will demonstrate these new technologies on military applications. Third, a fabrication phase will demonstrate the scalable performance of a 16-node system of custom graph processors for accelerating the most demanding military analytics applications.

The project has three technical areas: graph analytics processor, graph analytics tool kits, and system evaluator. The graph analytics processor (technical area one) will design a new chip architecture from scratch, focusing on the memory wall and on parallelization of multinode systems. The memory wall will create new memory architectures that allow for non-uniform memory access (NUMA).

Parallelization, meanwhile, will enable machines to work closely together, rather than working in parallel but running independently. Technical area one, in essence, has to move from today's single instruction multi-data (SIMD) world to one that allows for multiple instruction multi-data (MIMD) execution, DARPA officials say.

This component of the HIVE program will create an accelerator architecture and processor pipeline that processes graph primitives in a native sparse matrix format; develop a chip architecture that moves data quickly and efficiently from memory or I/Os to the accelerators; and develop an external memory controller that uses data mapping tools to handle random and sequential memory accesses on memory transfers as small as 8 to 32 bytes.

The prototype phase will develop the new chip architecture on a printed circuit board and low-level software that will emulate the future chip.

Graph analytics toolkits (technical area two) aims to develop the fundamental software technologies to translate existing graph algorithms into the new hardware by developing micro-code to match the microarchitecture of the new chips. Micro-code must support the data format and graph primitives of existing graph algorithms and not force them to re-write their algorithms.

System evaluator (technical area three) aims to identify and develop static and streaming graph analytics to solve five types of problem areas: anomaly detection, domain specific search, dependency mapping, N-x contingency analysis, and causal modeling of events.

Companies interested should submit proposals no later than 19 Oct. 2016 to the DARPA website at https://baa.darpa.mil. E-mail questions or concerns to DARPA's Trung Tran, the HIVE program manager, at DARPA-BAA-16-52@darpa.mil.

More information is online at www.fbo.gov/spg/ODA/DARPA/CMO/DARPA-BAA-16-52/listing.html.

More in Computers