Aerospace and defense engineers adopt software development, verification, and test tools with an eye toward cost, efficiency, and scalability.
As manned and unmanned military platforms continue to increase in complexity and functionality, the lines of software code in a system grow exponentially. This trend, in turn, cultivates the need for efficient and effective mission- and safety-critical software development, verification, and test tools.
"The combination of development, verification, and certification activities, tools, and artifacts of safety- critical software is the greatest challenge for our customers," says Chris Murray, vice president of business development at LDRA in Wirral, England. "Most clients in the mission- and safety-critical markets would prefer to solve as many of these challenges as possible with as few solutions as necessary. Essentially, they're looking for more value and capabilities from a smaller number of vendor partners."
Customers continue asking LDRA to extend its solutions beyond verification, code coverage, and analysis to requirements traceability, integrations with modeling, requirements engineering, continuous integration, and even product lifecycle management (PLM) tools. "Most recently we have added a certification services team to provide the underlying processes and guidance to tie everything together," Murray explains.
|The TBmanager interface to the LDRA tool suite shows the verification of object code, the fulfilment of requirements, and the pass/fail of tests-all requirements of DO-178C Level A.|
"The beauty of having all this capability from a single vendor is not just the ease of purchase, but that the tools work together easily and consistently," Hicken adds. "A developer moving from a Java project to C++ doesn't have any learning curve. In addition, many of the tools have a synergistic effect; for example, you can connect penetration testing directly to static analysis and runtime detection, so that when a pen test succeeds at breaching your system, you know which lines of code are at risk, as well as the root cause of which static analysis rules have been violated. It's really fulfilling the promise of the Development Testing dream."
Capability and scalability
"From the management side, scalable, workable project management takes you from system requirements to the actual tasks that developers do," Hicken says. "It's coordinated with the reporting system so that a manager can quickly tell which projects are tracking well not only from a finished code perspective, but also from a quality and security perspective. The reporting system helps control ad-hoc tool usage by allowing you to apply a consistent policy across multiple tools from different vendors, produce useful aggregated reports, and inject tasks into the developer IDE (integrated development environment)."
Such workflow integration further enables change-based testing, which is helpful to aerospace and defense engineers. "If a requirement or code or test changes, we can tell you which tests need to be rerun based on which ones are actually affected. We also provide full traceability, so you can easily handle audits; you know what was tested, by whom, when, what the results were, who signed off on it, and all the other information that is necessary for safety and compliance environments."
When selecting a tool, Hicken cautions against doing "a simple bake-off of features. Rather, take a look at what you actually need, and find out which tool does that well. You also need to consider cost, updates, maturity, support, partnership with and responsiveness of the vendor, and how it fits in your actual workflow," he says. "Having a bad workflow can kill a tool, no matter how much money was spent on it and how great its results are."
In today's climate of tightening budgets, cost is a key concern. "The most pressing issue related to mission- and safety-critical software is cost containment over the entire process," says George Romanski, CEO of Verocel Inc. in Westford, Mass.
Romanski uses the example of unmanned aerial vehicles (UAVs), which were developed and deployed quickly. "A lot of the testing and verification was performed at the system level, and software grew to accommodate the complex functionality that was required," he says. "To support the short deployment deadlines, the software just 'grew' without specific certification requirements-those would be added later. This example is typical, and software is very often developed to a set of specifications, and 'understood' intended behavior. The system is then tried and debugged until it works, then the certification process is started using reverse engineering. In a recent study, it was found that over 75 percent of safety- critical system used reverse engineering to some extent."
At some time, these systems with a huge code base are considered ready for certification, Romanski continues. "Estimates are developed and panic sets in. The certification effort can be as high as the development effort, depending on the design assurance level. Budget and schedule pressures mount to get the certification done. Teams are put in place to develop verification evidence, which means a large number of tests. For the most-critical software, there are typically more than three lines of test for every line of source code. The tests are used to verify the behavior, and their completeness is measured to assess how much of the code they actually cover," he says.
On a project that is behind schedule and over budget, tremendous temptation exists to reuse tests rather than developing new ones, Romanski says. "By using a coverage monitoring tool and just switching the system on, some systems may show that 50 percent of the code has been executed and people think they are 50 percent done." Yet, RTCA DO-178C has clarified the intent, and this type of "accidental" coverage is not accepted, he says.
"Tests based on low-level requirements must be used to measure and test the intent expressed in the requirements for the individual functions. Any additional code that is not executed by these tests must be identified and analyzed to show that it does not introduce additional and unexpected behavior," Romanski explains. "Companies that have managed to obtain certification credit using the high-level test coverage will find it more difficult in the future as DO-178C becomes the primary safety standard version."
Coupling and modularity
Low-level or module testing often is performed by isolating a source- level function and using tools or additional hand-written test code to produce a test environment. The test environment is a simulation of the real interfaces between the modules, Romanski explains. "This is a good approach for testing large systems, because we can test them a module at a time and assemble all the evidence. What this technique does not do is show that the coupling between the modules has been verified.
"Prior to DO-178C, there was some ambiguity as to what coupling really meant," Romanski says. "Some people interpreted this as an analysis of the use of shared data between modules. They would examine this by reviewing the code. With the publication of DO-178C, this coupling objective has been clarified, and it is now an objective to be verified by test. Unless projects are prepared for this, it could be one of the biggest cost drivers in new certifications."
The approach taken at Verocel always has been to test what flies, Romanski insists. "If a system is made up of many modules, then the modules are linked to form one integrated image and all tests are performed on that integrated image. Even though high-level and low-level requirement-based tests are used, the final image that flies is tested," he says. To accomplish this, Verocel developed test harnesses and coverage tools that do not use any additional hardware and work on the actual target computer.
"VerOCode is a coverage tool that measures coverage, module by module, on the target computer at the executable instruction level without instrumenting the code and without use of special debug probes," Romanski says. Often these debug probes only appear in development hardware; when they use final conformed hardware, the boxes are sealed and debug ports are not visible. "The same tests are run in the same manner with and without instrumentation. We check that the results are correct and that they are the same," he says.
The DO-178C document has clarified many activities and objectives in the certification process, Romanski says. "Companies that have been interpreting the old document badly will have to modify their approach to be in line with the new document. Those companies that have been interpreting DO-178B the way it was intended will need minor adjustments to their process plans. The proper interpretation and, thus, testing of low-level requirements and coupling in DO-178B will make migration to new or clarified requirements in DO-178C much simpler."
The size of future systems will increase as more sophistication is demanded, Romanski predicts. There is an approach to composing large systems made up of smaller systems through the use of integrated modular avionics (IMA), which is used successfully on most of the new, large aircraft, he says.
"There is an expectation to break this down even further, so that certification evidence can be developed for smaller modules independently. Several specifications and standards are under development in an attempt to lower the overall costs of deployment of these large systems," Romanski continues.
The Future Airborne Capability Environment (FACE) Consortium, an Open Group Managed Consortium, and the UAS Control Segment (UCS) Working Group are moving in this direction, Romanski says. "While the composition of the software modules into larger systems is understood, the composition and preservation of the different elements of certification evidence is still a work in progress. The issue of how to test coupling in such a modular environment, how to take credit with these modular sets of test results, has not been fully defined."
Parasoft's Hicken frequently sees "organizations that start with great ideas and plans, and then as deadlines get close or pass, they start taking shortcuts. Saving time is better handled by working smarter and being more efficient," he advises. "Rather than say 'this needs the most test' meaning 'you can probably skip this other thing,' I'd prefer to have tools in place that help me understand what is tested, what isn't, and what I actually need to retest as my code, requirements, and tests change. We need to move out of the 'let's test quality in' methodology to 'let's build quality in.'
"This requires real business policies, requirements, and risk management. It's not uncommon to talk to an organization and ask, for example, what their static analysis policy is. The answer is 'we want people to use it.' A policy like that is the same as saying 'you can use it if you want' or even 'you don't have to use it.' I prefer a policy that says 'you must use it, report on it, and actually fix things that it finds' down to the level of when/how suppressions are allowed, what percentage of items need to get fixed, what severity of items must be fixed, what items can be ignored when you release, and other such details. This is a more proactive approach, rather than 'we'll fix it when it's a problem.'"
Consider the complete solution necessary to solve the problem at hand, advises LDRA's Murray. "You may not implement it all at once, but it's best to have a plan from the start on how all these tools, environments, and processes will work together. If you can simplify the environment and decrease the amount of integrations, then you will have less chance of impacting your program's schedule, budget, and quality.
"If you only focus on a code coverage tool, for example, without considering how that relates to requirements-based testing, and how it helps create your certification artifacts, then you may only solve one problem while missing your program's schedule," Murray cautions. "Minimize the number of vendors and brittle integrations by taking a broader view of the task at hand."
NASA Eclipse specification editing and discovery tool
NASA officials needed a specification editing and discovery tool (SPEEDY) for C/C++ code analysis. They found their solution at GrammaTech Inc. in Ithaca, N.Y., contracting the company to develop a prototype plug-in to the Eclipse integrated development environment (IDE) to assist software developers with modular formal verification tasks.
SPEEDY will provide automated suggestions of specifications for given contexts, with user interface features that aid developers in generating, editing, and checking specifications. It will support the needs of NASA's software-development teams and Independent Verification and Validation (IV&V) groups in evaluating the safety and robustness of software in production or under review, including embedded next-generation avionics and space software.
"SPEEDY will essentially be able to look over your shoulder, using machine-checkable specifications to automate sound verification and warn you if something isn't right," GrammaTech CEO Tim Teitelbaum explains. "The user interface features and underlying automation in SPEEDY will facilitate the use of formal methods by all software developers, improving efficiency and accuracy of development teams."
"Even if your program does not require certification today, you should still start to implement some of the best practices that are referenced in all the certification standards," recommends Chris Murray, vice president of business development at LDRA in Wirral, England.
"Coding standards are a good place to start as the return on investment (ROI) is significant." For example, teams can implement a subset of MISRA-a software development standard for the C programming language developed by the Motor Industry Software Reliability Association-and eliminate common coding errors early in the development life cycle, saving time and money before they become costly errors downstream. Also, Murray adds, "consider how a coding standard solution implemented today ties into future tools and processes."