Estimating the effort and cost of a DO-254 program

By Michelle Lange, Logicircuit Inc.

Up until now, no tool or even quantitative guidance has been publicly available for companies to use for this purpose. This paper, which is based on data from numerous projects by a leading DO-254 services company, explains the factors that drive DO-254 effort and expense. It provides the background that is utilizes in a new DO-254 Effort Estimator tool that is available from Logicircuit, Inc. The target audience is program managers and/or engineering leads responsible for estimating the cost of new DO-254 programs.

DO-254 Basics
RTCA/DO-254, which was developed in the late 1990’s and early 2000’s, was invoked as policy by the Federal Aviation Administration (FAA), the European Aviation Safety Agency (EASA), and other worldwide aviation safety agencies in 2005. The DO-254 standard (called ED-80 in Europe) is titled “Design Assurance Guidance for Airborne Electronic Hardware.” The intent of this standard is to ensure safety of in-flight hardware by imposing structured development processes.

Understanding what DO-254 entails can be tricky. First, the scope is somewhat unclear. If you read DO-254, it appears that it applies to all levels of hardware, from component through board or even LRU. However, the document that actually invoked DO-254 as policy, AC20-152, bounded the application of DO-254 to “custom micro-coded components…such as ASICs, PLDs, and FPGAs.” In 2008, the FAA published Order 8110-105, which helped to clarify some aspects of the application of DO-254 for custom micro-coded components, since the document was not initially written exclusively for this scope.

This same year the FAA published the “Conducting Airborne Electronic Hardware Reviews Job Aid,” which provided certification authorities and their designated representatives with guidance as to how to hold DO-254 audits and what to look for during these audits. Even though this was not the intended audience, this document also helped to clarify DO-254 process expectations for DO-254 applicants themselves. Despite earlier limited scoping, the trend today is to push the scope of DO-254 closer to its original intent. The recent (2012) EASA Certification Memo CM-SWCEH-001 does just that. From this, it is clear that DO-254 expectations are not easily understood, and in fact, are constantly evolving.

The second major factor of confusion about DO-254 is that it really doesn’t tell you what to do. RTCA documents by design are not prescriptive and do not provide examples. They are instead objectives based, meaning they provide a number of somewhat abstract objectives that the applicant must determine how best to meet. Of course over time, common methods of meeting the objectives evolve, but this sort of “tribal knowledge” is not easy to come by without experience. This experience is typically obtained either through specialist consultants or simply the pain of going through the first few programs, getting things wrong, and
learning was is expected.

DO-254 can be a big concern to businesses because compliance can be quite costly. Companies with development programs requiring DO-254 compliance can see cost increases upwards of 400%. Bringing down this cost means not only gaining experience and establishing process efficiencies, but also understanding the key factors affecting the cost of a DO-254 program.

Cost/Effort Factors
The following list identifies and describes the items that affect the effort and/or cost of a DO-254 program.
The items are presented roughly in order of significance.

• Team Experience: DO-254 involves a difficult learning curve. First programs are often painful and
  expensive. The objective-based format of DO-254 and additional guiding documents can be difficult
  for first-timers to grasp. Training is a good start, but no substitute provides the learning of going
  through that first project or two. Experiencing audits and resolving “findings” (non-compliance
  issues) is the best, though the most expensive and oftentimes painful, learning. In general, highly
  experienced teams can be several times more productive on DO-254 projects than those with lesser
  or no experience.
  
• DAL: The work required in a DO-254 program is modulated (a common word spoken in “DO” circles)
  by the development assurance level (DAL) of the program. In other words, the effort for a DAL A or B
  program is far greater than the effort for a DAL C or D program.
  Design complexity: Complex designs will require more effort than less complex designs. Complexity
  can be measured in terms of number of requirements (for new designs being developed to
  implement those requirements) or by lines of code (LOC, for existing designs that are being reengineered
  to be DO-254 compliant).
  
• Quality system: Companies that have already achieved some sort of quality process certification
  (such as AS9100C or ISO 9000) fare better in DO-254 programs than those who do not. These sorts of
  processes tend to instill a culture of structure and control that DO-254 mandates. Companies that
  already abide by these principles have an easier time adapting their processes to be DO-254
  compliant.
  
• Existing or new?: If a design already exists, this can cut down on a significant portion of the
  development cycle. However, a certain amount of certification overhead exists even for pre-existing
  designs. And these designs often require modification as issues are typically found during the
  extensive verification required by DO-254.
  
• Military programs: While military aviation programs are starting to adopt DO-254, oftentimes they
  do so with a reduced DAL level or with relaxed requirements even for the same DAL. Military
  programs tend not to go through the same level of audit scrutiny as commercial programs. Since the
  time to prepare, hold, and address issues found in audits is a key time factor in DO-254 programs,
  military programs typically require less effort.
  
• Multiple certification agencies: While the US, EU, and numerous other countries have an agreement
  to honor the same certification standards, variants and compliance preferences often do come into
  play. Oftentimes what passes one certification agency, might require modification or additional work
  to pass with another certification agency. Therefore, programs having audits with more than one
  certification agency may require additional effort. Specifically, EASA may mandate some level of
  circuit board or LRU compliance that the FAA today does not require.
  
• Engineering costs: The cost of an engineer can vary based on several factors including their experience and also their geography. This factor is not always as straightforward as calculating hourly rate multiplied by a fixed number of anticipated hours. Oftentimes there is a reverse correlation between the two variables where lower salary might mean higher number of hours. In other words, sometimes less expensive engineers can actually end up being more expensive due to factors like lack of experience. Sometimes a higher salary or hourly rate can mean more experience and higher productivity.
  
• Tool sets: A team may need to supplement their tool set with development and verification tools that are tuned to support DO-254 processes. Using new tools means training and ramp-up time for the engineers using them. These tools also need to go through a tool assessment (and possibly qualification) process as per DO-254 section 11.2. The first time through this process tends to be expensive. Subsequent uses of the tool set may leverage experience and even work done from earlier programs.
  
• Use of IP: In general, intellectual property (IP) or reusable hardware components are used to improve productivity and efficiency of the design process. While this can be true also for DO-254 programs, the use of IP requires careful planning and a clear strategy. The DO-254 related policy is somewhat murky and evolving. IP is allowed only if the appropriate level of design assurance can be demonstrated. Several alternatives are provided in terms of compliance options, but most of them require substantial work—to the point where the use of IP may no longer be cost effective. The best bet if you choose to use IP is to use DO-254 compliant IP. This eliminates the extra work or unexpected requirements that could arise at the end of your program. It also provides the best strategy for design assurance and certification approvals.

All of these factors can interplay, affecting each other and affecting the overall effort and cost of a DO-254 program. While there is no single formula for determining the cost or effort of a DO-254 program, through experience, companies can begin to collect data on these (and potentially other) factors and begin to develop methods that can guide them in estimating the cost of a new DO-254 program. Companies that are able to do this have an advantage during both the bidding process and in the management of their programs, over those that do not understand the factors affecting DO-254 program cost and effort.

Reducing Cost of Compliance
It’s fairly easy to deduce from the cost/effort factors, which are in your control and which are not. Reducing cost of compliance involves adjusting those factors that are in your control and/or looking for creative alternatives to deal with those that are not.
For example, one factor that is in your control is developing team experience. Certainly for a new team and a new project, there will be a learning curve. However, proper training, or potentially hiring an experienced consultant to assist on a first project can help lower this cost factor. Even though the cost of hiring an expert can be high, the experience that person brings can be invaluable to the program, saving far more money to the program than the consulting cost.

Likewise, while you may not be able to actually reduce the complexity of the design you are chartered to develop, you may be able to build this design in part out of existing IP. Using IP can accelerate the development process, and using DO-254 compliant IP can reduce portions of the certification effort. If the IP is reused in additional programs, the savings continues to grow. (See Logicircuit WP001 “Assuring Compliant and Cost-Effective Use of IP in DO-254 Designs” for more information on DO-254 compliant IP).

Some companies have even determined, given all these factors, it is actually more expensive to build and maintain an in-house team of experienced DO-254 engineers than it is to contract this work out to DO-254 compliance specialty firms. This is especially true when companies may serve other markets as well, where not all programs need to be DO-254 compliant. It is hard to find and keep good, experienced, DO-254 savvy engineers, and if you can, it’s usually not an inexpensive proposition. If you build this internal expertise, you want to ensure you can keep these folks busy and focused on this type of work so they remain in tune with the latest policy and acceptable processes.

If you decide to build the expertise in-house, one stepping stone that can ease the move to developing DO-254 processes is by first instilling a general quality process. As mentioned earlier, quality processes like ISO 9000 (and similar standards) can lay the foundations for the discipline and rigor required for DO-254 development work.

Summary
DO-254 programs are notoriously costly. Numerous, interacting factors drive up the cost. Understanding these factors can help program managers estimate the cost of DO-254 compliant programs. Some companies build estimation tools out of data they’ve collected through their own experiences, based on these and other factors. For the first time, there is now a publicly available estimation tool that can provide a rough estimate of the effort and cost of a DO-254 program. (Visit www.logicircuit.com/resources to request it.) This of course is only a starting point, as each program will have its own unique set of factors and challenges.