Adding security to speed: safeguarding the RTOS

Military system integrators increasingly demand built-in security to protect their real-time operating systems from information attacks, as the nation becomes more and more security conscious. Meanwhile Linux providers claim their operating system is secure despite its open-source nature.

by John McHale

Click here to enlarge image

Software engineers who design real-time operating systems (RTOSs) say security is the biggest concern among their military customers. Military designers want the protect the RTOSs that run their mission-critical applications against application bugs and information attacks that can eat up system memory and eventually crash the operating system.

Part of the increased demand for RTOS security comes not only from the U.S. government's high state of alert from the ongoing war on terrorism, but also from the desire of many system integrators to comply with Federal Aviation Administration (FAA) safety regulations.

Information security has become a crucial topic following since Sept. 11, says Greg Rose director of product management at LynuxWorks in San Jose, Calif. There are more security requirements than ever before from cellular telephones to large workstation applications, he adds.

Many people fear the possibility of virus infection with their RTOS, however most viruses and not targeted toward RTOSs in particular, explains Bill Weinberg, director of strategic marketing for MontaVista Software, an embedded Linux provider in San Jose, Calif. Most hackers are not aware of many RTOSs, because they are obscure when compared to Microsoft Windows and even Linux, he continues. Windows is the biggest target because hackers really enjoy messing with Microsoft, Weinberg adds.

E-mail is the most common way for a virus to enter the operating system and attack an application, Rose says. If a virus infects an application, the virus can throw the whole system down, which is why most RTOSs have protection built in to isolate the kernel form rogue viruses and bugs, be they malicious or benign, he adds.

Shared memory vs. message passing
The main evolution in security and memory protection in RTOSs was the move from a shared memory model for interprocess communication to a message-passing interface, explains Richard Blackburn, United Kingdom manager for OSE in San Jose, Calif.

"The traditional RTOS will generally use a "share memory" model for inter-process communication (IPC)," Blackburn says. "Most developers are aware of not only the complexities of managing this type of IPC but also the many risks of failure due to one process either holding onto access to data or corrupting data vital to another process. The system works, but it requires many system calls, careful software management and maybe a longer period to iron out bugs. A further limitation of shared memory IPC is that it does not lend itself to a distributed system or the use of memory management. It has been made to work, but there are better ways."

An analogy for explaining the security risks associated with shared memory is to pretend that a shared memory model is a white board in a room that is accessible by all the programmers in a company, Blackburn says. They use it to pass messages to one another, with programmer A coming in and writing a message for programmer B then leaving, he continues. However, because the board is open to everyone another programmer can come in and change the message before programmer B ever gets it. If this continues it can corrupt the information and can shut down a whole company or system, Blackburn says.

However, with message passing the communication is limited to two processes sending encrypted messages, which isolates corrupt messages from the rest of the system, he explains.

Many modern RTOSs use a message-passing model for Inter-process control, Blackburn continues. "In this model, a process will communicate with another process by sending a message. This may takes as few as four system calls. So it is simple, robust, and messages can be traced as they pass through the system during run time, making development and debugging refreshingly straightforward. The message-passing IPC also means that the application can be seamlessly distributed over multiple CPUs and digital signal processors. It also means that it is easy to break the application down into different tasks for multiple developers."

One downside is that "traditional RTOS vendors may claim that this method is slower," Blackburn says. "In certain conditions this may be true but in the overall performance of a system, it rarely proves to be the case."

Another way to make sure that an RTOS is robust and secure is to allocate memory properly, so as to avoid fragmentation that can slow an RTOS, Blackburn says. Sometimes rogue programs can eat up system memory, so the RTOS needs to allocate specific amounts of memory to each task to isolate it from these bugs, he continues, It is similar to the defragmentation tools available in Windows desktop systems, Blackburn adds.

The OSE RTOS protects memory by requiring the software engineer to "specify a 'pool' of memory," Blackburn says. "He will also specify eight sizes of memory that can be allocated from this pool. Then, when he needs memory, the RTOS will give him the best fit from this range of eight sizes. When, the memory is no longer required, it goes onto a linked-list of free buffers. The advantage here is that memory fragmentations is zero, allocation times are deterministic, and the memory is either in the linked-list or at the start of free memory."

Trap operating systems
LynuxWorks's RTOS, called LynxOS, has the basic security features such as password protection, but also has had memory protection since 1988, Rose claims.

LynxOS, a trap operation system, not only uses the memory-management unit for message passing and address protection, bit also keeps the kernel isolated, Rose says. This prevents rogue applications from bringing down the RTOS.

"It allows the system to switch modes from user mode to supervisor mode," Rose says. "Using this as a part of an operating system design allows users then to be able to tag — via the on-chip memory management unit (MMU) — all the physical memory used by the operating system as supervisor mode read only or read/write access and user mode/no access, Rose explains. This means that an applications program — which usually executes in user mode in this type of operating system — cannot read or write the addresses that are a part of the operating system. This helps limit the damage of a stray pointer that points to operating system memory from inflicting damage on the operating system; when that memory location access is attempted, the MMU traps the access as a 'protection violation.'

"It also allows users to only make things visible to the user applications that you want to be public information or to verify that an individual applications program has the proper permission level to access," Rose continues. "Say that only a user program with the 'root' permission, or programs from a specific process grouping can perform. This allows segmentation of individual user programs to multiple groups with security permissions associated.

"This capability acts as a mechanism to allow the operating system to truly act as a supervisor and its critical tables and routines can only be accessed when the system has 'trapped' into system mode," Rose says. "From a security aspect this protection limits the damage that can be done to the operating system from a user application, whether it be a malicious attack or a software bug. Without this capability any program can execute anything because there is really not a concept of different levels of applications programs. Like in one popular RTOS an application program makes a call to an OS function and it is executed and then returns. There is not any checking of any permission or security levels."

JSF software
Security was one of the main reasons that Lockheed Martin officials in Fort Worth, Texas, chose the Integrity RTOS from Green Hills Software in Santa Barbara, Calif., for the company's U.S. F-35 Joint Strike Fighter (JSF) aircraft contract.

The fact that Integrity is an "embedded RTOS with virtual memory support and the capability to meet security standards was a key enabler in the design of our mission-critical software," says John Ledyard, software manager at Lockheed Martin.

RTOS security was an important consideration for the JSF team because even small bugs in software can bring down a whole system, Carbone says. For example, if an RTOS has several tasks running at once, a bug in one task can affect the other tasks and eventually corrupt the entire system, he explains.

In terms of security there are about three levels of RTOSs — zero, one, and two, Carbone says. Level zero would contain RTOSs such as the Wind River VxWorks 5.4, which Carbone claims has very little memory protection and is susceptible to bugs. RTOSs at level zero have virtually nothing to protect them in these situations, Carbone says.

The above diagram from OSE illustrates the differences between a shared-memory model and a message-passing one.
Click here to enlarge image

RTOSs at level one, such as VxWorks AE, QNX, LynxOS, and Linux have memory protection that protects simultaneously running tasks from one another by enabling tasks to run independently, Carbone says. Still, Carbone admits, the corrupted task can affect the uncorrupted tasks indirectly by draining the resources of the RTOS, he says.

Integrity, however, is secure in even this circumstance, which is why it stands alone in level two, Carbone claims. Green Hills engineers have accomplished this through guaranteed resource availability, which enables designers to set the specific time domain and space domain of each task, he says.

In other words each task is allotted a certain amount of time (time domain) to perform its function and a certain amount of memory (space domain), Carbone explains. The time and memory are for that task and that task alone; if another task is corrupted, it is not only contained through memory protection but cannot bring the system down by draining time and memory from other tasks, he says.

For example sometimes a rogue tasks can gobble up tons of memory and get the RTOS stuck in an endless loop, Carbone says.

Each task is budgeted a certain amount of time and memory much the same way different elements of a corporation are budgeted with money, Carbone explains. However, unlike a corporation they cannot infringe on one another, he adds.

Integrity also enables customers to run high-security applications alongside low-security applications without an error or bug in one affecting the other, Carbone says. The operating system isolates an application subjected to either hacking or a virus and locks it out of the rest of the system, he adds.

At the lowest level, the Integrity kernel employs an object-oriented design and access verification to protect against inadvertent and malicious kernel access problems such as invalid kernel addresses and invalid system call parameters, Green Hills officials say.

The kernel design also guarantees bound-ed computation times by eliminating the need for features such as dynamic memory allocation and heuristic scheduling. Underlying hardware mechanisms provide full system memory protection of all components, including user applications, device drivers, and inter-address space communications. Access permissions protect clocks and timers, which are implemented entirely in software, company officials say.

The partitioning of applications from the kernel is the main key to a secure RTOS and Wind River's latest RTOS products offer just that, says Steve Blackman, director of marketing and business development for Wind River's aerospace and defense business unit.

Wind River offers two RTOSs in addition to their market-leading VxWorks RTOS, that offer enhanced security features through use of the memory management unit and robust partitioning for mission- and safety-critical applications, he says.

The first, VxWorks AE, was released about a year and a half ago. It has features such as memory protection, resource management, system partitioning, and resource reclamation, but still allows the developer to obtain similar performance levels of a traditional RTOS and keep control over how the application and operating system interact, Blackman says. Developers can dial in the amount of protection necessary, where they need it, and when they need it, company officials say.

VxWorks AE's security aspects derive from protection domains technology. Previous protection environments included proprietary extensions to COTS offerings and were often based on variants of the traditional UNIX process model or message-passing paradigms, Wind River officials say. These approaches lacked the scalability, real-time performance, and flexible operation necessary for an embedded environment, company officials claim. VxWorks AE retains its original environment while still adding the security features.

Green Hills Software's INTEGRITY DO-178B RTOS supports an ARINC-653 compliant application executive API that assures that high security tasks will not be impacted by the failure of tasks operating at a lower security level.
Click here to enlarge image

A protection domain provides a logical resource "container" that defines an execution environment: each protection domain exists in its own address space and, depending on the configuration, may or may not be visible to other domains, Wind River officials say. VxWorks AE extends the single, flat physical address space of the VxWorks environment to multiple, virtual address partitions in which VxWorks applications — as they are written today — simply run, company officials say. The kernel has its own domain. A developer loads an application with resources such as memory, tasks, queues, and semaphores into a protection domain, thereby isolating and protecting the application from applications in other protection domains, Wind River officials say. The protection domain also defines the basis for automated resource reclamation.

The protection domain defines access to memory through hardware-enforced mechanisms of the memory management unit of modern CPUs and ownership of resources within the container, company officials say. Software engineers can create protection domains either at system startup, or dynamically at run time, to encapsulate resources within a system, Wind River officials say.

Wind River's latest RTOS, VxWorks AE653, is designed specifically for safety/ mission-critical systems that have multiple applications running on a single processor, Blackman says. It is based on Wind River's VxWorks AE RTOS. VxWorks AE's protection domains technology implements spatial partitioning required by the ARINC-653 avionics standard and Avionics Computing Resource Specification (RTCA/DO-255, Wind River officials say). In addition, VxWorks AE653 provides temporal partitioning.

Temporal partitioning enables users to control how much processor time the operating system allocates for each application running on it, Wind River officials say.

Compliance with the FAA's safety regulations has also made security a hot topic among RTOS designers and VxWorks AE653 targets those types of applications, Blackman says. Avionics integrators are demanding that RTOSs used for aircraft flying over U.S. airspace be certified under a standard called RTCA/DO-178B as adopted by the FAA, he continues.

Officials of the Federal Aviation Administration (FAA) in Washington require that software in airborne systems and equipment be certified under RTCA/DO-178B, "Software considerations in Airborne Systems and Equipment Certification," to provide guidance in the production of software for airborne systems.

RTCA Inc., the Radio Technical Commission for Aeronautics in Washington, is a private, not-for-profit corporation that develops consensus-based recommendations regarding communications, navigation, surveillance, and air-traffic management system issues. DO-178B provides guidelines for the software development process yet is not a development standard or process document.

The military does not require compliance with DO-178B. Yet military leaders recognize that they need 100 percent reliability of life- and mission-critical flight software especially with the growing use of commercial-off-the-shelf (COTS) technology.

DO-178B recognizes that different applications and systems have different potential for failure. Accordingly, the standard defines five software levels (A through E) to show certification compliance in accordance with the different hazard classifications. The effort required for each level is on a sliding scale with the Level A, the most safety-critical, requiring the most scrutiny and Level E, the least critical, requiring the least scrutiny.

Operating systems themselves are not certified to the standard, but can be certifiable based on the needs of a particular application, LynuxWorks's Rose explains. LynxOS leaders say they expect by the end of this year to have an application certified to DO-178B Level A, Rose says. LynxOS has been certified to DO-178B Level C for an air navigation application for Aerospatial in Toulouse, France, he adds.

Green Hills officials expect to have an application certified to DO-178B by the end of the year, Green Hills' Carbone says.

The terrorist attacks on Sept. 11 have made compliance with FAA regulations even more necessary for all kinds passenger jets and military aircraft, Blackman says. System integrators for these platforms also want RTOSs that can are designed to be certifiable for different avionics applications, Blackman says.

Wind River officials just recently announced a partnership with Smiths Aerospace in Cheltenham, England, and Ada Core Technologies in New York City, involving VxWorks AE653 for use on Seattle-based Boeing's new safety-critical platform on the U.S. Air Force C130AMP and 767 Tanker Transport programs.

The solution features Wind River's new Tornado for Safety Critical Systems platform, supporting the development of a range of safety/mission-critical embedded and desktop systems including aerospace avionics applications, military ground vehicle platforms, military and civil ground-control stations as well as automotive and telecommunication devices.

Tornado for Safety Critical Systems is a commercial-off-the-shelf certification solution that provides developers with spatial and temporal partitioning in accordance with the ARINC-653 standard. It includes a wide range of software development tools, a certifiable version of Wind River's VxWorks AE653 RTOS, and the documentation required to support the certification process for DO-178B, Level A.

With the protection that the VxWorks AE653 RTOS offers, individual applications and memory cannot corrupt each other. This is a key requirement for safety/mission-critical systems that cannot afford to fail under any possible circumstance.

Linux, the open-source operating system created by Linus Torvalds, while not prevalent among many real-time applications, is seeing military use as a development tool and needs to have some security features, software developers say.

Many people automatically hear that Linux is open source and assume that it is full of security holes because anybody can get the source code, LynuxWorks' Rose says.

However, this is not so, Rose claims. In the first place, Linux has built-in security because it is a trap operating system like LynxOS, Rose continues. LynxOS and Linux have many of the features of UNIX, which was a trap operating system, he says. Next, Linux also uses a memory management unit to help handle memory allocation, Rose continues.

Another big challenge with Linux security is configuration management, Rose says. Linux needs to be tested doing multiple complicated tasks at once to check for "obscure or malignant code than can come up and bite the user when he's not looking," he says.

Operating systems such as Linux "can offer more points of vulnerability, through shells, network daemons, web interfaces, scripting languages, etc., but they also feature greater resiliency and stronger defenses against rogue programs, Trojan Horses, and other programmed exploits," MontaVista's Weinberg says.

"Buffer overflows and other exploits in a traditional embedded operating system immediately yield access to all program instructions, user and system data, and I/O devices in the, "flat" address spaces common to RTOSs and embedded executives," Weinberg says. "The virtually addressed, protected program and data in Linux/POSIX processes isolates programs from one another. This model also helps to limit the scope of exploit damage: exploit-inspired failures can be isolated to single, restartable processes instead of requiring costly system-wide rebooting.

"Detractors of open source claim that the exposed code bases of Linux and other open-source operating systems are an invitation to hack and to crack," Weinberg says. However, One can only wonder how many undocumented security holes exist in systems like VxWorks, LynxOS, and the many other RTOSs in the industry, he continues.

"The events of 9/11 have also reignited the debate over whether open or proprietary systems are inherently more secure, with opinion-makers from the public and private sectors weighing in on both sides of the issue," Weinberg says. "Traditionally, embedded systems have relied on simplicity and secrecy and to ensure security.

Proprietary operating system developers such as Wind River and Microsoft in Redmond, Wash., created this secrecy by providing little or no access to source code and by relying on the "relative obscurity of embedded hardware architectures and the obscure nature of embedded devices themselves," Weinberg continues. The problem with proprietary operating systems is that so few people get to analyze and fix the problems whereas open-source systems benefit from quick disclosure and repair of security holes, Weinberg says. People in the open-source community say they believe that many of the closed-source suppliers are interested in hiding all knowledge of implementation and history of exploits, Weinberg says.

Weinberg says he likes to use a game theory example to demonstrate the strength of open systems security. For example, "both open and closed systems are subject to attack and have been cracked with regularity by 'hawks'. Access to source code may offer some small advantages to potential exploiters (hawks), but binary only products, like Windows, have fallen victim as often to malicious hacks. The real advantage to open source lies with the 'doves.' While there is no guarantee that the open-source community doves will repair security holes and exploits in a given time period, there does at least exist the potential for such amelioration; with closed systems, there is no potential for broad-based defensive efforts outside the vendor's own team, and security holes often remain unplugged for prolonged periods, even years."

Another concern with open source is the lack of security from patches and upgrades that can come from anyone anywhere in the world, Rose says. Many customers will buy the LynuxWorks package, but then get patches and add-ons from other individuals not connected to LynuxWorks, he continues. On one hand this is great because so many people are working to improve Linux, but on the other hand when someone gets a Linux patch from just anybody, there can be security risks to the OS, Rose says.

MontaVista and LynuxWorks offer their customers complete certification and testing of their products and look to provide continued support so the customers do not have to go to outside uncertified sources for their Linux patches.

Company information
Green Hills Software
Santa Barbara, Calif.

San Jose, Calif.

MontaVista Software
San Jose, Calif.

OSE Systems
San Jose, Calif.

Ottawa, Ontario

Wind River Systems
Alameda, Calif.

To access this Article, go to: