Controversy still swirls around using Linux for real-time applications
The notion of "real-time" Linux remains a controversial topic. Some in industry who offer what they clam are real-time versions of the Linux operating system, insist their modifications have put any lingering doubts to rest.
By J.R. Wilson
The notion of "real-time" Linux remains a controversial topic. Some in industry who offer what they clam are real-time versions of the Linux operating system, insist their modifications — whether directly to Linux itself or with the intervention of an external real-time kernel — have put any lingering doubts to rest. Others disagree.
The fact remains, there are several companies offering "real-time" Linux, with software engineers from each insisting his approach provides a true real-time operating system (RTOS) and his competitors do not. But the man at the center of Linux, creator Linus Torvalds, is said to have a very strong position on the matter.
Torvalds and his group are interested in improving the general support and performance of Linux, but are making no efforts to turn it into a true RTOS. This is according to Dr. ir. Herman Bruyninckx, an assistant professor in the Department of Mechanical Engineering, Division of Production Engineering, Machine Design & Automation (PMA) at Katholieke Universiteit Leuven in Belgium. Torvalds, Bruyninckx writes in "Real Time and Embedded HOWTO," has mentioned two reasons why he does not want to make Linux into a real-time operating system:
"Computers are getting faster all the time, such that a general-purpose operating system will satisfy the requirements of more and more real-time users. (Second), offering hard real-time features to all applications will quickly result in 'bad behaviour' of application programmers: they will all want their application to perform best and program it with high priority, which will lead to incompatible timing constraints between different applications rather sooner than later."
The Real-Time Linux Foundation, still in formation, identifies Linux as "a powerful operating system challenging the OS and computer market, because it is open source and adaptable to different hardware and computing problems. As a general-purpose operating system, however, Linux optimizes average performance, but is not appropriate for hard real-time." That shortcoming as led to the various modifications being offered to change or enhance Linux for real-time applications.
The RealTime Linux Group, formed in October 2000, takes a view directly opposite that of Torvalds: "In the fields of telecommunication, measurement, control systems, factory automation, robotics, embedded computing, etc., [research and development] regarding real-time processing functions has progressed tremendously. This in turn has allowed for development that has made real-time systems which function on the Linux platform a reality."
Even here, however, they refer to "real-time systems that function on the Linux platform" — not a specific reference to Linux itself as a legitimate real-time operating system.
"Linux is a general-purpose operating system — it wants to give all tasks a fair share of the resources (processor, memory, peripheral devices, etc.)," Bruyninckx writes. "Therefore, its basis scheduler is of the time slicing type: It gives more or less equal time slices to different tasks. (At least in user space, because the kernel routines cannot be pre-empted by user space routines.) It is possible to change the priorities of tasks to some extent (using the nice command), but not enough to make the scheduling deterministic."
Bruyninckx says Linux is a poor RTOS because of unpredictable delays caused by:
- accessing the hard disk;
- accessing a network, especially with the TCP/IP protocol, which re-sends packets in case of transmission errors;
- low-resolution timing (10 milliseconds on most processors compared to a 1 millisecond time slice for most general-purpose operating systems);
- low-resolution timer data structure;
- non-real-time device drivers; and
- memory allocation and management — after a task has asked for more memory, the time the memory allocation task needs to fulfill the request is unpredictable.
Bruyninckx notes the magnitude of such delays differs significantly with different hardware. Additionally, the importance of each varies according to the application involved. For some, context switch time is most important (e.g., for sampling audio signals at 44 kHz), while other applications require high computational performance at lower scheduling frequencies (e.g., robot motion control at 1 kHz). Some tasks, such as speech processing, require both.
"All the remarks above hold for all modern general-purpose operating systems, such as Windows, AIX, IRIX, HP-UX, Solaris, etc. It might sound strange at first, but 'good old' DOS was much closer to being an RTOS than Linux, because its scheduler was much less 'fair' and advanced and it had fewer system services to look after. (However, this is only true if there is only one real-time task!)," Bruyninckx writes.
"Because none of the desktop or server operating systems is a good candidate for real-time and/or embedded applications, several companies have started to develop real-time operating systems. Most of them are UNIX-like, but none are mutually compatible. The market is very fragmented, with several dozens of RTOSs but none of them holds a majority of the market.
At least, that was the case before Linux appeared on the radar of real-time and embedded system developers, he adds: "Since about the year 2000, the market has seen many mergers and acquisitions and substantial efforts from the established RTOS companies to become as 'Linux-compliant' as possible. The fact that Microsoft tries to enter the market, too, (with its PocketPC/Windows CE product line) is only accelerating this dynamic, because history has learned that the fragmented UNIX desktop and server markets were easy targets for Microsoft."
Among those working in this realm are engineers at Finite State Machine Labs (FSMLabs) of Socorro N.M., who say their RTLinux technique "allows a general-purpose operating system such as Linux to run as a pre-emptable sub-thread of a lean, fast, real-time operating system."
RTLinux.org, a non-commercial RTLinux site for the open-source user and developer community, adds: "RTLinux is a hard real-time operating system that handles time-critical tasks and runs Linux as its lowest priority execution thread. In RTLinux, a small hard-real-time kernel shares one or more processors with standard Linux. This allows the system to run accurately timed applications to perform data acquisition, systems control, and robotics, while still serving as a standard Linux workstation."
In other words, RTLinux is actually a separate operating system that does not try to change Linux into an RTOS, but instead provides a "homemade" RTOS kernel that remains small and streamlined while Linux itself remains the basis for common applications.
Designers are taking a somewhat different approach at Lineo Inc. in Salt Lake City. Founded in 1998, Lineo is on an aggressive track, having announced since May 2000 the acquisition or intent to acquire eight companies involved in embedded software, RTOS, high-availability solutions ,and digital television.
Those actions include the January 2001 acquisition of Embedded Power, which specialized in general-purpose and digital-signal-processing real-time operating systems and x86 development toolkits, and the recently announced letter of intent to acquire Convergence Integrated Media GmbH in Berlin.
Lineo officials say their Embedix RealTime differs from the soft real-time schedulers of standard Linux by offering real-time systems designers a base for building any application requiring response time guarantees — on time, every time regardless of anything else the hardware might be doing to service standard Linux tasks and services. Embedix RealTime is described as "pre-emptive, deterministic, high speed, guaranteed 'hard' real-time Linux" that includes a suite of tools that provide "a viable real-time development environment for Linux."
Yet another approach comes from specialists at OnCore Systems Corp. of Half Moon Bay, Calif., who are offering their own Linux for Real-Time and the OnCore OS, which can run several different operating systems (including one or more copies of Linux) simultaneously on one computer.
"With the OnCore OS as a foundation, standard Linux application binaries, from Red Hat or other compatible sources, immediately run on top of OnCore's standard Linux model," the company says. "The Linux OS and Linux applications run in their own MMU-protected memory partition and can run in concert with real-time applications in other partitions. This feature allows designers to run Linux applications in an embedded environment and still have their real-time applications run reliably in higher priority contexts on the same silicon. With OnCore's unique approach, designers can build their applications on a generic Linux desktop, then take that binary immediately to an embedded platform."
Douglass Locke, vice president-technology at TimeSys in Pittsburgh says none of those approaches is really Linux, but TimeSys Linux/RT is, with a slight modification. "With other approaches, applications are running in the kernel, not in Linux, so you're doing real-time and Linux, not real-time Linux," he says. "We modified enough of Linux to get real-time within the Linux kernel itself. It creates a resource kernel that acts as a device driver and fully pre-emptable scheduler, all within the Linux kernel.
"This way, you can have applications running real-time within the kernel; that's mandatory," Locke says. "We've extended the kernel in fundamental ways. And we don't do everything within the Linux kernel. We can run on MIPS or PowerPC, etc., with an additional board support package, and will target to any board anybody is interested in."
As a result, TimeSys officials claim, the incorporation of real-time technology directly within pure Linux enhances standard Linux applications and device drivers in real-time responsiveness and quality-of-service. The TimeSys Resource Kernel (RK) that enables that includes microprocessor and network reservations, priority inheritance, priority ceiling protocol, a high-resolution timer, 256 priority levels, and periodic thread support. Timesys Linux/RT also features such enhancements as kernel pre-emption and Support Symmetric Multi-Processing (SMP).
RK's support for a temporal firewall allows an application to reserve CPU and network capacity. Future releases of the TimeSys RK will include support for disk, network reservations and capacity reservations) that can be guaranteed to remain available to the application, even if another high-priority application attempts to monopolize the CPU.
Dave Beal, senior product marketing manager at Lineo, argues Linux does not need to be modified if systems designers can make it work efficiently in concert with a separate RTOS.
"Lineo believes the definition of a real-time operating system is completely independent from the operating system providing those services," he says. "Although there are many definitions of 'real-time', Lineo focuses on control or acquisition type usage and breaks real-time performance into the two primary categories: 'Soft' and 'Hard' real-time."
By Lineo's definition, soft real-time:
- can be susceptible to external and unpredictable events;
- can be susceptible to system activity such as disk or network access;
- cannot provide absolute performance guarantees;
- must be coordinated closely for best performance to ensure that a prohibited system call from one element of the system is not performed during a time-critical period; and
- provides performance whose AVERAGE value is acceptable, under conditions in which an occasional missed deadline is not mission critical.
For soft real-time and Linux, meanwhile:
- standard Linux provides acceptable soft real-time performance for many applications;
- low latency and pre-emptive patches to the Linux kernel and scheduler provide enhanced soft real-time scheduling capability under many, but not all, circumstances; and
- these patches are not native Linux (i.e., they do not appear in the "official" Linux kernel source tree).
Lineo defines hard real-time as:
- deterministic by providing performance that can be determined in advance of the application running;
- bounded with an absolute minimum/maximum scheduling latency value that can be applied for every situation and for every execution of a task, regardless of any other activities that might be requested by the system; and
- pre-emptable with the ability to pause any task in progress to execute a task of a higher priority.
For hard real-time and Linux, Lineo says:
- Linux cannot provide hard real-time performance;
- a small real-time sub-kernel (which runs Linux as its lowest priority task) can provide hard real-time performance;
- this real-time sub-kernel is not native Linux (i.e., it does not appear in the "official" kernel source tree);
- this sub-kernel approach architecturally isolates time critical tasks (e.g., data acquisition) from those that effectively support the time-critical tasks (e.g., GUI, disk access);
- the sub-kernel works collaboratively with Linux to offer a full operating system solution that is capable of providing hard real-time performance for tasks with very high iteration rates, while providing simultaneous access to all of standard Linux services and features to non-time-critical tasks; and
- development of the real-time and support elements can proceed independently.
"Something which should not be lost in this discussion is although none of the scheduler or sub-kernel approaches appears in the 'official' Linux kernel source tree, the availability of Linux source code invites engineers and organizations to contribute enhancements and services which are focused specifically to their own audiences," Beal says. "This should be viewed as a prime strength and advantage of Linux. In sharp contrast to proprietary offerings, Linux users can make their own decisions regarding what features to include — and what not to include — in their final system solution."
Bruyninckx argues the disadvantage of a kernel patch is that it has to be maintained by the developers over evolving kernel versions and applied by users each time they upgrade their kernel.