Changing Times for RTOS

Embedded real-time operating systems (RTOSs) have changed, says Hank Hogan, adding greater connectivity while meeting new safety and security certifications and considerations.

Embedded real-time operating systems

keep planes, trains, and automobiles – as

well as factory floor equipment – running

on time. At the heart of an embedded

RTOS is determinism, the absolute guarantee that

when the clock ticks or an interrupt arrives, the

system will respond appropriately.

But today that’s not enough. Safety, security, and

communications rank among other requirements.

Consider the experience of industrial control supplier

Siemens Energy and Automation (SEA). The

company wanted to deploy a PC on its customer’s

factory floor. According to Eric Kaczor, Product

Marketing Manager for engineering software

products at SEA, engineers quickly learned that a

standard office PC wouldn’t work. To avoid crashes

and freezes, they decided to go with an RTOS.

However, the need to preserve office PC software

applications played a role in RTOS selection. “When

we chose the real-time operating system, we were

looking for an environment that we could easily port

our office applications to,” says Kaczor.

More functions than a PLC

Today an RTOS has to handle connectivity of

various types, meet safety certifications, satisfy

security requirements and, in essence, look like an

office personal computer. RTOS vendors have

responded with added features and capabilities,

which puts pressure on memory and other constraints.

Fortunately, it’s possible to slim down an RTOS

considerably, ensuring that it and any applications

fit in the space available. A look at the current

RTOS landscape shows how it’s changing and what

to look for in industrial, automation and control

applications.

For its part, SEA went with the RTX RTOS from

Ardence, a Citrix company, in its Simatic Microbox

420 industrial PC. RTOS capabilities extend beyond

what’s available to the casual viewer. “They look at

this, and they think it’s a PLC…, not realizing that

it actually has more functionality than a PLC,” says

Kaczor.

Part of that extra capability is connectivity. Paul

Chen, VxWorks product line manager at RTOS

vendor Wind River Systems of Alameda, CA, notes

that linking up with the outside world is a large

requirement for state-of-the-art embedded real-time

operating systems. This includes such user-expected

technologies as USB, Ethernet, and wireless. Endusers

also expect such standards such as nextgeneration

Internet (IPv6), the various 802.x wireless

flavors, MIPv4 and MIPv6 for mobile applications,

along with IPsec and HTTPS for security.

Customers guide RTOS vendors. “If the real-time

operating system software wasn’t providing these

technologies, our customers would be encoding those

layers themselves,” says Chen.

The danger is that end-user or original equipment

manufacturer (OEM) add-ons could impact the

function of the software scheduler–the all-important

part of an RTOS that makes it deterministic. Since

they know the code, RTOS suppliers can add the

capabilities without removing the real-time part of

their product.

The same is true for safety and security

certifications. The former shows up in

airborne systems, industrial applications,

and medical software under regulations that

begin with three-letter acronyms: FAA DO-

178B, IEC 61508, and FDA 510(k).

The added connectivity makes the job

of an embedded RTOS more difficult,

especially when it comes to security. One

version of security is the type used in the

military, which in the past has kept different

security levels on different systems. Today

there’s an effort to bring all levels onto

one piece of hardware, which means the

hardware/software combination has to

keep “top secret” information top secret.

The other version of security, though, is

the one familiar to anyone with an office

computer, where programs can blunder

into each other’s memory space, and outside

forces can attack.

Chen notes that hardware improvements,

the third big driver behind the-state-of-the-art

in embedded RTOS, offers some help

in preventing such problems. For one thing,

semiconductor manufacturers are adding

processing elements to their chips so that

dedicated functions can be off-loaded from

software.

Some functions include encryption for

security and pattern matching for network

virus detection. “Dedicated hardware is

usually faster than software, so RTOS

software needs to be able to support and

leverage the various hardware engines,”

says Chen.

He adds that multicore processors are

now becoming available for embedded

applications. By splitting a processor

into homogeneous units or cores,

microprocessor manufacturers can run

each at a lower clock rate, thereby

cutting total power consumption while

achieving better performance. Thus, an

embedded RTOS could benefit from such

hardware – if the software supports

multicore processors.

Robert Day, Vice President of Marketing

for Lynuxworks of San Jose, CA, notes

that modern microprocessor cores offer

dedicated memory sections, providing

brick-wall partitioning that keeps everything

separate. The concept can be extended by

using a small RTOS as a hyper supervisor,

allowing an RTOS and its applications to

run in one part of the memory while non-real-time applications run in another.

Dan Mender, Director of Business

development for RTOS supplier Green

Hills Software, CA, notes that the use of

such a separation kernel makes for a secure

system and pays other dividends. He says,

“The same principles that isolate and

contain attacks on one area of the system

can be used to create systems that are safe

from inadvertent programming errors.”

Bare necessities

Aside from being able to support and take

advantage of such hardware innovations, an

RTOS has to meet connectivity, security,

and safety demands. However, Day is

quick to point to other functions of the

software that are absolute necessities. In

particular, there has to be some form of

communication so that an application with

multiple tasks running at once doesn’t get in

the way of others.

“Most applications that have multiple

tasks also require some form of intertask

communication, such as queues,

semaphores and Mutexs, with the latter

being important to applications that cannot

afford to have a priority inversion issue that

allows lower priority tasks to block out vital

system functions,” says Day.

For industrial automation applications,

Day adds that the RTOS should permit

rate-monotonic scheduling and also time

and space partitioning. The first supplies

a ticking clock so that time windows are

evenly spaced while the second ensures that

critical tasks get a constant time window.

There are also some communication

protocols and considerations specific

to automation applications and devices.

These may or may not be required because

other standards, like IPv4 and IPv6, may be

enough. However, having these protocols

could certainly be helpful. The list includes

the Controller Area Network (CAN),

OPC, the Distributed Common Object

Model, or DCOM, industrial WLAN for

wireless communications, Profinet or other

industrial Ethernet protocol, and Web

services powered by XML and SOAP

(which forms the foundation layer of the

Web services stack, providing a basic

messaging framework that more abstract

layers can build on). There are also new

technologies and standards, such as ZigBee

in the wireless arena, that could be part of

the RTOS bag of tricks.

Mark Hamilton, a field application

engineer at middleware vendor Real-

Time Innovations (RTI), has had extensive

experience designing, developing and

deploying real-time systems for the

intelligence and military communities.

While an extensive list of protocols and other

features are nice and perhaps necessary, he

cautions that there’s a limit as to how much

can be piled onto an RTOS.

As more functions become part of the

system, there will come a point at which

the scheduler simply can’t run the software

threads. In that case, RTOS determinism

will suffer. Then the system may offer the

functionality of a desktop computer and

behave like one.

Not surprisingly, he points out that

middleware can help avoid such problems

by managing the communication between

software applications. This capability can

allow the RTOS to unload some burdens

via its connectivity and not bump up against

scheduler constraints. Middleware even can

add value by supplying some ready-made

solutions to common problems.

“Middleware may provide the ability to

move data very efficiently between CPUs

that are a part of your system in a way

where you don’t have to now worry about

writing a bunch of networking code,” says

Hamilton.

Such off-loading and partitioning may

be necessary for another reason beyond

scheduler constraints. An embedded RTOS

typically operates in an environment

where memory is constrained and storage

restricted. At the same time, as devices add

new features, applications tend to get bigger.

The result is that the memory footprint of the

RTOS must remain small while supporting

growing device functionality.

RTOS vendors have taken several

approaches to meet these competing

demands. One method puts the final

decision in the hands of developers and endusers.

The ultimate RTOS configuration is

determined by adding or removing modules

during build time, allowing fine-grained

control over the trade-off between capability

and footprint. Potential savings with this

approach can be substantial.

For example, during development and

testing, users may elect to have diagnostic

information points placed within the code.

When development is done, these diagnostics

can be removed, sometimes saving double-digit

percentages in footprint size.

A corollary is that the modules themselves

have to be small. That ensures that the

final build and the RTOS itself will be

compact. However, another solution may

be to shrink the application space through a

careful choice of programming language and

compilers. That can have a dramatic impact

on the application size. National Instruments

LabView software, for instance, has options

to develop, integrate, re-use existing code

then port RTOS programming to a variety

of targets.

Electronic amnesia

For systems running an embedded real-time

operating system (RTOS), nonvolatile

memory holds the operating system,

running applications, and collected data.

Memory choices abound, and candidates

include battery-backed-up SRAM (serial

random access memory), flash memory

of various forms, and serial EEPROM

(electrically erasable and programmable

read-only memory).

So what’s best? “The answer to the

memory question is ‘some combination of

the above,’” says Wind River’s Chen.

As for why no one size fits all, consider

the static and runtime requirements of a

situation. The RTOS and any applications,

which could total hundreds of megabytes,

may be accessed during system startup

and largely untouched afterwards. Process

data, on the other hand, may be only a

handful of bytes but collected and stored

continuously.

One approach simply replaces the

spinning hard drive found in a PC with

one constructed out of flash memory

chips. Solid-state drives are small,

environmentally rugged, and consume little

power. They can be replaced as needed

and even can take some of the load off the

processor running the RTOS by handling

read/write housekeeping.

Solid state drives have also become

more affordable. Gary Drossel, vice president of planning at solid-state drive

maker SiliconSystems of Aliso Viejo, CA,

recalls when solid-state disks cost 40 times

what similarly sized rotating ones did.

That’s no longer the case. “We’re starting

to see maybe a two- to three-times price

premium,” he says.

Not all solid-state drives are the same,

with some offering error correction

and wear-leveling. The latter extends

disk life by ensuring bits don’t wear out

prematurely through overuse. Some

devices have the ability to write protect

specific areas, thus keeping the RTOS and

applications pristine.

But solid-state drives share something

with the rotating variety. There’s a

minimum size, which currently can run in

the tens of megabytes.

For smaller storage needs, users can

go with flash memory, battery backed up

RAM, or serial EEPROM. Several factors

play a role in the choice, such as the

amount of data involved and how often

data is written.

Typically the RTOS and applications

will be stored in flash, so the question

becomes what to do with the process data.

If there’s little process data involved, then

serial EEPROM may be the answer, notes

Martin Bowman, Senior Application

Engineer in the memory products

division of Microchip Technology. That’s

particularly true for industrial applications,

where long equipment lifetimes are a must.

“One of the advantages of using serial

EEPROM technology is the actual number

of erase/write cycles,” says Bowman.

While bit errors start to appear in flash

memory after 10,000 such cycles, serial

EEPROM can handle a million. The

downside is that serial EEPROM is slower

and isn’t as cost effective for larger storage

as flash.

A final solution would be to store

information remotely via an Ethernet

or wireless connection, using battery-backed-up SRAM for those times when

the connection is out. There’s no bit wear

problem, and writing and reading data is

fast. But there is the issue of the battery,

which has to be monitored and maintained.

Otherwise, there’s a chance for electronic

amnesia to occur and data to be lost.

It’s the ecosystem

As the interplay between RTOS and

application space indicates, whatever

steps are taken to fit within the constraints

will be a function of the integration

between the RTOS, applications, software

tools, and hardware. Todd Brian, product

marketing manager for the Nucleus

Kernals product line at electronic design

automation supplier Mentor Graphics

Corp. of Wilsonville, OR, notes that

no RTOS vendor is an island. All work

within an eco-system and that fact is

crucial to getting the best performance

from an embedded RTOS. “A vendor

needs partners that have integrated with

the vendor’s OS. That way, they are not

burdened by the integration process,”

Brian says.

Such integration benefits RTOS

suppliers and end-users, and is something

to look for in an embedded RTOS and

its vendor. “The complexity of device

design is growing so quickly that delivering

software is not enough. That software has

to be integrated so the customer is not left

with that burden,” says Todd Brian.

An RTOS has to support and take advantage of embedded hardware innovations.

Solid state drives are becoming more affordable for embedded systems storage.

Towards Real-Time Linux On March 14 it was announced that IBM and Red Hat are collaborating to deliver a new “Real-Time Linux” application development and deployment platform in order to provide customers the ability to run systems that can perform at greater processing rates with high levels of reliability. The new platform includes IBM WebSphere Real Time, a real-time J2SE Java Virtual Machine, with a real-time version of Red Hat Enterprise Linux 5 (RHEL 5) running on IBM System x and BladeCenter AMD and Intel-based servers. The computing infrastructure enables Java programmers to develop applications that can execute and provide predictable, to the millisecond granularity execution times. The US Navy is an early adopter of IBM’s Real-Time Java technology running on a customized Real-Time Linux platform deployed by IBM, Raytheon, and Red Hat. Applications ranging from command-and-control, navigation, targeting, weapons control, and radar systems. The Navy is looking to cut the cost of developing and maintaining traditionally expensive real-time systems. Commenting on the IBM/Red Hat collaboration, which also includes integrating improved virtualization features and security enhancements into the Linux kernel, Bob Mick of ARC Advisory Group, said, “Real-time is needed for many more application areas to provide real-time performance for manufacturing and other businesses. Linux has done well in the server markets, and this will strengthen its position further.”

Defining Real What does it really mean to be real? The folks from National Instruments provide some clarifications. Real-time operating systems originated with the need to deal with event response applications and with closed loop control systems. Event response applications require a response to a stimulus in a determined amount of time; an example of such a system is an automotive airbag system. Closed loop control systems continuously process feedback in order to adjust an output; an automotive cruise control system is an example. Both of these types of systems require the completion of an operation within a specific deadline. This type of performance is referred to as determinism. Familiar operating systems such as Microsoft Windows and Mac OS are designed for general purpose use and optimized to run a variety of applications simultaneously. As a result, high-priority tasks can be preempted by lower priority tasks, making it impossible to guarantee a responsetime for critical applications. In contrast, real-time operating systems give users the ability to prioritize tasks so that the most critical one can take control of the processor when needed. Real-time systems can be classified as “soft” or “hard”. Soft real-time typically means the utility of a system is inversely proportionate to how long it has been since a deadline is missed. For example, when pressing a mobile phone button to answer an incoming call, the connection must be established soon after the button has been pressed. However, the deadline is not mission-critical and small delays can be tolerated. But for hard real-time systems the utility of a system becomes zero in the event of a missed deadline. An automotive engine control unit (ECU) must process incoming signals and calculate spark plug timing within a deadline. If the deadline is missed, the engine will fail to operate correctly. The usefulness of a task after a deadline is missed is depends on whether the system is a soft real-time or a hard realtime system. Control & response Most real-time control systems monitor a physical system, compare the current state with the desired state, and then adjust the physical system based on the comparison. The time it takes to traverse this loop is considered the “loop cycle time”, which varies based on the complexity of the system. Determinism measures the consistency of the specified time interval between events. Control algorithms, such as PID (proportional-integral-derivative), require very deterministic behavior. For example, an elevator gradually moves to the correct floor because of the deterministic behavior of the control loop. Without the determinism, the elevator would still reach the correct floor but without stability. With all real-time systems, there is some amount of error called jitter. Jitter is another way of measuring the determinism of a real-time system. You can calculate it as the maximum difference between any individual time delay and the desired time delay in a system. With real-time event response, you can respond to a single event within a given and guaranteed amount of time. The realtime system guarantees some maximum response time to the single event. The event can be either periodic or random. For example, if a process plant enters a dangerous state, a real-time system monitoring must respond to the “danger” event within a guaranteed amount of time. “Latency” is used to describe the time it takes to respond to an event. Real technology National Instruments offers a variety of real-time hardware targets that contain an embedded processor running a realtime OS for maximum reliability and deterministic performance. It is possible to integrate a wide array of I/O with modular hardware that scales to accommodate high-channel-count data acquisition and control, industrial signal conditioning, and safety isolation. For software, the LabVIEW Real- Time Module and LabWindows/CVI Real-Time Module are used to achieve reliable deterministic execution on dedicated hardware targets. For increased determinism needs, the LabVIEW FPGA Module combined with hardware that includes reconfigurable I/O technology, offers nanosecond hardware response. And in November 2006, National Instruments announced the release of the NI cRIO-9012 high-performance, real-time controller. The controller is based on Freescale’s MPC5200 processor built on Power Architecture technology, and Wind River VxWorks real-time operating system. The aim: to deliver fast performance while maintaining the ruggedness, reliability and low cost of the NI’s CompactRIO platform. Running on the MPC5200, the VxWorks RTOS delivers dependable performance and a fault-tolerant file system that provides reliable data logging, making it possible for engineers to operate the controller for long periods of time in remote applications using a battery or solar power. The LabVIEW Real-Time Module provides shared variable technology for simplified network communication between distributed systems as well as the new LabVIEW Project, which streamlines code control and application deployment to multiple CompactRIO systems. With the technologies in the new cRIO-9012 real-time controller, the CompactRIO embedded system is said to be ideal for applications such as machine control and monitoring, in-vehicle logging and embedded system prototyping. The collaboration on the controller also illustrates the three companies’ ongoing strategic relationship which aims to improve the development of embedded devices. CEA The utility of a hard real-time system becomes zero in the event of a missed deadline. A variety of offerings for real-time applications. The cRIO-9012 controller real-time from NI incorporates technology from Freescale and Wind River.