Constructing Solid Control System Software

Hank Hogan takes a look at tools for improving control system software reliability and at what controller manufacturers are doing to ensure high quality code.

Eelco van der Wal, managing director of Netherlands-based industrial control organization PLCopen, is on a crusade – one that impacts applications and, hence, the part of software reliability control engineers are responsible for. His efforts, though, will only provide part of the solution.

Reliable control software requires a firm foundation, something only vendors can provide. With programs expanding from a few to thousands or more of lines of code, van der Wal sees ballooning costs and rising risks. He also notes that changes, before or after an application is deployed on a controller, are almost guaranteed.

The key to reducing the costs and the risks, according to van der Wal, is structured programming via the interface specifi ed in the global industrial control programming standard IEC 61131-3. While the use of structured programming represents a change from the traditional approach, van der Wal says the payoff can be improved software reliability and more. When multiple projects are considered, the benefi ts are even greater, he says.

“If there’s a certain overlap between the first project and the second project, you will see a dramatic decrease in cost and time and risk factors, and an increase in [software] quality,” says van der Waal, who also indicates that while the cost savings vary, 40 percent is a good fi gure to use. Thus, more reliable software may also be less costly to produce.

A look shows what control engineers can do to improve software reliability also reveals some of the tools that exist to make the job easier. In addition, discussions with controller manufacturers reveal how they’re responding to coding challenges by improving their own methods for ensuring software reliability.

Structured approach

While advocating a standards-based, structured programming method, van der Wal notes that the benefi ts aren’t free and they do require a change in the traditional way of developing code. “You have to have a software development philosophy in place,” he says.

For control engineers, this means adopting a structured approach. Problems have to be decomposed into individual components, which then can be handled by function blocks, which have defi ned inputs and known outputs.

Function blocks are designed using one of the graphical or textual languages in the IEC 61131-3 standard. Internally, they have defi ned variables and data types, such as integer, real, Boolean, or array. With the function block designed, it is simulated off-line. Then, when it meets requirements, it is integrated with other function blocks into the fi nal application, which is then deployed and maintained.

It’s the function blocks that give rise to the increase in reliability and the decrease in cost, says van der Wal. The former happens because the function blocks are relatively small and simple. Thus, it’s possible to exercise them completely in tests and understand what exactly what they’re doing. The cost reduction is a consequence of being able to reuse the fi nished function block in another project. Other savings result from being able to make changes more easily, and thus maintain an application for less.

A structured programming approach does require more upfront work, particularly on the fi rst project. It also means it’s not possible to dive right into a problem, no matter how tempting that is. The control problem partitioning task, however, is helped by such tools as the sequential function chart.

Model-based design tools

Mathworks has a different but conceptually similar approach for reliable control system software. Along with technical computing software, the company develops and supplies model-based design tools, which are the basis for its control software. Its model-based approach to developing control systems, like that of structured programming, requires the problem be defi ned and broken down into smaller steps. With that done, the company’s Simulink process modeling software can create function block analogs.

Paul Barnard, Mathworks’ Marketing Director for control design automation, explains that once a process is modeled, the software can be used for code development. “You can graphically design or describe an algorithm and through automatic code generation, generate C code that then is compiled and runs usually on some type of embedded target.”

National Instruments’ LabView takes a similar approach. The benefi t of graphical development methods is that they abstract the control problem and let a single individual developer manage more functionality. Like compilers that produce machine instructions, automatic code generation removes the human element from the process. The resulting code is more uniform than that produced by people – and might be more reliable, although that’s not guaranteed.

Brett Murphy, Mathworks’ Technical Marketing Manager for verifi cation, validation and test, notes that Mathworks is aware that automatic code generation is not infallible and so provides ways to test the generated solution and the models it springs from. “We have modeling checking standards, for example,” he says.

Other tools provide formal analysis of both model and code, thereby proving if it’s possible to reach a particular failure mode. Such tools allow checking at the component or model level, but they can’t at present be scaled up to large systems.

No machine is an island

That lack of scalability is unfortunate because applications run on platforms and often on systems that are part of a larger network. The combination brings with it an entirely new set of issues that can affect software reliability in areas that are sometimes beyond a control engineer’s domain.

Roy Kok, Product Marketing Manager for Profi cy HMI/SCADA software at GE Fanuc Automation, notes that the platform operating system (in this case, Microsoft Windows, NT or Vista) needs to be up-to-date and, if possible, fully patched. But applying a patch before it’s had a chance to be seasoned in non-critical areas can adversely impact software reliability, he says. So vendors like GE Fanuc have to supply the expertise to know when the patch is just right.

Likewise, other software and components can hurt an automation application – even if there’s no problem initially. “Updates of ancillary software can have a direct impact on the operation and reliability of the primary solution,” says Kok.

While software reliability can be improved by removing noncritical or untested extras, that’s not enough, Kok continues. What’s needed is a diagnostic that can spot changes in third party components and elsewhere.

For example, a production line may be down and it may be possible to bring it back up if a piece of code referencing a limit switch is bypassed. There’s a tremendous incentive to make the change immediately, bypass the switch, and fi x the problem software later. Kok says it’s important that the temporary fix isn’t allowed to slip in unnoticed, become permanent, and cause a failure some time afterward.

GE Fanuc’s change management system can do periodic and automatic scans of controller software, compare the results to a backup fi le, recognize the change, and thereby document it. Improving software reliability requires changes from programmers and software suppliers, but the payoff could be greater reliability, higher quality, and lower cost.

Trust, but verify

Jeffrey Harding, Director of System Software Architecture at ABB Process Automation, similarly notes the company’s 800xA software has diagnostic capabilities that check system operation and health. The product, for example, monitors installed software components and compares them to tested and known good confi gurations.

“This allows the system to detect if an untested version of a component, including a third party component, has been installed,” says Harding.

Eric Kaczor, responsible for product marketing for engineering software products at Siemens Engineering and Automation, notes that end users are confronted by many different pieces of software. That situation makes version control and version compatibility difficult to achieve and manage. One technique that’s used freezes time – at least as far as software is concerned – to reduce the problem.

Thus, Siemens will offer a single “golden DVD” that contains software needed by users for its equipment. Kaczor says that the software in such a setup will lag the most current versions somewhat, but the combination offers something the most recent software might not be able to match. “It’s all guaranteed to work together.”

For its part, Kaczor says that Siemens tries to ensure that software as-shipped is compatible with any combination of all released pieces of hardware. The company does this through extensive regression testing, trying each new prospective software release on rooms full of equipment. Such a check is thorough, but not 100 percent exhaustive, because it’s diffi cult and perhaps impossible to do verification against all third party software.

Opto 22, a supplier of I/O hardware, controllers, and software, does regression testing before releasing software to customers. Roger Herrscher, senior engineer in the Opto 22 quality assurance group, says the company recently reorganized its QA group to better test its software against the range of possible hardware.

Previously, the QA team had been composed of engineers dedicated solely to product testing. Now Opto 22 has brought software and hardware designers into the group. The effort increases detailed product knowledge, improves testing, and ultimately quality.

Such efforts come with a caveat with regards to software, because of its nature: “When using software, there’s almost always more than one way to accomplish a given task,” notes Herrscher.

A natural tendency, he says, is for the designer of software to have a preferred way to do things. Thus, the designer might not consider and exercise all the avenues a user might follow. By having someone else design the test function, that problem can be circumvented, Herrscher says.

Like other suppliers, Schneider Electric does regression testing with every release of its Unity Pro software package. In particular, Rich Hutton, automation product manager for Schneider Electric, notes that the communications layer that handles traffic is key to reliability and consistent operation. Diagnostics can catch memory failures and other problems, but adding them takes resources. If done to an extreme, diagnostics can actually impact the reliability of the main application, he says.

While Schneider Electric works to make its software reliable, the company is also taking steps to ensure that the programming done by end users to create a desired application is also reliable. Hutton notes that the company’s development tool supports fi ve IEC 61131-3 languages in which to program. This is done in part because it helps Schneider Electric’s customers do their part to ensure software reliability.

A choice of standard programming languages “improves the reliability of the designers, because they can utilize a standard programming style,” says Hutton. “That makes their code writing usually more reliable.”

Build quality in

Controlling software quality in the age of open systems and rapid change is a challenge for all developers. At this year’s Honeywell User Group meeting in Phoenix, Arizona, user questions turned to software upgrades and patch management. Honeywell Vice President of Technology DonnaLee Scaggs addressed the crowd and talked about continuous improvement.

“When you’re talking about software quality, you’re talking about three things. The fi rst thing you want to talk about is preventing defects from happening in the fi rst place,” she said. When you’re developing any software program, there are risky elements and there are other elements that are not so risky, Skagg explained. “We want to identify and understand the riskiest elements and test those out.

“Second, if there are defects, you don’t want them going out. You want to fi nd them and patch them internally, before the software goes out to the customer.” And third, said Skagg, “you want to look at any defects you do fi nd and not just fi x them. You want to see what’s causing them and do root cause analysis to make sure they don’t happen again elsewhere.”

Scaggs said Honeywell is moving to an iterative software development strategy that will deploy over next year and a half. Iterative development means software functions are developed, evaluated and tested more rapidly, and earlier in the lifecycle of a new release. Customers are brought in sooner to review the functionality and, testing is done earlier so the results can be fed back into the other functions being developed.

“Ultimately, we don’t want to test in quality; we want to build quality in from the start. An iterative process allows us to do that,” Scaggs said.

What about the OS?

While control engineers and vendors struggle with growing software complexity and therefore decreasing reliability, there are signs that the basis for everything – the operating system – is being increasingly scrutinized.

The second Tuesday of every month now comes with security patches from Microsoft, with some applying to core components in the operating system.

Not installing patches is less and less of an option, due to increasing connectivity and concerns about cyber security. Vendors must therefore do regression testing of patches in a timely manner.

Stephen Briant, Product Manager for the FactoryTalk services platform from Rockwell Automation says that Microsoft doesn’t make its core living in industrial automation. Thus, he says, Microsoft’s software requirements aren’t necessarily aligned with those of control system users. Using a non-Microsoft operating system, he says, could improve overall software reliability – but the trade-offs have to be carefully considered.

Briant also notes the high reliability of today’s controller software overall, and predicts it might have a far reaching impact: “The market is going to trend toward the need for everything else to operate at the same level of integrity as that control solution,” he says.

IEC 61131-3 Software Model

The Beauty of Blocks Arun Veeramani & Todd Walter on how intuitiveness, code reuse, and parallel execution make function block diagrams (FBDs) an ideal tool for control systems programming. A picture is worth a thousand words is a familiar proverb that asserts that complex stories can be told with a single still image, or that an image may be more influential than a substantial amount of text. It also aptly characterizes the goals of visualization-based software in industrial control. Graphical programming is an intuitive way of specifying system functionality by assembling and connecting “function blocks”, with a function block diagram (FBD) replacing thousands of lines from a textual program. FBDs were introduced by IEC 61131-3 to overcome the weaknesses associated with textual programming and ladder diagrams. FBDs are a graphical way of representing a control program and are a dataflow programming model. An FBD network primarily comprises interconnected functions and function blocks to express system behavior. Function blocks were introduced to address the need to reuse common tasks such as proportional-integral-derivative (PID) control, counters, and timers at different parts of an application or in different projects. A function block is a packaged element of software that describes the behavior of data, a data structure and an external interface defined as a set of input and output parameters. Functions v blocks A “function” is a software element that, when executed with a set of input values, produces one primary result and does not have any internal storage. “Function blocks”, on the other hand, do have internal storage and may have multiple outputs. Some examples of functions are trigonometric functions like sin() and cos(), arithmetic functions like add and multiply, and string handling functions. Function blocks include PID, counters, and timers. An FBD is a program constructed by connecting multiple functions and function blocks resulting in one block that becomes the input for the next. Unlike textual programming, no variables are necessary to pass data from one subroutine to another because the wires connecting different blocks automatically encapsulate and transfer data. An FBD can be used to express the behavior of function blocks, as well as programs and also to describe steps, actions, and transitions within sequential function charts (SFCs). A function block is not evaluated unless all inputs that come from other elements are available. When a function block executes, it evaluates all its variables, including input and internal variables as well as output variables. During its execution, the algorithm creates new values for the output and internal variables. Outputs of function blocks are updated as a result of function block evaluations. Changes of signal states and values therefore naturally propagate from left to right across the FBD network. The signal also can be fed back from function block outputs to inputs of the preceding blocks. FBD benefits Some of the major strengths associated with FBDs are described below: • Intuitive & easy to program Because FBDs are graphical, it is easy for system designers without extensive programming training to understand and program control logic. This benefits domain experts who may not necessarily be experts at writing specific control algorithms in textual languages but understand the logic of the control algorithm. They can use existing function blocks to easily construct programs for data acquisition, and process and discrete control. • Extensive code reuse System designers can use existing function blocks such as PIDs and filters or encapsulate custom logic and easily reuse this code throughout programs. Since separate copies are made every time these function blocks are called, system designers do not risk accidentally overwriting data. Additionally, function blocks also can be invoked from ladder diagrams and even textual languages such as structured text, making them highly portable among different models of computation. • Parallel execution With the introduction of multi-processor systems, programmable automation controllers and PCs now can execute multiple functions at the same time. Graphical programming languages, such as FBDs, can efficiently represent parallel logic. While textual programmers use specific threading and timing libraries to take advantage of multithreading, graphical, FBD, and dataflow languages (such as National Instruments’ LabView) can automatically execute parallel function blocks in different threads. This helps in applications requiring advanced control, including multiple PIDs in parallel. Some cautionary points But it’s not all smooth sailing; function block diagrams do have some weaknesses. For example, low-level functions and mathematical algorithms are traditionally represented in text functions. However, function blocks abstract the intricacies of an algorithm, making it difficult for domain experts to learn the details of advanced control and signal processing techniques. And there can also be limited execution control. Execution of an FBD network is left to right, which is suitable for continuous behavior. While system designers can control the execution of a network through “jump” constructs and also by using data dependency between two function blocks, FBDs are not ideal for solving sequencing problems. For instance, going from “tank fill” to “tank stir” state requires evaluation of all the current states. Depending on the output, a transition action has to take place before moving to the next state. While this can be achieved using data dependency of function blocks, such sequencing might require significant time and effort. With businesses increasingly seeking ways to connect modern factory floors to the enterprise, connectivity to the Web and databases has become extremely important. While textual programs have database-logging capabilities and source code control features, FBDs generally are unable to integrate natively with IT systems. Filling the gaps The intuitiveness, ease of use, and code reuse of FBDs make them very popular with engineers, and they are ideal for complex applications with parallel execution and for continuous processing. They also effectively fill gaps in ladder logic, such as encapsulation and code reuse. To overcome some of their weaknesses, engineers should employ mixed models of computation: use in conjunction with textual programming for algorithms and IT integration; improve batch and discrete operations by adding SFCs (sequential function charts). Typical Function Block with Inputs and Outputs FBD Network showing feedback and execution control Execute parallel function blocks in different threads Arun Veeramani is Product Manager and Todd Walter is Group Manager at National Instruments, www.ni.com.