The Change Agent

Management of change (MOC) is a subject that process industry manufacturers know well. It is a core process for managing modifications to such things as plant infrastructure and equipment, and although it can be a significant consumer of time, most acknowledge that it improves safety.

While most countries have recognized the need to manage change in process plants and established regulations governing the MOC process, the proper management of automation changes is generally not fully addressed in MOC regulations and as a result, most industrial plants live with significant vulnerabilities associated with poor automation change management. Many are unaware of these vulnerabilities.

Undocumented and unapproved changes to automation systems have been identified as contributing factors to a number of process industry incidents and accidents around the world. To learn more about this, we at PAS recently conducted a survey of its petroleum, power generation, and chemical industry clients asking the question: “Have modifications to automation systems ever led to an incident or process disturbance at your site?”

Shockingly, nearly 65 percent of the respondents said that they had! This data, while not definitive, indicates that proper management and documentation of automation configuration changes, while an essential element of control system security and plant safety, is not being practiced as it should be.

Automation systems in process industry plants contain an astronomical amount of data from disparate systems, measurements and applications. And since these systems are primary vehicles for continuous improvement of the process, they are continually undergoing change.

In the same survey, we asked those clients: “How often do you make one or more modifications to the configuration of your automation systems?” Nearly 38 percent responded that they made changes to their automation configuration at least once per day, and another 35 percent responded that they made changes at a frequency of between one day and one week.

We also asked the question: “Do automation system modifications currently require an MOC process at your facility?” Forty-seven percent responded that it sometimes does and 53 percent responded that it always does.

Taken together, these three questions tell a chilling story about how automation is managed in these facilities. While automation systems are modified often, an MOC process for those changes is only sometimes required, and automation modifications have led to incidents and/or lost production at nearly two-thirds of the sites polled.

The genesis of MOC
To understand why the MOC process is only sporadically applied to automation, we must consider its development. MOC has long been deemed best practice by many companies, but only became law in the United States in 1992 when Congress passed the 29 CFR 1910.119 Process Safety Management (PSM) regulation.

This regulation, which is aimed at preventing or minimizing the consequences of catastrophic releases of toxic, reactive, flammable or explosive chemicals, mandated that all manufacturers of these chemicals have a compliant MOC process in place by 1997.

Other governments around the world quickly saw the benefits of this regulation and enacted similar laws within their jurisdictions. For example, oil plants in Singapore have since 2001 been mandated to implement safety & health management systems in accordance with Singapore Standard SS506: Part 3: 2006.

In general, most PSM regulations are quite rigorous for physical plant and procedural changes, but are much less stringent for automation systems.

Another consideration when evaluating the sporadic use of MOC with automation is the relative ease of modifying automation system configuration. Generally, it is possible to make modifications to automation systems that have significant consequences by simply typing in database parameters – changes to critical alarm limits or re-ranging an instrument are accomplished in this way.

The traditional MOC process takes significantly more time and energy to implement than the time required to make the actual changes, and so seems excessive by comparison. As such, the ratio of work to benefits is perceived to be skewed heavily toward work rather than benefits, so MOC for these types of changes is often excluded from the work process.

This practice opens the door for unapproved and undocumented changes to be introduced into automation configuration, and these can carry with them the potential for disastrous results, as our client survey indicates. It should be noted that a robust MOC process is almost always followed if an automation configuration modification affects a drawing, procedure or any other entity currently under change control – even though the parametric changes that are currently overlooked could have equally serious consequences.

Only through a program of rigorous MOC that reconciles automation changes to the MOC cases that authorized them can unapproved or undocumented changes be eliminated as a cause of incidents.

Overwhelming impact
As we learned from our client survey, automation changes occur so frequently that, unlike other MOC scenarios, the sheer volume of automation changes can quickly become overwhelming. But there is now software available (such as from PAS) that can import the entire automation configuration database for all systems at regular intervals and compare it to the database image taken at the last import. It then tracks and reports on all changes made between import intervals, both within a specific system and also among interoperable systems.

While most automation system manufacturers are able to track changes within their own systems, this software does so across system boundaries, and is therefore able to track signal genealogy throughout integrated systems. A common example of a signal crossing the boundaries of interoperable systems is a safety transmitter that is connected to an SIS, which passes the measured value to a DCS workstation for display, which in turn passes it to a historian for archiving.

Process manufacturers almost always standardize their underlying Microsoft Windows-based automation systems infrastructure by implementing a uniform configuration of hardware and software, often referred to as a common operating environment (COE). Because most sites have a multitude of disparate automation systems, administrators accommodate the differences among them by defining multiple COEs within the facility and auditing them for compliance at regular intervals.

Managing modifications to this underlying automation system infrastructure is as important to safety and security as managing modifications to the actual configuration. Given that it is commonly understood that the Stuxnet virus was initially introduced through a removable thumb drive, an important consideration for process automation COEs is whether to disable the USB ports into which thumb drives may be inserted.

Modifications to such things as the USB port enable/disable status or Windows active directory security settings represent a significant security concern and must therefore be under MOC control.

Software tools can make this achievable by tracking compliance to COE specifications for the servers, work stations and desktop computers on process automation networks including such items as:

• Software installed on each machine and its versions and patches
• System hardware enabled/disabled statuses
• Correct operation of active directories
• Compliance to available drive capacity guidelines
• Disabled and stale user accounts
• Application and operating system services status

To ensure that no unauthorized or malicious changes are lurking within an automation system, each change, whether it be to the actual configuration or to the underlying infrastructure, must be under strict MOC control. Furthermore, it is essential that automation databases be continually checked for changes and compared to approved MOC cases to determine if those changes were actually authorized. Any changes that do not reconcile, would therefore be unapproved and require investigation.

Streamlined process
As previously mentioned, the MOC process is generally considered to be very time consuming; one client actually described it as so onerous as to be stifling to innovation within their plant. It follows that if the MOC process is to be applied fully to automation, it is a necessity that it is both greatly simplified and made very easy to use.

For most automation system configuration changes there is no need to change drawings or procedures, and thus the traditional MOC process would be overkill. However, to ensure that only safe and appropriate modifications are made to configurations, each change must be evaluated, approved and tracked. A shorter, more streamlined, MOC process that does not entail more effort than the modification itself is clearly needed for most parameter-only automation configuration modifications.

In these cases, PAS suggests a special classification of the MOC process that is initiated, approved and closed out by the automation professional that actually makes the change This process would employ a simple checklist form that would be completed after the modification is made and electronically signed by the person making the change. It is assigned a tracking number and saved as an MOC case by the software, and then on the next import of the configuration database, the modifications are reconciled with the MOC case that authorized them.

A report of all unreconciled modifications is generated so that unauthorized changes are immediately identified and communicated to the responsible parties. Those unreconciled changes may then be either accepted as they are or a new MOC case may then be created to authorize changing them back to an acceptable state. With minimal effort, this process ensures that no unauthorized changes remain in the automation systems.

Clearly this simplified MOC process is not appropriate for all MOC classifications, so software is also available that provides the ability to develop different templates for various types of workflows.For example, there is a class of automation modifications that affect drawings, procedures and other items, and therefore requires a longer, more complicated workflow. These other variations in workflow templates are then associated with the appropriate work classifications, and are selectable from a menu as MOC cases are initiated. Depicted above is an example of one of these longer workflow templates under construction. The MOC workflow comprises elements called “states” and “transitions”. States are represented by rounded rectangular boxes, while transitions are represented by the arrows. Each state may have a checklist associated with it that is composed of various user-selected elements such as checkboxes, data-entry fields, attachment boxes for attaching documents, expression boxes for calculations and electronic signature boxes for recording approvals and completions. Transitions contain the logic that either moves the case to the next state or routes it to an alternate path.

Engineers also spend a substantial amount of time acquiring approvals in order to initiate cases, move them to the next phase, or close them out. The time spent gathering approvals can be reduced by using Web 2.0 technology to “push” them to the approvers. User-defined approval logic may be applied so that for example, if a primary approver is unavailable, a secondary approver would automatically receive the approval request. And, if the approvers don't respond in a timely manner, the software reminds them that they have an outstanding approval task. These features provide a significant reduction in time spent on non-value-added activities associated with the MOC process.

Next generation MOC software also helps reduce engineering time spent on preparing initial requests for change (RFC) by automatically identifying and cross-referencing all configuration linkages and places where an automation parameter is used. This feature significantly accelerates the preparation of RFCs by reducing the time spent researching the scope of the change and also ensures that no critical use of the parameter under change is omitted from the process.

---

The Perils of Unmanaged Change

An example of an incident caused by unmanaged automation change occurred at a petrochemical plant operated by one of our clients. Plant personnel were conducting a routine safety integrity level (SIL) test of an interlock in a safety instrumented system (SIS).

As provided for in the test procedure, they bypassed the output of the SIS so that when they exercised the logic to test the system, the valve would not actually trip. However, when the logic was actuated, to their astonishment, the plant tripped.

Subsequent investigations determined that a well-intentioned individual had made a perfectly logical automation configuration change, but had not documented or communicated it. He had added some operator startup assistance logic in the distributed control system (DCS) that sensed when that specific interlock had tripped in the SIS and then placed all the DCS controllers on the unit in manual and their outputs at 0% to facilitate restart of the unit. His assumption was that if that particular signal in the SIS were true, that the SIS would have already shut the unit down.

Because this startup assistance logic was undocumented, and hence not reflected in the SIL test procedure, it was not disabled prior to the test. The result: the logic shut the unit down and a significant flaring event occurred. The plant not only lost production during the shutdown, but incurred stiff environmental fines from the flaring as well.

There is little doubt that proper automation system MOC would have prevented this incident by requiring peer reviews and approvals of the modifications before implementation, proper documentation of the new logic, modification of the SIL test procedure and a pre-startup safety review.