Don’t Replace it Yet! How One Plant Cut Downtime by 96%
PDA helped a manufacturer stabilize a troubled automation system, dramatically reduce production losses, and bridge the gap to a future controls modernization.
When manufacturers talk about difficult controls modernization projects, the picture that often comes to mind is an aging facility filled with obsolete hardware, outdated software, and decades of accumulated programming changes. This wasn’t that kind of project. The manufacturing facility was only a few years old. But a critical production system had struggled almost from the beginning.
The automation was overly complex in places, portions of the control logic were incomplete, operators had difficulty identifying the root cause of faults, and recovering from some failures required intervention from controls and maintenance personnel. Problems within the slurry system could affect extrusion and ultimately interrupt production across the facility.
The manufacturer knew the long-term answer would eventually involve a much more comprehensive controls modernization. That project, however, was still several years away. Production couldn’t wait. The immediate challenge was different: Could the existing system be made reliable enough to carry the plant until that larger modernization could take place? That was the assignment given to Process and Data Automation (PDA).
Diagnose Before You Program
The company needed an integrator that understood not only the underlying controls technologies, but also its established automation standards and the slurry process itself. PDA brought both. PDA was already familiar with the manufacturer through work at other facilities. That experience mattered. Rather than immediately beginning to modify code, PDA started with an investigation phase. The team documented issues, evaluated the existing architecture, and worked with plant and corporate personnel to prioritize the problems creating the greatest operational impact.
The resulting project was deliberately constrained. This was not going to be a plant-wide rewrite, nor was PDA being asked to install a new controls platform. The objective was to improve the system already in place.
“We knew going in that we weren’t designing the system we would build from scratch,” said Dan Cochran, PDA project manager. “Our job was to understand what was there, identify what was really hurting production and make the right changes without creating another layer of complexity.”
The investigation uncovered several recurring themes. Single events could generate multiple alarms, making it difficult for operators to determine which condition had actually initiated a failure. Interlocks between upstream and downstream equipment were incomplete or weren’t always clearly represented on the HMI. Phase Manager faults could leave portions of the process in states operators couldn’t easily recover from. Some process loops and parameters needed additional tuning. Even routine operator interaction had become a source of frustration. Some Phase Manager pop-up screens required more than 90 seconds to load.
Individually, each problem created delays. Collectively, they made a critical production area difficult to operate and support.
Making Problems Easier to Find and Fix
One of PDA’s priorities was giving operators better information when something went wrong. The team performed an alarm audit, corrected alarm-to-HMI relationships, and aligned descriptions with the manufacturer’s established alarm standards. PDA also added functionality that allowed operators to move directly from an alarm to the appropriate equipment detail screen.
More importantly, the team addressed cascading alarm conditions so operators could more readily identify the fault that initiated an event rather than sorting through the secondary alarms that followed it. “The system could tell you a lot of things were wrong without making it obvious which one happened first,” Cochran said. “If we could make the original fault clear and give the operator a direct path to the affected equipment, we could take a lot of time out of troubleshooting.”
PDA took a similar approach to Phase Manager recovery. Incomplete logic could cause batching and extrusion processes to deadlock, and attempts to recover from a fault could sometimes leave phases out of sequence. In one documented situation, fully resetting affected equipment could take at least two hours. PDA modified the recovery logic so that, after an alarm condition was resolved, the Phase Manager could return to the appropriate step or hold index and restart in the proper sequence. The goal wasn’t simply to prevent every possible fault. It was also to make the inevitable ones easier for plant personnel to understand and recover from without escalating every problem to specialized controls resources.
Improving the System Without Replacing It
The work extended beyond alarms and fault recovery. PDA reviewed interlocks between upstream and downstream systems and reconciled what existed in the PLC with what operators could see through the HMI. Automation and process personnel worked together to make sure the interlocks reflected how the equipment actually needed to operate.
The team also tuned process loops and operating parameters, establishing appropriate actions around low and high level conditions to reduce problems such as overfilling or running equipment empty.
Another problem involved Clean-in-Place systems that could interfere with one another when operating concurrently. PDA corrected sequencing and token-handling logic, added retry and reset functionality, and improved arbitration so multiple CIP processes could operate without deadlocking over shared equipment.
And those 90-second Phase Manager screens? PDA reorganized and optimized the pop-ups so less information needed to load at once and portions could load in parallel. Load time fell from more than 90 seconds to approximately seven.
None of those changes required replacing the underlying controls infrastructure and that was precisely the point. “This wasn’t about showing the customer everything we could redesign,” Cochran said. “Success meant getting the performance they needed from the assets they already had and doing it in a way their operators and controls people could support.”
From 548 Minutes to 21
The effect on production was dramatic. Shortly before the project, the manufacturer recorded approximately 548 minutes of production loss attributed to controls issues in a single month. The following month saw 487 minutes, followed by 427 minutes the month after that. As PDA’s changes were progressively deployed, the numbers began falling. Controls-related losses dropped to 204 minutes, then 93, 97, and ultimately just 21 minutes per month. From the beginning to the end of that period, controls-related lost minutes declined by approximately 96 percent.
The manufacturer’s EUPS measure, used to track the impact of unplanned production stops, improved from 10.49 percent to 2.1 percent. The internal project analysis calculated $1.42 million in savings from June through the end of the reporting period against a project cost of approximately $330,000. The economics are particularly significant in a process where downtime compounds quickly. If the slurry system isn’t operating, downstream production is affected as well. Every minute that product isn’t reaching packaging represents lost production opportunity.
But the improvement wasn’t measured only in dollars or downtime. The project also reduced escalation calls to factory controls personnel, improved operator effectiveness and sharply decreased extrusion downtime associated with the slurry operation.
Giving the System Back to the Operators
That may be one of the project’s most important outcomes. A production system can technically function and still perform poorly if operators can’t determine why it stopped, maintenance personnel can’t quickly diagnose problems or every abnormal condition requires help from a controls specialist.
PDA’s work attacked those issues together. Better alarms made problems easier to understand. First-fault identification made root causes easier to find. Direct navigation helped operators get to the affected equipment faster. Improved interlocks prevented some problems from occurring in the first place. Recovery logic made others easier to clear. Faster HMI performance reduced the friction operators experienced simply interacting with the system.
The manufacturer’s internal assessment identified quicker fault resolution, reduced manual effort, and less dependence on specialized troubleshooting expertise among the strategic benefits of the project. For PDA, that human element was just as important as improving the code. “You can make a lot of changes inside a PLC, but if the people running the process still can’t tell what’s happening, you haven’t solved the whole problem,” Cochran said. “We wanted the system to help operators operate the plant instead of forcing them to become controls engineers every time something stopped.”
A Bridge to What’s Next
The project wasn’t intended to make the manufacturer’s long-term modernization plans unnecessary. Eventually, the facility is expected to move toward an automation architecture more closely aligned with their own worldwide programming standards. PDA’s job was to help make the existing system productive until that transition makes sense. That distinction is important.
In industrial automation, the technically ideal solution isn’t always the right solution for the moment. Capital schedules, production requirements, available resources, and larger corporate initiatives all influence when a comprehensive modernization can occur. Sometimes the greatest value an integrator can provide is knowing what not to replace.
PDA entered this project knowing the existing automation had limitations. The team didn’t attempt to turn a bridge project into a complete redesign. Instead, it investigated the system, identified the problems creating the greatest operational losses and concentrated engineering effort where it could produce measurable results.
The result was a dramatic reduction in controls-related downtime, improved operator effectiveness, fewer demands on specialized support personnel and more than $1 million in documented savings, all while continuing to use the plant’s existing controls infrastructure. The manufacturer’s internal project assessment specifically noted that the performance improvements required no significant hardware expenditure.
For a facility that ultimately expects to replace much of the automation, that might seem counterintuitive. But waiting several years for the perfect system while absorbing today’s production losses wasn’t much of a strategy. PDA helped the manufacturer make the system it had work, buying the time it needed to eventually build the system it wants.
