It’s hard to address flaws in an existing system if you don’t know what they are. And the more information that a facility has about existing weaknesses in a system, the better able they will be to target the most important areas for updating. Conducting a comprehensive risk assessment that identifies possible “failure modes”— that is, how a piece of a system could fail to perform its duty— can provide exactly that kind of targeting.
As a best practice, risk assessments have proven value. When they’re detailed and comprehensive, they shed light on what the most pressing system issues are, and make it easier for facility managers to establish priorities for their upgrades— to direct their retrofit efforts to address the most important concerns, and prioritize other areas for subsequent improvement.
Genentech conducted a major risk assessment in late 2007, after becoming clear that at least some of their manufacturing systems would require upgrades to sustain continued plant operations. Their risk assessment showed that critical components of their manufacturing systems were either already obsolete, or getting very close. The impact of the failure of those parts of the system had the potential to be disastrous for the facility, and the company’s overall manufacturing supply chain. The risk assessment results clearly highlighted those software applications and system hardware components that were the most important and necessary priorities for the retrofit.
#2: Review vendor offerings for compatibility with your legacy systems, databases, and software applications.
Many companies using “legacy” equipment— that is, equipment that’s either already out of date or well on its way— will find that their vendors are now providing updated, upgraded versions of the same. As a best practice, it is recommended that companies carefully review these potential upgrade candidates, to determine if those new offerings have maintained compatibility with the facility’s legacy systems, to reduce the need to re-train plant personnel and hopefully cut down on the impact and scope of the retrofit effort..
Finding compatible systems can have big benefits for pharmaceutical manufacturers. By adopting an upgrade path supplied by an existing vendor, companies can make the retrofit process faster, more affordable, and can often minimize production downtime while the new systems and components are implemented.
Vendors often also still have resources and employees dedicated to maintaining or upgrading legacy systems. Planning a retrofit is always easier when the vendor is already familiar with the legacy systems and how best they can be replaced by, or integrated with new ones. Finally, because vendor product roadmaps often anticipate customer upgrades, suppliers may already have transition strategies and resources in place for performing system upgrades and replacements.
Genentech, during their upgrade, moved to new application software from the same vendor— WonderWare, a branch of Schneider Electric. The replacement software was developed in a newer technology, but had been derived from the original design, so the old and new systems maintained a high degree of compatibility, allowing the transfer process between versions to be expedited and made significantly easier.
In our last blog, we provided an introduction to the first two of eight best practices for effectively retrofitting manufacturing systems in a pharmaceutical production facility. In this one, we’ll explore the next two best practices and how they’ve been used effectively in our Genentech case study.
#3: Explore automated migration tools to transfer data between systems.
Retrofitting legacy systems can be challenging due to the amount of configuration information that exists in the old system and will need to be re-entered into the new system. Fortunately, there is a wealth of software tools available that are specifically designed to move configuration and systems data from one platform to another, and often the major software applications will already have similar capabilities built in, such as XML export/import functions. Utilizing existing data transfer tools, and exploring opportunities to fill in gaps with customized tools, can help maintain data integrity between legacy and new systems and smooth the retrofit process.
The benefits of this practice are clear, and numerous. Automated transfer of data eliminates the potential for transcription errors or omissions, reducing online testing requirements and overall system downtime. It also makes transferred data more secure and safeguards intellectual property such as recipe procedures, product structures, and other information essential for manufacturing. Testing a software migration tool is typically less expensive and less time-consuming than verifying manually-entered data. And an automated migration can be performed multiple times, so system managers can build confidence that the critical data has been migrated reliably and correctly.
Genentech successfully applied this practice during their retrofit, creating a database migration tool to transfer all the relevant configuration information from the legacy software application to the new application, which made the data transfer both more reliable and efficient, as creating and testing the software tool took significantly less time than it would have to manually re-enter the data.
#4: Plan to synchronize parallel system development and documentation systems.
When a retrofit occurs, one of the biggest challenges facing system managers is the need to maintain the legacy system, and keep it operating at fullest capacity, while simultaneously dealing with development of the new system. The existing system will still require regular updates to ensure reliable functionality and maintain compliance with regulatory and safety requirements. System managers, then, should have a working plan to manage development of both systems concurrently, at least until after the transition is complete.
Managing parallel development and documentation streams
The major benefit of having such a plan is that it provides the business with flexibility, so it can continue adapting to changing user requirements, compliance regulations, and market conditions. In short, it keeps the business operable and responsive even while upgrades are being developed.
In Genentech’s case, because the retrofit process was expected to take two to three years, it was unfeasible for modifications and updates to existing systems to be completely suspended. The team executing the retrofit employed multiple synchronized “development cycles” to ensure ongoing changes to user requirements and system functionality were implemented in both old and new systems.
In our last installment, we covered the third and fourth best practices adopted by Genentech and proven to be effective in retrofitting pharmaceutical manufacturing facilities.
#5: Benchmark current system performance, and conduct performance testing of the new system prior to cutover.
A retrofit project has the benefit of having access to years of historical operating data. Instead of starting from scratch, and installing a system with limited idea of what its full-scale performance characteristics need to be, a retrofit has the advantage of being able to look back through years of actual data, to see how the system needs to perform, both cumulatively and on an everyday basis. And companies can look not only at one area of performance, but at larger trends throughout the system’s history.
By defining performance “benchmarks” based on the legacy system’s operational history, a company can ensure that the new system will be at least as effective, and hopefully more, than the system being replaced. Establishing these benchmarks and collecting performance metrics from the legacy system prior to, or early in the development of the new system can be extremely useful. Thorough study of the existing system can help those working on the retrofit project to achieve a more complete understanding of how the new system will need to function. It can also help system developers in making functional and structural changes in the new system to increase its overall efficiency, while decreasing the likelihood of inconvenient last-minute design changes to address performance issues.
During Genentech’s retrofit, they prioritized the goal of ensuring that the new manufacturing system either met or exceeded the performance capabilities of the legacy system. They conducted benchmark testing, which allowed them to verify those performance targets in the legacy system and ensure they were achieved in the new system. Early performance testing on the new system also helped in determining the optimum system architecture and infrastructure configuration parameters.
#6: Involve users and key stakeholders early and extensively throughout the process.
Retrofits also have the advantage of being able to leverage a knowledgeable and experienced staff, who are very familiar with the operation of the plant. By getting those individuals involved in the retrofit effort, the facility can benefit from their expertise by assisting in pilot studies, benchmark testing, and design reviews, as well as utilizing their experience and knowledge for drafting initial plans for the retrofit.
The benefits of engaging these stakeholders early and throughout the retrofit effort are clear, and significant. Involving experienced operators and system users can help identify important gaps or weaknesses in the legacy system’s functionality, and reduce the possibility of misunderstandings about system requirements. And technical specifications for legacy systems rarely provide a complete and thorough understanding of a system’s functionality; pulling in experienced users will allow the retrofit project to better understand not only ‘how’ the system operates, but ‘why’ as well, and provide a more complete view of what’s needed to improve the plant.
Genentech involved key stakeholders in the retrofit process from its start, creating a “user team” with representatives from a many different departments. User team members helped with conducting pilot studies, benchmark testing, and design reviews of the new and old systems. Close collaboration with the retrofit team ensured that all site personnel were engaged with, and informed about, the retrofit process; that they understood and approved of changes being made to system functionality; and the overall importance of the retrofit project to the facility.
Throughout this series, we’ve covered six of the eight best practices for retrofitting an automated pharmaceutical manufacturing facility. In this installment, we’ll cover the last two practices to make your retrofit as efficient and effective as possible.
#7: Leverage computing virtualization technologies to save space, time, and money.
Manufacturing control and I/O rooms are generally packed full with computer racks, equipment, and cabling. Finding the physical space for staging a new computer system as part of the retrofit process can be challenging. But computer virtualization technology has advanced significantly in the past several years, and facilities can take advantage of it to greatly reduce the physical space requirements of the new system. Virtualized computing platforms are capable of running multiple operating systems and software applications at the same time. This saves space, time, and expenses for the company doing the retrofit, and increases system administration efficiency both during the retrofit project and subsequent lifecycle of the new system.
Using virtualization technologies delivers significant benefits. Centralized management of the manufacturing system— from maintaining anti-virus protections to updating the underlying operating system software— makes system administration simpler and more efficient. And employing standardized server configurations helps to ensure the new systems adhere to design and technical specifications. The savings in physical space, power, and cooling are significant, as are also the reduced costs during both implementation and subsequent maintenance of the system.
Genentech’s retrofit project required deployment of a large number of new computer server platforms, which would have introduced considerable increases in both system administration complexity and physical space requirements. Their use of virtualization technology allowed them to save substantial time and space, and helped reduce costs for both the retrofit project and long-term support.
#8: Develop “fall back” contingency plans for recovering from critical post-deployment problems.
You’re probably familiar with Murphy’s Law: whatever can go wrong, will go wrong. Even the best-laid plans can’t anticipate every possible problem or pitfall when installing a new manufacturing system. Contingency planning, while not a fun step for project teams, is undoubtedly a necessary one. Having backup plans in place is a prudent and forward-thinking move for facilities, especially given that regulatory agencies can shut down a pharmaceutical manufacturing plant if the new systems don’t perform according to their functional requirements and approved licenses.
By creating contingency plans for reversing the retrofit in advance, project managers can ensure that even if disaster strikes after the new system has been implemented, they have a viable path forward for trying to solve the problem. Such plans ensure that if the new system needs to be backed out, the plant will be able to resume operations with the legacy system as quickly as possible, ensuring the least possible impact on both the company’s global supply chain and overall profitability.
Genentech developed such a plan, which specified the activities and proper sequence for reversing all of the retrofit upgrades and ensuring that the legacy manufacturing system could be restored with full functionality. Although never employed, its existence itself was an important and valuable asset for the company, and would have proved essential if deployment if the new system had encountered unexpected catastrophic results.
These best practices, when put together, helped in creating a comprehensive strategy for carrying out as efficient and effective a retrofit project as possible, and the resounding success of Genentech’s project only served to reinforce their benefit. If your pharmaceutical manufacturing facility is considering manufacturing system upgrades, it would be wise to keep these practices in mind when preparing your plans.
Continue at: https://blog.schneider-electric.com/life-science/2016/08/17/getting-started-best-practices-1-2-pharmaceutical-manufacturing-upgrade-project/
The text above is owned by the site above referred.
Here is only a small part of the article, for more please follow the link