#Industry News
Expect the Unexpected: Disaster Prevention and Recovery
An expert team creates a robust plan, and is on the ground when you need them most
We live in a world of uncertainty. Disaster events, such as fires, severe weather, blackouts, cyber-crime, accidental damage, and more recently, the COVID-19 pandemic, occur with little or no warning and the effects can be catastrophic— especially for laboratories. These events bring operations to a halt, creating significant delays in production, increasing time to market, creating drug shortages, and contributing to potentially devastating financial losses. While it is not possible to anticipate every potential disaster, developing a recovery plan and partnering with a highly trained team of experts to execute recovery efforts will ensure business continuity when a serious event occurs. PerkinElmer’s OneSource® Lab Services offers an integrated solution, combining business continuity and disaster recovery planning to help laboratories manage or eliminate potential risks, with a team of experts ready to deploy a robust response in the event of a disaster.
Instruments, Computer Systems, and Compliance Are at Risk
The laboratory environment is very different from an office environment. It contains instrumentation and computer systems on their networks or managed separately from the rest of the company. Without proper planning and recovery strategies, natural or man-made disasters can have devastating, long-lasting consequences for laboratories. When instruments, computer systems, or compliance efforts are compromised, significant downtime, issues with quality, and financial losses result. It can take years to recover from such an event, and even longer if the organization’s reputation is affected.
Sophisticated instruments are highly valuable assets and can be impacted by disaster events. In addition to direct damage in the event of a fire, for example, indirect damage can occur as authorities work to control the fire. Or, during a ransomware cyber attack instrumentation is directly impacted and computer systems become inoperable. Depending on the situation, a plan is needed to perform an assessment of the damage and then define what is needed to recommission the instruments, such as careful cleaning, repairing damaged components, or declaring a complete loss that requires replacement.
Many of these instruments also have dedicated computers to initiate and control analysis, as well as store and manipulate large amounts of data. In the event of a cyber attack or other disaster that compromises computer systems, irreplaceable data may be lost. Proper planning, including implementing cloud or virtualization technology or offline backup solutions can prevent data loss, and engaging a reliable recovery team with specialized laboratory IT expertise can accelerate the process of recovery and restoration.
Laboratory remediation and potential relocation following a disaster event also poses challenges for organizations aiming to maintain compliance. Following remediation, laboratory systems must be audited and revalidated to ensure data integrity. A credible partner can determine a compliance quality audit post-remediation.
Many disasters render the facility no longer fit for use, sometimes resulting in the construction of a new facility. Recovery plans need to take this into account and have built-in solutions for dealing with such situations. Requirements need to be defined so that staging facilities can be identified to support laboratory operations while the new laboratory is under construction. This is one of the more important requirements to minimize disruption to R&D. Further, instrument, computer, and compliance states must be assessed to create a disaster baseline. Relocation requires instrument decommissioning, packing and transport to a new or temporary location and upon arrival, instruments require recommissioning and potentially re-qualification. The entire relocation process must also be completed quickly, to minimize the significant costs associated with downtime, and requires a recovery team with expert knowledge, processes, and tools.
Disaster does not discriminate between laboratory assets. Costly, sophisticated instruments, irreplaceable data, and compliance efforts are all at risk. Developing a comprehensive business continuity and disaster recovery plan protects these assets, and directs restoration efforts to bring critical operations back online.
Creating a Business Continuity and Disaster Recovery Plan
Laboratories are complex, fast-paced environments, with numerous personnel, instruments, and workflows. Following a disaster event, it is not feasible to restore every individual laboratory function in the short-term. Rather, the goal of a business continuity and disaster recovery plan is to restore critical functions to support business continuity and minimize losses, while working toward full long-term recovery.
One of the first steps in creating a plan is to identify critical functions within the lab, as well as the responsible personnel, the frequency at which they are performed, and whether they could be performed at another site. Next, it is important to determine the costs incurred if the laboratory was unable to perform these tasks. In addition to lost revenue, fines or penalties, legal liability, personal damage, public harm, and poor public image should also be considered. Determining whether these critical functions could perform at reduced operating efficiency, and whether they require other external resources, technology, or space, will be helpful for developing a plan.
With unique knowledge across a wide range of laboratory technology, compliance, and information services, OneSource can help laboratories develop a business continuity and disaster recovery plan, as well as perform key workflow and process assessments, lab readiness assessments, risk assessments, gaps analysis, assign priority listings, identify critical off-site temperature controlled storage requirements, and more. With a plan in place, the team can mobilize quickly to begin remediation efforts to maintain lab functionality and integrity.
Putting Plans Into Action
PerkinElmer’s OneSource® Professional Services offers an integrated solution to combine business continuity and disaster recovery planning with a qualified team prepared to execute recovery solutions.
Following a devastating fire in a large cancer research center, the PerkinElmer team was enlisted to take over full project management of the recovery effort. These efforts have many moving parts, including creating a complete inventory, triaging instruments and performing cleaning, testing, repair, decommissioning and relocation, as well as identifying instruments requiring original equipment manufacturer (OEM) support and engaging the appropriate companies. The team successfully carried out these tasks in just four months, and completed recommission and re-qualification within 12 months.
The team was also enlisted as an IT partner to accelerate restoration and recovery efforts following a cyber attack that affected 6000 customer research systems. Activities such as rebuilding computer configurations and installing software are all part of the recovery plan— and require mobilization of laboratory IT specialists around the globe. In this case, the team was able to accelerate the recovery time from years to just six weeks.
Uncertain Times and a “New Normal”
The ongoing COVID-19 pandemic has affected laboratories and organizations around the globe. For those deemed essential services, the pandemic poses new challenges to maintaining day-to-day operations. To support pharmaceutical laboratories and manufacturing sites, PerkinElmer is providing instrument maintenance services and IT support, as well as performing compliance work (such as instrument qualifications and system validations). PerkinElmer scientific staff are also working in laboratories to maintain operations, and working with laboratory leaders to develop a “new normal”, by developing processes to maintain production while keeping staff safe.
Disaster events can cause significant financial losses, compliance risks, brand damage, and threaten the future of the laboratory organization. The overall goal of disaster prevention and recovery measures is to maintain business continuity, and support long-term recovery. PerkinElmer’s OneSource® Professional Services makes it possible to plan for the unexpected, and initiate a response to keep your lab functioning and mitigate losses if the unthinkable happens.
