Business Continuity Plan Tested: 100% Recovery Success

Business Continuity Plan Tested: 100% Recovery Success

Executive Summary

Pacific Ridge, a growing RIA managing over $750 million in assets, faced a critical challenge: a business continuity plan that was documented but never rigorously tested. This left them vulnerable to potentially catastrophic disruptions. Golden Door Asset helped them implement a robust, testable plan leveraging tools like Datto and a framework for regular tabletop exercises. The result? A simulated disaster recovery scenario achieved 100% recovery of data and operations, demonstrating the plan's effectiveness and significantly reducing potential financial risk.

The Challenge

Pacific Ridge had experienced rapid growth in recent years, increasing their AUM from $300 million to over $750 million in just five years. While they had a business continuity plan (BCP) on paper, it hadn't been thoroughly tested or updated to reflect their current operational realities. Their biggest concerns revolved around data loss, operational downtime, and the potential reputational damage associated with a prolonged disruption.

Specifically, the existing BCP lacked:

• Defined Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs): The plan didn’t specify how quickly they needed to recover critical systems and data, nor how much data loss was acceptable. Without these, assessing the effectiveness of the plan was impossible. An estimate from an external audit placed potential losses during a 3-day outage at approximately $1.25 million in revenue due to inability to execute trades and service clients.

• Regular Testing and Drills: The BCP was viewed as a compliance document rather than a living, breathing process. It hadn't been tested in over two years, meaning critical staff members were unfamiliar with their roles and responsibilities during a crisis. This lack of preparedness translated to a heightened risk of errors and delays during an actual disaster.

• Insufficient Data Backup and Recovery Infrastructure: Their reliance on a legacy backup system meant recovery times were unacceptably long, potentially leading to significant client dissatisfaction and churn. A conservative estimate suggested a 10% client attrition rate following a week-long service disruption, potentially costing the firm over $75 million in AUM and associated management fees.

• Inadequate Communication Plan: The plan lacked a clear communication strategy for notifying clients, employees, and regulatory bodies in the event of a disruption. The absence of a proactive communication plan risked fueling client anxiety and damaging the firm’s reputation.

The regulatory landscape also contributed to the pressure. SEC regulations mandate that RIAs have business continuity plans in place. A deficient plan could result in regulatory scrutiny, fines, and potential reputational harm. A mock audit revealed that their current BCP would receive a deficiency, which put them on the path for penalties down the line.

The Approach

Golden Door Asset worked closely with Pacific Ridge to develop a comprehensive and actionable business continuity plan. The approach involved several key steps:

1. Risk Assessment and Business Impact Analysis (BIA): The first step involved a thorough risk assessment to identify potential threats (e.g., natural disasters, cyberattacks, hardware failures) and a BIA to understand the impact of disruptions on critical business functions. This assessment prioritized functions based on their criticality to the firm's revenue and regulatory compliance. For instance, trading and portfolio management were identified as "mission-critical" with a RTO of 4 hours.

2. Defining RTOs and RPOs: Based on the BIA, realistic and measurable RTOs and RPOs were established for each critical business function. For trading, the RTO was set at 4 hours and RPO at 15 minutes, reflecting the need for near real-time data recovery.

3. Developing a Detailed Recovery Plan: A step-by-step recovery plan was developed outlining the specific actions required to restore each critical business function. This included procedures for data recovery, system restoration, communication with stakeholders, and relocation to an alternative workspace if necessary.

4. Selecting and Implementing Data Backup and Disaster Recovery Solutions: After evaluating several options, Datto was selected as the primary data backup and disaster recovery solution. Datto’s image-based backup technology allowed for rapid restoration of entire servers, significantly reducing recovery times. The solution was configured to perform incremental backups every 15 minutes, ensuring minimal data loss.

5. Creating a Communication Plan: A detailed communication plan was created outlining the procedures for notifying clients, employees, and regulatory bodies in the event of a disruption. The plan included pre-drafted templates for email and phone communications, as well as contact information for key stakeholders.

6. Conducting Regular Testing and Drills: The BCP was designed to be tested regularly through tabletop exercises and simulated disaster scenarios. These exercises involved key staff members working through various disaster scenarios to identify gaps in the plan and refine recovery procedures.

7. Documentation and Training: All aspects of the BCP were thoroughly documented, and training was provided to all relevant staff members. This ensured that everyone understood their roles and responsibilities during a crisis.

Technical Implementation

The technical implementation of the BCP focused on leveraging technology to automate data backup and simplify the recovery process:

• Datto SIRIS Implementation: Datto SIRIS was deployed as the primary data backup and disaster recovery solution. Datto's Inverse Chain Technology™ allowed for efficient and reliable backups, minimizing storage space and network bandwidth. The system was configured to perform image-based backups of all critical servers and workstations every 15 minutes, ensuring a RPO of 15 minutes.

• Cloud Replication: Backups were replicated to Datto's secure cloud infrastructure, providing an offsite backup location in case of a local disaster. This replication process was automated and monitored to ensure data integrity.

• Virtualization: Datto's Instant Virtualization technology allowed for the rapid restoration of servers as virtual machines, either on-site or in the cloud. This significantly reduced the RTO, enabling Pacific Ridge to quickly restore critical services. For example, the firm's trading platform was able to be virtualized and accessible within two hours during the test.

• Tabletop Exercises: Annual tabletop exercises were conducted to test the BCP's effectiveness. These exercises involved simulating various disaster scenarios, such as a ransomware attack or a building fire, and having key staff members walk through the recovery procedures. The exercises were facilitated by a third-party consultant with expertise in business continuity planning. These sessions included various "what-if" scenarios to test the plan's agility.

• Documentation and Training: Detailed documentation was created for all aspects of the BCP, including procedures for data recovery, system restoration, communication, and relocation. Training was provided to all relevant staff members to ensure they understood their roles and responsibilities during a crisis. The training emphasized how to utilize the Datto platform for data restoration.

Results & ROI

The implementation of the comprehensive BCP resulted in significant improvements in Pacific Ridge's disaster preparedness and operational resilience.

• 100% Data and Operations Recovery: During a simulated disaster scenario, the firm was able to recover 100% of its data and restore all critical business functions within the defined RTOs. The RTO for the trading platform was achieved in under 2 hours, significantly faster than the 4-hour target.

• Reduced Recovery Time: The implementation of Datto significantly reduced recovery times. The firm was able to restore a failed server in less than 30 minutes, compared to several hours with their previous backup system.

• Improved Employee Awareness and Preparedness: The regular testing and drills increased employee awareness and preparedness. Staff members were more confident in their ability to respond to a crisis and follow the recovery procedures. Post-exercise surveys revealed a 40% increase in employee confidence in their ability to execute the BCP.

• Enhanced Regulatory Compliance: The comprehensive BCP ensured that Pacific Ridge was compliant with all relevant SEC regulations. This reduced the risk of regulatory scrutiny and potential fines.

• Reduced Financial Risk: By mitigating the risk of data loss and operational downtime, the BCP significantly reduced the firm’s financial risk. The firm estimated that the BCP would save them over $500,000 in potential losses in the event of a major disaster. Furthermore, they insured against the disruption with a specific cyber policy.

Specifically, the ROI analysis revealed the following:

• Cost of BCP Implementation: $25,000 (including consulting fees, software licenses, and training).
• Potential Loss Avoided (Major Disaster): $500,000.
• ROI: (500,000 - 25,000) / 25,000 = 19 or 1900%.

Key Takeaways

1. Test, Test, Test: A documented plan is useless without regular testing. Conduct tabletop exercises and simulated disaster scenarios at least annually to identify gaps and ensure readiness.

2. Invest in robust backup and recovery solutions: Cloud-based solutions like Datto offer rapid recovery times and offsite data storage, minimizing downtime and data loss.

3. Define clear RTOs and RPOs: These metrics provide a roadmap for recovery efforts and allow you to measure the effectiveness of your BCP.

4. Communicate clearly: Establish a communication plan to notify clients, employees, and regulators promptly in the event of a disruption.

5. Regularly review and update your BCP: Your BCP should be a living document that is regularly reviewed and updated to reflect changes in your business and the threat landscape. As AUM changes, so must the protection plans.

About Golden Door Asset

Golden Door Asset builds AI-powered intelligence tools for RIAs. Our platform helps advisors automate compliance tasks and manage risk. Visit our tools to see how we can help your practice.