Three Companies that had to Prove their Disaster Recovery Plan would Work

Pat O’Day, CTO, Bluelock
392
632
127

Pat O’Day, CTO, Bluelock

Given how much modern business relies on technology, you know how important it is to have a Disaster Recovery (DR) plan in place for your IT systems. Imagine if someone asked you today to prove your DR plan works. Worse yet, what if you had to prove it not only internally, but to your constituents? In a decade of working with clients at Bluelock, I’ve found it increasingly common – and in some cases required–that companies provide solid evidence of a successful recovery to auditors, insurers, investors, board members, regulators and customers.

More security incidents and system outages have occurred and made media news. It’s clear that our clients are responding to a broader set of risks than their original disaster recovery plans were designed to address. Security penetrations, DOS attacks, crypto lockers, prolonged network or power outages, and hardware or software failures to heavily-centralized virtual systems have a far greater likelihood and systemic impact than the traditionally-isolated systems and weather-related events most plans are designed around. Consequently, clients have been required to prove recovery success, recovery quality (data) and recovery time to various groups.

Based on Bluelock’s body of knowledge working with companies in modernizing their recovery approaches every day, I’ve compiled some specific areas of questioning that may help you better assess your plan to deter­mine if you are at more risk than you know. To tease out a bit more color and nuance around each point, I’ve included examples from Bluelock’s cli­ent base as a Disaster Recovery-as-a-Service (DRaaS) provider, scrubbed for privacy and security.

  Only a consistently successful recovery of people, processes, and technology will satisfy external parties that the recovery plan is effective 

If you’re a business leader this should help you open a dialogue with your technology team. If you’re on the technology team, this should verify confidence or at least increase awareness. If you happen to be reading this as one of the constituents I mentioned, it’s my hope that these examples will encourage a more productive dialogue with your business and technology leaders. In the end, we are all on the same team and want the best for business and the technology used to support our success.

Proof of Protection

Shortcuts and DR tend to go hand-in-hand. The shortcuts are possible because most companies don’t have to prove their DR plans have worked. DIY recovery environments tend to be outdated or under-scaled due to budget pressure and views of DR as insurance for something that will likely never happen. These shortcuts also impact security in the DR environment. The challenge is that whether a business is using its recovery environment at that moment or not, it most likely contains a complete copy of production data. Thus, protection and security for your recovery environment is the last thing on which to cut corners. As evidenced by some of the more recent large scale data breaches, it’s clear that the intruder community knows it’s easier to attack systems that are adjacent to production than a heavily-guarded proverbial front door. Proving the protection and security of your production environment can be daunting enough, but proving it in recovery can be even more challenging given the system is offline 99 percent of the time.

One of our clients in the legal industry faced external scrutiny from one of their largest customers when an audit of that customer’s internal DR plans realized a significant dependency on third-party services. While third-party services were originally out of scope, this newly-identified risk placed legal services as also accountable for recovery planning, so our client had to prove their DR plan worked and was secure.

For these reasons, they decided to deploy a hybrid configuration where they own and operate their network and security infrastructure, but deploy it around their DR environment at Bluelock. This best-of-both-worlds approach allowed them to leverage their strengths combined with those of a provider with a track record of working with broad sets of applications, security requirements and their own PMO office for secure change management.

Proof of Compliance

HIPAA/HITECH is pretty clear on disaster recovery. You have to have it for any system that is part of the overall healthcare delivery system. It has to work. It has to be proven by the time you are audited.

Doing this while maintaining rapid company growth can be difficult. This was true for a major healthcare provider whose expanding technology environments threatened the risk of non-compliance. They needed to prove to regulators that their environments met requirements, using only existing staff­ing and expertise. They also wanted to leverage the funding that was going to be liberated when they closed down their secondary datacenter site.

The IT department explored doing DR in-house, but realized the task of building a second site for recovery was too time consuming with other priorities. They needed a DR solution that would meet their HIPAA/ HITECH regulations, yet flexible enough to scale new technologies for innovative healthcare.

Leveraging Bluelock’s existing recovery expertise and our elastic pay-for-what-you-use recovery platform achieved not only compliance, but a position for growth and extreme efficiency. Having the test certification process signed off by both parties and fully documented with HIPAA/HITECH controls certified and audited under our SSAE16 Type2 SOC2 handily exceeded audit requirements.

Proof of Recovery

Testing is not proof. Having a copy of your data somewhere else does not ensure that applications can return to service. Being able to “power on” systems at the recovery site does not guarantee they will come back online within the specific time period that the business requires. Only a consistently successful recovery of people, processes, and technology will satisfy external parties that the recovery plan is effective.

One of our clients, a national research firm, needed to gain an insurance renewal that included business impact insurance, which protects revenue in the event of a disaster. They needed to prove continuously successful disaster recovery tests for key systems that supported their client-facing business applications. Each test, done every six months, had to verify that the most recent data and systems could recover within a given timeframe. To show this proof, both our recovery team and someone from the client’s leadership signed the testing certificates, which verified consistent recovery success.

Conclusion

If you haven’t already noticed the trend, effective and flexible DRaaS solutions and expertise help a variety of companies deploy and prove their DR strategies. While the right choice for DR is often dictated by business-specific priorities, timing, budget, and expertise, a good provider alleviates a lot of these pain points.

Read Also

Moving Towards a More Agile and Outcomes Based Model

Raiford Smith, Vice President, Energy Technology and Analytics, Entergy