Business Continuity Policy

Purpose

Keep the services we operate available through disruptions, and restore them quickly when they fail, so learners and partners are affected as little as possible.

Scope

The production services in scope of these policies, along with the systems, data flows, people and processes that deliver them. The Disaster Recovery Policy sets the technical recovery targets and backup rules that support this policy.

Out of scope:

• Loss of an entire AWS region. All production systems run in AWS us-east-1. If that region is unavailable as a whole, service is restored once AWS restores the region, and the recovery targets in DR-1 don't apply. Failure of a single availability zone is in scope, and production is built to survive it (BC-2).
• External sites that MIT Learn imports course and content listings from, such as edX.org, YouTube and podcast feeds. We don't run them, so we can't plan their recovery. If one is unavailable, its import fails and MIT Learn keeps showing the listings from the last successful import.
• Local development environments.

Roles and responsibilities

Critical services

These are the services and platform components that delivering the services depends on. Their containers, data stores and dependencies are described in System Architecture.

Requirements

BC-1. Plans are maintained and available on request. This policy, the Disaster Recovery Policy and the Disaster Recovery Runbook together make up our business continuity and disaster recovery plans. They are published on this site and kept current through the review process in Policies. When a partner or client asks for copies, Platform Engineering provides the current versions, with any updates since the last copy, within five working days of the request.

BC-2. Production is built to tolerate the failure of a single component. Production databases run Multi-AZ. The EKS clusters span several availability zones, and user-facing services (web applications, APIs and the API gateway) run as multiple replicas, so the loss of one server or availability zone doesn't take a critical service down. Background workers that run as a single replica are restarted automatically on another node. Applications are stateless, and their state lives in the data stores covered by DR-2.

BC-3. Everything needed to rebuild production is kept in code. Infrastructure, deployment pipelines and configuration are defined in code and deployed through reviewed pipelines (CM-5, CM-6). Container images for every release are kept in a registry, so any service can be redeployed or rebuilt from source without relying on undocumented manual steps.

BC-4. No single person is required for recovery. Production is supported by a weekly on-call rotation in Rootly (IR-1, IR-2). Recovery procedures are written so that any Platform Engineering engineer with the standard production access can carry them out. If on-call can't be reached, anyone can page Platform Engineering through the Emergency Oncall Break Glass Procedure. Platform Engineering checks at least once a year that every member of the rotation has the access the runbook needs.

BC-5. Disasters are handled as incidents. A disaster is any SEV1 incident where a critical service must be restored from backups or rebuilt, rather than fixed in place. The incident leader declares it in the Rootly incident and follows the Disaster Recovery Runbook. Everything else about the response, including roles, severity, escalation and post-mortems, follows the Incident Response Policy.

BC-6. Users and partners are kept informed. During a disruption the communicator updates the status page as required by IR-5. If a SEV1 is expected to last more than two hours, the communicator also notifies MIT Open Learning leadership, who inform affected partners as their agreements require.

BC-7. Third-party providers are part of the plan. We rely on the resilience of the providers listed in Dependencies. An outage at one of them is handled as an incident.

BC-8. The plans are exercised every year. Platform Engineering runs a tabletop exercise at least once a year, walking through a disaster scenario with the runbook, alongside the restore test in DR-7. Gaps found in an exercise or a real incident are filed as GitHub issues with an owner, as for post-mortem actions (IR-9).

Dependencies

Exceptions

Handled as described in Policies: Exceptions.

Related documents

• Disaster Recovery Policy
• Disaster Recovery Runbook
• Incident Response Policy
• Change Management Policy
• OL Production Incident Response Runbook
• Emergency Oncall Break Glass Procedure
• System Architecture

Review