Incident Response Policy

Purpose

Detect, respond to and recover from incidents that affect availability, integrity or confidentiality of our systems quickly and consistently, and learn from every significant incident.

Scope

Any unplanned interruption or degradation of a production service in scope of these policies, and any suspected or confirmed security incident: unauthorized access, credential compromise, data exposure, or malicious activity against our systems or accounts.

Roles and responsibilities

These are the incident roles in Rootly, described in the OL Production Incident Response Runbook.

Severity levels

Requirements

IR-1. Alerts route to on-call through Rootly. Production alerts from Grafana, Sentry and uptime checks are sent to Rootly, which pages the on-call engineer according to the escalation policies kept in code (Managing Rootly). Responders follow the On-Call Runbook for known alerts.

IR-2. Acknowledge within the target time.

IR-3. Open an incident when a problem isn't mitigated quickly. If a production problem isn't mitigated within five minutes, or any security incident is suspected, open an incident with /rootly new in #team-engineering. Rootly creates the incident channel, timeline and action items. All incident communication happens in that channel.

IR-4. Follow the incident phases. Incidents move through Triage, Active, Mitigated and Resolved in Rootly. The leader sets the severity during triage and updates it if the impact changes.

IR-5. Keep users informed. For SEV1 and SEV2 incidents, the communicator updates the status page with /rootly update_status_page when the incident starts, when it's mitigated and when it's resolved.

IR-6. Contain security incidents first and preserve evidence. For a suspected compromise, follow the Compromised Account Response Runbook. In order:

1. Contain the attacker's access: disable sessions, revoke tokens and rotate exposed credentials.
2. Preserve evidence: keep the logs, the affected accounts and any other artifacts the investigation needs. Disable affected accounts; don't delete or retire them. Export the logs for the whole period of the incident, starting from the earliest suspected activity, before their retention period can remove them: application and gateway logs in Grafana, CloudTrail, and any S3 access logs and Keycloak or Vault audit logs that cover the affected systems. Ask MIT IS&T whether anything else must be kept.
3. Investigate and recover only once access is contained and the evidence is safe.

IR-7. Escalate security incidents. Confirmed security incidents, including any exposure of personal data or credentials, are reported to MIT IS&T Security (security@mit.edu) by the incident leader as soon as possible, and no later than 24 hours after confirmation. Platform Engineering decides with MIT IS&T whether users or other parties must be notified. When personal data may have been exposed, the incident leader gathers what that decision needs:

• how many users are affected;
• which kinds of personal data were exposed or accessed;
• where the affected users live, since notification rules depend on it;
• whether payment card data was involved; and
• how the access happened, for example a stolen password or token, a vulnerability, or a misconfiguration.

IR-8. Write a post-mortem. A blameless post-mortem is required for every SEV1 incident (which includes every confirmed security incident). For SEV2 incidents, one is written if it's warranted, as decided by the incident leader. For SEV3 and SEV4 it's optional. The incident leader publishes it in Runbooks & Post-Mortems within five business days of resolution. It covers the summary, impact, detection, root cause, timeline, mitigation and corrective actions. The 2026-08-10 post-mortem is a good example.

IR-9. Track corrective actions to completion. Corrective actions from a post-mortem are filed as GitHub issues with an owner, and Platform Engineering reviews open actions until they're closed.

Exceptions

Handled as described in Policies: Exceptions.

Related documents

• OL Production Incident Response Runbook
• On-Call Runbook
• Compromised Account Response Runbook
• Emergency Oncall Break Glass Procedure
• Managing Rootly
• Change Management Policy
• Business Continuity Policy
• Disaster Recovery Policy

Review