Skip to content

Incident Playbook — Denial of Service

One page, for reading during an incident. Full process: Incident Response Procedure.

What this looks like at Soon

Soon's customers use the platform to run shifts. An outage during a schedule publish or a shift handover is disproportionately painful, which is why availability is treated as a security property here.

Three distinct causes wear the same costume, and telling them apart is the first job:

  1. Volumetric attack — traffic flood from outside.
  2. Application-layer abuse — expensive queries, scraping, or a runaway integration hammering the API.
  3. Self-inflicted — a bad deploy, a migration holding locks, an unbounded query. This is the most likely one, and the response is completely different.

Detection signals

  • Status page checks failing; customer reports
  • CloudWatch alarms on latency, error rate or database CPU
  • Sentry error spike
  • ALB metrics: request count far above baseline, or 5xx climbing
  • Cloudflare analytics showing an unusual traffic pattern

First 30 minutes

# Action Who
1 Declare. Slack #Security, start a timeline Whoever notices
2 Establish which of the three it is before mitigating — check whether a deploy or migration went out in the last hour. Blocking traffic that is actually your own bug wastes the window Thomas
3 If self-inflicted: roll back. Fastest correct action Thomas
4 If external: engage Cloudflare — rate limiting, bot protection, or "Under Attack" mode on the affected hostnames Thomas
5 If application-layer: identify the source in ALB and application logs, then rate-limit or block that key, tenant or IP range Thomas
6 Communicate. Update the status page early. Customers forgive an outage far more readily than silence Olaf
7 Scale if it helps — ECS/Fargate autoscaling; watch the database, which is usually the real ceiling Melvin

Then

  • Consider whether it is a distraction. A noisy outage can be cover for something quieter. Check GuardDuty and CloudTrail for authentication or data-access anomalies during the same window.
  • Record what the customer impact actually was — duration, tenants affected, and whether it breached the 99.5% commitment. That feeds the SLA position and the availability review.
  • Raise a corrective action for the underlying cause, not just the mitigation.

Standing mitigations

Cloudflare in front of the platform; ECS/Fargate autoscaling; Multi-AZ RDS; CloudWatch alarms routed to the team; a public status page at status.soon.works.

Change log

Version Date Author Comments
0.1 2026-08-22 ISMS First version — separates the three causes, since the most likely one is self-inflicted and the response differs.