Skip to content

Incident Playbook — Denial of Service

One page, for reading during an incident. Full process: Incident Response Procedure.

What this looks like at Soon

Soon's customers use the platform to run shifts. An outage during a schedule publish or a shift handover is disproportionately painful, which is why availability is treated as a security property here.

Three distinct causes wear the same costume, and telling them apart is the first job:

  1. Volumetric attack — traffic flood from outside.
  2. Application-layer abuse — expensive queries, scraping, or a runaway integration hammering the API.
  3. Self-inflicted — a bad deploy, a migration holding locks, an unbounded query. This is the most likely one, and the response is completely different.

Detection signals

  • Status page checks failing; customer reports
  • CloudWatch alarms on latency, error rate or database CPU
  • Sentry error spike
  • ALB metrics: request count far above baseline, or 5xx climbing
  • Cloudflare analytics showing an unusual traffic pattern

First 30 minutes

# Action Who
1 Declare. Slack #Security, start a timeline Whoever notices
2 Establish which of the three it is before mitigating — check whether a deploy or migration went out in the last hour. Blocking traffic that is actually your own bug wastes the window Thomas
3 If self-inflicted: roll back. Fastest correct action Thomas
4 If external: engage Cloudflare — rate limiting, bot protection, or "Under Attack" mode on the affected hostnames (front-end hosts only until the API is proxied — S-426) Thomas
5 If application-layer: identify the source in ALB and application logs, then rate-limit or block that key, tenant or IP range Thomas
6 Communicate. Update the status page early. Customers forgive an outage far more readily than silence Olaf
7 Scale if it helps — ECS/Fargate autoscaling; watch the database, which is usually the real ceiling Thomas

Then

  • Consider whether it is a distraction. A noisy outage can be cover for something quieter. Check GuardDuty and CloudTrail for authentication or data-access anomalies during the same window.
  • Record what the customer impact actually was — duration, tenants affected, and whether it breached the 99.5% commitment. That feeds the SLA position and the availability review.
  • Raise a corrective action for the underlying cause, not just the mitigation.

Standing mitigations

Cloudflare in front of the web front end and marketing site (the production API hostname is not yet proxied — S-426); ECS/Fargate autoscaling; Multi-AZ RDS; CloudWatch alarms routed to the team; a public status page at status.soon.works.

Change log

Version Date Author Comments
0.1 2026-08-22 ISMS First version — separates the three causes, since the most likely one is self-inflicted and the response differs.