Skip to content

Operating Procedure — Production Database Access

Purpose. Defines how engineers reach the production database and how that access stays least-privilege and auditable, plus the one-time runbook to remove the public endpoint. Satisfies A.5.37 (documented operating procedures), A.8.2, A.8.20, A.8.22 and SOC 2 CC6.1 / CC6.6.

Status: DRAFT — the remediation has LANDED. Written 2026-08-08 while the production database was still publicly reachable; the public endpoint was removed on 2026-08-19 and the dsync Lambda moved inside the VPC, which also forced production CI migrations in-VPC (soon-server#1675). §2 is now the live procedure. §3 is retained as the record of what was done and as the rollback reference — do not re-run it.

Still outstanding: capture the verification evidence (§3 step 4), and finish the housekeeping in §3 step 5 (deleting the unattached legacy security groups). Linear S-299 remains open because it also covers EC2 public IPs, RDS deletion protection and backups.

1. The rule

The production database must not be reachable from the internet. It accepts connections only from:

  1. Application workloads inside the VPC (ECS/Fargate tasks, in-VPC Lambdas), by security-group reference — never by IP range; and
  2. Named engineers, via a tunnelled session (§2) — never by opening a port.

Direct database credentials are never shared, never stored outside Enpass, and never placed in a security-group allow-list as a substitute for a tunnel.

2. How an engineer connects (the supported method)

AWS Systems Manager Session Manager port forwarding. No bastion host, no inbound port, no VPN client — and every session is recorded in CloudTrail, which is exactly the audit trail an assessor wants.

One-time prerequisites (Thomas — do this alongside §3): - An SSM-managed target inside the VPC: either an existing ECS/EC2 host with the SSM Agent and an instance profile containing AmazonSSMManagedInstanceCore, or a small dedicated t4g.nano "jump" instance in a private subnet (no public IP, no inbound rules at all). - The engineer's IAM user/role needs ssm:StartSession on that target and the AWS-StartPortForwardingSessionToRemoteHost document. - Install the Session Manager plugin locally (brew install --cask session-manager-plugin).

Every time you need the database:

aws ssm start-session \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["rds-soon-soon-prod.cq3bqogqbqvs.eu-west-1.rds.amazonaws.com"],"portNumber":["3306"],"localPortNumber":["3307"]}' \
  --region eu-west-1

Leave that running, then point TablePlus / DBeaver / MySQL Workbench at 127.0.0.1:3307 with the normal production credentials. Everything works exactly as before — the difference is the traffic goes through an authenticated, logged AWS channel instead of the open internet. Close the terminal to end the session.

Decided 2026-08-13: SSM Session Manager is the adopted method, because it adds no vendor and logs every session to CloudTrail, which doubles as audit evidence. A Tailscale subnet router in the VPC would achieve the same outcome and remains a documented alternative if the team later wants always-on private networking; session logging would then be Tailscale's rather than CloudTrail's.

Not permitted: adding a home/office IP to the database security group as a shortcut, re-enabling PubliclyAccessible, or sharing a tunnel with someone else.

3. Runbook — removing the public endpoint (S-299)

One-time change, tracked in Linear S-299. Do this in a maintenance window — step 3 restarts the database.

Blocker to clear first: the dsync Lambda has no VpcConfig, so it currently reaches the database over the public endpoint. Attach it to the VPC (private subnets + the app security group, with a NAT gateway if it needs internet egress) before step 2, or dsync breaks. This is the reason the public endpoint still exists.

Step 0 — capture the current state (evidence + rollback reference). 

aws ec2 describe-security-groups --group-ids sg-06e00f2565f145c41 --region eu-west-1 \
  > before-sg.json
aws rds describe-db-instances --db-instance-identifier rds-soon-soon-prod --region eu-west-1 \
  > before-rds.json

Step 1 — allow the application by security group (not by IP). 

# <app-sg-id> = the security group attached to the ECS/Fargate tasks and the in-VPC dsync Lambda
aws ec2 authorize-security-group-ingress \
  --group-id sg-06e00f2565f145c41 \
  --protocol tcp --port 3306 \
  --source-group <app-sg-id> \
  --region eu-west-1

Step 2 — remove the world-open rules. (Verify step 1 works first — this is the cut-over.) 

# IPv4 0.0.0.0/0
aws ec2 revoke-security-group-ingress \
  --group-id sg-06e00f2565f145c41 --protocol tcp --port 3306 --cidr 0.0.0.0/0 \
  --region eu-west-1

# IPv6 ::/0  (must be revoked separately — this is the one people forget)
aws ec2 revoke-security-group-ingress \
  --group-id sg-06e00f2565f145c41 \
  --ip-permissions 'IpProtocol=tcp,FromPort=3306,ToPort=3306,Ipv6Ranges=[{CidrIpv6=::/0}]' \
  --region eu-west-1

Step 3 — turn off public accessibility. (Causes a brief restart — schedule it.) 

aws rds modify-db-instance \
  --db-instance-identifier rds-soon-soon-prod \
  --no-publicly-accessible \
  --apply-immediately \
  --region eu-west-1

Step 4 — verify. 

# expect: "public": false
aws rds describe-db-instances --db-instance-identifier rds-soon-soon-prod --region eu-west-1 \
  --query 'DBInstances[].{public:PubliclyAccessible,status:DBInstanceStatus}'

# expect: no 0.0.0.0/0 and no ::/0 on 3306
aws ec2 describe-security-groups --group-ids sg-06e00f2565f145c41 --region eu-west-1 \
  --query 'SecurityGroups[].IpPermissions[?FromPort==`3306`]'
Then confirm the application is healthy, and re-run the GRC collector — CHK-NET-01 and CHK-PUB-01 should turn green: 
gh workflow run collect.yml --repo SoonHQ/soon-grc

Step 5 — housekeeping. Delete the unused legacy security groups that also permit world-open database ports (rds-launch-wizard-1/3/4, Database2-test, Production-DB-SG, mjssh, gpkvk) once confirmed unattached, and move the instance to a private subnet group as part of the OpenTofu/Terraform migration.

Rollback: re-add the previous ingress rule from before-sg.json and --publicly-accessible. Prefer fixing forward — reopening the database to the internet is a last resort and must be recorded as a decision.

4. Ongoing verification

Check Where Frequency
No 0.0.0.0/0 or ::/0 on sensitive ports CHK-NET-01 (soon-grc) Daily, automated
Account + bucket S3 public access blocked CHK-PUB-01 Daily, automated
RDS encrypted, backups retained CHK-ENC-01, CHK-BACKUP-01 Daily, automated
Anomalous database logins AWS GuardDuty (RDS login events, enabled 2026-08-08) → SNS → security@soon.works Continuous
Who connected, when CloudTrail / Session Manager history Reviewed quarterly with access review

5. Roles

  • Thomas Picauly — owns this procedure and the remediation; sets up the SSM target and IAM permissions.
  • Melvin Jacobson — infrastructure/IaC support; maintenance window.
  • ISM (Olaf) — confirms closure, records the evidence, reports at management review.

Change log

Version Date Author Comments
0.1 2026-08-08 ISMS First draft — supported access method (SSM Session Manager port forwarding, Tailscale alternative), the S-299 runbook to remove the public endpoint (incl. the IPv6 rule and the dsync VpcConfig blocker), verification and rollback.