Skip to content

Evidence Collection and Management Procedure

Purpose. Defines how Soon collects, stores, protects and retrieves the records (evidence) that prove its ISMS controls are actually operating — the material an ISO/IEC 27001 Stage 2 auditor and a SOC 2 examiner will sample. It covers what evidence to capture, how to capture it (screenshots, config exports, logs, reports), where it lives (a hardened AWS S3 store), and the cadence that keeps it fresh. Satisfies clause 7.5 (documented information), clause 9.1 (retaining evidence of monitoring results) and control A.5.33 (protection of records).

Status: DRAFT — the store is LIVE as of 2026-08-13. Built and verified in a dedicated log-archive account:
Account 404379474355 (Soon-LogArchive), in the Security OU, separate from production
Evidence bucket s3://soon-isms-evidence (eu-west-1)
Object Lock Governance, 1095 days (3 years) — matches the Records Retention Policy
Encryption SSE-KMS, alias/soon-isms-evidence, annual rotation enabled
Other protections Versioning on · account-level Block Public Access (all four) · bucket policy denying non-TLS and unencrypted uploads
Guardrail Service control policy ProtectSecurityLogs on the Security OU denies deleting the buckets, weakening Object Lock, bypassing governance retention, deleting the backup vault or its recovery points, disabling the KMS key, and leaving the organisation — even for a full administrator in that account
Verified Two functional tests, not just configuration checks: deleting a stored object was refused ("object protected by object lock"), and after the guardrail was corrected, an attempt to weaken retention from 1095 days to 1 day was refused by the SCP

Note on the guardrail's limits. SCPs never apply to the management account, and the organisation trail soon-org-trail is defined there — so the SCP cannot prevent that trail being stopped. What it does protect is the delivered logs: the destination bucket cannot be deleted or weakened from the log-archive account, so historic evidence survives even if collection were interrupted. Stopping a trail is itself a logged, alertable event.

Scope note. This procedure is about audit/compliance evidence (proof that controls run). It is not A.5.28 Collection of evidence, which concerns forensic evidence gathered during a security incident — that lives in the Incident Response Procedure.

1. Why evidence matters (and what counts)

ISO 27001 certification is a two-stage audit (see ROADMAP.md §1):

  • Stage 1 — design. The auditor checks the ISMS is designed: the policies, the Statement of Applicability, risk assessment. This is mostly the documents in this repo.
  • Stage 2 — operation. The auditor checks the ISMS is operating: they pick controls from the SoA and ask "show me proof." Proof = records/evidence.

SOC 2 works the same way — a Type 1 report tests design at a point in time; a Type 2 report tests that controls operated over a period (typically 3–12 months). Either way, no evidence = the control is treated as not working. Before booking Stage 2 the ISMS must have been operating ~3 months so there is a body of evidence to sample.

Evidence is any objective artefact that shows a control ran:

Kind Examples at Soon
Configuration state AWS Block Public Access on; RDS backups enabled; MFA enforced; TLS 1.2+; branch protection
Records of an activity Quarterly access review; backup restore test; internal audit; management review minutes
Tickets / change trail GitHub PRs + CI runs; Linear issues; Slack #security threads; incident records
Reports Aikido vulnerability scans; GetAgency pentest report; supplier SOC 2 reports on file
Signed records Signed contracts / NDAs; policy acknowledgements; training completions

2. Principles

Every piece of evidence should be:

  1. Authentic — a true capture of the real system, not edited or reconstructed. Crop and redact only; never alter a value.
  2. Attributable — records who captured it and when (the manifest, §5.4).
  3. Timely / fresh — captured close to the review date; recurring evidence shows multiple cycles, not a single old screenshot.
  4. Complete & self-explanatory — an auditor who has never seen the system can tell what it shows (which system, which account, which setting).
  5. Tamper-evident — stored so it cannot be silently changed or deleted (versioning
  6. Object Lock in S3; git history for the index).
  7. Minimised — contains only the personal data / secrets strictly needed. Redact the rest (GDPR — see Privacy Policy).

3. Where things live — the site, the repo, and the store

Decided 2026-08-23: auditors and assessors read the evidence on the ISMS site. We do not package and send a folder. A named, time-boxed login to isms.soon.works gives them the policies and the artefacts side by side, in one place, with the trail of who captured what and when — and it means there is exactly one version of the truth rather than a snapshot that starts ageing the moment it is zipped.

That gives three locations with three different jobs:

  ISMS site (isms.soon.works)        ISMS repo (GitHub)             S3 store (log-archive acct)
  ─────────────────────────          ──────────────────             ──────────────────────────
  What a reader SEES:                What is VERSIONED:             What is PROVABLE:
  • policies and procedures          • the documents (Markdown)     • the original artefact
  • the Evidence page —              • the capture log + index      • Object Lock, 3 years
    artefacts grouped by CC          • published artefact copies    • SSE-KMS, versioned
  • behind Cloudflare Access         • git = tamper-evident history • CloudTrail data events
  • The S3 store remains the system of record. Object Lock, per-object retention and a SHA-256 recorded at capture are what make an artefact evidence rather than a picture.
  • The repo holds a published copy of each artefact under evidence/<CC series>/, which the site renders. The hash printed beside it on the site ties it back to the immutable original, so anyone can confirm the two are the same file.
  • Both are produced by the same command. tools/evidence.py stamps, hashes, uploads and publishes in one step (§5.4), so the two copies cannot drift.

This is a deliberate change from "artefacts never enter the repo". Publishing a copy is what makes the site usable by an auditor, and the site is the delivery mechanism we have chosen. The constraint that replaces it is narrower and firmer: no secrets, no credentials, and no customer personal data in any artefact — which means redaction happens before capture, not afterwards. Git history is permanent, so an artefact committed in error cannot be quietly withdrawn. If it happens, treat it as an incident and rotate whatever leaked. Use --no-site for anything that must stay in S3 only.

The Evidence Register remains the control-by-control plan of what we intend to hold, keyed by ISO clause and Annex A control. The Evidence Capture Log records what we actually captured, and the site's Evidence page presents the same records grouped by SOC 2 criteria series — which is the cross-walk this section previously flagged as missing.

4. The evidence store — is S3 safe? (yes, when hardened)

Short answer: yes. S3 is an appropriate, auditor-accepted home for compliance evidence — provided it is configured for confidentiality, integrity and immutability. An unconfigured bucket is not; a hardened one is arguably stronger than a shared drive because S3 Object Lock gives true write-once-read-many (WORM) immutability that auditors recognise. It also keeps evidence in the EU on Soon's existing AWS footprint (eu-west-1), consistent with our GDPR posture.

4.1 Why S3 (alternatives considered)

Option Verdict for Soon
Hardened AWS S3 (this design) Recommended. AWS-native, EU-resident, cheap (pennies/month at our volume), WORM immutability via Object Lock, full access logging.
Google Workspace shared drive Acceptable fallback — Soon already uses Workspace; access-controlled + versioned. Weaker immutability story than Object Lock; check data region.
GRC platform (Vanta / Drata / Secureframe) Strong complement — auto-collects a lot of this evidence via read-only integrations and maps it to ISO/SOC 2. Costs money; still needs a store for manual evidence. Worth evaluating once budget allows. TODO(owner): decide.
This git repo No. Evidence contains personal data / config detail — prohibited here. Repo holds only the index + manifests.

TODO(owner): GetAgency (our SOC 2 Type 1 vendor — see OPEN-QUESTIONS.md Q10) may prefer evidence uploaded to their portal. Confirm with Aman whether the S3 store feeds their portal or replaces it.

4.2 Bucket configuration (the hardening baseline)

The evidence bucket must be created with all of the following. This is the control — the bucket's own config export is itself evidence (see this document's evidence: list).

# Setting Requirement Why
1 Region eu-west-1 (Ireland) EU data residency (GDPR)
2 Account A dedicated log-archive AWS account, separate from production (confirmed by the owner 2026-08-13) A prod compromise, a rogue admin or a bad script cannot reach or delete the evidence; clean IAM boundary. Create under AWS Organizations; the collector writes cross-account with put-only rights
3 Block Public Access All four settings ON, at account and bucket level Evidence is never internet-reachable
4 Default encryption SSE-KMS with a dedicated customer-managed key (CMK) Encryption at rest + per-key access control + CloudTrail on every decrypt
5 TLS-only Bucket policy denies requests where aws:SecureTransport = false Encryption in transit enforced
6 Versioning Enabled No silent overwrite/loss; every version retained
7 Object Lock (WORM) Enabled (requires versioning) with a default retention Immutability — evidence can't be tampered with or deleted early (see §4.4)
8 Lifecycle Transition to Glacier/Deep Archive after the active audit cycle; expire only after the retention period Cost control without breaking retention
9 Access logging CloudTrail S3 data events → a separate WORM-locked log bucket (the tamper-proof trail). Optional S3 server-access logging → a separate bucket without Object Lock (§4.6) Chain of custody (§4.6)
10 Tags data-classification=Confidential, purpose=isms-evidence Discoverability & DLP

4.3 Access & separation of duties

  • Least privilege. A small isms-evidence-writers IAM group/role gets s3:PutObject, s3:GetObject and list — not s3:DeleteObject and not permission to change Object Lock retention or the bucket policy.
  • Separate the admin. Bucket / KMS / lifecycle administration is a different role from routine uploaders where feasible. Soon is a 4-person founder team, so full separation is hard — the compensating controls are MFA on the account, Object Lock (nobody can delete locked objects), and CloudTrail logging of every action.
  • MFA is required for the account and for any privileged/delete-type operation.
  • Auditor access is read-only, time-boxed, and revoked after the audit (mirrors the read-only repo access in the Access Control Policy).

4.4 Retention & Object Lock mode

  • Object Lock has two modes: Governance (a specially-privileged role can override, and every override is logged) and Compliance (nobody — not even the account root — can delete or shorten retention until it expires).
  • Default: Governance mode with a retention period aligned to the Records Retention and Protection Policy (A.5.33). Governance is the pragmatic choice for a small team — it protects against accident and tampering while allowing a logged correction of a genuine mistake. Use Compliance mode for the highest-value records if required.
  • Retention: 3 years (current + previous certification cycle), matching the Records Retention & Protection Policy, confirmed 2026-08-13. Keep longer only where law or a legal hold requires it.

4.5 Naming convention (so evidence maps to controls)

s3://<evidence-bucket>/<year>/<control-or-clause>/<YYYY-MM-DD>__<control>__<short-desc>__<capturer>.<ext>

examples:
  2026/A.8.13/2026-07-10__A.8.13__rds-automated-backups-enabled__melvin.png
  2026/A.8.13/2026-07-10__A.8.13__restore-test-record__melvin.pdf
  2026/A.5.18/2026-07-01__A.5.18__q3-aws-iam-access-review__olaf.csv
  2026/9.2/2026-06-20__clause-9.2__internal-audit-report__andrea.pdf
  • One folder per control/clause, sortable by date, capturer named. Every object has a companion manifest (§5.4). The register (ISMS-FORM-07-2) links the S3 key.

4.6 Logging & chain of custody

  • Enable CloudTrail S3 data events for the bucket, delivered to a separate, Object-Lock/WORM log bucket — this is the tamper-proof chain-of-custody trail (CloudTrail can write to a locked bucket). It records who uploaded or read what, and when — what an auditor may ask for.
  • Optionally also enable S3 server-access logging, but its target bucket must not have Object Lock — S3 log delivery fails to a locked bucket. Protect that bucket with versioning, a restrictive policy and the separate log-archive account instead.
  • Independently, we record a SHA-256 hash of each artefact at capture time in its manifest (§5.3). If anyone ever questions integrity, the hash proves the file is unchanged — regardless of the store.

4.7 Chain of custody — who did what (no database needed)

The evidence system records who created, changed or accessed each piece of evidence, and when — the attribution an auditor asks for — without any database, login system or user store. That trail comes from tools already in place:

  • The index (this repo). Every change to the Evidence Register or a manifest lands as a Pull Request by a named GitHub user. Git history is an immutable, peer-reviewed record of who recorded or amended which evidence and when (governed by the Control of Documented Information procedure).
  • The artefacts (S3 store). CloudTrail data events + access logs (§4.6) record the IAM principal and timestamp of every upload and read.
  • Automated collection. When the GRC collectors run, they do so as a named identity (a specific GitHub Actions run assuming a specific role) and propose changes by PR — logged end to end, never self-merged.

This is deliberate: attribution lives in Git, CloudTrail and WORM S3 — harder to tamper with than a self-managed database, and nothing extra to run or secure. Out of scope here: logging of end-users of the Soon product is application logging (Logging & Monitoring, A.8.15/16), and access control for any published read-only dashboard is handled at the hosting layer (SSO / Cloudflare Access) — neither needs a user database either.

5. How to capture evidence (the runbook)

5.1 Screenshots — the golden rules

  1. Capture the whole window, including the browser URL bar or CLI prompt, so the system and account are visible (e.g. the AWS console shows the account/region).
  2. Make sure a date is visible — the system clock, or a timestamp in the UI. If none is visible, the manifest date stands, but a visible date is stronger.
  3. Show the actual setting/state, not a cropped fragment that could be anything.
  4. Redact secrets and unnecessary PII (access keys, tokens, customer names not needed as evidence) — redact by drawing over, don't crop out context. Never edit values.
  5. Save as PNG, name per §4.5, then log it (§6).

5.2 Config / CLI exports (preferred where possible)

Machine-readable exports are stronger than screenshots — they're the raw truth and can be re-verified. Run the read-only command, save the output verbatim, record the exact command and timestamp in the manifest. (These are commands a human runs against Soon's real infra — illustrative here; no infrastructure code or credentials live in this repo.)

# examples — the OUTPUT is the evidence, saved as .json/.txt
aws s3api get-bucket-encryption        --bucket <evidence-bucket>      # SSE-KMS on?
aws s3api get-public-access-block      --bucket <evidence-bucket>      # BPA all true?
aws s3api get-object-lock-configuration --bucket <evidence-bucket>     # WORM on?
aws rds describe-db-instances --query 'DBInstances[].{id:DBInstanceIdentifier,backup:BackupRetentionPeriod,enc:StorageEncrypted}'
gh api repos/SoonHQ/soon-server/branches/main/protection                # branch protection

5.3 Logs, reports & integrity

  • Log extracts: export the relevant window (CloudWatch, Sentry) as a file; note the query/time range in the manifest. Don't paste secrets.
  • Reports: save the PDF as-issued (Aikido scan, GetAgency pentest, supplier SOC 2).
  • Hash every artefact and record it: shasum -a 256 <file>.

5.4 The tool — tools/evidence.py

Everything in §5.1–§5.3 is done for you by one command, so that capturing evidence is cheaper than not capturing it:

python3 tools/evidence.py shot  --control A.8.13 --desc rds-backups-on --soc2 60 --cc CC7.5
python3 tools/evidence.py add   ~/Desktop/mfa.png --control A.5.17 --desc mfa-github
python3 tools/evidence.py run   --control A.8.13 --desc rds-config -- \
    aws rds describe-db-instances --region eu-west-1
python3 tools/evidence.py check     # is the store still hardened?
python3 tools/evidence.py list      # what have we filed?
python3 tools/evidence.py clock     # show date+time in macOS screenshots

For every artefact the tool:

  1. Burns a visible timestamp bar onto the image — UTC and local time, the control, the SOC 2 row, who captured it — before hashing, so the timestamp is inside the hashed artefact and cannot be swapped for a different one.
  2. Hashes it (SHA-256).
  3. Uploads it to the WORM store under the §4.5 key, and — where --soc2 is given — a second copy under soc2-type1/CC<n>/<row>-<ref>-<slug>.png.
  4. Publishes a copy to evidence/<CC series>/ and regenerates the site's Evidence page, so an auditor sees it next to the policy it supports. Use --no-site to keep an artefact in S3 only.
  5. Appends a row to the Evidence Capture Log with the hash and the S3 key.

Object Lock means uploads are permanent. The tool refuses to write a key that already exists rather than silently failing against the lock. If a capture was wrong, file a new, better one with a different --desc; do not try to delete the old one. A superseded artefact is normal and harmless — an evidence store that can be edited is the problem.

Independent of the tool, run evidence.py clock once on each machine so the macOS menu-bar clock shows the date and seconds. Then a full-screen capture carries its own timestamp even without the tool — belt and braces, and it is what an auditor looks for first.

5.5 The manifest (chain of custody)

Each artefact (or each collection batch) gets a short manifest recording control, description, S3 key, capturer, date, method, command (if any) and SHA-256 — copy ISMS-FORM-07-3 Evidence Capture Manifest. tools/evidence.py writes this for you into the Evidence Capture Log; the manifest form remains for batches captured by hand or by a third party. The manifest lives in the repo (no sensitive content); the artefact lives in S3.

6. The collection workflow

 capture ──▶ redact/minimise ──▶ name (§4.5) ──▶ hash ──▶ upload to S3
    │                                                          │
    └────────────▶ fill manifest (ISMS-FORM-07-3) ◀───────────┘
                                │
                                ▼
              add / update a row in the Evidence Register (ISMS-FORM-07-2)
                                │
                                ▼
                    commit index+manifest via PR → main
  1. Capture the artefact per §5.
  2. Redact / minimise personal data and secrets.
  3. Name it per §4.5 and compute its SHA-256.
  4. Upload to the S3 store (aws s3 cp or console); the bucket default applies Object Lock retention automatically.
  5. Manifest — fill ISMS-FORM-07-3.
  6. Register — add/update the row in ISMS-FORM-07-2 Evidence Register: control, description, S3 key, owner, date collected, next-due.
  7. PR the index + manifest into main (per CONTRIBUTING.md). The artefact itself is not committed — only its pointer.

7. Cadence (keeping evidence fresh)

Each item has an owner and a frequency in the Evidence Register. Summary of the rhythm:

Frequency Examples
Continuous (as it happens) PRs/CI, deployments, incident records, Slack #security threads
Monthly Vulnerability scan review (Aikido); backup success check
Quarterly Access reviews (AWS IAM, GitHub, WorkOS, Google Workspace); training refresher log; supplier/sub-processor check
At least annual Backup restore test; internal audit; management review; risk review; pentest; policy re-approval
On event Joiner/mover/leaver; incident; significant change; new supplier

The full recurring calendar belongs in the ISMS Regular Activity Schedule (ISMS-FORM-10-2, to create — see ROADMAP.md §4.8); this procedure and the register are its evidence-collection half.

8. Roles & responsibilities

  • ISM (Olaf) — owns this procedure and the evidence store; grants/revokes access; maintains the register; presents evidence to auditors. (Per AGENTS.md, Olaf owns evidence.)
  • Control owners (e.g. Melvin — backups/infra; Thomas — platform/access) — capture the evidence for the controls they own, on cadence, and log it.
  • Independent auditor (Andrea, CyberSquad) — samples evidence during the internal audit; must not be the sole capturer of evidence they later audit.
  • All staff — provide evidence on request (e.g. signed acknowledgements, training).

9. Appendix — one-time store setup runbook

To stand the store up (do once; the resulting config is itself evidence). Real names are TODO(owner) — do not invent them.

  1. ~~Choose the account and bucket name.~~ Done 2026-08-13: account 404379474355 (Soon-LogArchive) in the Security OU; bucket soon-isms-evidence.
  2. Create the bucket in eu-west-1 with versioning on and Object Lock enabled. Enabling Object Lock at creation is simplest; AWS also lets you enable it later on any versioning-enabled bucket (PutObjectLockConfiguration). Versioning is the true prerequisite; once Object Lock is on it cannot be turned off.
  3. Turn on Block Public Access (all four) at account + bucket level.
  4. Create a dedicated KMS CMK; set the bucket default encryption to SSE-KMS with it.
  5. Attach a bucket policy: deny non-TLS; deny PutObject without KMS encryption; restrict writes to the isms-evidence-writers role.
  6. Set the Object Lock default retention (mode + period per §4.4) — TODO(owner).
  7. Configure CloudTrail S3 data events → separate WORM-locked log bucket (chain of custody). If also using S3 server-access logging, send it to a separate bucket without Object Lock (see §4.6).
  8. Add a lifecycle rule (transition to Glacier; expire only after retention).
  9. Create the isms-evidence-writers IAM role/group (no delete, no lock-change) and the separate admin role; enforce MFA.
  10. Capture the setup as the first evidence: export items 3, 4, 6 (get-public-access-block, get-bucket-encryption, get-object-lock-configuration) and log them in the register under A.5.33 / this procedure.

This is a procedure, not infrastructure code. Implement it via Soon's normal IaC / change process in the infra repo — not here.

10. Flags for the owner (resolve before approval)

  1. ~~AWS account + bucket name.~~ Resolved 2026-08-13 — see the status block above.
  2. Object Lock mode (Governance vs Compliance) and retention period, aligned to the Records Retention Policy — TODO(owner: Olaf/Andrea).
  3. GetAgency portal vs S3 for the SOC 2 Type 1 evidence — confirm with Aman (TODO(owner: Olaf)).
  4. GRC tool (Vanta/Drata/Secureframe) — adopt to automate collection, or defer? TODO(owner).
  5. Restore test (Q5, OPEN-QUESTIONS.md) — a backup restore has reportedly never been tested; the first restore-test record is high-priority evidence for A.8.13 — TODO(owner: Melvin).
  6. Classification scheme. Many ISMS docs use a Protected value that isn't in the scheme defined in 07-3 §5 (Public/Internal/ Confidential/Restricted). These evidence docs use scheme-valid Confidential. TODO(owner): reconcile the scheme repo-wide (add the Protected tier or reclassify).

Change log

Version Date Author Comments
0.1 2026-07-12 ISMS First draft — evidence system: repo-index-vs-S3-store split, hardened S3 design (BPA, SSE-KMS, versioning, Object Lock/WORM, access logging), capture runbook (screenshots/exports/hashing), collection workflow, cadence, roles, setup runbook. Owner flags for account/bucket/retention/GRC tool.