Skip to content

Staging Runbook

Staging Runbook: Terraform + CloudTrail Verification

This checklist covers a staging apply for the S3 + CloudTrail stack and the post‑apply verification steps.

Preconditions

  • AWS credentials with permissions to create S3, KMS, IAM, CloudTrail.
  • Terraform state backend configured (S3 + DynamoDB lock recommended).
  • Staging variables ready (env, region, bucket_prefix, org_slug, retention_years).

Terraform Apply Checklist

  1. cd infra/terraform
  2. terraform init
  3. terraform validate
  4. terraform plan -var-file=terraform.tfvars
  5. terraform apply -var-file=terraform.tfvars
  6. Capture outputs: terraform output

Post‑Apply Verification

Buckets + Object Lock 1. Confirm buckets exist and are versioned: - aws s3api get-bucket-versioning --bucket <bucket> 1. Confirm Object Lock enabled on required buckets: - aws s3api get-object-lock-configuration --bucket <bucket> 1. Confirm lifecycle rules and retention: - aws s3api get-bucket-lifecycle-configuration --bucket <bucket>

KMS 1. Verify KMS keys created with rotation: - aws kms describe-key --key-id alias/calcbridge/<env>/<purpose> - aws kms get-key-rotation-status --key-id alias/calcbridge/<env>/<purpose>

IAM Role Matrix 1. Verify IAM roles exist: - aws iam get-role --role-name calcbridge-<env>-api - aws iam get-role --role-name calcbridge-<env>-worker - aws iam get-role --role-name calcbridge-<env>-analytics 1. Validate attached inline policies: - aws iam get-role-policy --role-name calcbridge-<env>-worker --policy-name calcbridge-<env>-worker-s3

IAM Policy Examples (Env-Specific) Use these as baselines for staging/prod and tailor bucket ARNs, KMS keys, and secret ARNs per environment.

Worker role (evidence bucket + object lock + KMS): 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EvidenceBucketReadWrite",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:PutObjectRetention",
        "s3:PutObjectLegalHold",
        "s3:PutObjectTagging"
      ],
      "Resource": [
        "arn:aws:s3:::calcbridge-<env>-evidence",
        "arn:aws:s3:::calcbridge-<env>-evidence/*"
      ]
    },
    {
      "Sid": "EvidenceBucketKmsEncrypt",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:<region>:<account-id>:key/<kms-key-id>"
    }
  ]
}

API role (read-only evidence for verify, no delete): 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EvidenceBucketReadOnly",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::calcbridge-<env>-evidence",
        "arn:aws:s3:::calcbridge-<env>-evidence/*"
      ]
    }
  ]
}

Secrets Manager access (TSA + HMAC): 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadSigningAndTsaSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:<region>:<account-id>:secret:calcbridge/<env>/tsa/*",
        "arn:aws:secretsmanager:<region>:<account-id>:secret:calcbridge/<env>/hmac/*"
      ]
    }
  ]
}

AWS CLI Verification Scripts These are explicit commands you can run in staging/prod to validate IAM + Object Lock + CloudTrail.

Assume a role and validate S3 Object Lock retention: 

ROLE_ARN=arn:aws:iam::<account-id>:role/calcbridge-<env>-worker
SESSION_NAME=calcbridge-verify
CREDS=$(aws sts assume-role --role-arn "$ROLE_ARN" --role-session-name "$SESSION_NAME")
export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r .Credentials.AccessKeyId)
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r .Credentials.SecretAccessKey)
export AWS_SESSION_TOKEN=$(echo "$CREDS" | jq -r .Credentials.SessionToken)

BUCKET=calcbridge-<env>-evidence
KEY=object-lock-smoke-test.txt
echo "object-lock-smoke-test" > /tmp/$KEY
aws s3api put-object --bucket "$BUCKET" --key "$KEY" --body /tmp/$KEY
aws s3api put-object-retention \
  --bucket "$BUCKET" \
  --key "$KEY" \
  --retention '{"Mode":"GOVERNANCE","RetainUntilDate":"2030-01-01T00:00:00Z"}'
aws s3api get-object-retention --bucket "$BUCKET" --key "$KEY"

Validate Secrets Manager access: 

aws secretsmanager get-secret-value --secret-id "calcbridge/<env>/hmac/primary"
aws secretsmanager get-secret-value --secret-id "calcbridge/<env>/tsa/username"

Validate CloudTrail data events are delivered: 

aws cloudtrail get-event-selectors --trail-name calcbridge-<env>-trail
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject \
  --max-results 5

CloudTrail Data Events 1. Verify trail status: - aws cloudtrail get-trail-status --name calcbridge-<env>-trail 1. Verify event selectors include evidence/audit buckets: - aws cloudtrail get-event-selectors --trail-name calcbridge-<env>-trail 1. Smoke‑test a write and confirm event delivery: - Upload a test object to evidence bucket. - Check CloudTrail log delivery in the logs bucket: - aws s3 ls s3://<logs-bucket>/AWSLogs/<account-id>/CloudTrail/

Service Integrations (Feature‑Flagged)

Signing Service (OOXML/PDF) 1. Enable flags: - CALCBRIDGE_SIGNING_ENABLED=true - CALCBRIDGE_SIGNING_BASE_URL=https://<signing-service> 2. Health check: - curl -f https://<signing-service>/health 3. Smoke‑test signing via integration test: - RUN_REAL_SIGNING_SERVICE=1 PYTHONPATH=. python3.11 -m pytest tests/integration/test_signing_service_real.py -v

TSA + Secrets Manager 1. Create TSA secrets (username/password/api key) in Secrets Manager. 2. Enable flags: - CALCBRIDGE_TSA_ENABLED=true - CALCBRIDGE_TSA_SECRETS_MANAGER_ENABLED=true - CALCBRIDGE_TSA_USERNAME_ARN=... / CALCBRIDGE_TSA_PASSWORD_ARN=... / CALCBRIDGE_TSA_API_KEY_ARN=... 3. Smoke‑test timestamping by running a compliance commitment flow and verifying a receipt row is created.

HMAC Secret via Secrets Manager 1. Create secret containing {"hmac_secret":"<value>"} in Secrets Manager. 2. Enable flags: - CALCBRIDGE_HMAC_SECRETS_MANAGER_ENABLED=true - CALCBRIDGE_HMAC_SECRET_ARN=... 3. Verify export/verify flow succeeds with the secret.

Verify Sandbox (Container) 1. Build container image: - docker build -t calcbridge-verify-sandbox:latest services/verify-sandbox 2. Enable flags: - CALCBRIDGE_VERIFY_SANDBOX_ENABLED=true - CALCBRIDGE_VERIFY_SANDBOX_MODE=container - Optional: CALCBRIDGE_VERIFY_SANDBOX_CONTAINER_RUNTIME=runsc (gVisor) 3. Smoke‑test: - RUN_VERIFY_CONTAINER_TESTS=1 PYTHONPATH=. python3.11 -m pytest tests/integration/test_export_verify_container_sandbox.py -v

Evidence Storage Validation (Optional)

  1. Ensure Object Lock + versioning enforced for the evidence bucket.
  2. Verify PutObjectRetention is permitted by IAM for the worker role.
  3. Confirm upload path with a local integration test:
  4. RUN_OBJECT_LOCK_TESTS=1 USE_TESTCONTAINERS=1 PYTHONPATH=. pytest tests/integration/test_compliance_object_lock_localstack.py -v

Rollback

  1. If resources are invalid, run:
  2. terraform destroy -var-file=terraform.tfvars
  3. Record any policy or resource drift for follow‑up.

Phase 2 Remaining Real-Service Validations

The following checks require live services/environments and are not fully verifiable in local/CI: 1. Real OOXML/PDF signing service (iText) end-to-end signature validation. 2. AWS S3 Object Lock delete-prevention enforcement and CloudTrail data event delivery. 3. TSA + Secrets Manager operational validation in AWS (live secrets + receipt generation). 4. Staging performance P95 verification under representative load.