fm_be/docs/file-lifecycle-s3-cleanup.md

File Lifecycle and S3 Cleanup

Lifecycle States (DB source of truth)

file_files.lifecycle_status:

• temp: uploaded but not attached
• active: referenced by one or more business entities
• orphan_pending: currently unreferenced, waiting grace period
• deleted: backend cleanup has deleted object(s) and marked terminal state

Supporting timestamps:

• created_at: existing file creation time
• attached_at: set when file becomes active
• orphaned_at: set when file becomes unreferenced
• lifecycle_deleted_at: set when cleanup marks lifecycle deleted

Notes:

• Existing status column (uploaded/processing/validated/ready/trashed) remains for processing/trash flow.
• deleted_at is still used by trash semantics. Cleanup lifecycle uses lifecycle_deleted_at.

S3 Tags

Single tag key: gc

• gc=temp: temporary uploads
• gc=keep: active or protected files
• gc=delete: optional marker immediately before backend delete

S3 tags are markers only. DB lifecycle state and DB reference checks are authoritative.

Cleanup Job

Scheduler: API process loop (startFileManagerLifecycleCleanupLoop).

Behavior:

1. Query DB for orphan_pending files where orphaned_at <= now - ORPHAN_GRACE_PERIOD_DAYS.
2. Process in batches (FILE_CLEANUP_BATCH_SIZE).
3. Re-check DB references per file before deletion.
4. If referenced again: mark active, set/keep gc=keep.
5. If still unreferenced:
   • optional gc=delete tag
   • delete original and thumbnail S3 objects
   • mark lifecycle deleted and set lifecycle_deleted_at
6. Per-file error handling; one failure does not stop the batch.

The job is idempotent and retry-safe.

Environment Variables

• ENABLE_S3_TAGGING (default true)
• ENABLE_FILE_CLEANUP_JOB (default false)
• TEMP_FILE_TTL_DAYS (default 7)
• ORPHAN_GRACE_PERIOD_DAYS (default 30)
• FILE_CLEANUP_BATCH_SIZE (default 100)
• FILE_CLEANUP_DRY_RUN (default true)

Backfill (Dry-Run First)

Command:

go run ./cmd/file-lifecycle-backfill

Optional:

• BACKFILL_DRY_RUN=true|false (default true)

Output report includes:

• active files
• probable orphan files
• missing S3 key
• invalid references
• skipped records

Required IAM Permissions

Minimum required:

• s3:PutObjectTagging
• s3:GetObjectTagging
• s3:DeleteObject
• s3:GetObject
• s3:HeadObject

Only if infra automation updates lifecycle rules:

• s3:PutLifecycleConfiguration

S3 Lifecycle Rules (Manual if no IaC)

Because this repository does not define Terraform/CDK/CloudFormation for S3 bucket lifecycle, configure lifecycle rules in your infra/deployment stack:

1. Rule gc=temp:

   • Filter: object tag gc=temp
   • Expire after TEMP_FILE_TTL_DAYS (recommended default 7 days)

2. Optional rule gc=delete:

   • Filter: object tag gc=delete
   • Expire after 1 day

Do not use lifecycle expiration as the orphan grace mechanism. Grace period is enforced by DB orphaned_at.

Rollout Plan

1. Deploy schema + code with defaults (cleanup job off, dry-run on).
2. Enable tagging for new uploads (ENABLE_S3_TAGGING=true).
3. Validate attach/detach transitions in logs.
4. Run backfill dry-run and review report.
5. Enable cleanup job in dry-run mode.
6. Switch cleanup dry-run off after validation.
7. Apply S3 lifecycle rules for gc=temp (and optional gc=delete).

Rollback Plan

1. Set ENABLE_FILE_CLEANUP_JOB=false.
2. Set FILE_CLEANUP_DRY_RUN=true.
3. Optionally set ENABLE_S3_TAGGING=false.
4. Pause/remove lifecycle rule gc=delete if enabled.