init push

docs/file-lifecycle-s3-cleanup.md

@@ -0,0 +1,124 @@
+# File Lifecycle and S3 Cleanup
+
+## Lifecycle States (DB source of truth)
+
+`file_files.lifecycle_status`:
+
+- `temp`: uploaded but not attached
+- `active`: referenced by one or more business entities
+- `orphan_pending`: currently unreferenced, waiting grace period
+- `deleted`: backend cleanup has deleted object(s) and marked terminal state
+
+Supporting timestamps:
+
+- `created_at`: existing file creation time
+- `attached_at`: set when file becomes active
+- `orphaned_at`: set when file becomes unreferenced
+- `lifecycle_deleted_at`: set when cleanup marks lifecycle `deleted`
+
+Notes:
+
+- Existing `status` column (`uploaded/processing/validated/ready/trashed`) remains for processing/trash flow.
+- `deleted_at` is still used by trash semantics. Cleanup lifecycle uses `lifecycle_deleted_at`.
+
+## S3 Tags
+
+Single tag key: `gc`
+
+- `gc=temp`: temporary uploads
+- `gc=keep`: active or protected files
+- `gc=delete`: optional marker immediately before backend delete
+
+S3 tags are markers only. DB lifecycle state and DB reference checks are authoritative.
+
+## Cleanup Job
+
+Scheduler: API process loop (`startFileManagerLifecycleCleanupLoop`).
+
+Behavior:
+
+1. Query DB for `orphan_pending` files where `orphaned_at <= now - ORPHAN_GRACE_PERIOD_DAYS`.
+2. Process in batches (`FILE_CLEANUP_BATCH_SIZE`).
+3. Re-check DB references per file before deletion.
+4. If referenced again: mark `active`, set/keep `gc=keep`.
+5. If still unreferenced:
+   - optional `gc=delete` tag
+   - delete original and thumbnail S3 objects
+   - mark lifecycle `deleted` and set `lifecycle_deleted_at`
+6. Per-file error handling; one failure does not stop the batch.
+
+The job is idempotent and retry-safe.
+
+## Environment Variables
+
+- `ENABLE_S3_TAGGING` (default `true`)
+- `ENABLE_FILE_CLEANUP_JOB` (default `false`)
+- `TEMP_FILE_TTL_DAYS` (default `7`)
+- `ORPHAN_GRACE_PERIOD_DAYS` (default `30`)
+- `FILE_CLEANUP_BATCH_SIZE` (default `100`)
+- `FILE_CLEANUP_DRY_RUN` (default `true`)
+
+## Backfill (Dry-Run First)
+
+Command:
+
+```bash
+go run ./cmd/file-lifecycle-backfill
+```
+
+Optional:
+
+- `BACKFILL_DRY_RUN=true|false` (default `true`)
+
+Output report includes:
+
+- active files
+- probable orphan files
+- missing S3 key
+- invalid references
+- skipped records
+
+## Required IAM Permissions
+
+Minimum required:
+
+- `s3:PutObjectTagging`
+- `s3:GetObjectTagging`
+- `s3:DeleteObject`
+- `s3:GetObject`
+- `s3:HeadObject`
+
+Only if infra automation updates lifecycle rules:
+
+- `s3:PutLifecycleConfiguration`
+
+## S3 Lifecycle Rules (Manual if no IaC)
+
+Because this repository does not define Terraform/CDK/CloudFormation for S3 bucket lifecycle, configure lifecycle rules in your infra/deployment stack:
+
+1. Rule `gc=temp`:
+   - Filter: object tag `gc=temp`
+   - Expire after `TEMP_FILE_TTL_DAYS` (recommended default 7 days)
+
+2. Optional rule `gc=delete`:
+   - Filter: object tag `gc=delete`
+   - Expire after 1 day
+
+Do not use lifecycle expiration as the orphan grace mechanism. Grace period is enforced by DB `orphaned_at`.
+
+## Rollout Plan
+
+1. Deploy schema + code with defaults (`cleanup job off`, `dry-run on`).
+2. Enable tagging for new uploads (`ENABLE_S3_TAGGING=true`).
+3. Validate attach/detach transitions in logs.
+4. Run backfill dry-run and review report.
+5. Enable cleanup job in dry-run mode.
+6. Switch cleanup dry-run off after validation.
+7. Apply S3 lifecycle rules for `gc=temp` (and optional `gc=delete`).
+
+## Rollback Plan
+
+1. Set `ENABLE_FILE_CLEANUP_JOB=false`.
+2. Set `FILE_CLEANUP_DRY_RUN=true`.
+3. Optionally set `ENABLE_S3_TAGGING=false`.
+4. Pause/remove lifecycle rule `gc=delete` if enabled.