125 lines
3.5 KiB
Markdown
125 lines
3.5 KiB
Markdown
# File Lifecycle and S3 Cleanup
|
|
|
|
## Lifecycle States (DB source of truth)
|
|
|
|
`file_files.lifecycle_status`:
|
|
|
|
- `temp`: uploaded but not attached
|
|
- `active`: referenced by one or more business entities
|
|
- `orphan_pending`: currently unreferenced, waiting grace period
|
|
- `deleted`: backend cleanup has deleted object(s) and marked terminal state
|
|
|
|
Supporting timestamps:
|
|
|
|
- `created_at`: existing file creation time
|
|
- `attached_at`: set when file becomes active
|
|
- `orphaned_at`: set when file becomes unreferenced
|
|
- `lifecycle_deleted_at`: set when cleanup marks lifecycle `deleted`
|
|
|
|
Notes:
|
|
|
|
- Existing `status` column (`uploaded/processing/validated/ready/trashed`) remains for processing/trash flow.
|
|
- `deleted_at` is still used by trash semantics. Cleanup lifecycle uses `lifecycle_deleted_at`.
|
|
|
|
## S3 Tags
|
|
|
|
Single tag key: `gc`
|
|
|
|
- `gc=temp`: temporary uploads
|
|
- `gc=keep`: active or protected files
|
|
- `gc=delete`: optional marker immediately before backend delete
|
|
|
|
S3 tags are markers only. DB lifecycle state and DB reference checks are authoritative.
|
|
|
|
## Cleanup Job
|
|
|
|
Scheduler: API process loop (`startFileManagerLifecycleCleanupLoop`).
|
|
|
|
Behavior:
|
|
|
|
1. Query DB for `orphan_pending` files where `orphaned_at <= now - ORPHAN_GRACE_PERIOD_DAYS`.
|
|
2. Process in batches (`FILE_CLEANUP_BATCH_SIZE`).
|
|
3. Re-check DB references per file before deletion.
|
|
4. If referenced again: mark `active`, set/keep `gc=keep`.
|
|
5. If still unreferenced:
|
|
- optional `gc=delete` tag
|
|
- delete original and thumbnail S3 objects
|
|
- mark lifecycle `deleted` and set `lifecycle_deleted_at`
|
|
6. Per-file error handling; one failure does not stop the batch.
|
|
|
|
The job is idempotent and retry-safe.
|
|
|
|
## Environment Variables
|
|
|
|
- `ENABLE_S3_TAGGING` (default `true`)
|
|
- `ENABLE_FILE_CLEANUP_JOB` (default `false`)
|
|
- `TEMP_FILE_TTL_DAYS` (default `7`)
|
|
- `ORPHAN_GRACE_PERIOD_DAYS` (default `30`)
|
|
- `FILE_CLEANUP_BATCH_SIZE` (default `100`)
|
|
- `FILE_CLEANUP_DRY_RUN` (default `true`)
|
|
|
|
## Backfill (Dry-Run First)
|
|
|
|
Command:
|
|
|
|
```bash
|
|
go run ./cmd/file-lifecycle-backfill
|
|
```
|
|
|
|
Optional:
|
|
|
|
- `BACKFILL_DRY_RUN=true|false` (default `true`)
|
|
|
|
Output report includes:
|
|
|
|
- active files
|
|
- probable orphan files
|
|
- missing S3 key
|
|
- invalid references
|
|
- skipped records
|
|
|
|
## Required IAM Permissions
|
|
|
|
Minimum required:
|
|
|
|
- `s3:PutObjectTagging`
|
|
- `s3:GetObjectTagging`
|
|
- `s3:DeleteObject`
|
|
- `s3:GetObject`
|
|
- `s3:HeadObject`
|
|
|
|
Only if infra automation updates lifecycle rules:
|
|
|
|
- `s3:PutLifecycleConfiguration`
|
|
|
|
## S3 Lifecycle Rules (Manual if no IaC)
|
|
|
|
Because this repository does not define Terraform/CDK/CloudFormation for S3 bucket lifecycle, configure lifecycle rules in your infra/deployment stack:
|
|
|
|
1. Rule `gc=temp`:
|
|
- Filter: object tag `gc=temp`
|
|
- Expire after `TEMP_FILE_TTL_DAYS` (recommended default 7 days)
|
|
|
|
2. Optional rule `gc=delete`:
|
|
- Filter: object tag `gc=delete`
|
|
- Expire after 1 day
|
|
|
|
Do not use lifecycle expiration as the orphan grace mechanism. Grace period is enforced by DB `orphaned_at`.
|
|
|
|
## Rollout Plan
|
|
|
|
1. Deploy schema + code with defaults (`cleanup job off`, `dry-run on`).
|
|
2. Enable tagging for new uploads (`ENABLE_S3_TAGGING=true`).
|
|
3. Validate attach/detach transitions in logs.
|
|
4. Run backfill dry-run and review report.
|
|
5. Enable cleanup job in dry-run mode.
|
|
6. Switch cleanup dry-run off after validation.
|
|
7. Apply S3 lifecycle rules for `gc=temp` (and optional `gc=delete`).
|
|
|
|
## Rollback Plan
|
|
|
|
1. Set `ENABLE_FILE_CLEANUP_JOB=false`.
|
|
2. Set `FILE_CLEANUP_DRY_RUN=true`.
|
|
3. Optionally set `ENABLE_S3_TAGGING=false`.
|
|
4. Pause/remove lifecycle rule `gc=delete` if enabled.
|