# File Lifecycle and S3 Cleanup

## Lifecycle States (DB source of truth)

`file_files.lifecycle_status`:

- `temp`: uploaded but not attached
- `active`: referenced by one or more business entities
- `orphan_pending`: currently unreferenced, waiting grace period
- `deleted`: backend cleanup has deleted object(s) and marked terminal state

Supporting timestamps:

- `created_at`: existing file creation time
- `attached_at`: set when file becomes active
- `orphaned_at`: set when file becomes unreferenced
- `lifecycle_deleted_at`: set when cleanup marks lifecycle `deleted`

Notes:

- Existing `status` column (`uploaded/processing/validated/ready/trashed`) remains for processing/trash flow.
- `deleted_at` is still used by trash semantics. Cleanup lifecycle uses `lifecycle_deleted_at`.

## S3 Tags

Single tag key: `gc`

- `gc=temp`: temporary uploads
- `gc=keep`: active or protected files
- `gc=delete`: optional marker immediately before backend delete

S3 tags are markers only. DB lifecycle state and DB reference checks are authoritative.

## Cleanup Job

Scheduler: API process loop (`startFileManagerLifecycleCleanupLoop`).

Behavior:

1. Query DB for `orphan_pending` files where `orphaned_at <= now - ORPHAN_GRACE_PERIOD_DAYS`.
2. Process in batches (`FILE_CLEANUP_BATCH_SIZE`).
3. Re-check DB references per file before deletion.
4. If referenced again: mark `active`, set/keep `gc=keep`.
5. If still unreferenced:
   - optional `gc=delete` tag
   - delete original and thumbnail S3 objects
   - mark lifecycle `deleted` and set `lifecycle_deleted_at`
6. Per-file error handling; one failure does not stop the batch.

The job is idempotent and retry-safe.

## Environment Variables

- `ENABLE_S3_TAGGING` (default `true`)
- `ENABLE_FILE_CLEANUP_JOB` (default `false`)
- `TEMP_FILE_TTL_DAYS` (default `7`)
- `ORPHAN_GRACE_PERIOD_DAYS` (default `30`)
- `FILE_CLEANUP_BATCH_SIZE` (default `100`)
- `FILE_CLEANUP_DRY_RUN` (default `true`)

## Backfill (Dry-Run First)

Command:

```bash
go run ./cmd/file-lifecycle-backfill
```

Optional:

- `BACKFILL_DRY_RUN=true|false` (default `true`)

Output report includes:

- active files
- probable orphan files
- missing S3 key
- invalid references
- skipped records

## Required IAM Permissions

Minimum required:

- `s3:PutObjectTagging`
- `s3:GetObjectTagging`
- `s3:DeleteObject`
- `s3:GetObject`
- `s3:HeadObject`

Only if infra automation updates lifecycle rules:

- `s3:PutLifecycleConfiguration`

## S3 Lifecycle Rules (Manual if no IaC)

Because this repository does not define Terraform/CDK/CloudFormation for S3 bucket lifecycle, configure lifecycle rules in your infra/deployment stack:

1. Rule `gc=temp`:
   - Filter: object tag `gc=temp`
   - Expire after `TEMP_FILE_TTL_DAYS` (recommended default 7 days)

2. Optional rule `gc=delete`:
   - Filter: object tag `gc=delete`
   - Expire after 1 day

Do not use lifecycle expiration as the orphan grace mechanism. Grace period is enforced by DB `orphaned_at`.

## Rollout Plan

1. Deploy schema + code with defaults (`cleanup job off`, `dry-run on`).
2. Enable tagging for new uploads (`ENABLE_S3_TAGGING=true`).
3. Validate attach/detach transitions in logs.
4. Run backfill dry-run and review report.
5. Enable cleanup job in dry-run mode.
6. Switch cleanup dry-run off after validation.
7. Apply S3 lifecycle rules for `gc=temp` (and optional `gc=delete`).

## Rollback Plan

1. Set `ENABLE_FILE_CLEANUP_JOB=false`.
2. Set `FILE_CLEANUP_DRY_RUN=true`.
3. Optionally set `ENABLE_S3_TAGGING=false`.
4. Pause/remove lifecycle rule `gc=delete` if enabled.