# Wucher Backend

Wucher is a Go backend built with GoFiber. The runtime uses MySQL for persistence and transient auth state, Amazon SQS for asynchronous email jobs, and Amazon SES v2 for email delivery.

## Prerequisites

- Go 1.25+
- MySQL
- LocalStack or AWS SQS/SES access for queue testing

## Project Structure

- `cmd/api/` API bootstrap
- `cmd/worker/` SQS worker bootstrap
- `internal/app/` application wiring
- `internal/resilience/` circuit breaker executor and dependency error classifiers
- `internal/service/` business logic
- `internal/transport/http/` GoFiber handlers and middleware
- `internal/repository/mysql/` MySQL repositories and transient stores
- `internal/repository/sqs/` SQS publisher/consumer adapter
- `internal/queue/` queue contracts, serializer, retry policy, and worker loop

## Runtime Architecture

- HTTP handlers stay thin and delegate to services.
- Auth and other business writes persist to MySQL first.
- API/service code never sends email synchronously.
- Email jobs are written to `email_outbox_messages` when outbox is enabled, or published directly to SQS when outbox is disabled.
- Worker publishes outbox rows to the SQS send queue, consumes SQS messages with long polling, and sends email through Amazon SES v2.
- SES bounce, complaint, and delivery tracking should use a separate flow:
  - SES Configuration Set
  - SNS topic subscription
  - dedicated SQS event queue
- Circuit breakers are applied only at outbound dependency boundaries:
  - Microsoft Entra token exchange
  - SES delivery
  - SQS publish/consume/ack calls
- Transient auth state previously kept in Redis is now stored in MySQL tables:
  - `auth_runtime_tokens`
  - `queue_idempotency_records`

`gorm` auto-migration creates these tables on startup.

## Getting Started

Install dependencies:

```bash
go mod tidy
```

Create local config from the example:

```bash
cp .env.example .env
```

Run the API:

```bash
go run ./cmd/api
```

Run the worker:

```bash
go run ./cmd/worker
```

Worker monitor endpoints:

```bash
curl http://127.0.0.1:9090/health
curl http://127.0.0.1:9090/metrics
curl http://127.0.0.1:9091/health
curl http://127.0.0.1:9091/metrics
```

- `/metrics` exposes Prometheus format for API and worker processes.
- `/health` remains JSON health/worker snapshot.

## Environment Variables

Example values are in `.env` for local development.

Important groups:

- Email delivery
  - `EMAIL_PROVIDER`
  - `EMAIL_MOCK_DIR`
- MySQL
  - `MYSQL_DSN`
  - `MYSQL_MAX_OPEN_CONNS`
  - `MYSQL_MAX_IDLE_CONNS`
  - `MYSQL_CONN_MAX_LIFETIME`
  - `MYSQL_CONN_MAX_IDLE_TIME`
- Auth
  - `AUTH_EMAIL_VERIFY_URL`
  - `AUTH_PASSWORD_RESET_URL`
  - `AUTH_SECURITY_PIN_RESET_URL`
  - `AUTH_TOTP_CHALLENGE_TTL`
  - `AUTH_SSO_STATE_TTL`
  - `AUTH_JWT_ACCESS_SECRET`
  - `AUTH_JWT_REFRESH_SECRET`
  - `AUTH_WEBAUTHN_*`
- Queue / worker
  - `QUEUE_MESSAGE_SCHEMA_VERSION`
  - `QUEUE_ENABLE_LEGACY_PAYLOAD_FALLBACK`
  - `QUEUE_IDEMPOTENCY_KEY_PREFIX`
  - `QUEUE_IDEMPOTENCY_TTL`
  - `QUEUE_PROCESSING_TTL`
  - `QUEUE_WORKER_CONCURRENCY`
  - `QUEUE_WORKER_MAX_BATCH_SIZE`
  - `QUEUE_WORKER_PROCESS_TIMEOUT`
  - `QUEUE_WORKER_BACKOFF_MIN`
  - `QUEUE_WORKER_BACKOFF_MAX`
  - `QUEUE_WORKER_PERMANENT_FAILURE_DELAY`
  - `QUEUE_OUTBOX_ENABLED`
  - `QUEUE_OUTBOX_POLL_INTERVAL`
  - `QUEUE_OUTBOX_BATCH_SIZE`
  - `QUEUE_OUTBOX_DISPATCH_TIMEOUT`
  - `QUEUE_OUTBOX_LOCK_TTL`
  - `QUEUE_OUTBOX_MAX_ATTEMPTS`
- File queue / file worker
  - `FILE_QUEUE_MESSAGE_SCHEMA_VERSION`
  - `FILE_QUEUE_IDEMPOTENCY_KEY_PREFIX`
  - `FILE_QUEUE_IDEMPOTENCY_TTL`
  - `FILE_QUEUE_PROCESSING_TTL`
  - `FILE_QUEUE_WORKER_CONCURRENCY`
  - `FILE_QUEUE_WORKER_MAX_BATCH_SIZE`
  - `FILE_QUEUE_WORKER_PROCESS_TIMEOUT`
  - `FILE_QUEUE_WORKER_SHUTDOWN_TIMEOUT`
  - `FILE_QUEUE_WORKER_BACKOFF_MIN`
  - `FILE_QUEUE_WORKER_BACKOFF_MAX`
  - `FILE_QUEUE_WORKER_PERMANENT_FAILURE_DELAY`
  - `FILE_QUEUE_WORKER_MONITOR_ADDR`
  - `FILE_QUEUE_WORKER_HEALTH_ERROR_THRESHOLD`
  - `FILE_QUEUE_WORKER_DEPTH_POLL_INTERVAL`
  - `FILE_QUEUE_OUTBOX_ENABLED`
  - `FILE_QUEUE_OUTBOX_POLL_INTERVAL`
  - `FILE_QUEUE_OUTBOX_BATCH_SIZE`
  - `FILE_QUEUE_OUTBOX_DISPATCH_TIMEOUT`
  - `FILE_QUEUE_OUTBOX_LOCK_TTL`
  - `FILE_QUEUE_OUTBOX_MAX_ATTEMPTS`
- Circuit breakers
  - `CB_DEFAULT_ENABLED`
  - `CB_DEFAULT_MAX_REQUESTS`
  - `CB_DEFAULT_INTERVAL`
  - `CB_DEFAULT_BUCKET_PERIOD`
  - `CB_DEFAULT_TIMEOUT`
  - `CB_DEFAULT_MIN_REQUESTS`
  - `CB_DEFAULT_FAILURE_RATIO`
  - `CB_DEFAULT_CONSECUTIVE_FAILURES`
  - `CB_MICROSOFT_SSO_ENABLED`
  - `CB_SES_ENABLED`
  - `CB_SQS_ENABLED`
- AWS / SES
  - `AWS_REGION`
  - `SES_REGION`
  - `SES_FROM`
  - `SES_CONFIGURATION_SET`
  - `SES_SEND_TIMEOUT`
  - `SES_ENDPOINT`
  - `SES_ALLOW_INSECURE_ENDPOINT`
- AWS / SQS
  - Email queue:
  - `AWS_REGION`
  - `AWS_ACCESS_KEY_ID`
  - `AWS_SECRET_ACCESS_KEY`
  - `SQS_ENDPOINT`
  - `SQS_ALLOW_INSECURE_ENDPOINT`
  - `SQS_QUEUE_URL`
  - `SQS_QUEUE_TYPE`
  - `SQS_MESSAGE_GROUP_ID`
  - `SQS_MAX_MESSAGES`
  - `SQS_WAIT_TIME_SECONDS`
  - `SQS_VISIBILITY_TIMEOUT`
  - File queue:
  - `FILE_SQS_REGION`
  - `FILE_SQS_ENDPOINT`
  - `FILE_SQS_QUEUE_URL`
  - `FILE_SQS_QUEUE_TYPE`
  - `FILE_SQS_MESSAGE_GROUP_ID`
  - `FILE_SQS_MAX_MESSAGES`
  - `FILE_SQS_WAIT_TIME_SECONDS`
  - `FILE_SQS_VISIBILITY_TIMEOUT`
  - `FILE_SQS_ALLOW_INSECURE_ENDPOINT`
- File storage (S3)
  - `FILE_STORAGE_PROVIDER`
  - `FILE_S3_BUCKET`
  - `FILE_S3_REGION`
  - `FILE_S3_ENDPOINT`
  - `FILE_S3_FORCE_PATH_STYLE`
  - `FILE_S3_ALLOW_INSECURE_ENDPOINT`
  - `FILE_S3_AUTO_CREATE_BUCKET`
  - `FILE_OBJECT_KEY_PREFIX`
  - `FILE_S3_PRESIGN_PUT_TTL`
  - `FILE_S3_PRESIGN_GET_TTL`
- WOPI / Collabora
  - `WOPI_TOKEN_SECRET`
  - `WOPI_PROOF_SECRET`
  - `WOPI_DISCOVERY_URL`
  - `WOPI_PUBLIC_BASE_URL`
  - `WOPI_EDIT_ACTION_URL`
  - `WOPI_TOKEN_TTL`
  - `WOPI_MAX_BODY_BYTES`
  - `WOPI_REQUIRE_PROOF`
- Worker mode selector
  - `WORKER_JOB=email`
  - `WORKER_JOB=file_processing`

## WOPI / Collabora Hardening

- Keep WOPI endpoints behind the dedicated `/wopi` router and do not expose them through the session-authenticated app middleware chain.
- Use HTTPS end to end in production.
- Restrict network access to the Collabora host or private network allowlist.
- Set Collabora `aliasgroup`/host configuration so it can only call back to this backend.
- Keep `WOPI_TOKEN_SECRET` and `WOPI_PROOF_SECRET` in env or a secrets manager and rotate them periodically.
- Scope IAM to the minimum S3 bucket/prefix needed by file storage.
- Keep direct S3 URLs disabled for editor access; the backend should proxy file bytes.
- Ensure backend logs capture open/save/exit-save events for auditability.
- Verify `/hosting/discovery` reachability and S3 connectivity in health checks before enabling Collabora traffic.

Per-dependency overrides are supported with the same shape as the default block:

- `CB_MICROSOFT_SSO_*`
- `CB_SES_*`
- `CB_SQS_*`

Example: `CB_SES_TIMEOUT=45s` or `CB_SQS_CONSECUTIVE_FAILURES=3`.

## Circuit Breakers

- Breakers are scoped per outbound dependency, not per HTTP endpoint and not global middleware.
- MySQL repositories and internal business logic are intentionally not wrapped.
- Only dependency-health failures count toward opening the breaker.
  - Microsoft SSO: network errors, timeouts, HTTP `429`, and `5xx`
  - SES: network errors, throttling, and AWS server faults
  - SQS: network errors, throttling, and AWS server faults
- Client/config/input errors are excluded so they do not open the breaker.
- State transitions are logged through the standard application logger.

## Queue Behavior

- Long polling is enabled through `SQS_WAIT_TIME_SECONDS`.
- Messages are deleted only after successful processing or explicit ack of duplicates.
- Transient failures use `ChangeMessageVisibility` with backoff.
- Permanent failures are left unacked so SQS redrive policy can move them to the DLQ.
- The default producer schema is `v2`; legacy payload fallback remains available for older messages.
- Configure DLQ on the queue itself in AWS/LocalStack. The application no longer publishes directly to a DLQ URL.

## Prometheus Metrics

Scrape endpoints:

- API: `http://<api-host>:8080/metrics`
- Email worker monitor: `http://<worker-host>:9090/metrics`
- File worker monitor: `http://<worker-host>:9091/metrics`

Example Prometheus scrape config:

```yaml
scrape_configs:
  - job_name: "wucher-api"
    static_configs:
      - targets: ["127.0.0.1:8080"]
  - job_name: "wucher-email-worker"
    static_configs:
      - targets: ["127.0.0.1:9090"]
  - job_name: "wucher-file-worker"
    static_configs:
      - targets: ["127.0.0.1:9091"]
```

Metrics and labels:

- `app_aws_requests_total{component,service,operation,result}`
- `app_aws_request_duration_seconds{component,service,operation}`
- `app_aws_errors_total{component,service,operation,error_class}` where `error_class` is `4xx`, `5xx`, or `other`
- `app_aws_transferred_bytes_total{component,service,operation,direction}`
- `app_queue_messages_received_total{component,queue_name}`
- `app_queue_messages_deleted_total{component,queue_name}`
- `app_queue_visibility_changes_total{component,queue_name,result}`
- `app_queue_send_total{component,queue_name,result}`
- `app_queue_retries_total{component,queue_name,result}`
- `app_queue_message_bytes_total{component,queue_name,direction}`
- `app_queue_depth{component,queue_name}`
- `app_worker_jobs_processed_total{component,job_type,result}`
- `app_worker_job_duration_seconds{component,job_type}`
- `app_email_send_total{component,provider,result}`
- `app_email_send_duration_seconds{component,provider}`
- `app_file_upload_total{component,result}`
- `app_file_delete_total{component,result}`
- `app_file_head_total{component,result}`
- `app_file_presign_download_total{component,result}`

PromQL examples:

- Error rate per AWS operation:

```promql
sum by (component, service, operation) (rate(app_aws_requests_total{result="error"}[5m]))
/
clamp_min(sum by (component, service, operation) (rate(app_aws_requests_total[5m])), 1e-9)
```

- P95 latency per AWS operation:

```promql
histogram_quantile(
  0.95,
  sum by (le, component, service, operation) (
    rate(app_aws_request_duration_seconds_bucket[5m])
  )
)
```

- Queue depth:

```promql
max by (component, queue_name) (app_queue_depth)
```

- Worker success/failure count:

```promql
sum by (component, job_type, result) (increase(app_worker_jobs_processed_total[15m]))
```

- SES send failure rate:

```promql
sum by (component) (rate(app_email_send_total{provider="ses", result="error"}[5m]))
/
clamp_min(sum by (component) (rate(app_email_send_total{provider="ses"}[5m])), 1e-9)
```

- S3 4xx/5xx error rate:

```promql
sum by (operation, error_class) (rate(app_aws_errors_total{service="s3"}[5m]))
```

- S3 GetObject transferred bytes (estimated from presign download flow):

```promql
sum by (component) (rate(app_aws_transferred_bytes_total{service="s3",operation="getobject",direction="download"}[5m]))
```

- SQS message bytes in/out:

```promql
sum by (component, queue_name, direction) (rate(app_queue_message_bytes_total[5m]))
```

- SQS retry/requeue count:

```promql
sum by (component, queue_name, result) (increase(app_queue_retries_total[15m]))
```

## File Manager

- Queue separation:
  - Email queue: `app-queue` (+ `app-queue-dlq`)
  - File queue: `app-file-queue` (+ `app-file-queue-dlq`)
- `FILE_SQS_QUEUE_URL` must be different from `SQS_QUEUE_URL`.
- API startup validates queue separation and refuses startup on misconfiguration.
- Upload writes file metadata + file outbox durably; file queue publish is handled asynchronously by the file outbox dispatcher.

Local run checklist:

```bash
# 1) Start localstack (queue + bucket bootstrap script will run)
cd ../localstack && docker compose up -d

WORKER_JOB=file_processing go run ./cmd/worker
```

## SES Delivery Notes

- `EMAIL_PROVIDER=mock` is the recommended local/dev default when you do not have real AWS SES access.
- Mock delivery writes one JSON file per delivered email into `EMAIL_MOCK_DIR`.
- `SES_FROM` must be a verified identity or domain in Amazon SES.
- `SES_ENDPOINT` should be blank in production and only overridden for local/dev stacks such as LocalStack.
- Do not point SES events to the main send queue. Use `SES_CONFIGURATION_SET` with SNS -> separate SQS event queue.
- The codebase includes an SES event parser for SNS-wrapped notifications, but an event consumer is not started by default.

## Migration Notes

Detailed migration notes are in [docs/queue-sqs-migration.md](docs/queue-sqs-migration.md).
Data migration tracking wiki is in [docs/data-migration-wiki.md](docs/data-migration-wiki.md).

Main breaking changes:

- Redis runtime support has been removed from the codebase.
- Redis env vars are obsolete and should be removed from deployment config.
- SMTP runtime env/config has been removed in favor of SES.
- Worker now relies on MySQL for queue idempotency and auth/runtime state persistence.
- SQS must have a redrive policy and DLQ configured outside the application.

## CI / Build Selection

The pipeline always builds and pushes the `api` image on every push to `development` or `staging`.

To also build and push the worker or file-worker image, include the corresponding tag anywhere in the commit message:

| Tag | Image built | Docker target |
|---|---|---|
| `#PUSH-WORKER` | `mybitlab/wfm:<tag>-worker` | `worker` |
| `#PUSH-FILE-WORKER` | `mybitlab/wfm:<tag>-file-worker` | `file-worker` |

Both tags can appear in the same commit message to build all three in one pipeline run:

```
ci: deploy all services #PUSH-WORKER #PUSH-FILE-WORKER
```

Omitting a tag skips that image entirely — it will not be built or pushed.

## Testing

Run the full test suite:

```bash
go test ./...
```

Generate coverage:

```bash
go test ./... -coverprofile=coverage.out
go tool cover -func=coverage.out
go tool cover -html=coverage.out -o coverage.html
```