158 lines
4.5 KiB
Markdown
158 lines
4.5 KiB
Markdown
# Queue Migration Notes
|
|
|
|
This project no longer supports Redis as a runtime dependency. Queue publishing and consumption are SQS-only, while transient auth/runtime state is stored in MySQL.
|
|
|
|
## Current Flow
|
|
|
|
1. API writes business data to MySQL.
|
|
2. If `QUEUE_OUTBOX_ENABLED=true`, email jobs are persisted to `email_outbox_messages`.
|
|
3. Worker polls the outbox and publishes serialized messages to SQS.
|
|
4. Worker consumes SQS messages with long polling and sends email through Amazon SES v2.
|
|
5. Successful processing deletes the SQS message.
|
|
6. Failed processing leaves the message unacked so SQS visibility timeout and DLQ redrive policy handle retries.
|
|
|
|
## What Changed
|
|
|
|
- Removed Redis producer, consumer, and Redis-backed idempotency paths.
|
|
- Removed Redis token store used for:
|
|
- refresh/session bookkeeping
|
|
- SSO state
|
|
- TOTP/WebAuthn challenges
|
|
- security PIN lock/cooldown state
|
|
- Added MySQL-backed transient stores:
|
|
- `auth_runtime_tokens`
|
|
- `queue_idempotency_records`
|
|
- Simplified bootstrap:
|
|
- API uses MySQL + optional direct SQS publisher when outbox is disabled.
|
|
- Worker uses MySQL + SQS only.
|
|
|
|
## Required Environment
|
|
|
|
Queue and worker settings:
|
|
|
|
```bash
|
|
EMAIL_PROVIDER=mock
|
|
EMAIL_MOCK_DIR=.local/emails
|
|
|
|
QUEUE_MESSAGE_SCHEMA_VERSION=v2
|
|
QUEUE_ENABLE_LEGACY_PAYLOAD_FALLBACK=true
|
|
QUEUE_IDEMPOTENCY_KEY_PREFIX=queue:idempotency:email:
|
|
QUEUE_IDEMPOTENCY_TTL=24h
|
|
QUEUE_PROCESSING_TTL=15m
|
|
QUEUE_WORKER_CONCURRENCY=4
|
|
QUEUE_WORKER_MAX_BATCH_SIZE=10
|
|
QUEUE_WORKER_PROCESS_TIMEOUT=30s
|
|
QUEUE_WORKER_SHUTDOWN_TIMEOUT=30s
|
|
QUEUE_WORKER_BACKOFF_MIN=1s
|
|
QUEUE_WORKER_BACKOFF_MAX=5m
|
|
QUEUE_WORKER_PERMANENT_FAILURE_DELAY=30s
|
|
QUEUE_WORKER_MONITOR_ADDR=127.0.0.1:9090
|
|
QUEUE_WORKER_HEALTH_ERROR_THRESHOLD=3
|
|
QUEUE_WORKER_DEPTH_POLL_INTERVAL=30s
|
|
QUEUE_OUTBOX_ENABLED=true
|
|
QUEUE_OUTBOX_POLL_INTERVAL=1s
|
|
QUEUE_OUTBOX_BATCH_SIZE=10
|
|
QUEUE_OUTBOX_DISPATCH_TIMEOUT=10s
|
|
QUEUE_OUTBOX_LOCK_TTL=1m
|
|
QUEUE_OUTBOX_MAX_ATTEMPTS=20
|
|
```
|
|
|
|
SES settings:
|
|
|
|
```bash
|
|
AWS_REGION=ap-southeast-1
|
|
SES_REGION=
|
|
SES_FROM=no-reply@example.com
|
|
SES_CONFIGURATION_SET=
|
|
SES_SEND_TIMEOUT=10s
|
|
SES_ENDPOINT=
|
|
SES_ALLOW_INSECURE_ENDPOINT=false
|
|
```
|
|
|
|
Local/dev recommendation when you do not have AWS SES:
|
|
|
|
- set `EMAIL_PROVIDER=mock`
|
|
- keep SQS on LocalStack if needed
|
|
- inspect delivered mock emails in `EMAIL_MOCK_DIR`
|
|
|
|
AWS / SQS settings:
|
|
|
|
```bash
|
|
AWS_REGION=ap-southeast-1
|
|
SQS_ENDPOINT=
|
|
SQS_ALLOW_INSECURE_ENDPOINT=false
|
|
SQS_QUEUE_URL=https://sqs.ap-southeast-1.amazonaws.com/123456789012/wucher-email
|
|
SQS_QUEUE_TYPE=standard
|
|
SQS_MESSAGE_GROUP_ID=email
|
|
SQS_MAX_MESSAGES=10
|
|
SQS_WAIT_TIME_SECONDS=20
|
|
SQS_VISIBILITY_TIMEOUT=60s
|
|
```
|
|
|
|
## SQS Expectations
|
|
|
|
Configure the queue outside the application with:
|
|
|
|
- long polling enabled
|
|
- sensible visibility timeout
|
|
- DLQ redrive policy
|
|
- `maxReceiveCount` aligned with retry tolerance
|
|
|
|
Recommended baseline:
|
|
|
|
- `ReceiveMessageWaitTimeSeconds=20`
|
|
- `VisibilityTimeout=60`
|
|
- `maxReceiveCount=5`
|
|
|
|
## IAM
|
|
|
|
API needs `sqs:SendMessage` only when `QUEUE_OUTBOX_ENABLED=false`.
|
|
|
|
Worker needs:
|
|
|
|
- `sqs:SendMessage` to the main queue when dispatching outbox messages
|
|
- `sqs:ReceiveMessage`
|
|
- `sqs:DeleteMessageBatch`
|
|
- `sqs:ChangeMessageVisibility`
|
|
- `sqs:GetQueueAttributes`
|
|
- `ses:SendEmail`
|
|
|
|
If SES event tracking is enabled, provision it separately with:
|
|
|
|
- SES configuration set event destinations
|
|
- SNS topic publish permissions
|
|
- a dedicated SQS event queue and consumer policy
|
|
|
|
## Breaking Changes
|
|
|
|
- Remove these Redis env vars from deployment:
|
|
- `REDIS_ADDR`
|
|
- `REDIS_PASSWORD`
|
|
- `REDIS_DB`
|
|
- `QUEUE_PRIMARY_BACKEND`
|
|
- `QUEUE_SECONDARY_BACKEND`
|
|
- `QUEUE_SECONDARY_REQUIRED`
|
|
- `QUEUE_CONSUMER_BACKEND`
|
|
- `QUEUE_IDEMPOTENCY_BACKEND`
|
|
- `QUEUE_PRODUCER_MODE`
|
|
- `QUEUE_REDIS_DLQ_KEY`
|
|
- `AUTH_EMAIL_QUEUE_KEY`
|
|
- `SQS_DLQ_URL`
|
|
- Worker now requires MySQL even when it is only consuming SQS, because idempotency state is persisted in MySQL.
|
|
- Old Redis backlog is not consumed by this version. Drain or discard legacy Redis queues before deploy.
|
|
|
|
## Deployment Checklist
|
|
|
|
1. Ensure MySQL migration/AutoMigrate runs and creates `auth_runtime_tokens` and `queue_idempotency_records`.
|
|
2. Create or verify the SQS queue and DLQ redrive policy.
|
|
3. Verify the SES identity/domain and optional configuration set.
|
|
4. Remove Redis and SMTP config from runtime manifests and secrets.
|
|
5. Deploy API and worker.
|
|
6. Verify:
|
|
- worker `/health`
|
|
- worker `/metrics`
|
|
- SQS queue depth
|
|
- outbox rows are progressing from `pending` to `published`
|
|
- SES send acceptance/message IDs in worker logs
|
|
7. Manually delete Redis infrastructure after confirming there is no remaining backlog you still need.
|