fm_be/docs/queue-sqs-migration.md

Queue Migration Notes

This project no longer supports Redis as a runtime dependency. Queue publishing and consumption are SQS-only, while transient auth/runtime state is stored in MySQL.

Current Flow

1. API writes business data to MySQL.
2. If QUEUE_OUTBOX_ENABLED=true, email jobs are persisted to email_outbox_messages.
3. Worker polls the outbox and publishes serialized messages to SQS.
4. Worker consumes SQS messages with long polling and sends email through Amazon SES v2.
5. Successful processing deletes the SQS message.
6. Failed processing leaves the message unacked so SQS visibility timeout and DLQ redrive policy handle retries.

What Changed

• Removed Redis producer, consumer, and Redis-backed idempotency paths.
• Removed Redis token store used for:
  • refresh/session bookkeeping
  • SSO state
  • TOTP/WebAuthn challenges
  • security PIN lock/cooldown state
• Added MySQL-backed transient stores:
  • auth_runtime_tokens
  • queue_idempotency_records
• Simplified bootstrap:
  • API uses MySQL + optional direct SQS publisher when outbox is disabled.
  • Worker uses MySQL + SQS only.

Required Environment

Queue and worker settings:

EMAIL_PROVIDER=mock
EMAIL_MOCK_DIR=.local/emails

QUEUE_MESSAGE_SCHEMA_VERSION=v2
QUEUE_ENABLE_LEGACY_PAYLOAD_FALLBACK=true
QUEUE_IDEMPOTENCY_KEY_PREFIX=queue:idempotency:email:
QUEUE_IDEMPOTENCY_TTL=24h
QUEUE_PROCESSING_TTL=15m
QUEUE_WORKER_CONCURRENCY=4
QUEUE_WORKER_MAX_BATCH_SIZE=10
QUEUE_WORKER_PROCESS_TIMEOUT=30s
QUEUE_WORKER_SHUTDOWN_TIMEOUT=30s
QUEUE_WORKER_BACKOFF_MIN=1s
QUEUE_WORKER_BACKOFF_MAX=5m
QUEUE_WORKER_PERMANENT_FAILURE_DELAY=30s
QUEUE_WORKER_MONITOR_ADDR=127.0.0.1:9090
QUEUE_WORKER_HEALTH_ERROR_THRESHOLD=3
QUEUE_WORKER_DEPTH_POLL_INTERVAL=30s
QUEUE_OUTBOX_ENABLED=true
QUEUE_OUTBOX_POLL_INTERVAL=1s
QUEUE_OUTBOX_BATCH_SIZE=10
QUEUE_OUTBOX_DISPATCH_TIMEOUT=10s
QUEUE_OUTBOX_LOCK_TTL=1m
QUEUE_OUTBOX_MAX_ATTEMPTS=20

SES settings:

AWS_REGION=ap-southeast-1
SES_REGION=
SES_FROM=no-reply@example.com
SES_CONFIGURATION_SET=
SES_SEND_TIMEOUT=10s
SES_ENDPOINT=
SES_ALLOW_INSECURE_ENDPOINT=false

Local/dev recommendation when you do not have AWS SES:

• set EMAIL_PROVIDER=mock
• keep SQS on LocalStack if needed
• inspect delivered mock emails in EMAIL_MOCK_DIR

AWS / SQS settings:

AWS_REGION=ap-southeast-1
SQS_ENDPOINT=
SQS_ALLOW_INSECURE_ENDPOINT=false
SQS_QUEUE_URL=https://sqs.ap-southeast-1.amazonaws.com/123456789012/wucher-email
SQS_QUEUE_TYPE=standard
SQS_MESSAGE_GROUP_ID=email
SQS_MAX_MESSAGES=10
SQS_WAIT_TIME_SECONDS=20
SQS_VISIBILITY_TIMEOUT=60s

SQS Expectations

Configure the queue outside the application with:

• long polling enabled
• sensible visibility timeout
• DLQ redrive policy
• maxReceiveCount aligned with retry tolerance

Recommended baseline:

• ReceiveMessageWaitTimeSeconds=20
• VisibilityTimeout=60
• maxReceiveCount=5

IAM

API needs sqs:SendMessage only when QUEUE_OUTBOX_ENABLED=false.

Worker needs:

• sqs:SendMessage to the main queue when dispatching outbox messages
• sqs:ReceiveMessage
• sqs:DeleteMessageBatch
• sqs:ChangeMessageVisibility
• sqs:GetQueueAttributes
• ses:SendEmail

If SES event tracking is enabled, provision it separately with:

• SES configuration set event destinations
• SNS topic publish permissions
• a dedicated SQS event queue and consumer policy

Breaking Changes

• Remove these Redis env vars from deployment:
  • REDIS_ADDR
  • REDIS_PASSWORD
  • REDIS_DB
  • QUEUE_PRIMARY_BACKEND
  • QUEUE_SECONDARY_BACKEND
  • QUEUE_SECONDARY_REQUIRED
  • QUEUE_CONSUMER_BACKEND
  • QUEUE_IDEMPOTENCY_BACKEND
  • QUEUE_PRODUCER_MODE
  • QUEUE_REDIS_DLQ_KEY
  • AUTH_EMAIL_QUEUE_KEY
  • SQS_DLQ_URL
• Worker now requires MySQL even when it is only consuming SQS, because idempotency state is persisted in MySQL.
• Old Redis backlog is not consumed by this version. Drain or discard legacy Redis queues before deploy.

Deployment Checklist

1. Ensure MySQL migration/AutoMigrate runs and creates auth_runtime_tokens and queue_idempotency_records.
2. Create or verify the SQS queue and DLQ redrive policy.
3. Verify the SES identity/domain and optional configuration set.
4. Remove Redis and SMTP config from runtime manifests and secrets.
5. Deploy API and worker.
6. Verify:
   • worker /health
   • worker /metrics
   • SQS queue depth
   • outbox rows are progressing from pending to published
   • SES send acceptance/message IDs in worker logs
7. Manually delete Redis infrastructure after confirming there is no remaining backlog you still need.