fm_be/README.md

Wucher Backend

Wucher is a Go backend built with GoFiber. The runtime uses MySQL for persistence and transient auth state, Amazon SQS for asynchronous email jobs, and Amazon SES v2 for email delivery.

Prerequisites

• Go 1.25+
• MySQL
• LocalStack or AWS SQS/SES access for queue testing

Project Structure

• cmd/api/ API bootstrap
• cmd/worker/ SQS worker bootstrap
• internal/app/ application wiring
• internal/resilience/ circuit breaker executor and dependency error classifiers
• internal/service/ business logic
• internal/transport/http/ GoFiber handlers and middleware
• internal/repository/mysql/ MySQL repositories and transient stores
• internal/repository/sqs/ SQS publisher/consumer adapter
• internal/queue/ queue contracts, serializer, retry policy, and worker loop

Runtime Architecture

• HTTP handlers stay thin and delegate to services.
• Auth and other business writes persist to MySQL first.
• API/service code never sends email synchronously.
• Email jobs are written to email_outbox_messages when outbox is enabled, or published directly to SQS when outbox is disabled.
• Worker publishes outbox rows to the SQS send queue, consumes SQS messages with long polling, and sends email through Amazon SES v2.
• SES bounce, complaint, and delivery tracking should use a separate flow:
  • SES Configuration Set
  • SNS topic subscription
  • dedicated SQS event queue
• Circuit breakers are applied only at outbound dependency boundaries:
  • Microsoft Entra token exchange
  • SES delivery
  • SQS publish/consume/ack calls
• Transient auth state previously kept in Redis is now stored in MySQL tables:
  • auth_runtime_tokens
  • queue_idempotency_records

gorm auto-migration creates these tables on startup.

Getting Started

Install dependencies:

go mod tidy

Create local config from the example:

cp .env.example .env

Run the API:

go run ./cmd/api

Run the worker:

go run ./cmd/worker

Worker monitor endpoints:

curl http://127.0.0.1:9090/health
curl http://127.0.0.1:9090/metrics
curl http://127.0.0.1:9091/health
curl http://127.0.0.1:9091/metrics

• /metrics exposes Prometheus format for API and worker processes.
• /health remains JSON health/worker snapshot.

Environment Variables

Example values are in .env for local development.

Important groups:

• Email delivery
  • EMAIL_PROVIDER
  • EMAIL_MOCK_DIR
• MySQL
  • MYSQL_DSN
  • MYSQL_MAX_OPEN_CONNS
  • MYSQL_MAX_IDLE_CONNS
  • MYSQL_CONN_MAX_LIFETIME
  • MYSQL_CONN_MAX_IDLE_TIME
• Auth
  • AUTH_EMAIL_VERIFY_URL
  • AUTH_PASSWORD_RESET_URL
  • AUTH_SECURITY_PIN_RESET_URL
  • AUTH_TOTP_CHALLENGE_TTL
  • AUTH_SSO_STATE_TTL
  • AUTH_JWT_ACCESS_SECRET
  • AUTH_JWT_REFRESH_SECRET
  • AUTH_WEBAUTHN_*
• Queue / worker
  • QUEUE_MESSAGE_SCHEMA_VERSION
  • QUEUE_ENABLE_LEGACY_PAYLOAD_FALLBACK
  • QUEUE_IDEMPOTENCY_KEY_PREFIX
  • QUEUE_IDEMPOTENCY_TTL
  • QUEUE_PROCESSING_TTL
  • QUEUE_WORKER_CONCURRENCY
  • QUEUE_WORKER_MAX_BATCH_SIZE
  • QUEUE_WORKER_PROCESS_TIMEOUT
  • QUEUE_WORKER_BACKOFF_MIN
  • QUEUE_WORKER_BACKOFF_MAX
  • QUEUE_WORKER_PERMANENT_FAILURE_DELAY
  • QUEUE_OUTBOX_ENABLED
  • QUEUE_OUTBOX_POLL_INTERVAL
  • QUEUE_OUTBOX_BATCH_SIZE
  • QUEUE_OUTBOX_DISPATCH_TIMEOUT
  • QUEUE_OUTBOX_LOCK_TTL
  • QUEUE_OUTBOX_MAX_ATTEMPTS
• File queue / file worker
  • FILE_QUEUE_MESSAGE_SCHEMA_VERSION
  • FILE_QUEUE_IDEMPOTENCY_KEY_PREFIX
  • FILE_QUEUE_IDEMPOTENCY_TTL
  • FILE_QUEUE_PROCESSING_TTL
  • FILE_QUEUE_WORKER_CONCURRENCY
  • FILE_QUEUE_WORKER_MAX_BATCH_SIZE
  • FILE_QUEUE_WORKER_PROCESS_TIMEOUT
  • FILE_QUEUE_WORKER_SHUTDOWN_TIMEOUT
  • FILE_QUEUE_WORKER_BACKOFF_MIN
  • FILE_QUEUE_WORKER_BACKOFF_MAX
  • FILE_QUEUE_WORKER_PERMANENT_FAILURE_DELAY
  • FILE_QUEUE_WORKER_MONITOR_ADDR
  • FILE_QUEUE_WORKER_HEALTH_ERROR_THRESHOLD
  • FILE_QUEUE_WORKER_DEPTH_POLL_INTERVAL
  • FILE_QUEUE_OUTBOX_ENABLED
  • FILE_QUEUE_OUTBOX_POLL_INTERVAL
  • FILE_QUEUE_OUTBOX_BATCH_SIZE
  • FILE_QUEUE_OUTBOX_DISPATCH_TIMEOUT
  • FILE_QUEUE_OUTBOX_LOCK_TTL
  • FILE_QUEUE_OUTBOX_MAX_ATTEMPTS
• Circuit breakers
  • CB_DEFAULT_ENABLED
  • CB_DEFAULT_MAX_REQUESTS
  • CB_DEFAULT_INTERVAL
  • CB_DEFAULT_BUCKET_PERIOD
  • CB_DEFAULT_TIMEOUT
  • CB_DEFAULT_MIN_REQUESTS
  • CB_DEFAULT_FAILURE_RATIO
  • CB_DEFAULT_CONSECUTIVE_FAILURES
  • CB_MICROSOFT_SSO_ENABLED
  • CB_SES_ENABLED
  • CB_SQS_ENABLED
• AWS / SES
  • AWS_REGION
  • SES_REGION
  • SES_FROM
  • SES_CONFIGURATION_SET
  • SES_SEND_TIMEOUT
  • SES_ENDPOINT
  • SES_ALLOW_INSECURE_ENDPOINT
• AWS / SQS
  • Email queue:
  • AWS_REGION
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • SQS_ENDPOINT
  • SQS_ALLOW_INSECURE_ENDPOINT
  • SQS_QUEUE_URL
  • SQS_QUEUE_TYPE
  • SQS_MESSAGE_GROUP_ID
  • SQS_MAX_MESSAGES
  • SQS_WAIT_TIME_SECONDS
  • SQS_VISIBILITY_TIMEOUT
  • File queue:
  • FILE_SQS_REGION
  • FILE_SQS_ENDPOINT
  • FILE_SQS_QUEUE_URL
  • FILE_SQS_QUEUE_TYPE
  • FILE_SQS_MESSAGE_GROUP_ID
  • FILE_SQS_MAX_MESSAGES
  • FILE_SQS_WAIT_TIME_SECONDS
  • FILE_SQS_VISIBILITY_TIMEOUT
  • FILE_SQS_ALLOW_INSECURE_ENDPOINT
• File storage (S3)
  • FILE_STORAGE_PROVIDER
  • FILE_S3_BUCKET
  • FILE_S3_REGION
  • FILE_S3_ENDPOINT
  • FILE_S3_FORCE_PATH_STYLE
  • FILE_S3_ALLOW_INSECURE_ENDPOINT
  • FILE_S3_AUTO_CREATE_BUCKET
  • FILE_OBJECT_KEY_PREFIX
  • FILE_S3_PRESIGN_PUT_TTL
  • FILE_S3_PRESIGN_GET_TTL
• WOPI / Collabora
  • WOPI_TOKEN_SECRET
  • WOPI_PROOF_SECRET
  • WOPI_DISCOVERY_URL
  • WOPI_PUBLIC_BASE_URL
  • WOPI_EDIT_ACTION_URL
  • WOPI_TOKEN_TTL
  • WOPI_MAX_BODY_BYTES
  • WOPI_REQUIRE_PROOF
• Worker mode selector
  • WORKER_JOB=email
  • WORKER_JOB=file_processing

WOPI / Collabora Hardening

• Keep WOPI endpoints behind the dedicated /wopi router and do not expose them through the session-authenticated app middleware chain.
• Use HTTPS end to end in production.
• Restrict network access to the Collabora host or private network allowlist.
• Set Collabora aliasgroup/host configuration so it can only call back to this backend.
• Keep WOPI_TOKEN_SECRET and WOPI_PROOF_SECRET in env or a secrets manager and rotate them periodically.
• Scope IAM to the minimum S3 bucket/prefix needed by file storage.
• Keep direct S3 URLs disabled for editor access; the backend should proxy file bytes.
• Ensure backend logs capture open/save/exit-save events for auditability.
• Verify /hosting/discovery reachability and S3 connectivity in health checks before enabling Collabora traffic.

Per-dependency overrides are supported with the same shape as the default block:

• CB_MICROSOFT_SSO_*
• CB_SES_*
• CB_SQS_*

Example: CB_SES_TIMEOUT=45s or CB_SQS_CONSECUTIVE_FAILURES=3.

Circuit Breakers

• Breakers are scoped per outbound dependency, not per HTTP endpoint and not global middleware.
• MySQL repositories and internal business logic are intentionally not wrapped.
• Only dependency-health failures count toward opening the breaker.
  • Microsoft SSO: network errors, timeouts, HTTP 429, and 5xx
  • SES: network errors, throttling, and AWS server faults
  • SQS: network errors, throttling, and AWS server faults
• Client/config/input errors are excluded so they do not open the breaker.
• State transitions are logged through the standard application logger.

Queue Behavior

• Long polling is enabled through SQS_WAIT_TIME_SECONDS.
• Messages are deleted only after successful processing or explicit ack of duplicates.
• Transient failures use ChangeMessageVisibility with backoff.
• Permanent failures are left unacked so SQS redrive policy can move them to the DLQ.
• The default producer schema is v2; legacy payload fallback remains available for older messages.
• Configure DLQ on the queue itself in AWS/LocalStack. The application no longer publishes directly to a DLQ URL.

Prometheus Metrics

Scrape endpoints:

• API: http://<api-host>:8080/metrics
• Email worker monitor: http://<worker-host>:9090/metrics
• File worker monitor: http://<worker-host>:9091/metrics

Example Prometheus scrape config:

scrape_configs:
  - job_name: "wucher-api"
    static_configs:
      - targets: ["127.0.0.1:8080"]
  - job_name: "wucher-email-worker"
    static_configs:
      - targets: ["127.0.0.1:9090"]
  - job_name: "wucher-file-worker"
    static_configs:
      - targets: ["127.0.0.1:9091"]

Metrics and labels:

• app_aws_requests_total{component,service,operation,result}
• app_aws_request_duration_seconds{component,service,operation}
• app_aws_errors_total{component,service,operation,error_class} where error_class is 4xx, 5xx, or other
• app_aws_transferred_bytes_total{component,service,operation,direction}
• app_queue_messages_received_total{component,queue_name}
• app_queue_messages_deleted_total{component,queue_name}
• app_queue_visibility_changes_total{component,queue_name,result}
• app_queue_send_total{component,queue_name,result}
• app_queue_retries_total{component,queue_name,result}
• app_queue_message_bytes_total{component,queue_name,direction}
• app_queue_depth{component,queue_name}
• app_worker_jobs_processed_total{component,job_type,result}
• app_worker_job_duration_seconds{component,job_type}
• app_email_send_total{component,provider,result}
• app_email_send_duration_seconds{component,provider}
• app_file_upload_total{component,result}
• app_file_delete_total{component,result}
• app_file_head_total{component,result}
• app_file_presign_download_total{component,result}

PromQL examples:

• Error rate per AWS operation:

sum by (component, service, operation) (rate(app_aws_requests_total{result="error"}[5m]))
/
clamp_min(sum by (component, service, operation) (rate(app_aws_requests_total[5m])), 1e-9)

• P95 latency per AWS operation:

histogram_quantile(
  0.95,
  sum by (le, component, service, operation) (
    rate(app_aws_request_duration_seconds_bucket[5m])
  )
)

• Queue depth:

max by (component, queue_name) (app_queue_depth)

• Worker success/failure count:

sum by (component, job_type, result) (increase(app_worker_jobs_processed_total[15m]))

• SES send failure rate:

sum by (component) (rate(app_email_send_total{provider="ses", result="error"}[5m]))
/
clamp_min(sum by (component) (rate(app_email_send_total{provider="ses"}[5m])), 1e-9)

• S3 4xx/5xx error rate:

sum by (operation, error_class) (rate(app_aws_errors_total{service="s3"}[5m]))

• S3 GetObject transferred bytes (estimated from presign download flow):

sum by (component) (rate(app_aws_transferred_bytes_total{service="s3",operation="getobject",direction="download"}[5m]))

• SQS message bytes in/out:

sum by (component, queue_name, direction) (rate(app_queue_message_bytes_total[5m]))

• SQS retry/requeue count:

sum by (component, queue_name, result) (increase(app_queue_retries_total[15m]))

File Manager

• Queue separation:
  • Email queue: app-queue (+ app-queue-dlq)
  • File queue: app-file-queue (+ app-file-queue-dlq)
• FILE_SQS_QUEUE_URL must be different from SQS_QUEUE_URL.
• API startup validates queue separation and refuses startup on misconfiguration.
• Upload writes file metadata + file outbox durably; file queue publish is handled asynchronously by the file outbox dispatcher.

Local run checklist:

# 1) Start localstack (queue + bucket bootstrap script will run)
cd ../localstack && docker compose up -d

WORKER_JOB=file_processing go run ./cmd/worker

SES Delivery Notes

• EMAIL_PROVIDER=mock is the recommended local/dev default when you do not have real AWS SES access.
• Mock delivery writes one JSON file per delivered email into EMAIL_MOCK_DIR.
• SES_FROM must be a verified identity or domain in Amazon SES.
• SES_ENDPOINT should be blank in production and only overridden for local/dev stacks such as LocalStack.
• Do not point SES events to the main send queue. Use SES_CONFIGURATION_SET with SNS -> separate SQS event queue.
• The codebase includes an SES event parser for SNS-wrapped notifications, but an event consumer is not started by default.

Migration Notes

Detailed migration notes are in docs/queue-sqs-migration.md. Data migration tracking wiki is in docs/data-migration-wiki.md.

Main breaking changes:

• Redis runtime support has been removed from the codebase.
• Redis env vars are obsolete and should be removed from deployment config.
• SMTP runtime env/config has been removed in favor of SES.
• Worker now relies on MySQL for queue idempotency and auth/runtime state persistence.
• SQS must have a redrive policy and DLQ configured outside the application.

CI / Build Selection

The pipeline always builds and pushes the api image on every push to development or staging.

To also build and push the worker or file-worker image, include the corresponding tag anywhere in the commit message:

Tag	Image built	Docker target
#PUSH-WORKER	mybitlab/wfm:<tag>-worker	worker
#PUSH-FILE-WORKER	mybitlab/wfm:<tag>-file-worker	file-worker

Both tags can appear in the same commit message to build all three in one pipeline run:

ci: deploy all services #PUSH-WORKER #PUSH-FILE-WORKER

Omitting a tag skips that image entirely — it will not be built or pushed.

Testing

Run the full test suite:

go test ./...

Generate coverage:

go test ./... -coverprofile=coverage.out
go tool cover -func=coverage.out
go tool cover -html=coverage.out -o coverage.html