Architecture

Synapse is a finite-run ETL engine for AIS / NMEA and related ISR feeds. You run four application workloads plus Postgres and Kafka.

Runtime processes

Process	Listens	Who authenticates	Notes
API (/server)	HTTP	OIDC ID token (console) or X-API-Key (workers, tooling)	WORKER_MODE=cluster
Worker (/worker)	none	outbound X-API-Key only	No HTTP probes
Console	HTTP	OIDC provider in the browser; API calls use the ID token
Docs	HTTP	none

Health:

• API: unauthenticated GET /api/health
• Console / docs: GET /healthz
• Workers have no HTTP listener and no probes

Data path of one execution

A pipeline run is not a long-lived Kafka consumer. POST /api/v1/pipelines/{id}/executions inserts a row (scheduled). A worker claims it, receives a JobBundle (pipeline + decrypted datasource + schemas — no DB on the worker), then:

1. Discover items from the source (S3, Kafka, HTTP poll, TCP/UDP dial or listen, or a filesystem path if one is mounted)
2. Read bytes with the pipeline reader (nmea or json)
3. Run classifier / transformer steps
4. Stop when the source is idle for 5 seconds (TCP/UDP listen streams to the sink until its listen window ends or the run is cancelled)
5. Write successful records to the sink (S3 or Kafka)
6. Finish the execution (completed / failed / cancelled)

warning

Partial sink writes from abandoned runs are not rolled back.

Worker mode

The API must run with FALCONTALON_WORKER_MODE=cluster. In that mode the API only queues work, extends leases, and reaps stale workers. Remote /worker processes poll HTTP to claim jobs.

Do not set the API to embedded while workers are also claiming the same database — jobs can execute twice.