Datasources

A datasource is an ingestion configuration: name + source type + source settings + sink type + sink settings. Create one before you can select it as a pipeline input stage.

Open Datasources → New. Configure the Source tab and Sink tab, then Save. Credentials tagged for encryption are stored AES-GCM encrypted and decrypted again when you reopen the editor.

Source types the engine runs

Type	Behavior
S3	List objects in a bucket/prefix on a poll interval. Access key + secret, optional custom endpoint.
Local	Folder path, glob, optional recursive. Polls about every 15s for new/changed files.
Kafka	Consume a topic (group, earliest/latest, SASL PLAIN/SCRAM, TLS + optional PEMs).
HTTP	Polling GET or POST. Emits when the response body fingerprint changes. Auth: none, bearer, API key query, or API key header. Rate limit and 10 MiB body cap. Private/loopback URLs are rejected unless the operator enabled FALCONTALON_ALLOW_PRIVATE_URLS.
TCP	Dial or listen. Dial connects to host:port (optional TLS). Listen accepts connections for the run. Framing newline emits one item per line (NMEA); raw emits one idle burst. 10 MiB cap.
UDP	Dial or listen. Dial writes one empty datagram, then emits each reply from that peer. Listen emits every datagram received on the run's port. Default max datagram 2048 bytes.

HTTP is not a webhook. The in-app help may say “push”; the engine is a poller.

Dial mode is an outbound client. RFC1918 and loopback socket hosts are allowed; cloud metadata, link-local, and multicast are rejected. This is independent of FALCONTALON_ALLOW_PRIVATE_URLS.

Listen mode ignores host. The worker binds an ephemeral port for listen_timeout seconds (0 stays open until the execution is cancelled) and, when listener_services is on, publishes ClusterIP Service ft-exec-<execution id> on the configured port. Senders use ft-exec-<id>.<namespace>.svc.cluster.local:<port> while the execution is running. The execution sheet shows that address for a scheduled, claimed, or running listen run. Several listen runs can be active at once; each has its own Service. The run holds a worker slot until the window ends. Records are written to the sink continuously (every few seconds) while the run is listening, and anything pending is flushed when the run is cancelled.

Sink types the engine runs

Type	Behavior
S3	Write objects under a prefix; optional gzip.
Kafka	Produce JSON records; optional partition key field (for example mmsi); batch size.

Sources coming soon

• SFTP
• GCS
• CAN

TCP / UDP fields

• Mode — client (dial) or listen
• Host / port — dial destination, or the ClusterIP port in listen mode (default port 10110)
• Read timeout — dial: seconds without data before the source closes (default 2). Listen: per-connection idle, which does not end the run
• Listen timeout — listen mode only. Seconds the port stays open. 0 waits until cancel
• Framing (TCP) — newline or raw
• TLS (TCP dial) — optional CA / client cert PEMs
• Max datagram (UDP) — bytes per datagram (default 2048)

Dial-mode UDP only receives from the dialed peer. It will not see multicast or broadcasts. Listen-mode UDP accepts datagrams from any sender.

Kafka fields

• Bootstrap brokers — the host:port list for the Kafka cluster this datasource should use.
• Topic / group / start offset
• SASL — PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
• TLS — optional CA / client cert PEMs
• Partition key (sink) — record field used as the Kafka key
• Batch size (sink)

How a run uses the datasource

Each execution drains newly discovered items until the source is idle for 5 seconds, then writes successful records to the sink. It is not a long-lived stream. Kafka consumer-group offsets persist across runs.