Pipelines

A pipeline is: input stage (existing datasource) + reader + ordered steps.

Rules

1. At least one step
2. First step must be a classifier
3. A classifier cannot follow another classifier (put a transformer in between)
4. Classifier → schema id; transformer → type + nested config

The UI enforces the same connectivity rules as the API.

Readers

Type	UI	Behavior
nmea	Sentence checkboxes	NMEA 0183 lines. AIS decode needs VDM (usually also VDO). Also GGA, RMC, GLL, VTG, ZDA, HDT (all selected by default). GGA accepts fix qualities 0–8 (including manual 7 and simulation 8 from AIS receivers). Multi-part VDM is reassembled.
json	JSON Lines or JSON array	One JSON object per line, or a JSON array of objects.
binary	Framing, prefix width or frame size, encoding	Raw bytes split into frames. No decoding: each frame becomes one record with a single hex or base64 field, for a transformer to parse. See Binary reader.

Binary reader

Use binary for binary protocols (for example a captured UDP message stream). The reader only frames the stream; parsing the payload is the job of a transformer step.

Setting	Values	Notes
Framing	length_prefix (default), fixed	length_prefix: each frame is preceded by a big-endian length. fixed: every frame is exactly the configured size.
Length prefix width	1, 2 or 4 bytes (default 4)	Used with length_prefix. The prefix is not included in the record.
Frame size	bytes	Required with fixed. The UI will not save a fixed-size reader without it.
Encoding	hex (default), base64	The record field is named after the encoding: {"hex": "1e0000..."} or {"base64": "..."}.
max_record_bytes	default 1 MiB	API only. Caps one frame so a corrupt length prefix cannot allocate a huge buffer.

Empty frames are skipped. A truncated frame, a length above max_record_bytes, or a trailing partial fixed-size frame stops the read with an error.

Example config (API):

{ "type": "binary", "binary": { "framing": "length_prefix", "length_bytes": 4, "encoding": "hex" } }

Pair it with a custom transformer that reads the hex field, for example struct.unpack on bytes.fromhex(record["hex"]).

The Playground cannot test binary readers because it sends text; run the pipeline against a datasource instead.

Built-in transformers

Type	What it does
passthrough	Copy fields
pick	Project output_name → source_key
ais	Decode NMEA VDM/VDO raw into AIS fields (mmsi, msg_type, position, AisLib fields). Only message types 1, 2, 3, 4, 5, 11, 14, 18, 19, 20, 21, 24 (part A and B), 27 are emitted; other types are dropped. AisLib fields are emitted at the top level with the field names and raw ITU-R M.1371 values of the dma-ais AisLib message classes (e.g. sog in 1/10 knot, pos.rawLatitude in 1/10000 minute; message 27 uses sog in knots and pos.rawLatitude in 1/10 minute; AisLib has no message 20 class, so its offsetN/numSlotsN/timeoutN/incrementN reservation blocks follow ITU-R M.1371); top-level lat/lon/speed/heading are normalized to degrees and knots. packetDate is the decode time as a string of epoch milliseconds (e.g. "1790941097800"), and pos carries AisLib's bitCorrection (0) and resolution (10000.0; 10.0 for message 27). Other UI NMEA types (GGA, RMC, GLL, VTG, ZDA, HDT) pass through unchanged.
custom	Saved Python transformer by id

Create and run

1. Pipelines → New → pick datasource, reader, steps → Save
2. List Run enqueues POST /api/v1/pipelines/{id}/executions (HTTP 202). The UI does not send priority, max_retries, or run_at (API defaults: retries 3, run immediately). There is no cron UI.

Each execution loads the definition, decrypts the datasource, expands custom steps, drains items (idle 5s), writes the sink, and records counters.

Use the Playground before the first real run.