108. Configuring the Evaluation Pipeline flow

A flow configuration file declares which pipeline stages to run and how each is configured. It is a YAML file that you pass to the run, run-single, and validate subcommands with --flow. Stages execute in the order listed.

A minimal flow configuration looks like this:

stages: [explore, parse, denoise, map, ingest, match, post_match]

user_top_file: $FTX/path/to/user_config_top.osc

explore:
  plugin: common.pandaset_explorer

parse:
  plugin: common.pandaset_parser
  max_clip_length_s: 30          # common parser option
  round_times: true              # common parser option

denoise:
  config: /path/to/denoiser_config.yaml

map:
  plugin: common.standard_map

ingest:
  timeout_s: 600

match:
  timeout_s: 3600

post_match:
  timeout_s: 1800

Each stage you list in stages may need a configuration block of the same name. Stages that load a plugin reference it with plugin: <library>.<plugin_name>. For information on how plugin references resolve and which libraries ship with the installer, see Using plugins.

108.1 Top-level fields

108.2 Available stages

You choose which stages to run by listing them in stages. Only explore is mandatory — every other stage is optional, and you include just the ones your evaluation needs. The pipeline runs the stages in the order you list them, and a few ordering rules apply (noted below). A typical end-to-end evaluation runs all of explore, parse, denoise, map, ingest, match, and post_match.

The following table lists each stage in its typical order, what it does, and whether it is mandatory.

• The ingest, match, and post_match stages run Foretify and therefore require a user_top_file. The map stage, when listed, must appear before ingest.
• Stages that load a plugin or a config file — explore, parse, denoise, map, ext_match, and post_process — need their configuration block when listed; the ingest, match, and post_match blocks are optional and may be omitted to run with defaults.

108.3 User top file

The user_top_file is an OSC2 file that imports and instantiates the evaluation/match scenarios. It is required whenever the flow includes the ingest, match, or post_match stages.

The top file can be hosted on cloud storage (s3://bucket/prefix/top.osc). When hosted on cloud storage, any .osc files it imports can also reside on cloud storage, provided they are located under the same prefix (folder) as the top file. The pipeline downloads all .osc files from that prefix so that relative imports resolve correctly.

108.4 Stage configuration blocks

108.4.1 Plugin-based stages

The explore, parse, ext_match, and post_process stages load a plugin. They reference a library fragment with plugin: <library>.<plugin_name> and can optionally override plugin-specific options in a nested config: block, plus the stage's common options — shared by every plugin of that stage — at the top level:

parse:
  plugin: common.pandaset_parser   # the plugin to load
  max_clip_length_s: 30            # common stage option — applies to every parser
  round_times: true                # common stage option — applies to every parser
  config:                          # optional plugin-specific overrides
    conversion_to_ms: 1000

Values you set under config: override the plugin's own defaults; any key you omit keeps its default value.

108.4.2 Subprocess-based stages

The ingest, match, and post_match stages run Foretify as a subprocess and accept timeout and retry settings:

ingest:
  timeout_s: 600
  max_retries: 1

108.4.3 Accepting benign ingest findings

By default, the pipeline stops after ingest when the Foretify run ends in a non-COMPLETED result status — including a rules-of-the-road SUT_ERROR, which is a checker finding rather than a tool failure. To run match and post_match on such logs, list the acceptable findings under ingest.accepted_issues, keyed by issue category and then by kind.

The reserved keyword all is a catch-all: all as a kind accepts every kind in that category, and all as a category (all: [all]) accepts every issue regardless of category or kind. When the run's reported error issues are all covered, the pipeline continues; any unlisted error issue still stops the pipeline, and genuine tool failures (timeout, non-zero exit) always fail the run.

ingest:
  accepted_issues:
    sut:                                            # accept specific kinds...
      - r_0002b_following_box_truck_caravan_violation
      - all                                         # ...or every kind in this category
    # all: [all]                                    # or accept every issue, any category/kind

A SUT_ERROR from the drive data is a property of the recording, so match and post_match replay the same data and re-detect it. To let those stages pass as well, repeat the allow-list under match.accepted_issues and post_match.accepted_issues (same format as above).

108.4.4 Denoise stage configuration block

The denoise stage takes a direct path to the denoiser configuration file. It does not use a plugin, so you specify the config file path directly:

denoise:
  config: $FTX/path/to/denoiser_config.yaml

The same denoiser is also available as a standalone subcommand — see Denoise an object list.

For information on denoiser functionality and the full set of configuration options, see Denoiser and Denoiser configuration file.

108.4.5 Map stage configuration block

The map stage is a dedicated step that must run before ingest. It ensures a usable map (in the configured format, default xodr) is available to all downstream Foretify stages. Its provider plugin follows a fixed priority list:

1. Existing map — if a map of the requested format is already available (from the input list or explorer discovery), use it as-is.
2. Map generation — if map.enable_mapgen: true and no existing map was found, run the configured map-generation plugin.
3. Third-party download — if map generation is disabled or does not pass its quality gate, and map.enable_third_party: true, the configured third-party provider downloads a map from its vendor. No third-party provider is included by default — supply your own plugin to enable downloads.
4. Stop with an error — if none of the above produces a map, the pipeline halts with a stage failure.

A minimal map block, which relies on an existing map and fails if none is found:

map:
  plugin: common.standard_map
  format: xodr                    # default; the format the provider looks for
  enable_mapgen: false
  enable_third_party: false

108.4.6 ext_match stage configuration block

The ext_match stage enriches a run with match intervals from an external source — the matches_data input — instead of, or in addition to, the match stage. Like other plugin-based stages, it references a plugin, and you list it in stages wherever the external intervals should be applied:

stages: [explore, parse, denoise, map, ingest, ext_match, post_match]

ext_match:
  plugin: my.ext_matcher          # your external-matcher plugin

No external matcher ships in the common library — supply one as a custom plugin.

108.5 Environment variable expansion

All string values in the flow YAML are expanded when the file is loaded. Use $FTX, $HOME, or any other environment variable in paths to keep the configuration portable.

108.6 Runtime overrides

Use --flow_override to override specific fields without editing the YAML file. The override is a JSON object that is deep-merged into the flow configuration:

$FTX_EVALUATE_EXE run-single \
    --drive_data /data/recordings/recording_001.parquet \
    --output_path /tmp/eval_output \
    --flow /path/to/flow_config.yaml \
    --flow_override '{"parse": {"config": {"utm_crs": "EPSG:32632"}}}'

This is useful for changing one setting per run — a timeout, a parser option, or the user top file — while keeping a single shared flow file under version control.