104. Running the Evaluation Pipeline

The Evaluation Pipeline is driven by a single command-line tool, ftx_evaluate, that ships with the Foretify installer. It processes recorded driving data and simulation output through an ordered sequence of stages — converting raw logs into the Foretellix object list format, denoising the data, ingesting it into Foretify, matching evaluation scenarios, and uploading the results to Foretify Manager.

This section assumes the pipeline is already set up for you — a flow configuration file and a set of evaluation scenarios are in place — and shows how to run it.

• To learn how to write a flow configuration, see Configuring the flow.
• For the conceptual overview, see Introducing the Evaluation Pipeline.

104.1 Evaluation Pipeline prerequisites

Ensure the following are in place before you run the pipeline.

• Foretify — Must be installed and on your PATH. Run foretify --version to verify.
• FTX environment variable — Must point to the Foretify installation root. Flow configurations use $FTX/... paths extensively.
• ftx_evaluate binary — Ships with the Foretify installer. You do not need to build it from source. The setup script exposes it as $FTX_EVALUATE_EXE — see below.
• Flow configuration file — A YAML file that declares which stages to run and how each is configured. See Configuring the evaluation pipeline flow.
• Evaluation scenarios — A set of evaluation scenarios must be defined. See Introducing the Evaluation Pipeline.
• Cloud storage credentials (optional) — Required only when an input path or the output path uses an s3:// URL. Configure them through the standard mechanisms (credentials file, environment variables, or an IAM role).

The installer sets $FTX and sources the Foretify setup script — see Foretify installation. Source the setup script once per shell. It puts foretify on your PATH, points the pipeline at its plugin libraries, and exposes the pipeline binary as $FTX_EVALUATE_EXE:

source $FTX/bin/ftx_setup.sh

With the script sourced, both foretify --version and $FTX_EVALUATE_EXE resolve. The examples on this page invoke the pipeline through $FTX_EVALUATE_EXE.

To confirm an installed release works end to end, run the smoke tests.

104.2 Input requirements

The pipeline requires two inputs to run:

• An input list — specifies which data packages to process. You can point at a single recording or a batch. See Evaluation Pipeline input list.
• A flow configuration — a YAML file that declares which stages to run and how each is configured. See Configuring the Evaluation Pipeline flow.

The run-single subcommand is the exception — it takes a single recording directly with --drive_data and needs no input list.

Foretify ships ready-made flow configurations you can copy as a starting point — for example $FTX/evaluate/library/config/flow_configs/ftlx_pandaset_flow_config.yaml for Pandaset-format recordings. Supply your own recordings and, for a batch, an input list.

104.3 ftx_evaluate command overview

ftx_evaluate exposes the following subcommands.

104.4 Validate the pipeline

Run validate first to catch configuration and accessibility problems before running to a full batch. It performs a series of categorized pre-flight checks without executing any pipeline stage.

There are three mutually exclusive ways to supply drive data:

# 1) Full input-list mode — every row is checked individually
$FTX_EVALUATE_EXE validate \
    --flow /path/to/flow_config.yaml \
    --input_list /path/to/input_list.json

# 2) Folder mode — only the directory's accessibility is checked
$FTX_EVALUATE_EXE validate \
    --flow /path/to/flow_config.yaml \
    --input_directory /data/recordings

# 3) Config-only mode — the drive-data check is skipped entirely
$FTX_EVALUATE_EXE validate \
    --flow /path/to/flow_config.yaml

--input_list and --input_directory are mutually exclusive.

Example output when every check passes:

Evaluation Pipeline — Validate
──────────────────────────────────────────────────────────────
Input list:      input_list.json
Flow:            flow_config.yaml
Stages:          explore → parse → denoise → map → ingest → match → post_match

Checks:
  ✓ License checkout
  ✓ Flow config
  ✓ User top file  (user_config_top.osc)
  ✓ Denoiser config  (denoiser_config.yaml)
  ✓ Other flow config files
  ✓ YAML !include targets
  ✓ Drive data  (2/2 input(s) accessible)
  ✓ User top imports
  ✓ Compile

Inputs: 2 data package(s)
  [1] recording_001.parquet   ✓  hash: k7m2p9x4qn
  [2] recording_002.parquet   ✓  hash: r3n8w1v5jt

──────────────────────────────────────────────────────────────
All checks passed. OK (exit code 0)

On success, validate returns exit code 0. On failure it returns a value between 1 and 255 that encodes which categories failed — each bit corresponds to one category, so multiple failures are reported in a single run.

For example, exit code 10 means bits 1 and 3 are set — drive data missing and user top file missing. The human-readable report always lists the failing categories by name regardless of the exit code.

104.5 Run a single recording

Use run-single to process one recording. This is the quickest way to test a flow configuration.

$FTX_EVALUATE_EXE run-single \
    --drive_data /data/recordings/recording_001.parquet \
    --output_path /tmp/eval_output \
    --flow /path/to/flow_config.yaml

You can supply optional inputs directly on the command line — --map_data, --matches_data, and --video_data — when they are not discoverable from the recording directory.

104.6 Run a batch

Use run to process every input in an input list. Set --max_workers to process several recordings in parallel.

$FTX_EVALUATE_EXE run \
    --input_list /path/to/input_list.json \
    --output_path /tmp/eval_batch \
    --flow /path/to/flow_config.yaml \
    --max_workers 4

The pipeline writes each input to its own deterministic hash directory under the output path, so the same input always maps to the same directory.

104.7 Check status

Inspect the outcome of a previous run at any time. status works with both local and s3:// output paths.

$FTX_EVALUATE_EXE status --output_path /tmp/eval_batch

It displays a summary of succeeded and failed inputs, along with per-stage pass rates.

104.8 Resume a failed run

If a batch fails partway through, resume it from a chosen stage instead of re-running everything. Stages that already succeeded before the resume point are not re-run.

$FTX_EVALUATE_EXE resume \
    --output_path /tmp/eval_batch \
    --from_stage parse \
    --flow /path/to/flow_config.yaml

To resume a single hash directory:

$FTX_EVALUATE_EXE resume-single \
    --output_path /tmp/eval_batch/k7m2p9x4qn \
    --from_stage parse

The validate report and status summary show the hash directory name next to each input.

If the run's checkpoint includes the flow configuration, --flow is optional on resume — the pipeline reuses the checkpoint configuration unless you override it.

104.9 Denoise an object list

The denoiser is available as a standalone subcommand. Use it to denoise an object list .pb file outside a full pipeline run — for example, to tune a denoiser configuration before adding it to a flow.

$FTX_EVALUATE_EXE denoise \
    --object_list /data/object_lists/recording_001.pb \
    --output_path /tmp/denoised \
    --config /path/to/denoiser_config.yaml

--output_path accepts either a directory (the denoised file is auto-named) or an exact .pb file path. When you omit --config, the pipeline uses a default denoiser configuration that ships with the installer. For the concepts behind denoising and the full configuration reference, see Denoiser and Denoiser configuration file.

104.10 Upload results to Foretify Manager

After a run completes, upload the results so you can explore coverage and KPIs in Foretify Manager. The pipeline does not upload automatically — run upload as a separate step. A completed batch prints the exact upload command to run.

$FTX_EVALUATE_EXE upload \
    --output_path /tmp/eval_batch \
    --host fmanager.example.com \
    --port 443 \
    --https \
    --project_id my_project \
    --user my_username \
    --tsr_name "Fleet Evaluation 2026-04"

• --tsr_name is the name of the Test Suite Result (Test Run Group) created in Foretify Manager.
• --project_id is the ID of the Foretify Manager project the results are uploaded to — see creating a project.
• Authentication uses credentials from ~/.ftxpass when available; otherwise you are prompted interactively. See managing credentials.

104.11 Writing results to cloud storage

Set --output_path to an s3:// URL to write results directly to cloud storage. Use --temp_work_dir to control where the local working directory is created during the run; without it, the system temporary directory is used.

$FTX_EVALUATE_EXE run \
    --input_list /path/to/input_list.json \
    --output_path s3://my-bucket/evaluations/batch_001 \
    --flow /path/to/flow_config.yaml \
    --temp_work_dir /tmp/eval_work

104.12 Understanding the output

104.12.1 Console summary

After each run, the pipeline prints a summary to the terminal:

Evaluation Pipeline
──────────────────────────────────────────────────────────
Output: /tmp/eval_output
Stages: explore → parse → denoise → map → ingest → match → post_match

  ✓ explore          0.1s
  ✓ parse            3.2s
  ✓ denoise          1.5s
  ✓ map              0.0s
  ✓ ingest          12.0s
  ✓ match            8.3s
  ✓ post_match       2.1s

──────────────────────────────────────────────────────────
Result: succeeded  (28.1s)

104.12.2 Output directory layout

Each input gets a deterministic hash directory under the output path:

<output_path>/
  <hash_id>/                # one directory per input
    context.json            # checkpoint with run state (read by resume)
    explore/                # each stage's output and log
    parse/
    denoise/
    map/
    ingest/
    match/
    post_match/
  final/
    manifest.json           # one entry per input, used by upload (status scans the per-input context.json files)

The pipeline derives the <hash_id> from the input's drive_data path, so the same input always maps to the same directory. Each hash directory also holds a pipeline.log and resolved configs/ and inputs/ directories.

run-single writes that same per-input set — context.json, pipeline.log, configs/, inputs/, and the stage directories — directly under --output_path, with no hash subdirectory and no final/manifest.json.

104.12.3 Log formats

• --log_format human (default) — human-readable log lines with timestamps.
• --log_format json — structured JSON log lines, one object per line, suitable for ingestion into a log-aggregation system. Per-stage log files remain human-readable in both modes.

104.12.4 Intermediate results

By default, the pipeline removes non-log intermediate artifacts after a run, keeping only the last stage's output. Use --keep_intermediate_results <mode> to control what is preserved:

104.12.5 Debugging tips

• Add --debug for verbose log output.
• Add --keep_intermediate_results raw to preserve every stage directory so you can inspect each stage's output.
• Check a stage's log file at <output_path>/<hash_id>/<stage>/<stage>.log.
• Run validate before a full batch to catch configuration and accessibility issues early.