Managing Baselines

A baseline is a snapshot of your project's performance at a known-good point in time. It captures aggregated stats (mean, median, p95, etc.) for each tracked metric from a Gauntlet capture run.

When you later compare a new capture against the baseline, PerfGuard calculates deltas and flags any stat that exceeds your configured threshold. Without a baseline, there's nothing to compare against — you're flying blind.

Choose a build that represents your target performance — typically a release milestone, or the last build before a known-problematic change. Run the Gauntlet capture, then record it as a baseline:

# Step 1: Run the capture
UnrealEditor YourProject.uproject -game \
    -gauntlet=PerfGuardGauntletController \
    -scenario=MainMenu_Flythrough \
    -csvprofile -RenderOffScreen -unattended -log

# Step 2: Record the baseline
python3 perfguard_cli.py baseline record MainMenu_Flythrough \
    --csv Saved/Profiling/CSV/Profile_2026.01.15-10.30.00.csv \
    --platform Linux \
    --warmup 60

:: Step 1: Run the capture
UnrealEditor-Cmd.exe YourProject.uproject -game ^
    -gauntlet=PerfGuardGauntletController ^
    -scenario=MainMenu_Flythrough ^
    -csvprofile -unattended -log

:: Step 2: Record the baseline
python perfguard_cli.py baseline record MainMenu_Flythrough ^
    --csv Saved\Profiling\CSV\Profile_2026.01.15-10.30.00.csv ^
    --platform Win64 ^
    --warmup 60

Baselines are stored as JSON files in the baselines/ directory. Here's what the structure looks like:

{
  "scenario": "MainMenu_Flythrough",
  "platform": "Win64",
  "revision": "a1b2c3d",
  "recorded_at": "2026-01-15T10:35:00Z",
  "warmup_frames": 60,
  "frame_count": 842,
  "stats": {
    "FrameTime": {
      "mean": 14.23,
      "median": 13.89,
      "p95": 18.42,
      "min": 11.02,
      "max": 22.67
    },
    "GPUTime": { ... }
  }
}

The JSON is human-readable and version-control friendly. Each stat stores multiple aggregations so you can compare against whichever is most meaningful for your use case.

Performance varies significantly between platforms. A frame that takes 14ms on Win64 might take 20ms on Linux with a different GPU driver. PerfGuard supports per-platform baselines so you can maintain separate reference points.

# Record Win64 baseline
python3 perfguard_cli.py baseline record MyScenario --csv win64_capture.csv --platform Win64

# Record Linux baseline
python3 perfguard_cli.py baseline record MyScenario --csv linux_capture.csv --platform Linux

:: Record Win64 baseline
python perfguard_cli.py baseline record MyScenario --csv win64_capture.csv --platform Win64

:: Record Linux baseline
python perfguard_cli.py baseline record MyScenario --csv linux_capture.csv --platform Linux

When comparing, PerfGuard automatically selects the baseline matching the current platform. You can also specify the platform explicitly.

Export all baselines to a directory for committing to your repository. This ensures the entire team shares the same reference data.

python3 perfguard_cli.py baseline export ./baselines-export/

# Then commit to version control
git add baselines-export/
git commit -m "Update performance baselines for v0.3"

python perfguard_cli.py baseline export .\baselines-export\

:: Then commit to version control
git add baselines-export\
git commit -m "Update performance baselines for v0.3"

Import baselines from a directory — whether pulled from version control, downloaded from a shared drive, or synced from a build farm.

python3 perfguard_cli.py baseline import ./baselines-export/

python perfguard_cli.py baseline import .\baselines-export\

This copies the JSON baseline files into PerfGuard's working baselines directory, ready for comparison.

See what baselines are currently available:

python3 perfguard_cli.py baseline list

# Output:
# MainMenu_Flythrough  Win64  2026-01-15  a1b2c3d  842 frames
# Arena_CombatView     Win64  2026-01-15  a1b2c3d  1204 frames
# OpenWorld_Drive      Linux  2026-01-14  e5f6g7h  2031 frames

python perfguard_cli.py baseline list

:: Output:
:: MainMenu_Flythrough  Win64  2026-01-15  a1b2c3d  842 frames
:: Arena_CombatView     Win64  2026-01-15  a1b2c3d  1204 frames
:: OpenWorld_Drive      Linux  2026-01-14  e5f6g7h  2031 frames

You can also open the JSON files directly to inspect individual stat values or verify the correct revision was captured.

Thresholds determine how much a stat can regress before PerfGuard flags it. There are two types:

• Percentage threshold — e.g., 5% means a stat can be up to 5% worse than baseline
• Absolute threshold — e.g., 2.0ms means the stat can be up to 2ms worse

python3 perfguard_cli.py baseline compare MyScenario \
    --csv new_capture.csv \
    --threshold-percent 5.0  # 5% regression tolerance

python perfguard_cli.py baseline compare MyScenario ^
    --csv new_capture.csv ^
    --threshold-percent 5.0  :: 5% regression tolerance

Budget presets let you set pass/fail based on target frame rates. If the mean frame time exceeds the budget, the comparison fails regardless of regression percentage.

• 60fps — 16.67ms frame budget
• 30fps — 33.33ms frame budget
• vr90fps — 11.11ms frame budget

python3 perfguard_cli.py baseline compare MyScenario \
    --csv new_capture.csv \
    --budget 60fps  # Fail if mean FrameTime > 16.67ms

python perfguard_cli.py baseline compare MyScenario ^
    --csv new_capture.csv ^
    --budget 60fps  :: Fail if mean FrameTime > 16.67ms

Baselines should be updated when the performance change is intentional and expected:

• After optimization work — Update to capture the new (better) performance level
• After feature additions — New visual features legitimately cost performance; re-baseline to the new normal
• After engine upgrades — UE version changes can shift baseline numbers
• After hardware changes — New CI machines need new baselines
• After content changes — Large asset additions that change scene complexity