# Test Scenario Schema & Runner — CLAUDE.md

Parent: [../CLAUDE.md](../CLAUDE.md)

## Overview

Automated PLC I/O validation through JSON-defined test scenarios.
The test runner is Python 3.10+ and communicates with Ignition exclusively
through the WebDev API. It never touches tags, files, or Modbus directly.

## Scenario Schema

Every scenario JSON file has exactly five top-level sections.

```json
{
  "identity": {
    "id": "PMP-101-START-001",
    "version": "1.0.0",
    "description": "Verify P-101 starts when start command is issued and permissives are met",
    "author": "engineer@company.com",
    "created": "2026-03-14",
    "tags": ["pump", "start", "P-101", "plantpax"]
  },
  "device": {
    "type": "Pump",
    "udt_path": "[default]Devices/Pumps/P-101",
    "plc_connection": "ControlLogix",
    "related_devices": [
      {
        "type": "Valve",
        "udt_path": "[default]Devices/Valves/XV-1001",
        "role": "discharge_valve"
      }
    ]
  },
  "initial_state": {
    "tags": [
      {
        "path": "[default]Devices/Pumps/P-101/Status/Running",
        "value": false
      },
      {
        "path": "[default]Devices/Pumps/P-101/Status/Faulted",
        "value": false
      },
      {
        "path": "[default]Devices/Pumps/P-101/Status/Ready",
        "value": true
      },
      {
        "path": "[default]Devices/Valves/XV-1001/Status/FullOpen",
        "value": true
      }
    ]
  },
  "sequence": [
    {
      "step": 1,
      "action": "write",
      "description": "Issue start command",
      "path": "[default]Devices/Pumps/P-101/Command/Start",
      "value": true,
      "delay_ms": 0
    },
    {
      "step": 2,
      "action": "wait",
      "description": "Allow PLC scan time for logic execution",
      "delay_ms": 2000
    },
    {
      "step": 3,
      "action": "read",
      "description": "Check running status",
      "path": "[default]Devices/Pumps/P-101/Status/Running",
      "delay_ms": 0
    }
  ],
  "expected_outcomes": [
    {
      "path": "[default]Devices/Pumps/P-101/Status/Running",
      "expected": true,
      "tolerance": null,
      "comparison": "equals",
      "description": "Pump should be running after start command"
    },
    {
      "path": "[default]Devices/Pumps/P-101/Status/Faulted",
      "expected": false,
      "tolerance": null,
      "comparison": "equals",
      "description": "Pump should not be faulted"
    },
    {
      "path": "[default]Devices/Pumps/P-101/Status/Speed",
      "expected": 60.0,
      "tolerance": 2.0,
      "comparison": "within_tolerance",
      "description": "Pump speed should be near setpoint"
    }
  ]
}
```

### Section Details

#### identity

Required fields: `id`, `version`, `description`
Optional fields: `author`, `created`, `tags`

ID convention: `{DEVICE_PREFIX}-{NUMBER}-{ACTION}-{SEQUENCE}`
Example: `PMP-101-START-001`, `VLV-1001-OPEN-002`

Version follows semver. Bump on any change to sequence or expected outcomes.

#### device

Required fields: `type`, `udt_path`
Optional fields: `plc_connection`, `related_devices`

`udt_path` is the full Ignition tag path to the device UDT instance.
`related_devices` lists other devices involved in the test with their role.

#### initial_state

Array of `{path, value}` objects. These tags are written (via WebDev API)
before the test sequence begins. The runner writes all initial_state tags,
waits 500ms for propagation, then verifies they were set correctly.

#### sequence

Ordered array of steps. Each step has:
- `step` (integer) — execution order
- `action` — one of: `write`, `read`, `wait`, `assert`
- `description` — human-readable step description
- `path` — tag path (required for `write`, `read`, `assert`)
- `value` — value to write (required for `write`)
- `delay_ms` — milliseconds to wait before executing this step

Actions:
- `write` — write `value` to `path` via WebDev tagWrite
- `read` — read `path` via WebDev tagRead, store result for later assertion
- `wait` — pause execution for `delay_ms` (no tag interaction)
- `assert` — immediately check `path` against `value` (mid-sequence validation)

#### expected_outcomes

Array of final assertions evaluated after the sequence completes.
Each outcome has:
- `path` — tag to check
- `expected` — expected value
- `tolerance` — numeric tolerance (null for exact match)
- `comparison` — one of: `equals`, `within_tolerance`, `greater_than`, `less_than`, `not_equals`
- `description` — what this assertion validates

## Scenario File Rules

1. **One scenario per file.** Named: `{identity.id}.json`
2. **Validate JSON before writing.** Use sorted keys, 2-space indent.
3. **Store in `testing/scenarios/` directory.**
4. **Back up before overwriting:** `cp file.json file.json.bak`

## Test Runner

### Architecture

```
runner/
├── main.py              ← CLI entry point (Python 3.10+)
├── executor.py          ← scenario execution engine
├── api_client.py        ← WebDev API client wrapper
├── validator.py         ← scenario JSON schema validation
├── reporter.py          ← results formatting and output
└── config.py            ← runner configuration (gateway URL, timeouts)
```

### Execution Flow

1. Load and validate scenario JSON against schema.
2. Connect to Ignition WebDev API (health check first).
3. Write `initial_state` tags → wait 500ms → verify writes.
4. Execute `sequence` steps in order, respecting `delay_ms`.
5. After sequence completes, evaluate `expected_outcomes`.
6. Generate results report (JSON + human-readable summary).

### API Client

The runner talks to Ignition through WebDev endpoints only:

```python
# env: python3.10+ (external tooling — NOT Ignition)

class IgnitionAPIClient:
    def __init__(self, base_url: str = "http://localhost:8088/system/webdev"):
        self.base_url = base_url

    def tag_write(self, path: str, value) -> dict:
        """POST /tagWrite {"path": path, "value": value}"""

    def tag_read(self, path: str) -> dict:
        """GET /tagRead?path=<path>"""

    def tag_browse(self, root: str = "") -> dict:
        """GET /tagBrowse?root=<root>"""

    def health(self) -> dict:
        """GET /health"""
```

## Results Format

```json
{
  "scenario_id": "PMP-101-START-001",
  "scenario_version": "1.0.0",
  "timestamp": "2026-03-14T10:30:00Z",
  "duration_ms": 4523,
  "result": "PASS",
  "initial_state_verification": {
    "status": "OK",
    "details": []
  },
  "sequence_log": [
    {
      "step": 1,
      "action": "write",
      "path": "[default]Devices/Pumps/P-101/Command/Start",
      "value": true,
      "response": {"status": "ok"},
      "elapsed_ms": 45
    }
  ],
  "outcomes": [
    {
      "path": "[default]Devices/Pumps/P-101/Status/Running",
      "expected": true,
      "actual": true,
      "result": "PASS",
      "description": "Pump should be running after start command"
    }
  ],
  "summary": {
    "total_outcomes": 3,
    "passed": 3,
    "failed": 0
  }
}
```

Results are stored in `testing/results/` as `{scenario_id}_{timestamp}.json`.

## File Structure

```
testing/
├── CLAUDE.md              ← you are here
├── scenarios/
│   ├── PMP-101-START-001.json
│   ├── PMP-101-STOP-001.json
│   └── VLV-1001-OPEN-001.json
├── runner/
│   ├── main.py
│   ├── executor.py
│   ├── api_client.py
│   ├── validator.py
│   ├── reporter.py
│   └── config.py
└── results/
    └── (timestamped result files)
```