framework-ignition-docker/testing/CLAUDE.md

Test Scenario Schema & Runner — CLAUDE.md

Parent: ../CLAUDE.md

Overview

Automated PLC I/O validation through JSON-defined test scenarios. The test runner is Python 3.10+ and communicates with Ignition exclusively through the WebDev API. It never touches tags, files, or Modbus directly.

Scenario Schema

Every scenario JSON file has exactly five top-level sections.

{
  "identity": {
    "id": "PMP-101-START-001",
    "version": "1.0.0",
    "description": "Verify P-101 starts when start command is issued and permissives are met",
    "author": "engineer@company.com",
    "created": "2026-03-14",
    "tags": ["pump", "start", "P-101", "plantpax"]
  },
  "device": {
    "type": "Pump",
    "udt_path": "[default]Devices/Pumps/P-101",
    "plc_connection": "ControlLogix",
    "related_devices": [
      {
        "type": "Valve",
        "udt_path": "[default]Devices/Valves/XV-1001",
        "role": "discharge_valve"
      }
    ]
  },
  "initial_state": {
    "tags": [
      {
        "path": "[default]Devices/Pumps/P-101/Status/Running",
        "value": false
      },
      {
        "path": "[default]Devices/Pumps/P-101/Status/Faulted",
        "value": false
      },
      {
        "path": "[default]Devices/Pumps/P-101/Status/Ready",
        "value": true
      },
      {
        "path": "[default]Devices/Valves/XV-1001/Status/FullOpen",
        "value": true
      }
    ]
  },
  "sequence": [
    {
      "step": 1,
      "action": "write",
      "description": "Issue start command",
      "path": "[default]Devices/Pumps/P-101/Command/Start",
      "value": true,
      "delay_ms": 0
    },
    {
      "step": 2,
      "action": "wait",
      "description": "Allow PLC scan time for logic execution",
      "delay_ms": 2000
    },
    {
      "step": 3,
      "action": "read",
      "description": "Check running status",
      "path": "[default]Devices/Pumps/P-101/Status/Running",
      "delay_ms": 0
    }
  ],
  "expected_outcomes": [
    {
      "path": "[default]Devices/Pumps/P-101/Status/Running",
      "expected": true,
      "tolerance": null,
      "comparison": "equals",
      "description": "Pump should be running after start command"
    },
    {
      "path": "[default]Devices/Pumps/P-101/Status/Faulted",
      "expected": false,
      "tolerance": null,
      "comparison": "equals",
      "description": "Pump should not be faulted"
    },
    {
      "path": "[default]Devices/Pumps/P-101/Status/Speed",
      "expected": 60.0,
      "tolerance": 2.0,
      "comparison": "within_tolerance",
      "description": "Pump speed should be near setpoint"
    }
  ]
}

Section Details

identity

Required fields: id, version, description Optional fields: author, created, tags

ID convention: {DEVICE_PREFIX}-{NUMBER}-{ACTION}-{SEQUENCE} Example: PMP-101-START-001, VLV-1001-OPEN-002

Version follows semver. Bump on any change to sequence or expected outcomes.

device

Required fields: type, udt_path Optional fields: plc_connection, related_devices

udt_path is the full Ignition tag path to the device UDT instance. related_devices lists other devices involved in the test with their role.

initial_state

Array of {path, value} objects. These tags are written (via WebDev API) before the test sequence begins. The runner writes all initial_state tags, waits 500ms for propagation, then verifies they were set correctly.

sequence

Ordered array of steps. Each step has:

• step (integer) — execution order
• action — one of: write, read, wait, assert
• description — human-readable step description
• path — tag path (required for write, read, assert)
• value — value to write (required for write)
• delay_ms — milliseconds to wait before executing this step

Actions:

• write — write value to path via WebDev tagWrite
• read — read path via WebDev tagRead, store result for later assertion
• wait — pause execution for delay_ms (no tag interaction)
• assert — immediately check path against value (mid-sequence validation)

expected_outcomes

Array of final assertions evaluated after the sequence completes. Each outcome has:

• path — tag to check
• expected — expected value
• tolerance — numeric tolerance (null for exact match)
• comparison — one of: equals, within_tolerance, greater_than, less_than, not_equals
• description — what this assertion validates

Scenario File Rules

1. One scenario per file. Named: {identity.id}.json
2. Validate JSON before writing. Use sorted keys, 2-space indent.
3. Store in testing/scenarios/ directory.
4. Back up before overwriting: cp file.json file.json.bak

Test Runner

Architecture

runner/
├── main.py              ← CLI entry point (Python 3.10+)
├── executor.py          ← scenario execution engine
├── api_client.py        ← WebDev API client wrapper
├── validator.py         ← scenario JSON schema validation
├── reporter.py          ← results formatting and output
└── config.py            ← runner configuration (gateway URL, timeouts)

Execution Flow

1. Load and validate scenario JSON against schema.
2. Connect to Ignition WebDev API (health check first).
3. Write initial_state tags → wait 500ms → verify writes.
4. Execute sequence steps in order, respecting delay_ms.
5. After sequence completes, evaluate expected_outcomes.
6. Generate results report (JSON + human-readable summary).

API Client

The runner talks to Ignition through WebDev endpoints only:

# env: python3.10+ (external tooling — NOT Ignition)

class IgnitionAPIClient:
    def __init__(self, base_url: str = "http://localhost:8088/system/webdev"):
        self.base_url = base_url

    def tag_write(self, path: str, value) -> dict:
        """POST /tagWrite {"path": path, "value": value}"""

    def tag_read(self, path: str) -> dict:
        """GET /tagRead?path=<path>"""

    def tag_browse(self, root: str = "") -> dict:
        """GET /tagBrowse?root=<root>"""

    def health(self) -> dict:
        """GET /health"""

Results Format

{
  "scenario_id": "PMP-101-START-001",
  "scenario_version": "1.0.0",
  "timestamp": "2026-03-14T10:30:00Z",
  "duration_ms": 4523,
  "result": "PASS",
  "initial_state_verification": {
    "status": "OK",
    "details": []
  },
  "sequence_log": [
    {
      "step": 1,
      "action": "write",
      "path": "[default]Devices/Pumps/P-101/Command/Start",
      "value": true,
      "response": {"status": "ok"},
      "elapsed_ms": 45
    }
  ],
  "outcomes": [
    {
      "path": "[default]Devices/Pumps/P-101/Status/Running",
      "expected": true,
      "actual": true,
      "result": "PASS",
      "description": "Pump should be running after start command"
    }
  ],
  "summary": {
    "total_outcomes": 3,
    "passed": 3,
    "failed": 0
  }
}

Results are stored in testing/results/ as {scenario_id}_{timestamp}.json.

File Structure

testing/
├── CLAUDE.md              ← you are here
├── scenarios/
│   ├── PMP-101-START-001.json
│   ├── PMP-101-STOP-001.json
│   └── VLV-1001-OPEN-001.json
├── runner/
│   ├── main.py
│   ├── executor.py
│   ├── api_client.py
│   ├── validator.py
│   ├── reporter.py
│   └── config.py
└── results/
    └── (timestamped result files)