272 lines
7.4 KiB
Markdown
272 lines
7.4 KiB
Markdown
# Test Scenario Schema & Runner — CLAUDE.md
|
|
|
|
Parent: [../CLAUDE.md](../CLAUDE.md)
|
|
|
|
## Overview
|
|
|
|
Automated PLC I/O validation through JSON-defined test scenarios.
|
|
The test runner is Python 3.10+ and communicates with Ignition exclusively
|
|
through the WebDev API. It never touches tags, files, or Modbus directly.
|
|
|
|
## Scenario Schema
|
|
|
|
Every scenario JSON file has exactly five top-level sections.
|
|
|
|
```json
|
|
{
|
|
"identity": {
|
|
"id": "PMP-101-START-001",
|
|
"version": "1.0.0",
|
|
"description": "Verify P-101 starts when start command is issued and permissives are met",
|
|
"author": "engineer@company.com",
|
|
"created": "2026-03-14",
|
|
"tags": ["pump", "start", "P-101", "plantpax"]
|
|
},
|
|
"device": {
|
|
"type": "Pump",
|
|
"udt_path": "[default]Devices/Pumps/P-101",
|
|
"plc_connection": "ControlLogix",
|
|
"related_devices": [
|
|
{
|
|
"type": "Valve",
|
|
"udt_path": "[default]Devices/Valves/XV-1001",
|
|
"role": "discharge_valve"
|
|
}
|
|
]
|
|
},
|
|
"initial_state": {
|
|
"tags": [
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
|
"value": false
|
|
},
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Faulted",
|
|
"value": false
|
|
},
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Ready",
|
|
"value": true
|
|
},
|
|
{
|
|
"path": "[default]Devices/Valves/XV-1001/Status/FullOpen",
|
|
"value": true
|
|
}
|
|
]
|
|
},
|
|
"sequence": [
|
|
{
|
|
"step": 1,
|
|
"action": "write",
|
|
"description": "Issue start command",
|
|
"path": "[default]Devices/Pumps/P-101/Command/Start",
|
|
"value": true,
|
|
"delay_ms": 0
|
|
},
|
|
{
|
|
"step": 2,
|
|
"action": "wait",
|
|
"description": "Allow PLC scan time for logic execution",
|
|
"delay_ms": 2000
|
|
},
|
|
{
|
|
"step": 3,
|
|
"action": "read",
|
|
"description": "Check running status",
|
|
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
|
"delay_ms": 0
|
|
}
|
|
],
|
|
"expected_outcomes": [
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
|
"expected": true,
|
|
"tolerance": null,
|
|
"comparison": "equals",
|
|
"description": "Pump should be running after start command"
|
|
},
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Faulted",
|
|
"expected": false,
|
|
"tolerance": null,
|
|
"comparison": "equals",
|
|
"description": "Pump should not be faulted"
|
|
},
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Speed",
|
|
"expected": 60.0,
|
|
"tolerance": 2.0,
|
|
"comparison": "within_tolerance",
|
|
"description": "Pump speed should be near setpoint"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Section Details
|
|
|
|
#### identity
|
|
|
|
Required fields: `id`, `version`, `description`
|
|
Optional fields: `author`, `created`, `tags`
|
|
|
|
ID convention: `{DEVICE_PREFIX}-{NUMBER}-{ACTION}-{SEQUENCE}`
|
|
Example: `PMP-101-START-001`, `VLV-1001-OPEN-002`
|
|
|
|
Version follows semver. Bump on any change to sequence or expected outcomes.
|
|
|
|
#### device
|
|
|
|
Required fields: `type`, `udt_path`
|
|
Optional fields: `plc_connection`, `related_devices`
|
|
|
|
`udt_path` is the full Ignition tag path to the device UDT instance.
|
|
`related_devices` lists other devices involved in the test with their role.
|
|
|
|
#### initial_state
|
|
|
|
Array of `{path, value}` objects. These tags are written (via WebDev API)
|
|
before the test sequence begins. The runner writes all initial_state tags,
|
|
waits 500ms for propagation, then verifies they were set correctly.
|
|
|
|
#### sequence
|
|
|
|
Ordered array of steps. Each step has:
|
|
- `step` (integer) — execution order
|
|
- `action` — one of: `write`, `read`, `wait`, `assert`
|
|
- `description` — human-readable step description
|
|
- `path` — tag path (required for `write`, `read`, `assert`)
|
|
- `value` — value to write (required for `write`)
|
|
- `delay_ms` — milliseconds to wait before executing this step
|
|
|
|
Actions:
|
|
- `write` — write `value` to `path` via WebDev tagWrite
|
|
- `read` — read `path` via WebDev tagRead, store result for later assertion
|
|
- `wait` — pause execution for `delay_ms` (no tag interaction)
|
|
- `assert` — immediately check `path` against `value` (mid-sequence validation)
|
|
|
|
#### expected_outcomes
|
|
|
|
Array of final assertions evaluated after the sequence completes.
|
|
Each outcome has:
|
|
- `path` — tag to check
|
|
- `expected` — expected value
|
|
- `tolerance` — numeric tolerance (null for exact match)
|
|
- `comparison` — one of: `equals`, `within_tolerance`, `greater_than`, `less_than`, `not_equals`
|
|
- `description` — what this assertion validates
|
|
|
|
## Scenario File Rules
|
|
|
|
1. **One scenario per file.** Named: `{identity.id}.json`
|
|
2. **Validate JSON before writing.** Use sorted keys, 2-space indent.
|
|
3. **Store in `testing/scenarios/` directory.**
|
|
4. **Back up before overwriting:** `cp file.json file.json.bak`
|
|
|
|
## Test Runner
|
|
|
|
### Architecture
|
|
|
|
```
|
|
runner/
|
|
├── main.py ← CLI entry point (Python 3.10+)
|
|
├── executor.py ← scenario execution engine
|
|
├── api_client.py ← WebDev API client wrapper
|
|
├── validator.py ← scenario JSON schema validation
|
|
├── reporter.py ← results formatting and output
|
|
└── config.py ← runner configuration (gateway URL, timeouts)
|
|
```
|
|
|
|
### Execution Flow
|
|
|
|
1. Load and validate scenario JSON against schema.
|
|
2. Connect to Ignition WebDev API (health check first).
|
|
3. Write `initial_state` tags → wait 500ms → verify writes.
|
|
4. Execute `sequence` steps in order, respecting `delay_ms`.
|
|
5. After sequence completes, evaluate `expected_outcomes`.
|
|
6. Generate results report (JSON + human-readable summary).
|
|
|
|
### API Client
|
|
|
|
The runner talks to Ignition through WebDev endpoints only:
|
|
|
|
```python
|
|
# env: python3.10+ (external tooling — NOT Ignition)
|
|
|
|
class IgnitionAPIClient:
|
|
def __init__(self, base_url: str = "http://localhost:8088/system/webdev"):
|
|
self.base_url = base_url
|
|
|
|
def tag_write(self, path: str, value) -> dict:
|
|
"""POST /tagWrite {"path": path, "value": value}"""
|
|
|
|
def tag_read(self, path: str) -> dict:
|
|
"""GET /tagRead?path=<path>"""
|
|
|
|
def tag_browse(self, root: str = "") -> dict:
|
|
"""GET /tagBrowse?root=<root>"""
|
|
|
|
def health(self) -> dict:
|
|
"""GET /health"""
|
|
```
|
|
|
|
## Results Format
|
|
|
|
```json
|
|
{
|
|
"scenario_id": "PMP-101-START-001",
|
|
"scenario_version": "1.0.0",
|
|
"timestamp": "2026-03-14T10:30:00Z",
|
|
"duration_ms": 4523,
|
|
"result": "PASS",
|
|
"initial_state_verification": {
|
|
"status": "OK",
|
|
"details": []
|
|
},
|
|
"sequence_log": [
|
|
{
|
|
"step": 1,
|
|
"action": "write",
|
|
"path": "[default]Devices/Pumps/P-101/Command/Start",
|
|
"value": true,
|
|
"response": {"status": "ok"},
|
|
"elapsed_ms": 45
|
|
}
|
|
],
|
|
"outcomes": [
|
|
{
|
|
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
|
"expected": true,
|
|
"actual": true,
|
|
"result": "PASS",
|
|
"description": "Pump should be running after start command"
|
|
}
|
|
],
|
|
"summary": {
|
|
"total_outcomes": 3,
|
|
"passed": 3,
|
|
"failed": 0
|
|
}
|
|
}
|
|
```
|
|
|
|
Results are stored in `testing/results/` as `{scenario_id}_{timestamp}.json`.
|
|
|
|
## File Structure
|
|
|
|
```
|
|
testing/
|
|
├── CLAUDE.md ← you are here
|
|
├── scenarios/
|
|
│ ├── PMP-101-START-001.json
|
|
│ ├── PMP-101-STOP-001.json
|
|
│ └── VLV-1001-OPEN-001.json
|
|
├── runner/
|
|
│ ├── main.py
|
|
│ ├── executor.py
|
|
│ ├── api_client.py
|
|
│ ├── validator.py
|
|
│ ├── reporter.py
|
|
│ └── config.py
|
|
└── results/
|
|
└── (timestamped result files)
|
|
``` |