Initial scaffold: CLAUDE.md modules, directory structure, .gitignore
This commit is contained in:
272
testing/CLAUDE.md
Normal file
272
testing/CLAUDE.md
Normal file
@@ -0,0 +1,272 @@
|
||||
# Test Scenario Schema & Runner — CLAUDE.md
|
||||
|
||||
Parent: [../CLAUDE.md](../CLAUDE.md)
|
||||
|
||||
## Overview
|
||||
|
||||
Automated PLC I/O validation through JSON-defined test scenarios.
|
||||
The test runner is Python 3.10+ and communicates with Ignition exclusively
|
||||
through the WebDev API. It never touches tags, files, or Modbus directly.
|
||||
|
||||
## Scenario Schema
|
||||
|
||||
Every scenario JSON file has exactly five top-level sections.
|
||||
|
||||
```json
|
||||
{
|
||||
"identity": {
|
||||
"id": "PMP-101-START-001",
|
||||
"version": "1.0.0",
|
||||
"description": "Verify P-101 starts when start command is issued and permissives are met",
|
||||
"author": "engineer@company.com",
|
||||
"created": "2026-03-14",
|
||||
"tags": ["pump", "start", "P-101", "plantpax"]
|
||||
},
|
||||
"device": {
|
||||
"type": "Pump",
|
||||
"udt_path": "[default]Devices/Pumps/P-101",
|
||||
"plc_connection": "ControlLogix",
|
||||
"related_devices": [
|
||||
{
|
||||
"type": "Valve",
|
||||
"udt_path": "[default]Devices/Valves/XV-1001",
|
||||
"role": "discharge_valve"
|
||||
}
|
||||
]
|
||||
},
|
||||
"initial_state": {
|
||||
"tags": [
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
||||
"value": false
|
||||
},
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Faulted",
|
||||
"value": false
|
||||
},
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Ready",
|
||||
"value": true
|
||||
},
|
||||
{
|
||||
"path": "[default]Devices/Valves/XV-1001/Status/FullOpen",
|
||||
"value": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"sequence": [
|
||||
{
|
||||
"step": 1,
|
||||
"action": "write",
|
||||
"description": "Issue start command",
|
||||
"path": "[default]Devices/Pumps/P-101/Command/Start",
|
||||
"value": true,
|
||||
"delay_ms": 0
|
||||
},
|
||||
{
|
||||
"step": 2,
|
||||
"action": "wait",
|
||||
"description": "Allow PLC scan time for logic execution",
|
||||
"delay_ms": 2000
|
||||
},
|
||||
{
|
||||
"step": 3,
|
||||
"action": "read",
|
||||
"description": "Check running status",
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
||||
"delay_ms": 0
|
||||
}
|
||||
],
|
||||
"expected_outcomes": [
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
||||
"expected": true,
|
||||
"tolerance": null,
|
||||
"comparison": "equals",
|
||||
"description": "Pump should be running after start command"
|
||||
},
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Faulted",
|
||||
"expected": false,
|
||||
"tolerance": null,
|
||||
"comparison": "equals",
|
||||
"description": "Pump should not be faulted"
|
||||
},
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Speed",
|
||||
"expected": 60.0,
|
||||
"tolerance": 2.0,
|
||||
"comparison": "within_tolerance",
|
||||
"description": "Pump speed should be near setpoint"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Section Details
|
||||
|
||||
#### identity
|
||||
|
||||
Required fields: `id`, `version`, `description`
|
||||
Optional fields: `author`, `created`, `tags`
|
||||
|
||||
ID convention: `{DEVICE_PREFIX}-{NUMBER}-{ACTION}-{SEQUENCE}`
|
||||
Example: `PMP-101-START-001`, `VLV-1001-OPEN-002`
|
||||
|
||||
Version follows semver. Bump on any change to sequence or expected outcomes.
|
||||
|
||||
#### device
|
||||
|
||||
Required fields: `type`, `udt_path`
|
||||
Optional fields: `plc_connection`, `related_devices`
|
||||
|
||||
`udt_path` is the full Ignition tag path to the device UDT instance.
|
||||
`related_devices` lists other devices involved in the test with their role.
|
||||
|
||||
#### initial_state
|
||||
|
||||
Array of `{path, value}` objects. These tags are written (via WebDev API)
|
||||
before the test sequence begins. The runner writes all initial_state tags,
|
||||
waits 500ms for propagation, then verifies they were set correctly.
|
||||
|
||||
#### sequence
|
||||
|
||||
Ordered array of steps. Each step has:
|
||||
- `step` (integer) — execution order
|
||||
- `action` — one of: `write`, `read`, `wait`, `assert`
|
||||
- `description` — human-readable step description
|
||||
- `path` — tag path (required for `write`, `read`, `assert`)
|
||||
- `value` — value to write (required for `write`)
|
||||
- `delay_ms` — milliseconds to wait before executing this step
|
||||
|
||||
Actions:
|
||||
- `write` — write `value` to `path` via WebDev tagWrite
|
||||
- `read` — read `path` via WebDev tagRead, store result for later assertion
|
||||
- `wait` — pause execution for `delay_ms` (no tag interaction)
|
||||
- `assert` — immediately check `path` against `value` (mid-sequence validation)
|
||||
|
||||
#### expected_outcomes
|
||||
|
||||
Array of final assertions evaluated after the sequence completes.
|
||||
Each outcome has:
|
||||
- `path` — tag to check
|
||||
- `expected` — expected value
|
||||
- `tolerance` — numeric tolerance (null for exact match)
|
||||
- `comparison` — one of: `equals`, `within_tolerance`, `greater_than`, `less_than`, `not_equals`
|
||||
- `description` — what this assertion validates
|
||||
|
||||
## Scenario File Rules
|
||||
|
||||
1. **One scenario per file.** Named: `{identity.id}.json`
|
||||
2. **Validate JSON before writing.** Use sorted keys, 2-space indent.
|
||||
3. **Store in `testing/scenarios/` directory.**
|
||||
4. **Back up before overwriting:** `cp file.json file.json.bak`
|
||||
|
||||
## Test Runner
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
runner/
|
||||
├── main.py ← CLI entry point (Python 3.10+)
|
||||
├── executor.py ← scenario execution engine
|
||||
├── api_client.py ← WebDev API client wrapper
|
||||
├── validator.py ← scenario JSON schema validation
|
||||
├── reporter.py ← results formatting and output
|
||||
└── config.py ← runner configuration (gateway URL, timeouts)
|
||||
```
|
||||
|
||||
### Execution Flow
|
||||
|
||||
1. Load and validate scenario JSON against schema.
|
||||
2. Connect to Ignition WebDev API (health check first).
|
||||
3. Write `initial_state` tags → wait 500ms → verify writes.
|
||||
4. Execute `sequence` steps in order, respecting `delay_ms`.
|
||||
5. After sequence completes, evaluate `expected_outcomes`.
|
||||
6. Generate results report (JSON + human-readable summary).
|
||||
|
||||
### API Client
|
||||
|
||||
The runner talks to Ignition through WebDev endpoints only:
|
||||
|
||||
```python
|
||||
# env: python3.10+ (external tooling — NOT Ignition)
|
||||
|
||||
class IgnitionAPIClient:
|
||||
def __init__(self, base_url: str = "http://localhost:8088/system/webdev"):
|
||||
self.base_url = base_url
|
||||
|
||||
def tag_write(self, path: str, value) -> dict:
|
||||
"""POST /tagWrite {"path": path, "value": value}"""
|
||||
|
||||
def tag_read(self, path: str) -> dict:
|
||||
"""GET /tagRead?path=<path>"""
|
||||
|
||||
def tag_browse(self, root: str = "") -> dict:
|
||||
"""GET /tagBrowse?root=<root>"""
|
||||
|
||||
def health(self) -> dict:
|
||||
"""GET /health"""
|
||||
```
|
||||
|
||||
## Results Format
|
||||
|
||||
```json
|
||||
{
|
||||
"scenario_id": "PMP-101-START-001",
|
||||
"scenario_version": "1.0.0",
|
||||
"timestamp": "2026-03-14T10:30:00Z",
|
||||
"duration_ms": 4523,
|
||||
"result": "PASS",
|
||||
"initial_state_verification": {
|
||||
"status": "OK",
|
||||
"details": []
|
||||
},
|
||||
"sequence_log": [
|
||||
{
|
||||
"step": 1,
|
||||
"action": "write",
|
||||
"path": "[default]Devices/Pumps/P-101/Command/Start",
|
||||
"value": true,
|
||||
"response": {"status": "ok"},
|
||||
"elapsed_ms": 45
|
||||
}
|
||||
],
|
||||
"outcomes": [
|
||||
{
|
||||
"path": "[default]Devices/Pumps/P-101/Status/Running",
|
||||
"expected": true,
|
||||
"actual": true,
|
||||
"result": "PASS",
|
||||
"description": "Pump should be running after start command"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"total_outcomes": 3,
|
||||
"passed": 3,
|
||||
"failed": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Results are stored in `testing/results/` as `{scenario_id}_{timestamp}.json`.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
testing/
|
||||
├── CLAUDE.md ← you are here
|
||||
├── scenarios/
|
||||
│ ├── PMP-101-START-001.json
|
||||
│ ├── PMP-101-STOP-001.json
|
||||
│ └── VLV-1001-OPEN-001.json
|
||||
├── runner/
|
||||
│ ├── main.py
|
||||
│ ├── executor.py
|
||||
│ ├── api_client.py
|
||||
│ ├── validator.py
|
||||
│ ├── reporter.py
|
||||
│ └── config.py
|
||||
└── results/
|
||||
└── (timestamped result files)
|
||||
```
|
||||
Reference in New Issue
Block a user