PowerSolver 2.0 — Tests User Guide

PowerSolver is tested at four levels: the engine's JUnit suite, an independent checker that re-computes every solution's score in Python, the wizard's node tests (imports, messages, workbooks, Results), and a corpus run against a deployed server. scripts/quality-gate.sh (or .ps1) runs the first two on hundreds of generated problems.

1. Engine — JUnit suite (powersolver-api/src/test/java/)

Add a test under src/test/java/com/powersolver/<package>/; Surefire picks it up. Use AssertJ and small JSON builders as the existing tests do.

1b. Independent checker — powersolver-verify/

verify_solution.py problem.json solution.json re-computes every rule's violations without the engine and reports total_hard_violations, feasible and the per-rule counts. The pytest suite (test_differential.py, test_corpus_all_types.py, test_rule_examples.py, test_rule_reference.py, test_verify.py) checks:

• every solution exported by DifferentialExportTest (shipped demos, rule fixtures, generated problems): the engine's per-rule counts equal the checker's;
• every demo's reference solution is feasible according to the checker;
• the rule examples in rule-examples/ (one or more per rule and mode) give the expected counts;
• RULE_REFERENCE.md is up to date with catalog.json (render_rule_reference.py regenerates it).

1c. Wizard — node tests (powersolver-wizard/tests/)

Also run node tools/check-i18n.mjs (every text key used exists in every locale).

1d. Template corpus — powersolver-template/

One folder per problem type (cloud-capacity, employee-rostering, job-shop, maintenance-scheduling, school-timetabling, task-assignment, workorder-assignment: basic mode; vehicle-routing: sequence mode). A basic-mode folder holds 1-variable … 4-variables cases, N being the number of decisions; the sequence folder holds 1-variable only. Each case has three files:

Regenerate everything with python -I powersolver-template/_generate.py (deterministic). Used by TemplateCorpusSolveTest (every case solves to a feasible plan), ScoreInvariantTest and DifferentialExportTest.

2. Corpus + assertion harness

Each corpus case is a folder under powersolver-api/test-cases/ containing:

test-cases/
└── <NN_descriptive_name>/
    ├── problem.json     ← the §5bis spec to POST
    ├── assertions.json  ← structural + performance assertions
    └── README.md        ← (optional) what this case proves

problem.json is exactly what you'd POST to /api/solve. assertions.json declares what must be true about the result.

The cases shipped today:

Assertion vocabulary

assertions.json shape:

{
  "feasibility": "feasible" | "infeasible" | "any",
  "structural": [ /* assertions, see below */ ],
  "performance": { "max_solve_seconds": <number> }
}

Active assertion kinds (the harness fails the case if any of these are violated):

Deferred kinds (logged-and-skipped, will graduate when a real case needs them):

Adding a new case

mkdir powersolver-api/test-cases/08_my_new_case
# Create problem.json and assertions.json

CorpusTest discovers folders dynamically — no Java code changes needed.

Running the in-process corpus

cd powersolver-api
mvn -B test -Dtest=CorpusTest

Each case shows up as a separate test method.

3. Running the corpus against a deployed environment

Two Python scripts under powersolver-api/scripts/:

run_corpus_against_env.py

Walks every case in test-cases/, POSTs to --solver-url/api/solve, verifies the response against assertions.json using the same harness as the in-process JUnit run.

cd powersolver-api
python scripts/run_corpus_against_env.py --solver-url https://api.planningpowertools.com

Output:

Running 6 case(s) against https://api.planningpowertools.com

  [PASS] 01_basic_single_host_schedule      (1.8s)  score=0hard/-15soft
  [PASS] 02_multi_host_schedule             (3.1s)  score=0hard/-50soft
  [PASS] 03_empty_input                     (0.2s)  score=0hard/0soft
  [PASS] 04_back_to_back_schedule           (2.4s)  score=0hard/-21soft
  [PASS] 05_time_window_schedule            (3.0s)  score=0hard/-13soft
  [PASS] 06_stress_50_tasks                 (5.4s)  score=0hard/-150soft

All cases passed.

Exit code 0 if all green, 1 if any failed.

perf_sanity.py

Runs the stress case N times against the deployed endpoint and reports min/median/mean/max/stdev. Gates on the case's performance.max_solve_seconds budget.

python scripts/perf_sanity.py --solver-url https://api.planningpowertools.com --repeats 5

Output:

Stress case: 06_stress_50_tasks  budget: 15s  solver timeout: 5s
Repeats: 5  target: https://api.planningpowertools.com

  run 1:    5.42s  score=0hard/-150soft     feasible=True
  run 2:    5.39s  score=0hard/-150soft     feasible=True
  run 3:    5.41s  score=0hard/-160soft     feasible=True
  run 4:    5.40s  score=0hard/-145soft     feasible=True
  run 5:    5.43s  score=0hard/-155soft     feasible=True

min     5.39s
median  5.41s
mean    5.41s
max     5.43s
stdev   0.02s

PASSED: all 5 runs within 15s budget.

Where these scripts run

These are pure-Python tools (no Java, no Maven). Run them from any host that can reach the solver URL — your workstation against testing.planningpowertools.com, a CI runner against a PR-environment endpoint, a cron loop against production for the 7-day burn-in.

4. The end-to-end LLM quality test

powerai-api/tests/test_v2_quality_e2e.py runs 20 held-out problem descriptions through PowerAI's run_generation, POSTs each result to the solver, and asserts ≥ 80% feasibility.

cd D:/PowerProjects/PowerAI/powerai-api
POWERAI_E2E_ENABLED=1 POWERSOLVER_API_URL=https://api.planningpowertools.com \
  python -m pytest tests/test_v2_quality_e2e.py

Opt-in via POWERAI_E2E_ENABLED=1 — it costs LLM API calls (~20 per run). The POWERSOLVER_API_URL env var picks which solver instance to hit.

5. Test patterns to copy

Asserting a value-level invariant

If your problem produces a numeric derived shadow that should be monotonic along the sequence, your assertions.json includes:

{
  "structural": [
    {"kind": "monotonic", "shadow": "startTime", "direction": "increasing"},
    {"kind": "monotonic", "shadow": "endTime",   "direction": "increasing"}
  ]
}

If it should stay below a deadline:

{
  "structural": [
    {"kind": "bound", "shadow": "endTime", "max": "2026-06-01T18:00:00"}
  ]
}

Verifying feasibility AND a soft target

{
  "feasibility": "feasible",
  "structural": [
    {"kind": "all_assigned", "on": "tasks"},
    {"kind": "unique_per_host", "on": "tasks"}
  ],
  "performance": {"max_solve_seconds": 10}
}

Catching an infeasible case on purpose

If you want a test that proves the solver correctly reports infeasibility:

{"feasibility": "infeasible"}

(Don't combine feasibility: infeasible with all_assigned — they're contradictory.)

6. Where to go next

• API Reference — to author problem.json from scratch
• Concepts Guide — to understand what the assertions check
• Developer Guide — to extend the harness with new assertion kinds