What is change-impact testing?
Change-impact testing is a differential check between two explicit rule versions. It asks not merely whether the proposed version calculates, but exactly what it changes and whether each permitted change propagates consistently.
NIST configuration-management guidance is written for information systems rather than marketing quizzes, but its control pattern is useful: document a change, analyze its functional impact, test and approve it, confirm the implemented result, update the baseline, and retain earlier baselines for traceability and rollback. For a quiz, that means no invisible edit to a live scoring table and no comparison against a moving target.
Related: Govern the scoring-rule change first · Test one rule version at its boundaries
Sources: NIST SP 800-128, Guide for Security-Focused Configuration Management
Freeze the comparison record
Record the request and both rule versions before the first run. Do not compare screenshots alone: keep stable identifiers and machine-visible values beside participant-visible text so another reviewer can reproduce the decision.
| Field | Required evidence |
|---|---|
| Change ID | Stable request identifier and reason |
| Old rule | Immutable version, effective window, formula, boundaries, and precedence |
| Proposed rule | New immutable version with the same fields |
| Input case | Stable question and answer IDs, including missing values |
| Expected difference | Exact output or action allowed to change |
| Forbidden difference | Outputs and actions that must remain equal |
| Evidence | Raw old output, raw proposed output, and comparison result |
| Decision | Pass, block, approved exception, owner, and timestamp |
Use the downstream impact matrix
Run the matrix for every synthetic input. It exposes partial failures: a new result can be correct in the browser while the CRM receives the old route or an email sequence uses the wrong segment.
| Surface | Compare | Blocking failure |
|---|---|---|
| Inputs | Question IDs, answer IDs, required state | An undeclared input change |
| Calculation | Components, transforms, rounding, total | An unexplained component or precision change |
| Decision | Band, result, qualification, hard stop | A changed decision absent from the request |
| Routing | Owner, queue, integration, retry key | Decision and destination disagree |
| Storage | Rule version, score, result, consent state | The applied version cannot be reconstructed |
| Template, recipient, permission, deduplication | A newly ineligible person receives a sequence | |
| Analytics | Event name, parameters, version marker | Reports cannot separate old and new cohorts |
| Participant view | Result, explanation, error, status | Visible text conflicts with the stored decision |
Related: Test downstream lead assignment · Validate analytics event behavior
Build a compact synthetic test set
Keep the set small enough to review and complete for the changed logic. It is not a substitute for broader journey testing when questions, branches, integrations, or consent behavior also change.
- One ordinary case expected to remain unchanged.
- The value immediately below, at, and above every changed boundary.
- Every tie and precedence path touched by the change.
- A missing optional answer and a blocked required answer.
- Every hard-stop or override condition.
- A branch that reaches the changed rule and one that cannot.
- An in-progress journey present at cutover time.
- An integration timeout followed by a retry.
- A rollback run against the preserved old version.
Run the comparison in five passes
Calculation pass: compare each component before the final total so offsetting errors cannot hide behind an equal result. Decision pass: classify every result, qualification, exclusion, tie, or precedence difference as predicted, forbidden, or unexplained.
Action pass: compare destination, owner, sequence, suppression, retry key, and stored fields; a changed decision must trigger the declared action exactly once. Experience pass: verify keyboard flow, error identification, focus, progress, and status announcements on affected paths. W3C guidance calls for clear validation and programmatically determinable status messages.
Reporting pass: confirm every completion stores its applied rule version and reports can split cohorts at the effective time. Google Analytics recommends distinct lead-stage events including generate_lead, qualify_lead, and disqualify_lead; emit only the event that represents the actual operational transition, with the authoritative rule version stored alongside the record.
Sources: W3C, Validating input · W3C, Understanding status messages · Google Analytics, Recommended events
Worked example: one threshold moves
Old Version 3 requires fit of at least 8 and readiness of at least 6. Proposed Version 4 lowers only readiness to 5; both retain the same hard stop. The comparison set fixes fit at 7, 8, and 9, readiness at 4, 5, and 6, plus one missing-readiness case and one hard-stop case.
Only cases with fit of at least 8, readiness exactly 5, and no hard stop may change from nurture to review. Each expected change must also update the review queue, stored version, result email, qualification event, and report cohort. If the browser shows review but the stored route remains nurture, release fails. These values are synthetic—not recommended thresholds or performance benchmarks.
Release and rollback checklist
A release decision should be reproducible from the evidence record, not from memory or a preferred outcome distribution.
- Old and proposed rules are immutable and identifiable.
- The approved request names every permitted difference.
- Ordinary, boundary, tie, missing, hard-stop, branch, cutover, retry, and rollback cases passed.
- Each result matches its route, stored fields, email, event, and report cohort.
- Participant explanations and accessible status messages match the decision.
- In-progress journey behavior at cutover is explicit.
- Historical records retain the rule that made their original decision.
- The effective timestamp, owner, approver, monitoring window, rollback trigger, and preserved prior baseline are recorded.
- Every unexpected difference blocks release or has an approved, documented exception.
Sources: NIST SP 800-128, Guide for Security-Focused Configuration Management
What this matrix cannot prove
The matrix checks deterministic differences and propagation. It does not prove that either rule predicts revenue, eliminate response bias, validate a psychological construct, or establish causality for later outcomes. It does not replace legal or specialist review for regulated decisions. The worked example is synthetic, and this method is not evidence of a product benchmark.
Findings and sources were reverified on October 18, 2026. Request a factual correction through the contact page.
Evidence
How to reproduce the method
A reproducible two-version evidence record combining an eight-surface impact matrix, nine-case synthetic set, five-pass comparison method, worked threshold example, and release-and-rollback checklist.
Limitation
Where this conclusion stops
This deterministic QA method does not establish predictive validity, causality, legal compliance, or a performance benchmark. The worked values are synthetic.
Sources and verification
What this guide relies on
- NIST SP 800-128, Guide for Security-Focused Configuration Management
- W3C, Validating input
- W3C, Understanding status messages
- Google Analytics, Recommended events
- The Lead Quiz Review editorial methodology
Sources and method checked October 18, 2026. External standards are linked to their primary publishers. Request a factual correction.