What is manual-review QA?
Manual-review QA verifies that a quiz case requiring judgment reaches the right queue with enough evidence, remains observable while pending, and exits exactly once through an authorized decision. It covers both the participant experience and the operational record; seeing a case in a dashboard is not proof that ownership, evidence, notification, or downstream actions work.
Define the states before testing. A practical minimum is auto_decided, review_requested, assigned, in_review, more_information_needed, approved, declined, returned_to_automation, timed_out, and closed. Use the system's own labels, but map every state to permitted transitions and an accountable owner.
Related: Define when review is warranted · Test the eventual route
Freeze the evidence record before the decision
Use synthetic identities and answers. Do not test production permissions by changing a real lead. The record should preserve stable identifiers and source values rather than only the labels currently visible in the builder.
| Field | Required evidence |
|---|---|
| Case | Stable synthetic test ID and review reason |
| Configuration | Exact quiz and rule version that produced the proposal |
| Original decision | Answer IDs, derived values, score, result, qualification, proposed route |
| Assignment | Queue, owner, assigned time, service target, reassignment history |
| Participant state | Visible status, promised next step, correction or information path |
| Reviewed decision | Permitted outcome, reason, authority, timestamp |
| Delivery and closure | Storage, CRM, email, analytics, retry, final state, retention class |
Sources: NIST SP 800-92, Guide to Computer Security Log Management
Run ten blocking review-workflow tests
Execute every case from a known version and compare the expected state, visible experience, saved record, owner, delivery, event, and recovery evidence.
| Case | Expected result | Release-blocking failure |
|---|---|---|
| Ordinary automatic | Clear case follows its normal route once | Unnecessary review case |
| Borderline trigger | Declared boundary enters with the exact reason | Wrong band or missing reason |
| Missing evidence | Case blocks safely and requests a correction | Reviewer invited to guess |
| Conflicting evidence | Both facts and sources remain visible | One value silently wins |
| Duplicate request | One case and one active owner | Duplicate case or action |
| Unauthorized reviewer | Decision and restricted evidence remain unavailable | Role can decide or expose data |
| Reassignment | Both owners and notices remain in history | Lost owner or alert |
| Timeout | One escalation and an accurate pending status | Silent decline or duplicate escalation |
| Decision plus retry | One final state across storage, CRM, email, and analytics | Duplicate lead, message, or event |
| Later correction | Linked new decision preserves the original | History rewritten |
Related: Test score-boundary cases · Validate the CRM handoff
Prove that the reviewer can perform a defined job
The reviewer should see the relevant evidence, understand the rule that produced the proposal, and have authority to choose among declared outcomes. NIST's AI Risk Management Framework Playbook recommends measuring and documenting human oversight of AI outputs. Many quiz decisions are deterministic rather than AI-based; the transferable QA principle is to define who oversees which output and retain evidence of what they could inspect and change.
Do not treat an approval click as proof of better decisions, fairness, or compliance. The entry policy, reviewer capability, evidence, and permitted action must fit the actual context.
Test the participant's pending, error, and final states
If review delays a promised result, state that the case is pending, explain the next step, and provide a way to correct an input or supply requested information. Do not display an immediate definitive result and replace it silently later.
W3C guidance calls for accessible validation and feedback about form-submission results. Test keyboard-only completion, text error identification, preserved values, status announcement without unnecessary focus movement, an additional-information request, and the final notice. Color, a spinner, or email alone is not a complete review status.
Sources: W3C, Validating input · W3C, User notification in forms · W3C, Understanding status messages
Reconcile the final decision across every destination
Compare the queue decision with the authoritative stored record, CRM route, email, analytics, and participant notice. Preserve the automated proposal and the reviewed operational decision as distinct facts. Use stable decision and delivery keys so a retry cannot create a second lead, result email, or qualification transition.
Google Analytics recommends separate lifecycle events including generate_lead, qualify_lead, disqualify_lead, and working_lead. Emit an operational event only when the corresponding state changes, not when a reviewer merely opens or refreshes a case.
Related: Track the broader lead lifecycle
Sources: Google Analytics, Recommended events
Worked example: conflicting budget answers
A synthetic services lead selects a $10k–$25k budget band and later enters $2,500 in a free-text field. The declared conflict rule creates one review case instead of allowing answer order to choose the route.
The reviewer sees both source values, the released rule version, proposed nurture route, and conflict reason. After requesting clarification, the reviewer records the corrected value and final route. The CRM receives one final qualification transition, the original proposal remains in history, and the participant receives accurate pending and final messages. The figures are synthetic and are not recommended thresholds or performance benchmarks.
Manual-review release checklist
Release only when the state transitions, permissions, participant messages, final operational record, and recovery behavior agree for every synthetic case.
- Every trigger has one stable reason code and expected evidence set
- Every state has an owner, timestamp, and permitted next states
- Reviewers have the declared authority and least-necessary data access
- Duplicate requests, decisions, deliveries, and retries are idempotent
- Timeouts escalate rather than silently decide
- Pending, error, information-request, and final messages are accessible
- Original and reviewed decisions remain separately reconstructable
- CRM, email, analytics, and reports agree with the final state
- Corrections create linked records without rewriting history
- Test data is removed or retained under a documented rule
Limitations
This checklist tests the declared workflow. It does not prove that a reviewer makes better decisions, select a universal service target, remove bias, or satisfy every legal duty. Human review can add inconsistency and delay as well as catch exceptions. The example is synthetic, and no product, conversion, accuracy, or response-time benchmark is claimed.
Findings and primary sources were reverified on October 21, 2026. Request a factual correction through the contact page.
Related: Read the editorial method · Request a factual correction
Evidence
How to reproduce the method
Original evidence: a ten-case manual-review state-transition matrix, frozen evidence record, synthetic conflict example, permission and retry tests, downstream reconciliation method, and blocking release checklist.
Limitation
Where this conclusion stops
This is a vendor-neutral QA method, not a product test, legal opinion, performance benchmark, or claim that human reviewers are inherently more accurate.
Sources and verification
What this guide relies on
- NIST AI RMF Playbook, measure human oversight
- NIST SP 800-92, Guide to Computer Security Log Management
- W3C, Validating input
- W3C, User notification in forms
- W3C, Understanding status messages
- Google Analytics, Recommended events
- The Lead Quiz Review editorial methodology
Sources and method checked October 21, 2026. External standards are linked to their primary publishers. Request a factual correction.