Act as an AI application evaluation engineer. Build The AI Workflow Context Sufficiency Tester, a workbench for testing whether an AI workflow receives enough relevant information to complete a task reliably.
Allow users to define a task, required facts, optional context documents, constraints, and expected outputs.
The application should:
Represent required facts as an explicit checklist.
Compare each test case's supplied context against those requirements.
Identify missing, duplicated, contradictory, and irrelevant information.
Simulate context truncation and document-removal scenarios.
Test whether critical requirements remain available after context reduction.
Compare full-context and reduced-context test results when outputs are supplied.
Track which missing facts correlate with specific failure types.
Generate minimal context packages for individual tasks.
Export test cases and a context coverage report.
Include examples for customer support, code review, and document summarization. Distinguish measurable context coverage from actual answer correctness. Do not claim that the presence of required facts guarantees that a model will use them correctly. Make the workbench functional with manually supplied data and repeatable tests.
0 Comments