The AI Evaluation Factory

 Build an AI Evaluation Factory that automatically creates rigorous evaluation suites for AI-powered applications. Let users provide a product description, representative inputs, expected behaviors, prohibited behaviors, and quality requirements. Generate normal cases, adversarial inputs, edge cases, and regression tests. Support configurable graders, human review, repeatable model comparisons, and evaluation runs across prompt versions. Track correctness, consistency, latency, token consumption, and estimated cost. Identify weak evaluation cases that cannot reliably distinguish good outputs from bad ones. Never grade an output as correct without an explicit criterion, and clearly distinguish actual model results from simulated demonstrations.