Create The AI Structured Extraction Adversarial Tester, a testing workbench for applications that extract structured data from invoices, contracts, emails, forms, and other documents.
Allow developers to define a JSON schema, required fields, validation rules, sample source documents, and expected outputs.
The application should:
Generate edge cases involving missing fields, ambiguous dates, conflicting amounts, unusual currencies, and malformed input.
Test whether extracted outputs conform to the schema.
Distinguish syntactic validity from semantic correctness.
Detect unsupported values, incorrect units, inconsistent totals, and fabricated fields.
Compare expected and actual outputs field by field.
Calculate field-level accuracy, completeness, and error severity.
Generate regression tests from failed cases.
Export a reusable evaluation dataset and machine-readable report.
Provide sample invoice, shipping-address, and contract-metadata scenarios. Include deterministic validators and manually supplied model outputs so the application works without external AI APIs. Make every failure traceable to a specific field and rule. Do not treat valid JSON as proof that the extracted information is correct.
0 Comments