Act as an AI evaluation researcher. Build The AI Evaluation Dataset Contamination Checker, a tool that helps researchers identify suspicious overlap between evaluation examples and a reference collection of training or development materials.
Allow users to import evaluation datasets and reference documents in supported text or JSON formats.
The application should:
Normalize text while preserving the original records.
Detect exact duplicates and near-duplicate examples.
Compare configurable text segments using transparent similarity methods.
Identify overlapping question wording, copied passages, and repeated answer patterns.
Let researchers adjust similarity thresholds and inspect matched excerpts.
Separate exact matches from approximate matches.
Produce a record-level contamination review queue.
Export flagged pairs with similarity scores and explanations.
Generate a report describing the method, thresholds, and limitations.
Include synthetic datasets with intentional duplicates, paraphrases, and unrelated examples. Do not claim that textual similarity proves training-set contamination, or that no detected overlap proves a benchmark is uncontaminated. Keep the analysis reproducible and make it usable without uploading data to external AI services.
0 Comments