Build an evaluation environment for personal knowledge bases that tests whether stored notes, documents, and saved references can actually answer realistic questions.
Users can import a collection of notes and documents, create evaluation questions, and identify the source passages that should support each answer.
The application must:
Generate candidate questions from the available documents while marking them as AI-generated.
Allow users to create questions that require combining evidence from multiple sources.
Test exact keyword search, metadata filtering, and semantic retrieval when supported by the configured search system.
Measure whether the expected source documents appear in the retrieved results.
Detect duplicate documents, outdated versions, ambiguous titles, and missing metadata that make retrieval difficult.
Compare retrieval configurations using the same evaluation dataset.
Highlight questions for which no adequate evidence exists.
Export evaluation results and a reusable benchmark dataset.
Provide a query simulator, retrieved-passage viewer, relevance-rating interface, and comparison dashboard.
Do not judge retrieval quality solely by whether an AI produces a plausible answer. Separate source retrieval quality from answer quality, avoid fabricating expected evidence, and keep private documents local unless the user explicitly configures an external service.
0 Comments