Build an interactive debugging workspace that helps data engineers reconstruct failed data-processing runs and test recovery strategies without rerunning expensive production workflows.
Allow users to define pipeline stages, dependencies, input datasets, transformation rules, checkpoints, schedules, and recorded execution logs. Represent each run as a directed graph with stage status, duration, input/output counts, and available error details.
Identify where records disappeared, duplicated, changed unexpectedly, or failed validation. Compare successful and failed runs to detect possible schema changes, late-arriving data, dependency failures, and non-idempotent transformations.
Let users simulate retrying a failed stage, replaying from a checkpoint, rebuilding a downstream dataset, or correcting a transformation rule. Show which outputs would need recomputation and which may be reused safely.
Generate reproducible test fixtures and a recovery checklist. Clearly label simulated outcomes, protect sensitive records, and never trigger production writes automatically. Support exportable run histories and lineage diagrams.
0 Comments