A Lakeflow Spark Declarative Pipelines (DLT) pipeline ingests customer records. The team wants to enforce that customer_id IS NOT NULL and age BETWEEN 0 AND 120. Invalid records should be excluded from the curated table (not written), but the pipeline should continue processing valid records and produce data quality metrics for how many were dropped. Which expectation action/operator is the best fit?
Answer & explanation
Correct answer: D
WHY D: expect_or_drop corresponds to the “drop on violation” behavior: invalid rows are dropped before writing, and the pipeline records metrics (including dropped counts) for observability. WHY NOT A: expect (warn/default) retains invalid records in the target while logging metrics; it doesn’t exclude them. WHY NOT C: expect_or_fail fails the update when invalid records are detected; in addition, when a flow fails, metrics are not recorded for that failed update in the same way as warn/drop. WHY NOT B: Expectation constraints must be valid SQL boolean expressions and do not allow custom Python functions or external calls in the constraint logic.