A data engineer has a Delta table with deletion vectors enabled. After processing a GDPR 'right-to-be-forgotten' request, they run:
DELETE FROM catalog1.schema1.users WHERE user_id = 42;
VACUUM catalog1.schema1.users;After running these commands, a compliance auditor is concerned that the deleted user's data might still be physically readable in the Parquet files. Is the auditor's concern valid, and why?
Answer & explanation
Correct answer: C
WHY C: When deletion vectors are enabled on a Delta table, DELETE operations are 'soft deletes': they write a deletion vector (a metadata bitmap file) marking which rows are logically deleted, but the Parquet data files containing the original rows are NOT rewritten or removed, so the raw user data physically remains in the current files. VACUUM removes older data files that are no longer referenced by the transaction log, but it does not rewrite current files to remove soft-deleted data. The GDPR purge workflow is: (1) DELETE the records, (2) run REORG TABLE catalog1.schema1.users APPLY (PURGE) to physically rewrite the current Parquet files without the soft-deleted rows, and then (3) run VACUUM to remove the older (pre-REORG) files once they are outside the retention window.
WHY NOT A: Deletion vectors exist to avoid rewriting Parquet files for performance, so DELETE does NOT rewrite files at commit time when they are enabled. The auditor's concern is valid.
WHY NOT B: VACUUM removes older files that are no longer referenced by the transaction log beyond the retention threshold. It does not rewrite current, actively referenced Parquet files, so the user's rows (soft-deleted via deletion vector) remain in them after VACUUM alone.
WHY NOT D: The concern is valid, but the reason given is inaccurate: a command to rewrite the current files does exist. After REORG TABLE ... APPLY (PURGE) the data is physically removed from the current files, and VACUUM then removes the now-old pre-REORG files, so the data does not remain forever.
WHY NOT E: VACUUM does not encrypt data; it deletes files. Encryption at rest is a storage-layer protection that is unrelated to whether raw data bytes are readable by someone with direct cloud storage access.