You need to programmatically determine whether a Lakeflow Spark Declarative Pipeline update is currently RUNNING, FAILED, or COMPLETED, and you also want to retrieve pipeline details. Which REST API surface is the best fit?
Answer & explanation
Correct answer: D
WHY D: Databricks provides a dedicated Lakeflow Spark Declarative Pipelines API (Pipelines API) to create, start, and view details about pipelines, including update/run details that expose state and progress.
WHY NOT A: Jobs can run pipeline tasks, but pipeline-specific state and metadata are exposed via the Pipelines API and the pipeline event log, not job runs.
WHY NOT C: Cluster APIs do not provide pipeline lifecycle state at the pipeline abstraction layer.
WHY NOT B: Pipelines are not just SQL warehouse queries; they run as managed pipeline updates.