A data engineering team needs to programmatically trigger a Databricks Job from an external orchestration system (Apache Airflow running on-premises) after an upstream data delivery is confirmed. The job has two required parameters: run_date (a date string) and env (either prod or staging). The team is using a service principal with the Can Manage Run permission on the job. Which approach correctly triggers the job run with the required parameters using the Databricks REST API?
Answer & explanation
Correct answer: B
WHY B: /api/2.2/jobs/run-now is the correct endpoint for triggering an existing pre-configured job. The job_parameters field passes named key-value pairs that are accessible to all tasks in the job as job-level parameters, replacing the older task-type-specific param fields. This is the modern, recommended approach for parameterized job runs. WHY NOT A: /api/2.2/jobs/runs/submit is for submitting a one-off run with a fully self-contained run spec (including cluster config and task definitions). It is not used to trigger an existing job by ID with parameter overrides. WHY NOT C: Modifying the job definition before each run is an anti-pattern that risks concurrent modification conflicts, pollutes job version history, and requires additional API calls. Parameters should be passed at run time, not written to the job definition. WHY NOT D: While the Databricks CLI can trigger jobs, it does not provide automatic token refresh beyond what the configured credential profile supports. The --params flag syntax described is also not the correct CLI parameter format for job parameters. WHY NOT E: /api/2.0/jobs/run-now is a legacy API version (two versions behind the current 2.2). notebook_params only passes parameters to Notebook tasks, not to Python, SQL, or DLT tasks in a multi-task job. job_parameters in the current API 2.2 is the correct multi-task-aware field.