Home / GenAI Engineer practice test / Evaluation and Monitoring / Question

Databricks Generative AI Engineer Associate · free practice question

A team evaluating an AI agent for legal document analysis observes that the platform's standard LLM evaluation metrics (e.g., for faithfulness, relevance…

Exam-style question on Evaluation and Monitoring from the free GenAI Engineer practice test, with the answer and a full explanation below.

Updated November 11, 2026

Evaluation and Monitoring

A team evaluating an AI agent for legal document analysis observes that the platform's standard LLM evaluation metrics (e.g., for faithfulness, relevance, toxicity) do not adequately assess a crucial domain-specific quality: whether the agent fully identifies all mandatory legal provisions pertinent to their jurisdiction (termed 'clause completeness'). What is the appropriate method to integrate this specific evaluation criterion within Databricks Agent Evaluation?

  1. AOpen a support request asking Databricks to add clause completeness as a new built-in judge, and evaluate with the standard judges until it is available
  2. BApproximate clause completeness with the built-in relevance judge by adding instructions that redefine what relevance means for these documents
  3. CWrite a custom scorer, a Python function decorated with @scorer that checks the response for each required clause, and pass it to the evaluation run
  4. DLeave clause completeness out of the automated evaluation, since Agent Evaluation supports its predefined judges and not custom criteria
Answer & explanation

Correct answer: C

WHY C is correct: Databricks Agent Evaluation's custom Scorers API allows teams to define domain-specific quality dimensions as Python functions decorated with @scorer. The scorer receives the agent's input and response, applies custom logic (clause presence checks via regex or an LLM call against a list of mandatory clauses), and returns a pass/fail value, a number or a Feedback object. This custom scorer is passed in the scorers list of mlflow.genai.evaluate() alongside the built-in judges, integrating clause completeness into the standard evaluation pipeline.

WHY NOT A: Databricks custom Scorers are designed precisely so teams don't need to wait for Databricks to add new judges. The self-service custom scorer API addresses domain-specific needs without platform modifications.

WHY NOT B: Repurposing the relevance score with prompt reinterpretation is semantically incorrect — relevance measures topical alignment to the query, not completeness of specific required content. This would produce unreliable, misleading scores for clause completeness.

WHY NOT D: Databricks Agent Evaluation explicitly supports custom Scorers as a first-class feature for extending evaluation beyond built-in judges. The claim that only pre-built judges are supported is factually incorrect.

Take the full GenAI Engineer practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Generative AI Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →