Evaluation and Monitoring: Databricks Generative AI Engineer Associate Practice Questions

Correct answer: D

WHY D is correct: The core advantage of continuous production monitoring over periodic offline evaluation is coverage of live, dynamic traffic. Production queries differ from curated evaluation sets — they include unanticipated phrasings, edge cases, domain shifts, and adversarial patterns. Agent Monitoring detects regressions caused by changes in user behavior, corpus drift, or downstream model updates in real time, allowing the team to respond before widespread user impact occurs. WHY NOT A: Agent Monitoring does not inherently use a 'faster' LLM judge model. Both offline and online evaluation can use the same judge, and cost differences depend on sampling rate, not a fixed 50% reduction. WHY NOT B: Both offline evaluation results (in MLflow) and Agent Monitoring results can be stored with versioning in Databricks. The storage mechanism is not the primary functional difference between them. WHY NOT C: Agent Monitoring and offline evaluation serve complementary roles in the Gen AI application lifecycle. Offline evaluation provides a controlled, reproducible quality gate before deployment. Production monitoring is not a replacement — the two phases are explicitly distinguished in the Gen AI lifecycle framework.

Correct answer: A

WHY A is correct: AI Gateway rate limiting configures per-route token-per-minute caps that are enforced at the gateway layer — requests exceeding the limit receive a 429 response without reaching the LLM, preventing budget overrun. AI Gateway Usage Tables capture cumulative token consumption per route/key/application in a structured, query-able format that finance teams can aggregate per application for monthly cost allocation. Both capabilities are native AI Gateway features. WHY NOT B: Model Signatures define the input/output schema structure for validation purposes. They do not enforce rate limits and do not log token consumption to a cost-allocation data store. WHY NOT C: MLflow Experiment tags are associated with training and evaluation runs, not with production AI Gateway serving requests. AI Gateway requests are not automatically tagged to MLflow experiments. WHY NOT D: Filtering inference table columns controls data access visibility but does not enforce rate limits, and inference tables log endpoint-level request/response data — not necessarily structured by AI Gateway application routing keys in the format needed for per-application cost allocation.

Correct answer: D

WHY D is correct: Production RAG monitoring requires a multi-dimensional metric set: faithfulness and relevance catch quality degradation from document corpus drift; latency (p50/p95) detects serving infrastructure issues; token consumption drives cost monitoring; safety flag rate detects abuse pattern changes; and retrieval quality helps isolate whether degradation originates in the retriever vs the generator. WHY NOT A: Training loss and GPU hours are training-time metrics. They are irrelevant to a deployed, serving endpoint that has already been trained. Production monitoring tracks inference-time behavior, not training process health. WHY NOT B: UI page load time and index storage size are infrastructure concerns, not LLM quality or safety metrics. Daily active users is a product engagement metric, not a model quality signal for an AI chatbot. WHY NOT C: BLEU score requires ground-truth reference texts to compute — it cannot be calculated in real-time on live production traffic without pre-prepared references. Embedding model training set size and Delta write throughput are static configuration attributes, not ongoing monitoring signals.

Correct answer: D

WHY D is correct: The Databricks review app integrates with Unity Catalog to persist SME annotations. The recommended workflow is: (1) stored annotations become new labeled evaluation examples (thumbs-down + free-text correction = question + expected_response pair), (2) these augment the offline evaluation dataset registered in MLflow Agent Evaluation, (3) re-running mlflow.evaluate() with the augmented dataset against the updated agent measures whether the specific failure modes identified by SMEs have been resolved, (4) high-quality annotation pairs can be used as supervised fine-tuning data for future model versions. WHY NOT A: Exporting to CSV and sharing with an external vendor is operationally slow, introduces data governance risks, and bypasses the Databricks-native annotation-to-evaluation pipeline that the review app is designed to support. WHY NOT B: RLHF (Reinforcement Learning from Human Feedback) is a valid but complex technique that requires significant infrastructure for reward model training. Using thumbs-down annotations exclusively for RLHF while abandoning offline evaluation and prompt engineering ignores simpler, faster improvement levers available in the Databricks platform. WHY NOT C: Ignoring the free-text corrections discards the most actionable signal in the SME annotations — the expert's statement of what the correct response should have been. Binary ratings alone provide a quality signal but not the expected response content needed to build evaluation examples or fine-tuning pairs.

Correct answer: D

WHY D is correct: MLflow custom metrics (Scorers) are created using mlflow.metrics.make_metric(), which accepts an eval_fn that receives a row object with predictions and targets fields, and returns a MetricValue. The metric is given a name and a greater_is_better direction. It is passed to mlflow.evaluate() via the extra_metrics parameter. This is the correct API for integrating custom scoring logic into MLflow's evaluation framework. WHY NOT A: A plain Python function passed directly to extra_metrics without the make_metric wrapper is not recognized by MLflow as a valid custom metric. The extra_metrics parameter expects EvalMetric objects created with make_metric, not raw callables. WHY NOT B: Returning True/False (boolean) instead of a numeric MetricValue, and using mlflow.log_metric() to log a single number outside of the evaluate() framework, does not integrate the scorer into per-example evaluation tables or aggregate evaluation reports. WHY NOT C: A LangChain Runnable chain that writes to a Delta table is a data pipeline, not an MLflow evaluation scorer. It does not integrate with mlflow.evaluate() and cannot produce per-example evaluation results within the MLflow evaluation framework.

Correct answer: A

WHY A is correct: Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production. Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production. WHY NOT B: Hourly traffic volume analysis is operational capacity planning, not retrieval quality assessment. It tells nothing about whether the retrieved chunks are relevant to user queries. WHY NOT C: Response verbosity over time may indicate model behavior drift but does not measure retrieval relevance. A brief response can still be grounded in highly relevant retrieved context, or a verbose response may be based on irrelevant chunks. WHY NOT D: Counting NULL responses measures endpoint reliability and error rate — an important operational metric but entirely separate from retrieval quality. NULL responses indicate system failures, not relevance degradation.

Correct answer: D

WHY D is correct: mlflow.evaluate() invokes LLM judges (configured quality scorers) against each example in the evaluation dataset, producing per-example scores for metrics like faithfulness, answer correctness, and relevance. It aggregates these into dataset-level statistics and logs them to the MLflow experiment run, enabling comparison across model versions. None of this — judge scoring, aggregation, and run comparison — is produced by simple batch inference, which only generates raw model outputs. WHY NOT A: Raw model response text is produced by both batch inference and mlflow.evaluate(). It is not unique to the evaluation function. WHY NOT B: Wall-clock inference time can be measured by any timed batch inference loop. MLflow logs latency metrics more precisely, but wall-clock time measurement alone is not unique to mlflow.evaluate(). WHY NOT C: Package names in the evaluation environment are logged as part of MLflow's run environment metadata — a side effect of any MLflow run, not a unique feature of mlflow.evaluate().

Correct answer: BCD

WHY BCD is correct: Faithfulness (B) requires only the response and its retrieved context — both available without ground truth. Answer Relevance (C) requires only the question and the response — both available at inference time. Retrieval Precision (D) evaluates whether each retrieved chunk is relevant to the query — requires only the query and retrieved chunks, no labeled answer. WHY NOT A: Answer Correctness explicitly compares the generated answer against a labeled reference (ground truth). Without expected_response labels in the dataset, this judge cannot compute a score — it has nothing to compare against. WHY NOT E: Document Recall measures what fraction of the 'correct' or relevant documents were retrieved. To calculate this, you must know which documents are considered correct for each query — a form of ground truth. Without labeled relevant document sets, recall cannot be computed.