Home / GenAI Engineer practice test / Evaluation and Monitoring / Question

Databricks Generative AI Engineer Associate · free practice question

A team is evaluating their RAG application, which is deployed with inference logging on a Databricks Model Serving endpoint. After a week of production use…

Exam-style question on Evaluation and Monitoring from the free GenAI Engineer practice test, with the answer and a full explanation below.

Updated November 11, 2026

Evaluation and Monitoring

A team is evaluating their RAG application, which is deployed with inference logging on a Databricks Model Serving endpoint. After a week of production use, they want to determine if the retriever component is successfully providing relevant contextual information for incoming user queries. Which of the following analytical approaches, utilizing the inference logs, would be most appropriate for this assessment?

  1. ACount the requests served per hour from the log records and plot the hourly totals to see whether retrieval load peaks at certain times of day
  2. BExtract the chunks logged with each request, sample 100 query-chunk pairs and score their relevance with an LLM judge
  3. CCompare the average response length in tokens across time intervals to see whether answers are becoming longer as the retrieved context grows
  4. DFind the log entries with blank or null response fields, which indicate failed requests, and compute the failure rate of the serving endpoint
Answer & explanation

Correct answer: B

WHY B is correct: Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production. Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production.

WHY NOT A: Hourly traffic volume analysis is operational capacity planning, not retrieval quality assessment. It tells nothing about whether the retrieved chunks are relevant to user queries.

WHY NOT C: Response verbosity over time may indicate model behavior drift but does not measure retrieval relevance. A brief response can still be grounded in highly relevant retrieved context, or a verbose response may be based on irrelevant chunks.

WHY NOT D: Counting NULL responses measures endpoint reliability and error rate — an important operational metric but entirely separate from retrieval quality. NULL responses indicate system failures, not relevance degradation.

Take the full GenAI Engineer practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Generative AI Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →