Following the deployment of a Generative AI assistant, the team establishes a continuous performance tracking system that assesses a daily sample of actual user inquiries and system responses using the identical LLM-based quality indicators (e.g., factual accuracy, harmfulness). What key distinction exists between this ongoing monitoring activity and the initial pre-release offline assessment phase?
Answer & explanation
Correct answer: A
WHY A is correct: Offline evaluation is a static, pre-deployment quality gate run on a fixed benchmark dataset to decide whether to release. Online monitoring is continuous and post-deployment, applying evaluation logic (including LLM judges) to live production samples to detect quality regression, data drift, and safety violations over time. The key differences are: data source (static benchmark vs. live traffic), frequency (point-in-time vs. continuous), and purpose (deployment gate vs. production health tracking).
WHY NOT C: Databricks explicitly supports using the same LLM judge configuration — including faithfulness — across both offline evaluation and online monitoring. Faithfulness measurement is not restricted to offline use.
WHY NOT B: Modern Gen AI monitoring systems (including Databricks Agent Evaluation) use LLM judges to assess quality dimensions well beyond system-level latency and error rates.
WHY NOT D: While the same LLM judge configuration is commonly reused across both phases, the processes differ fundamentally in dataset type, cadence, and purpose. Stating they are identical would eliminate the distinction that the Gen AI application lifecycle explicitly makes between them.