A company employing a Retrieval Augmented Generation (RAG) system is facing unexpectedly high monthly expenditures for its Large Language Model (LLM) API invocations. Analysis indicates that a significant portion (40%) of user queries are straightforward, frequently asked questions (e.g., 'What are your operating hours?') that do not necessitate an LLM's processing. Considering a Databricks-friendly setup, what architectural change would yield the GREATEST cost savings specifically addressing this prevalent query pattern?
Answer & explanation
Correct answer: C
WHY C: Query routing bypasses the LLM entirely for the 40% of queries that are simple FAQs, reducing LLM costs by approximately that proportion. Serving FAQ answers from a cached Delta table or lookup eliminates both prompt and completion tokens for those queries. This is the highest-leverage cost optimization: not calling the LLM at all is always cheaper than calling a smaller or more restricted LLM.
WHY NOT A: Switching to a small model (for example a 7B model) for the whole workload may degrade quality on complex reasoning questions. It also still incurs LLM costs for the 40% of queries that do not need an LLM at all. Routing to no LLM is better than routing to a cheaper LLM for that category.
WHY NOT B: Halving max_tokens reduces output token costs but not input/prompt token costs. For short FAQ answers, output tokens are already low; the input prompt (system prompt + retrieved chunks + user question) dominates the cost. This delivers minimal savings for the FAQ category.
WHY NOT D: Increasing top-k from 5 to 10 doubles the number of retrieved chunks, which increases the prompt token count and therefore increases LLM costs, the opposite of the goal.