An evaluation was conducted on three LLMs for an internal document question-answering system, yielding the following performance metrics:
- Model A: EM=0.72, F1=0.81, latency=120 ms
- Model B: EM=0.68, F1=0.83, latency=85 ms
- Model C: EM=0.75, F1=0.79, latency=310 ms
If the primary criteria for deployment are a response latency below 150 milliseconds and a stronger emphasis on F1 scores (for partial correctness) compared to EM scores (for exact matches), which model is the optimal pick?
Answer & explanation
Correct answer: B
WHY B is correct: The application has two requirements: (1) latency < 150 ms and (2) prioritize F1 over EM. Model B satisfies both: 85 ms latency is well within the 150 ms SLA, and F1=0.83 is the highest across all models — meaning it produces the most semantically complete answers even when not verbatim exact.
WHY NOT A: Model A meets the latency SLA (120 ms) but F1=0.81 is lower than Model B's 0.83. Since F1 is the prioritized metric, Model B is the better choice.
WHY NOT C: Model C has the highest EM=0.75, but the team explicitly prioritizes F1 over EM. More critically, Model C's 310 ms latency violates the 150 ms SLA, disqualifying it entirely.
WHY NOT D: 'Balancing EM and F1 equally' is not a stated requirement. Model A has lower F1 than Model B and meets the SLA only because it was designed for that threshold — it is still second-best on the prioritized metric.