Imagine designing a Retrieval-Augmented Generation (RAG) system tasked with analyzing extensive legal documents, each approximately 50 pages long. Within these documents, critical information often resides in passages spanning between 2,000 and 8,000 tokens. Users typically submit concise queries, usually 1 to 3 sentences, rarely exceeding 50 tokens. Given these parameters, what is the most suitable strategy for determining the embedding model's context length?
Answer & explanation
Correct answer: D
WHY D is correct: Source sections of 2,000–8,000 tokens require chunking to fit within any practical embedding model context window. A 512+ token window allows reasonably large, semantically coherent chunks. Asymmetric embedding models (for example the E5 family, which uses separate query and passage prefixes) use separate optimized representations for short queries and long passages, improving retrieval quality when the query–document length gap is large.
WHY NOT A: A 128-token window cannot hold a 50-page document. Without chunking, everything past the first 128 tokens is truncated, so almost none of the document is represented in the embedding.
WHY NOT B: A 512-token window still truncates a 50-page document to its first 512 tokens. Embedding the document as one unit does not preserve its structure; it discards nearly all of it.
WHY NOT C: Optimizing the embedding model for 50-token queries produces a model poorly calibrated for encoding paragraph-length document chunks. Short-context models produce degraded embeddings for longer inputs.