An AI developer is establishing a chatbot for HR policy queries, leveraging PDF documents (originally Word, 10-30 pages long) as source material. Early tests reveal the chatbot frequently references superfluous page elements like footers ('Page X of Y — Restricted — DOC-REF'), navigational headers ('SECTION 3: BENEFITS GUIDE | Table of Contents | Summary'), and document revision histories ('Build 2.1 | Approved by: D. Jones | Date: 09/01/2023') when generating answers. These non-pertinent details appear in retrieved text segments and lead the LLM to mistakenly incorporate them. Among the following, which data cleaning strategy offers the BEST way to remove this non-essential content prior to text segmentation?
Answer & explanation
Correct answer: C
WHY C is correct: Extraneous content (footers, headers, revision tables, navigation elements) must be removed at the data preparation stage, BEFORE text is chunked and embedded. Once boilerplate is embedded into the vector index, it becomes a noise signal that pulls retrieval toward non-informational content. A pre-processing pipeline is the correct intervention point: regex and rule-based matchers can reliably identify and strip recurring structural patterns (page number regex: Page \d+ of \d+; revision table detection via known column headers; navigation sections by header-only content detection). Clean text → chunking → embedding produces a higher-quality index with greater signal-to-noise ratio per chunk.
WHY NOT A: Increasing chunk size dilutes boilerplate but does NOT remove it. A 1,024-token chunk that contains two paragraphs of substantive policy PLUS a full footer and revision table still has those extraneous elements as tokens, which influence the chunk's embedding vector. The vector is biased toward the boilerplate, degrading retrieval quality for policy-specific queries.
WHY NOT E: Asking the LLM to ignore boilerplate context in the system prompt is unreliable. LLMs receive the full context window including the boilerplate text, and instruction-following to selectively ignore specific text patterns within a mixed passage is inconsistent. This approach also wastes context window tokens on non-informational content, reducing the space available for substantive policy text. Filtering at the source is always preferred over asking the LLM to compensate.
WHY NOT D: BM25 penalizes high-frequency terms across documents (low IDF), but 'Confidential' may appear on every page of every document — it does have low IDF and would be deprioritized. However, regulatory policy terms like 'FMLA', 'leave', 'termination' also appear frequently across HR documents and would also be penalized. BM25 switching is not a targeted fix for boilerplate and introduces separate retrieval quality trade-offs. Dense vector retrieval with clean data is superior for semantic HR policy queries.
WHY NOT B: The problem is not scanner resolution — the PDFs are Word document exports (digital PDFs), not scanned images. The issue is the document processing layer (e.g., pdfminer, PyPDF2) extracting text in reading order without semantic page zone separation. Higher DPI is irrelevant for digital PDFs. The fix is pre-processing the extracted text, not changing the export/scan settings.