A data engineer is constructing an information retrieval pipeline for a knowledge management system, which necessitates extracting textual content from several different document types. The input sources comprise: (1) digital PDF files that inherently contain selectable text (e.g., reports), (2) scanned PDF documents where text is embedded as images and not directly selectable (e.g., archived papers), (3) HTML web pages sourced from an internal network, and (4) Microsoft Word files (.docx format). When using Python, what combination of libraries would yield the most accurate and comprehensive solution for programmatically extracting this textual data from all these varied formats?
Answer & explanation
Correct answer: D
WHY D: Each format gets a tool that actually handles it: pypdf reads the text layer of digital PDFs, pdf2image turns scanned pages into images that pytesseract can OCR, BeautifulSoup extracts clean text from HTML, and python-docx reads Word documents.
WHY NOT A: PyPDF2 only reads PDFs that already carry a text layer. It has no OCR, so scanned PDFs come back empty, and it does not parse HTML or Word files at all.
WHY NOT B: Most of this stack is sound, but Pillow cannot rasterize PDF pages, so pytesseract has no images to read from a scanned PDF. A converter such as pdf2image is needed first, which leaves the scanned documents uncovered.
WHY NOT C: scrapy is a web crawler that parses HTML; it has no built-in parser for PDF or Word content. textract can read some binary files but still depends on external OCR backends for scanned PDFs, so the combination is incomplete.
WHY NOT E: unstructured can route files by type, but OCR for scanned PDFs works only after optional packages and system tools such as Tesseract are installed and configured. The claim that nothing else needs installing is what makes this choice wrong.