<p>This project enhances document retrieval by combining semantic search (FAISS) and keyword-based ranking (BM25). It enables efficient search across PDF documents, using vector embeddings and language model-driven content generation for improved accuracy.</p>

AI-driven document retrieval system using FAISS, BM25 and LLMs for fast, accurate search in legal, academic, corporate and research applications with citations.

Fusion Retrieval: Combining Vector Search and BM25 for Enhanced Document Retrieval

<h2>Project Overview</h2><p>The system processes PDFs using PyMuPDF, extracts text and splits it into manageable chunks with <strong><a href="https://www.langchain.com/" target="_blank">LangChain&rsquo;s</a> RecursiveCharacterTextSplitter.</strong> Text chunks are then embedded using Hugging Face&rsquo;s MiniLM model and stored in FAISS for fast similarity searches. Additionally, <strong><a href="https://python.langchain.com/docs/integrations/retrievers/bm25/" target="_blank">BM25</a></strong><a href="https://python.langchain.com/docs/integrations/retrievers/bm25/" target="_blank"> </a>scoring enhances retrieval by ranking documents based on keyword relevance. An LLM-powered hypothetical document is generated using <strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" target="_blank">DeepSeek-R1-Distill-Qwen-1.5B</a></strong> to improve responses, refining search results with context-aware insights. Retrieved documents are displayed with citations, making this system ideal for academic research, legal analysis, enterprise knowledge retrieval, and AI-driven Q&amp;A solutions.</p><h2>Prerequisites</h2><ul><li><strong><a href="https://colab.research.google.com/" target="_blank">Google Colab</a> or a local Python environment</strong> to run the code.  </li><li><strong>Python 3.8+</strong> installed.  </li><li><strong>Libraries &amp; Dependencies:</strong>  <ul><li>langchain, langchain-community, langchain-openai, langchain-cohere  </li><li>sentence-transformers, faiss-cpu, PyMuPDF, rank-bm25  </li><li>openai, transformers, torch, accelerate, pypdf  </li></ul></li><li><strong>Google Drive access</strong> (if running on Colab) for storing and retrieving PDFs.  </li><li><strong>A pre-trained language model</strong> (e.g., <strong>DeepSeek-R1-Distill-Qwen-1.5B</strong>) for LLM-based generation.  </li><li><strong>Basic understanding of FAISS, BM25 and text embeddings</strong> for fine-tuning retrieval.</li></ul><h2>Approach</h2><p>The system starts by processing PDF documents with <strong>PyMuPDF</strong> (fitz) to extract the text. This text is then divided into manageable chunks using <strong>LangChain&rsquo;s RecursiveCharacterTextSplitter</strong>. To maintain clean formatting, tab characters (\t) are replaced with spaces. The text chunks are transformed into vector embeddings using <strong><a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2" target="_blank">Hugging Face&rsquo;s MiniLM</a></strong> model and stored in <strong>FAISS</strong>, which enables quick semantic similarity searches. Additionally, <a href="https://python.langchain.com/docs/integrations/retrievers/bm25/" target="_blank">BM25 </a>ranking is utilized to improve keyword-based retrieval, ensuring that documents are ranked according to both contextual meaning and exact keyword matches. A hybrid retrieval approach merges FAISS-based semantic search with BM25 scoring for better accuracy. To further enhance the results, an LLM-powered hypothetical document is generated using DeepSeek-R1-Distill-Qwen-1.5B, offering context-aware search improvements. Finally, the retrieved documents are presented with citations, ensuring transparency and traceability for research, enterprise search and AI-driven Q\&amp;A applications.</p><h2>Workflow and Methodology</h2><h3>Workflow:</h3><h4>Step 1: Data Ingestion</h4><ul><li>Manually upload PDF files or load them from Google Drive in Colab.</li></ul><h4>Step 2: Text Extraction &amp; Cleaning</h4><ul><li>Utilize PyMuPDF (fitz) to extract text from PDFs.  </li><li>Replace tab characters (\t) with spaces for better formatting.</li></ul><h4>Step 3: Text Splitting &amp; Chunking</h4><ul><li>Apply <strong>LangChain&rsquo;s RecursiveCharacterTextSplitter</strong> to divide text into <strong>manageable chunks</strong> while maintaining context.</li></ul><h4>Step 4: Embedding Generation &amp; Vector Storage</h4><ul><li>Convert text chunks into <strong>vector embeddings</strong> using <strong>Hugging Face&rsquo;s MiniLM model</strong>.  </li><li>Store these embeddings in <strong>FAISS</strong>, enabling <strong>fast semantic similarity searches</strong>.</li></ul><h4>Step 5: Keyword-Based Retrieval with BM25</h4><ul><li>Compute <strong>BM25 scores</strong> to rank documents based on <strong>keyword similarity</strong>.</li></ul><h4>Step 6: Hybrid Retrieval (FAISS + BM25)</h4><ul><li>Perform <strong>FAISS-based similarity search</strong> to retrieve <strong>semantically relevant documents</strong>.  </li><li>Refine search results using <strong>BM25 keyword ranking</strong> to ensure <strong>both semantic and keyword relevance</strong>.</li></ul><h4>Step 7: LLM-Powered Hypothetical Document Generation</h4><ul><li>Use <strong>DeepSeek-R1-Distill-Qwen-1.5B</strong> to generate a <strong>hypothetical document</strong> based on the user&rsquo;s query.  </li><li>Use this document to enhance retrieval by adding <strong>context-aware results</strong>.</li></ul><h4>Step 8: Retrieval with Citations</h4><ul><li>Attach <strong>source citations</strong> to retrieved documents, including <strong>page numbers (if available)</strong>.  </li><li>Display results in a structured format for <strong>easy interpretation and traceability</strong>.</li></ul><h3>Methodology</h3><ul><li><strong>PDF Processing and Text Cleaning:</strong> Get text from PDFs and format it by removing a few characters that are not necessary to chunk it into smaller parts.  </li><li><strong>Vector Embeddings:</strong> Translate and encode these text chunks as packed vectors utilizing MiniLM for embedding conversion.  </li><li><strong>FAISS-Based Semantic Search:</strong> It embeds the embeddings in FAISS and retrieves them through similarity-based searching.   </li><li><strong>BM25 Keyword Matching:</strong> The BM25 ranking is used to advance searches by keyword relevance.   </li><li><strong>Hybrid Retrieval:</strong> A combination of FAISS search in similarity and BM25 scoring to improve accuracy.   </li><li><strong>Hypothetical Answer Generation:</strong> The DeepSeek-R1-Distill-Qwen-1.5B is employed to produce context-rich documents that enhance retrieval quality.   </li><li><strong>Citations of Results:</strong> Create source references in retrieved documents for traceability and validation.</li></ul><h2>Data Collection and Preparation</h2><h3>Data Collection</h3><ol><li>Users manually upload PDF files through Google Colab using files.upload().  </li><li>Alternatively, files can be loaded from Google Drive by mounting them in Colab.</li></ol><h3>Data Preparation Workflow</h3><ol><li>Upload PDFs via Google Colab or Google Drive.  </li><li>Extract text using PyMuPDF (fitz).  </li><li>Clean text by removing tab characters (<code>\t</code>).  </li><li>Split text into chunks using LangChain&rsquo;s RecursiveCharacterTextSplitter.  </li><li>Generate embeddings with Hugging Face&rsquo;s MiniLM model.  </li><li>Store embeddings in FAISS for fast semantic search.</li></ol><p>This process ensures <strong>clean, structured and retrievable data</strong> for the system. </p>


<h2>Code Explanation</h2><h3><span style="color: inherit; font-family: inherit; font-size: 1.75rem;">Installing Required Libraries</span></h3><p>This code installs essential libraries for building an AI-powered retrieval system. It includes LangChain, FAISS, PyMuPDF, Sentence Transformers and Rank-BM25 for text embeddings, vector search and document processing. It also installs PyPDF for handling PDF files. </p>


<h3>Importing Libraries for Document Processing &amp; AI Retrieval</h3><p>This code imports essential libraries for PDF reading, text processing and AI-powered retrieval. It uses PyMuPDF (fitz) for PDFs, LangChain for text splitting and embeddings, FAISS for vector storage and BM25 for keyword-based search. It also includes HuggingFace Transformers for LLM-based text generation and retrieval. </p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.75rem;">Mounting Google Drive</span></h3><p>This code mounts Google Drive to Colab, allowing access to files stored in Drive. The mounted directory is /content/drive, enabling seamless file handling.</p>


<h3>Uploading PDF Files in Colab</h3><p>This code allows manual PDF file uploads in Google Colab. It uses files.upload() to let users select files, then extracts their filenames into pdf_paths for further processing. </p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.75rem;">Loading a Pretrained LLM for Text Generation</span></h3><p>This code loads the <strong>DeepSeek-R1-Distill-Qwen-1.5B</strong> model for <strong>text generation</strong> using Hugging Face. It initializes the tokenizer, loads the model with optimized settings (float16 precision and automatic device mapping) and creates an inference pipeline for generating text. </p>


<h3>Processing PDFs and Creating a FAISS Vector Store</h3><p>This code processes multiple PDFs into a FAISS vector store for efficient similarity search. It loads PDFs using PyPDFLoader, splits the text into chunks with overlap for better retrieval and cleans the text by replacing tab characters (\t) with spaces. The cleaned text is then converted into embeddings using HuggingFace's MiniLM model and stored in FAISS for fast retrieval. Finally, it confirms the successful creation of the vector store. </p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.75rem;">Hybrid Retrieval Using FAISS and BM25</span></h3><p>This code combines <strong>FAISS</strong> (semantic search) and BM25 (keyword-based search) for better document retrieval. The bm25_retrieval function scores documents based on keyword relevance and returns the top k matches. The hybrid_retrieval function first retrieves documents using FAISS, then refines the results using BM25 for a balanced mix of semantic and keyword-based relevance. This approach improves search accuracy by leveraging both methods</p>


<h3>Generating Hypothetical Documents Using LLM</h3><p>This function generates a <strong>hypothetical document</strong> using the <strong>DeepSeek/Qwen</strong> model. It constructs a prompt based on the given query and requests a response with a specified character limit (chunk_size). The model generates text in response to the query, ensuring a detailed and relevant answer. This is useful for <strong>enhancing retrieval systems</strong> by creating additional context. </p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.75rem;">Retrieving Documents with Citations</span></h3><p>This function retrieves relevant documents from a <strong>FAISS vector store</strong> and attaches <strong>source citations</strong>. It first generates a <strong>hypothetical document</strong> using an LLM based on the query, then finds similar documents using <strong>semantic search</strong>. The results are formatted with citations, including <strong>page numbers</strong> when available. This approach enhances document retrieval by providing <strong>contextual answers with references</strong>.</p>


<h3>Testing the Retrieval System</h3><p>This code tests the <strong>retrieval system</strong> by querying it with <em>"What is the main cause of climate change?"</em>. It first generates a <strong>hypothetical document</strong> using an LLM, then retrieves <strong>similar documents</strong> from the FAISS vector store based on semantic similarity. Finally, it attached <strong>citations</strong> to the retrieved documents, ensuring accurate and contextually relevant results with proper references. </p>

<h3>Displaying the Hypothetical Answer</h3><p>This code prints the <strong>hypothetical answer</strong> generated by the LLM in a readable format. It uses textwrap.fill() to wrap the text at <strong>100 characters per line</strong>, ensuring better readability. This helps present the AI-generated response in a structured way. </p>

<h3>Displaying Retrieved Documents with Citations</h3><p>This code prints the <strong>retrieved documents</strong> along with their <strong>citations</strong> for better reference. It loops through the results, displaying the <strong>source identifier</strong>(e.g., [1] (Page X)) and neatly formatting the text using textwrap.fill() to maintain readability. This ensures the retrieved content is structured and easy to review.</p>

<h2>Conclusion</h2><p>This project efficiently combines FAISS-based semantic search, BM25 keyword ranking and LLM-powered text generation to enhance document retrieval from PDFs. By leveraging text embeddings, hybrid retrieval and citation-based results, it ensures accurate, context-aware and transparent search outcomes. The integration of DeepSeek-R1-Distill-Qwen-1.5B further improves retrieval by generating hypothetical responses for better context matching. This system is highly useful for academic research, legal document retrieval, enterprise knowledge search and AI-powered Q\&amp;A applications, making it a scalable and intelligent retrieval solution.</p><h2>Challenges New Coders Might Face</h2><ul><li><p><strong>Challenge: Handling Large PDF Files</strong><br /><strong>Solution:</strong> Use batch processing, split documents into smaller chunks and utilize Colab's high-RAM runtime for better performance.  </p></li><li><p><strong>Challenge:  Slow Embedding Generation</strong><br /><strong>Solution:</strong> Use optimized models like all-MiniLM-L6-v2, process text in parallel and enable GPU acceleration in Colab.  </p></li><li><p><strong>Challenge: FAISS Search Accuracy Issues</strong><br /><strong>Solution</strong>: Adjust <strong>chunk size &amp; overlap</strong>, fine-tune <strong>embedding models</strong> and combine FAISS with <strong>BM25 for hybrid retrieval</strong>.  </p></li><li><p><strong>Challenge: Query Mismatch in Retrieval</strong><br /><strong>Solution:</strong> Use LLM-generated hypothetical documents to refine search queries and improve retrieval accuracy.  </p></li><li><p><strong>Challenge: Dependency Installation Issues</strong><br /><strong>Solution:</strong> Ensure Python 3.8+, install dependencies with !pip install --upgrade and use virtual environments for package management</p></li></ul><h2>FAQ</h2><p><strong>Question 1: How does this project combine FAISS and BM25 for document retrieval?</strong><br /><strong>Answer:</strong> This project uses FAISS for semantic similarity search and BM25 for keyword-based ranking. The hybrid retrieval approach ensures both contextual meaning and exact keyword matches, improving search accuracy.</p><p><strong>Question 2: What preprocessing steps are applied to PDFs before retrieval?</strong><br /><strong>Answer:</strong> The PDF text is extracted using PyMuPDF (fitz), cleaned by removing tab characters (\t) and then split into manageable chunks using LangChain’s RecursiveCharacterTextSplitter for efficient processing.</p><p><strong>Question 3: How are embeddings generated for document retrieval?</strong><br /><strong>Answer:</strong> Text chunks are converted into vector embeddings using Hugging Face’s MiniLM model, which enables fast and accurate semantic search using FAISS.</p><p><strong>Question 4: How does the LLM-generated hypothetical document improve retrieval?</strong><br /><strong>Answer</strong>: The LLM (DeepSeek-R1-Distill-Qwen-1.5B) generates a context-aware response based on the query. This hypothetical document helps retrieve more relevant documents by providing a better context match for FAISS.</p><p><strong>Question 5. Why are some retrieved documents not relevant to my query?</strong><br /><strong>Answer</strong>: FAISS relies on vector similarity, which may not always align with keyword intent. Combining it with BM25 ranking and using hypothetical document generation refines the search for better results.</p>

Fusion Retrieval: Combining Vector Search and BM25 for Enhanced Document Retrieval

Project Outcomes

Requirements:

Project Description