<p>In the rapidly evolving field of artificial intelligence, the ability to retrieve accurate information and generate informed responses is paramount, especially for specialized topics like image recognition using deep neural networks. This project implements a Corrective Retrieval-Augmented Generation (CRAG) system that leverages a combination of document retrieval, relevance evaluation, and web search to answer queries intelligently. Integrating a pre-loaded PDF document with real-time web data ensures robust and contextually rich responses, adaptable to varying levels of document relevance.</p>

Corrective Retrieval-Augmented Generation (RAG) enhances response accuracy by dynamically adjusting the retrieval process, ensuring relevant, up-to-date information.

Corrective Retrieval-Augmented Generation (RAG) with Dynamic Adjustments

<h2>Project Overview</h2><p>The project builds a sophisticated query-processing pipeline using Python, powered by libraries like <a href="https://www.langchain.com/" target="_blank">LangChain</a>, <a href="https://openai.com/" target="_blank">OpenAI&rsquo;s GPT-4o</a>, Sentence Transformers, and <a href="https://duckduckgo.com/" target="_blank">DuckDuckGo</a> search. It begins by loading and vectorizing a PDF file ("Image Recognition Using Deep Neural Network") into a Chroma vector store, then employs a FAISS index for similarity-based document retrieval, evaluated by a custom relevance scoring mechanism driven by GPT-4o. Depending on the relevance score, the system either uses the retrieved document, fetches refined knowledge from the web, or combines both, generating a final response with sourced citations&mdash;demonstrated through example queries about neural network-based image recognition and object detection training.</p><h2>Prerequisites</h2><ul><li>Python (version 3.8+) is required to run the project scripts and manage dependencies.  </li><li>Libraries like langchain, openai, <a href="https://www.trychroma.com/" target="_blank">chromadb</a>, tiktoken, pypdf, langchain-openai, langchain-community, sentence_transformers, and duckduckgo-search must be installed via pip.  </li><li>An OpenAI API key needs to be configured in <a href="https://colab.research.google.com/" target="_blank">Google Colab</a> secrets or a .env file for GPT-4o access.  </li><li>The PDF file "Image Recognition Using Deep Neural Network.pdf" must be accessible at the specified path (e.g., Google Drive).  </li><li>Familiarity with a code editor like Google Colab is essential for coding and debugging.  </li><li>Basic understanding of NLP concepts such as embeddings and vector stores is needed to follow the workflow.  </li><li>Sufficient system resources (CPU/GPU, RAM) are required for efficient document processing and model inference.</li></ul><h2>Approach</h2><p>The project adopts a systematic approach to implement a Corrective Retrieval-Augmented Generation (CRAG) system by first setting up a Python environment with necessary libraries and an OpenAI API key, then loading and vectorizing a PDF document ("Image Recognition Using Deep Neural Network.pdf") into a Chroma vector store using PyPDFLoader and SentenceTransformerEmbeddings for efficient retrieval. A FAISS index facilitates similarity-based document retrieval, followed by a relevance evaluation step powered by GPT-4o, which scores document relevance on a 0-1 scale to determine the next action. The system refines retrieved or web-sourced knowledge into key points, parses search results into title-link pairs, and generates a final response with citations using a structured prompt and the language model, ensuring adaptability and accuracy. This workflow is executed through modular functions, culminating in the crag_process that dynamically adjusts based on query needs, as demonstrated with example queries about image recognition and object detection training.</p><h2>Workflow and Methodology</h2><h3><span style="font-size: 18px;">Workflow</span></h3><ul><li>Set up the Python environment by importing required libraries and configuring the OpenAI API key.  </li><li>Load and vectorize the PDF file into a Chroma vector store using encode_pdf.  </li><li>Initialize the GPT-4o language model (llm) and DuckDuckGo search tool (search) for response generation and web queries.  </li><li>Define a query to process through the system.  </li><li>Retrieve relevant documents from the vector store using <a href="https://faiss.ai/index.html" target="_blank">FAISS</a> similarity search with retrieve_documents.  </li><li>Evaluate the retrieved documents&rsquo; relevance to the query using evaluate_documents and GPT-4o scoring.  </li><li>Decide the action based on the highest relevance score.  </li><li>Perform a web search with perform_web_search if needed, rewriting the query and refining results into key points.  </li><li>Combine or select the final knowledge and sources, then generate a response with generate_response.  </li><li>Print the query and final answer with citations for review and validation.</li></ul><h3><span style="font-size: 18px;">Methodology</span></h3><ul><li><strong>Environment Setup:</strong> Configure Python with libraries like LangChain and OpenAI, securing an API key for GPT-4o access.  </li><li><strong>Document Processing:</strong> Use PyPDFLoader to load the PDF, split it into chunks with RecursiveCharacterTextSplitter, and vectorize it with SentenceTransformerEmbeddings into a Chroma store.  </li><li><strong>Retrieval Mechanism:</strong> Employ a FAISS index for a fast similarity search to fetch top-k relevant document chunks based on the query.  </li><li><strong>Relevance Evaluation:</strong> Implement retrieval_evaluator with GPT-4o to score document relevance on a 0-1 scale, guiding the system&rsquo;s decision-making.  </li><li><strong>Dynamic Adjustment:</strong> Design a corrective logic in crag_process to choose between document use, web search, or a hybrid approach based on score thresholds (0.7 and 0.3).  </li><li><strong>Web Search Integration:</strong> Rewrite queries with rewrite_query, fetch results via DuckDuckGo, and parse them into title-link pairs with parse_search_results.  </li><li><strong>Knowledge Refinement:</strong> Extract key points from documents or web results using knowledge_refinement for concise, usable information.  </li><li><strong>Response Generation:</strong> Format knowledge and sources into a prompt, leveraging GPT-4o in generate_response to produce a coherent answer with citations.  </li><li><strong>Execution and Output:</strong> Run the full pipeline with crag_process, logging steps and displaying the query and response for transparency.</li></ul><h2>Data Collection and Preparation</h2><h3><span style="font-size: 18px;">Data Collection</span></h3><ul><li>Obtain the PDF file "Image Recognition Using Deep Neural Network.pdf" as the primary data source.  </li><li>Store the PDF in a specified directory (e.g., "/content/drive/MyDrive/...") accessible to the script.</li></ul><h3><span style="font-size: 18px;">Data Preparation Workflow</span></h3><ul><li>Load the PDF using <a href="https://python.langchain.com/docs/integrations/document_loaders/pypdfloader/" target="_blank">PyPDFLoader</a> from the defined path.  </li><li>Split the document into chunks of 1000 characters with RecursiveCharacterTextSplitter.  </li><li>Generate embeddings for chunks using SentenceTransformerEmbeddings (model: "all-mpnet-base-v2").  </li><li>Create a Chroma vector store with encode_pdf to store the vectorized document.  </li><li>Assign the vector store to vectorstore for retrieval in the CRAG process.</li></ul>


<h2>Code Explanation</h2><h4>Mounting Google Drive</h4><p>This code mounts Google Drive to Colab, allowing access to files stored in Drive. The mounted directory is /content/drive, enabling seamless file handling.</p>

<p>This command installs five Python libraries using pip: langchain for building language model applications, openai for accessing OpenAI's API, chromadb for working with a vector database, tiktoken for tokenizing text efficiently, and pypdf for handling PDF file operations. These tools are commonly used together for tasks like natural language processing, document retrieval, or AI-powered search systems</p>

<h4>Installing Necessary Packages</h4><p>These commands install langchain-openai, langchain-community, sentence_transformers, and an updated duckduckgo-search, enabling OpenAI integration with LangChain, community tools, sentence embeddings, and web search functionality for advanced NLP and data retrieval tasks.</p>

<h4>Setting Up OpenAI API and Environment</h4><p>This code sets up a Python environment by importing essential libraries and modules like os, json, and LangChain tools, then retrieves an <a href="https://www.aionlinecourse.com/blog/openai-api-error-the-api-key-client-option-must-be-set-either-by-passing-api-key" target="_blank">OpenAI API key</a> from Google Colab secrets or an environment variable. It configures the API key in the environment for use with ChatOpenAI, raising an error if the key is missing, and appends a parent directory to the system path for additional module access. The script ensures proper setup for an <a href="https://www.aionlinecourse.com/ai-basics/natural-language-processing-nlp" target="_blank">NLP</a> project reliant on OpenAI's API, suppressing warnings for cleaner output.</p>

<h4>Defining File Path for Project Resource</h4><p>This code assigns a string to the variable path, specifying the location of a PDF file titled "Image Recognition Using Deep Neural Network" within a nested directory structure under Google Drive. The path points to a resource likely used in a generative AI project focused on Corrective <a href="https://www.aionlinecourse.com/ai-projects/playground/graph-enhanced-retrieval-augmented-generation-graph-rag" target="_blank">Retrieval-Augmented Generation (RAG)</a> with dynamic adjustments, indicating the file’s relevance to the project’s context.</p>

<h4>Processing and Vectorizing PDF Content</h4><p>This code defines a function encode_pdf that takes a PDF file path, loads the document using PyPDFLoader, and splits it into chunks of 1000 characters with no overlap using RecursiveCharacterTextSplitter. It then generates embeddings for these chunks using the SentenceTransformerEmbeddings model "all-mpnet-base-v2" and stores them in a Chroma vector database, enabling efficient retrieval and analysis of the PDF content for tasks like Retrieval-Augmented Generation (RAG).</p>

<h4>Creating Vector Store from PDF</h4><p>This code calls the encode_pdf function with the previously defined path variable, processing the specified PDF file and returning a Chroma vector store containing the document’s text chunks and their embeddings. The result is assigned to vectorstore, making the PDF’s content searchable and usable for downstream NLP tasks like retrieval or generation.</p>

<h4>Initializing OpenAI Chat Model</h4><p>This code creates an instance of ChatOpenAI named llm, configured to use the "gpt-4o" model from OpenAI with a maximum output of 1000 tokens and a temperature of 0 for deterministic responses. It sets up a language model for generating precise, controlled text, likely intended for use in tasks like answering queries or processing data from the vector store.</p>

<h4>Setting Up Web Search Tool</h4><p>This code initializes a DuckDuckGoSearchResults object named search, enabling web searches via the DuckDuckGo engine. It provides a tool to retrieve online information, which can complement tasks like Retrieval-Augmented Generation by fetching additional context or data as needed.</p>

<h4>Evaluating Document Relevance to Query</h4><p>This code defines a retrieval_evaluator function that assesses how relevant a document is to a given query, returning a score between 0 and 1. It uses a PromptTemplate to format the query and document for the llm (a ChatOpenAI instance), which outputs a structured RetrievalEvaluatorInput object containing the relevance score, leveraging the language model’s judgment for retrieval accuracy in tasks like RAG.</p>

<h4>Extracting Key Information from Document</h4><p>This code defines a knowledge_refinement function that processes a document to extract its key points as a list of strings, using a PromptTemplate to instruct the llm (ChatOpenAI) to summarize the content in bullet points. The function structures the output with KnowledgeRefinementInput, invokes the language model with the document, and returns a cleaned list of key points by splitting and stripping the resulting text, enhancing document comprehension for applications like RAG.</p>

<h4>Optimizing Query for Web Search</h4><p>This code defines a rewrite_query function that takes a user query and rephrases it using a PromptTemplate and the LLM (ChatOpenAI) to make it more effective for web searching. The function structures the output with QueryRewriterInput, processes the query through the language model, and returns a stripped, rewritten version tailored for better search engine results, useful for enhancing retrieval in tools.</p>

<h4>Parsing JSON Search Results into Title-Link Pairs</h4><p>This code defines a parse_search_results function that takes a JSON-formatted string of search results, parses it, and returns a list of tuples containing each result’s title and link. It uses a try-except block to handle potential json.JSONDecodeError exceptions, returning an empty list if parsing fails, and defaults missing titles to "Untitled" or links to an empty string, making it robust for processing web search outputs.</p>

<h4>Retrieving Relevant Documents with FAISS</h4><p>This code defines a retrieve_documents function that uses a FAISS index to perform a similarity search for a given query, retrieving the top k (defaulting to 3) most relevant documents. It returns a list of the documents’ content extracted from their page_content attributes, enabling efficient vector-based document retrieval for tasks like Retrieval-Augmented Generation (RAG).</p>

<h4>Assessing Document Relevance Scores</h4><p>This code defines an evaluate_documents function that takes a query and a list of documents, then uses the retrieval_evaluator function to compute a relevance score (between 0 and 1) for each document relative to the query. It returns a list of these scores, providing a quantitative measure of how well each document matches the query, which is useful for ranking or filtering in retrieval systems.</p>

<h4>Executing and Refining Web Search Results</h4><p>This code defines a perform_web_search function that takes a query, rewrites it for better web searchability using rewrite_query, and retrieves results using the search tool. It refines the raw search results into key points with knowledge_refinement and parses them into title-link pairs with parse_search_results, returning both the refined knowledge and source references as a tuple for use in applications like RAG.</p>

<h4>Crafting Query Response with Knowledge and Sources</h4><p>This code defines a generate_response function that uses a PromptTemplate to create a response to a query by combining refined knowledge and a list of sources (title-link pairs). It formats the sources as a readable string, feeds the query, knowledge, and sources into the llm (ChatOpenAI), and returns the generated answer, complete with citations, suitable for delivering informed and sourced responses in a Retrieval-Augmented Generation system.</p>

<h4>Implementing Corrective Retrieval-Augmented Generation (CRAG)</h4><p>This code defines a crag_process function that processes a query by retrieving documents from a FAISS index, evaluating their relevance with scores, and dynamically deciding whether to use the best document (if score &gt; 0.7), perform a web search (if score \&lt; 0.3), or combine both (if score is between 0.3 and 0.7). It refines knowledge accordingly, tracks sources, and generates a final response using the generate_response function, printing intermediate steps for transparency in this adaptive Retrieval-Augmented Generation workflow.</p>

<h4>Executing CRAG Process for Image Recognition Query</h4><p>This code sets a query string asking about "Image Recognition Using Deep Neural Network," then calls the crag_process function with this query and a previously created vectorstore (a Chroma vector store from a PDF). It processes the query through retrieval, evaluation, and response generation, storing the result in result, and finally prints both the query and the generated answer for review.</p>

<h4>Running CRAG Process for Object Detector Training Query</h4><p>This code defines a query asking about building an object detector with multi-step training, then invokes the crag_process function using this query and the existing vectorstore (from a PDF). It executes the full CRAG workflow—retrieval, evaluation, and response generation—storing the outcome in result, and prints the query and answer for inspection.</p>

<h2>Conclusion</h2><p>This project successfully implements a Corrective Retrieval-Augmented Generation (CRAG) system, integrating deep neural network insights from a PDF with dynamic web search capabilities to deliver accurate, sourced responses. By leveraging GPT-4o, FAISS, and Chroma vector stores, it demonstrates robust document retrieval, relevance evaluation, and knowledge refinement for queries like image recognition and object detection. The adaptive workflow, powered by SentenceTransformerEmbeddings and DuckDuckGo, ensures flexibility and precision, making it a valuable tool for NLP and AI-driven research.</p><h2>Challenges New Coders Might Face</h2><ul><li><p><strong>Challenge: Large Document Processing</strong><br /><strong>Solution:</strong> Use text splitting (e.g., RecursiveCharacterTextSplitter) to divide the document into smaller chunks. Additionally, storing the embeddings in chromadb ensures that only the most relevant chunks are searched, reducing the load on the system.</p></li><li><p><strong>Challenge: API Key Configuration</strong><br /><strong>Solution:</strong> Double-check the API key setup in Colab secrets or .env and test with a simple OpenAI call to confirm functionality.  </p></li><li><p><strong>Challenge: Model Incompatibility or Version Mismatch</strong><br /><strong>Solution</strong>: Ensure that the required versions of the libraries and models are properly installed using version management tools like <strong>pip</strong> or <strong>conda</strong>. It's also helpful to document the versions of libraries used for consistency across environments.</p></li><li><p><strong>Challenge: Resource Limitations (RAM/Storage)</strong><br /><strong>Solution:</strong> Use FAISS's disk-based index options, such as FAISS’s IVFPQ (Inverted File with Product Quantization), which allows for efficient disk storage and retrieval without overloading memory.  </p></li><li><p><strong>Challenge: Relevance Scoring Variability</strong><br /><strong>Solution:</strong> Adjust thresholds (e.g., 0.7 to 0.8) or average multiple GPT-4o evaluations to stabilize relevance scoring.  </p></li></ul><h2>FAQ</h2><p><strong>Question 1. Is this project suitable for real-time AI-driven research applications?</strong><br /><strong>Answer:</strong> Yes, with optimization for speed (e.g., caching) and robust NLP tools, it supports AI research on topics like image recognition.</p><p><strong>Question 2. How can I improve the relevance scoring for retrieved documents?</strong><br /><strong>Answer:</strong> Fine-tune the thresholds (e.g., 0.7 to 0.8) in crag_process or average multiple GPT-4o evaluations for consistency.</p><p><strong>Question 3. What role does LangChain play in this project?</strong><br /><strong>Answer:</strong> LangChain orchestrates the CRAG system by handling PDF loading and splitting, managing Chroma and FAISS vector stores for retrieval, and structuring prompts and chains for GPT-4o to evaluate the relevance, refine knowledge, and generate responses efficiently.</p><p><strong>Question 4. How does the text-splitting process work?</strong><br /><strong>Answer:</strong> The RecursiveCharacterTextSplitter splits large text into smaller chunks of a defined size (e.g., 1000 characters), with overlapping sections to maintain context. This ensures that even long documents are handled efficiently while preserving meaning across chunks.</p><p><strong>Question 5. Can I use a different model instead of GPT-4o for relevance evaluation?</strong><br /><strong>Answer:</strong> Yes, modify the ChatOpenAI instance in llm to use another NLP model, adjusting parameters like max_tokens as needed.</p>

Corrective Retrieval-Augmented Generation (RAG) with Dynamic Adjustments

Project Outcomes

Requirements:

Project Description