TOPIC: Parsing PDFs table contents for RAG supporting notebook

README.md

@@ -44,6 +44,7 @@ The [`notebooks`](notebooks/README.md) folder contains a range of executable Pyt
 - [`Document Chunking with Ingest Pipelines`](./notebooks/document-chunking/with-index-pipelines.ipynb)
 - [`Document Chunking with LangChain Splitters`](./notebooks/document-chunking/with-langchain-splitters.ipynb)
 - [`Calculating tokens for Semantic Search (ELSER and E5)`](./notebooks/document-chunking/tokenization.ipynb)
+- [`Fetch surrounding chunks`](./supporting-blog-content/fetch-surrounding-chunks/fetch-surrounding-chunks.ipynb)
 
 ### Search
 

supporting-blog-content/alternative-approach-for-parsing-pdfs-in-rag/README.md

@@ -0,0 +1,34 @@
+
+# PDF Parsing - Table Extraction
+
+ Python notebook demonstrates an alternative approach to parsing PDFs, particularly focusing on extracting and converting tables into a format suitable for search applications such as Retrieval-Augmented Generation (RAG). The notebook leverages Azure OpenAI to process and convert table data from PDFs into plain text for better searchability and indexing.
+
+## Features
+- **PDF Table Extraction**: The notebook identifies and parses tables from PDFs.
+- **LLM Integration**: Calls Azure OpenAI models to provide a text representation of the extracted tables.
+- **Search Optimization**: The parsed table data is processed into a format that can be more easily indexed and searched in Elasticsearch or other vector-based search systems.
+
+## Getting Started
+
+### Prerequisites
+- Python 3.x
+- Output Directory
+  - Example
+    - `/tmp`
+  - Parsed output file name
+    - Example
+      - `parsed_file.txt`
+- Azure Account
+  - OpenAI deployment
+  - Key
+    - Example
+      - a330xxxxxxxde9xxxxxx
+  - Completions endpoint such as GPT-4o
+    - Example
+      - https://exampledeploy.openai.azure.com/openai/deployments/gpt-35-turbo-16k/chat/completions?api-version=2024-08-01-preview
+  - For more information on getting started with Azure OpenAI, check out the official [Azure OpenAI ChatGPT Quickstart](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart?tabs=command-line%2Ctypescript%2Cpython-new&pivots=programming-language-studio).
+
+
+## Example Use Case
+This notebook is ideal for use cases where PDFs contain structured tables that need to be converted into plain text for indexing and search applications in environments like Elasticsearch or similar search systems.
+

supporting-blog-content/alternative-approach-for-parsing-pdfs-in-rag/alternative-approach-for-parsing-pdfs-in-rag.ipynb

@@ -0,0 +1,216 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e9-GuDRKCz_1"
+   },
+   "source": [
+    "# PDF Parsing - Table Extraction\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "MBdflc9G0ICc"
+   },
+   "source": [
+    "## Objective\n",
+    "This Python script extracts text and tables from a PDF file, converts the tables into a human-readable text format using Azure OpenAI, and writes the processed content to a text file. The script uses pdfplumber to extract text and table data from each page of the PDF. For tables, it sends a cleaned version (handling any missing or None values) to Azure OpenAI, which generates a natural language summary of the table. The extracted non-table text and the summarized table text are then saved to a text file for easy search and readability."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "QBwz0_VNL1p6"
+   },
+   "outputs": [],
+   "source": [
+    "!pip install pdfplumber pandas"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "QC37eVM90few"
+   },
+   "source": [
+    "This code imports necessary libraries for PDF extraction, data processing, and interacting with Azure OpenAI via API calls. It retrieves the Azure OpenAI API key and endpoint from Google Colab's userdata storage, sets up the required headers, and prepares for sending requests to the Azure OpenAI service."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "X3vuHZTjK6l7"
+   },
+   "outputs": [],
+   "source": [
+    "import pdfplumber\n",
+    "import pandas as pd\n",
+    "import requests\n",
+    "import base64\n",
+    "import json\n",
+    "from getpass import getpass\n",
+    "import io  # To create an in-memory file-like object\n",
+    "import os\n",
+    "\n",
+    "# Endpoint example\n",
+    "# https://my-deploymet.openai.azure.com/openai/deployments/gpt-4o-global/chat/completions?api-version=2024-08-01-preview\n",
+    "ENDPOINT = getpass(\"Azure OpenAI Completions Endpoint: \")\n",
+    "\n",
+    "API_KEY = getpass(\"Azure OpenAI API Key: \")\n",
+    "\n",
+    "\n",
+    "##Directory where parsed output file will be written to\n",
+    "PARSED_PDF_DIRECTORY = getpass(\"Output directory for parsed PDF: \")\n",
+    "\n",
+    "##Name of output parsed file\n",
+    "PARSED_PDF_FILE_NAME = getpass(\"PARSED PDF File Name: \")\n",
+    "\n",
+    "\n",
+    "headers = {\n",
+    "    \"Content-Type\": \"application/json\",\n",
+    "    \"api-key\": API_KEY,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "79VOKKam0leA"
+   },
+   "source": [
+    "This code defines two functions: extract_table_text_from_openai and parse_pdf. The extract_table_text_from_openai function sends a table's plain text to Azure OpenAI for conversion into a human-readable description by building a request payload and handling the response. The parse_pdf function processes a PDF file page by page, extracting both text and tables, and sends the extracted tables to Azure OpenAI for summarization, saving all the content (including summarized tables) to a text file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "CdMm1AKJLKbA"
+   },
+   "outputs": [],
+   "source": [
+    "def extract_table_text_from_openai(table_text):\n",
+    "    # Payload for the Azure OpenAI request\n",
+    "    payload = {\n",
+    "        \"messages\": [\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": [\n",
+    "                    {\n",
+    "                        \"type\": \"text\",\n",
+    "                        \"text\": \"You are an AI assistant that helps convert tables into a human-readable text.\",\n",
+    "                    }\n",
+    "                ],\n",
+    "            },\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": f\"Convert this table to a readable text format:\\n{table_text}\",\n",
+    "            },\n",
+    "        ],\n",
+    "        \"temperature\": 0.7,\n",
+    "        \"top_p\": 0.95,\n",
+    "        \"max_tokens\": 4096,\n",
+    "    }\n",
+    "\n",
+    "    # Send the request to Azure OpenAI\n",
+    "    try:\n",
+    "        response = requests.post(ENDPOINT, headers=headers, json=payload)\n",
+    "        response.raise_for_status()  # Raise error if the request fails\n",
+    "    except requests.RequestException as e:\n",
+    "        raise SystemExit(f\"Failed to make the request. Error: {e}\")\n",
+    "\n",
+    "    # Process the response\n",
+    "    return (\n",
+    "        response.json()\n",
+    "        .get(\"choices\", [{}])[0]\n",
+    "        .get(\"message\", {})\n",
+    "        .get(\"content\", \"\")\n",
+    "        .strip()\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "def parse_pdf_from_url(file_url):\n",
+    "    # Download the PDF file from the URL\n",
+    "    response = requests.get(file_url)\n",
+    "    response.raise_for_status()  # Ensure the request was successful\n",
+    "\n",
+    "    # Open the PDF content with pdfplumber using io.BytesIO\n",
+    "    pdf_content = io.BytesIO(response.content)\n",
+    "\n",
+    "    # Ensure the directory exists and has write permissions\n",
+    "    os.makedirs(PARSED_PDF_DIRECTORY, mode=0o755, exist_ok=True)\n",
+    "\n",
+    "    with pdfplumber.open(pdf_content) as pdf, open(\n",
+    "        os.path.join(PARSED_PDF_DIRECTORY, PARSED_PDF_FILE_NAME), \"w\"\n",
+    "    ) as output_file:\n",
+    "        for page_num, page in enumerate(pdf.pages, 1):\n",
+    "            print(f\"Processing page {page_num}\")\n",
+    "\n",
+    "            # Extract text content\n",
+    "            text = page.extract_text()\n",
+    "            if text:\n",
+    "                output_file.write(f\"Page {page_num} Text:\\n\")\n",
+    "                output_file.write(text + \"\\n\\n\")\n",
+    "                print(\"Text extracted:\", text)\n",
+    "\n",
+    "            # Extract tables\n",
+    "            tables = page.extract_tables()\n",
+    "            for idx, table in enumerate(tables):\n",
+    "                print(f\"Table {idx + 1} found on page {page_num}\")\n",
+    "\n",
+    "                # Convert the table into plain text format (handling None values)\n",
+    "                table_text = \"\\n\".join(\n",
+    "                    [\n",
+    "                        \"\\t\".join(\n",
+    "                            [str(cell) if cell is not None else \"\" for cell in row]\n",
+    "                        )\n",
+    "                        for row in table[1:]\n",
+    "                    ]\n",
+    "                )\n",
+    "\n",
+    "                # Call Azure OpenAI to convert the table into a text representation\n",
+    "                table_description = extract_table_text_from_openai(table_text)\n",
+    "\n",
+    "                # Write the text representation to the file\n",
+    "                output_file.write(\n",
+    "                    f\"Table {idx + 1} (Page {page_num}) Text Representation:\\n\"\n",
+    "                )\n",
+    "                output_file.write(table_description + \"\\n\\n\")\n",
+    "                print(\"Text representation of the table:\", table_description)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "7ig9NSSnLMGt"
+   },
+   "outputs": [],
+   "source": [
+    "# URL of the PDF file\n",
+    "file_url = \"https://raw.githubusercontent.com/elastic/elasticsearch-labs/refs/heads/sunman/supporting-blog-content/alternative-approach-for-parsing-pdfs-in-rag/quarterly_report.pdf\"\n",
+    "\n",
+    "# Call the function to parse the PDF from the URL\n",
+    "parse_pdf_from_url(file_url)"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}