job_scraping_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alexfazio/firecrawl-quickstart/blob/main/job_scraping_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "intro-section"
      },
      "source": [
        "# Job Board Scraping with Firecrawl and OpenAI\n",
        "\n",
        "By Alex Fazio (https://twitter.com/alxfazio)\n",
        "\n",
        "Github repo: https://github.com/alexfazio/firecrawl-cookbook\n",
        "\n",
        "This Jupyter notebook demonstrates how to build an automated job scraping pipeline using Firecrawl and OpenAI. By combining Firecrawl's web scraping capabilities with OpenAI's Structured Outputs feature, we can efficiently extract and analyze job listings to find the best matches for your skills.\n",
        "\n",
        "Structured Outputs is a powerful capability that ensures the model will always generate responses that adhere to our specified JSON Schema. This means we can:\n",
        "- Extract job details with guaranteed schema compliance\n",
        "- Get reliable, structured responses every time\n",
        "- Process the data efficiently without worrying about format inconsistencies\n",
        "- Build robust pipelines for job matching and analysis\n",
        "\n",
        "By the end of this notebook, you'll be able to:\n",
        "\n",
        "1. Set up a scraping environment with Firecrawl and OpenAI\n",
        "2. Extract structured data from job boards using Firecrawl\n",
        "3. Use OpenAI models with Structured Outputs to analyze job listings and match them with your resume\n",
        "4. Process job data at scale with reliable, schema-validated outputs\n",
        "5. Build type-safe applications using Pydantic models with OpenAI's responses\n",
        "\n",
        "This cookbook is designed for developers who want to automate their job search and leverage AI for better job matching, while maintaining robust data structures and type safety throughout their application.\n",
        "\n",
        "Note: Structured Outputs feature requires specific OpenAI models (gpt-4o-mini-2024-07-18, gpt-4o-2024-08-06, or later). For earlier models, we'll demonstrate alternative approaches using standard JSON formatting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "requirements-section"
      },
      "source": [
        "## Requirements\n",
        "\n",
        "Before proceeding, ensure you have:\n",
        "\n",
        "- Python 3.7 or higher\n",
        "- API keys for both Firecrawl and OpenAI\n",
        "- Required Python packages\n",
        "\n",
        "First, let's install the necessary packages:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ljjBOLUnYOjd",
        "outputId": "69da1051-204e-4eaa-e239-492f5dd98509"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/386.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m \u001b[32m378.9/386.9 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m386.9/386.9 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/76.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.4/76.4 kB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.0/78.0 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m325.2/325.2 kB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "%pip install requests python-dotenv openai --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup-section"
      },
      "source": [
        "## Step 1: Set Up Your Environment\n",
        "\n",
        "Let's set up our environment variables. In Google Colab, we'll use direct input for API keys instead of a .env file:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IDX_i0qyYOjd",
        "outputId": "e130eafb-9b6f-4fb3-d82b-812032f7ba15"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your Firecrawl API key: ··········\n",
            "Enter your OpenAI API key: ··········\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import requests\n",
        "import json\n",
        "from getpass import getpass\n",
        "from openai import OpenAI  # Add this import\n",
        "\n",
        "# Securely get API keys\n",
        "firecrawl_api_key = getpass(\"Enter your Firecrawl API key: \")\n",
        "openai_api_key = getpass(\"Enter your OpenAI API key: \")\n",
        "\n",
        "# Initialize OpenAI client\n",
        "client = OpenAI(api_key=openai_api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c7UcjMm3YOjd"
      },
      "source": [
        "## Step 2: Define the Jobs Page URL and Resume\n",
        "\n",
        "Now, let's specify the job board URL and your resume content:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "GARr0_ALYOjd"
      },
      "outputs": [],
      "source": [
        "# @title Job Search Configuration\n",
        "\n",
        "jobs_page_url = \"https://openai.com/careers/search\"  # @param {type:\"string\"}\n",
        "resume_paste = \"\"\"\n",
        "**John Doe**\n",
        "123 Main Street, Anytown, USA\n",
        "(123) 456-7890 | john.doe@email.com | [linkedin.com/in/johndoe](https://linkedin.com/in/johndoe) | [github.com/johndoe](https://github.com/johndoe)\n",
        "\n",
        "---\n",
        "\n",
        "### Objective\n",
        "Passionate and motivated Machine Learning Engineer with a strong foundation in computer science, statistics, and mathematics. Eager to apply data-driven solutions to real-world problems. Seeking an entry-level position where I can leverage my skills in machine learning, data analysis, and software development.\n",
        "\n",
        "---\n",
        "\n",
        "### Education\n",
        "**Bachelor of Science in Computer Science**\n",
        "University of California, Berkeley, CA — May 2024\n",
        "- Relevant coursework: Machine Learning, Data Structures and Algorithms, Probability and Statistics, Linear Algebra, Artificial Intelligence, Database Systems\n",
        "\n",
        "---\n",
        "\n",
        "### Skills\n",
        "- **Programming Languages:** Python, R, Java, C++\n",
        "- **Machine Learning:** Scikit-learn, TensorFlow, Keras, PyTorch, XGBoost\n",
        "- **Data Analysis:** Pandas, NumPy, Matplotlib, Seaborn\n",
        "- **Tools & Technologies:** Git, Docker, Jupyter Notebooks, SQL, AWS (S3, EC2)\n",
        "- **Mathematics:** Linear Algebra, Probability, Statistics, Calculus\n",
        "- **Software Development:** Agile methodologies, version control (Git)\n",
        "\n",
        "---\n",
        "\n",
        "### Experience\n",
        "\n",
        "**Machine Learning Intern**\n",
        "Tech Innovations Inc., San Francisco, CA — June 2023 to August 2023\n",
        "- Assisted in building and deploying predictive models for customer segmentation, achieving a 20% increase in targeted marketing efficiency.\n",
        "- Preprocessed large datasets using Pandas and NumPy to clean, normalize, and handle missing values.\n",
        "- Developed machine learning algorithms in Python with Scikit-learn and TensorFlow, achieving up to 85% accuracy on classification tasks.\n",
        "- Created visualizations using Matplotlib and Seaborn to communicate data insights to stakeholders.\n",
        "\n",
        "**Data Science Project (Academic)**\n",
        "University of California, Berkeley — January 2024\n",
        "- Built a recommendation system for an e-commerce platform using collaborative filtering techniques.\n",
        "- Conducted exploratory data analysis to identify key user behavior trends and improve the model's performance.\n",
        "- Implemented the model in Python using Scikit-learn, resulting in a 30% increase in user engagement metrics.\n",
        "\n",
        "---\n",
        "\n",
        "### Projects\n",
        "\n",
        "**Spam Detection Using Natural Language Processing (NLP)**\n",
        "- Implemented a spam detection system using a Naive Bayes classifier and TF-IDF vectorization.\n",
        "- Achieved an accuracy of 92% on a public spam dataset.\n",
        "- Utilized NLTK for text preprocessing, including tokenization, stopword removal, and stemming.\n",
        "\n",
        "**Predicting House Prices Using Regression Analysis**\n",
        "- Built a multiple linear regression model to predict house prices based on various features such as location, size, and number of bedrooms.\n",
        "- Utilized Python libraries (Pandas, NumPy, Scikit-learn) to preprocess data and evaluate the model's performance.\n",
        "- Optimized the model using regularization techniques to prevent overfitting.\n",
        "\n",
        "---\n",
        "\n",
        "### Certifications\n",
        "- **Machine Learning Specialization** – Coursera (Andrew Ng, September 2023)\n",
        "- **Deep Learning Specialization** – Coursera (Andrew Ng, October 2023)\n",
        "- **AWS Certified Solutions Architect – Associate** – Amazon Web Services (November 2023)\n",
        "\n",
        "---\n",
        "\n",
        "### Additional Activities\n",
        "- **Hackathons:** Participated in HackUC, developing a real-time object detection application using TensorFlow.\n",
        "- **AI Club Member:** Actively involved in UC Berkeley's AI Club, organizing workshops and study sessions on machine learning topics.\n",
        "\n",
        "---\n",
        "\n",
        "### Languages\n",
        "- English (Fluent)\n",
        "- Spanish (Proficient)\n",
        "\"\"\"\n",
        "\n",
        "assert jobs_page_url != \"\", \"Error: jobs_page_url should not be empty\"\n",
        "assert resume_paste != \"\", \"Error: resume_paste should not be empty\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bYe-sJ6KYOjd"
      },
      "source": [
        "## Step 3: Scrape the Jobs Page\n",
        "\n",
        "Let's use Firecrawl to extract content from the jobs page:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7HTqjZY7YOje",
        "outputId": "ec5ca5e0-90ef-4cd0-ccab-7d8a650243e9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Scraped content length: 409 characters\n"
          ]
        }
      ],
      "source": [
        "def scrape_jobs_page(url):\n",
        "    try:\n",
        "        response = requests.post(\n",
        "            \"https://api.firecrawl.dev/v1/scrape\",\n",
        "            headers={\n",
        "                \"Content-Type\": \"application/json\",\n",
        "                \"Authorization\": f\"Bearer {firecrawl_api_key}\"\n",
        "            },\n",
        "            json={\n",
        "                \"url\": url,\n",
        "                \"formats\": [\"markdown\"]\n",
        "            }\n",
        "        )\n",
        "        if response.status_code == 200:\n",
        "            result = response.json()\n",
        "            if result.get('success'):\n",
        "                return result['data']['markdown']\n",
        "        return \"\"\n",
        "    except Exception as e:\n",
        "        print(f\"Error scraping jobs page: {str(e)}\")\n",
        "        return \"\"\n",
        "\n",
        "# Scrape the jobs page\n",
        "html_content = scrape_jobs_page(jobs_page_url)\n",
        "print(f\"Scraped content length: {len(html_content)} characters\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZwS71vMKYOje"
      },
      "source": [
        "## Step 4: Extract Job Links\n",
        "\n",
        "Use OpenAI's model to extract application links from the scraped content:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MqrOy7KNYOje",
        "outputId": "0e47fd54-3d0f-410c-eefe-9823f8302cb2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 2 job listings\n"
          ]
        }
      ],
      "source": [
        "def extract_apply_links(content):\n",
        "    if not content:\n",
        "        return []\n",
        "\n",
        "    prompt = f\"\"\"\n",
        "Extract up to 30 job application links from the given markdown content.\n",
        "Return the result as a JSON object with a single key 'apply_links' containing an array of strings (the links).\n",
        "The output should be a valid JSON object, with no additional text.\n",
        "\n",
        "Markdown content:\n",
        "{content[:100000]}\n",
        "\"\"\"\n",
        "\n",
        "    try:\n",
        "        completion = client.chat.completions.create(\n",
        "            model=\"gpt-4\",  # Changed from \"gpt-4o\" to \"gpt-4\"\n",
        "            messages=[{\n",
        "                \"role\": \"user\",\n",
        "                \"content\": prompt\n",
        "            }]\n",
        "        )\n",
        "        if completion.choices:\n",
        "            result = json.loads(completion.choices[0].message.content.strip())\n",
        "            return result['apply_links']\n",
        "    except Exception as e:\n",
        "        print(f\"Error extracting links: {str(e)}\")\n",
        "    return []\n",
        "\n",
        "# Extract the links\n",
        "apply_links = extract_apply_links(html_content)\n",
        "print(f\"Found {len(apply_links)} job listings\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N0ODaleMYOje"
      },
      "source": [
        "## Step 5: Extract Details from Each Job\n",
        "\n",
        "Now let's get detailed information about each job using Firecrawl's extraction capabilities:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gKJ3-o0eYOje",
        "outputId": "17b5e4ad-3784-4c87-daa4-c4d71c200f4e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracted details for: Account Director\n",
            "Extracted details for: Account Director - Japan\n",
            "Successfully extracted details for 2 jobs\n"
          ]
        }
      ],
      "source": [
        "# Define the extraction schema\n",
        "schema = {\n",
        "    \"type\": \"object\",\n",
        "    \"properties\": {\n",
        "        \"job_title\": {\"type\": \"string\"},\n",
        "        \"sub_division_of_organization\": {\"type\": \"string\"},\n",
        "        \"key_skills\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n",
        "        \"compensation\": {\"type\": \"string\"},\n",
        "        \"location\": {\"type\": \"string\"},\n",
        "        \"apply_link\": {\"type\": \"string\"}\n",
        "    },\n",
        "    \"required\": [\"job_title\", \"sub_division_of_organization\", \"key_skills\", \"compensation\", \"location\", \"apply_link\"]\n",
        "}\n",
        "\n",
        "def extract_job_details(url):\n",
        "    try:\n",
        "        response = requests.post(\n",
        "            \"https://api.firecrawl.dev/v1/scrape\",\n",
        "            headers={\n",
        "                \"Content-Type\": \"application/json\",\n",
        "                \"Authorization\": f\"Bearer {firecrawl_api_key}\"\n",
        "            },\n",
        "            json={\n",
        "                \"url\": url,\n",
        "                \"formats\": [\"extract\"],\n",
        "                \"actions\": [{\n",
        "                    \"type\": \"click\",\n",
        "                    \"selector\": \"#job-overview\"\n",
        "                }],\n",
        "                \"extract\": {\n",
        "                    \"schema\": schema\n",
        "                }\n",
        "            }\n",
        "        )\n",
        "        if response.status_code == 200:\n",
        "            result = response.json()\n",
        "            if result.get('success'):\n",
        "                return result['data']['extract']\n",
        "    except Exception as e:\n",
        "        print(f\"Error extracting job details: {str(e)}\")\n",
        "    return None\n",
        "\n",
        "# Extract details for each job\n",
        "extracted_data = []\n",
        "for link in apply_links:\n",
        "    job_data = extract_job_details(link)\n",
        "    if job_data:\n",
        "        extracted_data.append(job_data)\n",
        "        print(f\"Extracted details for: {job_data['job_title']}\")\n",
        "\n",
        "print(f\"Successfully extracted details for {len(extracted_data)} jobs\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pprint\n",
        "pprint.pprint(extracted_data)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RNl25-X4bK67",
        "outputId": "6a7e8b1a-db3e-4457-bf2b-4a4ea0ea89d5"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'apply_link': '/openai/e09c71f0-1be2-4141-8ab7-d4f38e583c7e/application',\n",
            "  'compensation': '',\n",
            "  'job_title': 'Account Director',\n",
            "  'key_skills': ['7+ years selling platform-as-a-service and/or '\n",
            "                 'software-as-a-service',\n",
            "                 'Native-level Japanese language proficiency',\n",
            "                 'Achieving revenue targets >$1M per year for more than 3 '\n",
            "                 'years',\n",
            "                 'Designing and executing complex deal strategies',\n",
            "                 'Supporting the growth of fast-growing, high-performance '\n",
            "                 'companies',\n",
            "                 'Working directly with c-level executives',\n",
            "                 'Communicating technical concepts to customers and internal '\n",
            "                 'stakeholders',\n",
            "                 'Leading high-visibility customer events (CAB, conferences, '\n",
            "                 'product launches, etc.)',\n",
            "                 'Gathering, distilling, and processing complex market '\n",
            "                 '(industry, competitor, customer, prospect) intelligence'],\n",
            "  'location': 'Tokyo, Japan',\n",
            "  'sub_division_of_organization': 'Go To Market'},\n",
            " {'apply_link': '/openai/e09c71f0-1be2-4141-8ab7-d4f38e583c7e/application',\n",
            "  'compensation': '',\n",
            "  'job_title': 'Account Director - Japan',\n",
            "  'key_skills': ['7+ years selling platform-as-a-service and/or '\n",
            "                 'software-as-a-service',\n",
            "                 'Native-level Japanese language proficiency',\n",
            "                 'Achieving revenue targets >$1M per year for more than 3 '\n",
            "                 'years',\n",
            "                 'Designing and executing complex deal strategies',\n",
            "                 'Supporting the growth of fast-growing, high-performance '\n",
            "                 'companies',\n",
            "                 'Working directly with c-level executives',\n",
            "                 'Communicating technical concepts to customers and internal '\n",
            "                 'stakeholders',\n",
            "                 'Leading high-visibility customer events (CAB, conferences, '\n",
            "                 'product launches, etc.)',\n",
            "                 'Gathering, distilling, and processing complex market '\n",
            "                 '(industry, competitor, customer, prospect) intelligence'],\n",
            "  'location': 'Tokyo, Japan',\n",
            "  'sub_division_of_organization': 'Go To Market'}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "chTqArp0YOje"
      },
      "source": [
        "## Step 6: Match Jobs to Resume\n",
        "\n",
        "Finally, let's use OpenAI's model to find the best matches for your resume:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F83lply4YOje",
        "outputId": "34a78acd-5a97-4afd-cf72-bc12bdc21dec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Raw model response:\n",
            "[]\n",
            "\n",
            "No job recommendations could be generated.\n"
          ]
        }
      ],
      "source": [
        "def get_job_recommendations(resume, jobs):\n",
        "    prompt = f\"\"\"\n",
        "You are a job matching assistant. Analyze the resume and job listings below, and return ONLY a JSON array containing the top 3 roles that best fit the candidate's experience and skills.\n",
        "\n",
        "The response must be a valid JSON array that can be parsed by json.loads(). Include only the job title, compensation, and apply link for each recommended role.\n",
        "\n",
        "Example format:\n",
        "[\n",
        "  {{\n",
        "    \"job_title\": \"Senior Software Engineer\",\n",
        "    \"compensation\": \"$150,000 - $180,000\",\n",
        "    \"apply_link\": \"https://example.com/jobs/123\"\n",
        "  }},\n",
        "  ...\n",
        "]\n",
        "\n",
        "Resume:\n",
        "{resume}\n",
        "\n",
        "Job Listings:\n",
        "{json.dumps(jobs, indent=2)}\n",
        "\n",
        "Remember: Respond ONLY with the JSON array, no additional text or explanations.\n",
        "\"\"\"\n",
        "\n",
        "    try:\n",
        "        completion = client.chat.completions.create(\n",
        "            model=\"gpt-4\",\n",
        "            messages=[{\n",
        "                \"role\": \"user\",\n",
        "                \"content\": prompt\n",
        "            }]\n",
        "        )\n",
        "\n",
        "        # Debug: print raw response content\n",
        "        response_content = completion.choices[0].message.content.strip()\n",
        "        print(\"\\nRaw model response:\")\n",
        "        print(response_content)\n",
        "\n",
        "        try:\n",
        "            parsed_response = json.loads(response_content)\n",
        "            return parsed_response\n",
        "        except json.JSONDecodeError as json_err:\n",
        "            print(f\"\\nJSON parsing error: {str(json_err)}\")\n",
        "            print(f\"Failed to parse content: {response_content}\")\n",
        "            return []\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"\\nError getting recommendations: {str(e)}\")\n",
        "        return []\n",
        "\n",
        "# Get and display recommendations\n",
        "recommended_jobs = get_job_recommendations(resume_paste, extracted_data)\n",
        "if recommended_jobs:\n",
        "    print(\"\\nTop 3 Recommended Jobs:\")\n",
        "    print(json.dumps(recommended_jobs, indent=2))\n",
        "else:\n",
        "    print(\"\\nNo job recommendations could be generated.\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pydantic import BaseModel\n",
        "from openai import OpenAI\n",
        "\n",
        "class JobRecommendation(BaseModel):\n",
        "    class Job(BaseModel):\n",
        "        job_title: str\n",
        "        compensation: str\n",
        "        apply_link: str\n",
        "\n",
        "    recommendations: list[Job]\n",
        "\n",
        "def get_job_recommendations(resume, jobs):\n",
        "    print(\"\\nDEBUG: Starting job recommendations\")\n",
        "    print(f\"DEBUG: Number of jobs to analyze: {len(jobs)}\")\n",
        "\n",
        "    prompt = f\"\"\"\n",
        "You are a job matching expert. Your task is to analyze a resume and available job listings to find the best matches.\n",
        "\n",
        "Instructions:\n",
        "1. Analyze the candidate's resume carefully\n",
        "2. Review each job listing's requirements and details\n",
        "3. Match the candidate's skills and experience with job requirements\n",
        "4. Always return at least one recommendation if there are any jobs available\n",
        "5. Use empty string for compensation if not specified in job listing\n",
        "6. Use the exact apply_link from the job listing\n",
        "\n",
        "Return your analysis as a valid JSON object with this exact structure:\n",
        "{{\n",
        "    \"recommendations\": [\n",
        "        {{\n",
        "            \"job_title\": \"exact job title from listing\",\n",
        "            \"compensation\": \"compensation from listing or empty string if not specified\",\n",
        "            \"apply_link\": \"exact apply link from listing\"\n",
        "        }}\n",
        "    ]\n",
        "}}\n",
        "\n",
        "Resume to analyze:\n",
        "{resume}\n",
        "\n",
        "Available job listings:\n",
        "{json.dumps(jobs, indent=2)}\n",
        "\n",
        "Remember: You must return at least one recommendation if any jobs are available, choosing the best match based on the candidate's qualifications.\n",
        "\"\"\"\n",
        "\n",
        "    try:\n",
        "        print(\"DEBUG: Making API call...\")\n",
        "        completion = client.chat.completions.create(\n",
        "            model=\"gpt-4\",\n",
        "            messages=[\n",
        "                {\n",
        "                    \"role\": \"system\",\n",
        "                    \"content\": \"You are a job matching expert that ALWAYS responds with valid JSON containing at least one job recommendation if jobs are available.\"\n",
        "                },\n",
        "                {\n",
        "                    \"role\": \"user\",\n",
        "                    \"content\": prompt\n",
        "                }\n",
        "            ]\n",
        "        )\n",
        "\n",
        "        print(\"DEBUG: Got API response\")\n",
        "        response_text = completion.choices[0].message.content\n",
        "        print(f\"DEBUG: Raw response:\\n{response_text}\")\n",
        "\n",
        "        try:\n",
        "            parsed = JobRecommendation.parse_raw(response_text)\n",
        "            print(f\"DEBUG: Successfully parsed response into {len(parsed.recommendations)} recommendations\")\n",
        "            return parsed\n",
        "        except Exception as parse_err:\n",
        "            print(f\"DEBUG: Error parsing response: {str(parse_err)}\")\n",
        "            raise\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"DEBUG: Error in job recommendations: {str(e)}\")\n",
        "        print(f\"DEBUG: Error type: {type(e)}\")\n",
        "        return JobRecommendation(recommendations=[])\n",
        "\n",
        "# Get and display recommendations\n",
        "print(\"\\nStarting recommendation process...\")\n",
        "recommended_jobs = get_job_recommendations(resume_paste, extracted_data)\n",
        "if recommended_jobs and recommended_jobs.recommendations:\n",
        "    print(\"\\nTop 3 Recommended Jobs:\")\n",
        "    for job in recommended_jobs.recommendations:\n",
        "        print(f\"\\nTitle: {job.job_title}\")\n",
        "        print(f\"Compensation: {job.compensation}\")\n",
        "        print(f\"Apply Link: {job.apply_link}\")\n",
        "else:\n",
        "    print(\"\\nNo recommendations were generated. Could you check if the resume content is provided and not empty?\")\n",
        "    print(f\"DEBUG: Resume content (first 100 chars): {resume_paste[:100] if resume_paste else 'Empty resume'}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vuXgLkL9b7H7",
        "outputId": "f63f663b-504f-4813-965a-5ef718d61682"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Starting recommendation process...\n",
            "\n",
            "DEBUG: Starting job recommendations\n",
            "DEBUG: Number of jobs to analyze: 2\n",
            "DEBUG: Making API call...\n",
            "DEBUG: Got API response\n",
            "DEBUG: Raw response:\n",
            "{\n",
            "    \"recommendations\": []\n",
            "}\n",
            "DEBUG: Successfully parsed response into 0 recommendations\n",
            "\n",
            "No recommendations were generated. Could you check if the resume content is provided and not empty?\n",
            "DEBUG: Resume content (first 100 chars):   # @param {type:\"string\"}\n",
            "**John Doe**  \n",
            "123 Main Street, Anytown, USA  \n",
            "(123) 456-7890 | john.doe@\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.0"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}