map_endpoint_quickstart.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alexfazio/firecrawl-quickstart/blob/main/firecrawl_map_endpoint_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeI-WY-jFSWx"
      },
      "source": [
        "# Map Endpoint Tutorial (Alpha)\n",
        "\n",
        "By Alex Fazio (https://x.com/alxfazio)\n",
        "\n",
        "Github repo: https://github.com/alexfazio/firecrawl-cookbook\n",
        "\n",
        "This notebook demonstrates how to use Firecrawl's new Map endpoint to efficiently transform a single URL into a comprehensive sitemap. By the end of this tutorial, you'll be able to:\n",
        "\n",
        "1. Set up the Firecrawl environment\n",
        "2. Transform a single URL into a complete sitemap\n",
        "3. Process and display the mapping results\n",
        "\n",
        "This cookbook is designed for developers who need to quickly gather and analyze website structures at scale."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xGtSCC9eFSWy"
      },
      "source": [
        "## Requirements\n",
        "\n",
        "Before proceeding, ensure you have the following:\n",
        "\n",
        "- **Firecrawl API key**: Essential for accessing the Firecrawl service\n",
        "\n",
        "The Map endpoint is extremely useful when you need to:\n",
        "- Quickly know the links on a website\n",
        "- Scrape pages of a website that are related to a specific topic (using the search parameter)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c95ZaBn4FSWy"
      },
      "source": [
        "## Setup\n",
        "\n",
        "First, let's set up our environment with the necessary imports and initializations:\n",
        "\n",
        "This notebook requires the following library:\n",
        "- `firecrawl-py`: For web mapping and URL extraction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nkREos9sFSWy",
        "outputId": "6ab338f3-f79c-4b31-bbb6-de77eba419a5"
      },
      "source": [
        "%pip -q install firecrawl-py"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/164.1 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m163.8/164.1 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m164.1/164.1 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "fsRoe6V2FSWy"
      },
      "source": [
        "from firecrawl.firecrawl import FirecrawlApp"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u94UIBRhFSWz"
      },
      "source": [
        "## Initialize Firecrawl\n",
        "\n",
        "Let's initialize our Firecrawl instance with your API key:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "-gjtssnpFSWz",
        "outputId": "a649a674-29c4-43e8-cd3b-729a59476241",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Initialize the app with your API key\n",
        "# Replace \"fc-YOUR_API_KEY\" with your actual API key\n",
        "from getpass import getpass\n",
        "\n",
        "# Securely get API keys\n",
        "firecrawl_api_key = getpass(\"Enter your Firecrawl API key: \")\n",
        "\n",
        "# Initialize FirecrawlApp\n",
        "app = FirecrawlApp(api_key=firecrawl_api_key)"
      ],
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your Firecrawl API key: ··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9kwQLnDBFSWz"
      },
      "source": [
        "## Using the Map Endpoint\n",
        "\n",
        "Now let's use the Map endpoint to generate a sitemap. The response will include a comprehensive list of URLs found on the website."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ad8pvQWaFSWz",
        "outputId": "f8dfd1e3-0ffe-4cce-c8c6-7a4dd16b4375"
      },
      "source": [
        "# Map a website\n",
        "map_result = app.map_url('https://techcrunch.com/category/startups/')\n",
        "print(map_result)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'success': True, 'links': ['https://techcrunch.com/category/startups', 'https://techcrunch.com/category/startups/page/113', 'https://techcrunch.com/category/startups/page/46', 'https://techcrunch.com/category/startups/page/114', 'https://techcrunch.com/category/startups/page/131', 'https://techcrunch.com/category/startups/page/258', 'https://techcrunch.com/category/startups/page/317', 'https://techcrunch.com/category/startups/page/21', 'https://techcrunch.com/category/startups/page/238', 'https://techcrunch.com/category/startups/page/35', 'https://techcrunch.com/category/startups/page/44', 'https://techcrunch.com/category/startups/page/90', 'https://techcrunch.com/category/startups/page/237', 'https://techcrunch.com/category/startups/page/226', 'https://techcrunch.com/category/startups/page/85', 'https://techcrunch.com/category/startups/page/25', 'https://techcrunch.com/category/startups/page/266', 'https://techcrunch.com/category/startups/page/38', 'https://techcrunch.com/category/startups/page/50', 'https://techcrunch.com/category/startups/page/139', 'https://techcrunch.com/category/startups/page/276', 'https://techcrunch.com/category/startups/page/127', 'https://techcrunch.com/category/startups/page/55', 'https://techcrunch.com/category/startups/page/26', 'https://techcrunch.com/category/startups/page/87', 'https://techcrunch.com/category/startups/page/20', 'https://techcrunch.com/category/startups/page/135', 'https://techcrunch.com/category/startups/page/126', 'https://techcrunch.com/category/startups/page/310', 'https://techcrunch.com/category/startups/page/319', 'https://techcrunch.com/category/startups/page/236', 'https://techcrunch.com/category/startups/page/150', 'https://techcrunch.com/category/startups/page/259', 'https://techcrunch.com/category/startups/page/111', 'https://techcrunch.com/category/startups/page/141', 'https://techcrunch.com/category/startups/page/22', 'https://techcrunch.com/category/startups/page/186', 'https://techcrunch.com/category/startups/page/311', 'https://techcrunch.com/category/startups/page/152', 'https://techcrunch.com/category/startups/page/240', 'https://techcrunch.com/category/startups/page/161', 'https://techcrunch.com/category/startups/page/34', 'https://techcrunch.com/category/startups/page/122', 'https://techcrunch.com/category/startups/page/177', 'https://techcrunch.com/category/startups/page/227', 'https://techcrunch.com/category/startups/page/224', 'https://techcrunch.com/category/startups/page/295', 'https://techcrunch.com/category/startups/page/68', 'https://techcrunch.com/category/startups/page/13', 'https://techcrunch.com/category/startups/page/250', 'https://techcrunch.com/category/startups/page/89', 'https://techcrunch.com/category/startups/page/31', 'https://techcrunch.com/category/startups/page/204', 'https://techcrunch.com/category/startups/page/318', 'https://techcrunch.com/category/startups/page/277', 'https://techcrunch.com/category/startups/page/175', 'https://techcrunch.com/category/startups/page/220', 'https://techcrunch.com/category/startups/page/63', 'https://techcrunch.com/category/startups/page/76', 'https://techcrunch.com/category/startups/page/305', 'https://techcrunch.com/category/startups/page/274', 'https://techcrunch.com/category/startups/page/329', 'https://techcrunch.com/category/startups/page/144', 'https://techcrunch.com/category/startups/page/321', 'https://techcrunch.com/category/startups/page/24', 'https://techcrunch.com/category/startups/page/200', 'https://techcrunch.com/category/startups/page/289', 'https://techcrunch.com/category/startups/page/243', 'https://techcrunch.com/category/startups/page/95', 'https://techcrunch.com/category/startups/page/163', 'https://techcrunch.com/category/startups/page/210', 'https://techcrunch.com/category/startups/page/211', 'https://techcrunch.com/category/startups/page/136', 'https://techcrunch.com/category/startups/page/51', 'https://techcrunch.com/category/startups/page/232', 'https://techcrunch.com/category/startups/page/205', 'https://techcrunch.com/category/startups/page/151', 'https://techcrunch.com/category/startups/page/101', 'https://techcrunch.com/category/startups/page/168', 'https://techcrunch.com/category/startups/page/159', 'https://techcrunch.com/category/startups/page/42', 'https://techcrunch.com/category/startups/page/79', 'https://techcrunch.com/category/startups/page/64', 'https://techcrunch.com/category/startups/page/198', 'https://techcrunch.com/category/startups/page/202', 'https://techcrunch.com/category/startups/page/188', 'https://techcrunch.com/category/startups/page/268', 'https://techcrunch.com/category/startups/page/74', 'https://techcrunch.com/category/startups/page/207', 'https://techcrunch.com/category/startups/page/190', 'https://techcrunch.com/category/startups/page/288', 'https://techcrunch.com/category/startups/page/172', 'https://techcrunch.com/category/startups/page/97', 'https://techcrunch.com/category/startups/page/267', 'https://techcrunch.com/category/startups/page/327', 'https://techcrunch.com/category/startups/page/138', 'https://techcrunch.com/category/startups/page/8', 'https://techcrunch.com/category/startups/page/60', 'https://techcrunch.com/category/startups/page/33', 'https://techcrunch.com/category/startups/page/12', 'https://techcrunch.com/category/startups/page/247', 'https://techcrunch.com/category/startups/page/324', 'https://techcrunch.com/category/startups/page/325', 'https://techcrunch.com/category/startups/page/57', 'https://techcrunch.com/category/startups/page/16', 'https://techcrunch.com/category/startups/page/11', 'https://techcrunch.com/category/startups/page/308', 'https://techcrunch.com/category/startups/page/252', 'https://techcrunch.com/category/startups/page/96', 'https://techcrunch.com/category/startups/page/71', 'https://techcrunch.com/category/startups/page/133', 'https://techcrunch.com/category/startups/page/146', 'https://techcrunch.com/category/startups/page/62', 'https://techcrunch.com/category/startups/page/307', 'https://techcrunch.com/category/startups/page/48', 'https://techcrunch.com/category/startups/page/108', 'https://techcrunch.com/category/startups/page/287', 'https://techcrunch.com/category/startups/page/199', 'https://techcrunch.com/category/startups/page/223', 'https://techcrunch.com/category/startups/page/320', 'https://techcrunch.com/category/startups/page/272', 'https://techcrunch.com/category/startups/page/88', 'https://techcrunch.com/category/startups/page/129', 'https://techcrunch.com/category/startups/page/56', 'https://techcrunch.com/category/startups/page/328', 'https://techcrunch.com/category/startups/page/3', 'https://techcrunch.com/category/startups/page/166', 'https://techcrunch.com/category/startups/page/263', 'https://techcrunch.com/category/startups/page/40', 'https://techcrunch.com/category/startups/page/18', 'https://techcrunch.com/category/startups/page/36', 'https://techcrunch.com/category/startups/page/249', 'https://techcrunch.com/category/startups/page/121', 'https://techcrunch.com/category/startups/page/75', 'https://techcrunch.com/category/startups/page/86', 'https://techcrunch.com/category/startups/page/242', 'https://techcrunch.com/category/startups/page/257', 'https://techcrunch.com/category/startups/page/82', 'https://techcrunch.com/category/startups/page/102', 'https://techcrunch.com/category/startups/page/99', 'https://techcrunch.com/category/startups/page/94', 'https://techcrunch.com/category/startups/page/103', 'https://techcrunch.com/category/startups/page/332', 'https://techcrunch.com/category/startups/page/109', 'https://techcrunch.com/category/startups/page/6', 'https://techcrunch.com/category/startups/page/278', 'https://techcrunch.com/category/startups/page/304', 'https://techcrunch.com/category/startups/page/155', 'https://techcrunch.com/category/startups/page/230', 'https://techcrunch.com/category/startups/page/10', 'https://techcrunch.com/category/startups/page/54', 'https://techcrunch.com/category/startups/page/285', 'https://techcrunch.com/category/startups/page/73', 'https://techcrunch.com/category/startups/page/170', 'https://techcrunch.com/category/startups/page/302', 'https://techcrunch.com/category/startups/page/72', 'https://techcrunch.com/category/startups/page/181', 'https://techcrunch.com/category/startups/page/98', 'https://techcrunch.com/category/startups/page/93', 'https://techcrunch.com/category/startups/page/191', 'https://techcrunch.com/category/startups/page/140', 'https://techcrunch.com/category/startups/page/158', 'https://techcrunch.com/category/startups/page/112', 'https://techcrunch.com/category/startups/page/9', 'https://techcrunch.com/category/startups/page/142', 'https://techcrunch.com/category/startups/page/261', 'https://techcrunch.com/category/startups/page/312', 'https://techcrunch.com/category/startups/page/194', 'https://techcrunch.com/category/startups/page/119', 'https://techcrunch.com/category/startups/page/222', 'https://techcrunch.com/category/startups/page/61', 'https://techcrunch.com/category/startups/page/255', 'https://techcrunch.com/category/startups/page/160', 'https://techcrunch.com/category/startups/page/100', 'https://techcrunch.com/category/startups/page/66', 'https://techcrunch.com/category/startups/page/264', 'https://techcrunch.com/category/startups/page/149', 'https://techcrunch.com/category/startups/page/19', 'https://techcrunch.com/category/startups/page/314', 'https://techcrunch.com/category/startups/page/43', 'https://techcrunch.com/category/startups/page/256', 'https://techcrunch.com/category/startups/page/273', 'https://techcrunch.com/category/startups/page/297', 'https://techcrunch.com/category/startups/page/134', 'https://techcrunch.com/category/startups/page/234', 'https://techcrunch.com/category/startups/page/323', 'https://techcrunch.com/category/startups/page/148', 'https://techcrunch.com/category/startups/page/39', 'https://techcrunch.com/category/startups/page/23', 'https://techcrunch.com/category/startups/page/173', 'https://techcrunch.com/category/startups/page/284', 'https://techcrunch.com/category/startups/page/178', 'https://techcrunch.com/category/startups/page/7', 'https://techcrunch.com/category/startups/page/116', 'https://techcrunch.com/category/startups/page/156', 'https://techcrunch.com/category/startups/page/330', 'https://techcrunch.com/category/startups/page/137', 'https://techcrunch.com/category/startups/page/107', 'https://techcrunch.com/category/startups/page/291', 'https://techcrunch.com/category/startups/page/2', 'https://techcrunch.com/category/startups/page/315', 'https://techcrunch.com/category/startups/page/83', 'https://techcrunch.com/category/startups/page/92', 'https://techcrunch.com/category/startups/page/195', 'https://techcrunch.com/category/startups/page/28', 'https://techcrunch.com/category/startups/page/162', 'https://techcrunch.com/category/startups/page/118', 'https://techcrunch.com/category/startups/page/293', 'https://techcrunch.com/category/startups/page/244', 'https://techcrunch.com/category/startups/page/143', 'https://techcrunch.com/category/startups/page/153', 'https://techcrunch.com/category/startups/page/104', 'https://techcrunch.com/category/startups/page/130', 'https://techcrunch.com/category/startups/page/262', 'https://techcrunch.com/category/startups/page/14', 'https://techcrunch.com/category/startups/page/59', 'https://techcrunch.com/category/startups/page/132', 'https://techcrunch.com/category/startups/page/296', 'https://techcrunch.com/category/startups/page/91', 'https://techcrunch.com/category/startups/page/298', 'https://techcrunch.com/category/startups/page/167', 'https://techcrunch.com/category/startups/page/105', 'https://techcrunch.com/category/startups/page/58', 'https://techcrunch.com/category/startups/page/209', 'https://techcrunch.com/category/startups/page/294', 'https://techcrunch.com/category/startups/page/283', 'https://techcrunch.com/category/startups/page/78', 'https://techcrunch.com/category/startups/page/254', 'https://techcrunch.com/category/startups/page/180', 'https://techcrunch.com/category/startups/page/77/', 'https://techcrunch.com/category/startups/page/233/', 'https://techcrunch.com/category/startups/page/281/', 'https://techcrunch.com/category/startups/page/226/', 'https://techcrunch.com/category/startups/page/218/', 'https://techcrunch.com/category/startups/page/35/', 'https://techcrunch.com/category/startups/page/55/', 'https://techcrunch.com/category/startups/page/168/', 'https://techcrunch.com/category/startups/page/157/', 'https://techcrunch.com/category/startups/page/88/', 'https://techcrunch.com/category/startups/page/235/']}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-xMYbPODFSWz"
      },
      "source": [
        "## Processing the Results\n",
        "\n",
        "Let's process and display the results in a more readable format:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aG_vG7lCFSWz",
        "outputId": "22b6bdde-eca4-445a-c682-cf906c84ca1f"
      },
      "source": [
        "import pprint\n",
        "\n",
        "pprint.pprint(map_result)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'links': ['https://techcrunch.com/category/startups',\n",
            "           'https://techcrunch.com/category/startups/page/113',\n",
            "           'https://techcrunch.com/category/startups/page/46',\n",
            "           'https://techcrunch.com/category/startups/page/114',\n",
            "           'https://techcrunch.com/category/startups/page/131',\n",
            "           'https://techcrunch.com/category/startups/page/258',\n",
            "           'https://techcrunch.com/category/startups/page/317',\n",
            "           'https://techcrunch.com/category/startups/page/21',\n",
            "           'https://techcrunch.com/category/startups/page/238',\n",
            "           'https://techcrunch.com/category/startups/page/35',\n",
            "           'https://techcrunch.com/category/startups/page/44',\n",
            "           'https://techcrunch.com/category/startups/page/90',\n",
            "           'https://techcrunch.com/category/startups/page/237',\n",
            "           'https://techcrunch.com/category/startups/page/226',\n",
            "           'https://techcrunch.com/category/startups/page/85',\n",
            "           'https://techcrunch.com/category/startups/page/25',\n",
            "           'https://techcrunch.com/category/startups/page/266',\n",
            "           'https://techcrunch.com/category/startups/page/38',\n",
            "           'https://techcrunch.com/category/startups/page/50',\n",
            "           'https://techcrunch.com/category/startups/page/139',\n",
            "           'https://techcrunch.com/category/startups/page/276',\n",
            "           'https://techcrunch.com/category/startups/page/127',\n",
            "           'https://techcrunch.com/category/startups/page/55',\n",
            "           'https://techcrunch.com/category/startups/page/26',\n",
            "           'https://techcrunch.com/category/startups/page/87',\n",
            "           'https://techcrunch.com/category/startups/page/20',\n",
            "           'https://techcrunch.com/category/startups/page/135',\n",
            "           'https://techcrunch.com/category/startups/page/126',\n",
            "           'https://techcrunch.com/category/startups/page/310',\n",
            "           'https://techcrunch.com/category/startups/page/319',\n",
            "           'https://techcrunch.com/category/startups/page/236',\n",
            "           'https://techcrunch.com/category/startups/page/150',\n",
            "           'https://techcrunch.com/category/startups/page/259',\n",
            "           'https://techcrunch.com/category/startups/page/111',\n",
            "           'https://techcrunch.com/category/startups/page/141',\n",
            "           'https://techcrunch.com/category/startups/page/22',\n",
            "           'https://techcrunch.com/category/startups/page/186',\n",
            "           'https://techcrunch.com/category/startups/page/311',\n",
            "           'https://techcrunch.com/category/startups/page/152',\n",
            "           'https://techcrunch.com/category/startups/page/240',\n",
            "           'https://techcrunch.com/category/startups/page/161',\n",
            "           'https://techcrunch.com/category/startups/page/34',\n",
            "           'https://techcrunch.com/category/startups/page/122',\n",
            "           'https://techcrunch.com/category/startups/page/177',\n",
            "           'https://techcrunch.com/category/startups/page/227',\n",
            "           'https://techcrunch.com/category/startups/page/224',\n",
            "           'https://techcrunch.com/category/startups/page/295',\n",
            "           'https://techcrunch.com/category/startups/page/68',\n",
            "           'https://techcrunch.com/category/startups/page/13',\n",
            "           'https://techcrunch.com/category/startups/page/250',\n",
            "           'https://techcrunch.com/category/startups/page/89',\n",
            "           'https://techcrunch.com/category/startups/page/31',\n",
            "           'https://techcrunch.com/category/startups/page/204',\n",
            "           'https://techcrunch.com/category/startups/page/318',\n",
            "           'https://techcrunch.com/category/startups/page/277',\n",
            "           'https://techcrunch.com/category/startups/page/175',\n",
            "           'https://techcrunch.com/category/startups/page/220',\n",
            "           'https://techcrunch.com/category/startups/page/63',\n",
            "           'https://techcrunch.com/category/startups/page/76',\n",
            "           'https://techcrunch.com/category/startups/page/305',\n",
            "           'https://techcrunch.com/category/startups/page/274',\n",
            "           'https://techcrunch.com/category/startups/page/329',\n",
            "           'https://techcrunch.com/category/startups/page/144',\n",
            "           'https://techcrunch.com/category/startups/page/321',\n",
            "           'https://techcrunch.com/category/startups/page/24',\n",
            "           'https://techcrunch.com/category/startups/page/200',\n",
            "           'https://techcrunch.com/category/startups/page/289',\n",
            "           'https://techcrunch.com/category/startups/page/243',\n",
            "           'https://techcrunch.com/category/startups/page/95',\n",
            "           'https://techcrunch.com/category/startups/page/163',\n",
            "           'https://techcrunch.com/category/startups/page/210',\n",
            "           'https://techcrunch.com/category/startups/page/211',\n",
            "           'https://techcrunch.com/category/startups/page/136',\n",
            "           'https://techcrunch.com/category/startups/page/51',\n",
            "           'https://techcrunch.com/category/startups/page/232',\n",
            "           'https://techcrunch.com/category/startups/page/205',\n",
            "           'https://techcrunch.com/category/startups/page/151',\n",
            "           'https://techcrunch.com/category/startups/page/101',\n",
            "           'https://techcrunch.com/category/startups/page/168',\n",
            "           'https://techcrunch.com/category/startups/page/159',\n",
            "           'https://techcrunch.com/category/startups/page/42',\n",
            "           'https://techcrunch.com/category/startups/page/79',\n",
            "           'https://techcrunch.com/category/startups/page/64',\n",
            "           'https://techcrunch.com/category/startups/page/198',\n",
            "           'https://techcrunch.com/category/startups/page/202',\n",
            "           'https://techcrunch.com/category/startups/page/188',\n",
            "           'https://techcrunch.com/category/startups/page/268',\n",
            "           'https://techcrunch.com/category/startups/page/74',\n",
            "           'https://techcrunch.com/category/startups/page/207',\n",
            "           'https://techcrunch.com/category/startups/page/190',\n",
            "           'https://techcrunch.com/category/startups/page/288',\n",
            "           'https://techcrunch.com/category/startups/page/172',\n",
            "           'https://techcrunch.com/category/startups/page/97',\n",
            "           'https://techcrunch.com/category/startups/page/267',\n",
            "           'https://techcrunch.com/category/startups/page/327',\n",
            "           'https://techcrunch.com/category/startups/page/138',\n",
            "           'https://techcrunch.com/category/startups/page/8',\n",
            "           'https://techcrunch.com/category/startups/page/60',\n",
            "           'https://techcrunch.com/category/startups/page/33',\n",
            "           'https://techcrunch.com/category/startups/page/12',\n",
            "           'https://techcrunch.com/category/startups/page/247',\n",
            "           'https://techcrunch.com/category/startups/page/324',\n",
            "           'https://techcrunch.com/category/startups/page/325',\n",
            "           'https://techcrunch.com/category/startups/page/57',\n",
            "           'https://techcrunch.com/category/startups/page/16',\n",
            "           'https://techcrunch.com/category/startups/page/11',\n",
            "           'https://techcrunch.com/category/startups/page/308',\n",
            "           'https://techcrunch.com/category/startups/page/252',\n",
            "           'https://techcrunch.com/category/startups/page/96',\n",
            "           'https://techcrunch.com/category/startups/page/71',\n",
            "           'https://techcrunch.com/category/startups/page/133',\n",
            "           'https://techcrunch.com/category/startups/page/146',\n",
            "           'https://techcrunch.com/category/startups/page/62',\n",
            "           'https://techcrunch.com/category/startups/page/307',\n",
            "           'https://techcrunch.com/category/startups/page/48',\n",
            "           'https://techcrunch.com/category/startups/page/108',\n",
            "           'https://techcrunch.com/category/startups/page/287',\n",
            "           'https://techcrunch.com/category/startups/page/199',\n",
            "           'https://techcrunch.com/category/startups/page/223',\n",
            "           'https://techcrunch.com/category/startups/page/320',\n",
            "           'https://techcrunch.com/category/startups/page/272',\n",
            "           'https://techcrunch.com/category/startups/page/88',\n",
            "           'https://techcrunch.com/category/startups/page/129',\n",
            "           'https://techcrunch.com/category/startups/page/56',\n",
            "           'https://techcrunch.com/category/startups/page/328',\n",
            "           'https://techcrunch.com/category/startups/page/3',\n",
            "           'https://techcrunch.com/category/startups/page/166',\n",
            "           'https://techcrunch.com/category/startups/page/263',\n",
            "           'https://techcrunch.com/category/startups/page/40',\n",
            "           'https://techcrunch.com/category/startups/page/18',\n",
            "           'https://techcrunch.com/category/startups/page/36',\n",
            "           'https://techcrunch.com/category/startups/page/249',\n",
            "           'https://techcrunch.com/category/startups/page/121',\n",
            "           'https://techcrunch.com/category/startups/page/75',\n",
            "           'https://techcrunch.com/category/startups/page/86',\n",
            "           'https://techcrunch.com/category/startups/page/242',\n",
            "           'https://techcrunch.com/category/startups/page/257',\n",
            "           'https://techcrunch.com/category/startups/page/82',\n",
            "           'https://techcrunch.com/category/startups/page/102',\n",
            "           'https://techcrunch.com/category/startups/page/99',\n",
            "           'https://techcrunch.com/category/startups/page/94',\n",
            "           'https://techcrunch.com/category/startups/page/103',\n",
            "           'https://techcrunch.com/category/startups/page/332',\n",
            "           'https://techcrunch.com/category/startups/page/109',\n",
            "           'https://techcrunch.com/category/startups/page/6',\n",
            "           'https://techcrunch.com/category/startups/page/278',\n",
            "           'https://techcrunch.com/category/startups/page/304',\n",
            "           'https://techcrunch.com/category/startups/page/155',\n",
            "           'https://techcrunch.com/category/startups/page/230',\n",
            "           'https://techcrunch.com/category/startups/page/10',\n",
            "           'https://techcrunch.com/category/startups/page/54',\n",
            "           'https://techcrunch.com/category/startups/page/285',\n",
            "           'https://techcrunch.com/category/startups/page/73',\n",
            "           'https://techcrunch.com/category/startups/page/170',\n",
            "           'https://techcrunch.com/category/startups/page/302',\n",
            "           'https://techcrunch.com/category/startups/page/72',\n",
            "           'https://techcrunch.com/category/startups/page/181',\n",
            "           'https://techcrunch.com/category/startups/page/98',\n",
            "           'https://techcrunch.com/category/startups/page/93',\n",
            "           'https://techcrunch.com/category/startups/page/191',\n",
            "           'https://techcrunch.com/category/startups/page/140',\n",
            "           'https://techcrunch.com/category/startups/page/158',\n",
            "           'https://techcrunch.com/category/startups/page/112',\n",
            "           'https://techcrunch.com/category/startups/page/9',\n",
            "           'https://techcrunch.com/category/startups/page/142',\n",
            "           'https://techcrunch.com/category/startups/page/261',\n",
            "           'https://techcrunch.com/category/startups/page/312',\n",
            "           'https://techcrunch.com/category/startups/page/194',\n",
            "           'https://techcrunch.com/category/startups/page/119',\n",
            "           'https://techcrunch.com/category/startups/page/222',\n",
            "           'https://techcrunch.com/category/startups/page/61',\n",
            "           'https://techcrunch.com/category/startups/page/255',\n",
            "           'https://techcrunch.com/category/startups/page/160',\n",
            "           'https://techcrunch.com/category/startups/page/100',\n",
            "           'https://techcrunch.com/category/startups/page/66',\n",
            "           'https://techcrunch.com/category/startups/page/264',\n",
            "           'https://techcrunch.com/category/startups/page/149',\n",
            "           'https://techcrunch.com/category/startups/page/19',\n",
            "           'https://techcrunch.com/category/startups/page/314',\n",
            "           'https://techcrunch.com/category/startups/page/43',\n",
            "           'https://techcrunch.com/category/startups/page/256',\n",
            "           'https://techcrunch.com/category/startups/page/273',\n",
            "           'https://techcrunch.com/category/startups/page/297',\n",
            "           'https://techcrunch.com/category/startups/page/134',\n",
            "           'https://techcrunch.com/category/startups/page/234',\n",
            "           'https://techcrunch.com/category/startups/page/323',\n",
            "           'https://techcrunch.com/category/startups/page/148',\n",
            "           'https://techcrunch.com/category/startups/page/39',\n",
            "           'https://techcrunch.com/category/startups/page/23',\n",
            "           'https://techcrunch.com/category/startups/page/173',\n",
            "           'https://techcrunch.com/category/startups/page/284',\n",
            "           'https://techcrunch.com/category/startups/page/178',\n",
            "           'https://techcrunch.com/category/startups/page/7',\n",
            "           'https://techcrunch.com/category/startups/page/116',\n",
            "           'https://techcrunch.com/category/startups/page/156',\n",
            "           'https://techcrunch.com/category/startups/page/330',\n",
            "           'https://techcrunch.com/category/startups/page/137',\n",
            "           'https://techcrunch.com/category/startups/page/107',\n",
            "           'https://techcrunch.com/category/startups/page/291',\n",
            "           'https://techcrunch.com/category/startups/page/2',\n",
            "           'https://techcrunch.com/category/startups/page/315',\n",
            "           'https://techcrunch.com/category/startups/page/83',\n",
            "           'https://techcrunch.com/category/startups/page/92',\n",
            "           'https://techcrunch.com/category/startups/page/195',\n",
            "           'https://techcrunch.com/category/startups/page/28',\n",
            "           'https://techcrunch.com/category/startups/page/162',\n",
            "           'https://techcrunch.com/category/startups/page/118',\n",
            "           'https://techcrunch.com/category/startups/page/293',\n",
            "           'https://techcrunch.com/category/startups/page/244',\n",
            "           'https://techcrunch.com/category/startups/page/143',\n",
            "           'https://techcrunch.com/category/startups/page/153',\n",
            "           'https://techcrunch.com/category/startups/page/104',\n",
            "           'https://techcrunch.com/category/startups/page/130',\n",
            "           'https://techcrunch.com/category/startups/page/262',\n",
            "           'https://techcrunch.com/category/startups/page/14',\n",
            "           'https://techcrunch.com/category/startups/page/59',\n",
            "           'https://techcrunch.com/category/startups/page/132',\n",
            "           'https://techcrunch.com/category/startups/page/296',\n",
            "           'https://techcrunch.com/category/startups/page/91',\n",
            "           'https://techcrunch.com/category/startups/page/298',\n",
            "           'https://techcrunch.com/category/startups/page/167',\n",
            "           'https://techcrunch.com/category/startups/page/105',\n",
            "           'https://techcrunch.com/category/startups/page/58',\n",
            "           'https://techcrunch.com/category/startups/page/209',\n",
            "           'https://techcrunch.com/category/startups/page/294',\n",
            "           'https://techcrunch.com/category/startups/page/283',\n",
            "           'https://techcrunch.com/category/startups/page/78',\n",
            "           'https://techcrunch.com/category/startups/page/254',\n",
            "           'https://techcrunch.com/category/startups/page/180',\n",
            "           'https://techcrunch.com/category/startups/page/77/',\n",
            "           'https://techcrunch.com/category/startups/page/233/',\n",
            "           'https://techcrunch.com/category/startups/page/281/',\n",
            "           'https://techcrunch.com/category/startups/page/226/',\n",
            "           'https://techcrunch.com/category/startups/page/218/',\n",
            "           'https://techcrunch.com/category/startups/page/35/',\n",
            "           'https://techcrunch.com/category/startups/page/55/',\n",
            "           'https://techcrunch.com/category/startups/page/168/',\n",
            "           'https://techcrunch.com/category/startups/page/157/',\n",
            "           'https://techcrunch.com/category/startups/page/88/',\n",
            "           'https://techcrunch.com/category/startups/page/235/'],\n",
            " 'success': True}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y26qr7_VFSWz"
      },
      "source": [
        "## What's Next?\n",
        "\n",
        "The Map endpoint is just the beginning! This powerful feature opens up exciting possibilities for future integrations and advanced web scraping projects. Some potential use cases include:\n",
        "\n",
        "- Automated site structure analysis\n",
        "- Content discovery and mapping\n",
        "- SEO analysis and optimization\n",
        "- Custom web crawling solutions"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.0"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}