Forward-propagation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e161390e-1a35-4eab-b1b6-7e991078df6b",
   "metadata": {},
   "source": [
    "# Intro\n",
    "\n",
    "Hey Everyone, In this notebook i am gonna show you how Neural Networks actually work. I will simplify all the Jargon and the complicated words.\n",
    "\n",
    " We will go step by step and at the end we will summarize everything and by the end of this notebook you will be able to perform a forward propagation step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "994b700b-c67b-4578-8e18-1d988ce51844",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30aab340-4f6f-4450-95d5-3cfd182ef166",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4995268f-de27-44ad-9351-40f9e780a88c",
   "metadata": {},
   "source": [
    "## Step 1: Define the Structure of the Network\r\n",
    "\r\n",
    "We will build a simple neural network with:\r\n",
    "- 2 neurons in the input layer.\r\n",
    "- 2 neurons in the hidden layer.\r\n",
    "- 1 neuron in the output layer.\r\n",
    "\r\n",
    "This is the most basic structure to understand how a neural network functions, using just plain Python without any external libraries.\r\n",
    "\r\n",
    "We'll start by initializing the weights and biases for each layer.\r\n",
    "like gradient descent.\r\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4a25c1c-8f75-4209-81bc-aabb829fd3ab",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3c433484-c73d-49e5-84db-12bfd9547f6b",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5024a860-c765-485c-9aba-699a76846b3a",
   "metadata": {},
   "source": [
    "## Step 2: Initialize Weights and Biases\n",
    "\n",
    "### What are Weights?\n",
    "\n",
    "Weights are the most important parameters in a neural network. They determine how much influence each input will have on the final prediction of the network. Each connection between neurons is associated with a weight, and this weight tells the network how important that connection is.\n",
    "\n",
    "#### Example:\n",
    "In our network, the weights between the input layer and the hidden layer are represented as follows:\n",
    "\n",
    "`weights_input_to_hidden = [[0.15, 0.2], [0.25, 0.3]]`\n",
    "\n",
    "This means:\n",
    "- `0.15` is the weight between the first input neuron and the first hidden neuron.\n",
    "- `0.25` is the weight between the second input neuron and the first hidden neuron.\n",
    "\n",
    "The weights adjust the strength of the input signals before they reach the next layer.\n",
    "\n",
    "### What are Biases?\n",
    "\n",
    "Biases are additional parameters that help the network adjust its outputs. They are added to the weighted sum of inputs before passing the value to the activation function. Biases allow the network to make predictions even when the input is zero.\n",
    "\n",
    "#### Example:\n",
    "We define the biases for the hidden layer like this:\n",
    "\n",
    "`bias_hidden = [0.35, 0.35]`\n",
    "\n",
    "This means each hidden neuron will add 0.35 to its total input before passing it to the activation function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bdef61c3-b202-4813-89a3-3a9a055036be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code for weights and biases\n",
    "weights_input_to_hidden = [[0.15, 0.2],  # Weights between input and hidden layer\n",
    "                           [0.25, 0.3]]  # Each value corresponds to the strength of the connection between neurons\n",
    "\n",
    "bias_hidden = [0.35, 0.35]  # Biases for the two neurons in the hidden layer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b40ed9d0-7ba6-4f2e-b96e-c5f5cdffbd6b",
   "metadata": {},
   "source": [
    "## Step 3: Activation Function\n",
    "\n",
    "An activation function is a mathematical function applied to the output of each neuron in the network. Its purpose is to introduce non-linearity into the network, enabling the model to learn and approximate more complex functions.\n",
    "\n",
    "Without an activation function, the network would just perform linear transformations, which limits its ability to model complex relationships in the data.\n",
    "\n",
    "### Sigmoid Activation Function\n",
    "\n",
    "The sigmoid function is one of the most commonly used activation functions. It \"squashes\" the input value into a range between 0 and 1, making it ideal for problems where the output needs to represent a probability.\n",
    "\n",
    "The sigmoid function is defined as:\n",
    "\n",
    "\\[\n",
    "sigmoid(x) = \\frac{1}{1 + e^{-x}}\n",
    "\\]\n",
    "\n",
    "Where `e` is Euler's number (approximately 2.718). The sigmoid function is a smooth curve that takes large negative inputs close to 0 and large positive inputs close to 1. This behavior is useful for binary classification tasks.\n",
    "\n",
    "### Why Do We Need Activation Functions?\n",
    "\n",
    "If we didn’t use activation functions, the network would only perform linear transformations on the inputs. This would prevent the network from learning complex, non-linear relationships in the data. The sigmoid function, by introducing non-linearity, allows the network to learn complex patterns and make better predictions.\n",
    "\n",
    "### Example:\n",
    "\n",
    "In our network, after calculating the weighted sum plus bias for each neuron, we pass the result through the sigmoid activation function to produce the output.\n",
    "\n",
    "For example, for a hidden neuron, the calculation would be:\n",
    "\n",
    "\\[\n",
    "hidden\\_output = sigmoid((input_1 \\times 0.15) + (input_2 \\times 0.25) + 0.35)\n",
    "\\]\n",
    "\n",
    "This produces an output between 0 and 1, depending on the input sum.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cb855309-66e5-49b6-94ac-ce616bf36438",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code for sigmoid activation function\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + 2.718 ** -x)  # Sigmoid function implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d997ddfd-215e-445c-b0b7-abd21ba91c06",
   "metadata": {},
   "source": [
    "## Step 4: Feedforward Process\n",
    "\n",
    "In the feedforward process, the input data passes through the network layer by layer. Each neuron in a layer performs the following operations:\n",
    "1. **Weighted Sum**: The input values are multiplied by their corresponding weights.\n",
    "2. **Add Bias**: A bias is added to the weighted sum.\n",
    "3. **Activation Function**: The activation function is applied to the result.\n",
    "\n",
    "This process starts from the input layer, passes through the hidden layers, and finally reaches the output layer.\n",
    "\n",
    "### Example:\n",
    "\n",
    "Let’s consider our inputs as `[4, 3]`. To calculate the value of the first hidden neuron, we multiply each input by its corresponding weight, add the bias, and then apply the sigmoid activation function.\n",
    "\n",
    "For example:\n",
    "\n",
    "\\[\n",
    "hidden\\_neuron\\_1\\_input = (4 \\times 0.15) + (3 \\times 0.25) + 0.35 = 1.75\n",
    "\\]\n",
    "\\[\n",
    "hidden\\_neuron\\_1\\_output = sigmoid(1.75) \\approx 0.851\n",
    "\\]\n",
    "\n",
    "The same process is repeated for all neurons in the hidden layer.\n",
    "\n",
    "The hidden layer outputs are then passed to the output layer, where the same operations (weighted sum, add bias, apply activation function) are performed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6a59cd0b-ca97-4963-9ff4-78ade636e36a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hidden layer outputs: [0.845511712519507, 0.8859261396850433]\n"
     ]
    }
   ],
   "source": [
    "# Code for feedforward process\n",
    "\n",
    "inputs = [4, 3]\n",
    "\n",
    "# Calculate the value at the hidden layer neurons\n",
    "hidden_layer = []\n",
    "for i in range(2):  # We have 2 hidden neurons\n",
    "    neuron_input = (inputs[0] * weights_input_to_hidden[0][i]) + (inputs[1] * weights_input_to_hidden[1][i]) + bias_hidden[i]\n",
    "    hidden_layer_output = sigmoid(neuron_input)\n",
    "    hidden_layer.append(hidden_layer_output)\n",
    "\n",
    "print(\"Hidden layer outputs:\", hidden_layer)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e59eac37-2d8b-4c02-aab4-01706ac6037f",
   "metadata": {},
   "source": [
    "## Step 5: Calculating Output\n",
    "\n",
    "After processing the inputs through the hidden layer, the final step is to calculate the network’s output. The hidden layer outputs become the inputs for the output neuron. This neuron performs the same steps as before:\n",
    "1. **Weighted Sum**: The hidden layer outputs are multiplied by their corresponding weights for the output layer.\n",
    "2. **Add Bias**: The output neuron’s bias is added to the sum.\n",
    "3. **Activation Function**: The sigmoid function is applied to produce the final output.\n",
    "\n",
    "### Example:\n",
    "\n",
    "Let’s assume that the hidden layer outputs are `[0.851, 0.8]`. These values are used as inputs for the output neuron.\n",
    "\n",
    "\\[\n",
    "output\\_neuron\\_input = (0.851 \\times 0.4) + (0.8 \\times 0.45) + 0.6 \\approx 1.39\n",
    "\\]\n",
    "\n",
    "Finally, the sigmoid function is applied:\n",
    "\n",
    "\\[\n",
    "final\\_output = sigmoid(1.39) \\approx 0.800\n",
    "\\]\n",
    "\n",
    "This final value represents the prediction made by the network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6caaeb66-8509-437d-bb5c-571b114d1a7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final output of the network: 0.7919521439569785\n"
     ]
    }
   ],
   "source": [
    "# Code for calculating the output\n",
    "\n",
    "output = 0  # Initialize output\n",
    "\n",
    "# Calculate the output layer neuron value based on the hidden layer outputs\n",
    "weights_hidden_to_output = [0.4, 0.45]  # Weights from hidden to output layer\n",
    "bias_output = 0.6  # Bias for output neuron\n",
    "\n",
    "for i in range(2):\n",
    "    output += hidden_layer[i] * weights_hidden_to_output[i]\n",
    "\n",
    "output += bias_output  # Add the bias for the output neuron\n",
    "final_output = sigmoid(output)\n",
    "\n",
    "print(\"Final output of the network:\", final_output)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a83a5de-fc3f-4e48-95ff-97eaf98e9235",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ecf5e59-bd43-4d08-88d7-aa82029d1515",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9c949a0d-07e3-4206-b864-3767865bd126",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84da6064-e026-4f08-94ad-fa7df9f0bc05",
   "metadata": {},
   "source": [
    "## Complete Explanation of Forward Propagation in Neural Networks\n",
    "\n",
    "### What is a Neural Network?\n",
    "\n",
    "At its core, a **neural network** is a computational model designed to mimic the way the human brain processes information. It is composed of layers of nodes called **neurons**. These neurons are connected by **weights** that represent the strength of connections between them. The network is organized into three main layers:\n",
    "1. **Input Layer**: This layer receives the initial data.\n",
    "2. **Hidden Layers**: These layers process the input data using weights and biases.\n",
    "3. **Output Layer**: This layer produces the final prediction or classification based on the processed information.\n",
    "\n",
    "Each neuron receives inputs, processes them, and then passes an output to the next layer in the network. The process of passing information forward through the network is called **forward propagation**.\n",
    "\n",
    "### The Forward Propagation Process\n",
    "\n",
    "In forward propagation, information flows through the network from the input layer to the output layer. The neurons in each layer perform three key operations:\n",
    "1. **Weighted Sum**: The inputs are multiplied by their corresponding weights, which determine how much influence each input has on the next neuron.\n",
    "2. **Add Bias**: A bias term is added to the weighted sum to shift the output of the neuron. Biases allow the network to adjust its predictions even when the inputs are zero.\n",
    "3. **Activation Function**: The weighted sum and bias are passed through an activation function, which introduces non-linearity into the network. This is crucial for modeling complex patterns in data.\n",
    "\n",
    "Let’s break down these steps in detail.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 1: Weighted Sum\n",
    "\n",
    "When an input is passed into the neural network, it first encounters a set of **weights**. Weights are the core parameters of the network. They control the strength of the connections between neurons. You can think of weights as filters that decide how important each input is to the network.\n",
    "\n",
    "**Mathematical Explanation**:\n",
    "If we have two inputs, \\( x_1 \\) and \\( x_2 \\), and their corresponding weights \\( w_1 \\) and \\( w_2 \\), the weighted sum for a neuron would be:\n",
    "\n",
    "\\[\n",
    "weighted\\_sum = (x_1 \\times w_1) + (x_2 \\times w_2)\n",
    "\\]\n",
    "\n",
    "The weights determine how much emphasis each input has in the final decision. Larger weights increase the impact of the corresponding input, while smaller weights reduce it.\n",
    "\n",
    "**Example**:\n",
    "Consider an input layer with two neurons, and let’s assume the inputs are 4 and 3, and the weights connecting the input neurons to a hidden neuron are 0.15 and 0.25, respectively. The weighted sum for the first hidden neuron would be:\n",
    "\n",
    "\\[\n",
    "weighted\\_sum = (4 \\times 0.15) + (3 \\times 0.25) = 0.6 + 0.75 = 1.35\n",
    "\\]\n",
    "\n",
    "This is the raw input to the neuron before the bias and activation function are applied.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 2: Adding the Bias\n",
    "\n",
    "In addition to the weighted sum, neural networks use a **bias** term. Biases are constants that are added to the weighted sum before passing it through the activation function. Biases allow the network to make predictions even when all inputs are zero. Essentially, the bias shifts the output of the activation function, helping the network fit the data more flexibly.\n",
    "\n",
    "**Mathematical Explanation**:\n",
    "If we have a bias term \\( b \\), the equation for the neuron’s input becomes:\n",
    "\n",
    "\\[\n",
    "neuron\\_input = weighted\\_sum + b\n",
    "\\]\n",
    "\n",
    "**Example**:\n",
    "Continuing from the previous example, let’s say the bias for the first hidden neuron is 0.35. The neuron’s input would be:\n",
    "\n",
    "\\[\n",
    "neuron\\_input = 1.35 + 0.35 = 1.7\n",
    "\\]\n",
    "\n",
    "This adjusted input is then passed through the activation function.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 3: Activation Function\n",
    "\n",
    "The **activation function** is a mathematical function applied to the neuron’s input. Its role is to introduce **non-linearity** into the network, which allows the model to learn more complex patterns in the data. Without an activation function, the network would behave like a linear model, and no matter how many layers it had, it wouldn’t be able to model non-linear relationships (which are crucial for tasks like image recognition or natural language processing).\n",
    "\n",
    "There are several types of activation functions, but for simplicity, we’ll focus on the **sigmoid** activation function, which is commonly used in binary classification problems.\n",
    "\n",
    "**Sigmoid Activation Function**:\n",
    "The sigmoid function takes the neuron’s input and squashes it into a value between 0 and 1. It is defined as:\n",
    "\n",
    "\\[\n",
    "sigmoid(x) = \\frac{1}{1 + e^{-x}}\n",
    "\\]\n",
    "\n",
    "Where \\( e \\) is Euler’s number (approximately 2.718). The sigmoid function is ideal when we want to interpret the output as a probability, such as when predicting whether an image contains a cat (1) or not (0).\n",
    "\n",
    "**Example**:\n",
    "For the first hidden neuron, the input is 1.7. The output of the sigmoid function would be:\n",
    "\n",
    "\\[\n",
    "neuron\\_output = sigmoid(1.7) = \\frac{1}{1 + e^{-1.7}} \\approx 0.845\n",
    "\\]\n",
    "\n",
    "The neuron’s output is now 0.845, which will be passed to the next layer of the network.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 4: Passing Information to the Output Layer\n",
    "\n",
    "Once the hidden layer has processed the inputs, it produces outputs (in this case, 0.845 and another value from the second hidden neuron). These outputs are then passed to the **output layer**, where the same steps are repeated: weighted sum, add bias, and apply activation function.\n",
    "\n",
    "For simplicity, let’s assume the output layer has only one neuron, and its task is to predict whether the input data belongs to a particular class (binary classification).\n",
    "\n",
    "**Mathematical Explanation**:\n",
    "Let the outputs from the hidden layer be \\( h_1 \\) and \\( h_2 \\), and let their corresponding weights for the output neuron be \\( w_3 \\) and \\( w_4 \\). The weighted sum for the output neuron would be:\n",
    "\n",
    "\\[\n",
    "output\\_neuron\\_input = (h_1 \\times w_3) + (h_2 \\times w_4) + b_{output}\n",
    "\\]\n",
    "\n",
    "Finally, this sum is passed through the sigmoid function to produce the final output.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 5: Calculating the Final Output\n",
    "\n",
    "The final output of the network is the prediction, typically a value between 0 and 1 when using the sigmoid activation function. This value represents the network’s confidence in its prediction. If the value is close to 1, the network is confident that the input belongs to the positive class (e.g., \"cat\"), while if the value is close to 0, it believes the input belongs to the negative class (e.g., \"not a cat\").\n",
    "\n",
    "**Example**:\n",
    "Let’s assume the hidden layer outputs are 0.845 and 0.8, and the weights for the output neuron are 0.4 and 0.45, with a bias of 0.6. The weighted sum for the output neuron would be:\n",
    "\n",
    "\\[\n",
    "output\\_neuron\\_input = (0.845 \\times 0.4) + (0.8 \\times 0.45) + 0.6 \\approx 1.39\n",
    "\\]\n",
    "\n",
    "Passing this through the sigmoid function:\n",
    "\n",
    "\\[\n",
    "final\\_output = sigmoid(1.39) \\approx 0.8\n",
    "\\]\n",
    "\n",
    "Thus, the network’s final prediction is 0.8, meaning the network is 80% confident that the input belongs to the positive class.\n",
    "\n",
    "---\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "In summary, forward propagation in a neural network involves passing the input data through a series of layers, where each neuron performs a weighted sum of its inputs, adds a bias, and applies an activation function. The result is passed forward until it reaches the output layer, where the final prediction is made.\n",
    "\n",
    "The key concepts in forward propagation include:\n",
    "- **Weights**: Control the importance of each input.\n",
    "- **Biases**: Shift the neuron’s output to allow the network more flexibility.\n",
    "- **Activation Functions**: Introduce non-linearity, enabling the network to model complex patterns.\n",
    "- **Output**: The final prediction of the network, often interpreted as a probability.\n",
    "\n",
    "Understanding forward propagation is the first step to understanding how neural networks learn and make predictions. As you dive deeper into neural networks, you’ll encounter more advanced topics like backpropagation, which is how the network learns from its errors and adjusts the weights and biases to improve its predictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0787c32d-9ee5-4c26-9f4b-ef0f5153c3f8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8e561a64-58e2-43a9-bc6a-839bc07b6d80",
   "metadata": {},
   "source": [
    "### connect with me on linkedin : https://www.linkedin.com/in/masood-ahmed-b42215226/\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "395ace25-580e-4607-9ae8-69565cd29b97",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}