chapter03_introduction-to-keras-and-tf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "# Introduction to Keras and TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## What's TensorFlow?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## What's Keras?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Keras and TensorFlow: a brief history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Setting up a deep-learning workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Jupyter notebooks: the preferred way to run deep-learning experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Using Colaboratory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### First steps with Colaboratory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Installing packages with `pip`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Using the GPU runtime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## First steps with TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Constant tensors and Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**All-ones or all-zeros tensors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "x = tf.ones(shape=(2, 1))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "x = tf.zeros(shape=(2, 1))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Random tensors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "x = tf.random.normal(shape=(3, 1), mean=0., stddev=1.)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "x = tf.random.uniform(shape=(3, 1), minval=0., maxval=1.)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**NumPy arrays are assignable**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "x = np.ones(shape=(2, 2))\n",
    "x[0, 0] = 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Creating a Variable**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "v = tf.Variable(initial_value=tf.random.normal(shape=(3, 1)))\n",
    "print(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Assigning a value to a Variable**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "v.assign(tf.ones((3, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Assigning a value to a subset of a Variable**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "v[0, 0].assign(3.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Using assign_add**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "v.assign_add(tf.ones((3, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Tensor operations: doing math in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**A few basic math operations**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "a = tf.ones((2, 2))\n",
    "b = tf.square(a)\n",
    "c = tf.sqrt(a)\n",
    "d = b + c\n",
    "e = tf.matmul(a, b)\n",
    "e *= d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### A second look at the `GradientTape` API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Using the GradientTape**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "input_var = tf.Variable(initial_value=3.)\n",
    "with tf.GradientTape() as tape:\n",
    "   result = tf.square(input_var)\n",
    "gradient = tape.gradient(result, input_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Using the GradientTape with constant tensor inputs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "input_const = tf.constant(3.)\n",
    "with tf.GradientTape() as tape:\n",
    "   tape.watch(input_const)\n",
    "   result = tf.square(input_const)\n",
    "gradient = tape.gradient(result, input_const)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Using nested gradient tapes to compute second-order gradients**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "time = tf.Variable(0.)\n",
    "with tf.GradientTape() as outer_tape:\n",
    "    with tf.GradientTape() as inner_tape:\n",
    "        position =  4.9 * time ** 2\n",
    "    speed = inner_tape.gradient(position, time)\n",
    "acceleration = outer_tape.gradient(speed, time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### An end-to-end example: a linear classifier in pure TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Generating two classes of random points in a 2D plane**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "num_samples_per_class = 1000\n",
    "negative_samples = np.random.multivariate_normal(\n",
    "    mean=[0, 3], cov=[[1, 0.5],[0.5, 1]], size=num_samples_per_class)\n",
    "positive_samples = np.random.multivariate_normal(\n",
    "    mean=[3, 0], cov=[[1, 0.5],[0.5, 1]], size=num_samples_per_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Stacking the two classes into an array with shape (2000, 2)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "inputs = np.vstack((negative_samples, positive_samples)).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Generating the corresponding targets (0 and 1)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "targets = np.vstack((np.zeros((num_samples_per_class, 1), dtype=\"float32\"),\n",
    "                     np.ones((num_samples_per_class, 1), dtype=\"float32\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Plotting the two point classes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(inputs[:, 0], inputs[:, 1], c=targets[:, 0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Creating the linear classifier variables**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "input_dim = 2\n",
    "output_dim = 1\n",
    "W = tf.Variable(initial_value=tf.random.uniform(shape=(input_dim, output_dim)))\n",
    "b = tf.Variable(initial_value=tf.zeros(shape=(output_dim,)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**The forward pass function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "def model(inputs):\n",
    "    return tf.matmul(inputs, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**The mean squared error loss function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "def square_loss(targets, predictions):\n",
    "    per_sample_losses = tf.square(targets - predictions)\n",
    "    return tf.reduce_mean(per_sample_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**The training step function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.1\n",
    "\n",
    "def training_step(inputs, targets):\n",
    "    with tf.GradientTape() as tape:\n",
    "        predictions = model(inputs)\n",
    "        loss = square_loss(predictions, targets)\n",
    "    grad_loss_wrt_W, grad_loss_wrt_b = tape.gradient(loss, [W, b])\n",
    "    W.assign_sub(grad_loss_wrt_W * learning_rate)\n",
    "    b.assign_sub(grad_loss_wrt_b * learning_rate)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**The batch training loop**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "for step in range(40):\n",
    "    loss = training_step(inputs, targets)\n",
    "    print(f\"Loss at step {step}: {loss:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "predictions = model(inputs)\n",
    "plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-1, 4, 100)\n",
    "y = - W[0] /  W[1] * x + (0.5 - b) / W[1]\n",
    "plt.plot(x, y, \"-r\")\n",
    "plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Anatomy of a neural network: understanding core Keras APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Layers: the building blocks of deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### The base `Layer` class in Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "class SimpleDense(keras.layers.Layer):\n",
    "\n",
    "    def __init__(self, units, activation=None):\n",
    "        super().__init__()\n",
    "        self.units = units\n",
    "        self.activation = activation\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        input_dim = input_shape[-1]\n",
    "        self.W = self.add_weight(shape=(input_dim, self.units),\n",
    "                                 initializer=\"random_normal\")\n",
    "        self.b = self.add_weight(shape=(self.units,),\n",
    "                                 initializer=\"zeros\")\n",
    "\n",
    "    def call(self, inputs):\n",
    "        y = tf.matmul(inputs, self.W) + self.b\n",
    "        if self.activation is not None:\n",
    "            y = self.activation(y)\n",
    "        return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "my_dense = SimpleDense(units=32, activation=tf.nn.relu)\n",
    "input_tensor = tf.ones(shape=(2, 784))\n",
    "output_tensor = my_dense(input_tensor)\n",
    "print(output_tensor.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "#### Automatic shape inference: building layers on the fly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers\n",
    "layer = layers.Dense(32, activation=\"relu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "model = models.Sequential([\n",
    "    layers.Dense(32, activation=\"relu\"),\n",
    "    layers.Dense(32)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model = keras.Sequential([\n",
    "    SimpleDense(32, activation=\"relu\"),\n",
    "    SimpleDense(64, activation=\"relu\"),\n",
    "    SimpleDense(32, activation=\"relu\"),\n",
    "    SimpleDense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### From layers to models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### The \"compile\" step: configuring the learning process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model = keras.Sequential([keras.layers.Dense(1)])\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"mean_squared_error\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=keras.optimizers.RMSprop(),\n",
    "              loss=keras.losses.MeanSquaredError(),\n",
    "              metrics=[keras.metrics.BinaryAccuracy()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Picking a loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Understanding the `fit` method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Calling `fit` with NumPy data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "    inputs,\n",
    "    targets,\n",
    "    epochs=5,\n",
    "    batch_size=128\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "history.history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Monitoring loss & metrics on validation data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**Using the validation data argument**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model = keras.Sequential([keras.layers.Dense(1)])\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=0.1),\n",
    "              loss=keras.losses.MeanSquaredError(),\n",
    "              metrics=[keras.metrics.BinaryAccuracy()])\n",
    "\n",
    "indices_permutation = np.random.permutation(len(inputs))\n",
    "shuffled_inputs = inputs[indices_permutation]\n",
    "shuffled_targets = targets[indices_permutation]\n",
    "\n",
    "num_validation_samples = int(0.3 * len(inputs))\n",
    "val_inputs = shuffled_inputs[:num_validation_samples]\n",
    "val_targets = shuffled_targets[:num_validation_samples]\n",
    "training_inputs = shuffled_inputs[num_validation_samples:]\n",
    "training_targets = shuffled_targets[num_validation_samples:]\n",
    "model.fit(\n",
    "    training_inputs,\n",
    "    training_targets,\n",
    "    epochs=5,\n",
    "    batch_size=16,\n",
    "    validation_data=(val_inputs, val_targets)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### Inference: using a model after training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "predictions = model.predict(val_inputs, batch_size=128)\n",
    "print(predictions[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Chapter summary"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "chapter03_introduction-to-keras-and-tf.i",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}