recreation-crowdsourced-bo-ui.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Recreating Experiment 1 in Crowdsourcing Interface Feature Design with Bayesian Optimization\n",
        "\n",
        "- Paper: https://jsonj.co.uk/resources/science/papers/chi2019-bayesian-optimization.pdf\n",
        "- Experimental Data: https://www.repository.cam.ac.uk/handle/1810/292232\n",
        "\n",
        "The code below is a **very simplified** attempt to recreate this paper's Experiment 1. See the discussion below the code for aspects of the paper that were simplified.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import scipy.stats\n",
        "from matplotlib import pyplot as plt\n",
        "import GPy; import GPyOpt"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def f_u(x):\n",
        "    print(x)\n",
        "    while True:\n",
        "        res = input('Completion Time: ')\n",
        "        res = float(res)\n",
        "        return res\n",
        "\n",
        "\n",
        "def run_bo(max_iter):\n",
        "    bounds = [{'name': 'x1', 'type': 'continuous', 'domain': (10, 200)},\n",
        "              {'name': 'x2', 'type': 'continuous', 'domain': (100, 1000)},\n",
        "              {'name': 'x3', 'type': 'continuous', 'domain': (0.5, 2.5)},\n",
        "              {'name': 'x4', 'type': 'continuous', 'domain': (100, 1000)},\n",
        "              {'name': 'x5', 'type': 'continuous', 'domain': (0.2,1)}]\n",
        "    myBopt = GPyOpt.methods.BayesianOptimization(\n",
        "        f=f_u, domain=bounds,\n",
        "        acquisition_type='EI',\n",
        "        exact_feval=False,\n",
        "        eps=1e-6,\n",
        "        normalize_Y=False,\n",
        "        initial_design_numdata=2)\n",
        "    myBopt.run_optimization(max_iter=max_iter - 2)\n",
        "\n",
        "    return myBopt\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    n_iter = 8\n",
        "    # run BO\n",
        "    bo = run_bo(n_iter)\n",
        "    bo_xs, bo_ys = bo.get_evaluations()\n",
        "\n",
        "    plt.plot(-bo_ys, 'k-', label='BO')\n",
        "    plt.xlabel('iterations')\n",
        "    plt.ylabel('completion time')\n",
        "    plt.show()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "scrolled": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## RESCALING OBSERVATIONS \n",
        "> We rescale the observation values to have zero mean and unit variance. To do this in the absence\n",
        "of prior data, we require a coarse approximation of the distribution of typical observation values.\n",
        "In this paper, the observation values are task completion times. Completion time is\n",
        "approximately the product of the number of inspections and\n",
        "time per inspection. Such products approach a log-normal\n",
        "distribution. Based on initial pilot testing and for consistency\n",
        "across the experiments and case study, we assume a mean\n",
        "completion time of approximately 30 s. To normalize for unit\n",
        "variance we expect typical task times might vary between\n",
        "15 s (half) and 60 s (double) which corresponds very approximately with a log-normal standard deviation of 0.7.\n",
        "\n",
        "<p>Time constraints didn't allow me to learn how to appropriately implement this.\n",
        "\n",
        "## BATCHING\n",
        "> Within each batch and for each\n",
        "user, however, the standard process of Bayesian optimization\n",
        "also incorporates the individual user’s prior performance.\n",
        "The hyperparameters are held constant during a batch and\n",
        "then updated based on all data up to and including the most\n",
        "recent batch. In the first batch with no prior data, the hyperparameters were all set to a nominal value (unity).\n",
        "\n",
        "<p>I've asked the authors to clarify as what constitutes an observation.\n",
        "\n",
        "## KERNEL AND ACQUISITION FUNCTION\n",
        ">In this study we employ the automatic relevance determination (ARD) kernel\n",
        "The literature provides many choices for acquisition function. We use a\n",
        "standard approach based on expected improvement (EI).\n",
        "\n",
        "<p>\n",
        "\n",
        ">From Brochu et al., 2010a\n",
        "Note that the conventional approach to setting\n",
        "RBF parameters (c,α) as well as the kernel parameters (θ,σ) is to maximize the log-likelihood of the\n",
        "model within a specific optimization run. This method works\n",
        "well for many application of GPs, but unfortunately for our\n",
        "application, it is known to work poorly when the number\n",
        "of training data is small (exactly the situation we are in).\n",
        "The sparsity of data in the space can lead to the likelihood\n",
        "function becoming very flat on some dimensions, or even\n",
        "monotonically increasing to infinity. This can lead to low quality models\n",
        "or ill-conditioned covariance matrices.\n",
        "\n",
        "<p>\"ARD Kernel\" could be a misnomer as it appears to be an attribute of multiple kernels in GPy. As a simplification, I used the default kernel.\n",
        "\n",
        "## OPTIMIZING THE ACQUISITION FUNCTION\n",
        "> The approach involves evaluating a candidate list of sample\n",
        "points that provide representative coverage of the design space.\n",
        "Appropriate bounds for each parameter are chosen\n",
        "and this sets the limits of the hypercube. The candidate list\n",
        "is then constructed by sampling from the parameter hypercube using a Sobol sequence as described by 26. We use\n",
        "1000 candidates sampled in this way.\n",
        "    \n",
        "<p>Time constraints didn't allow me to learn how to appropriately implement this."
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "jupytext": {
      "cell_metadata_filter": "-all",
      "main_language": "python",
      "notebook_metadata_filter": "-all"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "nteract": {
      "version": "0.14.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}