Lecture_1_CombinatorialOptimization_Oulasvirta.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "94692a50-d205-44a0-b4c2-ed6e63a6893f"
    }
   },
   "source": [
    "<img src=\"imgs/logo.png\" width=\"85%\">\n",
    "\n",
    "---------------\n",
    "# 5th Summer School on Computational Interaction\n",
    "\n",
    "## Part 1: Introduction to Combinatorial Optimization\n",
    "\n",
    "---\n",
    "\n",
    "#### <b>[Antti Oulasvirta](http://users.comnet.aalto.fi/oulasvir/)</b> / Aalto University\n",
    "\n",
    "*Combinatorial optimization refers to algorithmic search for combinations of design decisions that best meet stated design objectives. Published applications in HCI include keyboards and panels, menu systems, graphical user interfaces, visualizations, and input methods. In this block, we first learn three basic concepts: (1) design space (feasible set), (2) objective function, and (3) search method. We implement a plain vanilla optimizer. Our exercises apply it in layout design tasks of increasing difficulty. In the latter part, we focus on heuristic methods for improving search performance. The last part looks at advanced topics, such as the design of multi-objective, robust, and dynamic methods.*\n",
    "\n",
    "This notebook provides general introduction with focus on heuristic black-box optimization.  Another notebook by Anna Feit </a> focuses on integer programming, an exact method.  \n",
    "\n",
    "#### Primary sources\n",
    "1. Oulasvirta., A. & Karrenbauer, A. (2018). Combinatorial Optimization for Interface Design, in \"Computational Interaction\", Oxford University Press, 2018.1. Rao, S.R. Engineering Optimization: Theory and Practice. John Wiley & Sons, 2009.\n",
    "2. Oulasvirta, A. (2017) [User Interface Design with Combinatorial Optimization](http://users.comnet.aalto.fi/oulasvir/pubs/UIdesign_with_combinatorial_optimization_IEEEComputer_2017_Oulasvirta.pdf), IEEE Computer 50 (1), pp. 40-47. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "06e95d9c-25d9-42ea-b6e1-70f0d31b05c8"
    }
   },
   "source": [
    "## Learning objectives\n",
    "\n",
    "---\n",
    "Throughout, our emphasis is on conceptual understanding. Links will be offered to understand requirements for harder problems.\n",
    "\n",
    "### Targeted audience\n",
    "\n",
    "* HCI students and researchers with some exposure to algorithms and Python programming\n",
    "* Computer scientists with interest in but little experience with HCI problems\n",
    "* No prior knowledge on combinatorial optimization methods nor behavioral modeling is assumed.\n",
    "\n",
    "### In this unit, you will learn how to ...\n",
    "\n",
    "* define a simple layout design task as a combinatorial optimization problem\n",
    "* encode design goals and assumptions in its objective function\n",
    "* choose and tune a simple search algorithm (solver).\n",
    "\n",
    "We will also go through core assumptions behind the approach and review its position in HCI.\n",
    " \n",
    "### Practically, you will ...\n",
    "\n",
    "* implement a random search -based optimizer for layout problems of increasing difficulty: linear menus, toolbars, dialogues, and web layouts\n",
    "* understand what interface design 'means' from a combinatorial perspective (and what makes it hard)\n",
    "* explore the consequences of changing the objective function\n",
    "* boost efficiency with more advanced black box optimization methods (e.g., simulated annealing)\n",
    "\n",
    "The code offers a bare bones, extensible solver for layout-related design problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c042b825-38ba-4cac-b80b-194100103daf"
    }
   },
   "source": [
    "# 1. Motivation \n",
    "-----------------\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>In-class exercise.</h3> \n",
    "<br>\n",
    "    An optimization task with a human oracle and a human solver.\n",
    "</div>\n",
    "</p>\n",
    "\n",
    "\n",
    "### An executable formalism for interaction design \n",
    "\n",
    "*Optimization* refers to both the **outcome** (optimal design) and the **process** of systematically improving a design. Interface design, herein, is combination of discrete design decisions to obtain an optimal solution defined by a stated objective function. To *design interaction* is about improving design decisions algorithmically in the light of best information on what makes interaction good for end-users. \n",
    "\n",
    "*Model-based combinatorial optimization* is a [multidisciplinary approach to computational design](imgs/multidisciplinary-overview.png) inviting contributions from computer scientists, designers, and behavioral scientists. The objective function is a model (statistical, rule-based, neural,...) that links design choices to quantitative estimates of effects on end-users. Besides one-shot optimization, these methods can be used as [tools in user-centered design](imgs/ucd.png) and in [data-driven design](imgs/ucd-with-ML.png). \n",
    "\n",
    "The defining properties within HCI are: \n",
    "\n",
    "| Property  | Combinatorial optimization  |\n",
    "| ---------- | ------ |\n",
    "| **Constructive** |  The algorithm proposes how to construct a full or partial design   |\n",
    "| **Reductionist AND holist**  |  The 'goodness' of one design choice is considered in the context of the rest of the design  |\n",
    "| **Predictive**  |  In model-based optimization, the objective function is an explicit and empirically verifiable statement about the 'goodness' of design   |\n",
    "| **Systematic**  |  Exploits a wealth of efficient algorithms known in computer science |\n",
    "| **Explicit**  |  All assumptions are scrutinable  |\n",
    "\n",
    "There are strong rationale for this method in HCI. It offers:\n",
    "* a rigorous theoretical basis for design: defining central concepts such as ’design task', 'design space’, ’design objective’, 'design method', and ’optimal design’\n",
    "* efficient solutions: exploiting results of decades of research in computer science, and allowing you to attack complex problems with good results and pursue real-time interactive performance\n",
    "* a flexible toolbox for generating and adapting designs automatically and for supporting designers in combinatorially hard problems in design. \n",
    "\n",
    "**General benefits to practitioners** include:\n",
    "* Improved quality of solutions: Modern optimization methods support designers by offering a high chance of finding good or optimal solutions \n",
    "* Solutions to hard (large, multi-objective) problems\n",
    "* Estimates of practically achievable improvements possible for a given design\n",
    "* Information about the complexity and structure of a problem \n",
    "* Automating design or adaptation in a real-time system\n",
    "* Integratable with interactive design tools\n",
    "* Explicit, scrutinable approach that can support knowledge building in a team.\n",
    "\n",
    "Importantly, combinatorial optimization can be used not only to find an optimal design but to **assist a designer in various problems encountered in design**, such as finding:\n",
    "* a unique (maximally different but equally good) solution\n",
    "* a design that strikes the best compromise among competing objectives\n",
    "* a design that is robust to changes in assumptions (e.g., about conditions of use or users)\n",
    "* a design achieved by minimal changes to a current design\n",
    "* a design that is within some k% from the present design. \n",
    "\n",
    "### Challenges \n",
    "\n",
    "> *“The tortoise of accumulative science will win the hare of intuitive design.”* (Card, Moran, and Newell in Psychology of Human-Computer Interaction, 1983)\n",
    "\n",
    "Combinatorial optimization is a principled approach that insists on rigorous definitions of the design task and objectives. Be warned about:\n",
    "\n",
    "1. Ramp-up costs: Formal definition and development of search methods can be laborious and risky;\n",
    "2. At times exceedingly hard problems: While modest background in computer science suffices for first probes, real-world cases can be prohibitively hard and insist on professional support;\n",
    "3. Knowledge gap: Defining meaningful objective functions (e.g., aesthetics, learnability) may require support from a cognitive or behavioral scientist with mathematical abilities.\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>In-class demonstrations.</h3> \n",
    "<br>\n",
    "    Videos of recent applications.\n",
    "</div>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Basic concepts\n",
    "\n",
    "---------------\n",
    "\n",
    "We start by defining a **DESIGN TASK**, consisting of three elements:\n",
    "1. **design space** (search space; a.k.a. feasible set, candidate set): a finite set of alternative designs; \n",
    "2. **objective function**: defines what you mean by 'good' or 'desirable design';\n",
    "3. **task instance**: sets task-specific parameters.\n",
    "\n",
    "A task defined like this can then be attacked by a **solver**. It goes through the design space to find increasingly better designs. We use this structure in the following exercises.\n",
    "\n",
    "<img src=\"imgs/optimization.png\" width=\"95%\">\n",
    "\n",
    "### Design task\n",
    "\n",
    "In plain words, the task is to find the design that yields the highest value of a given objective (goodness) function. The following definition makes **the combinatorial structure of design** more explicit:\n",
    "$$\\mbox{Find } \\mathbf{x} =\n",
    "\t\\begin{pmatrix}\n",
    "\tx_1 \\\\\n",
    "\tx_2 \\\\\n",
    "\t\\vdots \\\\\n",
    "\tx_n\n",
    "\t\\end{pmatrix} \\in \\mathbf{X}\n",
    "\\mbox{ which maximizes } f(\\mathbf{x})$$\n",
    "where \n",
    "* $\\mathbf{x}$ is an *n*-dimensional **design vector**,  each dimension describing a **design variable**, and\n",
    "* $\\mathbf{X}$ is **the set of feasible designs** (all vectors that fulfill constraints).\n",
    "\n",
    "In plain words, a design is a vector in a multi-dimensional vector space. The objective function assigns each vector a value describing its goodness (or badness). When we optimize, we are looking for the vectors that obtain the highest value.\n",
    "\n",
    "### Design space\n",
    "\n",
    "A design space is the product of design variables under consideration. In a real design problem, some variables are fixed and some are open and to be decided by an optimizer. **A design variable** captures a decision using familiar types of variables, such as:\n",
    "* boolean (e.g., Do we offer auditory feedback for button press or not?)\n",
    "* integer (e.g., position of an element on x-axis)\n",
    "* real (e.g., hue)\n",
    "* categorical (e.g., type of an element).\n",
    "\n",
    "In combinatorial optimization continuous variables are discretized. \n",
    "\n",
    "Functional and other requirements that 'rule out' candidate designs are collectively called **design constraints**. Example: When designing a Web layout, we should not place elements such that they overlap.\n",
    "\n",
    "#### Analysis of problem size\n",
    "\n",
    "The first benefit of this approach is visible now: Defining a design space allows examining its size and structure. When we start doing this for UI design problems, we quickly realize that **design spaces in HCI are often very large**. For example, for $n$ functionality there are $2^n - 1$ ways to combine them to an application, which for only 50 functions means 1,125,899,906,842,623 possibilities. Further, assuming that 50 have been selected, they can be organized into a hierarhical menu in $100! \\approx 10^{158}$ ways. [Combinatorics](https://www.tug.org/texshowcase/cheat.pdf) can be used to analyze problem sizes. \n",
    "\n",
    "### Objective function\n",
    "\n",
    "The objective function $f( \\mathbf{x} )$ states what makes a design good. It is a function that assigns an **objective score (or value)** to each design candidate. The design candidate that obtains the highest (or lowest, when minimizing) score is the **optimum design**. The function can be about anything regarded desirable, such as surface features of the interface (e.g., minimal white space) or expected performance of users (e.g., 'task A should be completed as quickly as possible') and so on. In HCI, the quality of a interface is primarily determined by reference to end-users, for example user performance and experiences. \n",
    "\n",
    "There are many principled (and unprincipled) ways to obtain objective functions:\n",
    "* Literature: A model or theory from prior work\n",
    "* Data-driven approach: Use e.g. statistics or machine learning to obtain a model from data\n",
    "* Standards, guidelines, and design patterns\n",
    "* Heuristics, such as guidelines and best-found practices.\n",
    "\n",
    "### Solver\n",
    "\n",
    "After a task has been defined, a solver should be chosen and tuned to attack its instances. We will later look at solvers and their different properties. The process of 'solving' can be intuitively understood as search for highest peaks (when maximizing), or lowest valleys (when minimizing), in a vast 'landscape':\n",
    "\n",
    "<img src=\"imgs/landscapes.png\" width=\"80%\">\n",
    "\n",
    "### A word about 'optimality'\n",
    "\n",
    "To say that a design is optimal is to say that in the defined set of candidate designs, there is no better design (global optimum has been identified). The word 'optimal' is overused and misused. In order to claim that a design is optimal, one *must* answer the following questions:\n",
    "1. The best out of which options? (design space)\n",
    "2. The best for what? (design objectives and task instance)\n",
    "3. The best with what guarantees? (search method)\n",
    "\n",
    "A design is **globally optimal** when the search method *guarantees* that it achieves the highest objective value in the whole design space. Such guarantees are offered by exhaustive search and exact methods (e.g., integer programming). When we do not have access to those methods, for example because the complexity of the objective function, we cannot claim global optimality. A design is **approximately optimal** when its objective value is within some margin of the optimal design, or when there is a good chance that only marginally better designs exist. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "fc20e5dd-230e-443d-899a-d379c5dfaf42"
    }
   },
   "source": [
    "# 3. Our first optimizer: Linear menu design\n",
    "\n",
    "----------\n",
    "\n",
    "**Linear menu** is one of the most prevalent type of user interfaces, which we find in mobile and desktop applications, web services, and consumer electronics. This optimizer serves as a sipmle example illustrating the concepts introduced above.\n",
    "\n",
    "### Task definition\n",
    "\n",
    "The design of a linear menu is a so-called **assignment problem**: *n* commands are to be *assigned* to *n* predetermined locations (slots) in order to maximize usability, which here means **minimizing average selection time**. We insist that every slot must be filled with one command, and each command must be assigned only once (constraints). With these in mind, a linear menu with *n* items can be organized in *n!* ways. \n",
    "\n",
    "### Approach overview\n",
    "\n",
    "We implement a simple objective function emulating a novice user's search performance with a menu, and a random search algorithm as our solver. An overview of our design task:\n",
    "\n",
    "| Functionality | Description  | In code |\n",
    "|----------|------|------|\n",
    "| Design task | Given *n* menu items, decide their order to minimize expected selection time | |\n",
    "| Design space | All possible orderings of the items (*n!* in total)  | permutations of a list |\n",
    "| Objective function | Minimize expected selection time | linear_ST |\n",
    "| Task instance | Specification of elements and their relative importance | seed_layout, e_weights|\n",
    "| Solver | Random search | random_search |\n",
    "\n",
    "Note: We are simply ordering elements in this exercise. \n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Exercise 1: Implement a random search algorithm.</h3> \n",
    "<br>\n",
    "It should shuffle a design and consult the (given) objective function to ask how good it is. If it's better, it should become the new *incumbent* (best known design). Continue like this until max_iters. Return the incumbent. \n",
    "</div>\n",
    "</p>\n",
    "\n",
    "#### Introduction to code\n",
    "\n",
    "We offer a scaffolding for a plug-and-play black box optimizer:\n",
    "* Our design space -- all orderings of *n* elements -- is represented as a list in Python. The position of an element indicates its position in the menu.\n",
    "* Our *objective function* computes expected search time in $O(n^2)$ time.\n",
    "* Our *optimize* function takes any solver function and objective function as input. It then calls the solver using the objective function as its input. We use this 'oracle optimizer' later with different inputs.\n",
    "\n",
    "Note: This code **will not run** without the missing part filled in:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "a583c8ec-989d-46fb-a38c-b5bf0e9ec835"
    }
   },
   "outputs": [],
   "source": [
    "import math, random\n",
    "import render # For presenting layouts for rendering in Jupyter \n",
    "from IPython.display import SVG # SVG capabilities for showing layouts\n",
    "\n",
    "# Returns Euclidean (unit) distance between two element positions in a grid layout\n",
    "# Needed in our objective function (ST)\n",
    "def distance (columns, i, j):\n",
    "    return math.sqrt( abs(j / columns - i / columns)**2 + abs( i % columns - j % columns)**2) \n",
    "    \n",
    "# Our objective function: Selection time (time it takes for users to select an item). \n",
    "# - A weighted sum of individual selection times weighed by their importance/probability (weight)\n",
    "# - This simplifies the novice model of menu search time presented by Cockburn et al. CHI 2007 \n",
    "def linear_ST (layout, columns, o_inputs):\n",
    "    ST = 0.0\n",
    "    reading_cost = 0.4 # assumed that scanning a single item takes 400 ms\n",
    "    for i,element in enumerate(layout):\n",
    "        ST += o_inputs[0][layout[i]] * distance(columns,0,i) * reading_cost\n",
    "    return ST\n",
    "\n",
    "# Our solver: Random search method\n",
    "# - Shuffles a layout (list) and asks from objective function how good it is\n",
    "# - Updates incumbent (best known design) whenever an improvement is found\n",
    "# - Continues like this max_iters times\n",
    "def random_search(max_iters, *args):\n",
    "    columns = args[1] # Number of columns in this layout (=1) \n",
    "    obj_f = args[2]   # Handle to the objective function (=linear_ST)\n",
    "    o_inputs = args[3:] # Arguments simply passed on to the objective function\n",
    "    incumbent = args[0] # Best-known design thus far\n",
    "    incumbent_ov = obj_f(incumbent, columns, o_inputs) # Set initial objective value\n",
    "\n",
    "    for iter in range (0, max_iters):\n",
    "        # TASK: FIX THE NEXT LINE\n",
    "        candidate = incumbent # this is wrong \n",
    "        candidate_ov = obj_f(candidate, columns, o_inputs) # Then compute its objective value\n",
    "\n",
    "        if candidate_ov < incumbent_ov: # Update best known if an improvement was found\n",
    "            incumbent = candidate[:]\n",
    "            incumbent_ov = candidate_ov\n",
    "    return incumbent, incumbent_ov\n",
    "\n",
    "    \n",
    "# Our generic optimization service:\n",
    "# - Solver and objective function are given as arguments\n",
    "# - Used throughout the exercises\n",
    "def optimize (iters, solver, *args):\n",
    "    return solver(iters, *args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "3531538e-a779-418b-ae61-52e8a807d11a"
    }
   },
   "outputs": [],
   "source": [
    "# Task instance\n",
    "seed_layout =  ['Open', 'About','Quit','Help','Close','Save','Edit','Insert','Delete'] # initial layout\n",
    "e_weights =    {'Quit':0.5,'About':0.4,'Open':0.2,'Save':0.2,'Close':0.1,'Help':0.05,'Edit':0.1,'Insert':0.2,'Delete':0.2} # weights or 'importance' of elements\n",
    "\n",
    "# Call the optimizer \n",
    "winner, winner_score = optimize(10000, random_search, seed_layout, 1, linear_ST, e_weights)\n",
    "\n",
    "# Show result\n",
    "print (\"Objective value (expected selection time):\", winner_score)\n",
    "SVGlayout = render.SVGlayout(winner,1)\n",
    "SVG(SVGlayout.inSVG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection</h3> \n",
    "<br>\n",
    "Well, this was trivial. Why? \n",
    "</div>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f1eaff93-e8ab-4a5d-a189-dd2d51eebf5f"
    }
   },
   "source": [
    "# 4. Basic concepts: Revisited\n",
    "\n",
    "------------------\n",
    "\n",
    "To attack more interesting problems, we need to revisit the basic concepts:\n",
    "\n",
    "<img src=\"imgs/overview.png\" width=\"80%\">\n",
    "\n",
    "### Design space\n",
    "\n",
    "*Design space* is a representation of the candidate set of designs, laid out in a suitable format for some chosen search algorithm. [Some representations are illustrated here](imgs/representations.png). The choice of representation affects:\n",
    "* availability of sovers; for example, the formulation of the concept of search neighborhood, which is exploited in local search methods\n",
    "* computational efficiency\n",
    "* ease of extension.\n",
    "\n",
    "In the following, we use a *list* representation of layouts, that has its own pros and cons.\n",
    "\n",
    "### Objective function\n",
    "\n",
    "**An objective function** maps a design candidate $x \\in \\mathbf{X}$ to a real-valued predictor:\n",
    "$f: \\mathbf{X} \\rightarrow \\mathbb{R}$.\n",
    "The choice of modeling formalism affects validity (hit rate, quality) and computational efficiency. Moreover, to be relevant in interaction design, this function should quantify an aspect like usability, enjoyability, ergonomics, or errors. Several objectives may need to be combined for reasonable results ([example: web layout design; Todi et al. DIS 2016](imgs/multi-objective.png)). This table shows 11 examples spanning task, cognitive, motor, and aesthetic aspects:\n",
    "\n",
    "| Model | Example objective |  Application ref.  |\n",
    "|----------|------|-------|\n",
    "| Fitts' law  |  Minimize finger travel times | Zhai et al. UIST 2000 |\n",
    "| Search-Decide-Point | Minimize visual search and selection time in menus  | Bailly et al. UIST 2013 |\n",
    "| Coherence  |  Ensure related items are close to each other | Bailly et al. UIST 2013 |\n",
    "| Familiarity |  Ensure spatial resemblance with a prior design | Dunlop et al. CHI 2013 |\n",
    "| Keystroke-level model | Minimize task completion time for a sequential user task | Sarcar et al. IxAP 2016 |\n",
    "| Active vision | Minimize visual search time when items are colored | Todi et al. DIS 2016 |\n",
    "| Rosenholtz clutter model | Minimize visual clutter, maximize saliency | Rosenholtz et al. 2011 |\n",
    "| Color harmony theory | Ensure harmonious color sets | Todi et al. DIS 2016 |\n",
    "| Motor complexity theory | Minimize complexity of gesture sets | Sridhar et al. CHI 2015 |\n",
    "| Ergonomics | Minimize shoulder stress | Bachynskyi et al. TOCHI 2015 |\n",
    "| Internal clock model | Keep temporal task performance at desirable level | Lee et al. CHI 2016 |\n",
    "\n",
    "**Subjective goals and preferences** can be dealt with by two main means: (1) by calibrating weights manually to correspond to some distribution of preferences, be they designers' or users'; (2) using machine learning to parametrize models to best correspond with individual priorities expressed in data. \n",
    "\n",
    "#### Approaches to defining objective functions\n",
    "\n",
    "**Heuristics** are rules of thumb used in design,  such as 'white space should be minimized'.  Since most heuristics are 'weak' (carry little information), multiple are needed. This leads to the problem of **conflict resolution**: how to weigh the contribution of one heuristic against another. Another issue of heuristics is their questionable validity. (Note: not to be confused with heuristic optimization methods.)\n",
    "\n",
    "**Metrics** are expressed as a real-valued functions that capture some feature of a design and link it to predictions of outcomes to users. Example: Overplotting metrics in the design of information visualizations.\n",
    "\n",
    "**Regression** offers a more expressive yet quickly computable mathematical form of cost functions:\n",
    "$y = f (\\mathbf{x}, \\mathbf{\\beta} ) + \\epsilon$ . \n",
    "\n",
    "**Simulator-based models** are step-wise executed functions $M$ that map the model parameters $\\theta$ to data. In HCI, the parameters $\\theta$ capture both parameters of the user and those of the task or context. Simulators are **generative models** that predict not only the outcomes of interaction but intermediate steps, or the process of interaction. A simulator may address multiple design objectives in a coherent manner that require less weight-tuning. However, they are non-linear and significantly more expensive to execute, ruling out exact methods (e.g, integer programming).\n",
    "\n",
    "In addition, there is a vast number of machine learning of different types available.\n",
    "\n",
    "### Task instance\n",
    "\n",
    "*Task instance* refers to task-specific input values to the design task and objective function, such as:\n",
    "* Which elements are to be decided (sizes, colors etc)?\n",
    "* Relative importances of design objectives (e.g,. how much is usability more important than security?)\n",
    "* Properties of targeted user groups, such as goals, capabilities, and interaction styles of users (e.g., what do they want to achieve with the interface?)\n",
    "* Constraints of the interface (e.g., screen resolution, input method).\n",
    "\n",
    "### Optimization method (solver)\n",
    "\n",
    "Modern methods in combinatorial optimization can be divided into two main classes. An overview of [optimization methods in engineering optimization](imgs/optimizationmethods.png) is offered in Rao 2009:\n",
    "1. **Black-box or heuristic methods** do not necessitate assumptions about  the objective function, but rather consider it as a black-box -- like an oracle -- that tells them the objective value of a given candidate it has generated.  \n",
    "2. **Exact methods** such as integer programming offer mathematical guarantees for solutions but insist on analysis of the objective function for revealing simplifications and links to known tasks in the optimization literature. Integer programming is covered in the other part of the day.\n",
    "\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Group exercise.</h3> \n",
    "<br>\n",
    "Take an example paper from CHI or UIST and report the following: design task, design space, objective function, solver. Present the paper to the class.\n",
    "</div>\n",
    "</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d1a8d537-7f0e-4408-9e7e-70c298cafc3a"
    }
   },
   "source": [
    "# 5. Toolbar design\n",
    "\n",
    "---------------\n",
    "\n",
    "**Toolbars**, widely used in desktop and web applications, are essentially **grid menus**. To design a toolbar, ordering elements is not enough. One needs to consider how to ensure that related items are close to each other (grouping).\n",
    "\n",
    "| New aspect | Content  |\n",
    "|----------|------|\n",
    "| Design objectives | Minimum selection time; **Your objective function for grouping** |\n",
    "| Task instance | 12 elements in a productivity app |\n",
    "\n",
    "<img src=\"imgs/toolbar.png\" width=\"70%\">\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Exercise 2: Implement an objective function for element-grouping.</h3> \n",
    "<br>\n",
    "Implement an objective function for grouping. It should try to keep *related items close to each other*. **Tip:** Association value between Your function can utilize *distance* function defined in the earlier cell. Call syntax for distance function: 'distance(columns, i,j)' </div>\n",
    "</p>\n",
    "\n",
    "Note: The random search algorithm of Exercise 1 must work before starting on this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Objective function that sums up two sub-objectives: selection time and YOUR function\n",
    "def ST_and_myO (layout, columns, o_inputs):\n",
    "    return linear_ST (layout,columns,o_inputs[0:]) + 0.5 * myObjective (layout,columns,o_inputs[1:])\n",
    "\n",
    "def myObjective (layout, columns, o_inputs):\n",
    "    ov = 0.0\n",
    "    for i in range(0, len(layout)):\n",
    "        for j in range(i+1, len(layout)):\n",
    "            # association score: highest value in dict associations or 0.0\n",
    "            association = max(value for value in [0.0,o_inputs[0].get(layout[i]+layout[j]), o_inputs[0].get(layout[j] + layout[i])] if value is not None)\n",
    "            # TASK: ADD YOUR OBJECTIVE FUNCTION HERE\n",
    "            # Tip: Use these two: \n",
    "            # - distance(columns, i,j)  (distance between the two elements)\n",
    "            # - association (association between the two elements)\n",
    "            ov += 0.0      # This is wrong; fix this\n",
    "    return ov\n",
    "\n",
    "# Task instance\n",
    "seed_layout = ['Tasks','Word','Excel','PPT','Admin','Mail','Cal','Ppl','News','Drive','Sites','Notes']\n",
    "e_weights = {'Tasks':0.1,'Word':0.2,'Excel':0.15,'PPT':0.2,'Admin':0.05,'Mail':0.5,'Cal':0.4,'Ppl':0.4,'News':0.4,'Drive':0.2,'Sites':0.01, 'Notes':0.05}\n",
    "associations = {'WordExcel':0.5,'WordPPT':0.5,'MailCal':0.3,'PplCal':0.3,'TasksCal':0.2,'NotesTasks': 0.3}\n",
    "columns = 6\n",
    "\n",
    "# Optimization\n",
    "winner, winner_score = optimize(3000, random_search, seed_layout, columns, ST_and_myO, e_weights, associations)\n",
    "\n",
    "# Results\n",
    "print (\"Objective value:\", winner_score)\n",
    "SVGlayout = render.SVGlayout(winner,columns,10,associations)\n",
    "SVG(SVGlayout.inSVG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection</h3> \n",
    "<br>\n",
    "Should grouping be implemented as an *objective* OR a *constraint*?</div>\n",
    "</p>\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection</h3> \n",
    "<br>\n",
    "The weighted sum approach to multi-objective optimization is sensitive to the value of the weight. Find out what happens when the objective weight is changed (line 3, now set to 0.5). This weight trades off the relative importance of the two objectives in the optimation process. It determines how the two objectives are collapsed into one scalar. \n",
    "</div>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. A Better Solver: Simulated Annealing\n",
    "\n",
    "Results obtained thus far have been reasonable, because the problems have been small. To attack larger problems efficiently, and to introduce first concepts of heuristic methods, we turn to a classic: **simulated annealing**\n",
    "* [Main idea in three diagrams](imgs/sa.jpg)\n",
    "* [Pseudo-code](imgs/sa-pseudocode.png) \n",
    "\n",
    "We run the previous exercise using simulated annealing, observing that (on average) better objective value is obtained with the same computation budget. \n",
    "\n",
    "### Basic concepts\n",
    "\n",
    "**Local search** relies on a definition of a **neighborhood** of a given design. For example, the neighboring layouts of a keyboard layouts are those that can be constructed by swapping a pair of letters. Starting from an initial configuration, local search proceeds by chosing a neighboring configuration with an objective value that is at least as good as the current one. These choices could be, for example, greedy or randomized.\n",
    "\n",
    "**Simulated Annealing** exploits local and random search in a exploration/exploitation scheme. A neighboring configuration is not only chosen when it yields to a better objective value but also with a certain probability when it is worse. However, this probability decreases with the extent of the detoriation of the objective value (temperature). \n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Exercise 3: Define a 'neighboring design' for simulated annealing</h3> \n",
    "<br>\n",
    "To use SA, need to add a function that allows the simulated annealer to explore the *neighborhood* of a current design. While there are many possibilities to construct a neighborhood. Your task is to define one. Bonus: Define control parameter 'n' for distance in the neighborhood.</div>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returns a neighbor of a given layout (list); \n",
    "# Has a parameter 'n' to control distance in the neighborhood (optional) \n",
    "def neighbor(layout, n=1):\n",
    "    # This is wrong\n",
    "    return layout\n",
    "\n",
    "# Solver: Simulated annealing using exponential cooling schedule\n",
    "def anneal(k_max, *args):\n",
    "    s = args[0] # solution seed\n",
    "    columns = args[1] \n",
    "    obj_f = args[2]\n",
    "    o_inputs = args[3:]\n",
    "    s_ov = obj_f(s, columns, o_inputs)\n",
    "    T_min, T_initial, alpha = 0.0000001, 10000, 0.991 # Hyperparameters \n",
    "    converged = False\n",
    "    \n",
    "    for k in range (0, k_max):\n",
    "        T = max(T_min, T_initial * math.pow(alpha,k)) # exponential cooling schedule\n",
    "        s_new = neighbor(s[:], args[-1])\n",
    "        s_new_ov = obj_f(s_new, columns, o_inputs)\n",
    "\n",
    "        delta = s_new_ov - s_ov\n",
    "        if delta < 0: # accept the neighbor if it is better \n",
    "            s = s_new[:]\n",
    "            s_ov = s_new_ov\n",
    "        elif random.random() < math.exp(-delta/T): # if not, decide according to the Metropolis rule\n",
    "            s = s_new[:]\n",
    "            s_ov = s_new_ov\n",
    "    return s, s_ov\n",
    "\n",
    "# Task instance (same as above, copied here for reference)\n",
    "seed_layout = ['Tasks','Word','Excel','PPT','Admin','Mail','Cal','Ppl','News','Drive','Sites','Notes']\n",
    "e_weights = {'Tasks':0.1,'Word':0.2,'Excel':0.15,'PPT':0.2,'Admin':0.05,'Mail':0.5,'Cal':0.4,'Ppl':0.4,'News':0.4,'Drive':0.2,'Sites':0.01, 'Notes':0.05}\n",
    "associations = {'WordExcel':0.5,'WordPPT':0.5,'MailCal':0.3,'PplCal':0.3,'TasksCal':0.2,'NotesTasks': 0.3}\n",
    "columns = 6\n",
    "\n",
    "# Optimization\n",
    "winner, winner_score = optimize(10000, anneal, seed_layout, columns, ST_and_myO, e_weights, associations, 1)\n",
    "\n",
    "# Present results\n",
    "print (\"Objective value:\", winner_score)\n",
    "SVGlayout = render.SVGlayout(winner,columns,10,associations)\n",
    "SVG(SVGlayout.inSVG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Dialogue design\n",
    "----------\n",
    "\n",
    "**Dialogues and forms** are similar to grid menus, with more constraints added to the relative placing of elements. We can model that in the task instance without changes to our objective function. Here, *n* elements must be placed on *n* slots, such that related items (e.g., label 'Password' should be next to its text entry field) are close to each other. Relative order of elements can be defined (e.g., 'OK' must occur only after manipulating some other field).\n",
    "\n",
    "| New aspect | Content  |\n",
    "|----------|------|\n",
    "| Task instance | Labels and entry fields, OK and Cancel button; Associations among element-pairs |\n",
    "\n",
    "We here consider a simple type with only 8 elements. Our case imitates that of a regular **login dialogue** such as this:\n",
    "<img src=\"imgs/login-dialogue.png\" width=50%>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "seed_layout = ['LName','LDom','EDom','OK','EName','LPWD','EPWD','Cancel']\n",
    "e_weights = {'LName':0.6,'EName':0.5,'LPWD':0.3,'EPWD':0.2,'LDom':0.1,'EDom':0.05,'OK':0.03,'Cancel':0.02}\n",
    "associations = {'LNameEName':1.0,'LPWDEPWD':1.0,'LDomEDom':1.0,'OKCancel':1.0}\n",
    "columns = 2\n",
    "\n",
    "# Optimization\n",
    "winner, winner_score = optimize(10000, anneal, seed_layout, columns, ST_and_myO, e_weights, associations, 2)\n",
    "\n",
    "# Present results\n",
    "print (\"Objective value:\", winner_score)\n",
    "SVGlayout = render.SVGlayout(winner,columns,0,associations)\n",
    "SVG(SVGlayout.inSVG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection</h3> \n",
    "<br>\n",
    "This solution was heavily constrained by the input data. It dictated how elements should go next to each other. The optimizer only had one real solution. What might be some other design problems that are heavily constrained by nature?\n",
    "</div>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 8. Web layouts\n",
    "----------\n",
    "\n",
    "The design of a **web layouts** (or, any graphical widget layout) is an order of magnitude harder problem. First, we start combining requirements we saw for menu (= some elements more important to be found quickly than others) and for dialogues (= some elements must be manipulated in some order). The canvas is larger, providing way more options on how to place items. There may also be empty slots (white space). Many control panels, dialogues, and control panels tasks are similar ([example](imgs/cockpit.png)). We also must reconsider our objective function. For example, it is well known that, or reasons of visual appeal, the alignment of element groups is important. To that end, we have implemented a simplified version of a **grid alignment function**:\n",
    "* [Miniukovich, A. (2015). Computational Inteface Aesthetics. Proc. CHI'15](https://www.researchgate.net/profile/Aliaksei_Miniukovich/publication/300726008_Computation_of_Interface_Aesthetics/links/57d7e0b108ae5f03b49812c7.pdf)\n",
    "\n",
    "Our revised design task is:\n",
    "\n",
    "| New aspect | Content  |\n",
    "|----------|------|\n",
    "| Objective function | Selection time, Incoherence, **Misalignment** |\n",
    "| Task instance | 36 elements; groups and associations |\n",
    "\n",
    "As our task instance, we consider a design like the [CHI2017 homepage](imgs/chi-homepage.png). To desing a task instance, we annotate different element types of the page with [prefixes as follows](imgs/chi-homepage-annotated.png)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A tri-objective function: selection time (ST) and incoherence (I) and misalignment (M)\n",
    "def ST_and_myO_and_M (layout, columns, o_inputs):\n",
    "    try:\n",
    "        ST = linear_ST(layout,columns,o_inputs[0:])\n",
    "    except:\n",
    "        ST = 2.0\n",
    "        pass\n",
    "    myO = myObjective(layout,columns,o_inputs[1:])\n",
    "    M = misalignment(layout,columns,o_inputs[1:])\n",
    "    return ST + 0.5 * myO + 0.5 * M\n",
    "\n",
    "# Returns a misalignment score M\n",
    "# - Inspired by grid quality metrics (Miniukovich et al. CHI'15)\n",
    "# - An important difference is that the number of grid lines is not minimized\n",
    "def misalignment (layout, columns, o_inputs):\n",
    "    M = len(o_inputs[0].keys())+1 \n",
    "    for i in range(0, len(layout)):\n",
    "        verticals = 0.0\n",
    "        horizontals = 0.0\n",
    "        related = 0.0\n",
    "        for j in range(0, len(layout)):\n",
    "            if i==j: \n",
    "                next\n",
    "            association = max(value for value in [0,o_inputs[0].get(layout[i]+layout[j]), o_inputs[0].get(layout[j] + layout[i])] if value is not None)\n",
    "            if association == 0:\n",
    "                next \n",
    "            else:\n",
    "                related += 1\n",
    "                verticals += vertically_aligned(columns, i,j)\n",
    "                horizontals += horizontally_aligned(columns, i,j)\n",
    "                #print i,j,verticals,horizontals\n",
    "        if related > 0:\n",
    "            M -= 1.0 * (max(verticals/related, horizontals/related) - min(verticals/related, horizontals/related))\n",
    "    return M\n",
    "\n",
    "# Helper functions for the new misalignment value\n",
    "# Returns 1 if two elements are vertically aligned\n",
    "def vertically_aligned (columns, i, j):\n",
    "    #print \"vert\", i,j, abs(j-i), abs(j-i) % columns\n",
    "    if (abs(j-i) % columns) == 0:\n",
    "        return 1.0\n",
    "    else:\n",
    "        return 0.0\n",
    "\n",
    "# returns 1 if two elements are horizontally aligned\n",
    "def horizontally_aligned (columns, i, j):\n",
    "    #print \"hor\", i,j, abs(j-i), int(j/columns), int(i/columns)\n",
    "    if (abs(j-i) <= columns) and (int(j/columns) == int(i/columns)):\n",
    "        return 1.0\n",
    "    else:\n",
    "        return 0.0\n",
    "\n",
    "# Task instance\n",
    "# We compose the task in several pieces to make its structure clearer\n",
    "menu = ['MExh','MOrg','MHom','MAuth','MProg','MSpon']\n",
    "menu_weights = {'MHom':0.5}\n",
    "menu_associations = {'MHomMAuth':0.5,'MAuthMProg':0.5,'MProgMSpon':0.5,'MSponMExh':0.5,'MExhMOrg':0.5}\n",
    "\n",
    "header = ['H3','H4','H5','H6','H1','H2']\n",
    "header_associations = {'H1H2':0.5,'H2H3':0.5,'H3H4':0.5,'H4H5':0.5,'H5H6':0.5}\n",
    "header_weights = {'H1':0.3}\n",
    "\n",
    "links = ['L1','L2','L3','L4']\n",
    "links_associations = {'L1L2':0.5, 'L2L3':0.5, 'L3L4': 0.5}\n",
    "links_weights = {'L1': 0.1}\n",
    "\n",
    "content = ['C1','C2','C3','C4','C5','C6','C7','C8','C9','C10','C11','C12','C13','C14','C15','C16','C17','C18','C19','C20']\n",
    "\n",
    "# We here put these parts together to a single task instance\n",
    "seed_layout = menu + header + links + content\n",
    "random.shuffle(seed_layout)\n",
    "weights = menu_weights.copy() \n",
    "weights.update(header_weights)\n",
    "weights.update(links_weights)\n",
    "associations = menu_associations.copy()\n",
    "associations.update(header_associations)\n",
    "associations.update(links_associations)\n",
    "columns = 6\n",
    "\n",
    "# Optimization\n",
    "winner, winner_score = optimize(100, anneal, seed_layout, columns, ST_and_myO_and_M, weights, associations,1)\n",
    "\n",
    "# Present results\n",
    "print (winner_score)\n",
    "SVGlayout = render.SVGlayout(winner,columns,0,associations)\n",
    "SVG(SVGlayout.inSVG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you see, obtaining a good solution is very hard! [A result after 100K iterations using simulated annealing is still not very good.](imgs/result-100000.png)\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection</h3> \n",
    "<br>\n",
    "How could you get better results in the same time?\n",
    "</div>\n",
    "</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "9c1e4146-b7e7-4a23-9045-ebd97f19282d"
    }
   },
   "source": [
    "# 9. Heuristic and metaheuristic methods\n",
    "---------\n",
    "\n",
    "Here are some possibilities to improve: (1) One could try using more clever heuristics. They might boost performance, but which one to use? (2) The code could also be optimized (e.g., parallelization, precomputed lookup tables for objective function, ...). This might bring x10 or even x100 speed-ups. (3) One could also use domain knowledge to rewrite the optimizer, such as exploiting the groups of associated elements as units. (4) How about reformulating the problem for exact methods, such as integer programming (see <a href=\"exact methods.ipynb\">Anna's notebook</a>)? (5) The hyperparameters of the method might be tuned. (6) We could look at diagnostics telling something about how the optimizer works and what the search landscape looks like. We here look at the first option (heuristics).\n",
    "\n",
    "## Heuristic strategies\n",
    "\n",
    "􏰀A heuristic is a technique that seeks good (near-optimal) solutions at a reasonable computational cost without guaranteeing optimality. Often heuristics are problem-specific and cannot be transfered from one problem to another. The essence of heuristic methods is search by *trial and error* (Glover 1986). Heuristic methods are practical for non-linear problems where analytical solutions might not be obtained, in theory or in practice.  \n",
    "\n",
    "Key dimensions are: \n",
    "\n",
    "**Stochastic vs. deterministic:** Stochastic methods exploit randomness in choosing how to sample the design space. They offer an effective way to obtain solutions of good quality. However, a different solution might be found even if starting from the same seed. Deterministic methods end up with the same solution every time when starting from the same location.\n",
    "* Random search with restarts (stochastic)\n",
    "* Particle swarm optimization (stochastic)\n",
    "* Hill-climbing (deterministic).\n",
    "\n",
    "**Local vs. global**: Local methods search within some defined neighborhood of a solution. In other words, a solution can [communicate](imgs/communication.png) with other solutions. These methods can converge to a *local* optimum quickly. However, without additional capabilities, they may not find the global optimum. \n",
    "\n",
    "**Greedy:** divides the construction of a solution into multiple subsequent decisions that are solved locally before moving to the next one. These decisions are taken in a certain order and are not revoked later. A greedy algorithm would in each step choose the best available solution. Idea: Start with empty solution; iteratively add elements, each adding highest gain; stop when no more elements can be added. \n",
    "\n",
    "**Trajectory-based vs. population-based:** Trajectory-based algorithms investigate an agent (or sample point, or candidate solution) at a time. Its progress can be traced as the trace of changes to that solution as search evolves. Population-based methods, by contrast, maintain a larger number of candidate solutions and choose between them.\n",
    "* Hill-climbing (trajectory-based)\n",
    "* Genetic algorithms (population-based)\n",
    "\n",
    "See also this [overview diagram](imgs/methods.png).\n",
    "\n",
    "## Metaheuristics\n",
    "\n",
    "Some (not all) heuristic methods can get easily 'trapped' in a local optimum. Metaheuristics are optimization methods that modify their use of heuristic methods as optimization progresses: \n",
    "\n",
    "> \"master strategy that guides and modifies other heuristics to produce solutions beyond those that are normally generated in a quest for local optimality\" (Glover and Laguna 1997)\n",
    "\n",
    "> “A metaheuristic is formally defined as an iterative generation process which guides a subordinate heuristic by combin- ing intelligently different concepts for ex- ploring and exploiting the search space, learning strategies are used to struc- ture information in order to find effici- ently near-optimal solutions.” (Osman and Laporte 1996).\n",
    "\n",
    "Metaheuristics implement some tradeoff between local and global strategies, and they are therefore generally considered global optimization methods.\n",
    "\n",
    "<img src=\"imgs/local-global.png\" width=\"95%\"></img>\n",
    "\n",
    "### Some general strategies\n",
    "\n",
    "**Intensification and diversification** (a.k.a. exploitation and exploration): [Diversification](imgs/diversification.png) means generating diverse solutions to explore the global search space, and [intensification](imgs/intensification.png) focusing on a local region to find the best solution in that region. Controlling the balance between these strategies improves the rate of convergence. However, this never guarantees finding the global optimum. Simulated annealing belongs includes hyper parameters (temperature) to control how to diversify/intensify search with time. \n",
    "\n",
    "**Memory:** One common strategy is to keep a memory of good or bad previously seen solutions. [Tabu search](https://en.wikipedia.org/wiki/Tabu_search) is a classic example ([illustration](imgs/tabu.png)).\n",
    "\n",
    "**Construction:** Instead of sampling full solutions, one can try to combine element-level ideas from previously visited solutions. A method can try to build a proxy model of the domain to 'guess' where previously unseen good solutions might be (e.g., ant colonies implement a pheromone function). ([Example ACO](https://www.youtube.com/watch?v=0NtB7PQxizw).)\n",
    "\n",
    "**Biology-inspired meta-heuristics** such as *genetic algorithms* and *ant colony optimization* borrow ideas from nature.\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Competition: Improve the web layout optimizer with a metaheuristics approach.</h3> \n",
    "<br>\n",
    "Design, implement, and test at least one metaheuristic method. Report how much it improves your objective value. For competition: The group that obtains the lowest objective value in fixed time wins. Ensure your code outputs the objective value when it finds a new incumbent. \n",
    "</div>\n",
    "</p>\n",
    "\n",
    "A visual summary of [some often used metaheuristics in Scholarpedia](http://www.scholarpedia.org/article/Metaheuristic_Optimization). \n",
    "\n",
    "### Hyperparameter tuning\n",
    "A general problem with metaheuristics is the selection of hyperparameters (e.g., temperature cooling schedule in simulated annealing). These can be set manually, or one can try to search the best parameters (often by using grid search) for the task at hand.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10. Advanced topics\n",
    "----------\n",
    "Real-world problems are often characterized by\n",
    "* Existence of some prior knowledge about good designs (think: your favorite app)\n",
    "* Uncertain or inexact data (think: a human as the source of inputs)\n",
    "* Problems are high-dimensional (think: a web layout or office tool)\n",
    "* Changing conditions of use (think: different user groups and needs, changing contents)\n",
    "* Expensive computations (think: a machine learning model to parametrize your objective function)\n",
    "* Changing criteria for good design (think: a designer ideating)\n",
    "\n",
    "## Sensitivity analysis\n",
    "\n",
    "An optimal design is a *particular* set of design choices. However, in some cases we would like to know if it is specific to the task instance. How much would a change in the input parameters 'perturb' the best design? In other words, how sensitive is it to the underlying task? Consider the Qwerty keyboard: It is an acceptable design for English, but if we assume a different user population, how much would it change?\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection</h3> \n",
    "<br>\n",
    "How sensitive is the toolbar optimizer to small changes in the association matrix? \n",
    "</div>\n",
    "</p>\n",
    "\n",
    "## Robust optimization\n",
    "\n",
    "Robust optimization takes a more proactive approach to optimization. We generate a series of solutions and improve them to make the progressively less sensitive to task instances. We call the set of task instances a **scenario set**. The most robust design is the one that achieves the best objective value across the whole set. \n",
    "\n",
    "## Multi-objective optimization\n",
    "\n",
    "In real-world design problems, objectives often conflict with each other. \n",
    "A perfect multi-objective solution that is the best in terms of all objectives is mostly unattainable.\n",
    "\n",
    "#### Pareto optimality\n",
    "\n",
    "We must first revisit our what 'optimality' means in the case of multi-objective optimization. We need a new concept: A **pareto-optimal design** is one which cannot be changed to make one objective better without making at least one other objective worse off. [Pareto frontier](imgs/pareto.jpg) is the set of all pareto optimal designs. Finding Pareto frontiers is useful for making informed choices in design, focusing on the particular parameters that differentiate designs (as opposed to all of them).\n",
    "\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Exercise 5: Pareto frontier.</h3> \n",
    "<br>\n",
    "Implement a method to print all pareto-optimal designs in some easily solved problem. Tip: You can first compute objective values for a set of feasible designs, and then determine the Pareto frontier.\n",
    "</div>\n",
    "</p>\n",
    "\n",
    "\n",
    "\n",
    "#### Multi-objective function: Strategies\n",
    "\n",
    "Thus far, we have used the **weighted sum approach**, which collapses the multiple objectives into a single function by the use of weights. This has known limitations, such as fragility to weights. In addition, selecting a weight is a problem. Trying to fix a problem by changing a weight can cause unwanted [side effects](http://i.imgur.com/x501fi0.gifv). \n",
    "\n",
    "Like with heuristics, the choice of another function is case-specific. Some strategies that one can try ([see also this paper](http://web.cse.ohio-state.edu/~parent.1/classes/788/Au10/OptimizationPapers/MultiObjective/journal_survey.pdf)):\n",
    "* Inverted sum: as opposed to maximizing goodness, we want to minimize 'undesirability' (inverse of goodness value).\n",
    "* Normalization: Makes objectives less sensitive to weight selection.\n",
    "* Function transformations: For example, a logarithmic transformation de-emphasises one end of the variable.\n",
    "* The weighted min-max method (the weighted Tchebycheff): \"ensure that a minimum is reached, maximize after that\"\n",
    "* Bounded objective: We place a maximum and minimum level for each objective, which leads to prioritizing the most important function first. \n",
    "* Lexicographic method: Rank objectives and optimize them one at a time by starting from the most important objective. \n",
    "* Goal programming: Designer sets goals for each objective that should be ideally obtained. The best design is closest to those references.\n",
    "* Goal attainment method: Rank objectives, place goals to them, solve them one at a time in the ranked order, moving on to next objective after attaining the goal.\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Exercise 6: Multi-objective approach.</h3> \n",
    "<br>\n",
    "Study, design, implement, and test one multi-objective approach. \n",
    "</div>\n",
    "</p>\n",
    "\n",
    "## Interactive optimization\n",
    "\n",
    "The standard \"fire and forget\" paradigm of optimization methods is irreconcilable with design practice: A designer cannot be asked to provide all inputs to an optimizer within a decimal point and come back later for an answer. However, a considerable amount of information is needed to design anything. Which aspects can be automated and which should be left to the designer?\n",
    "\n",
    "[Interactive design tools](imgs/interactive-optimization.png) automate some aspects of design work but leave others to the designer. This introduces a tradeoff between the designer's and the optimizer's objectives. The **full-control** concept (MenuOptimizer) requires objectives and constraints to be specified upon any change in a designer's task. A **minimum-effort optimization** approach tries to infer some the design task and then reacts, e.g. by suggesting redesigns.\n",
    " \n",
    "Some dimensions of interactive optimization methods:\n",
    "* **Control before, during, or after optimization**\n",
    "* **Proactive vs. reactive optimization**: Should the optimizer try to encourage or even override the designer to improve their (assumed) quality?\n",
    "* **Locus of control:** objectives, constraints, examples?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11. Discussion\n",
    "\n",
    "---------------------\n",
    "\n",
    "This unit has introduced some basic concepts of model-based combinatorial optimization using a plain vanilla black-box optimizer demonstrated in layout design problems. \n",
    "\n",
    "Combinatorial optimization offers a powerful tool for interface design. \n",
    "Applications in realistic problems depend on success in: 1) formal definition of a design task, 2) encoding of meaningful design objectives and knowledge in the objective function, 3) choosing and tuning an efficient solver. After this, the algorithm can be used to support the creative and problem-solving activities of a designer. Different task instances can be solved with the approach once set up.\n",
    "\n",
    "The upside of this approach is that almost any problem can be attacked. The downside is that no guarantees can be given to the solution. The selection of a method is 'black art'. \n",
    "\n",
    "To scale up, at least the following challenges stand out:\n",
    "* Acquiring meaningful input data empirically or by using machine learning models\n",
    "* Calibrating parameters of objective functions (e.g., empirically)\n",
    "* Justifying the objective functions and design tasks against some empirical data or literature\n",
    "* Constructing an efficient solver\n",
    "* Developing a tool for designers to work with the optimization system\n",
    "* Carrying out diagnostics and visualizations to understand the results\n",
    "* Choosing and tuning a suitable multi-objective function\n",
    "* Carrying out critical empirical evaluation of optimized interfaces.\n",
    "\n",
    "<p>\n",
    "<div class=\"alert alert-block alert-success\"> \n",
    "<h3>Reflection.</h3> \n",
    "<br>\n",
    "What were the most surprising, hardest, or exciting things you learned?</div>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Selected further readings\n",
    "\n",
    "* Karrenbauer, A., Oulasvirta, A. (2014). [Keyboard optimization with integer programming](http://resources.mpi-inf.mpg.de/keyboardoptimization/). Proc. UIST'14. \n",
    "* Oulasvirta, A. (2017). [User Interface Design with Combinatorial Optimization](http://users.comnet.aalto.fi/oulasvir/pubs/UIdesign_with_combinatorial_optimization_IEEEComputer_2017_Oulasvirta.pdf), IEEE Computer 50 (1).\n",
    "* Bailly, G. et al. (2013). [MenuOptimizer:  Interactive Optimization of Menu Systems](https://www.gillesbailly.fr/publis/BAILLY_MenuOptimizer.pdf), Proc. UIST 2013. \n",
    "* Todi, K. et al. [Sketchplorer: Sketch and explore with an optimizer](http://www.kashyaptodi.com/sketchplore/), Proc. DIS'16. \n",
    "* Micallef, L. et al. (2017). [Towards perceptual optimization of the visual design of scatterplots](http://ieeexplore.ieee.org/document/7864468/), IEEE TGCV\n",
    "* Feit, A., Oulasvirta, A. (2105). [Exercises in assignment problems in keyboard design](https://www.slideshare.net/oulasvir/modelbased-user-interface-optimization-part-ii-letter-assignment-at-sicsa-summer-school-on-computational-interaction-2015?qid=4320d34c-8c76-42c8-9cd3-69b2260dbc3a&v=&b=&from_search=2), Lecture notes from 1st Computational Interaction Summer School 2015, Glasgow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}