Updated documentation with new changes

dfols/__init__.py

@@ -39,7 +39,7 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 # DFO-LS version
-__version__ = '1.5.0'
+__version__ = '1.5.1'
 
 # Main solver & exit flags
 from .solver import *

dfols/tests/test_model.py

@@ -211,8 +211,8 @@ def runTest(self):
         self.assertTrue(array_compare(model.rsave, np.array([0.0, 0.0])), 'Wrong rsave after saving 3')
         self.assertAlmostEqual(model.objsave, 0.0, 'Wrong fsave after saving 3')
         self.assertEqual(model.nsamples_save, 2, 'Wrong nsamples_save after saving 3')
-        self.assertEqual(model.eval_num_save, 7, 'Wrong eval_num_save after saving 2')
-        self.assertIsNone(model.jacsave_eval_nums, 'jacsave_eval_nums not none after saving 2')
+        self.assertEqual(model.eval_num_save, 7, 'Wrong eval_num_save after saving 3')
+        self.assertIsNone(model.jacsave_eval_nums, 'jacsave_eval_nums not none after saving 3')
 
 
 class TestAveraging(unittest.TestCase):

docs/advanced.rst

@@ -24,7 +24,7 @@ Initialization of Points
 ------------------------
 * :code:`init.random_initial_directions` - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default as of version 1.2 is :code:`False`.
 * :code:`init.random_directions_make_orthogonal` - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is :code:`True`.
-* :code:`init.run_in_parallel` - If using random directions, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`.
+* :code:`init.run_in_parallel` - If using random directions or non-random with input :code:`npt` at most :code:`len(x0)+1`, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`.
 
 Trust Region Management
 -----------------------

docs/build/html/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 8fe9d19f5645917bc7a4a240ff52ff51
+config: 57a11b064dfa88ccc9f8feaa9e7f040a
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/build/html/_sources/advanced.rst.txt

@@ -24,7 +24,7 @@ Initialization of Points
 ------------------------
 * :code:`init.random_initial_directions` - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default as of version 1.2 is :code:`False`.
 * :code:`init.random_directions_make_orthogonal` - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is :code:`True`.
-* :code:`init.run_in_parallel` - If using random directions, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`.
+* :code:`init.run_in_parallel` - If using random directions or non-random with input :code:`npt` at most :code:`len(x0)+1`, whether or not to ask for all :code:`objfun` to be evaluated at all points without any intermediate processing. Default is :code:`False`.
 
 Trust Region Management
 -----------------------

docs/build/html/_sources/history.rst.txt

@@ -66,3 +66,8 @@ Version 1.5.0 (11 Sep 2024)
 ---------------------------
 * Add support for (possibly nonsmooth) regularizer term.
 * Drop warning about infeasible initial point if the point is on an upper/lower bound.
+
+Version 1.5.1 (10 Oct 2024)
+---------------------------
+* Add return values :code:`soln.xmin_eval_num` and :code:`soln.jacmin_eval_nums`
+* Allow option for parallel initial evaluations for non-random directions if :code:`npt` not too large

docs/build/html/_sources/userguide.rst.txt

@@ -45,6 +45,8 @@ The output of :code:`dfols.solve` is an object containing:
 * :code:`soln.flag` - an exit flag, which can take one of several values (listed below), an Integer.
 * :code:`soln.msg` - a description of why the algorithm finished, a String.
 * :code:`soln.diagnostic_info` - a table of diagnostic information showing the progress of the solver, a Pandas DataFrame.
+* :code:`soln.xmin_eval_num` - an integer representing which evaluation point (i.e. same as :code:`soln.nx`) gave the solution :code:`soln.x`. Evaluation counts are 1-indexed, to match the logging information. 
+* :code:`soln.jacmin_eval_nums` - a NumPy integer array of length :code:`npt` with the evaluation point numbers (i.e. same as :code:`soln.nx`) used to build :code:`soln.jacobian` via linear interpolation to the residual values at these points. Evaluation counts are 1-indexed, to match the logging information. This array will usually, but not always, include :code:`soln.xmin_eval_num`.
 
 The possible values of :code:`soln.flag` are defined by the following variables:
 
@@ -149,8 +151,10 @@ Note that DFO-LS is a randomized algorithm: in its first phase, it builds an int
       Residual vector = [0. 0.]
       Objective value f(xmin) = 0
       Needed 33 objective evaluations (at 33 points)
-      Approximate Jacobian = [[-2.00180000e+01  1.00000000e+01]
-       [-1.00000000e+00  8.19971362e-16]]
+      Approximate Jacobian = [[-1.9982000e+01  1.0000000e+01]
+       [-1.0000000e+00  1.0079924e-14]]
+      Solution xmin was evaluation point 33
+      Approximate Jacobian formed using evaluation points [29 31 32]
       Exit flag = 0
       Success: Objective is sufficiently small
       ****************************
@@ -176,11 +180,13 @@ DFO-LS correctly finds the solution to the constrained problem:
   
       ****** DFO-LS Results ******
       Solution xmin = [0.9  0.81]
-      Residual vector = [3.10862447e-14 1.00000000e-01]
+      Residual vector = [0.  0.1]
       Objective value f(xmin) = 0.01
-      Needed 58 objective evaluations (at 58 points)
-      Approximate Jacobian = [[-1.79999999e+01  9.99999998e+00]
-       [-1.00000000e+00  8.62398179e-10]]
+      Needed 56 objective evaluations (at 56 points)
+      Approximate Jacobian = [[-1.79999999e+01  1.00000000e+01]
+       [-1.00000000e+00 -5.15519307e-10]]
+      Solution xmin was evaluation point 42
+      Approximate Jacobian formed using evaluation points [55 42 54]
       Exit flag = 0
       Success: rho has reached rhoend
       ****************************
@@ -212,8 +218,8 @@ And for the simple bounds example we can now see each evaluation of :code:`objfu
       Function eval 2 at point 2 has f = 14.337296 at x = [-1.08  0.85]
       Function eval 3 at point 3 has f = 55.25 at x = [-1.2   0.73]
       ...
-      Function eval 57 at point 57 has f = 0.010000001407575 at x = [0.89999999 0.80999999]
-      Function eval 58 at point 58 has f = 0.00999999999999997 at x = [0.9  0.81]
+      Function eval 55 at point 55 has obj = 0.0100000000000225 at x = [0.9        0.80999998]
+      Function eval 56 at point 56 has obj = 0.01 at x = [0.9  0.81]
       Did a total of 1 run(s)
 
 If we wanted to save this output to a file, we could replace the above call to :code:`logging.basicConfig()` with
@@ -310,6 +316,8 @@ gave an increase in the model. This is common in the case where multiple constra
       Needed 10 objective evaluations (at 10 points)
       Approximate Jacobian = [[-1.79826221e+01  1.00004412e+01]
        [-1.00000000e+00 -1.81976605e-15]]
+      Solution xmin was evaluation point 5
+      Approximate Jacobian formed using evaluation points [8 5 9]
       Exit flag = 5
       Warning (trust region increase): Either multiple constraints are active or trust region step gave model increase
       ****************************
@@ -395,6 +403,8 @@ The solution found by DFO-LS is:
        ...
        [ 4.50000000e+01  4.60000000e+01  4.70000000e+01  4.80000000e+01
          4.90000000e+01]]
+      Solution xmin was evaluation point 34
+      Approximate Jacobian formed using evaluation points [30 32 29 31 33 27]
       Exit flag = 0
       Success: rho has reached rhoend
       ****************************
@@ -404,7 +414,7 @@ Note that many LASSO-type algorithms can produce a solution with many entries be
 
 Example: Noisy Objective Evaluation
 -----------------------------------
-As described in :doc:`info`, derivative-free algorithms such as DFO-LS are particularly useful when :code:`objfun` has noise. Let's modify the previous example to include random noise in our objective evaluation, and compare it to a derivative-based solver:
+As described in :doc:`info`, derivative-free algorithms such as DFO-LS are particularly useful when :code:`objfun` has noise. Let's modify the previous example to include random noise in our objective evaluation, and compare it to SciPy's derivative-based solver (the below results came from using SciPy v1.13.0):
 
   .. code-block:: python
   
@@ -463,25 +473,27 @@ The output of this is:
       objfun(x0) = [-4.39545837  2.20903317]
       
       ****** DFO-LS Results ******
-      Solution xmin = [1.         1.00000003]
-      Residual vector = [ 1.59634974e-07 -4.63036198e-09]
-      Objective value f(xmin) = 2.550476524e-14
-      Needed 53 objective evaluations (at 53 points)
-      Approximate Jacobian = [[-1.98196347e+01  9.90335675e+00]
-       [-1.01941978e+00  4.24991776e-05]]
+      Solution xmin = [1.00000001 1.00000002]
+      Residual vector = [ 5.17481720e-09 -1.04150014e-08]
+      Objective value f(xmin) = 1.352509879e-16
+      Needed 35 objective evaluations (at 35 points)
+      Approximate Jacobian = [[-1.98079840e+01  1.00105722e+01]
+       [-9.93887907e-01 -3.06567570e-04]]
+      Solution xmin was evaluation point 35
+      Approximate Jacobian formed using evaluation points [30 33 34]
       Exit flag = 0
       Success: Objective is sufficiently small
       ****************************
       
       
       ** SciPy results **
-      Solution xmin = [-1.20000087  1.00000235]
-      Objective value f(xmin) = 23.95535774
-      Needed 6 objective evaluations
+      Solution xmin = [-1.2  1. ]
+      Objective value f(xmin) = 23.83907501
+      Needed 5 objective evaluations
       Exit flag = 3
       `xtol` termination condition is satisfied.
 
-DFO-LS is able to find the solution with 20 more function evaluations as in the noise-free case. However SciPy's derivative-based solver, which has no trouble solving the noise-free problem, is unable to make any progress.
+DFO-LS is able to find the solution, but SciPy's derivative-based solver, which has no trouble solving the noise-free problem, is unable to make any progress.
 
 As noted above, DFO-LS has an input parameter :code:`objfun_has_noise` to indicate if :code:`objfun` has noise in it, which it does in this case. Therefore we can call DFO-LS with
 
@@ -495,11 +507,13 @@ Using this setting, we find the correct solution faster:
   
       ****** DFO-LS Results ******
       Solution xmin = [1. 1.]
-      Residual vector = [-4.06227943e-08  2.51525603e-10]
-      Objective value f(xmin) = 1.650274685e-15
-      Needed 29 objective evaluations (at 29 points)
-      Approximate Jacobian = [[-1.99950530e+01  1.00670067e+01]
-       [-9.96161167e-01 -2.41166495e-04]]
+      Residual vector = [-6.56093684e-10 -1.17835345e-10]
+      Objective value f(xmin) = 4.443440912e-19
+      Needed 28 objective evaluations (at 28 points)
+      Approximate Jacobian = [[-1.98649933e+01  9.93403044e+00]
+       [-9.93112150e-01  5.78830812e-03]]
+      Solution xmin was evaluation point 28
+      Approximate Jacobian formed using evaluation points [27 25 26]
       Exit flag = 0
       Success: Objective is sufficiently small
       ****************************
@@ -551,20 +565,22 @@ The output of this is (noting that DFO-LS moves :math:`x_0` to be far away enoug
   
       ****** DFO-LS Results ******
       Solution xmin = [ 4.98830861e+02 -1.01256863e-01]
-      Residual vector = [-0.1816709   0.06098397  0.76276301  0.11962354 -0.26589796 -0.59788814
-       -1.02611897 -1.5123537  -1.56145452 -1.63266662]
+      Residual vector = [-0.1816709   0.06098396  0.76276296  0.11962351 -0.26589799 -0.59788816
+       -1.02611898 -1.51235371 -1.56145452 -1.63266662]
       Objective value f(xmin) = 9.504886892
-      Needed 79 objective evaluations (at 79 points)
-      Approximate Jacobian = [[-9.12897463e-01 -4.09843514e+02]
-       [-8.59085679e-01 -6.42808544e+02]
-       [-2.47252555e-01 -1.70205419e+03]
-       [-1.34676365e-01 -1.33017181e+03]
-       [-8.71355033e-02 -1.04752848e+03]
-       [-5.75304364e-02 -8.09280752e+02]
-       [-2.83184867e-02 -4.97239623e+02]
-       [-2.22992989e-03 -6.70749826e+01]
-       [-5.24129962e-04 -1.95045269e+01]
-       [-2.65956876e-04 -1.07858081e+01]]
+      Needed 111 objective evaluations (at 111 points)
+      Approximate Jacobian = [[-9.12901055e-01 -4.09843504e+02]
+       [-8.59087363e-01 -6.42808534e+02]
+       [-2.47254068e-01 -1.70205403e+03]
+       [-1.34676757e-01 -1.33017163e+03]
+       [-8.71358948e-02 -1.04752831e+03]
+       [-5.75309286e-02 -8.09280596e+02]
+       [-2.83185935e-02 -4.97239504e+02]
+       [-2.22997879e-03 -6.70749550e+01]
+       [-5.24146460e-04 -1.95045170e+01]
+       [-2.65964661e-04 -1.07858021e+01]]
+      Solution xmin was evaluation point 111
+      Approximate Jacobian formed using evaluation points [104 109 110]
       Exit flag = 0
       Success: rho has reached rhoend
       ****************************
@@ -641,11 +657,13 @@ The output of this is
   
       ****** DFO-LS Results ******
       Solution xmin = [ 0.09777309 -2.32510588]
-      Residual vector = [-1.45394186e-09 -1.95108811e-08]
-      Objective value f(xmin) = 3.827884295e-16
+      Residual vector = [-1.38601752e-09 -1.70204653e-08]
+      Objective value f(xmin) = 2.916172822e-16
       Needed 13 objective evaluations (at 13 points)
-      Approximate Jacobian = [[ 3.32499552  0.90216381]
-       [10.22664908 -1.00061604]]
+      Approximate Jacobian = [[ 3.32527052  0.90227531]
+       [10.22943034 -0.99958226]]
+      Solution xmin was evaluation point 13
+      Approximate Jacobian formed using evaluation points [ 8 11 12]
       Exit flag = 0
       Success: Objective is sufficiently small
       ****************************

docs/build/html/_static/documentation_options.js

@@ -1,5 +1,5 @@
 const DOCUMENTATION_OPTIONS = {
-    VERSION: '1.5.0',
+    VERSION: '1.5.1',
     LANGUAGE: 'en',
     COLLAPSE_INDEX: false,
     BUILDER: 'html',

docs/build/html/advanced.html

@@ -4,7 +4,7 @@
   <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
 
   <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  <title>Advanced Usage &mdash; DFO-LS v1.5.0 documentation</title>
+  <title>Advanced Usage &mdash; DFO-LS v1.5.1 documentation</title>
       <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
       <link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=19f00094" />
 
@@ -15,7 +15,7 @@
 
         <script src="_static/jquery.js?v=5d32c60e"></script>
         <script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
-        <script src="_static/documentation_options.js?v=e0a75244"></script>
+        <script src="_static/documentation_options.js?v=44dfd65d"></script>
         <script src="_static/doctools.js?v=888ff710"></script>
         <script src="_static/sphinx_highlight.js?v=dc90522c"></script>
         <script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
@@ -38,7 +38,7 @@
             DFO-LS
           </a>
               <div class="version">
-                1.5.0
+                1.5.1
               </div>
 <div role="search">
   <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
@@ -103,7 +103,7 @@
   <section id="advanced-usage">
 <h1>Advanced Usage<a class="headerlink" href="#advanced-usage" title="Link to this heading"></a></h1>
 <p>This section describes different optional user parameters available in DFO-LS.</p>
-<p>In the last section (<a class="reference internal" href="userguide.html"><span class="doc">Using DFO-LS</span></a>), we introduced <code class="code docutils literal notranslate"><span class="pre">dfols.solve()</span></code>, which has the optional input <code class="code docutils literal notranslate"><span class="pre">user_params</span></code>. This is a Python dictionary of user parameters. We will now go through the settings which can be changed in this way. More details are available in the papers <a class="reference internal" href="#cfmr2018" id="id1"><span>[CFMR2018]</span></a>, <a class="reference internal" href="#hr2022" id="id2"><span>[HR2022]</span></a> and <a class="reference internal" href="#llr2024" id="id3"><span>[LLR2024]</span></a>.</p>
+<p>In the last section (<a class="reference internal" href="userguide.html"><span class="doc">Using DFO-LS</span></a>), we introduced <code class="code docutils literal notranslate"><span class="pre">dfols.solve()</span></code>, which has the optional input <code class="code docutils literal notranslate"><span class="pre">user_params</span></code>. This is a Python dictionary of user parameters. We will now go through the settings which can be changed in this way. More details are available in the papers <a class="reference internal" href="userguide.html#cfmr2018" id="id1"><span>[CFMR2018]</span></a>, <a class="reference internal" href="userguide.html#hr2022" id="id2"><span>[HR2022]</span></a> and <a class="reference internal" href="userguide.html#llr2024" id="id3"><span>[LLR2024]</span></a>.</p>
 <p>The default values, used if no override is given, in some cases vary depending on whether <code class="code docutils literal notranslate"><span class="pre">objfun</span></code> has stochastic noise; that is, whether evaluating <code class="code docutils literal notranslate"><span class="pre">objfun(x)</span></code> several times at the same <code class="code docutils literal notranslate"><span class="pre">x</span></code> gives the same result or not. Whether or not this is the case is determined by the <code class="code docutils literal notranslate"><span class="pre">objfun_has_noise</span></code> input to <code class="code docutils literal notranslate"><span class="pre">dfols.solve()</span></code> (and not by inspecting <code class="code docutils literal notranslate"><span class="pre">objfun</span></code>, for instance).</p>
 <section id="general-algorithm-parameters">
 <h2>General Algorithm Parameters<a class="headerlink" href="#general-algorithm-parameters" title="Link to this heading"></a></h2>
@@ -128,7 +128,7 @@ <h2>Initialization of Points<a class="headerlink" href="#initialization-of-point
 <ul class="simple">
 <li><p><code class="code docutils literal notranslate"><span class="pre">init.random_initial_directions</span></code> - Build the initial interpolation set using random directions (as opposed to coordinate directions). Default as of version 1.2 is <code class="code docutils literal notranslate"><span class="pre">False</span></code>.</p></li>
 <li><p><code class="code docutils literal notranslate"><span class="pre">init.random_directions_make_orthogonal</span></code> - If building initial interpolation set with random directions, whether or not these should be orthogonalized. Default is <code class="code docutils literal notranslate"><span class="pre">True</span></code>.</p></li>
-<li><p><code class="code docutils literal notranslate"><span class="pre">init.run_in_parallel</span></code> - If using random directions, whether or not to ask for all <code class="code docutils literal notranslate"><span class="pre">objfun</span></code> to be evaluated at all points without any intermediate processing. Default is <code class="code docutils literal notranslate"><span class="pre">False</span></code>.</p></li>
+<li><p><code class="code docutils literal notranslate"><span class="pre">init.run_in_parallel</span></code> - If using random directions or non-random with input <code class="code docutils literal notranslate"><span class="pre">npt</span></code> at most <code class="code docutils literal notranslate"><span class="pre">len(x0)+1</span></code>, whether or not to ask for all <code class="code docutils literal notranslate"><span class="pre">objfun</span></code> to be evaluated at all points without any intermediate processing. Default is <code class="code docutils literal notranslate"><span class="pre">False</span></code>.</p></li>
 </ul>
 </section>
 <section id="trust-region-management">