Release 0.3.2 (#1121)

* Use cloudpickle for pickling dynesty sampler (#1094)

Cloudpickle handles ABC-derived classes properly, whereas the latest dill version on pypi (0.3.6) does not.

Cloudpickle is already installed with pypesto, so there seems to be no need to get dill on top.

* Restrict fval magnitude in waterfall with order_by_id (#1090)

* restrict waterfall with order_by_id to fvals < 1e100
* add constant for parameters/waterfall plot maximum value

* Add startpoint_method to Problem (#1093)

It's not intuitive that `x_guesses` is part of `Problem`, but `startpoint_method`s are handled separately. See also discussion in #1017.

This patch
* adds `startpoint_method` to `Problem`
* during PEtab import, sets `Problem.startpoint_method` based on the PEtab problem
* deprecates `startpoint_method` arguments where also `Problem` is passed

Closes #1035

* Small initialize fix (#1095)

Censoring bounds should not be reinitialized when calculators are reinitialized.

* Added Falco et al to bib. (#1100)

* fix storage (#1099)

* Hierarchical parameter plot fix (#1106)

* Fix parameter plot

If, in any case (because some inner optimization failed, simulation failed, or some other reason) and there is a`optimize_result`dictionary in `result.optmize_result.list` which doesn't contain the `INNER_PARAMETERS` key and its item, the parameter plot would fail at line 381.

This is a fix to make it robust to this. If any of the `optimize_result` objects has a `INNER_PARAMETER` key, then the other ones will be filled with NaN values, and plotted.

* Dilan review change

Co-authored-by: Dilan Pathirana <[email protected]>

* Update pypesto/visualize/parameters.py

Co-authored-by: Daniel Weindl <[email protected]>

* Add comments

* Add comment

Testing the quality check

* Fix quality check error

---------

Co-authored-by: Dilan Pathirana <[email protected]>
Co-authored-by: Daniel Weindl <[email protected]>

* Fix startpoint sampling for hierarchical optimization (#1105)

* Fix startpoint sampling for hierarchical optimization

See #1096

* fixed pars

* fixup

---------

Co-authored-by: Doresic <[email protected]>

* PetabJL integration (#1089)

* SingleCore engine works, multicore has problems with pickling even though manual pickling works

* intermediate commit

* Working tests.

* small changes needs verification

* created base test to check for integration into python.

* Added julia module file for test

* Added test for objective function evaluation

* Added values for evaluation

* Added installation of packages to Workflow.

* Readded the Julia files that for some reason were deleted.

* Removed PetabJL from default pypesto.petab import

* Temporariy added amici as dependency. Needs to be taken care of appropriately.

* Moved Importer to objective/julia

* Deactivated precompilation in test

* Integrated suggestions

* add comments

* add sundials to ci

---------

Co-authored-by: Yannik Schälte <[email protected]>

* Fix #1108 (#1109)

* enable setting the y limits of a waterfall plot by using the y_limits argument
* use default y limits for zoomed in starts
* fix #1108

Co-authored-by: Paul Jonas Jost <[email protected]>

* SacessOptimizer: retry reading, delay deleting (#1110)


* delete temporary files only after *all* were read successfully
* retry reading temp file upon error to work around potential filesystem latency issues

* SacessOptimizer: Fix logging with multiprocessing (#1112)

* SacessOptimizer: Fix logging with multiprocessing

* Fix: `ValueError: setting an array element with a sequence.`

* Other platform tests (#1113)

* blind attempt mac

* brew installs

* try python 3.11 numba

* try windows

* fix cache

* stuff

* stuff

* stuff

* stuff

* minimal test

* add basic workflow test

* Update .github/workflows/ci.yml

Co-authored-by: Daniel Weindl <[email protected]>

* close figures

---------

Co-authored-by: Daniel Weindl <[email protected]>

* SacessOptimizer: tmpdir option (#1115)

SacessOptimizer:
* Add option to pass a custom directory for temporary files
* Make the default temporary file paths unique, to avoid name collisions when running multiple SacessOptimizer instances from within the same working directory

* Notebook on differences (#1098)

* Notebook on differences

* simplify some parts; add some more details and visualizations

* minor edit

* add workflow comparison to docs

* integrated suggestions

* integrated suggestions 2

* fixup

* removed personal info.
changed nlop objective appropriatly

---------

Co-authored-by: Yannik Schälte <[email protected]>
Co-authored-by: Yannik Schaelte <[email protected]>

* Documentation fixes (#1120)

* document hierarchical optimization sources; add aesara+jax to objective docs

* document origin of the mh + pt samplers

* fix typo

* fix typo

* Update pypesto/sample/parallel_tempering.py

Typo

* streamline doc deps

* update setup req versions

* try cyipopt fix

* test

* tesT

* test

* fix test_visualize::test_optimization_scatter_with_x_None

* test

* fixup

---------

Co-authored-by: Paul Jonas Jost <[email protected]>

* update changelog+version

* Changed Codeowners and contact to accomodate changes in personell (#1123)

* Updated version and Changelog (#1122)

* Updated version and Changelog

* Updated Changelog

* Update CHANGELOG.rst

---------

Co-authored-by: Daniel Weindl <[email protected]>
Co-authored-by: Maren Philipps <[email protected]>
Co-authored-by: Doresic <[email protected]>
Co-authored-by: Polina Lakrisenko <[email protected]>
Co-authored-by: Dilan Pathirana <[email protected]>
Co-authored-by: Yannik Schälte <[email protected]>
Co-authored-by: Yannik Schaelte <[email protected]>

.github/CODEOWNERS

@@ -4,43 +4,43 @@
 * @ICB-DCM/pypesto-maintainers
 
 /doc/example/hdf5_storage.ipynb @PaulJonasJost
-/doc/example/hierarchical.ipynb @dilpath @dweindl @yannikschaelte
-/doc/example/julia.ipynb @yannikschaelte
+/doc/example/hierarchical.ipynb @dilpath @dweindl
+/doc/example/julia.ipynb @PaulJonasJost
 /doc/example/model_selection.ipynb @dilpath
 /doc/example/petab_import.ipynb @dweindl @FFroehlich
-/doc/example/sampler_study.ipynb @dilpath @yannikschaelte
-/doc/example/sampling_diagnostics.ipynb @dilpath @yannikschaelte
+/doc/example/sampler_study.ipynb @dilpath
+/doc/example/sampling_diagnostics.ipynb @dilpath
 /doc/example/store.ipynb @PaulJonasJost
 /doc/example/synthetic_data.ipynb @dilpath
 /docker/ @dweindl
-/pypesto/engine/ @yannikschaelte
+/pypesto/engine/ @PaulJonasJost
 /pypesto/engine/mpi_pool.py @PaulJonasJost
 /pypesto/ensemble/ @dilpath
-/pypesto/hierarchical/ @dilpath @dweindl @yannikschaelte
+/pypesto/hierarchical/ @dilpath @dweindl
 /pypesto/hierarchical/optimal_scaling_approach/ @doresic
 /pypesto/history/ @PaulJonasJost
-/pypesto/objective/ @yannikschaelte
-/pypesto/objective/amici/ @yannikschaelte @dweindl @FFroehlich
+/pypesto/objective/ @PaulJonasJost
+/pypesto/objective/amici/ @PaulJonasJost @dweindl @FFroehlich
 /pypesto/objective/jax/ @FFroehlich
 /pypesto/objective/aesara/ @FFroehlich
-/pypesto/optimize/ @yannikschaelte
+/pypesto/optimize/ @PaulJonasJost
 /pypesto/petab/ @dweindl @FFroehlich
 /pypesto/predict/ @PaulJonasJost @dilpath
-/pypesto/problem/ @yannikschaelte
+/pypesto/problem/ @PaulJonasJost
 /pypesto/profile/ @PaulJonasJost
-/pypesto/result/ @yannikschaelte
-/pypesto/sample/ @dilpath @yannikschaelte
+/pypesto/result/ @PaulJonasJost
+/pypesto/sample/ @dilpath
 /pypesto/select/ @dilpath
-/pypesto/startpoint/ @yannikschaelte
+/pypesto/startpoint/ @PaulJonasJost
 /pypesto/store/ @PaulJonasJost
 /pypesto/visualize/ @plakrisenko @stephanmg
-/test/base/ @yannikschaelte
-/test/doc/ @yannikschaelte
-/test/hierarchical/ @dilpath @dweindl @yannikschaelte
-/test/julia/ @yannikschaelte
-/test/optimize/ @yannikschaelte
+/test/base/ @PaulJonasJost
+/test/doc/ @PaulJonasJost
+/test/hierarchical/ @dilpath @dweindl
+/test/julia/ @PaulJonasJost
+/test/optimize/ @PaulJonasJost
 /test/petab/ @dweindl @FFroehlich
 /test/profile/ @PaulJonasJost
-/test/sample/ @dilpath @yannikschaelte
+/test/sample/ @dilpath
 /test/select/ @dilpath
 /test/visualize/ @plakrisenko @stephanmg

.github/workflows/ci.yml

@@ -17,8 +17,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        # update to 3.11 when numba allows (needed in umap)
-        python-version: ['3.9', '3.10']
+        python-version: ['3.9', '3.11']
 
     steps:
     - name: Check out repository
@@ -48,6 +47,65 @@ jobs:
         token: ${{ secrets.CODECOV_TOKEN }}
         file: ./coverage.xml
 
+  mac:
+    runs-on: macos-latest
+    strategy:
+      matrix:
+        python-version: ['3.11']
+
+    steps:
+    - name: Check out repository
+      uses: actions/checkout@v3
+
+    - name: Prepare python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Cache
+      uses: actions/cache@v3
+      with:
+        path: ~/.cache
+        key: ${{ runner.os }}-${{ matrix.python-version }}-ci
+
+    - name: Install dependencies
+      run: .github/workflows/install_deps.sh amici
+
+    - name: Run tests
+      timeout-minutes: 30
+      run: tox -e base
+
+    - name: Coverage
+      uses: codecov/codecov-action@v3
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        file: ./coverage.xml
+
+  windows:
+    runs-on: windows-latest
+    strategy:
+      matrix:
+        python-version: ['3.11']
+
+    steps:
+    - name: Check out repository
+      uses: actions/checkout@v3
+
+    - name: Prepare python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip
+        pip install tox
+
+    - name: Run tests
+      shell: bash
+      timeout-minutes: 10
+      run: tox -e windows
+
   petab:
     runs-on: ubuntu-latest
     strategy:
@@ -100,7 +158,7 @@ jobs:
     - name: Install julia
       uses: julia-actions/setup-julia@v1
       with:
-        version: 1.7
+        version: 1.9
 
     - name: Cache
       uses: actions/cache@v3
@@ -113,8 +171,11 @@ jobs:
     - name: Install dependencies
       run: .github/workflows/install_deps.sh
 
+    - name: Install PEtabJL dependencies
+      run: julia -e 'using Pkg; Pkg.add("PEtab"); Pkg.add("OrdinaryDiffEq"), Pkg.add("Sundials")'
+
     - name: Run tests
-      timeout-minutes: 20
+      timeout-minutes: 25
       run: tox -e julia
 
     - name: Coverage
@@ -127,8 +188,8 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        # update to 3.11 when nlopt allows
-        python-version: ['3.9', '3.10']
+        # ipopt does not work on 3.9 (https://github.com/mechmotum/cyipopt/issues/225)
+        python-version: ['3.11']
 
     steps:
     - name: Check out repository

.github/workflows/install_deps.sh

@@ -9,8 +9,14 @@ pip install wheel setuptools
 # Used to create local test environments
 pip install tox
 
-# Update apt
-sudo apt-get update
+# Update package lists
+if [ "$(uname)" == "Darwin" ]; then
+  # MacOS
+  brew update
+else
+  # Linux
+  sudo apt-get update
+fi
 
 # Check arguments
 for par in "$@"; do
@@ -22,8 +28,12 @@ for par in "$@"; do
 
     amici)
       # for amici
-      sudo apt-get install \
-        swig libatlas-base-dev libhdf5-serial-dev
+      if [ "$(uname)" == "Darwin" ]; then
+        brew install swig hdf5 libomp
+      else
+        sudo apt-get install \
+          swig libatlas-base-dev libhdf5-serial-dev
+      fi
     ;;
 
     ipopt)

.readthedocs.yml

@@ -17,7 +17,6 @@ python:
       path: .
       extra_requirements:
         - doc
-        - amici
 
 build:
   os: "ubuntu-20.04"

CHANGELOG.rst

@@ -6,8 +6,38 @@ Release notes
 ..........
 
 
-0.3.1 (2023-06-22)
+0.3.2 (2023-10-02)
 -------------------
+
+* Visualize:
+    * Restrict fval magnitude in waterfall with order_by_id (#1090)
+    * Hierarchical parameter plot fix (#1106)
+    * Fix y-limits on waterfall (#1109)
+* Sampling:
+    * Use cloudpickle for pickling dynesty sampler (#1094)
+* Optimize
+    * Small fix on hierarchical initialise (#1095)
+    * Fix startpoint sampling for hierarchical optimization (#1105)
+    * SacessOptimizer: retry reading, delay deleting (#1110)
+    * SacessOptimizer: Fix logging with multiprocessing (#1112)
+    * SacessOptimizer: tmpdir option (#1115)
+* Storage:
+    * fix storage (#1099)
+* Examples
+    * Notebook on differences (#1098)
+* Problem
+    * Add startpoint_method to Problem (#1093)
+* General
+    * Added new entry to bib (#1100)
+    * PetabJL integration (#1089)
+    * Other platform tests (#1113)
+    * Dokumentation fixes (#1120)
+    * Updated CODEOWNER (#1123)
+
+
+0.3.1 (2023-06-22)
+------------------
+
 * Visualize:
     * Parameter plot w/ hier. pars, noise estimation for splines (#1061)
 * Sampling:
@@ -34,7 +64,7 @@ Release notes
 
 
 0.3.0 (2023-05-02)
--------------------
+------------------
 
 New functionalities compared to 0.2.0:
 
@@ -61,8 +91,6 @@ Not supported functionalities and versions compared to 0.2.0:
 * **Changed parameter indexing from boolean to int in profiling routines**
 
 
-
-
 0.2 series
 ..........
 

doc/api.rst

@@ -12,6 +12,8 @@ API reference
    pypesto.history
    pypesto.logging
    pypesto.objective
+   pypesto.objective.aesara
+   pypesto.objective.jax
    pypesto.objective.julia
    pypesto.optimize
    pypesto.petab

doc/contact.rst

@@ -5,4 +5,4 @@ Contact
 Discovered an error? Need help? Not sure if something works as intended?
 Please contact us!
 
-- Yannik Schälte: `[email protected] <[email protected]>`_
+- Paul Jonas Jost: `[email protected] <[email protected]>`_

doc/example.rst

@@ -27,6 +27,7 @@ Getting started
 
    example/getting_started.ipynb
    example/custom_objective_function.ipynb
+   example/workflow_comparison.ipynb
 
 PEtab and AMICI
 ---------------

doc/example/boehm_JProteomeRes2014/Boehm_validation/Gradient.csv

@@ -0,0 +1,2 @@
+Epo_degradation_BaF3,k_exp_hetero,k_exp_homo,k_imp_hetero,k_imp_homo,k_phos,sd_pSTAT5A_rel,sd_pSTAT5B_rel,sd_rSTAT5A_rel
+-226.01431847863805,0.03263295726678998,0.05074978988074719,-273.3251522968395,-4.9353233689552945e-5,85.33308210815652,-57.82373022138993,6.47910944243548,1.5327970845176904

doc/example/boehm_JProteomeRes2014/Boehm_validation/Hessian.csv

@@ -0,0 +1,10 @@
+Epo_degradation_BaF3,k_exp_hetero,k_exp_homo,k_imp_hetero,k_imp_homo,k_phos,sd_pSTAT5A_rel,sd_pSTAT5B_rel,sd_rSTAT5A_rel
+2348.781845248937,-0.034169780647049425,-103.24202100672939,2772.675796487632,7.194796260472683e-5,-938.3560377189202,996.659728857624,31.143959976782426,13.030712117968559
+-0.034169780647049425,0.07739858784002018,-0.15899501912479183,0.7386693785162849,-4.187803035168086e-8,-0.17134098751539498,-0.2564122376627509,0.03372069395138544,0.07241122210747836
+-103.24202100672939,-0.15899501912479183,132.944794430612,95.49012010074641,-2.508362770407691e-6,-22.250883365424308,11.34374075166351,-3.886786248018129,-7.690665859911962
+2772.675796487632,0.7386693785162849,95.49012010074641,3563.661917245381,1.092950395076315e-5,-1110.2044314822922,1202.406495619125,41.717823566223906,14.58452306207908
+7.194796260472683e-5,-4.187803035168086e-8,-2.508362770407691e-6,1.092950395076315e-5,0.00011364116982094509,-0.00016740979955329535,0.0001089554236616266,0.00011027946613432109,8.045150373694503e-6
+-938.3560377189202,-0.17134098751539498,-22.250883365424308,-1110.2044314822922,-0.00016740979955329535,965.7762449769793,-416.0316410076297,-6.62255869644657,29.6808324053416
+996.659728857624,-0.2564122376627509,11.34374075166351,1202.406495619125,0.0001089554236616266,-416.0316410076297,435.9488576495015,0.0,0.0
+31.143959976782426,0.03372069395138544,-3.886786248018129,41.717823566223906,0.00011027946613432109,-6.62255869644657,0.0,139.82333729022352,0.0
+13.030712117968559,0.07241122210747836,-7.690665859911962,14.58452306207908,8.045150373694503e-6,29.6808324053416,0.0,0.0,162.6019478340014

doc/example/boehm_JProteomeRes2014/Boehm_validation/ObjectiveValue.csv

@@ -0,0 +1,2 @@
+ObjectiveValue
+149.05022585636098

doc/example/boehm_JProteomeRes2014/Boehm_validation/Parameter.csv

@@ -0,0 +1,2 @@
+Epo_degradation_BaF3,k_exp_hetero,k_exp_homo,k_imp_hetero,k_imp_homo,k_phos,sd_pSTAT5A_rel,sd_pSTAT5B_rel,sd_rSTAT5A_rel
+-1.665827601408055,-4.999704893599998,-2.2096987817000167,-1.78600654750001,4.9901140088,4.1977354885,0.5857552705999998,0.8189828191999999,0.49868440400000047

doc/example/conversion_reaction/PEtabJl_module.jl

@@ -0,0 +1,20 @@
+module MyPEtabJlModule
+
+using OrdinaryDiffEq
+using Sundials
+using PEtab
+
+pathYaml = "/Users/pauljonasjost/Documents/GitHub_Folders/pyPESTO/test/julia/../../doc/example/conversion_reaction/conversion_reaction.yaml"
+petabModel = readPEtabModel(pathYaml, verbose=true)
+
+# A full list of options for createPEtabODEProblem can be found at https://sebapersson.github.io/PEtab.jl/dev/API_choosen/#PEtab.setupPEtabODEProblem
+petabProblem = createPEtabODEProblem(
+	petabModel,
+	odeSolverOptions=ODESolverOptions(Rodas5P(), abstol=1e-08, reltol=1e-08, maxiters=Int64(1e4)),
+	gradientMethod=:ForwardDiff,
+	hessianMethod=:ForwardDiff,
+	sparseJacobian=nothing,
+	verbose=true
+)
+
+end