diff --git a/README.md b/README.md index dd619de7..b47d89d0 100644 --- a/README.md +++ b/README.md @@ -35,9 +35,9 @@ features include: to run the same task with different inputs. - **Select tasks via expressions.** Run only a subset of tasks with [expressions and marker expressions](https://pytask-dev.readthedocs.io/en/stable/tutorials/selecting_tasks.html). -- **Easily extensible with plugins**. pytask is built on top of - [pluggy](https://pluggy.readthedocs.io/en/latest/), a plugin management framework, - which allows you to adjust pytask to your needs. Plugins are available for +- **Easily extensible with plugins**. pytask is built on + [pluggy](https://pluggy.readthedocs.io/en/latest/), a plugin management framework that + allows you to adjust pytask to your needs. Plugins are available for [parallelization](https://github.com/pytask-dev/pytask-parallel), [LaTeX](https://github.com/pytask-dev/pytask-latex), [R](https://github.com/pytask-dev/pytask-r), and diff --git a/docs/source/changes.md b/docs/source/changes.md index b03b8898..0556fdc2 100644 --- a/docs/source/changes.md +++ b/docs/source/changes.md @@ -5,21 +5,16 @@ chronological order. Releases follow [semantic versioning](https://semver.org/) releases are available on [PyPI](https://pypi.org/project/pytask) and [Anaconda.org](https://anaconda.org/conda-forge/pytask). -## 0.5.0 - 2024-xx-xx - -- {pull}`544` requires sqlalchemy `>=2` and upgrades the syntax. - -## 0.4.5 - 2024-01-xx +## 0.4.5 - 2024-01-09 - {pull}`515` enables tests with graphviz in CI. Thanks to {user}`NickCrews`. - {pull}`517` raises an error when the configuration file contains a non-existing path - (fixes #514). It also warns if the path is configured as a string and not a list of + (fixes #514). It also warns if the path is configured as a string, not a list of strings. - {pull}`519` raises an error when builtin functions are wrapped with - {func}`~pytask.task`. Closes {issue}`512`.pull + {func}`~pytask.task`. Closes {issue}`512`. - {pull}`521` raises an error message when imported functions are wrapped with {func}`@task ` in a task module. Fixes {issue}`513`. - {func}`~pytask.task`. Closes {issue}`512`. - {pull}`522` improves the issue templates. - {pull}`523` refactors `_pytask.console._get_file`. - {pull}`524` improves some linting and formatting rules. @@ -35,6 +30,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and the signatures suggested. - {pull}`542` refactors the plugin manager. - {pull}`543` fixes imports in tests and related issues. +- {pull}`544` requires sqlalchemy `>=2` and upgrades the syntax. +- {pull}`545` finalizes the release. ## 0.4.4 - 2023-12-04 diff --git a/docs/source/developers_guide.md b/docs/source/developers_guide.md index cd5f40e3..1a987b0c 100644 --- a/docs/source/developers_guide.md +++ b/docs/source/developers_guide.md @@ -5,7 +5,7 @@ Run pytest to execute the test suite. The test suite creates many temporary directories. There is usually a limit on the -number of open file descriptors on Unix systems which causes some tests and the end of +number of open file descriptors on Unix systems, which causes some tests and the end of the test suite to fail. If that happens, increase the limit with the following command. ```console @@ -19,35 +19,23 @@ The following list covers all steps of a release cycle. - Start a new release cycle by opening a milestone. Assign all relevant issues and merge requests to this milestone. -- Every change is pushed to the `main` branch of the repository and will make it into - the next release. +- Every change is pushed to the repository's main branch and will make it into the next + release. -- Once all additions to a release are merged, prepare `changes.rst` in the source folder - of the documentation listing all changes which made it into the release. +- Once all additions to a release are merged, prepare `docs/source/changes.md` listing + all changes that made it into the release. - Update the version numbers in the animations if you create a new major or minor release. -- Go to the [release tab](https://github.com/pytask-dev/pytask/releases). Create a new - release by clicking on "Draft a new release" and add a tag named `vx.x.x` on `main` - and make it also the release title. Click on "Publish release". - - Creating a tag will trigger a pipeline which builds the package and uploads it to PyPI - which consequently triggers a new release on conda-forge. - -## Creating showcases on the command line - -- Replace prompt in powershell core with a simple arrow by typing - - ```console - $ function prompt {"> "} - ``` +- Go to the main branch and set a new tag with `git tag vx.x.x -m "vx.x.x"`. Then, push + the tag to the repository. -- Rename the tab with +- Creating a tag will trigger a pipeline that builds the package and uploads it to PyPI, + consequently triggering a new release on conda-forge. - ```console - $ $Host.UI.RawUI.WindowTitle = $title - ``` + The pipeline also creates a GitHub release under + [releases](https://github.com/pytask-dev/pytask/releases). Fill out the release notes. ## Profiling the application diff --git a/docs/source/how_to_guides/remote_files.md b/docs/source/how_to_guides/remote_files.md index e617a59b..abd8967d 100644 --- a/docs/source/how_to_guides/remote_files.md +++ b/docs/source/how_to_guides/remote_files.md @@ -13,9 +13,8 @@ easy to interact with files from an HTTP(S)-, Dropbox-, S3-, GCP-, Azure-based filesystem, and many more. ```{warning} -universal_pathlib does currently not support Python 3.12. To track progress, see [this -PR](https://github.com/fsspec/universal_pathlib/pull/152) and check the [releases -`>0.1.4`](https://github.com/fsspec/universal_pathlib/releases) +universal_pathlib does currently not support Python 3.12. To track progress, check the +[releases `>0.1.4`](https://github.com/fsspec/universal_pathlib/releases). ``` ## HTTP(S)-based filesystem @@ -86,6 +85,5 @@ Many files on the web do not provide an ETag like this version of the iris datas {'Server': 'nginx/1.25.3', 'Date': 'Sun, 10 Dec 2023 23:59:21 GMT', 'Connection': 'keep-alive'} ``` -In these instances, pytask does not recognize if the file has changed and only reruns -the task if other conditions are not met, like the product is missing, the task module -has changed, etc.. +In these instances, pytask does not recognize if the file has changed. If you want to +force rerunning the task, delete a product of the task. diff --git a/docs/source/tutorials/configuration.md b/docs/source/tutorials/configuration.md index a61788fc..7492e2b3 100644 --- a/docs/source/tutorials/configuration.md +++ b/docs/source/tutorials/configuration.md @@ -3,7 +3,7 @@ pytask can be configured via the command-line interface or permanently with a `pyproject.toml` file. -The file also indicates the root of your project where pytask stores information in a +The file also indicates the root of your project, where pytask stores information in a `.pytask` folder. ## The configuration file @@ -28,23 +28,22 @@ paths = "src" There are two ways to point pytask to the configuration file. -First, it is possible to pass the location of the configuration file via +The first option is to let pytask try to find the configuration itself. + +1. Find the common directory of all paths passed to pytask (default to the current + working directory). +1. Look at all parent directories from this directory and return the file if it exists. +1. Stop searching if a directory contains a `.git` directory/file or a valid + configuration file with the correct section. + +Secondly, it is possible to pass the location of the configuration file via {option}`pytask build -c` like ```console $ pytask -c config/pyproject.toml ``` -The second option is to let pytask try to find the configuration itself. - -1. Find the common base directory of all paths passed to pytask (default to the current - working directory). -1. Starting from this directory, look at all parent directories, and return the file if - it exists. -1. Stop searching if a directory contains a `.git` directory/file or a valid - configuration file with the right section. - ## The options -You can find all possible configuration values in the +All possible configuration values are found in the {doc}`reference guide on the configuration <../reference_guides/configuration>`. diff --git a/docs/source/tutorials/defining_dependencies_products.md b/docs/source/tutorials/defining_dependencies_products.md index 266125e2..0472b666 100644 --- a/docs/source/tutorials/defining_dependencies_products.md +++ b/docs/source/tutorials/defining_dependencies_products.md @@ -3,20 +3,24 @@ Tasks have dependencies and products that you must define to run your tasks. Defining dependencies and products also serves another purpose. By analyzing them, -pytask determines the order in which to run the tasks. +pytask determines the order to run the tasks. This tutorial offers you different interfaces. If you are comfortable with type -annotations or not afraid to try them, look at the `Python 3.10+` or `Python 3.8+` tabs. +annotations or are not afraid to try them, look at the `Python 3.10+` or `3.8+` tabs. You find a tutorial on type hints {doc}`here <../type_hints>`. If you want to avoid type annotations for now, look at the tab named `produces`. +```{warning} The `Decorators` tab documents the deprecated approach that should not be used anymore and will be removed in version v0.5. +``` +```{seealso} In this tutorial, we only deal with local files. If you want to use pytask with files online, S3, GCP, Azure, etc., read the {doc}`guide on remote files <../how_to_guides/remote_files>`. +``` First, we focus on defining products that should already be familiar to you. Then, we focus on how you can declare task dependencies. diff --git a/docs/source/tutorials/plugins.md b/docs/source/tutorials/plugins.md index a7eaba42..abf97366 100644 --- a/docs/source/tutorials/plugins.md +++ b/docs/source/tutorials/plugins.md @@ -6,6 +6,12 @@ maintainers to cater to all possible use cases. Therefore, pytask uses [pluggy](https://github.com/pytest-dev/pluggy), a plugin framework, to allow users to extend pytask. +## How to extend pytask + +A quick method to extend pytask is explained in the +{doc}`guide on extending pytask <../how_to_guides/extending_pytask>`. You will learn how +to add your own hook implementations or write your plugin. + ## Where can I find plugins? You can find plugins in many places. @@ -18,8 +24,3 @@ You can find plugins in many places. overview of repositories linked to pytask. - Search on [anaconda.org](https://anaconda.org/search?q=pytask) or [prefix.dev](https://prefix.dev) for related packages. - -## How to extend pytask - -Follow the {doc}`guide on extending pytask <../how_to_guides/extending_pytask>` to add -your own hook implementations or write your plugin. diff --git a/docs_src/tutorials/collecting_tasks.py b/docs_src/tutorials/collecting_tasks.py index 857b2c14..23155fb7 100644 --- a/docs_src/tutorials/collecting_tasks.py +++ b/docs_src/tutorials/collecting_tasks.py @@ -1,8 +1,11 @@ -# Content of task_module.py from pathlib import Path +from typing import Annotated + +from pytask import Product def task_write_file( - input_path: Path = Path("in.txt"), output_path: Path = Path("out.txt") + input_path: Path = Path("in.txt"), + output_path: Annotated[Path, Product] = Path("out.txt"), ) -> None: output_path.write_text(input_path.read_text())