L97: Python: Setuptools entrypoint for generating file

L97-python-setuptools-plugin.md

@@ -0,0 +1,177 @@
+L97 - Python: Setuptools entrypoint for generating file
+----
+* Author(s): Patrick Lannigan
+* Approver: lidizheng and gnossen
+* Status: Draft
+* Implemented in: Python
+* Last updated: 2022-02-18
+* Discussion at: https://groups.google.com/g/grpc-io/c/H617sSzoV-8
+
+## Abstract
+
+Implement a `setuptools` entrypoint for generating files when a Python distribution is being built.
+This type of integration would be called automatically by `setuptools` and would not require and
+explicit call by the project to generate the files. The entrypoint would be configured using an
+established python project configuration file and would only require the `grpcio-tools` package to be
+listed as a build time dependency.
+
+## Background
+
+Currently `grpc_tools` [provides a command][grpc_command] for integrating with `setuptools`. However,
+this requires custom integration on the part of the package (example:
+[grpc_health_checking][grpc_command_example_integration] and
+[explicitly calling the command][grpc_command_example_call] when building the wheel).
+
+With the introduction of [PEP 517][pep_517] and [PEP 518][pep_518], directly calling `setup.py` will
+start becoming less common. Additionally, the use of custom commands will not work with projects that
+use this mechanism and solely rely on the [declarative config in `setup.cfg`][declarative_config],
+which was added to `setuptools` in [v30.3.0][release_declarative_config] (Dec. 2016). Though, use of
+the functionality defined in this proposal does not require a project to use the declarative config.
+
+Custom commands are not the only integration point that is available. `setuptools` also supports
+entrypoints that allows for packages to act as a plugin and provide custom behavior. There are a few
+different options for entrypoint hooks ([1][setuptools_customize], [2][setuptools_file_finder]) which
+could be used to generate source files based on `.proto` files. (There is also an
+[open proposal][setuptools_sub_commands] to allow for adding `build` sub-commands. Which more directly
+represents what we are trying to achieve.)
+
+[Original Feature Request][original_feature_request]
+
+## Proposal
+
+`grpcio-tools` will register a new entrypoint function to respond to the
+`setuptools.finalize_distribution_options` hook. This hook will generate Python files based on the
+`.proto` files.
+
+### Configuration
+
+This functionality will use a [tool table][tool_table] in `pyproject.toml` named `tool.grpcio-tools`
+to accept configuration parameters.
+
+__TODO__: List configuration parameters and how they would be used. If anything is required, add it to
+the list in "How would a project use this feature".
+
+### Functional Implementation
+
+Add an entrypoint function to `grpcio-tools` that would register as a `file_finder` hook with
+`setuptools`. To simplify the implementation without introducing an additional code path, the
+entrypoint will target the same `main()` function used by the `grpc_tools.protoc` CLI.
+
+In order to prevent unintended behavior, the first thing the function will do is check the
+configuration file to see if the section has been configured (See [Configuration](#configuration)). If
+the file does not exist or the section is not configured, no additional steps will be taken. The
+function will return an empty list back to `setuptools` to indicate that there are no additional files
+to include in the distribution.
+
+Starting with Python 3.11, the configuration file will be readable using a
+[standard library module][pep_680]. For earlier Python versions, `tomli` will be added as an optional
+dependency.
+
+### How would a project use this feature
+
+A project that would like to use this `setuptools` integration would need to perform two actions to
+enable the functionality.
+
+* List `grpcio-tools` as a [build time dependency][build_dependency]. This will ensure that the library
+    is installed when the project's package is built.
+* Add a `tool.grpcio-tools` table to the file `pyproject.toml`.
+
+Minimal example for a project using [PEP 518][pep_518]:
+
+pyproject.toml
+```toml
+[build-system]
+requires = ["setuptools", "wheel", "grpcio-tools >= X.Y.Z"]
+build-backend = "setuptools.build_meta"
+
+[tool.grpcio-tools]
+example_parameter = "CHANGE ME"  # This is a place holder until the configuration parameters are defined
+```
+
+Minimal example for a project using `setup.py`:
+
+setup.py
+```python
+from setuptools import setup
+
+setup(
+    ...,
+    setup_requires=[
+        'grpcio-tools >= X.Y.Z'
+    ],
+)
+```
+
+pyproject.toml
+```toml
+[tool.grpcio-tools]
+example_parameter = "CHANGE ME"  # This is a place holder until the configuration parameters are defined
+```
+
+## Rationale
+
+A downside of this proposal is that it will introduce yet another way to get the Python modules that
+correspond to the proto files.
+
+The primary way generate the Python files is using the
+[`grpc_tools.protoc` command line interface][protoc_cli]. The advantage of this mechanism of this
+method is also its disadvantage. It is a completely standalone way to generate the source code for
+the python modules. While projects are allowed to decide how to work this generated files in the way
+that best suits the project, there is no integration point. This means the project **must** decide
+how and when this command is executed, likely needing to write a script to execute this CLI.
+
+For projects that use `setuptools` to publish a package, `grpcio-tools` includes a
+[custom command][grpc_command]. However, it also requires a specific call like the `grpc_tools.protoc`
+CLI and is not compatible with installing from a source distribution for projects that declare
+`setuptools` as build system that should be used. Additionally, one of the top result for "grpc
+setuptools command" is [this post][setuptools_gen_question], which suggests a custom command that
+doesn't rely on the one provided by `grpcio-tools`. This suggest that there is a desire for functionality
+that doesn't require a specific all and that the existing functionality is under-documented.
+
+gRPC also provides a way to get the Python [classes at runtime][runtime_classes]
+([Proposal][runtime_proposal]). However, it is still [marked as experimental][runtime_api] almost 2
+years later and requires `grpcio-tools` to be available at runtime. Similar to the current `setuptools`
+integration, it is not mentioned in the basic tutorial documentation.
+
+## Implementation
+
+[Pull Request][feature_pr]
+
+- [x] Proof of Concept with hard coded values.
+- [ ] Configuration file parsing & mapping to CLI arguments.
+- [ ] Document feature and how projects would use it.
+
+Could be done later:
+
+- [ ] Migrate existing `grpc*` packages to use this functionality.
+
+I am planning work on it. I will be able to spend a few hours of each work week on this effort.
+
+## Open issues (if applicable)
+
+- Looking at the implementation of the [custom command][grpc_command], it looks as though only a
+  sub-set of the CLI arguments supported by `grpc_tools.protoc` need to be supported. I believe this
+  will become more clear after starting on the implementation. However, there could always be an
+  "additional arguments" type parameter to cover this if a project has a specific need. This would
+  also allow for CLI arguments to be passed to any gRPC plugins that might be used by a project.
+
+[grpc_command]: https://github.com/grpc/grpc/blob/05e17e92390d4685f1418f535604a201a7f8e1a3/tools/distrib/python/grpcio_tools/grpc_tools/command.py#L50
+[grpc_command_example_integration]: https://github.com/grpc/grpc/blob/2d4f3c56001cd1e1f85734b2f7c5ce5f2797c38a/src/python/grpcio_health_checking/health_commands.py#L48
+[grpc_command_example_call]: https://github.com/grpc/grpc/blob/2d4f3c56001cd1e1f85734b2f7c5ce5f2797c38a/tools/run_tests/artifacts/build_artifact_python.sh#L196-L197
+[pep_517]: https://www.python.org/dev/peps/pep-0517/
+[pep_518]: https://www.python.org/dev/peps/pep-0518/
+[declarative_config]: https://setuptools.pypa.io/en/latest/userguide/declarative_config.html
+[release_declarative_config]: https://setuptools.pypa.io/en/latest/history.html#v30-3-0
+[setuptools_customize]: https://setuptools.pypa.io/en/latest/userguide/extension.html#customizing-distribution-options
+[setuptools_file_finder]: https://setuptools.pypa.io/en/latest/userguide/extension.html#adding-support-for-revision-control-systems
+[setuptools_sub_commands]: https://github.com/pypa/setuptools/issues/2591
+[original_feature_request]: https://github.com/grpc/grpc/issues/28662
+[build_dependency]: https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=setup_requires#build-system-requirement
+[tool_table]: https://www.python.org/dev/peps/pep-0518/#tool-table
+[pep_680]: https://www.python.org/dev/peps/pep-0680/
+[protoc_cli]: https://grpc.io/docs/languages/python/basics/#generating-client-and-server-code
+[setuptools_gen_question]: https://stackoverflow.com/q/52994857
+[runtime_classes]: https://github.com/grpc/grpc/blob/a72c8ebb7def13a317a1afc7c08455388d1fa2e4/src/python/grpcio/grpc/_runtime_protos.py#L1
+[runtime_proposal]: ./L64-python-runtime-proto-parsing.md
+[runtime_api]: https://grpc.github.io/grpc/python/grpc.html#runtime-protobuf-parsing
+[feature_pr]: https://github.com/grpc/grpc/pull/29541