[BLUEPRINT] Module registration

[oliverklee]

This is the plan for how module registration for phpList 4 should work. Comments and feedback welcome.

Principles

• All phpList modules are Composer packages that are recognized by the package type phplist-module defined in the composer.json.
• A phpList module package must only contain code, assets and configuration related to this package. The module-related files must also not be "piggy-backing" in some other package. Packages still can depend on other packages and make use of their code, though (by adding them to the require section of the composer.json).
• From the view of the administrator/user, a module is installed by requiring it in the root package composer.json and then running a composer update. It will not be possible for a package to be in an "installed, but not activated" state.
• It must be possible to test module packages on their own, i.e., without the base-distribution package, both with automated tests as well as manual tests. (The module package usually will depend on at least the phplist4-core package, though.)
• Modules that provide the same capabilities (e.g., the same routes) can be configured via their composer.json to conflict with each (or more specifically: with other packages that provide the same capabilities) (TODO: document how to do this)

Technical approach

• For the installed application, each bundle can bring its own configuration that will be auto-included once the bundle is included.
• There will be an application-wide configuration file that lists all module-provided bundles (i.e., their class names).
• Each module package must provide the class names of its provided bundles in the extra section of its composer.json.
• The phplist4-core package will provide a composer script that lists all installed modules (i.e., all installed composer packages that have the corresponding type). (The core might need to read vendor/composer/installed.json for this. I'll need to check this.) This script will be there mainly for convenience, and also to keep the PRs relatively small. This script will also be accessible via a bin/console command.
• The phplist4-core package will provide a composer script that runs through all installed module packages, reads their composer.json file, and generates the root application bundle classes file. This script will listen to the update and install events of all phplist 4 packages (including the core). It will also listen to the package creation event of the base-distribution package. (NB: Scripts will always only be executed for the current root package, not for any dependencies.)
• The "empty start page" will be moved to a separate module package (for which we still need to find a good name). This package will be included by default in the base-distribution package (as long as the web-frontend package does not provide any usable content). This package will conflict with the web-frontend package as both provide the / route (i.e., the root route). This package will also be used by the automated tests for module registration in the phplist4-core package.

(Planned) Pull requests

• [FEATURE] Parameter kernel.application_root for the application root #153 Add a function for provided the path to the application root.
• [FEATURE] Composer PackageRepository #155, [FEATURE] Composer script for listing all installed phpList modules #156 Add a composer script that lists all installed module packages to the phplist4-core package. This will include a PHP class for this that can also be used for other things.
• [FEATURE] Bundle configuration file #157 Add the not-yet-autogenerated bundles classes configuration file to the phplist4-core package, move the REST API configuration and the "empty start page" configuration there, and add it to Git. (At this stage, the phplist4-core still has hard-coded references to the rest-api package.) (This config file will be removed once auto-generation of the config file is added.)
• [FEATURE] Composer script for reading registered bundles #158 Add the script that auto-generates the root configuration file to the phplist4-core package and delete the old manual configuration file. Add composer listeners for update and install to the phplist4-core package and add the file to the .gitignore.
• Change the "empty start page" bundle to auto-include its configuration.
• For all module packages and the base-distribution package, add composer listeners for update and install to the phplist4-core package and add the file to the .gitignore.
• [FEATURE] Bundle routing configuration #168 Add a script that auto-generates a routes configuration file.
• [FEATURE] Routes configuration rest-api#40 Add the configuration file to the rest-api package.
• [CLEANUP] Clean up the "empty start page" bundle #183 Clean up the code for the empty start page in the phplist4-core package.
• [TASK] Provide the ApplicationStructure via dependency injection #179, [TASK] Use the Symfony dependency injection for the HashGenerator #205, [CLEANUP] Cleanup up the service configuration #207 Use dependency injection.
• [FEATURE] System configuration from modules #203 Allow modules to provide their own (non-service) configuration
• Use the FOSRestBundle for our REST module.

[oliverklee]

@samtuke, @michield: Could you please read this with a critical eye and provide feedback on what's missing, potential problems etc.? Thanks!

[samtuke]

Thanks for this Oliver; very informative.
Having a separate composer package just to provide an empty homepage / web root seems like overkill, though the technical merits make sense (easily disclude it when the full web ui is installed). Is there perhaps an option to avoid this otherwise redundant package?

[samtuke]

The ability to list all installed packages is not only useful for composer commands, but also for the Web UI and plugin management at a later date. Have you considered how the initial composer commands may relate to or interact with the Web UI related system in future, or will these be handled separately?

[oliverklee]

Having a separate composer package just to provide an empty homepage / web root seems like overkill, though the technical merits make sense (easily disclude it when the full web ui is installed). Is there perhaps an option to avoid this otherwise redundant package?

I don't see a problem with a package being relatively small an simple.

This package will conflict with the web-frontend package as both will provide the / route. So we cannot keep this package part of the core if we also want to be able to use the web-frontend package.

In addition, I need a relatively small package for the automated tests for the module registration code in the core. I'd like to use the "empty start page" module for this.

[oliverklee]

The ability to list all installed packages is not only useful for composer commands, but also for the Web UI and plugin management at a later date. Have you considered how the initial composer commands may relate to or interact with the Web UI related system in future, or will these be handled separately?

I'll make the code that finds out which modules are installed available as a class that can be used also outside the Composer scripts.

In our conference call with Michiel, we decided to not add a UI for Composer (i.e., also for the module management) at the moment.

[samtuke]

Both sound good to me. @michield Please can you review and comment?

[oliverklee]

Update on the approach after more research and some prototyping: It is not possible to use parameters (like %kernel.application_dir%) when importing configuration files. So the approach for the generated, application-wide configuration files will not work (and I'll need to find out a way to include an application-wide parameters file).

However, it is possible to have installed bundles automatically include their configuration. I'm planning to use this. I'm going to update the blueprint accordingly.

[michield]

Looks good. I think the most important part of modules is to have good documentation and an example one that does some basic things, possibly a few with different capabilities (eg https://github.com/michield/phplist-plugin-subscribeexample)

The threshold to start coding modules should be as low as possible.

The issue on conflicting modules should be handled by a module manager. For example only one module can offer a particular service and the choice which one is with the person managing it (either during config or possibly in the UI). For example the way that phpList3 has the "editorprovider" and now the "emailsenderplugin" and "authenticationplugin". There can only be one of each.

Modules should also be allowed to add to the commandline processes. I also think it will be good to standardise (and enforce) the way they are configured and log their output via a central system.

In general I would encourage the modules to use the rest api to do things.

[oliverklee]

@michield Thanks for the feedback!

[samtuke]

The issue on conflicting modules should be handled by a module manager

@oliverklee Can't we rely on composer for this?

In general I would encourage the modules to use the rest api to do things

@oliverklee Would it be simpler for third party modules to use Symfony routing, or the REST API? Or perhaps have the choice of either?

[oliverklee]

@oliverklee Can't we rely on composer for this?

Yes, that's the plan: to figure out and document how to use Composer for this.

@oliverklee Would it be simpler for third party modules to use Symfony routing, or the REST API? Or perhaps have the choice of either?

In general, they have the choice:

1. use the repositories etc. (without any additional HTTP requests)
2. use the REST API via HTTP
3. have their own routes (usually combined with 1.)

[samtuke]

In general, they have the choice:

I think there should be a clear recommendation from core developers to third party module developers about this, as to which is the better approach. If software is using the REST API to interact with phpList 4 core, then it would seem to be more of an app than a module -- less integrated, more flexible, with no requirement to run on the same server (presumably). Any opinions on this?

[oliverklee]

I think there should be a clear recommendation from core developers to third party module developers about this, as to which is the better approach.

Agreed. I'll take care adding this to the documentation.

[oliverklee]

This is now all done. Closing.