Draft for job decoupling discussion

[javiermtorres]

What's changing

A new generic create_job_alt is proposed. The key changes are:

• Job data are retrieved from a dict, which allows programmatic changes in the future, and a more general config and use right now
• Job config models are stored in the same dict from a statically imported file
  • Currently models are used in standalone scripts, so they need to be added to lumigator somehow
    • The image build process could include these files somehow (maybe using the mechanisms in https://hatch.pypa.io/latest/config/build/)
    • The models could be included in lumigator itself, and the scripts could import the lumigator_schemas package
  • More dynamic imports using importlib to load models at load-time or run-time could be done later on without changes to the current job creation
• Additional steps can be added as hooks, if needed, in the same way that the model code is added

The current code has not been checked; once the approach is validated, a full, working implementation will be provided.

How to test it

N/A; draft for discussion at the moment.

Additional notes for reviewers

N/A

I already...

• added some tests for any new functionality
• updated the documentation
• checked if a (backend) DB migration step was required and included it if required

[javiermtorres]

Closing this PR as the issue has been discussed internally and issues have been created.