339 support seq2seq models

[dpoulopoulos]

What's changing

Introduces the HuggingFaceModelClient to support models pulled from Hugging Face Hub.

The client expects a model and a task, and instantiates the corresponding pipeline. For example, if the task is "summarization" the client instantiates the following pipeline:

from transformers import pipeline

pipe = pipeline("summarization", model="...", **kwargs)

During the instantiation of the pipeline we consider several things, such as the dtype for the model weights and the device we will use during inference (i.e., cpu, cuda, or mps).

Closes #339

How to test it

Steps to test the changes:

1. Run make local-up to deploy Lumigator locally.

2. Upload a dataset using the OpenAPI docs page (localhost:8000/docs)

3. Create a new inference job, with the following body:

   {
     "name": "summarization",
     "model": "hf://facebook/nbart-large-cnn",
     "dataset": "...",
     "max_samples": 1,
     "task": "summarization",
     "accelerator": "auto",
     "revision": "main",
     "use_fast": true,
     "trust_remote_code": false,
     "torch_dtype": "auto"
   }

   for translation tasks provide the following body:

   {
     "name": "translation",
     "model": "hf://google-t5/t5-base",
     "dataset": "...",
     "max_samples": 1,
     "task": "translation",
     "accelerator": "auto",
     "revision": "main",
     "use_fast": true,
     "trust_remote_code": false,
     "torch_dtype": "auto"
   }

Additional notes for reviewers

In theory, this implementation can support any arbitrary task, but we're limited by the way we're loading the dataset. For example, "question-answering" tasks need the dataset to be in the following format:

qa_input = {
    'question': 'Why is model conversion important?',
    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
}

We treat this as a known limitation.

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required

[aittalam]

Thank you Dimitris! There are a couple of unresolved convos here but I do not think they are blocking, so I am going to pre-approve this and let you close them.