Handle Worker Resources

[PythonFZ]

Currently there is no way of setting the resources required per Node. There are ways to define this for the client.submit via https://distributed.dask.org/en/stable/resources.html.

My Idea would be, to add a dask4dvc.yaml file (or alternatively the meta key in dvc.yaml https://distributed.dask.org/en/stable/resources.html) to define the resources that should be acquired for the respective Node.

node1:
  GPU: 1
  MEMORY: 100e9

node2:
  MEMORY: 16e9

node3:
  GPU: 3
  MEMORY: 128e9

we parse this file, if it exists and use it here:

dask4dvc/dask4dvc/utils/graph.py

Lines 96 to 101 in 86396b2

	dask_node = client.submit(
	cmd,
	name=node, # required
	deps=deps, # required
	pure=False,
	**kwargs,

Furthermore, we might also want to have a general section e.g. for https://jobqueue.dask.org/en/latest/generated/dask_jobqueue.SLURMCluster.html

cluster:
   slurm:
      project: "my-project"