How can I use both docker containers and local software?

[jonbra]

Hi,
I am using docker containers for my different processes. Currently this is specified in the nextflow.config file. But I want to include a new process which needs access to a program installed locally (I'm not able to make that into a Dockerfile). Can I have some processes use docker containers and some use locally installed software? If yes, how?

Thanks, Jon

Here's my config file and I add the option --use_docker:

process {
  if (params.use_docker) {
      container = "nfcore/viralrecon:1.1.0"

      withName:CONSENSUS {
          container = "nfcore/viralrecon:1.1.0"
      }
      withName:MODIFY_FASTA {
          container = "darcyabjones/bbmap:latest"
      }
  }
}


docker {
    enabled = params.use_docker
}

[ewels]

Sure, I think that the docker scope also works at the process level:

process {
    withName:LOCAL_PROCESS {
        docker.enabled = false
    }
}

It's been a while, but I'm pretty sure that this is what we did / do in order to run MultiQC with our in-house plugins 🤔

[jonbra]

The config file: https://github.com/jonbra/learning_nextflow/blob/main/nextflow.config
The problem is the executable for the tanoti mapper in this module: https://github.com/jonbra/learning_nextflow/blob/main/modules/tanoti.nf
I also export to PATH in the .bashrc file, but still command not found...

[pditommaso]

This does not much sense

https://github.com/jonbra/learning_nextflow/blob/main/modules/tanoti.nf#L22

where tanoti is expected to be located? in a container or the host?

[jonbra]

Yes, most likely this does not make much sense! 😅 I added this line because I got command not found even with the executable available in my local path... How can Nextflow access software not in a container?

[pditommaso]

How can Nextflow access software not in a container

There's no special nextflow logic for that. It just delegates to the usual Linux PATH variable mechanism.

However, note that variable can behave differently depending on where it has been defined. See the post below

https://stackoverflow.com/questions/415403/whats-the-difference-between-bashrc-bash-profile-and-environment

To help with the troubleshooting 1) go in the failed task work directory 2) edit .command.sh and add echo $PATH at the beginning 3) run it with bash .command.run

[jonbra]

Ok, I see.. The program is not in the path that Nextflow sees and it works by using absolute paths.
Thanks! :)