Nextflow process not properly killed after error

[lbrandhoff-w]

Hi all,

I am running nextflow via slurm on an HPC using two slurm partitions.
One these contains only illumina dragen servers.

Whenever theres an error in the partition with the dragen servers it gets recognised correctly by nextflow which can be seen in the nextflow logfile:

Error executing process > 'dragen_basic (null)'

Caused by:
  Process `dragen_basic (null)` terminated with an error exit status (1)

Command executed:

  <command call>

Command exit status:
  1

Command output:
 <command output>

Command error:
  <error_message>

Work dir:
  <work_dir>

Then, all processes independant of the error process finish - as expected.
But then a process randomly pops up and keeps nextflow running for an unlimited amount of time. This can only be stopped by manually killing nextflow. The log entries look like this:

Jun-08 13:41:26.286 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor slurm > tasks in the submission queue: 1 -- tasks to be submitted are shown below
~> TaskHandler[jobId: null; id: 31; name: merge_func; status: NEW; exit: -; error: -; workDir: <workdir> started: -; exited: -; ]
Jun-08 13:45:44.284 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
...
Jun-09 12:11:37.465 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor slurm > tasks in the submission queue: 1 -- tasks to be submitted are shown below
~> TaskHandler[jobId: null; id: 31; name: merge_func; status: NEW; exit: -; error: -; workDir: <workdir> started: -; exited: -; ]
Jun-09 12:15:47.492 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled

Interestingly this is not the case if an error occurs in the other slurm partition.

The merge_func process is downstream of the dragen_basic process but there are multiple steps between them. So there is definitely no input for the merge_func process.
From debugging I found out that this code block generates a single empty instance of the merge_func input channel ch_res_func:

    ch_output_func
        .toSortedList { a, b -> a[0] <=> b[0] }
        .map { allPairs -> allPairs.collect{ chr, file -> file } }
        .set { ch_res_func }

This mpty channel instance is what keeps the process in the submission queue. Also,ch_res_func is the only input for the merge_func channel.
I also checked the ch_output_func channel, but nothing is contained there at any point.
I am a bit puzzled why this is happening, especially only with errors from the dragen slurm partition.

I use this software for running nextflow:

• nextflow version 20.10.0.5430
• Java(TM) SE Runtime Environment (build 1.8.0_92-b14), Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
• CentOS Linux release 7.9.2009 (Core)