Auto3DSeg KiTS23 Performance

[mibaumgartner]

Hi, I have trained Auto3DSeg on a single fold of the KiTS23 dataset with hierarchical labels (as defined in https://github.com/Project-MONAI/tutorials/pull/1627/files but without additional manual tuning of parameters). While SegResNet performed pretty well, DiNTs and Swin UNETR performed uncharacteristically bad (20-30 Dice points worse). Any ideas how to fix this?

Best,
Michael

task.yaml

datalist: ./data_list.json
dataroot: [path to dir here; removed for discussion on Github]
modality: CT

class_names:
  - { name: kidney_and_mass, index: [1,2,3] }
  - { name: mass, index: [2,3] }
  - { name: tumor, index: [2] }
sigmoid: true

[KumoLiu]

Hi @mibaumgartner, it makes sense that different algos have different performances. You can more details here.
https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg/tasks/kits23

Hope it helps, thanks.

[mibaumgartner]

Hi, considering that the results are significantly worse from what I would expect, a few more questions:

1. What is the expected compute budget for Auto3DSeg? We are currently using a single A100 GPU 40GB. Some of the hyperparameters seem to be quite dependent on the number of GPU used, e.g. batch size 1 seems like a rather odd default for a single GPU setup (while it is reasonable for a 8 GPU setup)
2. When checking the log file I noticed that:

Suggested network parameters: 
Batch size 1 => 2 
ROI size [224, 224, 144] => [224, 224, 144] 
init_filters 32 => 32 
aniso: False image_size_mm: [400.0, 400.0, 417.0] spacing: [0.78125, 0.78125, 1.0] levels: 5 

are determined from the auto scaling procedure (e.g. increasing batch size seems logical in this case) but the later message shows (immediately before the training):

Using num_epochs => 150
 Using start_epoch => 0
 batch_size => 1 
 num_crops_per_image => 2 
 num_steps_per_image => 1 
 num_warmup_epochs => 3 

[KumoLiu]

What is the expected compute budget for Auto3DSeg?

Just for your information.
https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg/tasks/kits23#validation-performance-nvidia-dgx-1-8x-v100-32g

Suggested network parameters:
Batch size 1 => 2
ROI size [224, 224, 144] => [224, 224, 144]
init_filters 32 => 32
aniso: False image_size_mm: [400.0, 400.0, 417.0] spacing: [0.78125, 0.78125, 1.0] levels: 5

These lines are suggested network parameters that may not modified before training.

[mibaumgartner]

Thanks for the quick answer.

(1) I saw the 8xV100 setup but the question is rather: Should Auto3dSeg only be run in 8xGPU setups in order to achieve good performance? - I would expect a certain gain from scaling the network etc. but not a huge leap in performance

(2) So am I expected to change the values manually in the config? I thought the goal is to have an "out-of-the box" experience