Estimate costs to run experiments

[WillemSpek]

I want to run on a small subsample of the data in order to measure the costs in terms of compute, memory and network. Of course, compute appears to be the most relevant here.

[elboyran]

Do you have some estimates now?

[WillemSpek]

Yes, as a matter of fact I wanted to introduce it as a seperate PR. So far:

MNIST:

• CPU 19:44s for 225 configurations @ AMD Ryzen 5 5500U
• GPU 10:20s for 225 configurations @ NVIDIA T400

GPU is faster but the speedup is not as good as expected presumably due to excessive CPU-GPU transfers which is a difficult issue to resolve (we would have to reimplement all methods to optimize for GPU which requires linearization of masks with an image batch to bind IO to the GPU device).