Epoch counter does not resume when resuming from start checkpoint.

[dillfrescott]

It seems to reset to 0 every time

[ZFTurbo]

Yes, it's not saved in model data anywhere. Actually config can be saved inside too... I need to think what to save.

It's not actually an error, just not enough functionality.

[dillfrescott]

Gotcha. I've been manually adjusting the "for epoch in range" values in train.py every resume which works i guess.

[jarredou]

I've started working on a more "resume-friendly" fork a while ago with the --resume CLI args, and saving optimizer, scheduler states + epoch, best_sdr and last training loss values within the "last_xxx.ckpt" saved model (+ wandb logging here).
main...jarredou:Music-Source-Separation-Training:wandb+resume

Code is not bulletproof.

[dillfrescott]

Ah, thank you! @jarredou

[dillfrescott]

You should do a PR for that

[jarredou]

It would require more work for a PR, like I said it's not bulletproof in its current state and can lead to some errors, but since few months, I don't have free time to spend on this, unfortunately.

[dillfrescott]

Ah, gotcha