Best weight check between training sessions and exported set_batch_network

[keko950]

Just modified detector.c in order to check the best weight map every time a training session is started as described in #3452.

If *best.weights exist in backup path, detector will do a map before start training, it will save it and initialize the best_map value, so now you can maintain your best weight across different training sessions :)

Edit:
Also updated network.h in order to add set_batch_network to the exported api, so you can use dynamic batching..

[skaldesh]

@keko950 could you please tell me, is the best file after training then always named _final.weights?

[keko950]

@keko950 could you please tell me, is the best file after training then always named _final.weights?

Nope, final and best weights are different.

[skaldesh]

Ok, I tested your PR and trained a model for 1000 batches. When the training was done, I got these 3 files: dnn_train_1000.weights, dnn_train_final.weights, dnn_train_last.weights
What happened to the _best.weights?

[skaldesh]

Could it be, that you first save a _best.weights, when the first comparison happened? So I should have used 2000 for max_batches.

[keko950]

Could it be, that you first save a _best.weights, when the first comparison happened? So I should have used 2000 for max_batches.

A _best.weights file is created after a map is done, so yes, probably you should increase your max_batches.

[skaldesh]

So I tried again, I have now the following files:

• darknet53.conv.74
• dnn_train_1000.weights
• dnn_train_2000.weights
• dnn_train_3000.weights
• dnn_train_final.weights
• dnn_train_last.weights

Still no _best.weights? My training is still running, does it need to be stopped to generate the is?

[keko950]

So I tried again, I have now the following files:

• darknet53.conv.74
• dnn_train_1000.weights
• dnn_train_2000.weights
• dnn_train_3000.weights
• dnn_train_final.weights
• dnn_train_last.weights

Still no _best.weights? My training is still running, does it need to be stopped to generate the is?

Just tested it again, working fine, trained 2000 iterations:
cam-lat_1000.weights
cam-lat_2000.weights
cam-lat_best.weights
cam-lat_final.weights
cam-lat_last.weights

What command are you running? Show me your ./darknet argument line
Also make sure you are running master version

[skaldesh]

What command are you running?

darknet detector train darknet.data dnn.cfg weights/darknet53.conv.74

Also make sure you are running master version

What do you mean? Of course I am running on your PR, since I want to test it

[keko950]

What command are you running?

darknet detector train darknet.data dnn.cfg weights/darknet53.conv.74

Also make sure you are running master version

What do you mean? Of course I am running on your PR, since I want to test it

Add -map option in order to generate best weights
Example:
./darknet detector train x.data y.cfg z.weights -map

[skaldesh]

got it! Thanks!

[skaldesh]

@AlexeyAB could you merge this please?

[skaldesh]

ping

[r0l1]

@AlexeyAB would appreciate this too. Useful feature!

[skaldesh]

@keko950 the appVeyor build failed, can you look into this?
@AlexeyAB is there interest in merging this? I personally do not want to constantly check my training output to find the best model