({{, (RefLink[TrainingProgressCheckpointing, paclet:ref/TrainingProgressCheckpointing]) 
is an option for (RefLink[NetTrain, paclet:ref/NetTrain]) that specifies how to save copies of the net during training.}})

OpenersNotesSection(Spacer1)Details

With the default value of (RefLink[TrainingProgressCheckpointing, paclet:ref/TrainingProgressCheckpointing])->(RefLink[None, paclet:ref/None]), no checkpointing is done.

For (RefLink[TrainingProgressCheckpointing, paclet:ref/TrainingProgressCheckpointing])->spec, the following specifications can be used:

All nets are saved in (StringTypeLink["WLNet", paclet:ref/format/WLNet]) format.

If checkpointing is enabled, it will be done by default once every training round.

The suboption "Interval"->(RefLink[Quantity, paclet:ref/Quantity])[n,"unit"] specifies the interval at which to do checkpointing. Possible forms for "unit" include:

The suboption "MinimumInterval"->n specifies that checkpoints should not be performed more frequently than once every n seconds. If unspecified, there is no limit.

Multiple files or directories can be saved to with (RefLink[TrainingProgressCheckpointing, paclet:ref/TrainingProgressCheckpointing])->{(Subscript[spec, 1]),(Subscript[spec, 2]),…}.

With (RefLink[TrainingProgressCheckpointing, paclet:ref/TrainingProgressCheckpointing])->{"Directory","dir"}, individual files are named "starttime_counter_round_batch_loss.wlnet", where starttime is generated by (RefLink[DateString, paclet:ref/DateString])["ISODateTime"] at the start of training, counter is a value incremented after each call to (RefLink[NetTrain, paclet:ref/NetTrain]), round is the current round, batch is the cumulative batch number and loss is the most recent round loss. If a (RefLink[ValidationSet, paclet:ref/ValidationSet]) was specified, the most recent validation loss will also be included in the file name.

The list of checkpointing files created, if any, can be retrieved from a (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) via the property "CheckpointingFiles".

OpenersPrimaryExamplesSection(Spacer1)Examples  (1)

OpenersExampleSection0(Spacer1)Basic Examples  (1)

Take periodic checkpoints of a convolutional network during training on the MNIST dataset:

dir=CreateDirectory[]

lenet=NetChain[{
ConvolutionLayer[10,4],Ramp,PoolingLayer[2,2],
FlattenLayer[],500,Ramp,10,SoftmaxLayer[]},
"Input"->NetEncoder[{"Image",{28,28},"Grayscale"}],
"Output"->NetDecoder[{"Class",Range[0,9]}]];

trainingData=RandomSample[ResourceData["MNIST","TrainingData"], 5000];

result=NetTrain[lenet,trainingData,All,MaxTrainingRounds-> 10,TrainingProgressCheckpointing->{"Directory",dir}]

List all created checkpoints:

files=result["CheckpointingFiles"]

Import the final checkpoint:

Import[Last[files]]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[NetTrain, paclet:ref/NetTrain])  ▪  (RefLink[TrainingProgressReporting, paclet:ref/TrainingProgressReporting])  ▪  (RefLink[TrainingProgressFunction, paclet:ref/TrainingProgressFunction])}})}})

({{DocumentationFooterIconsRelatedTutorial.png, ({{Tutorials}, {▪(RefLinkPlain[Neural Networks in the Wolfram Language, paclet:tutorial/NeuralNetworksOverview])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Neural Networks, paclet:guide/NeuralNetworks])}})}})