OneCycleLR#

classtorch.optim.lr_scheduler.OneCycleLR(optimizer,max_lr,total_steps=None,epochs=None,steps_per_epoch=None,pct_start=0.3,anneal_strategy='cos',cycle_momentum=True,base_momentum=0.85,max_momentum=0.95,div_factor=25.0,final_div_factor=10000.0,three_phase=False,last_epoch=-1)[source]#

Sets the learning rate of each parameter group according to the 1cycle learning rate policy.

The 1cycle policy anneals the learning rate from an initial learning rate to some maximumlearning rate and then from that maximum learning rate to some minimum learning rate muchlower than the initial learning rate.This policy was initially described in the paperSuper-Convergence:Very Fast Training of Neural Networks Using Large Learning Rates.

The 1cycle learning rate policy changes the learning rate after every batch.step should be called after a batch has been used for training.

This scheduler is not chainable.

Note also that the total number of steps in the cycle can be determined in oneof two ways (listed in order of precedence):

1. A value for total_steps is explicitly provided.

2. A number of epochs (epochs) and a number of steps per epoch(steps_per_epoch) are provided.In this case, the number of total steps is inferred bytotal_steps = epochs * steps_per_epoch

You must either provide a value for total_steps or provide a value for bothepochs and steps_per_epoch.

The default behaviour of this scheduler follows the fastai implementation of 1cycle, whichclaims that “unpublished work has shown even better results by using only two phases”. Tomimic the behaviour of the original paper instead, setthree_phase=True.

Parameters

• optimizer (Optimizer) – Wrapped optimizer.

• max_lr (float orlist) – Upper learning rate boundaries in the cyclefor each parameter group.

• total_steps (int) – The total number of steps in the cycle. Note thatif a value is not provided here, then it must be inferred by providinga value for epochs and steps_per_epoch.Default: None

• epochs (int) – The number of epochs to train for. This is used alongwith steps_per_epoch in order to infer the total number of steps in the cycleif a value for total_steps is not provided.Default: None

• steps_per_epoch (int) – The number of steps per epoch to train for. This isused along with epochs in order to infer the total number of steps in thecycle if a value for total_steps is not provided.Default: None

• pct_start (float) – The percentage of the cycle (in number of steps) spentincreasing the learning rate.Default: 0.3

• anneal_strategy (str) – {‘cos’, ‘linear’}Specifies the annealing strategy: “cos” for cosine annealing, “linear” forlinear annealing.Default: ‘cos’

• cycle_momentum (bool) – IfTrue, momentum is cycled inverselyto learning rate between ‘base_momentum’ and ‘max_momentum’.Default: True

• base_momentum (float orlist) – Lower momentum boundaries in the cyclefor each parameter group. Note that momentum is cycled inverselyto learning rate; at the peak of a cycle, momentum is‘base_momentum’ and learning rate is ‘max_lr’.Default: 0.85

• max_momentum (float orlist) – Upper momentum boundaries in the cyclefor each parameter group. Functionally,it defines the cycle amplitude (max_momentum - base_momentum).Note that momentum is cycled inverselyto learning rate; at the start of a cycle, momentum is ‘max_momentum’and learning rate is ‘base_lr’Default: 0.95

• div_factor (float) – Determines the initial learning rate viainitial_lr = max_lr/div_factorDefault: 25

• final_div_factor (float) – Determines the minimum learning rate viamin_lr = initial_lr/final_div_factorDefault: 1e4

• three_phase (bool) – IfTrue, use a third phase of the schedule to annihilate thelearning rate according to ‘final_div_factor’ instead of modifying the secondphase (the first two phases will be symmetrical about the step indicated by‘pct_start’).

• last_epoch (int) – The index of the last batch. This parameter is used whenresuming a training job. Sincestep() should be invoked after eachbatch instead of after each epoch, this number represents the totalnumber ofbatches computed, not the total number of epochs computed.When last_epoch=-1, the schedule is started from the beginning.Default: -1

Example

>>>data_loader=torch.utils.data.DataLoader(...)>>>optimizer=torch.optim.SGD(model.parameters(),lr=1e-4,momentum=0.9)>>>scheduler=torch.optim.lr_scheduler.OneCycleLR(...optimizer,max_lr=0.01,steps_per_epoch=len(data_loader),epochs=10...)>>>forepochinrange(10):>>>forbatchindata_loader:>>>train_batch(...)>>>optimizer.step()>>>scheduler.step()

get_last_lr()[source]#

Return last computed learning rate by current scheduler.

Return type

list[float]

get_lr()[source]#

Compute the learning rate of each parameter group.

Return type

list[float]

load_state_dict(state_dict)[source]#

Load the scheduler’s state.

Parameters

state_dict (dict) – scheduler state. Should be an object returnedfrom a call tostate_dict().

state_dict()[source]#

Return the state of the scheduler as adict.

It contains an entry for every variable in self.__dict__ whichis not the optimizer.

Return type

dict[str,Any]

step(epoch=None)[source]#

Perform a step.