LBFGS#

classtorch.optim.lbfgs.LBFGS(params,lr=1,max_iter=20,max_eval=None,tolerance_grad=1e-07,tolerance_change=1e-09,history_size=100,line_search_fn=None)[source]#

Implements L-BFGS algorithm.

Heavily inspired byminFunc.

Warning

This optimizer doesn’t support per-parameter options and parametergroups (there can be only one).

Warning

Right now all parameters have to be on a single device. This will beimproved in the future.

Note

This is a very memory intensive optimizer (it requires additionalparam_bytes*(history_size+1) bytes). If it doesn’t fit in memorytry reducing the history size, or use a different algorithm.

Parameters

• params (iterable) – iterable of parameters to optimize. Parameters must be real.

• lr (float,optional) – learning rate (default: 1)

• max_iter (int,optional) – maximal number of iterations per optimization step(default: 20)

• max_eval (int,optional) – maximal number of function evaluations per optimizationstep (default: max_iter * 1.25).

• tolerance_grad (float,optional) – termination tolerance on first order optimality(default: 1e-7).

• tolerance_change (float,optional) – termination tolerance on functionvalue/parameter changes (default: 1e-9).

• history_size (int,optional) – update history size (default: 100).

• line_search_fn (str,optional) – either ‘strong_wolfe’ or None (default: None).

add_param_group(param_group)[source]#

Add a param group to theOptimizer sparam_groups.

This can be useful when fine tuning a pre-trained network as frozen layers can be madetrainable and added to theOptimizer as training progresses.

Parameters

param_group (dict) – Specifies what Tensors should be optimized along with groupspecific optimization options.

load_state_dict(state_dict)[source]#

Load the optimizer state.

Parameters

state_dict (dict) – optimizer state. Should be an object returnedfrom a call tostate_dict().

Warning

Make sure this method is called after initializingtorch.optim.lr_scheduler.LRScheduler,as calling it beforehand will overwrite the loaded learning rates.

Note

The names of the parameters (if they exist under the “param_names” key of each param groupinstate_dict()) will not affect the loading process.To use the parameters’ names for custom cases (such as when the parameters in the loaded state dictdiffer from those initialized in the optimizer),a customregister_load_state_dict_pre_hook should be implemented to adapt the loaded dictaccordingly.Ifparam_names exist in loaded state dictparam_groups they will be saved and overridethe current names, if present, in the optimizer state. If they do not exist in loaded state dict,the optimizerparam_names will remain unchanged.

Example

>>>model=torch.nn.Linear(10,10)>>>optim=torch.optim.SGD(model.parameters(),lr=3e-4)>>>scheduler1=torch.optim.lr_scheduler.LinearLR(...optim,...start_factor=0.1,...end_factor=1,...total_iters=20,...)>>>scheduler2=torch.optim.lr_scheduler.CosineAnnealingLR(...optim,...T_max=80,...eta_min=3e-5,...)>>>lr=torch.optim.lr_scheduler.SequentialLR(...optim,...schedulers=[scheduler1,scheduler2],...milestones=[20],...)>>>lr.load_state_dict(torch.load("./save_seq.pt"))>>># now load the optimizer checkpoint after loading the LRScheduler>>>optim.load_state_dict(torch.load("./save_optim.pt"))

register_load_state_dict_post_hook(hook,prepend=False)[source]#

Register a load_state_dict post-hook which will be called afterload_state_dict() is called. It should have thefollowing signature:

hook(optimizer)->None

Theoptimizer argument is the optimizer instance being used.

The hook will be called with argumentself after callingload_state_dict onself. The registered hook can be used toperform post-processing afterload_state_dict has loaded thestate_dict.

Parameters

• hook (Callable) – The user defined hook to be registered.

• prepend (bool) – If True, the provided posthook will be fired beforeall the already registered post-hooks onload_state_dict. Otherwise,the providedhook will be fired after all the already registeredpost-hooks. (default: False)

Returns

a handle that can be used to remove the added hook by callinghandle.remove()

Return type

torch.utils.hooks.RemoveableHandle

register_load_state_dict_pre_hook(hook,prepend=False)[source]#

Register a load_state_dict pre-hook which will be called beforeload_state_dict() is called. It should have thefollowing signature:

hook(optimizer,state_dict)->state_dictorNone

Theoptimizer argument is the optimizer instance being used and thestate_dict argument is a shallow copy of thestate_dict the userpassed in toload_state_dict. The hook may modify the state_dict inplaceor optionally return a new one. If a state_dict is returned, it will be usedto be loaded into the optimizer.

The hook will be called with argumentself andstate_dict beforecallingload_state_dict onself. The registered hook can be used toperform pre-processing before theload_state_dict call is made.

Parameters

• hook (Callable) – The user defined hook to be registered.

• prepend (bool) – If True, the provided prehook will be fired beforeall the already registered pre-hooks onload_state_dict. Otherwise,the providedhook will be fired after all the already registeredpre-hooks. (default: False)

Returns

a handle that can be used to remove the added hook by callinghandle.remove()

Return type

torch.utils.hooks.RemoveableHandle

register_state_dict_post_hook(hook,prepend=False)[source]#

Register a state dict post-hook which will be called afterstate_dict() is called.

It should have the following signature:

hook(optimizer,state_dict)->state_dictorNone

The hook will be called with argumentsself andstate_dict after generatingastate_dict onself. The hook may modify the state_dict inplace or optionallyreturn a new one. The registered hook can be used to perform post-processingon thestate_dict before it is returned.

Parameters

• hook (Callable) – The user defined hook to be registered.

• prepend (bool) – If True, the provided posthook will be fired beforeall the already registered post-hooks onstate_dict. Otherwise,the providedhook will be fired after all the already registeredpost-hooks. (default: False)

Returns

a handle that can be used to remove the added hook by callinghandle.remove()

Return type

torch.utils.hooks.RemoveableHandle

register_state_dict_pre_hook(hook,prepend=False)[source]#

Register a state dict pre-hook which will be called beforestate_dict() is called.

It should have the following signature:

hook(optimizer)->None

Theoptimizer argument is the optimizer instance being used.The hook will be called with argumentself before callingstate_dict onself.The registered hook can be used to perform pre-processing before thestate_dictcall is made.

Parameters

• hook (Callable) – The user defined hook to be registered.

• prepend (bool) – If True, the provided prehook will be fired beforeall the already registered pre-hooks onstate_dict. Otherwise,the providedhook will be fired after all the already registeredpre-hooks. (default: False)

Returns

a handle that can be used to remove the added hook by callinghandle.remove()

Return type

torch.utils.hooks.RemoveableHandle

register_step_post_hook(hook)[source]#

Register an optimizer step post hook which will be called after optimizer step.

It should have the following signature:

hook(optimizer,args,kwargs)->None

Theoptimizer argument is the optimizer instance being used.

Parameters

hook (Callable) – The user defined hook to be registered.

Returns

a handle that can be used to remove the added hook by callinghandle.remove()

Return type

torch.utils.hooks.RemovableHandle

register_step_pre_hook(hook)[source]#

Register an optimizer step pre hook which will be called before optimizer step.

It should have the following signature:

hook(optimizer,args,kwargs)->Noneormodifiedargsandkwargs

Theoptimizer argument is the optimizer instance being used. Ifargs and kwargs are modified by the pre-hook, then the transformedvalues are returned as a tuple containing the new_args and new_kwargs.

Parameters

hook (Callable) – The user defined hook to be registered.

Returns

a handle that can be used to remove the added hook by callinghandle.remove()

Return type

torch.utils.hooks.RemovableHandle

state_dict()[source]#

Return the state of the optimizer as adict.

It contains two entries:

• state: a Dict holding current optimization state. Its content

  differs between optimizer classes, but some common characteristicshold. For example, state is saved per parameter, and the parameteritself is NOT saved.state is a Dictionary mapping parameter idsto a Dict with state corresponding to each parameter.

• param_groups: a List containing all parameter groups where each

  parameter group is a Dict. Each parameter group contains metadataspecific to the optimizer, such as learning rate and weight decay,as well as a List of parameter IDs of the parameters in the group.If a param group was initialized withnamed_parameters() the namescontent will also be saved in the state dict.

NOTE: The parameter IDs may look like indices but they are just IDsassociating state with param_group. When loading from a state_dict,the optimizer will zip the param_groupparams (int IDs) and theoptimizerparam_groups (actualnn.Parameter s) in order tomatch state WITHOUT additional verification.

A returned state dict might look something like:

{    'state': {        0: {'momentum_buffer': tensor(...), ...},        1: {'momentum_buffer': tensor(...), ...},        2: {'momentum_buffer': tensor(...), ...},        3: {'momentum_buffer': tensor(...), ...}    },    'param_groups': [        {            'lr': 0.01,            'weight_decay': 0,            ...            'params': [0]            'param_names' ['param0']  (optional)        },        {            'lr': 0.001,            'weight_decay': 0.5,            ...            'params': [1, 2, 3]            'param_names': ['param1', 'layer.weight', 'layer.bias'] (optional)        }    ]}

Return type

dict[str,Any]

step(closure)[source]#

Perform a single optimization step.

Parameters

closure (Callable) – A closure that reevaluates the modeland returns the loss.

zero_grad(set_to_none=True)[source]#

Reset the gradients of all optimizedtorch.Tensor s.

Parameters

set_to_none (bool,optional) –

Instead of setting to zero, set the grads to None. Default:True

This will in general have lower memory footprint, and can modestly improve performance.However, it changes certain behaviors. For example:

1. When the user tries to access a gradient and perform manual ops on it,a None attribute or a Tensor full of 0s will behave differently.

2. If the user requestszero_grad(set_to_none=True) followed by a backward pass,.gradsare guaranteed to be None for params that did not receive a gradient.

3. torch.optim optimizers have a different behavior if the gradient is 0 or None(in one case it does the step with a gradient of 0 and in the other it skipsthe step altogether).