Manual Warmup

class pytorch_warmup.base.BaseWarmup(optimizer, warmup_params, last_step=-1)

Base class for all warmup schedules

Parameters

• optimizer (Optimizer) – an instance of a subclass of Optimizer

• warmup_params (list) – warmup paramters

• last_step (int) – The index of last step. (Default: -1)

dampen(step=None)

Dampen the learning rates.

Parameters

step (int) – The index of current step. (Default: None)

load_state_dict(state_dict)

Loads the warmup scheduler’s state.

Parameters

state_dict (dict) – warmup scheduler state. Should be an object returned from a call to state_dict().

state_dict()

Returns the state of the warmup scheduler as a dict.

It contains an entry for every variable in self.__dict__ which is not the optimizer.

class pytorch_warmup.base.ExponentialWarmup(optimizer, warmup_period, last_step=-1)

Exponential warmup schedule.

Parameters

• optimizer (Optimizer) – an instance of a subclass of Optimizer

• warmup_period (int or list) – Effective warmup period

• last_step (int) – The index of last step. (Default: -1)

class pytorch_warmup.base.LinearWarmup(optimizer, warmup_period, last_step=-1)

Linear warmup schedule.

Parameters

• optimizer (Optimizer) – an instance of a subclass of Optimizer

• warmup_period (int or list) – Warmup period

• last_step (int) – The index of last step. (Default: -1)