Manual Warmup

class pytorch_warmup.base.BaseWarmup(optimizer, warmup_params, last_step=-1)[source]

Base class for all warmup schedules

Parameters:

  • optimizer (Optimizer) – an instance of a subclass of Optimizer

  • warmup_params (list) – warmup paramters

  • last_step (int) – The index of last step. (Default: -1)

dampen(step=None)[source]

Dampen the learning rates.

Parameters:

step (int) – The index of current step. (Default: None)

load_state_dict(state_dict)[source]

Loads the warmup scheduler’s state.

Parameters:

state_dict (dict) – warmup scheduler state. Should be an object returned from a call to state_dict().

state_dict()[source]

Returns the state of the warmup scheduler as a dict.

It contains an entry for every variable in self.__dict__ which is not the optimizer.

class pytorch_warmup.base.ExponentialWarmup(optimizer, warmup_period, last_step=-1)[source]

Exponential warmup schedule.

Parameters:

  • optimizer (Optimizer) – an instance of a subclass of Optimizer

  • warmup_period (int or list) – Effective warmup period

  • last_step (int) – The index of last step. (Default: -1)

class pytorch_warmup.base.LinearWarmup(optimizer, warmup_period, last_step=-1)[source]

Linear warmup schedule.

Parameters:

  • optimizer (Optimizer) – an instance of a subclass of Optimizer

  • warmup_period (int or list) – Warmup period

  • last_step (int) – The index of last step. (Default: -1)