RAdam Warmup¶

class pytorch_warmup.radam.RAdamWarmup(optimizer, last_step=-1)[source]¶

RAdam warmup schedule.

This warmup scheme is described in On the adequacy of untuned warmup for adaptive optimization.

Parameters

optimizer (Optimizer) – an Adam optimizer
last_step (int) – The index of last step. (Default: -1)