RAdam Warmup

class pytorch_warmup.radam.RAdamWarmup(optimizer, last_step=-1)

RAdam warmup schedule.

This warmup scheme is described in On the adequacy of untuned warmup for adaptive optimization.

Parameters:

• optimizer (Optimizer) – an Adam optimizer

• last_step (int) – The index of last step. (Default: -1)