pytorch/ignite
create_lr_scheduler_with_warmup does not change init_lr to proper value
Closed
#2,441 opened on Jan 24, 2022
docsenhancementhelp wantedquestion
Repository metrics
- Stars
- (4,313 stars)
- PR merge metrics
- (Avg merge 18d 23h) (19 merged PRs in 30d)
Description
Hi,
In order to get expected sequence of lrs from create_lr_scheduler_with_warmup's scheduler, one must not attach it to engine on event EPOCH_COMPLETED because it produces the lr passed to optimizer's constructor at the beginning and then warmup lrs. This hurts warmup procedure. As a workaround, one could use event EPOCH_STARTED but it might not be a good solution.
It seems there should be something like below in line 1017 of param_scheduler.py within the for loop .
param_group['lr'] = warmup_start_value
To reproduce current behaviour:
param = nn.Parameter(torch.Tensor([3.]))
optimizer = torch.optim.SGD([param], lr=1e-3)
scheduler = StepLR(optimizer, 3)
with_warmup_scheduler = create_lr_scheduler_with_warmup(scheduler, warmup_start_value=1e-5, warmup_duration=3)
def process_func(e,b):
param.grad = torch.Tensor([1.])
optimizer.step()
trainer = Engine(process_func)
@trainer.on(Events.EPOCH_COMPLETED)
def _():
print(op.param_groups[0]['lr'])
trainer.add_event_handler(Events.EPOCH_COMPLETED, with_warmup_scheduler)
output:
0.001
1e-05
0.000505
0.001
0.001
0.001
0.0001
0.0001
0.0001
1e-05