Skip to main content

​
lr_scheduler: <list[list[LRScheduler]] (Optional)

​
Description

A list of potential learning rate schedulers for AutoML to explore. A learning rate scheduler strategy can be empty (in which case no learning rate scheduling is applied), or can be configured through three parameters: name, interval and kwargs:
  • name: The name of the learning rate scheduler strategy. Valid learning rate scheduler strategies are:
    • constant_with_warmup: Uses a constant learning rate preceded by a warmup period which increases the learning rate from 0 to base_lr. Number of warmup steps can be specified through kwargs via warmup_ratio_or_steps.
    • linear_with_warmup: Decays the learning rate linearly from base_lr to 0, preceded by a warmup period which increases the learning rate from 0 to base_lr. Number of warmup steps can be specified through kwargs via warmup_ratio_or_steps. exponential: Decays the learning rate by gamma. gamma can be specified through kwargs.
    • cosine_with_warmup: Adjusts the learning rate between base_lr and 0 following a cosine function, preceded by a warmup period which increases the learning rate from 0 to base_lr. Number of warmup steps can be specified through kwargs via warmup_ratio_or_steps.
    • cosine_with_warmup_restarts: Adjusts the learning rate between base_lr and 0 following a cosine function, with several hard restarts. Preceded by a warmup period which increases the learning rate from 0 to base_lr. Number of hard restarts can be configured through kwargs via num_cycles (3 by default). Number of warmup steps can be specified through kwargs via warmup_ratio_or_steps.
  • interval: Specifies whether learning rate scheduling is applied per optimization step (step) or per epoch (epoch).
  • kwargs: Additional arguments depending on the chosen LR scheduler strategy. See above for detailed information.

​
Supported Task Types

  • All

​
Example

lr_scheduler:
- name: cosine_with_warumup_restarts
  interval: step
  kwargs:
    warmup_ratio_or_steps: 0.2
- name: exponential
  interval: epoch
  kwargs:
    gamma: 0.9

​
Default Values

lr_scheduler:
- name: cosine_with_warmup_restarts
  interval: step
  kwargs:
    warmup_ratio_or_steps: 0.1
- name: constant_with_warmup
  interval: step
  kwargs:
    warmup_ratio_or_steps: 0.1
- name: linear_with_warmup
  interval: step
  kwargs:
    warmup_ratio_or_steps: 0.1
- name: cosine_with_warmup
  interval: step
  kwargs:
    warmup_ratio_or_steps: 0.1