> ## Documentation Index
> Fetch the complete documentation index at: https://kumo.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# lr_scheduler

#### `lr_scheduler: <list[list[LRScheduler]]` (Optional)

## Description

A list of potential learning rate schedulers for AutoML to explore.

A learning rate scheduler strategy can be empty (in which case no learning rate scheduling is applied), or can be configured through three parameters: `name`, `interval` and `kwargs`:

* `name`: The name of the learning rate scheduler strategy. Valid learning rate scheduler strategies are:

  * `constant_with_warmup`: Uses a constant learning rate preceded by a warmup period which increases the learning rate from 0 to `base_lr`. Number of warmup steps can be specified through `kwargs` via `warmup_ratio_or_steps`.
  * `linear_with_warmup`: Decays the learning rate linearly from `base_lr` to `0`, preceded by a warmup period which increases the learning rate from 0 to `base_lr`. Number of warmup steps can be specified through `kwargs` via `warmup_ratio_or_steps`.
  * `exponential`: Decays the learning rate by `gamma`. `gamma` can be specified through `kwargs`.
  * `cosine_with_warmup`: Adjusts the learning rate between `base_lr` and `0` following a cosine function, preceded by a warmup period which increases the learning rate from 0 to `base_lr`. Number of warmup steps can be specified through `kwargs` via `warmup_ratio_or_steps`.
  * `cosine_with_warmup_restarts`: Adjusts the learning rate between `base_lr` and `0` following a cosine function, with several hard restarts. Preceded by a warmup period which increases the learning rate from `0` to `base_lr`. Number of hard restarts can be configured through `kwargs` via `num_cycles` (`3` by default). Number of warmup steps can be specified through `kwargs` via `warmup_ratio_or_steps`.

* `interval`: Specifies whether learning rate scheduling is applied per optimization step (`step`) or per epoch (`epoch`).

* `kwargs`: Additional arguments depending on the chosen LR scheduler strategy. See above for detailed information.

### Supported Task Types

* All

### Example

<CodeGroup>
  ```yaml yaml theme={null}
  lr_scheduler:
  - name: cosine_with_warmup_restarts
    interval: step
    kwargs:
      warmup_ratio_or_steps: 0.2
  - name: exponential
    interval: epoch
    kwargs:
      gamma: 0.9
  ```
</CodeGroup>

### Default Values

<CodeGroup>
  ```yaml yaml theme={null}
  lr_scheduler:
  - name: cosine_with_warmup_restarts
    interval: step
    kwargs:
      warmup_ratio_or_steps: 0.1
  - name: constant_with_warmup
    interval: step
    kwargs:
      warmup_ratio_or_steps: 0.1
  - name: linear_with_warmup
    interval: step
    kwargs:
      warmup_ratio_or_steps: 0.1
  - name: cosine_with_warmup
    interval: step
    kwargs:
      warmup_ratio_or_steps: 0.1
  ```
</CodeGroup>