In this article

Optimizers

|
  • Share via
  •  Edit this article

Optimizers

[class] AbstractOptimizer

AbstractOptimizer is the base class for all optimizers. One AbstractOptimizer corresponds to one Tensor class with :requires-grad=T. (optimizer-param optimizer) to get the corresponding tensor.

[generic] step-optimizer

NIL

[function] hook-optimizers

(hook-optimizers avm hooker)

This function is used to hook the optimizers in the recognised parameters in avm-params-to-optimize. hooker is an function that takes one argument, which is the tensor that requires-grad=T, returns the AbstractOptimizer.

A list of created optimizers are returned.

[function] zero-grad

(zero-grad optimizer)
Fills the gradient of the optimizer with zeros.

[class] SGD

Implements SGD Optimizer:

Param_{new}\gets{Param - Param_{grad}\times{lr}}

where the initarg :lr is the learning rate.

[function] SGD

(SGD :lr 1e-3)
Returns a lambda function that takes one argument, which is the parameter tensor, and returns an instance of SGD optimizer.