cl-waffe

Optimizers

SGD

sgd
OptionValue
Constructor:(sgd params &key (lr 0.001) &aux (model-ident (gensym W)) (params params) (lr lr))
Predicate:sgd-p
Copier:copy-sgd
Print Function:(lambda (m stream k) (declare (ignore k)) (render-simple-model-structure stream m))

cl-waffe's Optimizer: SGD

This structure is an cl-waffe object
Overview
Simple SGD.
How to Initialize
(SGD &key (lr 1e-3)) => [OPTIMIZER: SGD]
Update
Following defnition.
Call update
use deftrainer and (update), or (call-forward (SGD) &rest args)
Call zero-grad
use deftrainer and (zero-grad) or (call-backward (SGD))
Object's slots
  • model-ident
    OptionValue
    Type:symbol
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-model-ident
    Initform:#:w0
  • hide-from-tree
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-hide-from-tree
    Initform:nil
  • forward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-forward
    Initform:t
  • object-type
    OptionValue
    Type:keyword
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-object-type
    Initform::optimizer
  • backward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-backward
    Initform:t
  • parameters
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-parameters
    Initform:(quote (cl-waffe.optimizers::params cl-waffe.optimizers::lr))
  • params
    OptionValue
    Type:hash-table
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-params
    Initform:cl-waffe.optimizers::params
  • lr
    OptionValue
    Type:float
    Read Only:nil
    Accessor:cl-waffe.optimizers::sgd-lr
    Initform:cl-waffe.optimizers::lr

Momentum

momentum
OptionValue
Constructor:(momentum params &key (momentum 0.9) (lr 0.001) &aux (model-ident (gensym W)) (params params) (lr lr) (momentum momentum) (velocities (make-hash-table)))
Predicate:momentum-p
Copier:copy-momentum
Print Function:(lambda (m stream k) (declare (ignore k)) (render-simple-model-structure stream m))

cl-waffe's Optimizer: Momentum

This structure is an cl-waffe object
Overview
Simple Momentum
Note
This code isn't optimized and slow
How to Initialize
(Momentum &key (momentum 0.9)(lr 1e-3)) => [OPTIMIZER: Momentum]
Update
Following definition.
Call update
use deftrainer and (update), or (call-forward (Momentum) &rest args)
Call zero-grad
use deftrainer and (zero-grad) or (call-backward (Momentum))
Object's slots
  • model-ident
    OptionValue
    Type:symbol
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-model-ident
    Initform:#:w0
  • hide-from-tree
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-hide-from-tree
    Initform:nil
  • forward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-forward
    Initform:t
  • object-type
    OptionValue
    Type:keyword
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-object-type
    Initform::optimizer
  • backward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-backward
    Initform:t
  • parameters
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-parameters
    Initform:(quote (cl-waffe.optimizers::params cl-waffe.optimizers::lr cl-waffe.optimizers:momentum cl-waffe.optimizers::velocities))
  • params
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-params
    Initform:cl-waffe.optimizers::params
  • lr
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-lr
    Initform:cl-waffe.optimizers::lr
  • momentum
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-momentum
    Initform:cl-waffe.optimizers:momentum
  • velocities
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::momentum-velocities
    Initform:(make-hash-table)

AdaGrad

adagrad
OptionValue
Constructor:(adagrad params &key (lr 0.001) (epsilon 1.0e-7) &aux (model-ident (gensym W)) (params params) (lr lr) (h (make-hash-table)) (epsilon epsilon))
Predicate:adagrad-p
Copier:copy-adagrad
Print Function:(lambda (m stream k) (declare (ignore k)) (render-simple-model-structure stream m))

cl-waffe's Optimizer: AdaGrad

This structure is an cl-waffe object
Overview
Simple AdaGrad
Note
The codes aren't optimized and slow. Todo: Write docs
How to Initialize
(AdaGrad describe like &rest this) => [OPTIMIZER: AdaGrad]
Update
Nothing
Call update
use deftrainer and (update), or (call-forward (AdaGrad) &rest args)
Call zero-grad
use deftrainer and (zero-grad) or (call-backward (AdaGrad))
Object's slots
  • model-ident
    OptionValue
    Type:symbol
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-model-ident
    Initform:#:w0
  • hide-from-tree
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-hide-from-tree
    Initform:nil
  • forward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-forward
    Initform:t
  • object-type
    OptionValue
    Type:keyword
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-object-type
    Initform::optimizer
  • backward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-backward
    Initform:t
  • parameters
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-parameters
    Initform:(quote (cl-waffe.optimizers::params cl-waffe.optimizers::lr cl-waffe.optimizers::h cl-waffe.optimizers::epsilon))
  • params
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-params
    Initform:cl-waffe.optimizers::params
  • lr
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-lr
    Initform:cl-waffe.optimizers::lr
  • h
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-h
    Initform:(make-hash-table)
  • epsilon
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::adagrad-epsilon
    Initform:cl-waffe.optimizers::epsilon

RMSProp

rmsprop
OptionValue
Constructor:(rmsprop params &key (lr 0.001) (epsilon 1.0e-7) (decay-rate 0.99) &aux (model-ident (gensym W)) (params params) (lr lr) (h (make-hash-table)) (epsilon epsilon) (decay-rate decay-rate))
Predicate:rmsprop-p
Copier:copy-rmsprop
Print Function:(lambda (m stream k) (declare (ignore k)) (render-simple-model-structure stream m))

cl-waffe's Optimizer: RMSProp

This structure is an cl-waffe object
Overview
Simple RMSProp
Note
Not Optimized and slow, todo: write docs
How to Initialize
(RMSProp describe like &rest this) => [OPTIMIZER: RMSProp]
Update
Nothing
Call update
use deftrainer and (update), or (call-forward (RMSProp) &rest args)
Call zero-grad
use deftrainer and (zero-grad) or (call-backward (RMSProp))
Object's slots
  • model-ident
    OptionValue
    Type:symbol
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-model-ident
    Initform:#:w0
  • hide-from-tree
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-hide-from-tree
    Initform:nil
  • forward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-forward
    Initform:t
  • object-type
    OptionValue
    Type:keyword
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-object-type
    Initform::optimizer
  • backward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-backward
    Initform:t
  • parameters
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-parameters
    Initform:(quote (cl-waffe.optimizers::params cl-waffe.optimizers::lr cl-waffe.optimizers::h cl-waffe.optimizers::epsilon cl-waffe.optimizers::decay-rate))
  • params
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-params
    Initform:cl-waffe.optimizers::params
  • lr
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-lr
    Initform:cl-waffe.optimizers::lr
  • h
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-h
    Initform:(make-hash-table)
  • epsilon
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-epsilon
    Initform:cl-waffe.optimizers::epsilon
  • decay-rate
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::rmsprop-decay-rate
    Initform:cl-waffe.optimizers::decay-rate

Adam

adam
OptionValue
Constructor:(adam params &key (lr 0.001) (epsilon 1.0e-7) (beta1 0.9) (beta2 0.999) &aux (model-ident (gensym W)) (params params) (lr lr) (m (make-hash-table)) (v (make-hash-table)) (n 0) (epsilon epsilon) (beta1 beta1) (beta2 beta2))
Predicate:adam-p
Copier:copy-adam
Print Function:(lambda (m stream k) (declare (ignore k)) (render-simple-model-structure stream m))

cl-waffe's Optimizer: Adam

This structure is an cl-waffe object
Overview
Simple Adam. It invokes kernel directly.
How to Initialize
(Adam &key (lr 1e-3)(epsilon 1e-7)(beta1 0.9)(beta2 0.999)) => [OPTIMIZER: Adam]
Update
Following definition.
Call update
use deftrainer and (update), or (call-forward (Adam) &rest args)
Call zero-grad
use deftrainer and (zero-grad) or (call-backward (Adam))
Object's slots
  • model-ident
    OptionValue
    Type:symbol
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-model-ident
    Initform:#:w0
  • hide-from-tree
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-hide-from-tree
    Initform:nil
  • forward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-forward
    Initform:t
  • object-type
    OptionValue
    Type:keyword
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-object-type
    Initform::optimizer
  • backward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-backward
    Initform:t
  • parameters
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-parameters
    Initform:(quote (cl-waffe.optimizers::params cl-waffe.optimizers::lr cl-waffe.optimizers::m cl-waffe.optimizers::v cl-waffe.optimizers::n cl-waffe.optimizers::epsilon cl-waffe.optimizers::beta1 cl-waffe.optimizers::beta2))
  • params
    OptionValue
    Type:hash-table
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-params
    Initform:cl-waffe.optimizers::params
  • lr
    OptionValue
    Type:float
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-lr
    Initform:cl-waffe.optimizers::lr
  • m
    OptionValue
    Type:hash-table
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-m
    Initform:(make-hash-table)
  • v
    OptionValue
    Type:hash-table
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-v
    Initform:(make-hash-table)
  • n
    OptionValue
    Type:fixnum
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-n
    Initform:0
  • epsilon
    OptionValue
    Type:float
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-epsilon
    Initform:cl-waffe.optimizers::epsilon
  • beta1
    OptionValue
    Type:float
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-beta1
    Initform:cl-waffe.optimizers::beta1
  • beta2
    OptionValue
    Type:float
    Read Only:nil
    Accessor:cl-waffe.optimizers::adam-beta2
    Initform:cl-waffe.optimizers::beta2

AdamW

RAdam

; More will be added...