This macro defines a cl-waffe model as name.

At the same time, a constructor name is defined and you can initialize your model like:

(cl-waffe.nn:LinearLayer 100 20) ; => [Model: Linearlayer]

name

Your model and constructor name

args

The arguments of a constructor

parameters

The parameters your model has.

Every time you initialize the model, the parameters are initialized.

Note that defmodel behaves like class.

The arguments are the same as defstruct

Format Example: ((param-name param-initial-value &key (type your-type)))

optimize

when t, your forward slot is defined with (declare (optimize (speed 3)(space 0)(debug 0))). It helps faster training after you ensured debugged.

forward

Define here the forward propagation of your model.

When backward, Automatic differentiation applies.

This macro allows to sequentially call layers.

the argument input must be a tensor.

Refering each layers from (self) macro, destructively modifying x with the returned value.

Note: This macro supposes models to be returned a single tensor, not a list.

(defmodel MLP (activation)
   :parameters ((layer1   (denselayer (* 28 28) 512 T activation))
   	        (layer2   (denselayer 512 256 T activation))
	        (layer3   (linearlayer 256 10 T)))
   :forward ((x)
	     (with-calling-layers x
	       (layer1 x)
 	       (layer2 x)
               (layer3 x))))

For the different arguments.

(with-calling-layers x
     (layer1 x 1 1)
     (layer2 1 x 2)
     (layer3 x y))

Output: An last value of layers.

Defining computation nodes.

defnode is useful when you want to define the derivative yourself.Note that parameter tensors in :parameter won't updated by optimizers.

If you want to update params, define additional models.

regard-as-node

When the slot :regard-as-node is nil, an optimizer in cl-waffe.caches regards this as model (i.e. argument could be destructed.) Default is t.

Note that:

1. :backward must return list, where that length corresponds with the length of input's argument, otherwise an error occurs when backward.
2. In forward and backward, computation node isn't needed to be continuous.However, the last values of :forward and :backward step, must posses :thread-data, which can be obtained by (waffetensor-thread-data tensor)

Example:

(defnode AddTensor nil
  :optimize t
  :parameters nil
  :forward  ((x y)
	     (with-searching-calc-node :add x y))
  :backward ((dy)(list dy dy)))

(call (AddTensor) tensor1 tensor2)

Reading the target's device, this macro invokes property codes described in :mgl, :mgl-cuda etc...

Dynamically defining and caching cpu and cuda kernel.

Every time reaches this macro, cl-waffe caches the target (i.e. the target is allowed to be destructed).

This macro won't create computation nodes.

The available slot is in *kernels*

When :mgl-cuda is nil, automatically calls :mgl

This macro returns the last value of called slots.

The last value of :mgl, :mgl-cuda and so on, must be type of list (cons), or mgl-mat:mat, waffetensorcontenttype.

Note: the target's thread-data must be already created. (i.e. By the time tensors reach this macro, at least once they needed to be pathed through Trainer or Model.) So, use this macro when you defining :forward and :backward in defnode macro because in defnode, backprop is disabled and computation nodes isn't always required.

Inputs

target

an target tensor.

var

where an copied tensor of target will be assigned.

:mgl

mgl-mat, when using cpu.

:mgl-mat

mgl-mat, when using cuda.

Return: An tensor (where tensor is made by sysconst)

Example:

(with-kernel-case x o
     :mgl (progn
            (axpy! 1.0 a o)) ; axpy! = !add
     :mgl-cuda nil) => #Const(((0.0 1.0 ~ 2.0 3.0)        
                 ...
      (0.0 4.0 ~ 5.0 6.0)) :mgl t :shape (10 10))

 ; This is useful when defining :backward
 (with-kernel-case x o
     :mgl (progn
            (list 1 1)))

Defining trainer, which is made in order to call train function.

The slots you defined can be invoked by using (step-model model &rest args), (predict model &rest args). See below.

model

An model defined by (defmodel) which you want to train.

optimizer

An optimizer defined by (defoptimizer)

optimizer-args

An arguments for optimizer

step-model

For each batch step, :step-model is called in (train) function. Describe here forward step, backward, zero-grad, update for training.

predict

an code for predicting

These macro below are defined by macrolet and you can use them in :step-model, :predict

(self name)

access trainer's parameters.

(model)

access trainer's model, defined by :model keyword.

(zero-grad)

Find model's all parameters and constants, and initialize their grads. (i.e. call optimizer's backward)

(update)

Find model's all parameters, and call optimizer and change parameter's data. (i.e. call optimizer's forward)

This trainer macro is defined in order to integrate following works:

1. calling models
2. calling criterions
3. calling backward
4. calling optimizer
5. calling zero-grad
6. defining predict

Example:

(deftrainer MLPTrainer (activation lr)
  :model          (MLP activation)
  :optimizer      cl-waffe.optimizers:Adam ; Note: :optimizer requires a single variable.
  :optimizer-args (:lr lr) ; these arguments directly expanded to optimizer's args.
  :step-model ((x y)
	       (zero-grad) ; call zero-grad
	       (let ((out (cl-waffe.nn:softmax-cross-entropy (call (model) x) y))) ; get criterion
		 (backward out) ; backward
		 (update) ; call optimizer
		 out)) ; return loss
 :predict ((x)(call (model) x))) ;for predict

(setq trainer (MLPTrainer :relu 1e-4)) ; init your trainer

; Train:   (step-model trainer model-input-x model-input-y)
; Predict: (predict trainer model-input-x)

Defining dataset. (This is kinda pytorch's dataloader)

The slots you defined can be invoked by using (get-dataset dataset index)(get-length dataset).

parameters

parameters datasets have.

next

when function (get-dataset dataset index) is called, this slot invokes. Return waffetensor for the next batch in response to your task.

length

In this form, the function must return the total length of your datasets where the value is fixnum. (Not a batch, and not a current index.)

(defdataset Mnistdata (train valid batch-size)
  :parameters ((train train)(valid valid)(batch-size batch-size))
  :next    ((index)
	    (list (!set-batch (self train) index (self batch-size))
		  (!set-batch (self valid) index (self batch-size))))
  :length (()(car (!shape (self train)))))

cl-waffe excepts index to be 1, 2, 3, ... (dataset-maxlen)

So, please manage batch-sizes in args and :next slots.

Defining optimizers. Internally, This is paraphase of defmodel, which slot names are just different.

Note: by calling :backward slot, optimizers work as (zero-grad).

Name

The optimizer's structure and constructor will be defined based on name

Args

Initializer of the optimizer. The first value of initializer is the hash-table that collected model's parameter where the key is fixnum from 0 to n. You have to store it.

parameters

An parameters that it has.

update

when training and (update) is called, this slot is called and you optimizer your parameters.

optimize

when t, the :update slot is defined with (optimize (speed 3)(space 0)(debug 0)) Default: nil

document

docstring for optimizers. You can use string or (with-usage) macro

Example:

;defoptimizer's args must start with params (symbol-name doesn't matter) which receives hash-table whose key is 1..n

(defoptimizer SGD (params &key (lr 1e-3))
  :optimize t
  :parameters ((params params :type hash-table)
               (lr lr :type single-float))
  :update (()
       (dotimes (i (hash-table-count (self params)))
         ; W(n+1) = W(n) - n * grad
         (!modify (gethash i (self params))) :+=
               (!mul (self lr)(grad (gethash i (self params)))))))

In :document slot, (for `defmodel, defnode, defoptimizer deftrainer defdataset`) this macro will be useful.

Keyword: step-args, arguments for :forward or :step-model, :next.

cl-waffe automatically generate docstrings.

Todo:Write Docs