[package] cl-waffe2

The cl-waffe2 package provides utilities for a wide range needs: Object Convertion, Advance Network Construction, Trainer, and so on.

[Tensor Facet] Converting AbstractTensor <-> Anything

If you're looking for the way to create an AbstractTensor from a Common Lisp array or manipulate an AbstractTensor as a Common Lisp array, this section is perfect for you. Here we provide a common APIs for the conversion between AbstractTensor and other matrix types. The most basic method is a convert-tensor-facet method and we're welcome to add a new method by users. Other functions are macros work by assigning a method according to the type of object and the direction. Plus, conversions are performed while sharing pointers as much as possible. If turned out to be they can't be shared, the with-facet macro forces a copy to be performed and pseudo-synchronises them.

(convert-tensor-facet from to)

Converts the given object (anything is ok; from=AbstractTensor simple-array etc as long as declared) into the direction indicated in to.

Inputs

From[Anything] The object to be converted

To[Symbol] Indicates to where the object is converted

Adding an extension

Welcome to define the addition of method by users. For example, Fixnum -> AbstractTensor convertion can be written like:

(defmethod convert-tensor-facet ((from fixnum) (to (eql 'AbstractTensor)))
    (make-tensor from))

(print (change-facet 1 :direction 'AbstractTensor))

;;{SCALARTENSOR[float]   
;;    1.0
;;  :facet :exist
;;  :requires-grad NIL
;;  :backward NIL}

If any object to AbstractTensor conversion is implemented, it is strongly recommended to add it to this method.

Example

(convert-tensor-facet (randn `(3 3)) 'simple-array)

See also: convert-facet (more convenient API)

(change-facet (array-from &key (direction 'array)))

By calling the conver-tensor-facet method, this function can change the facet of given array-form into the direction. (Just an alias of (convert-tensor-facet array-from direction))

Standard Directions

We provide these symbols as a direction in standard.

array: Any Object -> Common Lisp Standard ND Array
simple-array: Any Object -> Common Lisp Simple-Array
AbstractTensor: Any Object -> AbstractTensor. If couldn't determine the dtype, dtype of the first element of array-from is used instead.

[function] device-as

(device-as tensor as)

Converts the given AbstractTensor as as.

Example

(device-as (randn `(3 3)) :as 'CPUTensor)

[function] ->tensor

Using the convert-tensor-facet method, converts the given object into AbstractTensor.

Example

(->tensor #2A((1 2 3) (4 5 6)))

(with-facet (var (object-from &key (direction 'simple-array)) &body body))

By using the convert-tensor-facetmethod, this macro changes the facet ofobject-frominto thedirection. If you want to apply any operations toobject-fromand ensure that modifications are applied to theobject-from, setsync=t and moves element forcibly (only available when direction='abstracttensor`). This is useful when editing AbstractTensor or calling other libraries without making copies.

For a more explict workflow, see below:

    [macro with-facet]
            ↓
[Binding var = (convert-tensor-facet object-from direction)] 
            ↓
      [Processing body]
            ↓
[If sync=t, (setf (tensor-vec object-from) (tensor-vec (convert-tensor-facet var 'AbstractTensor)))]

Example

(let ((a (randn `(3 3))))
    (with-facet (a* (a :direction 'simple-array))
        (print a*)
        (setf (aref a* 0) 10.0))
   a)

;; Operations called with simple-array a*, also effects on a.

#(0.92887694 -0.710253 1.2339028 -0.78008 1.6763965 0.93389416 -0.5691122
  1.6552123 -0.108502984) 
{CPUTENSOR[float] :shape (3 3)  
  ((10.0         -0.710253    1.2339028)
   (-0.78008     1.6763965    0.93389416)
   (-0.5691122   1.6552123    -0.108502984))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

Advanced Network Construction

Powerful macros in Common Lisp enabled me to provide an advanced APIs for make the construction of nodes more systematic, and elegant. Computational nodes that are lazy evaluated can be treated as pseudo-models, for example, even if they are created via functions. And, APIs in this section will make it easy to compose/compile several nodes.

[function] asnode

(asnode function &rest arguments)

Wraps the given function which is expected to create computation nodes with the Encapsulated-Node composite. That is, functions are regarded as a Composite and be able to use a variety of APIs (e.g.: call, call->, defmodel-as ...).

In principle, a function takes one argument and returns one value, but by adding more arguments the macro automatically wraps the function to satisfy it. For example, (asnode #'!add 1.0) is transformed into: #'(lambda (x) (!add x 1.0)). So the first arguments should receive AbstractTensor.

Usage: call->

It is not elegant to use call more than once when composing multiple models.

(call (AnyModel1)
      (call (AnyModel2)
             (call (AnyModel3) X)))

Instead, you can use the call-> function:

(call-> X
        (AnyModel1)
        (AnyModel2)
        (AnyModel3))

However, due to constrains of call, it is not possible to place functions here. asnode is exactly for this!

(call-> X
        (AnyModel1)
        (asnode #'!softmax)
        (asnode #'!view 0) ;; Slicing the tensor: (!view x 0 t ...)
        (asnode #'!add 1.0) ;; X += 1.0
        (asnode !matmul Y) ;; X <- Matmul(X, Y)
        )

Usage2: defmodel-as

The macro cl-waffe2/vm.nodes:defmodel-as is able to define new functions/nodes from existing Composite. However, this macro only needs the traced computation nodes information to do this. As the simplest case, compiling the AbstractNode SinNode (which is callable as !sin) into static function, matrix-sin.


;; The number of arguments is anything: (defmodel-as (asnode #'(lambda (x y z) ... is also ok

(defmodel-as (asnode #'!sin) :where (A[~] -> B[~]) :asif :function :named matrix-sin)

(matrix-sin (ax+b `(10 10) 0 1)) ;; <- No compiling overhead. Just works like Numpy

On a side note: Encapsulated-Node itself doesn't provide for :where declaration, but you can it with the keyword :where.

[macro] RepeatN

Creates an encapsulated node which repeats the given nodes for N times.

Example

(defsequence NCompose (N)
    (RepeatN N
        (asnode #'!sin)
        (asnode #'!cos)))

(cl-waffe2:dprint (call (NCompose 2) (randn `(3 10))))
Op:COSNODE{CPUTENSOR}
 |Op:SINNODE{CPUTENSOR}
   |Op:COSNODE{CPUTENSOR}
     |Op:SINNODE{CPUTENSOR}
       |<TMP:CPUTENSOR>TID398579(3 10)
       |Op:MOVETENSORNODE{CPUTENSOR}
         |<Input:CPUTENSOR>TID398582(3 10)
     |Op:MOVETENSORNODE{CPUTENSOR}
       |<Input:CPUTENSOR>TID398599(3 10)
   |Op:MOVETENSORNODE{CPUTENSOR}
     |<Input:CPUTENSOR>TID398620(3 10)
 |Op:MOVETENSORNODE{CPUTENSOR}
   |<Input:CPUTENSOR>TID398637(3 10)

[function] call->

(call-> input &rest nodes)

Starting from input, this macro applies a composed function.

(call-> (randn `(3 3))       ;; To the given input:
    (asnode #'!add 1.0)  ;;  |
    (asnode #'!relu)     ;;  | Applies operations in this order.
    (asnode #'!sum))))   ;;  ↓

nodes could be anything as long as the call method can handle, but I except node=Composite, AbstractNode, and (asnode function ...).

[macro] defsequence

(defsequence (name (&rest args) &optional docstring &rest nodes))

Defines a Composite that can be defined only by the call-> method.

Inputs

name[symbol] defines the new Composite after name

args[list] a list of arguments that used to initialize nodes. Not for call.

docstring[string] docstring

Example

(defsequence MLP (in-features)
    "Docstring (optional)"
    (LinearLayer in-features 512)
    (asnode #'!tanh)
    (LinearLayer 512 256)
    (asnode #'!tanh)
    (LinearLayer 256 10))

;; Sequence can receive a single argument.
(call (MLP 786) (randn `(10 786)))

Tips: Use (sequencelist-nth n sequence-model) to read the nth layer of sequence.

[macro] hooker

(hooker bind optimizer)

A convenient macro to hook AbstractOptimizers to each AbstractTensor. As the most straightforward explanation: this macro is expanded into this form.

`(lambda (,bind)
     (hook-optimizer! ,bind ,optimizer))

where bind is expected to be an AbstractTensor, optimizer is a creation form of AbstractOptimizer, and the function hook-optimizer! hooks the given optimizer into bind.

In cl-waffe2, one independent Optimizer must be initialised per parameter. This macro can be used to concisely describe the process of initialising the same Optimiser for many parameters.

Example

;; (model-parameters compiled-composite) to read the list of all parameters in the network

(let ((model (build (!matmul 
             (parameter (randn `(3 3)))
             (parameter (randn `(3 3)))))))

  (mapc (hooker x (Adam X :lr 1e-3)) (model-parameters model))

  (forward model)
  (backward model)

  (mapc #'call-optimizer! (model-parameters model)))

[macro] node->lambda

(node->lambda (&rest where) &body body)

Creates a lambda function obtained by tracing and compiling the computation node described in body.

Inputs

where declares the shape transforms. the tensor names used here are the same as those used in body. (i.e.: everything is AbstractTensor)

body Describe the construction of the computation node here.

Example

(node->lambda (A[~] -> B[~])
    (!sin (!cos a)))

(funcall * (randn `(3 3)))

Note

⚠️ Caches of functions are created for each location where this macro is located. Never place it inside a loop!

[macro] node->defun

(node->defun name (&rest where) &body body)

Defines a function obtained by tracing and compiling the computation node described in the body.

Inputs

name[symbol] the function is defined after it

where declares the shape transforms. the tensor names used here are the same as those used in body.

body Describe the construction of the computation node here.

Example

(node->defun log-softmax (A[~] -> OUT[~])
    (!softmax (!loge a) :axis 1))

(log-softmax (ax+b `(3 3) 0 1))
{CPUTENSOR[float] :shape (3 3) :id TID261835 
  ((0.33333334 0.33333334 0.33333334)
   (0.33333334 0.33333334 0.33333334)
   (0.33333334 0.33333334 0.33333334))
  :facet :input
  :belongs-to :memory-pool
  :requires-grad NIL
  :backward NIL}

[macro] node->defnode

(node->defun name (&rest where) &body body)

Defines a differentiable AbstractNode obtained by tracing and compiling the computation node described in the body.

Inputs

name[symbol] the function is defined after it

where declares the shape transforms. the tensor names used here are the same as those used in body.

body Describe the construction of the computation node here.

Example

(node->defnode log-softmax (A[~] -> OUT[~])
    (!softmax (!loge a) :axis 1))

(proceed (log-softmax (parameter (ax+b `(3 3) 0 1))))

[function] show-backends

(show-backends &key (stream t))

collects and displays the current state of devices to the given stream

Example

(show-backends)

─────[All Backends Tree]──────────────────────────────────────────────────

[*]CPUTENSOR: OpenBLAS=available *simd-extension-p*=available
    └[-]JITCPUTENSOR: compiler=gcc flags=(-fPIC -O3 -march=native) viz=NIL

[*]LISPTENSOR: Common Lisp implementation on matrix operations
    └[-]JITLISPTENSOR: To be deleted in the future release. do not use this.

[-]SCALARTENSOR: is a special tensor for representing scalar values.
    └[-]JITCPUSCALARTENSOR: Use with JITCPUTensor

([*] : in use, [-] : not in use.)
Add a current-backend-state method to display the status.
─────[*using-backend*]───────────────────────────────────────────────────

Priority: Higher <───────────────────>Lower
                  CPUTENSOR LISPTENSOR 

(use with-devices macro or set-devices-toplevel function to change this parameter.)

[function] set-devices-toplevel

(set-devices-toplevel &rest devices)

Declares devices to use.

[package] cl-waffe2

[Tensor Facet] Converting AbstractTensor <-> Anything

[generic] convert-tensor-facet

Inputs

Adding an extension

Example

[function] change-facet

Standard Directions

[function] device-as

Example

[function] ->tensor

Example

[macro] with-facet

Example

[macro] with-facets

Advanced Network Construction

[function] asnode

Usage: call->

Usage2: defmodel-as

[macro] RepeatN

Example

[function] call->

[macro] defsequence

Inputs

Example

[macro] hooker

Example

[macro] node->lambda

Inputs

Example

Note

[macro] node->defun

Inputs

Example

[macro] node->defnode

Inputs

Example

[function] show-backends

Example

[function] set-devices-toplevel