In this article

Model

|
  • Share via
  •  Edit this article

Models

TODO

[macro] defcall

(defcall (model-bind model) (&rest inputs) body0

A macro to write defmethod call in a more concise way.

Example

(defcall (model Transformer) (Tokens[Batch Seq-Len] Start-Pos[])
  (with-slots ((wte wte) (wpe wpe) (h h) (ln-f ln-f) (lm-head lm-head)) model
    (let* ((token-emb (forward wte tokens))
       (pos-emb   (forward wpe (!cast (!add start-pos (!index-components `(1 ,seq-len))) (dtype-of tokens))))
       (hi (!add token-emb pos-emb))
       (mask (!triu (!full `(1 1 ,seq-len ,(!+ start-pos (iconst seq-len))) (-inf)) :diagonal (!+ (iconst 1) start-pos)))
       (_ (dolist (hn h) (setf hi (forward hn hi mask start-pos))))
       (logits (forward lm-head (forward ln-f hi))))
      (declare (ignore _))
      ;; (!argmax (!view logits t -1 t))
      (!argmax logits))))

[macro] defsequence

(defsequence (name (&rest args) &optional docstring &rest nodes))

Defines a model which the definition is given as a sequence of nodes. Note that models defined by this macro only accept one input.

Example

(defsequence MLP (in-features hidden-dim out-features &key (activation #'!relu))
         (Linear in-features hidden-dim)
         (asnode activation)
         (Linear hidden-dim hidden-dim)
         (asnode activation)
         (Linear hidden-dim out-features))

[function] asnode

(asnode function &rest rest-args)

Wraps the function as a callable node by (forward ...). function is a function which takes one argument and returns one value.

rest-args is a place to pass additional arguments like: (asnode #'!leaky-relu :neg-slope 1e-2)