Distributions

Sampling matrices from distribution

cl-waffe2 provides a package :cl-waffe2/distributions which is used to sample matrices from the distributions.

Common Format to the APIs

All sampling functions are defined in the following format via define-tensor-initializer macro.

(function-name shape [Optional Arguments] &rest args &keys &allow-other-keys)

That is, arguments passed to the make-tensor function can also be passed directly to the initializer functions.

Example

(normal `(10 10) 0.0 1.0 :requires-grad t)

{CPUTENSOR[float] :shape (10 10)  
  ((0.019091532   1.077396      -0.17874254   ~ -0.7403823    0.15017667    -0.46685407)                     
   (0.5039429     0.2985424     -0.5985508    ~ 0.58595663    0.21627972    -0.12598784)   
                  ...
   (-1.534469     0.48690832    0.61201316    ~ 0.13288274    -0.96409506   0.3868878)
   (0.90485555    0.105599456   0.6912249     ~ -0.7889702    -1.148289     -0.050823983))
  :facet :exist
  :requires-grad T
  :backward NIL}

Example

(ax+b `(10 10) 1 0 :dtype :uint8)

{CPUTENSOR[uint8] :shape (10 10)  
  ((0  1  2  ~ 7  8  9)          
   (10 11 12 ~ 17 18 19)   
       ...
   (80 81 82 ~ 87 88 89)
   (90 91 92 ~ 97 98 99))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

define-tensor-initializer

(define-tensor-initializer (function-name (&rest args) initializer-lambda document &key (keep-order? nil)))

define-tensor-initializer is a macro which is used to define a initializer function.

Initializer function is a function whose arguments follow this format:

(function-name shape <Initializer's Arguments> &rest initargs &key &allow-other-keys)

Input:

function-name - the function is defined after this argument

args          - Initializer's Arguments

initializer-lambda - A form to be expanded as the sampling function, which must return a function of #'(lambda (i) ...) where i is the index of element.

keep-order? - set t if the index is needed to sampling matrices.

Example:

(define-initializer-function
    uniform-random
    (upfrom below)
  (let ((upfrom (coerce upfrom (dtype->lisp-type (dtype tensor))))
    (below  (coerce below  (dtype->lisp-type (dtype tensor)))))
    #'(lambda (i)
    (declare (ignore i))
    (sample-uniform-random upfrom below)))
    "")

(uniform-random `(10 10) 0.1 0.3 :requires-grad t)

{CPUTENSOR[float] :shape (10 10)  
  ((0.13149574  0.15135926  0.1569588   ~ 0.103781514 0.20610212  0.19365484)                   
   (0.2638953   0.12672275  0.21630599  ~ 0.16542184  0.10228193  0.12928057)   
                ...
   (0.20429519  0.12252951  0.17538154  ~ 0.22072719  0.18642941  0.11027551)
   (0.14372297  0.11097031  0.25514898  ~ 0.28739202  0.18398522  0.15176433))
  :facet :exist
  :requires-grad T
  :backward NIL}

(Note that new tensor is binded to tensor, being used to determined dtype etc...)

ax+b

(ax+b shape a b &rest initargs &key &allow-other-keys)

The function ax+b is a family of initializer functions, and samples matrices from arithmetic progression.

outn=an+b out_n = an + b

Inputs:

a, b - Coefficients of the above formula.

Example

(ax+b `(3 3) 1.0 0.0)

{CPUTENSOR[float] :shape (3 3)  
  ((0.0 1.0 2.0)
   (3.0 4.0 5.0)
   (6.0 7.0 8.0))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

beta

(beta shape alpha beta &rest initargs &key &allow-other-keys)

The function beta is a family of initializer functions, and sample matrices from beta distribution.

Reference

  1. Generating Beta Variates with Nonintegral Shape Parameters (R. C. H. Cheng University of Wales Institute of Science and Technology)

  2. https://dl.acm.org/doi/pdf/10.1145/359460.359482

Note: My implementation is unstable, being occurs floating-overflow constantly..., especially when min(alpha, beta) < 1.0 (i.e.: beta-bc)

Example

(beta `(3 3) 5.0 1.0)

{CPUTENSOR[float] :shape (3 3)  
  ((0.9526814  0.9772896  0.8532768)
   (0.7357619  0.92763716 0.8122957)
   (0.716915   0.900784   0.90241367))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

bernoulli

(bernoulli shape p &rest initargs &key &allow-other-keys)

The bernoulli is a family of initializer functions, and samples matrices from bernoulli distribution.

Inputs

p - Takes 1 with probability p and 0 with probalibity (1-p).

Example

(bernoulli `(3 3) 0.3)

{CPUTENSOR[float] :shape (3 3)  
  ((0.0 0.0 1.0)
   (1.0 0.0 0.0)
   (0.0 1.0 0.0))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

chisquare

(chisquare shape df &rest initargs &key &allow-other-keys)

The function chisquare is a family of initializer functions, and samples matrices from chisquare distributions.

Inputs

df - degree of freedom.

References

https://github.com/lvaruzza/cl-randist

Example

(chisquare `(3 3) 1.0)

{CPUTENSOR[float] :shape (3 3)  
  ((0.21670008  0.112398714 0.3001082)
   (0.6785706   0.7514469   0.37084237)
   (0.72724813  2.9567142   0.1113069))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

exponential

(exponential shape &rest initargs &key &allow-other-keys)

The function exponential is a family of initializer functions, and samples the exponential distribution using ziggurat algorithm with table-size=256.

References

  1. https://andantesoft.hatenablog.com/entry/2023/04/30/183032

  2. Marsaglia, G., & Tsang, W. W. (2000). The ziggurat method for generating random variables. Journal of statistical software.

  3. https://marui.hatenablog.com/entry/2023/01/23/194507

Example

(exponential `(3 3))

{CPUTENSOR[float] :shape (3 3)  
  ((0.18444276 1.1665336  0.5280462)
   (0.41465604 1.4009397  0.671173)
   (2.6738424  1.2597004  0.17590772))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

gamma

(gamma shape k &rest initargs &key &allow-other-keys)

The function gamma is a family of initializer functions, and samples matrices from the gamma distribution.

References

  1. https://github.com/lvaruzza/cl-randist

Example

(gamma `(3 3) 1.0)

{CPUTENSOR[float] :shape (3 3)  
  ((0.25004625  0.066562936 3.1535413)
   (0.42649975  1.1531353   0.9811732)
   (1.3254013   0.67709714  0.3043218))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

normal

(normal shape mean stddev &rest initargs &key &allow-other-keys)

The function normal is a family of initializer functions, and samples matrices from normal distribution.

Reference

  1. https://github.com/lvaruzza/cl-randist (seems to create ziggurat table with size=128)

Inputs

mean

stddev - Standard Deviation, σ.

Example

(normal `(3 3) 1.0 0.0)

{CPUTENSOR[float] :shape (3 3)  
  ((1.0 1.0 1.0)
   (1.0 1.0 1.0)
   (1.0 1.0 1.0))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

uniform-random

(uniform-random shape upfrom below &rest initargs &key &allow-other-keys)

The function uniform-random is a family of initializer functions, and samples matrices from uniform random distribution using Common Lisp's standard function, (random arg).

Input:

upfrom, below. Each elements of returned tensor is in the range of: `[upfrom, below)`

Example

(uniform-random `(3 3) 2 4)

{CPUTENSOR[float] :shape (3 3)  
  ((3.3453193 3.0378416 2.4349144)
   (2.584164  2.1954563 3.3052208)
   (2.4049528 2.1151443 2.4065173))
  :facet :exist
  :requires-grad NIL
  :backward NIL}

randn

(randn shape &rest initargs &key &allow-other-keys)

The function randn is a family of initializer functions, and samples the gaussian distributions using ziggurat algorithm with table-size=256.

References

  1. https://andantesoft.hatenablog.com/entry/2023/04/30/183032

  2. Marsaglia, G., & Tsang, W. W. (2000). The ziggurat method for generating random variables. Journal of statistical software.

  3. https://marui.hatenablog.com/entry/2023/01/23/194507

Example

(randn `(3 3))

{CPUTENSOR[float] :shape (3 3)  
  ((-0.50962603  0.970088     -0.23391165)
   (-0.045766525 -1.1019369   0.6617144)
   (-1.4078823   2.643538     0.81806624))
  :facet :exist
  :requires-grad NIL
  :backward NIL}