Distributions
Sampling matrices from distribution
cl-waffe2 provides a package :cl-waffe2/distributions
which is used to sample matrices from the distributions.
Common Format to the APIs
All sampling functions are defined in the following format via define-tensor-initializer
macro.
(function-name shape [Optional Arguments] &rest args &keys &allow-other-keys)
That is, arguments passed to the make-tensor
function can also be passed directly to the initializer functions.
Example
(normal `(10 10) 0.0 1.0 :requires-grad t)
{CPUTENSOR[float] :shape (10 10)
((0.019091532 1.077396 -0.17874254 ~ -0.7403823 0.15017667 -0.46685407)
(0.5039429 0.2985424 -0.5985508 ~ 0.58595663 0.21627972 -0.12598784)
...
(-1.534469 0.48690832 0.61201316 ~ 0.13288274 -0.96409506 0.3868878)
(0.90485555 0.105599456 0.6912249 ~ -0.7889702 -1.148289 -0.050823983))
:facet :exist
:requires-grad T
:backward NIL}
Example
(ax+b `(10 10) 1 0 :dtype :uint8)
{CPUTENSOR[uint8] :shape (10 10)
((0 1 2 ~ 7 8 9)
(10 11 12 ~ 17 18 19)
...
(80 81 82 ~ 87 88 89)
(90 91 92 ~ 97 98 99))
:facet :exist
:requires-grad NIL
:backward NIL}
define-tensor-initializer
(define-tensor-initializer (function-name (&rest args) initializer-lambda document &key (keep-order? nil)))
define-tensor-initializer
is a macro which is used to define a initializer function.
Initializer function is a function whose arguments follow this format:
(function-name shape <Initializer's Arguments> &rest initargs &key &allow-other-keys)
Input:
function-name - the function is defined after this argument
args - Initializer's Arguments
initializer-lambda - A form to be expanded as the sampling function, which must return a function of #'(lambda (i) ...) where i is the index of element.
keep-order? - set t if the index is needed to sampling matrices.
Example:
(define-initializer-function
uniform-random
(upfrom below)
(let ((upfrom (coerce upfrom (dtype->lisp-type (dtype tensor))))
(below (coerce below (dtype->lisp-type (dtype tensor)))))
#'(lambda (i)
(declare (ignore i))
(sample-uniform-random upfrom below)))
"")
(uniform-random `(10 10) 0.1 0.3 :requires-grad t)
{CPUTENSOR[float] :shape (10 10)
((0.13149574 0.15135926 0.1569588 ~ 0.103781514 0.20610212 0.19365484)
(0.2638953 0.12672275 0.21630599 ~ 0.16542184 0.10228193 0.12928057)
...
(0.20429519 0.12252951 0.17538154 ~ 0.22072719 0.18642941 0.11027551)
(0.14372297 0.11097031 0.25514898 ~ 0.28739202 0.18398522 0.15176433))
:facet :exist
:requires-grad T
:backward NIL}
(Note that new tensor is binded to tensor, being used to determined dtype etc...)
ax+b
(ax+b shape a b &rest initargs &key &allow-other-keys)
The function ax+b is a family of initializer functions, and samples matrices from arithmetic progression.
Inputs:
a, b - Coefficients of the above formula.
Example
(ax+b `(3 3) 1.0 0.0)
{CPUTENSOR[float] :shape (3 3)
((0.0 1.0 2.0)
(3.0 4.0 5.0)
(6.0 7.0 8.0))
:facet :exist
:requires-grad NIL
:backward NIL}
beta
(beta shape alpha beta &rest initargs &key &allow-other-keys)
The function beta is a family of initializer functions, and sample matrices from beta distribution.
Reference
-
Generating Beta Variates with Nonintegral Shape Parameters (R. C. H. Cheng University of Wales Institute of Science and Technology)
-
https://dl.acm.org/doi/pdf/10.1145/359460.359482
Note: My implementation is unstable, being occurs floating-overflow constantly..., especially when min(alpha, beta) < 1.0 (i.e.: beta-bc)
Example
(beta `(3 3) 5.0 1.0)
{CPUTENSOR[float] :shape (3 3)
((0.9526814 0.9772896 0.8532768)
(0.7357619 0.92763716 0.8122957)
(0.716915 0.900784 0.90241367))
:facet :exist
:requires-grad NIL
:backward NIL}
bernoulli
(bernoulli shape p &rest initargs &key &allow-other-keys)
The bernoulli is a family of initializer functions, and samples matrices from bernoulli distribution.
Inputs
p - Takes 1 with probability p and 0 with probalibity (1-p).
Example
(bernoulli `(3 3) 0.3)
{CPUTENSOR[float] :shape (3 3)
((0.0 0.0 1.0)
(1.0 0.0 0.0)
(0.0 1.0 0.0))
:facet :exist
:requires-grad NIL
:backward NIL}
chisquare
(chisquare shape df &rest initargs &key &allow-other-keys)
The function chisquare is a family of initializer functions, and samples matrices from chisquare distributions.
Inputs
df - degree of freedom.
References
https://github.com/lvaruzza/cl-randist
Example
(chisquare `(3 3) 1.0)
{CPUTENSOR[float] :shape (3 3)
((0.21670008 0.112398714 0.3001082)
(0.6785706 0.7514469 0.37084237)
(0.72724813 2.9567142 0.1113069))
:facet :exist
:requires-grad NIL
:backward NIL}
exponential
(exponential shape &rest initargs &key &allow-other-keys)
The function exponential is a family of initializer functions, and samples the exponential distribution using ziggurat algorithm with table-size=256.
References
-
https://andantesoft.hatenablog.com/entry/2023/04/30/183032
-
Marsaglia, G., & Tsang, W. W. (2000). The ziggurat method for generating random variables. Journal of statistical software.
-
https://marui.hatenablog.com/entry/2023/01/23/194507
Example
(exponential `(3 3))
{CPUTENSOR[float] :shape (3 3)
((0.18444276 1.1665336 0.5280462)
(0.41465604 1.4009397 0.671173)
(2.6738424 1.2597004 0.17590772))
:facet :exist
:requires-grad NIL
:backward NIL}
gamma
(gamma shape k &rest initargs &key &allow-other-keys)
The function gamma is a family of initializer functions, and samples matrices from the gamma distribution.
References
- https://github.com/lvaruzza/cl-randist
Example
(gamma `(3 3) 1.0)
{CPUTENSOR[float] :shape (3 3)
((0.25004625 0.066562936 3.1535413)
(0.42649975 1.1531353 0.9811732)
(1.3254013 0.67709714 0.3043218))
:facet :exist
:requires-grad NIL
:backward NIL}
normal
(normal shape mean stddev &rest initargs &key &allow-other-keys)
The function normal is a family of initializer functions, and samples matrices from normal distribution.
Reference
- https://github.com/lvaruzza/cl-randist (seems to create ziggurat table with size=128)
Inputs
mean
stddev - Standard Deviation, σ.
Example
(normal `(3 3) 1.0 0.0)
{CPUTENSOR[float] :shape (3 3)
((1.0 1.0 1.0)
(1.0 1.0 1.0)
(1.0 1.0 1.0))
:facet :exist
:requires-grad NIL
:backward NIL}
uniform-random
(uniform-random shape upfrom below &rest initargs &key &allow-other-keys)
The function uniform-random is a family of initializer functions, and samples matrices from uniform random distribution using Common Lisp's standard function, (random arg)
.
Input:
upfrom, below. Each elements of returned tensor is in the range of: `[upfrom, below)`
Example
(uniform-random `(3 3) 2 4)
{CPUTENSOR[float] :shape (3 3)
((3.3453193 3.0378416 2.4349144)
(2.584164 2.1954563 3.3052208)
(2.4049528 2.1151443 2.4065173))
:facet :exist
:requires-grad NIL
:backward NIL}
randn
(randn shape &rest initargs &key &allow-other-keys)
The function randn is a family of initializer functions, and samples the gaussian distributions using ziggurat algorithm with table-size=256.
References
-
https://andantesoft.hatenablog.com/entry/2023/04/30/183032
-
Marsaglia, G., & Tsang, W. W. (2000). The ziggurat method for generating random variables. Journal of statistical software.
-
https://marui.hatenablog.com/entry/2023/01/23/194507
Example
(randn `(3 3))
{CPUTENSOR[float] :shape (3 3)
((-0.50962603 0.970088 -0.23391165)
(-0.045766525 -1.1019369 0.6617144)
(-1.4078823 2.643538 0.81806624))
:facet :exist
:requires-grad NIL
:backward NIL}