\chapter{{\tt PatchAndGoInfo}: Pivot Modification Object}
\par
On occasion, an application will demand specific behavior
during a factorization.
We have written the {\tt PatchAndGoInfo} object to communicate
information to the {\tt Chv} object during a factorization of a front.
Most users can ignore this object.
However, if a different type of behavior is required, one could
extend this object by adding a new strategy to it and modifying the
{\tt Chv} methods that factor a front.
\par
Let us describe two strategies that we presently support.
\begin{itemize}
\item
Primal-dual linear programming may require repeated factorizations
of matrices of the form $AD^2A^T$, where $A$ comes from constraint
equations and $D$ is a diagonal matrix.
As the optimization proceeds,
$AD^2A^T$ becomes increasingly ill-conditioned because the
entries in $D$ go to zero or infinity.
Normally, when a small or zero pivot element is detected, we would
either signal an error (if we expected the matrix to be positive
definite) or pivot for stability.
However, in the primal-dual pivot context, a small or zero element
on the diagonal is not a calamity.
It signals that the variable associated with the small entry can be
``skipped'' in the solution process.
There are several ways to implement this behavior.
We have chosen a simple way: the diagonal entry is set to 1.0
and all off-diagonal entries in the corresponding column of $L$
are set to zero.
\item
In structural analysis, ``multi-point constraints'' are often
applied to a linear system. At times, applying these constraints
generates a matrix that is essentially singular. The singularity
may be benign, as in the following case.
$$
\left \lbrack \begin{array}{cc}
A_{1,1} & 0 \\
0 & A_{2,2}
\end{array} \right \rbrack
\left \lbrack \begin{array}{c}
X_1 \\
X_2
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{c}
0 \\
B_2
\end{array} \right \rbrack
$$
If $A_{1,1}$ is singular, the solution $X_1 = 0$ and
$X_2 = A_{2,2}^{-1} B_2$ is perfectly acceptable.
In other cases, the location of the singularity can be
communicated back to the user to supply useful information
about the finite element model.
One common practice is to not use pivoting, but to check the
magnitude of the diagonal entry as a row and column is to be eliminated.
If the magnitude is smaller than a user-supplied parameter,
the diagonal entry is set to some multiple of the largest
offdiagonal entry in that row and column of the front,
the location and perturbation is noted,
and the factorization proceeds.
\end{itemize}
\par
Other strategies can be added to the {\tt PatchAndGoInfo} object.
For example, if a matrix is being factored that is believed to be
positive definite, and a negative value is found in a pivot
element, one could abort the factorization, or perturb the element
so that it is positive.


Implements the Gr\"obner bases based procedure as presented in~\cite{JLCA_CAI13}. In general, this procedure can detect only the unsatisfiability of a conjunction of equations. This module also supports the usage of these equations to further simplify all constraints in the conjunction of constraints forming its input and passes these simplified constraints to its backends. However, it cannot be guaranteed that backends perform better on the simplified constraints than on the constraints before simplification.

\paragraph{Efficiency} The worst case complexity of the underlying procedure is exponential in the number of variables of the input constraints. In the case that the conjunction of constraints to check for satisfiability contains equations, this module can be more efficient than other modules for NRA on finding out inconsistency.


% Generated by Sphinx.
%\usepackage[utf8]{inputenc}
%\usepackage[T1]{fontenc}
%\usepackage{babel}
%\usepackage{times}
%\usepackage[Bjarne]{fncychap}

PyMC provides 35 built-in probability distributions. For each distribution, PyMC provides:
\begin{itemize}
    \item A function that evaluates its log-probability or log-density, for example \code{normal_like()}.
    \item A function that draws random variables, for example \code{rnormal()}.
    \item A function that computes the expectation associated with the distribution, for example \code{normal_expval()}.
    \item A \code{Stochastic} subclass generated from the distribution, for example \code{Normal}.
\end{itemize}

This section describes the likelihood functions of these distributions.
\index{pymc.distributions (module)}
\hypertarget{module-pymc.distributions}{}
%\declaremodule[pymc.distributions]{}{pymc.distributions}

\small

\section{Discrete distributions}
\index{bernoulli\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.bernoulli_like}{}\begin{funcdesc}{bernoulli\_like}{x, p}
The Bernoulli distribution describes the probability of successes (x=1) and
failures (x=0).
\begin{gather}
\begin{split}f(x \mid p) = p^{x} (1-p)^{1-x}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Series of successes (1) and failures (0). $x=0,1$

\item {}
\emph{p} : Probability of success. $0 < p < 1$.

\end{itemize}

\item[Example] \leavevmode
\begin{Verbatim}[commandchars=@\[\]]
@PYGaQ[@textgreater[]@textgreater[]@textgreater[] ]bernoulli@_like(@PYGZlb[]@PYGaw[0],@PYGaw[1],@PYGaw[0],@PYGaw[1]@PYGZrb[], @PYGbe[.]@PYGaw[4])
@PYGaa[-2.8542325496673584]
\end{Verbatim}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(x)= p$

\item {}
$Var(x)= p(1-p)$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{binomial\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.binomial_like}{}\begin{funcdesc}{binomial\_like}{x, n, p}
Binomial log-likelihood.  The discrete probability distribution of the
number of successes in a sequence of n independent yes/no experiments,
each of which yields success with probability p.
\begin{gather}
\begin{split}f(x \mid n, p) = \frac{n!}{x!(n-x)!} p^x (1-p)^{n-x}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} Number of successes, \textgreater{} 0.

\item {}
\emph{n} : {[}int{]} Number of Bernoulli trials, \textgreater{} x.

\item {}
\emph{p} : Probability of success in each trial, $p \in [0,1]$.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X)=np$

\item {}
$Var(X)=np(1-p)$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{categorical\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.categorical_like}{}\begin{funcdesc}{categorical\_like}{x, p}
Categorical log-likelihood. The most general discrete distribution.
\begin{gather}
\begin{split}f(x=i \mid p) = p_i\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
for $i \in 0 \ldots k-1$.
\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} $x \in 0\ldots k-1$

\item {}
\emph{p} : {[}float{]} $p > 0$, $\sum p = 1$

\end{itemize}

\end{description}
\end{funcdesc}
\index{discrete\_uniform\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.discrete_uniform_like}{}\begin{funcdesc}{discrete\_uniform\_like}{x, lower, upper}
Discrete uniform log-likelihood.
\begin{gather}
\begin{split}f(x \mid lower, upper) = \frac{1}{upper-lower}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} $lower \leq x \leq upper$

\item {}
\emph{lower} : Lower limit.

\item {}
\emph{upper} : Upper limit (upper \textgreater{} lower).

\end{itemize}

\end{description}
\end{funcdesc}
\index{geometric\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.geometric_like}{}\begin{funcdesc}{geometric\_like}{x, p}
Geometric log-likelihood. The probability that the first success in a
sequence of Bernoulli trials occurs on the x'th trial.
\begin{gather}
\begin{split}f(x \mid p) = p(1-p)^{x-1}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} Number of trials before first success (x \textgreater{} 0).

\item {}
\emph{p} : Probability of success on an individual trial, $p \in [0,1]$

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X)=1/p$

\item {}
$Var(X)=\frac{1-p}{p^2}$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{hypergeometric\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.hypergeometric_like}{}\begin{funcdesc}{hypergeometric\_like}{x, n, m, N}
Hypergeometric log-likelihood. Discrete probability distribution that
describes the number of successes in a sequence of draws from a finite
population without replacement.
\begin{gather}
\begin{split}f(x \mid n, m, N) = \frac{\binom{m}{x}\binom{N-m}{n-x}}{\binom{N}{n}}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} Number of successes in a sample drawn from a population.

\item {}
\emph{n} : {[}int{]} Size of sample drawn from the population.

\item {}
\emph{m} : {[}int{]} Number of successes in the population.

\item {}
\emph{N} : {[}int{]} Total number of units in the population.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}
$E(X) = \frac{n n}{N}$
\end{notice}
\end{funcdesc}
\index{negative\_binomial\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.negative_binomial_like}{}\begin{funcdesc}{negative\_binomial\_like}{x, mu, alpha}
Negative binomial log-likelihood. The negative binomial distribution describes a
Poisson random variable whose rate parameter is gamma distributed. PyMC's chosen
parameterization is based on this mixture interpretation.
\begin{gather}
\begin{split}f(x \mid \mu, \alpha) = \frac{\Gamma(x+\alpha)}{x! \Gamma(\alpha)} (\alpha/(\mu+\alpha))^\alpha (\mu/(\mu+\alpha))^x\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input data (x \textgreater{} 0).

\item {}
\emph{mu} : mu \textgreater{} 0

\item {}
\emph{alpha} : alpha \textgreater{} 0

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E[x]=\mu$

\item {}
In Wikipedia's parameterization,
$r=\alpha$
$p=\alpha/(\mu+\alpha)$
$\mu=r(1-p)/p$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{poisson\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.poisson_like}{}\begin{funcdesc}{poisson\_like}{x, mu}
Poisson log-likelihood. The Poisson is a discrete probability distribution.
It is often used to model the number of events occurring in a fixed period of
time when the times at which events occur are independent. The Poisson
distribution can be derived as a limiting case of the binomial distribution.
\begin{gather}
\begin{split}f(x \mid \mu) = \frac{e^{-\mu}\mu^x}{x!}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} $x \in {0,1,2,...}$

\item {}
\emph{mu} : Expected number of occurrences during the given interval, $\mu \geq 0$.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(x)=\mu$

\item {}
$Var(x)=\mu$

\end{itemize}
\end{notice}
\end{funcdesc}

\section{Continuous distributions}
\index{beta\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.beta_like}{}\begin{funcdesc}{beta\_like}{x, alpha, beta}
Beta log-likelihood. The conjugate prior for the parameter :math: \emph{p} of the binomial distribution.
\begin{gather}
\begin{split}f(x \mid \alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) \Gamma(\beta)} x^{\alpha - 1} (1 - x)^{\beta - 1}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : 0 \textless{} x \textless{} 1

\item {}
\emph{alpha} : alpha \textgreater{} 0

\item {}
\emph{beta} : beta \textgreater{} 0

\end{itemize}

\item[Example] \leavevmode
\begin{Verbatim}[commandchars=@\[\]]
@PYGaQ[@textgreater[]@textgreater[]@textgreater[] ]beta@_like(@PYGbe[.]@PYGaw[4],@PYGaw[1],@PYGaw[2])
@PYGaa[0.18232160806655884]
\end{Verbatim}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X)=\frac{\alpha}{\alpha+\beta}$

\item {}
$Var(X)=\frac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{cauchy\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.cauchy_like}{}\begin{funcdesc}{cauchy\_like}{x, alpha, beta}
Cauchy log-likelihood. The Cauchy distribution is also known as the
Lorentz or the Breit-Wigner distribution.
\begin{gather}
\begin{split}f(x \mid \alpha, \beta) = \frac{1}{\pi \beta [1 + (\frac{x-\alpha}{\beta})^2]}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{alpha} : Location parameter.

\item {}
\emph{beta} : Scale parameter \textgreater{} 0.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
Mode and median are at alpha.

\end{itemize}
\end{notice}
\end{funcdesc}
\index{chi2\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.chi2_like}{}\begin{funcdesc}{chi2\_like}{x, nu}
Chi-squared $\chi^2$ log-likelihood.
\begin{gather}
\begin{split}f(x \mid \nu) = \frac{x^{(\nu-2)/2}e^{-x/2}}{2^{\nu/2}\Gamma(\nu/2)}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : \textgreater{} 0

\item {}
\emph{nu} : {[}int{]} Degrees of freedom ( nu \textgreater{} 0 )

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X)=\nu$

\item {}
$Var(X)=2\nu$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{degenerate\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.degenerate_like}{}\begin{funcdesc}{degenerate\_like}{x, k}
Degenerate log-likelihood.
\begin{gather}
\begin{split}f(x \mid k) = \left\{ \begin{matrix} 1 \text{ if } x = k \\ 0 \text{ if } x \ne k\end{matrix} \right.\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input value.

\item {}
\emph{k} : Degenerate value.

\end{itemize}

\end{description}
\end{funcdesc}
\index{exponential\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.exponential_like}{}\begin{funcdesc}{exponential\_like}{x, beta}
Exponential log-likelihood.

The exponential distribution is a special case of the gamma distribution
with alpha=1. It often describes the time until an event.
\begin{gather}
\begin{split}f(x \mid \beta) = \frac{1}{\beta}e^{-x/\beta}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : x \textgreater{} 0

\item {}
\emph{beta} : Survival parameter (beta \textgreater{} 0).

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X) = \beta$

\item {}
$Var(X) = \beta^2$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{exponweib\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.exponweib_like}{}\begin{funcdesc}{exponweib\_like}{x, alpha, k, loc=0, scale=1}
Exponentiated Weibull log-likelihood.

The exponentiated Weibull distribution is a generalization of the Weibull
family. Its value lies in being able to model monotone and non-monotone
failure rates.
\begin{gather}
\begin{split}f(x \mid \alpha,k,loc,scale)  & = \frac{\alpha k}{scale} (1-e^{-z^k})^{\alpha-1} e^{-z^k} z^{k-1} \\
z & = \frac{x-loc}{scale}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : x \textgreater{} 0

\item {}
\emph{alpha} : Shape parameter

\item {}
\emph{k} : k \textgreater{} 0

\item {}
\emph{loc} : Location parameter

\item {}
\emph{scale} : Scale parameter (scale \textgreater{} 0).

\end{itemize}

\end{description}
\end{funcdesc}
\index{gamma\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.gamma_like}{}\begin{funcdesc}{gamma\_like}{x, alpha, beta}
Gamma log-likelihood.

Represents the sum of alpha exponentially distributed random variables, each
of which has mean beta.
\begin{gather}
\begin{split}f(x \mid \alpha, \beta) = \frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : math:\emph{x ge 0}

\item {}
\emph{alpha} : Shape parameter (alpha \textgreater{} 0).

\item {}
\emph{beta} : Scale parameter (beta \textgreater{} 0).

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X) = \frac{\alpha}{\beta}$

\item {}
$Var(X) = \frac{\alpha}{\beta^2}$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{half\_normal\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.half_normal_like}{}\begin{funcdesc}{half\_normal\_like}{x, tau}
Half-normal log-likelihood, a normal distribution with mean 0 limited
to the domain $x \in [0, \infty)$.
\begin{gather}
\begin{split}f(x \mid \tau) = \sqrt{\frac{2\tau}{\pi}}\exp\left\{ {\frac{-x^2 \tau}{2}}\right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : $x \ge 0$

\item {}
\emph{tau} : tau \textgreater{} 0

\end{itemize}

\end{description}
\end{funcdesc}
\index{hypergeometric\_like() (in module pymc.distributions)}

\begin{funcdesc}{hypergeometric\_like}{x, n, m, N}
Hypergeometric log-likelihood. Discrete probability distribution that
describes the number of successes in a sequence of draws from a finite
population without replacement.
\begin{gather}
\begin{split}f(x \mid n, m, N) = \frac{\binom{m}{x}\binom{N-m}{n-x}}{\binom{N}{n}}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int{]} Number of successes in a sample drawn from a population.

\item {}
\emph{n} : {[}int{]} Size of sample drawn from the population.

\item {}
\emph{m} : {[}int{]} Number of successes in the population.

\item {}
\emph{N} : {[}int{]} Total number of units in the population.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}
$E(X) = \frac{n n}{N}$
\end{notice}
\end{funcdesc}
\index{inverse\_gamma\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.inverse_gamma_like}{}\begin{funcdesc}{inverse\_gamma\_like}{x, alpha, beta}
Inverse gamma log-likelihood, the reciprocal of the gamma distribution.
\begin{gather}
\begin{split}f(x \mid \alpha, \beta) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} x^{-\alpha - 1} \exp\left(\frac{-\beta}{x}\right)\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : x \textgreater{} 0

\item {}
\emph{alpha} : Shape parameter (alpha \textgreater{} 0).

\item {}
\emph{beta} : Scale parameter (beta \textgreater{} 0).

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}
$E(X)=\frac{\beta}{\alpha-1}$  for $\alpha > 1$
$Var(X)=\frac{\beta^2}{(\alpha-1)^2(\alpha)}$  for $\alpha > 2$
\end{notice}
\end{funcdesc}
\index{laplace\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.laplace_like}{}\begin{funcdesc}{laplace\_like}{x, mu, tau}
Laplace (double exponential) log-likelihood.

The Laplace (or double exponential) distribution describes the
difference between two independent, identically distributed exponential
events. It is often used as a heavier-tailed alternative to the normal.
\begin{gather}
\begin{split}f(x \mid \mu, \tau) = \frac{\tau}{2}e^{-\tau |x-\mu|}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : $-\infty < x < \infty$

\item {}
\emph{mu} : Location parameter :math: \emph{-infty \textless{} mu \textless{} infty}

\item {}
\emph{tau} : Scale parameter $\tau > 0$

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X) = \mu$

\item {}
$Var(X) = \frac{2}{\tau^2}$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{logistic\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.logistic_like}{}\begin{funcdesc}{logistic\_like}{x, mu, tau}
Logistic log-likelihood.

The logistic distribution is often used as a growth model; for example,
populations, markets. Resembles a heavy-tailed normal distribution.
\begin{gather}
\begin{split}f(x \mid \mu, tau) = \frac{\tau \exp(-\tau[x-\mu])}{[1 + \exp(-\tau[x-\mu])]^2}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : $-\infty < x < \infty$

\item {}
\emph{mu} : Location parameter $-\infty < mu < \infty$

\item {}
\emph{tau} : Scale parameter (tau \textgreater{} 0)

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X) = \mu$

\item {}
$Var(X) = \frac{\pi^2}{3\tau^2}$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{lognormal\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.lognormal_like}{}\begin{funcdesc}{lognormal\_like}{x, mu, tau}
Log-normal log-likelihood. Distribution of any random variable whose
logarithm is normally distributed. A variable might be modeled as
log-normal if it can be thought of as the multiplicative product of many
small independent factors.
\begin{gather}
\begin{split}f(x \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}}\frac{
\exp\left\{ -\frac{\tau}{2} (\ln(x)-\mu)^2 \right\}}{x}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : x \textgreater{} 0

\item {}
\emph{mu} : Location parameter.

\item {}
\emph{tau} : Scale parameter (tau \textgreater{} 0).

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}
$E(X)=e^{\mu+\frac{1}{2\tau}}$
$Var(X)=(e^{1/\tau}-1)e^{2\mu+\frac{1}{\tau}}$
\end{notice}
\end{funcdesc}
\index{normal\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.normal_like}{}\begin{funcdesc}{normal\_like}{x, mu, tau}
Normal log-likelihood.
\begin{gather}
\begin{split}f(x \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left\{ -\frac{\tau}{2} (x-\mu)^2 \right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input data.

\item {}
\emph{mu} : Mean of the distribution.

\item {}
\emph{tau} : Precision of the distribution, which corresponds to $1/\sigma^2$ (tau \textgreater{} 0).

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X) = \mu$

\item {}
$Var(X) = 1/\tau$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{skew\_normal\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.skew_normal_like}{}\begin{funcdesc}{skew\_normal\_like}{x, mu, tau, alpha}
Azzalini's skew-normal log-likelihood
\begin{gather}
\begin{split}f(x \mid \mu, \tau, \alpha) = 2 \Phi((x-\mu)\sqrt{\tau}\alpha) \phi(x,\mu,\tau)\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where :math: Phi is the normal CDF and :math: phi is the normal PDF.
\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input data.

\item {}
\emph{mu} : Mean of the distribution.

\item {}
\emph{tau} : Precision of the distribution (\textgreater{} 0).

\item {}
\emph{alpha} : Shape parameter of the distribution.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}
See \href{http://azzalini.stat.unipd.it/SN/}{http://azzalini.stat.unipd.it/SN/}
\end{notice}
\end{funcdesc}
\index{t\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.t_like}{}\begin{funcdesc}{t\_like}{x, nu}
Student's T log-likelihood. Describes a zero-mean normal variable whose precision is
gamma distributed. Alternatively, describes the mean of several zero-mean normal
random variables divided by their sample standard deviation.
\begin{gather}
\begin{split}f(x \mid \nu) = \frac{\Gamma(\frac{\nu+1}{2})}{\Gamma(\frac{\nu}{2}) \sqrt{\nu\pi}} \left( 1 + \frac{x^2}{\nu} \right)^{-\frac{\nu+1}{2}}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input data.

\item {}
\emph{nu} : Degrees of freedom.

\end{itemize}

\end{description}
\end{funcdesc}
\index{truncnorm\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.truncnorm_like}{}\begin{funcdesc}{truncnorm\_like}{x, mu, tau, a, b}
Truncated normal log-likelihood.
\begin{gather}
\begin{split}f(x \mid \mu, \tau, a, b) = \frac{\phi(\frac{x-\mu}{\sigma})} {\Phi(\frac{b-\mu}{\sigma}) - \Phi(\frac{a-\mu}{\sigma})},\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where $\sigma^2=1/\tau$, \emph{phi} is the standard normal PDF and \emph{Phi} is the standard normal CDF.
\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input data.

\item {}
\emph{mu} : Mean of the distribution.

\item {}
\emph{tau} : Precision of the distribution, which corresponds to 1/sigma**2 (tau \textgreater{} 0).

\item {}
\emph{a} : Left bound of the distribution.

\item {}
\emph{b} : Right bound of the distribution.

\end{itemize}

\end{description}
\end{funcdesc}
\index{uniform\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.uniform_like}{}\begin{funcdesc}{uniform\_like}{x, lower, upper}
Uniform log-likelihood.
\begin{gather}
\begin{split}f(x \mid lower, upper) = \frac{1}{upper-lower}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : $lower \leq x \leq upper$

\item {}
\emph{lower} : Lower limit.

\item {}
\emph{upper} : Upper limit (upper \textgreater{} lower).

\end{itemize}

\end{description}
\end{funcdesc}
\index{von\_mises\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.von_mises_like}{}\begin{funcdesc}{von\_mises\_like}{x, mu, kappa}
von Mises log-likelihood.
\begin{gather}
\begin{split}f(x \mid \mu, k) = \frac{e^{k \cos(x - \mu)}}{2 \pi I_0(k)}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where \emph{I\_0} is the modified Bessel function of order 0.
\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : Input data.

\item {}
\emph{mu} : Mean of the distribution.

\item {}
\emph{kappa} : Dispersion of the distribution

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X) = \mu$

\end{itemize}
\end{notice}
\end{funcdesc}
\index{weibull\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.weibull_like}{}\begin{funcdesc}{weibull\_like}{x, alpha, beta}
Weibull log-likelihood
\begin{gather}
\begin{split}f(x \mid \alpha, \beta) = \frac{\alpha x^{\alpha - 1}
\exp(-(\frac{x}{\beta})^{\alpha})}{\beta^\alpha}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : $x \ge 0$

\item {}
\emph{alpha} : alpha \textgreater{} 0

\item {}
\emph{beta} : beta \textgreater{} 0

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(x)=\beta \Gamma(1+\frac{1}{\alpha})$

\item {}
$Var(x)=\beta^2 \Gamma(1+\frac{2}{\alpha} - \mu^2)$

\end{itemize}
\end{notice}
\end{funcdesc}

\section{Multivariate discrete distributions}
\index{multivariate\_hypergeometric\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.multivariate_hypergeometric_like}{}\begin{funcdesc}{multivariate\_hypergeometric\_like}{x, m}
The multivariate hypergeometric describes the probability of drawing x{[}i{]}
elements of the ith category, when the number of items in each category is
given by m.
\begin{gather}
\begin{split}\frac{\prod_i \binom{m_i}{x_i}}{\binom{N}{n}}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where $N = \sum_i m_i$ and $n = \sum_i x_i$.
\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : {[}int sequence{]} Number of draws from each category, (x \textless{} m).

\item {}
\emph{m} : {[}int sequence{]} Number of items in each categoy.

\end{itemize}

\end{description}
\end{funcdesc}
\index{multinomial\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.multinomial_like}{}\begin{funcdesc}{multinomial\_like}{x, n, p}
Multinomial log-likelihood. Generalization of the binomial
distribution, but instead of each trial resulting in ``success'' or
``failure'', each one results in exactly one of some fixed finite number k
of possible outcomes over n independent trials. `x{[}i{]}' indicates the number
of times outcome number i was observed over the n trials.
\begin{gather}
\begin{split}f(x \mid n, p) = \frac{n!}{\prod_{i=1}^k x_i!} \prod_{i=1}^k p_i^{x_i}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{description}
\item[x] \leavevmode{[}(ns, k) int{]}
Random variable indicating the number of time outcome i is
observed. $\sum_{i=1}^k x_i=n$, $x_i \ge 0$.

\item[n] \leavevmode{[}int{]}
Number of trials.

\item[p] \leavevmode{[}(k,) {]}
Probability of each one of the different outcomes.
$\sum_{i=1}^k p_i = 1)$, $p_i \ge 0$.

\end{description}

\end{description}

\begin{notice}{note}{Note:}\begin{itemize}
\item {}
$E(X_i)=n p_i$

\item {}
$Var(X_i)=n p_i(1-p_i)$

\item {}
$Cov(X_i,X_j) = -n p_i p_j$

\end{itemize}
\end{notice}
\end{funcdesc}

\section{Multivariate continuous distributions}
\index{dirichlet\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.dirichlet_like}{}\begin{funcdesc}{dirichlet\_like}{x, theta}
Dirichlet log-likelihood.

This is a multivariate continuous distribution.
\begin{gather}
\begin{split}f(\mathbf{x}) = \frac{\Gamma(\sum_{i=1}^k \theta_i)}{\prod \Gamma(\theta_i)}\prod_{i=1}^{k-1} x_i^{\theta_i - 1}\cdot\left(1-\sum_{i=1}^{k-1}x_i\right)^\theta_k\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{description}
\item[x] \leavevmode{[}(n, k-1) array {]}
Array of shape (n, k-1) where \emph{n} is the number of samples
and \emph{k} the dimension.
$0 < x_i < 1$,  $\sum_{i=1}^{k-1} x_i < 1$
\item[theta] \leavevmode{[}array{]}
An (n,k) or (1,k) array \textgreater{} 0.

\end{description}

\end{description}

\begin{notice}{note}{Note:}
Only the first \emph{k-1} elements of \emph{x} are expected. Can be used as a parent of Multinomial and Categorical
nevertheless.
\end{notice}
\end{funcdesc}
\index{inverse\_wishart\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.inverse_wishart_like}{}\begin{funcdesc}{inverse\_wishart\_like}{X, n, Tau}
Inverse Wishart log-likelihood. The inverse Wishart distribution is the conjugate
prior for the covariance matrix of a multivariate normal distribution.
\begin{gather}
\begin{split}f(X \mid n, T) = \frac{{\mid T \mid}^{n/2}{\mid X \mid}^{(n-k-1)/2} \exp\left\{ -\frac{1}{2} Tr(TX^{-1}) \right\}}{2^{nk/2} \Gamma_p(n/2)}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where $k$ is the rank of X.
\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{X} : Symmetric, positive definite matrix.

\item {}
\emph{n} : {[}int{]} Degrees of freedom (n \textgreater{} 0).

\item {}
\emph{Tau} : Symmetric and positive definite matrix.

\end{itemize}

\end{description}

\begin{notice}{note}{Note:}
Step method MatrixMetropolis will preserve the symmetry of Wishart variables.
\end{notice}
\end{funcdesc}
\index{mv\_normal\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.mv_normal_like}{}\begin{funcdesc}{mv\_normal\_like}{x, mu, tau}
Multivariate normal log-likelihood
\begin{gather}
\begin{split}f(x \mid \pi, T) = \frac{|T|^{1/2}}{(2\pi)^{1/2}} \exp\left\{ -\frac{1}{2} (x-\mu)^{\prime}T(x-\mu) \right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : (n,k)

\item {}
\emph{mu} : (k) Location parameter sequence.

\item {}
\emph{Tau} : (k,k) Positive definite precision matrix.

\end{itemize}

\end{description}

\strong{See Also:}

\hyperlink{pymc.distributions.mv_normal_chol_like}{\code{mv\_normal\_chol\_like()}}, \hyperlink{pymc.distributions.mv_normal_cov_like}{\code{mv\_normal\_cov\_like()}}

\end{funcdesc}
\index{mv\_normal\_chol\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.mv_normal_chol_like}{}\begin{funcdesc}{mv\_normal\_chol\_like}{x, mu, sig}
Multivariate normal log-likelihood.
\begin{gather}
\begin{split}f(x \mid \pi, \sigma) = \frac{1}{(2\pi)^{1/2}|\sigma|)} \exp\left\{ -\frac{1}{2} (x-\mu)^{\prime}(\sigma \sigma^{\prime})^{-1}(x-\mu) \right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : (n,k)

\item {}
\emph{mu} : (k) Location parameter.

\item {}
\emph{sigma} : (k,k) Lower triangular matrix.

\end{itemize}

\end{description}

\strong{See Also:}

\hyperlink{pymc.distributions.mv_normal_like}{\code{mv\_normal\_like()}}, \hyperlink{pymc.distributions.mv_normal_cov_like}{\code{mv\_normal\_cov\_like()}}

\end{funcdesc}
\index{mv\_normal\_cov\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.mv_normal_cov_like}{}\begin{funcdesc}{mv\_normal\_cov\_like}{x, mu, C}
Multivariate normal log-likelihood parameterized by a covariance
matrix.
\begin{gather}
\begin{split}f(x \mid \pi, C) = \frac{1}{(2\pi|C|)^{1/2}} \exp\left\{ -\frac{1}{2} (x-\mu)^{\prime}C^{-1}(x-\mu) \right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}\begin{description}
\item[Parameters] \leavevmode\begin{itemize}
\item {}
\emph{x} : (n,k)

\item {}
\emph{mu} : (k) Location parameter.

\item {}
\emph{C} : (k,k) Positive definite covariance matrix.

\end{itemize}

\end{description}

\strong{See Also:}

\hyperlink{pymc.distributions.mv_normal_like}{\code{mv\_normal\_like()}}, \hyperlink{pymc.distributions.mv_normal_chol_like}{\code{mv\_normal\_chol\_like()}}

\end{funcdesc}
\index{wishart\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.wishart_like}{}\begin{funcdesc}{wishart\_like}{X, n, Tau}
Wishart log-likelihood. The Wishart distribution is the probability
distribution of the maximum-likelihood estimator (MLE) of the precision
matrix of a multivariate normal distribution. If Tau=1, the distribution
is identical to the chi-square distribution with n degrees of freedom.

For an alternative parameterization based on $C=T{-1}$, see
\emph{wishart\_cov\_like}.
\begin{gather}
\begin{split}f(X \mid n, T) = {\mid T \mid}^{n/2}{\mid X \mid}^{(n-k-1)/2} \exp\left\{ -\frac{1}{2} Tr(TX) \right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where $k$ is the rank of X.
\begin{description}
\item[Parameters] \leavevmode\begin{description}
\item[X] \leavevmode{[}matrix{]}
Symmetric, positive definite.

\item[n] \leavevmode{[}int{]}
Degrees of freedom, \textgreater{} 0.

\item[Tau] \leavevmode{[}matrix{]}
Symmetric and positive definite

\end{description}

\end{description}

\begin{notice}{note}{Note:}
Step method MatrixMetropolis will preserve the symmetry of Wishart variables.
\end{notice}
\end{funcdesc}
\index{wishart\_cov\_like() (in module pymc.distributions)}

\hypertarget{pymc.distributions.wishart_cov_like}{}\begin{funcdesc}{wishart\_cov\_like}{X, n, C}
Wishart log-likelihood. The Wishart distribution is the probability
distribution of the maximum-likelihood estimator (MLE) of the covariance
matrix of a multivariate normal distribution. If C=1, the distribution
is identical to the chi-square distribution with n degrees of freedom.

For an alternative parameterization based on $T=C{-1}$, see
\emph{wishart\_like}.
\begin{gather}
\begin{split}f(X \mid n, C) = {\mid C^{-1} \mid}^{n/2}{\mid X \mid}^{(n-k-1)/2} \exp\left\{ -\frac{1}{2} Tr(C^{-1}X) \right\}\end{split}\notag\\\begin{split}\end{split}\notag
\end{gather}
where $k$ is the rank of X.
\begin{description}
\item[Parameters] \leavevmode\begin{description}
\item[X] \leavevmode{[}matrix{]}
Symmetric, positive definite.

\item[n] \leavevmode{[}int{]}
Degrees of freedom, \textgreater{} 0.

\item[C] \leavevmode{[}matrix{]}
Symmetric and positive definite

\end{description}

\end{description}
\end{funcdesc}
\normalsize


\section{Adversarial Training}

\transitionFrame{Adversarially Robust Training}

\begin{frame}{Solving the Outer Minimization}
  \onslide<+->{%
    \begin{equation}\label{eq:MinimaxTrainRepeat}
      \green{\min_{\params} \rho(\params)} \text{, where } \rho(\params) = \mathbb{E}_{(\X,\y) \sim \distr} \sbrack{\red{\max_{\delta \in \sPerturb} \loss (\X + \perturb, \y ; \params)}}
    \end{equation}
  }

  \begin{itemize}[<+->]
    \item So far, we have only talked about solving the \red{inner maximization} to create adversarial examples

    \vspace{10pt}
    \item \textbf{Question}: What do we need to solve to train adversarially robust networks?
    \vspace{3pt}
    \item \textbf{Answer}: Solve the \green{outer minimization}

    \vspace{10pt}
    \item \textbf{Question}: What algorithm can be used to solve the outer minimization?
    \vspace{3pt}
    \item \textbf{Answer}: Stochastic gradient descent (SGD) on the adversarial examples
      \begin{itemize}[<+->]
        \setlength{\itemsep}{4pt}
        \item \textit{Intuition}: SGD on adv.\ examples' loss reduces the \red{inner maximization}
        \item \textit{Takeaway}: Given an algorithm that transforms training examples into adv.\ examples (e.g.,~PGD), the rest of the \textbf{\blue{training process proceeds normally}}
      \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}{Why Should You Believe the Preceding Claims are True?}
  \onslide<+->{}
  \onslide<+->{\textbf{Answer}: You shouldn't.  The preceding explanation is definitely \red{not a proof}.}

  \vspace{20pt}
  \onslide<+->{\madry\ rely on \textbf{\blue{Danskin's Theorem}} that states gradients at inner maximizers (think adversarial examples) correspond to descent directions for the complete problem}

  \vspace{20pt}
  \onslide<+->{\textbf{Problem}: Multiple assumptions made by Danskin's Theorem's \red{do not apply} here, e.g.,~continuously differentiable function, only \red{\textit{approximate}} inner maximizers etc.}
  \begin{itemize}[<+->]
    \setlength{\itemsep}{6pt}
    \item Empirical results show that despite Danskin's not holding, \madry's approach is robust
    \item A more complete discussion of how Danksin's applies to this problem is in Appendix~A (see Arxiv version) and is beyond the scope of this talk
  \end{itemize}
\end{frame}


\chapter{Classification Trees}
\label{ch:classification-trees}

In the previous lesson, we used a classification tree,
\marginnote{Classification trees were hugely popular in the early years of machine learning, when they were first independently proposed by the engineer Ross Quinlan (C4.5) and a group of statisticians (CART), including the father of random forests Leo Brieman.}
one of the oldest, but still popular, machine learning methods. We like it since the method is easy to explain and gives rise to random forests, one of the most accurate machine learning techniques (more on this later). So, what kind of model is a classification tree?

Let us load \textit{iris} data set, build a tree (widget \widget{Tree}) and visualize it in a \widget{Tree Viewer}.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{workflow-tree-viewer.png}
\end{figure}

\begin{figure*}[h]
    \vspace{-0.4cm}
    \includegraphics[scale=0.35]{iris-data.png}
    \label{fig:classification-predictions}
\end{figure*}

\begin{wrapfigure}{o}{1.0\textwidth}
    \includegraphics[scale=0.35]{tree-viewer.png}
    \label{fig:classification-predictions}
\end{wrapfigure}

We read the tree from top to bottom. Looks like the column \textit{petal length} best separates the iris variety \textit{setosa} from the others, and in the next step, \textit{petal width} then almost perfectly separates the remaining two varieties.

Trees place the most useful feature at the root. What would be the most useful feature? The feature that splits the data into two purest possible subsets. It then splits both subsets further, again by their most useful features, and keeps doing so until it reaches subsets in which all data belongs to the same class (leaf nodes in strong blue or red) or until it runs out of data instances to split or out of useful features (the two leaf nodes in white).

We still have not been very explicit about what we mean by "the most useful" feature. There are many ways to measure the quality of features, based on how well they distinguish between classes. We will illustrate the general idea with information gain. We can compute this measure in Orange using the \widget{Rank} widget\marginnote{The \widget{Rank} widget can be used on its own to show the best predicting features. Say, to figure out which genes are best predictors of the phenotype in some gene expression data set.}, which estimates the quality of data features and ranks them according to how informative they are about the class. We can either estimate the information gain from the whole data set, or compute it on data corresponding to an internal node of the classification tree in the \widget{Tree Viewer}. In the following example we use the \textit{Sailing} data set.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{workflow-rank.png}
    \caption{The \widget{Datasets} widget is set to load the \textit{Sailing} data set. To use the second \widget{Rank}, select a node in the \widget{Tree Viewer}.}
\end{figure}

Besides the information gain, \widget{Rank} displays several other measures (including Gain Ratio and Gini), which are often quite in agreement and were invented to better handle discrete features with many different values.

\begin{figure}[h]
    \centering
    \vspace{-0.2cm}
    \includegraphics[scale=0.4]{rank.png}
    \caption{For the whole \textit{Sailing} data set, \textit{Company} is the most class-informative feature according to all measures shown.}
\end{figure}

\newpage

Here is an interesting combination of a \widget{Tree Viewer} and a \widget{Scatter Plot}. This time, use the \textit{Iris} data set. In the \widget{Scatter Plot}, we first find the best visualization of this data set, that is, the one that best separates the instances from different classes. Then we connect the \widget{Tree Viewer} to the \widget{Scatter Plot}. Data instances (particular irises) from the selected node in the \widget{Tree Viewer} are shown in the \widget{Scatter Plot}.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{workflow-inspection.png}
    \caption{Careful, the \widget{Data} widget needs to be connected to the \widget{Scatter Plot}'s \textit{Data} input, and \widget{Tree Viewer} to the \widget{Scatter Plot}'s \textit{Data Subset} input.}
\end{figure}

Just for fun, we have included a few other widgets in this workflow. In a way, a \widget{Tree Viewer} behaves like \widget{Select Rows}, except that the rules used to filter the data are inferred from the data itself and optimized to obtain purer data subsets.

\begin{figure*}[h]
  \infinitewidthbox{\includegraphics[scale=0.35]{tree-viewer-selection.png} \includegraphics[scale=0.35]{scatter-plot-subset.png}}
  \caption{In the \widget{Tree Viewer} we selected the rightmost node. All data instances coming to the selected node are highlighted in \widget{Scatter Plot}.}
\end{figure*}

Wherever possible, visualizations in Orange are designed to support selection and passing of the data that applies to it. Finding interesting data subsets and analyzing their commonalities is a central part of explorative data analysis, a data analysis approach favored by the data visualization guru Edward Tufte.


\section{The \copyi algorithm}
\Label{sec:copyi}

The \copyi  algorithm in the \cxx Standard Library \cite[\S 28.6.1]{cxx-17-draft} implements
a duplication algorithm for general sequences.
For our purposes we have modified
the generic implementation
to that of a range of type \valuetype.
The signature now reads:

\begin{lstlisting}[style=acsl-block]

  void copy(const value_type* a, size_type n, value_type* b);
\end{lstlisting}

Informally, the function copies every element from the source range \inl{a[0..n-1]} to the
destination range~\inl{b[0..n-1]}, as shown in Figure~\ref{fig:copy}.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.50\textwidth]{Figures/copy.pdf}
\caption{\Label{fig:copy} Effects of \copyi}
\end{figure}

\subsection{Formal specification of \copyi}

Figure~\ref{fig:copy} might suggest that the ranges \inl{a[0..n-1]} and \inl{b[0..n-1]}
must not overlap.
However, since the informal specification requires that elements are copied in the
order of increasing indices only a weaker condition is necessary.
To be more specific, it is required that the pointer~\inl{b} does not refer
to elements of \inl{a[0..n-1]} as shown in the example in Figure~\ref{fig:copy-overlap}.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.60\textwidth]{Figures/copy-overlap.pdf}
\caption{\Label{fig:copy-overlap} Possible overlap of \copyi ranges}
\end{figure}

\FloatBarrier

The specification of \copyi is shown in the following listing.
The \copyi algorithm expects that the ranges \inl{a} and \inl{b} are valid for reading
and writing, respectively.
Note the precondition~\inl{sep} that expresses the previously discussed non-overlapping property.

\input{Listings/copy.h.tex}

Again, we can use the \logicref{Equal} predicate to express that the
array~\inl{a} equals~\inl{b} after \copyi has been called.
Nothing else must be altered.
To state this we use the \inl{assigns}-clause.

%\clearpage

\subsection{Implementation of \copyi}

The following listing shows an implementation of the \copyi function.

\input{Listings/copy.c.tex}

For the postcondition \equal to be true, we must ensure that for every index
\inl{i}, the value \inl{a[i]} must not yet have been changed before it is
copied to \inl{b[i]}.
We express this by using the  \Unchanged predicate.\footnote{
Alternatively, this could also be expressed by changing the
\inl{loop assigns} clause to \inl{i, b[0..i-1]}; however,
\framac doesn't yet support \inl{loop assigns} clauses
containing the loop variable.
}

The \inl{assigns} clause ensures that nothing but the range \inl{b[0..n-1]}
and the loop variable \inl{i} is modified.
Keep in mind, however, that parts of the source range \inl{a[0..n-1]} might change
due to its potential overlap with the destination range.

\clearpage


\documentclass[10.5pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{setspace}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{listings}
\usepackage{wrapfig}
\usepackage{hyperref}
\usepackage[left=1cm, right=1cm, top=1.5cm, bottom=1.5cm]{geometry}
\usepackage[font=small]{caption, subcaption}

%\setlength{\parskip}{0em}
\setstretch{0.3}

\algdef{SE}[SUBALG]{Indent}{EndIndent}{}{\algorithmicend\ }%
\algtext*{Indent}
\algtext*{EndIndent}

\newcommand{\algorithmautorefname}{Algorithm}

\begin{document}
    \begin{center}
        \large \textbf{Problem, Search and beyond}

        \normalsize Chuanyuna Liu 884140, Zhuoqun Huang 908525
    \end{center}
    \section{Problem}
    \subsection{General Problem}
    We formulate our problem with the following five concepts:
    \begin{itemize}
        \itemsep0em
        \item \textbf{State($\mathbb{S}$)} including a \textbf{initial state ($s_0$)} the agent begins with
        \item \textbf{Actions($\mathbb{A}$)} available to agent at each State
        \item \textbf{Transition function ($F:\mathbb{S}\times \mathbb{A}\rightarrow \mathbb{S}$)} that takes a (state, action) pair and return a new state
        \item \textbf{Goal Test($GT:S\rightarrow \{True, False\}$)} that takes a state $s$ and return true if $s\in S_{goal}$, the \textbf{Goal State} of the problem.
        \item \textbf{Path Cost ($C:\mathbb{S}\times \mathbb{S}\rightarrow \mathbb{R}$)} That takes two states and return cost moving from one to another
    \end{itemize}
    \vspace{-15pt}
    \subsection{Single-player Chexers}
    In this section, we formally describe how the above framework fits the Chexers game.
    \begin{itemize}
        \itemsep0em
        \item Denote set of pieces with $\mathbb{P}=\{(r, q, t)\}$. \\
        where
        \begin{align*}
            \text{$r, q, -(r+q)\in [-3, 3]$ denotes the location on board.}\\
            \text{$t\in \{red, blue, green, block\}$ stands for type of piece.}
        \end{align*}

        \item $\mathbb{S} := \{p_i\in \mathbb{P}, t_b|i\le n\}$ where $n$ is number of pieces on board, where $t_b$ is the searched type.*
        \item $\mathbb{A} := (Move,p_i), (Jump, p_i), (Exit, p_i), \forall\ p_i $ where$\ t(p_i)=t_b.\ | \mathbb{A}_s| \le 6n_p\ \forall s$.
        \item $f(s, a) := s'$. where $s'$ differs exactly by one piece $r(p_{i,s}), q(p_{i,s}) \ne r(p_{i,s'}), q(p_{i,s'})$ or $p_i\notin s'$
        \item $c(s, s') := 1, \forall s, s'\ \text{if}\ \exists a\ \text{such that} f(s, a) = s'$
        \item $gt(s) = True$ if $\nexists p_i$ such that $t(p_i) = t_b$
    \end{itemize}
    \vspace{-5pt}
    *Initial State $s_0$ given by problem specification.
    \vspace{-5pt}
    \section{Search}
        \subsection{preliminary}
        We use the following \autoref{a_star}, to search for our goal. Based on our problem specification, we have all required components except for \textbf{H($node$)}, so we will propose one type of \textbf{H} we found to be most effective after comparing it against \textbf{Null $H=0$} and \textbf{depth first search}\footnote{In the following analysis, $b$ for \textbf{branching factor}, $d$ for \textbf{depth to optimal solution}}.
        \begin{algorithm}[ht]
            \footnotesize
            \caption{General A* algorithm}\label{a_star}
            \begin{algorithmic}[1]
                \Statex PRIORITY-QUEUE \Comment{\textbf{Min} Priority Queue (min key)}
                \Statex  \hskip2.0em ADD($q$, $key$, $value$), \Comment{Add (key, value) to q. If value exists, update the key}
                \Statex  \hskip2.0em POP($q$) \Comment{Pop the value with \textbf{least} key}
                \Statex  \hskip2.0em GET($q$, $value$) \Comment{Get the key associated with a value}
                \Statex \Comment{All above Queue operations can operate in $\Theta(1)$}
                \Statex
                \Statex NODE \Comment{stores associated \textbf{state} and its \textbf{parent}}
                \Require EXPAND($node$)\Comment{expand a \textbf{node} to get it's \textbf{children}}
                \Require G($node$)\Comment{get \textbf{total cost} arriving this node}
                \Require H($node$) \Comment{Computes an \textbf{admissible estimation} of cost to goal state}
                \Require C($node1$, $node2$) \Comment{Give Path cost arriving node 2 from node 1}
                \Procedure{A*}{$problem$, $initial$}
                \State $openSet \gets$ PRIORITY-QUEUE(H($initial$),  $initial$)
                \State $closedSet \gets$ \{\}
                \While{$openSet$ \textbf{is not} empty} \Comment{$O(b^d)$ repetitions}
                    \State $node \gets POP(openSet)$
                    \State $ADD(closedSet, node)$
                    \If {GOAL-TEST($node$) \textbf{is} True}
                        \State \textbf{return} $node$
                    \EndIf
                    \For {$child$ \textbf{in} EXPAND($node$)} \Comment{$O(b)$ repetitions*}
                        \State $cost$ = G($node$) + C($node$, $child$) + H($child$)
                        \If {$child$ \textbf{in} $closedSet$}
                            continue
                        \ElsIf {$child$ not in $openSet$ \textbf{or} cost $<$ GET($openSet$, $child$)}
                            \State ADD(openSet, cost, child)
                        \EndIf
                    \EndFor
                \EndWhile
                \State \textbf{return} no solution
                \EndProcedure\\
                All operations $O(1)$ unless explicitly stated, we denote the complexity of search for problem also here.\\
                *\textbf{Note} at $d$ level, we don't run this loop, making overall factor only $O(b^d)$.
            \end{algorithmic}
        \end{algorithm}
    \vspace{-10pt}
    \subsection{Heuristic}
        \paragraph{As a Problem} We propose the following problem definition for mapping a good heuristic value for each pieces and we define $H(s) = \sum_{p_i}h(p_i), \forall p_i$ that $t(p_i) = t_b$ after relaxing the problem to be: a piece can choose freely between \textbf{move}, \textbf{jump} disregarding normal constrains and \textbf{cannot} move on to blocks.
        \begin{itemize}
            \itemsep0em
            \item $\mathbb{S} := \{(cost_i, position_i)|position_i\in board\}$*,
            \item $\mathbb{A} :$ for $pos_i$ with $cost_i=min({cost_n})$, $\forall$ $pos_{j\ne i} $ reachable from $pos_i$ and $cost_j > cost_i+1$. Update $(pos_j, cost_i + 1)$. If $\nexists pos_j$, remove $(pos_i, cost_i)$ from $s$ and put $h(pos_i) = cost_i$
            \item $c(s, s')=0$ and $f(s,a)$ follows definition in $\mathbb{A}$
            \item $gt(s')=True$ if $s' = \emptyset$
        \end{itemize}
        \vspace{-5pt}
        *$s_0 = \{(1, pos_i)|can\_exit(pos_i) = True\} | \{(\infty, pos_i)|can\_exit(pos_i) = False\}$
        \vspace{-10pt}
        \paragraph{Heuristic problem solution} Finding heuristic is straight forward with the given above problem definition. Supporting all the given operations to \ref{a_star}, and you should end up with a complete heuristic map in constant time (due to small and constant search space $O(\mathbb{V}\times \mathbb{E}) = O(37\times 12)=O(1)$).
        \vspace{-10pt}
        \paragraph{Admissible?} The algorithm is guaranteed to provide us with a admissible cost estimation for each game state:\\
    - The relaxed rule let pieces always able to jump.\\
    - This will in all cases reduce the number of action taken, by always increasing the distance a piece can move.
    \vspace{-10pt}
    \subsection{Property of search}
        \paragraph{Efficiency} The efficiency of A* algorithm heavily depends on how accurate the heuristic is. As we can see from the example below. Our heuristic (shown in \hyperref[fig:heurstics]{this figure}) is very close to the real cost.
        \begin{wrapfigure}{r}{0.3\textwidth}
            \vspace{-32pt}
                \begin{center}
                    \includegraphics[width=0.3\textwidth]{heuristic.png}
                    \label{fig:heurstics}
                \end{center}
            \vspace{-10pt}
            \caption{Map formed by heuristic}
            \vspace{-100pt}
        \end{wrapfigure}
        \vspace{-40pt}
        \paragraph{Optimality} A* algorithm can only find the optimal solution if the heuristic is admissible. We relaxed the rule to allow pieces jumping freely without the need to leapfrog another piece. Because jump move allows pieces to move twice the normal move, our heuristic at most underestimates the real cost by a factor of $2$, and cannot be faster the real cost. This satisfied admissibility. Monotonicity is also met because each move adds $1$ unit of cost. Hence our program is optimal.
        \vspace{-10pt}
        \paragraph{Completeness} A* algorithm is complete if the graph contains finite nodes. For our problem, both the board and number of pieces are finite leading to a bounded size for state space, concludes the completeness of our algorithm.
    \section{Beyond}

        The complexity of problem is \textbf{exponentially} related to \textbf{number} of \textbf{moving pieces}. This is due to the effect of more moving options leading to a higher branching factor.\\
        The \textbf{further} pieces are from the \textbf{goal}, deeper the program has to search.\\
        A* algorithm stores each layer of nodes in a priority queue and only expend the $node$ $n$ with the least $g(n)+h(n)$. In the \textbf{worst case}, it expands all the nodes fully, store and sort them in order of cost, giving $O(b^d)$ complexity for both \textbf{space and time}. In the best case, our heuristic matches the real cost, and A only expand nodes with the correct path. This gives us $O(bd)$. The \textbf{high memory complexity} of A* ($O(b^d)$) can be avoided by using \textbf{iterative deepening A search}. This would reduce A* algorithm's \textbf{space complexity} to $O(bd)$ (by trading off some time complexity).\\
        The amount of free space on the board also affects the complexity of algorithm. From \ref{fig:moves} we can see that as the board becomes more empty, we have more actions to consider. Despite the low cost solution of \ref{fig:more_move}, its free space is drastically larger, leading to a significantly higher number of expanded nodes than that of \ref{fig:less_move}.
        \begin{wrapfigure}[b]{r}{0.7\textwidth}
            \begin{center}
                \begin{subfigure}[b]{0.25\textwidth}
                    \includegraphics[width=\textwidth]{SpaceComplexity1.png}
                    \caption{More options}
                    \label{fig:more_move}
                \end{subfigure}
                \begin{subfigure}[b]{0.25\textwidth}
                    \includegraphics[width=\textwidth]{SpaceComplexity2.png}
                    \caption{Less Options}
                    \label{fig:less_move}
                \end{subfigure}
            \end{center}
            \vspace{-10pt}
            \caption{Search space illustration}\label{fig:moves}
        \end{wrapfigure}
\end{document}


\documentclass[11pt]{article}
\setlength{\textwidth}{7in} \setlength{\textheight}{9.8in}
\setlength{\topmargin}{-0.8in} \setlength{\oddsidemargin}{-0.25in}
\setlength{\evensidemargin}{-0.25in}

\usepackage{graphicx}
\usepackage{epstopdf}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}
\usepackage{color}
\usepackage{hyperref}
\usepackage{amssymb}
\usepackage{amsmath}
%\usepackage{algpseudocode}

\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\rhead{Homework2  Page \thepage}
\lhead{Shen Qu}
\chead{STAT 671}

\DeclareMathOperator{\trace}{trace}

\begin{document}
%\includegraphics{../../../psulogo_horiz_bw.eps}\hfill\includegraphics{../../../deptlogo}

\section{Kernels}

\begin{enumerate}
\item let $(x,y) \in \mathbb{R}^+ \times \mathbb{R}^+$, where $\mathbb{R}^+=\{x \in \mathbb{R};x \geq 0\}$, the ``french positive\rq\rq{} real numbers.

\begin{enumerate}
\item  Verify that $\min(x,y) = \int_0^\infty \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt$
where  $\mathbb{I}_A =  \left\{
\begin{tabular}{ll}
1 & \mbox{ if A is true}\\
0 & \mbox{otherwise}
\end{tabular}\right.$

When $x\le y$,
\begin{align*}
\int_0^\infty \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt&=\int_0^x \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt+\int_x^y \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt+\int_y^\infty \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt\\
&=\int_0^x 1\cdot 1 dt+\int_x^y 0\cdot 1 dt+\int_y^\infty 0\cdot 0 dt=x\\
\end{align*}

By the same way, when $y\le x$, $\int_0^\infty \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt=y$.

Therefore, $\min(x,y) = \int_0^\infty \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt$
\vspace{2mm}
\item Use the previous question to show that $K(x,y)=\min(x,y)$ is a pd kernel over $\mathbb{R}^+$

\vspace{2mm}
$K(x,y)=\min(x,y)=\int_0^\infty \mathbb{I}_{t\leq x} \mathbb{I}_{t\leq y} dt=\min(y,x)=K(y,x)$ symmetric

$$\sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_j\min(x,y)=\int_0^\infty \sum_{i=1}^n\alpha_i\mathbb{I}_{t\leq x} \sum_{j=1}^n\alpha_j\mathbb{I}_{t\leq y} dt=\int_0^\infty (\sum_{i=1}^n\alpha_i\mathbb{I}_{t\leq x})^2dt\ge0$$

\item Show that $\max(x,y)$ is not a pd kernel over  $\mathbb{R}^+$.
\end{enumerate}

% When $x\le y$,  $\int_0^\infty \mathbb{I}_{t\geq x} \mathbb{I}_{t\geq y} dt=\int_y^\infty 1\cdot 1 dt=\left.t\right|_y^\infty\neq\max(x,y)$
% The Gram Matrix
% $$M(x,y)=\begin{bmatrix}\alpha_1\alpha_1\max(x, x) & \alpha_1\alpha_2\max(x, y)\\\alpha_2\alpha_1\max(x,y) & \alpha_2\alpha_2\max(y,y)\end{bmatrix} =
%       \begin{bmatrix}x &y \\y & y\end{bmatrix}=xy-y^2\le 0$$
% When $y\le x$, it is the same. $M(x,y)=xy-x^2\le 0$. For example,

Let $x_1=1,x_2=2,\alpha_1=1,\alpha_2=-1$

$\sum_{i=1}^2\sum_{j=1}^2\alpha_1\alpha_2 k(x_1,x_2)=\alpha_1\alpha_1 \max(1,1)+\alpha_1\alpha_2 \max(1,2)+\alpha_2\alpha_1 \max(2,1)+\alpha_2\alpha_2 \max(2,2)=1-2-2+2=-1<0$

Therefore, $\max(x,y)$ is not a p.d. kernel over  $\mathbb{R}^+$

\item Consider a probability space $(\Omega,\mathcal{A},P)$

\begin{enumerate}
\item Define for any two events $A$ and $B$, $K_1(A,B)=P(A \cap B)$
where $A \cap B$ is the intersection between the events A and B
Verify that $K_1$ is positive definite. Hint: $P(A)=E[\mathbb{I}_A]$

\vspace{2mm}
$K_1(A,B)=P(A \cap B)=P(B \cap A)=K_1(B,A)$ symmetric

$P(A)=E[\mathbb{I}_A]$; $P(B)=E[\mathbb{I}_B]$; $P(A\cap B)=E[\mathbb{I}_A\mathbb{I}_B]$

$$\sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jE[\mathbb{I}_{A_i}\mathbb{I}_{A_j}]=E[\sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_j\mathbb{I}_{A_i}\mathbb{I}_{A_j}]=E[(\sum_{i=1}^n\alpha_i\mathbb{I}_{A_i})^2]\ge0$$

\item Define for any two events $A$ and $B$,
$K_2(A,B)=P(A \cap B)-P(A)P(B)$
Verify that $K_2$ is positive definite.
\end{enumerate}

\vspace{2mm}
$K_2(A,B)=P(A \cap B)-P(A)P(B)=E[\mathbb{I}_A\mathbb{I}_B]-E[\mathbb{I}_A]E[\mathbb{I}_B]=Cov[\mathbb{I}_A,\mathbb{I}_B]$

$$\sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jCov[\mathbb{I}_{A_i},\mathbb{I}_{A_j}]=Cov[\sum_{i=1}^n\alpha_i\mathbb{I}_{A_i},\sum_{j=1}^n\alpha_j\mathbb{I}_{A_j}]=Var[\sum_{i=1}^n\alpha_i\mathbb{I}_{A_i}]\ge 0$$

\end{enumerate}

\section{Kernels and RKHS}
\begin{enumerate}
\item Define the RKHS  over $\mathbb{R}^d$ $K(x,y)=x^Ty+c$ where $c>0$.

\begin{enumerate}
\item What is the RKHS associated with the kernel $K$? no proof is required.

$$\mathcal{H} = \{f:\ \mathbb{R}^d\mapsto\mathbb{R};\ f_{w,w_0}(x)=w^Tx+w_0;\quad w\in\mathbb{R}^d,w_0\in\mathbb{R}\}$$

\item What is the inner product in this RKHS? no proof required.

$$\langle f_{v,v_0},f_{w,w_0}\rangle_{\mathcal{H}}=v^Tw+\frac1cv_0w_0\Rightarrow\langle f_{v,v_0},f_{v,v_0}\rangle=\|f_{v,v_0}\|^2_{\mathcal{H}}=\|v\|^2+\frac{v_0^2}c$$

\item Verify the reproducing property

$\mathcal{H}$ contains all the functions $k(\cdot,x): t\mapsto k(t,x)=t^Tx+c=f_t(x)$

$$\langle f_{w,w_0},k(\cdot,x)\rangle=\langle f_{w,w_0},f_{x,c}\rangle=x^Tw+\frac1ccw_0=w^Tx+w_0=f_w(x)$$

$\therefore\langle f,k(\cdot,x)\rangle_{\mathcal{H}}=f(x)$ for each $f\in\mathcal{H}$, $x\in\mathcal{X}$

\end{enumerate}

\item Define the RKHS  over $\mathbb{R}^d$
$K(x,y)=(x^Ty)^2$
The RKHS associated with the kernel $K$ is $\{f_S;f_S(x)=x^T S x\}$ where $S$ is a symmetric $(d,d)$ matrix. The inner product is
$<f_{S_1},f_{S_2}>=<S_1,S_2>_F$

\begin{enumerate}
\item Verify the reproducing property.

$\mathcal{H}$ contains all the functions
$k(\cdot,x): t\mapsto k(t,x)=(t^Tx)(t^Tx)=x^T\cdot (tt^T)\cdot x=f_t(x)$

$$ \langle f_{S},k(\cdot,x)\rangle_{\mathcal{H}}=\langle f_{S},f_{xx^T}\rangle_{\mathcal{H}}=\langle S,xx^T\rangle_{\mathcal{F}}=\trace[Sxx^T]=\trace[x^TSx]=x^TS x= f_{S}(x)$$

$\therefore\langle f_{S},k(\cdot,x)\rangle_{\mathcal{H}}=f_{S}(x)$ for each $f\in\mathcal{H}$, $x\in\mathcal{X}$

\item Why do we require that $S$ is symmetric?

$k(x,y)=(x^Ty)^2$ is a p.d. kernel. $\langle f_{S},k(\cdot,x)\rangle_{\mathcal{H}}= f_{S}(x)=x^TS x$ is a quadratic form over $\mathbb{R}^d$, where $x$ is the column vector and $S$ must be a symmetric $n\times n$ matrix by the definition of quadratic form.

% The matrices $S$ and $x^TSx$ are congruent where $x$ is an invertible matrix over the same field.
% Matrix congruence is an equivalence relation. Any matrix congruent to a symmetric matrix is again symmetric. Therefore, $S$ must be symmetric.

%Only if $\underset{(d,d)}{S}$ is a symmetric Matrix, $t^Tx=x^Tt$.

%$$[S_1]_{ij}[S_2]_{ij}=\trace[(x_i^Tx_j)(y_j^Ty_i)]=\trace[(y_ix_i^T)(x_jy_j^T)]=\langle x_iy_i^T,x_jy_j^T\rangle_{\mathcal{F}}=\langle z_i,z_j\rangle_{\mathbb{R}^{n^2}}$$
%If not, we can not complete the step of $(t^Tx)(t^Tx)=x^T\cdot (tt^T)\cdot x$ in (b).

\end{enumerate}

\item Define the RKHS  over $\mathbb{R}^d$ $K(x,y)=(x^Ty+c)^2$ where $c>0$.

\begin{enumerate}
\item What is the RKHS associated with the kernel $K$? no proof is required.

$$\{f_{S,s,s_0}: f_{S,s,s_{0}}(x)=x^T S x+2s_0 s^Tx+s_0^2;\quad S\in\mathbb{R}^{d\times d},s\in\mathbb{R}^{d},s_0\in\mathbb{R}\}$$

\item What is the inner product in this RKHS? no proof required.

$$\langle f_{S_1,s_1,s_{10}},f_{S_2,s_2,s_{20}}\rangle_{\mathcal{H}}=\langle S_1,S_2\rangle_{\mathcal{F}}+\frac{2s_{10}s_{20}}cs_1^Ts_2+(\frac{s_{10}s_{20}}c)^2$$

\item Verify the reproducing property

$\mathcal{H}$ contains all the functions
$k(\cdot,x_i): t\mapsto k(t,x)=(t^Tx+c)^2=x^T\cdot (tt^T)\cdot x+2ct^Tx+c^2=f_t(x)$

$$\langle f_{S,s,s_{0}},k(\cdot,x)\rangle_{\mathcal{H}}=\langle f_{S,s,s_{0}},f_{xx^T,x,c}\rangle_{\mathcal{H}}=\langle S,xx^T\rangle_{\mathcal{F}}+\frac{2s_{0}c}cs^Tx+(\frac{s_{0}c}c)^2$$
$$=x^T Sx+2s_{0}s^Tx+s_0^2= f_{S,s,s_{0}}(x)$$

$\therefore\langle f_{S,s,s_{0}},k(\cdot,x)\rangle_{\mathcal{H}}=f_{S,s,s_{0}}(x)$ for each $f\in\mathcal{H}$, $x\in\mathcal{X}$

\end{enumerate}

\end{enumerate}

\section{Fisher kernel}
Let $\theta \in \mathbb{R}$ be a parameter and let $p_\theta$ be a probabilistic model (i.e a point mass function or a density) over a set $\mathcal{X}$ indexed by $\theta$. Let $\theta_0 \in \mathbb{R}$ be a specific value for $\theta$.
Let us define the Fisher score at $x \in \mathcal{X}$ as
$\phi(x,\theta_0) = \frac{\delta}{\delta \theta} \ln p_\theta(x) \mbox{ evaluated at } \theta=\theta_0$
assuming that this quantity exists.
Define $I(\theta)$, the Fisher information associated with the parameter $\theta$, i.e.,
$I(\theta)=E[\phi^2(X,\theta)]$
where $E$ stands for expectation and $X$ is a random variable with distribution $p_\theta$.
The Fisher kernel is then
$k(x,x')=\frac{\phi(x,\theta_0)\phi(x',\theta_0)}{I(\theta_0)}$
where
\begin{enumerate}
\item Verify that $k(.,.)$ is a positive definite kernel over $\mathcal{X}$

$k(x,x')=\frac{\phi(x,\theta_0)\phi(x',\theta_0)}{I(\theta_0)}=\frac{\phi(x',\theta_0)\phi(x,\theta_0)}{I(\theta_0)}=k(x',x)$ symmetric.

For $I(\theta)=E[\phi^2(X,\theta)]\ge0$,

$k(x,x')=\frac{1}{I(\theta_0)}\sum_{i=1}^n\alpha_i\phi(x_i,\theta_0)\sum_{j=1}^n\alpha_j\phi(x_j,\theta_0)=\frac{1}{I(\theta_0)}[\sum_{i=1}^n\alpha_i\phi(x_i,\theta_0)]^2\ge0$

$\therefore k(.,.)$ is a positive definite kernel over $\mathcal{X}$

\item Consider the following model: $x \in \{0,1\}$, $X \sim Bernoulli(\theta)$, $0 < \theta < 1$, that is
$p_\theta(x)=\theta^x(1-\theta)^{(1-x)}$
We recall that in this case $E[X]=\theta$ and $Var[X]=E[(X-\theta)^2]=\theta(1-\theta)$
Compute $k(x,x')$

\begin{align*}
p_\theta(x)&=\theta^x(1-\theta)^{(1-x)} \\
  \ln p_\theta(x)&=x\ln\theta+(1-x)\ln(1-\theta)\\
  \frac{d}{d \theta} \ln p_\theta(x)&=\frac{x}{\theta}+\frac{1-x}{1-\theta}=\frac{x-\theta}{\theta(1-\theta)}
\end{align*}

$$I(\theta)=E[\phi^2(X,\theta)]=E[(\frac{X-\theta}{\theta(1-\theta)})^2]
=\frac{E[(X-\theta)^2]}{\theta^2(1-\theta)^2}=\frac{V[X]}{\theta^2(1-\theta)^2}
=\frac{\theta(1-\theta)}{\theta^2(1-\theta)^2}=\frac{1}{\theta(1-\theta)}$$

$$k(x,x')=\frac{\phi(x,\theta_0)\phi(x',\theta_0)}{I(\theta_0)}=\frac{(x-\theta_0)(x'-\theta_0)}{\theta_0^2(1-\theta_0)^2}\theta_0(1-\theta_0)=\frac{(x-\theta_0)(x'-\theta_0)}{\theta_0(1-\theta_0)} \qquad\hfill\square$$

\item Assume now $x=(x_1,x_2)$ with $x_1 \in \{0,1\}$ and $x_2 \in \{0,1\}$.
We consider the following model where $X=(X_1,X_2)$, $X_1$ and $X_2$ are independent with the same $Bernoulli(\theta)$ distribution.
Compute $k(x,x')$.

\begin{align*}
  p_\theta(\vec x)\underset{x_1\perp x_2}{=}&p_\theta(x_1)p_\theta(x_2)=\theta^{x_1+x_2}(1-\theta)^{2-x_1-x_2}\\
  \ln p_\theta(x)=&(x_1+x_2)\ln\theta+(2-x_1-x_2)\ln(1-\theta)\\
  \phi(\vec x,\theta)=& \frac{d}{d \theta} \ln p_\theta(x)=\frac{x_1+x_2}{\theta}+\frac{2-x_1-x_2}{1-\theta}=\frac{x_1+x_2-2\theta}{\theta(1-\theta)}
\end{align*}

\begin{align*}
I(\theta)=&E[\phi^2(\vec X,\theta)]=\frac{E[(X_1+X_2-2\theta)^2]}{\theta^2(1-\theta)^2}\\
\underset{x_1\perp x_2}{=}&\frac{E[(X_1-\theta)^2]+E[(X_2-\theta)^2]+2(E[X_1]-\theta)(E[X_2]-\theta)}{\theta^2(1-\theta)^2}\\
=&\frac{V[X_1]+V[X_2]-0}{\theta^2(1-\theta)^2}=\frac{2\theta(1-\theta)}{\theta^2(1-\theta)^2}=\frac{2}{\theta(1-\theta)}
\end{align*}

\begin{align*}
k(x,x')&=\frac{\phi(\vec x,\theta_0)\phi(\vec x',\theta_0)}{I(\theta_0)}=\frac{(x_1+x_2-2\theta_0)(x'_1+x'_2-2\theta_0)}{\theta_0^2(1-\theta_0)^2}\frac{\theta_0(1-\theta_0)}2\\
&=\frac{(x_1+x_2-2\theta_0)(x'_1+x'_2-2\theta_0)}{2\theta_0(1-\theta_0)}&& \square
\end{align*}

\end{enumerate}

\end{document}


\chapter{Date and time}
We often need to generate a date string in various formats. A common challenge encountered is getting that format just right.
Unless you have reasons to the contrary, using a format of year, month and date, all numeric, results in a sortable date which
makes it easier to find specific files in large collections.

\paragraph{Getting a date}

There are different mechanisms for acquiring a date. A popular approach is to use \verb+GETDATE()+ \href{http://msdn.microsoft.com/en-us/library/ms139875.aspx}{GETDATE(SSIS Expression)}

It's built into the SSIS Expression language but I find that the System scoped variable, \verb+StartTime+ provides a better reference point as it is evaulated only when the package begins. This is in contrast to GETDATE which is evaluated each time it is inspected. For a long running package, the drift in values may not be acceptable. It might also be \begin{em}exactly\end{em} what is desired. Assume we have a Derived Column transformation in a package which adds a column named InsertDate into our buffer.

Consider the following two queries, which would more correctly describe the data loaded from today's data load?

Quote
\begin{quote}
SELECT
    T.*

FROM
    MyTable AS T

WHERE
    T.InsertDate = '2015-01-17T12:24:48.123';
\end{quote}

VerbatimInput method

\VerbatimInput{Expressions/InsertDate.Scan.sql}

verbatiminput method

\verbatiminput{Expressions/InsertDate.Scan.sql}

\section{YYYY}
This expression defines how to get a 4 digit year. This is sometimes referred to as CCYY (century, century, year, year). The expression simply applies the \verb|YEAR()| function to a variable which returns an integer value.

\verbatiminput{Expressions/YYYY.exp}

To make use of it, it's been my experience that we will be using this value for string concatenation instead of mathematical operations: addition, subtraction, etc. Therefore, we explicitly cast the result of the \verb|YEAR()| as a unicode string, \verb|DT_WSTR| of length 4

Biml
\verbatiminput{Expressions/YYYY.biml}

\section{YY}
This expression defines how to get a 2 digit year. We apply the \verb|RIGHT()| function and get the last 2 digits. There is no need to explicitly cast to a string as it is handled by the function call.

\verbatiminput{Expressions/YY.exp}

Biml
\verbatiminput{Expressions/YY.biml}

\section{MM}
This expression defines how to get a 2 digit month such that March is 03 instead of 3. The leading zero is important for character based sorting. We will use the \verb|MONTH()| function which returns an integer from 1 to 12. We convert that to a string, prepend a zero to it and then shear off the last two characters using the \verb|RIGHT()| function. The trimming is only required for October, November, and December but the logic is cleaner to unconditionally apply \verb|RIGHT()|.

\verbatiminput{Expressions/MM.exp}

Biml
\verbatiminput{Expressions/MM.biml}

\section{DD}
This expression defines how to get a 2 digit day such that instead of 9 we get 09. It is the same process used in MM except with a call to  \verb|DAY()|. We continue to prepend a zero to our value and shear off the last two digits.

\verbatiminput{Expressions/DD.exp}

Biml
\verbatiminput{Expressions/DD.biml}

\section{YYYY-MM-DD}
The most common format is a year, month and day separated by a dash. This recipe uses what we've build in the YYYY, MM and DD sections.

verbatimimport

\verbatiminput{Expressions/YYYY-MM-DD.exp}

Biml
\verbatiminput{Expressions/YYYY-MM-DD.biml}

\section{YYYYMMDD}
A more compact version of the year month date string is represented below.

\verbatiminput{Expressions/YYYYMMDD.exp}

Biml
\verbatiminput{Expressions/YYYYMMDD.biml}