String matching is determining all the indices in a source string
where a given target string begins; for example, for source string
\texttt{ababab} and target \texttt{aba} the results of string
matching would be \texttt{[0, 2]}.

We now define a suitable monoid, @SM target@, for the codomain of
a string matching function, where @target@ is the string being looked
for.
%
Additionally, we will define a function @toSM :: RString -> SM target@
which does the string matching and is indeed a monoid morphism from
@RString@ to @SM target@ for a given @target@.

\subsubsection{String Matching Monoid}

We define the data type
@SM target@ to contain a refined string field @input@ and
a list of all the indices in @input@ where the
@target@ appears.
%
\begin{code}
  data SM (target :: Symbol) where
    SM :: input:RString
       -> indices:[GoodIndex input target]
       -> SM target
\end{code}
%
We use the string type literal~\footnote{\texttt{Symbol} is a kind and
target is effectively a singleton type.} to parameterize the monoid
over the target being matched. This encoding allows the type checker
to statically ensure that only searches for the same target can be
merged together.  The input field is a refined string, and the indices
field is a list of good indices.  For simplicity we present lists as
Haskell's built-in lists, but our implementation uses the reflected
list type, @L@, defined in \S~\ref{sec:haskell-proofs}.

A @GoodIndex input target@ is a refined type alias for a natural
number @i@ for which @target@ appears at position @i@ of @input@.  As
an example, the good indices of @"abcab"@ on @"ababcabcab"@ are
@[2,5]@.
%
\begin{code}
  type GoodIndex Input Target
    = {i:Nat | isGoodIndex Input (fromString Target) i }

  isGoodIndex :: RString -> RString -> Int -> Bool
  isGoodIndex input target i
    = (subString i (lenStr target) input  == target)
    && (i + lenStr target <= lenStr input)

  subString :: Int -> Int -> RString -> RString
  subString o l = takeStr l . dropStr o
\end{code}
%
\ignore{
\begin{code}
goodSM :: SM "abcab"
goodSM = SM "ababcabcab" [2, 5]

badSM  :: SM "abcab"
badSM  = SM "ababcabcab" [0, 7]
\end{code}
\ignore{
\NV{Liquid Haskell actually will reject both the above, as the lenStr and subString functions are uninterpreted}
}
}

\subsubsection{Monoid Methods for String Matching}~\label{subsec:monoid:methods}
Next, we define the mappend and identity elements for string matching.

The \textit{identity element} @mempty@ of @SM t@, for each target @t@, is
defined to contain the identity @RString@ (@stringMempty@) and the
identity @List@ (@listMempty@).
\begin{code}
  mempty:: forall (t :: Symbol). SM t
  mempty = SM stringMempty listMempty
\end{code}
%

\ignore{
The associative operator, @(mappend)@,
appends the two input strings.
The appended indices, as depicted in Figure~\ref{fig:mappend:indices},
are the concatenations of three list indices:
\begin{enumerate}
\item The indices @xis@ of the first input, casted to good indices in the new structure,
\item the new indices @xyis@ created when concatenating the two strings, and
\item the indices @yis@ of the second input, shifted right @lenStr y@ units.
\end{enumerate}
%
}

\begin{figure}[t]
\centering
\captionsetup{justification=centering}
\includegraphics[scale=0.5]{text/stringmatcher/makeIndices}
\caption{Mappend indices of String Matcher.}
\label{fig:mappend:indices}
\end{figure}
%
The Haskell definition of @<>@, the monoid operation for @SM t@, is as follows.
\begin{code}
  (mappend)::forall (t::Symbol). KnownSymbol t => SM t -> SM t -> SM t
  (SM x xis) mappend (SM y yis)
    = SM (x stringMappend y) (xis' listMappend xyis listMappend yis')
    where
      tg   = fromString (symbolVal (Proxy :: Proxy t))
      xis' = map (castGoodIndexLeft tg x y) xis
      xyis = makeNewIndices x y tg
      yis' = map (shiftStringRight tg x y) yis
\end{code}
Note again that capturing target as a type parameter is critical,
otherwise there is no way for the Haskell's type system to specify
that both arguments of @(mappend)@ are string matchers on the same target.

The action of @(<>)@ on the two @input@ fields is straightforward;
however, the action on the two @indices@ is complicated by the need to
shift indices and the possibility of new matches arising from the
concatenation of the two @input@
fields. Figure~\ref{fig:mappend:indices} illustrates the three pieces
of the new @indices@ field which we now explain in more detail.

\mypara{1. Casting Good Indices}
If @xis@ is a list of good indices for the string @x@ and the target
@tg@, then @xis@ is also a list of good indices for the string
@x stringMappend y@ and the target @tg@, for each @y@.
%
To prove this property we need to invoke the property
@subStrAppendRight@ on Refined Strings that establishes
substring preservation on string right appending.
%
\begin{code}
  assume subStrAppendRight
      :: sl:RString -> sr:RString -> j:Int
      -> i:{Int | i + j <= lenStr sl }
      -> {subString sl i j = subString (sl stringMappend sr) i j}
\end{code}
%
The specification of @subStrAppendRight@ ensures that for each
string @sl@ and @sr@ and each integer @i@ and @j@ whose sum is within @sl@,
the substring from @i@ with length @j@ is identical in @sl@ and in @(sl stringMappend sr)@.
%
The function @castGoodIndexLeft@ applies the above property to an index @i@
to cast it from a good index on @sl@ to a good index on @(sl stringMappend sr)@
%
\begin{code}
  castGoodIndexLeft
    :: tg:RString -> sl:RString -> sr:RString
    -> i:GoodIndex sl tg
    -> {v:GoodIndex (sl stringMappend sr) target | v = i}

  castGoodIndexLeft tg sl sr i
    = cast (subStrAppendRight sl sr (lenStr tg) i) i
\end{code}
%
Where @cast p x@ returns @x@, after enforcing the properties of @p@ in the logic
\begin{code}
  cast :: b -> x:a -> {v:a | v = x}
  cast _ x = x
\end{code}
%
Moreover, in the logic, each expression @cast p x@
is reflected as @x@,
thus allowing random (\ie non-reflected) Haskell expressions to appear in @p@.

\mypara{2. Creation of new indices}
The concatenation of two input strings @sl@ and @sr@
may create new good indices.
%
For instance, concatenation of
@"ababcab"@ with @"cab"@
leads to a new occurence of @"abcab"@ at index @5@ which
does not occur in either of the two input strings.
%
These new good indices can appear only at the last @lenStr tg@ positions
of the left input @sl@.
%
@makeNewIndices sl sr tg@ detects all such good new indices.
%
\begin{code}
  makeNewIndices
    :: sl:RString -> sr:RString -> tg:RString
    -> [GoodIndex {sl stringMappend sr} tg]

  makeNewIndices sl sr tg
    | lenStr tg < 2 = []
    | otherwise     = makeIndices (sl stringMappend sr) tg lo hi
    where
      lo = maxInt (lenStr sl - (lenStr tg - 1)) 0
      hi = lenStr sl - 1
\end{code}
If the length of the @tg@ is less than 2, then no new good indices are created.
%
Otherwise,
the call on @makeIndices@ returns all the good indices of the input
@sl stringMappend sr@ for target @tg@
in the range from @maxInt (lenStr sl-(lenStr tg-1)) 0@ to  @lenStr sl-1@.
%

Generally, @makeIndices s tg lo hi@ returns the good indices
of the input string @s@ for target @tg@ in the range from @lo@ to @hi@.
%
\begin{code}
  makeIndices :: s:RString -> tg:RString -> lo:Nat
              -> hi:Int -> [GoodIndex s tg]

  makeIndices s tg lo hi
    | hi < lo             = []
    | isGoodIndex s tg lo = lo:rest
    | otherwise           = rest
    where
      rest = makeIndices s tg (lo + 1) hi
\end{code}
%Note the similarity to the functional specification of string matching
%at the beginning of this section.

It is important to note that
@makeNewIndices@ does not scan all the input,
instead only searching at most @lenStr tg@ positions for new good indices.
%
Thus, the time complexity to create the new indices is linear
on the size of the target but independent of the size of the input.

\mypara{3. Shift Good Indices}
If @yis@ is a list of good indices on the string @y@ with target @tg@,
then we need to shift each element of @yis@ right @lenStr x@ units to
get a list of good indices for the string @x stringMappend y@.

%
To prove this property we need to invoke the property
@subStrAppendLeft@ on Refined Strings that establishes
substring shifting on string left appending.
%
\begin{code}
  assume subStrAppendLeft
    :: sl:RString -> sr:RString
    -> j:Int -> i:Int
    -> {subStr sr i j = subStr (sl stringMappend sr) (lenStr sl+i) j}
\end{code}
%
The specification of @subStrAppendLeft@ ensures that for each
string @sl@ and @sr@ and each integers @i@ and @j@,
the substring from @i@ with length @j@ on @sr@
is equal to the substring from @lenStr sl + i@
with length @j@ on @(sl stringMappend sr)@.
%
The function @shiftStringRight@ both shifts the input index @i@ by @lenStr sl@
and applies the @subStrAppendLeft@ property to it,
casting @i@ from a good index on @sr@ to a good index on @(sl stringMappend sr)@

Thus, @shiftStringRight@ both appropriately shifts the index
and casts the shifted index using the above theorem:
\begin{code}
  shiftStringRight
    :: tg:RString -> sl:RString -> sr:RString
    -> i:GoodIndex sr tg
    -> {v:(GoodIndex (sl stringMappend sr) tg) | v = i + lenStr sl}

  shiftStringRight tg sl sr i
    = subStrAppendLeft sl sr (lenStr tg) i `cast` i + lenStr sl
\end{code}

\subsubsection{String Matching is a Monoid}
Next we prove that the monoid methods @mempty@ and @(mappend)@ satisfy
the monoid laws.
%
\begin{theorem}[SM is a Monoid]\label{theorem:stringmatchers}
(@SM t@, @mempty@, @mappend@)
is a monoid.
\end{theorem}
%
\begin{proof}
According to the Monoid Definition~\ref{definition:monoid},
we prove that string matching is a monoid,
by providing safe implementations for the monoid law functions.
%
First, we prove \textit{left identity}.
\begin{code}
  idLeft :: x:SM t -> {mempty mappend x = xs}

  idLeft (SM i is)
    =   (mempty :: SM t) mappend (SM i is)
    =. (SM stringMempty listMempty) mappend (SM i is)
    =. SM (stringMempty <+> i) (is1 ++ isNew ++ is2)
       ? idLeftStr i
    =. SM i ([] ++ [] ++ is)
       ? (mapShiftZero tg i is && newIsNullRight i tg)
    =. SM i is
       ? idLeftList is
    ** QED
    where
      tg    = fromString (symbolVal (Proxy :: Proxy t))
      is1   = map (castGoodIndexRight tg i stringMempty) []
      isNew = makeNewIndices stringMempty i tg
      is2   = map (shiftStringRight tg stringMempty i) is
\end{code}

The proof proceeds by rewriting, using left identity of the monoid strings and lists,
and two more lemmata.
\begin{itemize}
\item Identity of shifting by an empty string.
\begin{code}
  mapShiftZero :: tg:RString -> i:RString
    -> is:[GoodIndex i target]
    -> {map (shiftStringRight tg stringMempty i) is = is}
\end{code}
The lemma is proven by induction on @is@ and
the assumption that empty strings have length 0.
\item No new indices are created.
\begin{code}
  newIsNullLeft :: s:RString -> t:RString
                -> {makeNewIndices stringMempty s t = []}
\end{code}
The proof relies on the fact that @makeIndices@
is called on the empty range from @0@ to @-1@
and returns @[]@.
\end{itemize}

Next, we prove \textit{right identity}.
\begin{code}
  idRight :: x:SM t -> {x mappend mempty = x}

  idRight (SM i is)
    =  (SM i is) mappend (mempty :: SM t)
    =. (SM i is) mappend (SM stringMempty listMempty)
    =. SM (i stringMappend stringMempty) (is1 listMappend isNew listMappend is2)
       ? idRightStr i
    =. SM i (is listMappend N listMappend N)
       ? (mapCastId tg i stringMempty is && newIsNullLeft i tg)
    =. SM i is
       ? idRightList is
    **  QED
    where
      tg    = fromString (symbolVal (Proxy :: Proxy t))
      is1   = map (castGoodIndexRight tg i stringMempty) is
      isNew = makeNewIndices i stringEmp tg
      is2   = map (shiftStringRight tg i stringMempty) []
\end{code}
The proof proceeds by rewriting,
using right identity on strings and lists and two more lemmata.
\begin{itemize}
\item Identity of casting is proven
\begin{code}
  mapCastId
    :: tg:RString -> x:RString -> y:RString
    -> is:[GoodIndex x tg] ->
    -> {map (castGoodIndexRight tg x y) is = is}
\end{code}
We prove identity of casts by induction on @is@ and
identity of casting on a single index.
\item No new indices are created.
\begin{code}
  newIsNullLeft :: s:RString -> t:RString
                -> {makeNewIndices s stringMempty t = listMempty}
\end{code}
The proof proceeds by case splitting
on the relative length of @s@ and @t@.
At each case we prove by induction that all
the potential new indices would be out of bounds and thus
no new good indices would be created.
\end{itemize}

- Finally we prove \textit{associativity}.
For space, we only provide a proof sketch.
The whole proof is available online~\cite{implementation}.
%
Our goal is to show equality of the left and right associative string matchers.
%
\begin{code}
  assoc :: x:SM t -> y:SM t -> z:SM t
        -> {x mappend (y mappend z) = (x mappend y) mappend z}
\end{code}
To prove equality of the two string matchers we show
that the input and indices fields are respectively equal.
%
Equality of the input fields follows by associativity of RStrings.
%
Equality of the index list proceeds in three steps.
%
\begin{enumerate}
\item Using list associativity and distribution of index shifting,
we group the indices in the five lists shown in
Figure~\ref{fig:mappend:assoc}: the indices of the input @x@, the new
indices from mappending @x@ to @y@, the indices of the input @y@, the
new indices from mappending @x@ to @y@, and the indices of the input
@z@.
\item The representation of each group depends on the order of appending.
For example, if @zis1@ (resp. @zis2@) is the group @zis@ when
right (resp. left) mappend happened first, then we have
\begin{code}
  zis1 = map (shiftStringRight tg xi (yi stringMappend zi))
             (map (shiftStringRight tg yi zi) zis)

  zis2 = map (shiftStringRight tg (xi stringMappend yi) zi) zis
\end{code}
That is, in right first, the indices of @z@ are first shifted
by the length of @yi@ and then by the length of @xi@,
while in the left first case, the indices of @z@ are shifted by the
length of @xi stringMappend yi@.
In this second step of the proof we prove, using lemmata,
the equivalence of the different group representations.
%
The most interesting lemma we use is called @assocNewIndices@ and proves
equivalence of all the three middle groups together
by case analysis on the relative lengths of the target @tg@ and the middle string @yi@.
\item After proving equivalence of representations,
we again use list associativity and distribution of casts to wrap the
index groups back in string matchers.
\end{enumerate}
\begin{figure}
\centering
\captionsetup{justification=centering}
\includegraphics[scale=0.5]{text/stringmatcher/AssociativeIndices}
\caption{Associativity of String Matching.}
\label{fig:mappend:assoc}
\end{figure}
We now sketch the three proof steps, while the whole proof
is available online~\cite{implementation}.
\begin{code}
  assoc x@(SM xi xis) y@(SM yi yis) z@(SM zi zis)
    -- Step 1: unwrapping the indices
    =   x <> (y <> z)
    =. (SM xi xis) <> ((SM yi yis) <> (SM zi zis))
                         ...
    -- via list associativity and distribution of shifts
    =. SM i (xis1 ++ ((xyis1 ++ yis1 ++ yzis1) ++ zis1))
    -- Step 2: Equivalence of representations
    =. SM i (xis2 ++ ((xyis1 ++ yis1 ++ yzis1) ++ zis1))
       ? castConcat tg xi yi zi xis
    =. SM i (xis2 ++ ((xyis1 ++ yis1 ++ yzis1) ++ zis2))
       ? mapLenFusion tg xi yi zi zis
    =. SM i (xis2 ++ ((xyis2 ++ yis2 ++ yzis2) ++ zis2))
       ? assocNewIndices y tg xi yi zi yis
    -- Step 3: Wrapping the indices
                         ...
    -- via list associativity and distribution of casts
    =. (SM xi xis <> SM yi yis) <> SM zi zis
    =. (x <> y) <> z
    ** QED
    where
      i     = xi stringMappend (yi stringMappend zi)

      yzis1 = map (shiftStringRight tg xi (yi <+> zi)) yzis
      yzis2 = makeNewIndices (xi <+> yi) zi tg
      yzis  = makeNewIndices yi zi tg
      ...
\end{code}
\cqed\end{proof}


\documentclass{subfile}

\begin{document}

	\section{SgMO}\label{sec:sgmo}

		\begin{problem}[Singapore Team Selection Test $2009$, problem $2$]
			If $a,b,c$ are three positive real numbers such that $ab+bc+ca=1$, prove that
				\begin{align*}
					\sqrt[3]{\dfrac{1}{a}+6b}+\sqrt[3]{\dfrac{1}{b}+6c}+\sqrt[3]{\dfrac{1}{c}+6a}
						& \leq\dfrac{1}{abc}
				\end{align*}
		\end{problem}

		\begin{problem}[Singapore Team Selection Test $2008$, problem $2$]
			Let $x_{1},\ldots,x_{n}$ be positive real numbers such that $x_{1}\cdots x_{n}=1$. Prove that
				\begin{align*}
					\sum_{i=1}^{n}\dfrac{1}{n-1+x_{i}}
						& \leq1
				\end{align*}
		\end{problem}

		\begin{problem}[Singapore Team Selection Test $2007$, problem $2$]
			Prove the inequality
				\begin{align*}
					\sum_{i<j}\dfrac{a_{i}a_{j}}{a_{i}+a_{j}}
						& \leq \dfrac{n}{2(a_{1}+\ldots+a_{n})}\sum_{i<j}a_{i}a_{j}
				\end{align*}
			for all positive real numbers $a_{1},\ldots,a_{n}$.
		\end{problem}

		\begin{problem}[Singapore $2006$, problem $2$]
			Let $n>1$ be an integer and $x_{1},\ldots,x_{n}$ be real numbers such that
				\begin{align*}
					|x_{1}|+\ldots+|x_{n}|
						& = 1\quad\mbox{and}\\
					x_{1}+\ldots+x_{n}
						& = 0
				\end{align*}
			Prove that
				\begin{align*}
					\left|\dfrac{x_{1}}{1}+\ldots+\dfrac{x_{n}}{n}\right|
						& \leq \dfrac{1}{2}\left(1-\dfrac{1}{n}\right)
				\end{align*}
		\end{problem}

		\begin{problem}[Singapore $2004$, problem $2$]
			Let $a,b,c$ be real numbers such that $ab+bc+ca=1$ and $0<a,b,c<1$. Prove that
				\begin{align*}
					\dfrac{a}{1-a^{2}}+\dfrac{b}{1-b^{2}}+\dfrac{c}{1-c^{2}}
						& \geq\dfrac{3\sqrt{3}}{2}
				\end{align*}
		\end{problem}
\end{document}


\documentclass[10pt,a4paper]{article}%{scrartcl}
    \newtheorem{defi}{Definition}[section]
    \usepackage[inner=1.0cm,outer=1.0cm,top=0cm,bottom=1cm,includeheadfoot]{geometry}
    \usepackage[utf8]{inputenc}
    \usepackage[english]{babel}
    \usepackage{xcolor}
    \usepackage[pdftex]{graphicx}
    \usepackage[colorlinks,linkcolor=black,citecolor=black]{hyperref}
    \usepackage[numbers]{natbib}
    \usepackage{tabularx}
    \usepackage{amsmath}
    \usepackage{amssymb}
    \usepackage{mathtools}
    \usepackage{txfonts}
    \usepackage{placeins}
    \usepackage[skip=0pt]{caption}
    \usepackage{multicol}

    \newlength\tindent
    \setlength{\tindent}{\parindent}
    \setlength{\parindent}{0pt}
    \renewcommand{\indent}{\hspace*{\tindent}}

\begin{document}

\section{Col/Row major}
\begin{align*}
\underbrace{\bordermatrix{
		i\ \text{\textbackslash} \ j
		& 0 &  1 & 2 & 3      \cr
		0 & 0  & 1  & 2  & 3  \cr
		1 & 4  & 5  & 6  & 7  \cr
		2 & 8  & 9  & 10 & 11 \cr
		3 & 12 & 13 & 14 & 15 \cr
		4 & 16 & 17 & 18 & 19 \cr
}}_{\textstyle\begin{array}{c}\text{row-major order}
\end{array}}
\qquad\quad
\underbrace{\bordermatrix{
		i\ \text{\textbackslash} \ j
		& 0 & 1 & 2 & 3     \cr
		0 & 0 & 5 & 10 & 15 \cr
		1 & 1 & 6 & 11 & 16 \cr
		2 & 2 & 7 & 12 & 17 \cr
		3 & 3 & 8 & 13 & 18 \cr
		4 & 4 & 9 & 14 & 19 \cr
}}_{\textstyle\begin{array}{c}\text{column-major order}
	\end{array}}
\end{align*}

\section{Cache aware data storage}
\begin{align*}
\boldsymbol{C}= &\boldsymbol{A} \times  \boldsymbol{B} \\
= & \quad\underbrace{\bordermatrix{
		i\ \text{\textbackslash} \ j
		& 0 &  1 &  2 &  3 & 4    \cr
		  0 &  0 &  1 &  2 & 3  & 4  \cr
		  1 &  5 &  6 &  7 & 8  & 9  \cr
		  2 & 10 & 11 & 12 & 13 & 14 \cr
		  3 & 15 & 16 & 17 & 18 & 19 \cr
}}_{\textstyle\begin{array}{c}\text{matrix A row-major}
	\end{array}}
\quad \times\quad
\underbrace{\bordermatrix{
		i\ \text{\textbackslash} \ j
		& 0 & 1 & 2 & 3     \cr
		0 & 0 & 5 & 10 & 15 \cr
		1 & 1 & 6 & 11 & 16 \cr
		2 & 2 & 7 & 12 & 17 \cr
		3 & 3 & 8 & 13 & 18 \cr
		4 & 4 & 9 & 14 & 19 \cr
}}_{\textstyle\begin{array}{c}\text{matrix B column-major}
	\end{array}}
\end{align*}

\end{document}


\documentclass{article}
\usepackage[letterpaper,top=2cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{ulem}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

\title{Expectation Maximization for Mixture Model}
\author{dw}

\begin{document}
\maketitle
\section{Mixture model}
In statistics, a mixture model is a probabilistic model for representing the presence of subpopulations within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observation belongs. It is common to think of probability mixture modeling as a missing data problem. One way to understand this is to assume that the data points under consideration have "membership" in one of the distributions we are using to model the data. When we start, this membership is unknown, or missing. \href{https://en.wikipedia.org/wiki/Mixture_model}{[1]}

Assuming $N$ observation (index $i$) are observed in $K$ distributions(index $K$). The basis model is:
\begin{equation}
	p(x_i|\theta) = \sum_k \alpha_k p(x_i| \theta_k)
\end{equation}

\begin{itemize}
	\item $\theta$ - parameters including $\theta_k$ and $\alpha_k$.
	\item $\theta_k$ - parameters of $k$-th distribution.
	\item $\alpha_k$ - weight or prior, equal to $p(z_{ik})$.
\end{itemize}

\textbf{Proof}:
Hidden variables $z_{ik}$s indicate which class the $i$-th observation belongs to, value 1 shows in while 0 shows not, and $\sum_k {z_ik} = 1$, which satisfy categarical distribution.
Likelihood including the hidden variables and observations is:

\begin{equation}
	\mathcal{L}(\theta|x_i, z_{ik}) \propto \prod_k p(x_i,z_{ik}|\theta_k)
\end{equation}

Using formula of total probability:

It just a trick to promise $z_{ik}=0$ make nonsense in the formula. Using categarical distribution, we have

\begin{equation}
	\begin{aligned}
		\mathcal{L}(\theta|x_i) \propto \sum_k p(x_i | z_{ik}, \theta) p(z_{ik} | \theta)
		  & = \sum_k \prod_k p^{z_{ik}}(x_i,z_{ik}|\theta_k) p(z_{ik}|\theta) \\
		  & = \sum_k \alpha_k p(x_i,z_{ik}|\theta_k)
	\end{aligned}
\end{equation}

Don't be confused that:
\begin{itemize}
	\item $p(x_i,z_{ik}|\theta) = \{\alpha_k p(x_i|\theta_k)\}^{z_{ik}}$
	\item $p(x_i|\theta,z_{ik}) = p^{z_{ik}}(x_i|\theta_k)$
	\item $p(z_{ik}|\theta) = \alpha_k$
\end{itemize}

\section{Expectation Maximization}
However, logsum is hard to optimize since all parameters are coupled, using Jensen Inequality we get the lower bound:
\begin{equation}
	\log\mathcal{L}_{i}(\theta|x_i) \propto \sum_{i} \log p(x_i|\theta) \geq \sum_{i} \sum_k q(z_{ik})\log\{\frac{p(x_i,z_{ik}|\theta)}{q(z_{ik})}\} = F(q,\theta)
\end{equation}

Here $q(z_{ik})$ is an arbitrary distribution of $z_{ik}$, satisfying $\sum_k q(z_{ik}) = 1$ only when
\begin{equation}
	q(z_{ik}) = \frac{\alpha_k p(x_i, z_{ik})}{\sum_{k'} p(x_i, z_{ik'})}
\end{equation}
the equality holds.

\textbf{Proof}:
By Jensen's inequality, for the concave function (such as $\log$) we have
\begin{equation}
	f(\frac{\sum_i a_i x_i}{\sum_i a_i}) \geq \frac{\sum_i a_i f(x_i)}{\sum_i a_i}
\end{equation}
When $\sum_i a_i = 1$, it becomes $f(\sum_i a_i x_i) \geq \sum_i a_i f(x_i)$
Replace $a_i$ by $q(z_{ik})$ by $\{\frac{p(x_i,z_{ik}|\theta)}{q(z_{ik})}\}$, we have

\begin{equation}
	\begin{aligned}
		\sum_i \log p(x_i|\theta) & = \sum_i \log \sum_k p(x_i, z_{i k}|\theta)                                 \\
		                          & = \sum_i \log \sum_k q(z_{ik}) \{\frac{p(x_i, z_{ik}|\theta)} {q(z_{ik})}\} \\
		                          & \geq \sum_i \sum_k q(z_{ik})\log\{\frac{p(x_i,z_{ik}|\theta)}{q(z_{ik})}\}
	\end{aligned}
	\label{eq: Ineq}
\end{equation}

when $\{\frac{p(x_i,z_{ik}|\theta)}{q(z_{ik})}\}$ is nothing to do with $k$ the inequality holds. Notice that
\begin{equation}
	p(z_{ik}|x_i,\theta) = \frac{p(x_i, z_{ik} |\theta)}{\sum_{k'} p(x_i , z_{ik'} | \theta)} = \frac{\alpha_k p(x_i|\theta_k)} {\sum_{k'} {\alpha_{k'} p(x_i|\theta_{k'})}}
\end{equation}

take $z_{ik}$ as $p(z_{ik}|x_i,\theta)$ and the $\sum_{k'} {\alpha_{k'} p(x_i|\theta_{k'})}$ is a constant.

\section{EM flow chart}
\begin{itemize}
	\item \textbf{Initial}: Giving an initial guess of $\theta_{[0]}: \alpha_{k}^{[0]}, \theta_{k}^{[0]}$
	\item \textbf{E-step}:
	      $q^{[\mathrm{new}]} \leftarrow \arg\max_q F(q,\theta^{[\mathrm{old}]})$
	      \begin{equation}
		      \mathcal{L}(\theta^{[\mathrm{old}]}|t) \geq F(q,\theta^{[\mathrm{old}]}) = \sum_i \sum_k q(z_{ik})\log\{\frac{p(t_{i},z_{ik}|\theta^{[\mathrm{old}]})}{q(z_{ik})}\}
	      \end{equation}

	      we get
	      \begin{equation}
		      q(z_{ik}) = \frac{\alpha_k p(x_i|\theta_k^{[\mathrm{old}]})} {\sum_{k'} {\alpha_{k'} p(x_i|\theta_{k'}^{[\mathrm{old}]})}}
	      \end{equation}

	\item \textbf{M-step}:
	      $\theta^{[\mathrm{new}]} \leftarrow \arg\max_{\theta} F(q^{[\mathrm{new}]},\theta)$\\
	      By this way, the parameters in different classes can be optimized individually, Notice that:
	      \begin{equation}
		      F(q,\theta) = \sum_i \sum_k q(z_{ik}) \log{p(x_{i}, z_{ik}|\theta)} - \sum_i \sum_k q(z_{ik}) \log{q(z_{ik})}
	      \end{equation}
	      Only the former part is dependent on $\theta$, noted as $Q(q,\theta)$
	      \begin{equation}
		      \begin{aligned}
			      Q(q^{[\mathrm{new}]},\theta) & = \sum_i \sum_k  q^{[\mathrm{new}]}(z_{ik})\log{p(x_i,z_{ik}|\theta)}               \\
			                                   & = \sum_k \left\{\sum_i q^{[\mathrm{new}]}(z_{ik})\log{p(x_i,z_{ik}|\theta)}\right\}
		      \end{aligned}
	      \end{equation}

	      The optimize target is change to single class.
	\item Repeat until converge.
\end{itemize}
\textbf{Proof of Convergence:}
1. In E-step, Jensen Inequality promise the convergence. \\
2. In M-step, the optimize step promise the convergence. \\
However, EM is not a global minimizer.

\section{Alternative}
E-step define the lower bounds of $\mathcal{L}(\theta|t) - \mathrm{KL}(q(z_{ik})||p(z_{ik}|x_i,\theta) $ \\
Difference between $\log\mathcal{L}(\theta|x) $ and $ F(q,\theta)$ is the KL-divergence of:
Proof:
\begin{equation}
	\begin{aligned}
		L(\theta) - F(q,\theta) & = \sum_i^N \sum_{k=1}^K \log\{{ p(x_i|\theta)}\} - \sum_i^N \sum_{k=1}^K q(z_{ik})\log\{\frac{p(x_i,z_{ik}|\theta)}{q(z_{ik})}\}          \\
		                        & = \sum_i^N \sum_{k=1}^K q(z_{ik})\log\{{ p(x_i|\theta)}\} - \sum_i^N \sum_{k=1}^K q(z_{ik})\log\{\frac{p(x_i,z_{ik}|\theta)}{q(z_{ik})}\} \\
		                        & = \sum_i^N \sum_{k=1}^K q(z_{ik})\log\{\frac{p(x_i|\theta)/p(x_i,z_{ik}|\theta)}{q(z_{ik})}\}                                             \\
		                        & = \sum_i^N \sum_{k=1}^K q(z_{ik})\log\{\frac{p(z_{ik}|x_i,\theta)}{q(z_{ik})}\}                                                           \\
		                        & = \mathrm{KL}(q(z_{ik})||p(z_{ik}|x_i,\theta))
	\end{aligned}
\end{equation}
The KL-divergence is always positive and have value 0 only if $q(z_{ik}) = p(z_{ik}|x_i,\theta)$.

\section{Apply: Gaussian mixture model}
\href{https://biarnes-adrien.medium.com/em-of-gmm-appendix-m-step-full-derivations-4ae95cdd40c9}{Proof}
\end{document}


\section{Quantifiers (Bonevac)}

\begin{itemize}
    \item Aristotle logic gives arguments restricted form. Every sentence is of form $\{\text{some,all,no}\}F\{\text{are,not}\}G$
    \item Syllogistic argument has two such setences as premises and one as a conclusion. The meshing of these is specific, limited.
    \item \ti{Sentential logic} takes sentences as basic analytical units, covers broader realm.
    \item Sentential logic does not solve problems of syllogistic, e.g. cannot explain why an argument is valid
    \item Divergence b/w syllogistic and sentential resolved by Friege and Peirce
    \begin{itemize}
        \item Introduced determiners (e.g. all, some, no, every, any, etc.)
        \item Universal quantifier $\forall$ and existential quantifier $\exists$
    \end{itemize}
\end{itemize}

\subsection{Constants and Quantifiers}

Atomic sentences consist of a main subject/noun phrase and verb phrase. Examples of verb phrases:

\begin{enumerate}
    \item is a man
    \item sleeps very soundly
    \item kicked the ball into the end zone
\end{enumerate}

Verb phrases are terms in syllogistic logic, are \tb{true or false} of individual objects. E.g. a man can sleep soundly or not.
Objects of which verb phrase \tb{satisfy} it, phrase will \tb{apply} to them.
The set of objects which make a verb phrase true are called \tb{extensions}.

Verb phrases combine with noun phrases to form \ti{sentences}. Noun phrases specify an object or groups of objects, since verb phrases describe their truth values.
Following examples are complete sentences.

\begin{itemize}
    \item Socrates is a man
    \item Mr. Hendley sleeps very soundly
    \item Nate have Fred a copy of the letter
\end{itemize}

Upper-case alphabet letters w/wout subscripts are \tb{predicates}. Each predicate has an assigned number.
Predicate with number $n$ is $n$-ary. Predicate yields a truth value when combined with certain number of objects.
Assigned value $n$ indicates of how many objects the predicate takes on this truth value.

For example, man has singulary predicates (true false of a single object). Binary predicates take on two objects. Example is Person 1 respects Person 2, but Person 1 respects makes no sense.

To structure sentences, take example.

\begin{example}
    Something is missing
\end{example}

Missing applies to an object.

\begin{example}
    Missing(something)
\end{example}

We use lowercase letters to denote constants, capital for predicates as discussed.
Socrates is a man can be translated to $Ma$ where $a$ is a constant symbolizing Socrates, $M$ means man.

\begin{example} Translation\\
    (for some $x$)($x$ is missing)\\
    $\exists x M x$
\end{example}

\section{Quantificational Logic}

Main idea is that we use predicates to describe properties of variables, which we say are \ti{quantified}.
Say we define $Lxy$ to be that $x$ likes $y$. $L$ is our predicate, and $x,y$ are variables, of which we can pass any number to an (appropriate) predicate.

Idea of quantifiers
\begin{itemize}
    \item $\exists x$: there is some individual $x$ such that
    \item $\forall x$: every individual $x$ such that
\end{itemize}

\subsection{Subtlety with quantifier order}

Difference between
\begin{gather*}
    1.\;\exists y\forall x (Lxy)\\
    2.\;\forall x\exists y(Lxy)
\end{gather*}

\ti{First}: there exists someone who likes all individuals.\\
\ti{Second}: for every individual, there exists someone who likes them.

\subsection{Rules and properties}

An identity predicate exists to test equality of two subjects.

\begin{definition} (Identity predicate)
    $$x=y:\text{$x$ is equal to $y$}$$
\end{definition}

We can obviously tell that identical subjects have the same properties, so

\begin{definition} (Leibniz' law)
    $$\forall x\forall y(x=y\then (Px\leftrightarrow Py))$$
\end{definition}

We can't ``for loop'' over all properties of subjects, so that must be done manually.

\subsection{Describing quantities}

Let $Px$ mean that $x$ has property $P$. Be aware that we can trivially extend the below equivalences to $n$ subjects.

\begin{tabular}{l|l}
    Statement & QL\\
    \hline
    \hline
    There is exactly one thing that is $P$ & $\exists x(Px\land \forall y(Py\then x=y))$\\
    There are exactly two things that are $P$ & $\exists x\exists y(Px\land Py\land x\neq y\land \forall z(Pz\then (z=x\lor z=y))$\\
    There are at least two things that are $P$ & $\exists x\exists y(Px\land Py\land x\neq y)$\\
    There are less than two things that are $P$ & $\forall x\forall y(\lnot Px\lor \lnot Py\lor x=y)$ (this is not identical to the first)\\
\end{tabular}

\subsection{Interpretations}

To make interpretations of sentences (i.e. constant values using names) we set $UD=\{a,b,c\}$ or whatever,
then interpret the predicates with these names $Pa,Pb,Pc$, etc. Here we can cherry pick $a,b,c$ to arbitrarily give truth values to $P$ when interpreted. Useful for informal proofs.

\subsection{Natural deduction}

Below is a list of inferences or tautologies for natural deduction.

\begin{tabular}{l|l|l}
    Name & Tautology or Inference & Code\\
    \hline
    \hline
    Quantifier Exchange (occurs both ways) & $\exists x(\dots x \dots)\leftrightarrow \forall x \lnot(\dots x\dots)$ & QEx\\
    Universal Instantiation & $\forall x(\dots x\dots)\therefore \dots a \dots$ where $a$ is a new name in proof & UI\\
    Universal Generalization & $\dots a \dots\; \therefore\forall x(\dots x\dots)$ iff $a$ was declared by UI & UG\\
    Existential Instantiation & $\exists x(\dots x\dots)\therefore \dots a\dots$ where $a$ is any name & EI\\
    Existential Generalization & $\dots a\dots \;\therefore\exists x(\dots x\dots)$ & EG\\
    Leibniz Law & $\forall x\forall y(x=y\then (Px\leftrightarrow Py))$ & Leibniz
\end{tabular}


\newtheorem{theorem}{Theorem}

\newcommand{\ff}{f}
\newcommand{\gf}{g}

\newcommand{\knl}{k}

\chapter{Properties of the effective field operator}
\chaptermark{Effective field operator}
\label{sec:properties-of-field-operators}

In \thisref{sec:properties-of-field-operators} we show some properties of the effective field operator that are used in \cref{sec:energy-cons}.
It seems possible that these properties could be proven for a general effective field using only the definition \cref{eq:62}, but we have not managed to find such a proof.

\section{Linear Symmetrical operator}
\label{sec:linear-symm-field-operators}

An operator $\lop$ is linear if
\begin{equation}
  \lop[\av + c\bv] = \lop[\av] + c\lop[\bv].
\end{equation}
The vector Laplace, magnetostatic field and magnetocrystalline anisotropy operators are all easily demonstrated to be linear because they involve only linear operations, such as derivatives, integrals and dot products.

An operator $\lop$ is symmetrical if
\begin{equation}
  \ip{\lop \av}{\bv} = \ip{\av}{\lop \bv},
\end{equation}
for all functions $\av, \bv \in $ in some function space on the real numbers where $\ip{\cdot}{\cdot}$ is defined.

In \thisref{sec:linear-symm-field-operators} we frequently make use of a consequence of the divergence theorem:
\begin{equation}
  \intd{\fv(\xv) \cdot \grad \gf(\xv)}
  = \intb{\gf(\xv) \, (\fv(\xv) \cdot \nv(\xv))} - \intd{\gf(\xv) \, \div \fv(\xv)},
  \label{eqn:grad-divergence}
\end{equation}
where $\fv : \real^d \rightarrow \real^d$ and $\gf : \real^d \rightarrow \real$.

Note that by substituting $\fv = \grad \ff$ into \cref{eqn:grad-divergence} we can derive
\begin{equation}
  \begin{aligned}
    \intd{(\lap \ff) \gf}
    &= \intb{\gf (\grad \ff \cdot \nv)} - \intd{\grad \ff \cdot \grad \gf}, \\
    &= \intb{\gf \ddn{\ff}} - \intd{\grad \ff \cdot \grad \gf}.
    \label{eqn:laplace-divergence}
  \end{aligned}
\end{equation}

\subsection{Applied field}

The applied field part of the effective field is in general \emph{not} linear or symmetrical because it is independent of $\mv$.

\subsection{Vector Laplace operator}

\begin{theorem}
  If $\lop$ is a symmetric operator on $v \in \ltwo$ then so is its ``vector equivalent'', $\bar{\lop}[\vv] = \threevec{\lop v_1}{\lop v_2}{\lop v_3}$.
\end{theorem}

\begin{proof}
  \begin{equation}
    \begin{aligned}
      \ip{\bar{\lop}\av}{\bv} &= \intd{ \bar{\lop} \av \cdot \bv}, \\
      &= \intd{\lop[a_1] b_1} + \intd{\lop[a_2] b_2} + \intd{\lop[a_3] b_3}, \\
      & = \ip{\lop[a_1]}{b_1} + \ip{\lop[a_2]}{b_2} + \ip{\lop[a_3]}{b_3}.
    \end{aligned}
  \end{equation}
  So $\bar{\lop}$ is symmetrical if and only if $\lop$ is symmetrical.
\end{proof}

\begin{theorem}[Symmetry of Laplace operator]
  If $m_i \in \ltwo$ and $\ddn{m_i} = 0$ on all of $\boundd$ then $\lap$ is a linear operator on $m_i$.
\end{theorem}
\begin{proof}
  Apply equation~\cref{eqn:laplace-divergence} twice: first with $\ff = a$, $\gf = b$, then the other way around.
  \begin{equation}
    \begin{aligned}
      \label{eq:93}
      \ip{\lap a}{b} &= \intd{\left(\lap a \right) b}, \\
      &= \intb{b \ddn{a}} - \intd{\grad a \cdot \grad b}, \\
      &= \intb{b \ddn{a}} + \intd{a (\lap b)} - \intb{a \ddn{b}}, \\
      &= \intd{a (\lap b)}.
    \end{aligned}
  \end{equation}
\end{proof}

From these two theorems we see that the vector Laplacian operator $\lap$ is symmetrical.
Note that the above does not include the case with surface anisotropy or when the length of $\mv$ is not constant because in either of these cases we do not necessarily have $\ddn{\mv} = 0$.
The case with periodic boundary conditions is true by the combination of \cref{eq:93,eq:95}.

\subsection{Magnetostatic field operator}

For simplicity we write
\begin{equation}
  \knl = \frac{1}{4\pi \abs{\xv - \xv'}}.
\end{equation}

\begin{theorem}[Symmetry of magnetostatic field operator]
  If $\av, \bv \in \ltwo$ and $\grad \av, \grad\bv \in \ltwo$  (\ie $\av, \bv \in \sob^1$) then the operator
  \begin{equation}
    \begin{aligned}
      \hmsop [\av](\xv) &= - \grad \phim[\av](\xv), \\
      &= -\grad \bigs{
              -\intd[\magd']{ \knl \grad' \cdot \av(\xv') }
              + \intd[\boundd']{\knl \av(\xv') \cdot \nv(\xv') }
            },
    \end{aligned}
  \end{equation}
  where primes denote another set of coordinates, is symmetrical.
  %% Do we also need some properties on k? Is k in ltwo -- yes int(k) something like [1/x]form -inf to inf = 0
\end{theorem}

\begin{proof}

  Essentially we apply identity~\cref{eqn:grad-divergence}, rearrange the result using the symmetry of the kernel, $\knl$, and apply the identity again in reverse. We drop the $\xv$ argument from $\phim$ where it is obvious.

  Using~\cref{eqn:grad-divergence} we get
  \begin{equation}
    \begin{aligned}
      \ip{\hmsop[\av]}{\bv} &= -\intd{\bv \cdot \grad \phim[\av] }, \\
      &= - \intb{\phim[\av] (\bv \cdot \nv)} + \intd{\phim[\av] (\div \bv) }, \\
      &= \intb{ \intd[\magd']{\knl (\nabla' \cdot \av(\xv')) (\bv(\xv) \cdot \nv(\xv))}} \\
      &- \intb{ \intd[\boundd']{\knl (\av(\xv') \cdot \nv(\xv')) (\bv(\xv) \cdot \nv(\xv))}} \\
      &- \intd{ \intd[\magd']{\knl (\nabla' \cdot \av(\xv')) (\div \bv(\xv))}} \\
      &+ \intd{ \intd[\boundd']{\knl (\av(\xv') \cdot \nv(\xv')) (\div \bv(\xv))}}.
    \end{aligned}
  \end{equation}

Changing the order of the integrals gives (since $\av$, $\bv$ and their derivatives are in $\ltwo$)
  \begin{equation}
    \begin{aligned}
      \ip{\hmsop[\av]}{\bv}
      &= \intd[\magd']{ \intb{\knl (\bv(\xv) \cdot \nv(\xv))} (\nabla' \cdot \av(\xv'))} \\
      &- \intd[\boundd']{ \intb{\knl (\bv(\xv) \cdot \nv(\xv))} (\av(\xv') \cdot \nv(\xv'))} \\
      &- \intd[\magd']{ \intd{\knl (\div \bv(\xv))} (\nabla' \cdot \av(\xv'))} \\
      &+ \intd[\boundd']{ \intd{\knl (\div \bv(\xv))} (\av(\xv') \cdot \nv(\xv'))}.
    \end{aligned}
  \end{equation}

  Finally we swap $\xv$ with $\xv'$ (which we can do because $\knl$ is symmetrical in its arguments) and collect terms with the same (outer) integral domain
  \begin{equation}
    \begin{aligned}
      \ip{\hmsop[\av]}{\bv} &= \intd{\phi[\bv] (\div \av)} - \intb{\phi[\bv] (\av \cdot \nv)}, \\
      & = \ip{\hmsop[\bv]}{\av}.
    \end{aligned}
  \end{equation}

\end{proof}

\subsection{Magnetocrystalline anisotropy}

Here we only examine the case of uniaxial anisotropy as it is the most technologically relevant
\begin{equation}
  \hca[\mv] = \kone (\mv \cdot \ev) \ev.
\end{equation}

We can easily see that the operator is symmetrical by writing out the definitions
\begin{equation}
  \begin{aligned}
    \ip{\hca[\av]}{\bv} &= \intd{ \kone (\av \cdot \ev) (\ev \cdot \bv)}, \\
    &= \intd{(\kone(\bv \cdot \ev) \ev) \cdot \av}, \\
    &= \ip{\hca[\bv]}{\av}.
  \end{aligned}
\end{equation}

\section{Effective field and energy}
\label{sec:energy-field-relation}

A useful relationship between the effective field and the energy is:
\begin{equation}
  \begin{aligned}
    \hop[\mv] &=  \lap \mv + \hms + \hca, \\
    -\frac{1}{2} \ip{\mv}{\hop[\mv]} &= - \frac{1}{2} \intd{\mv \cdot
      \hop[\mv]} = \e,
  \end{aligned}
\end{equation}
note that this does not include the applied field energy.

For the magnetostatic field this relationship is obvious from the definition of $\ems$, \cref{eq:nd-e-ms}.
For a uniaxial magnetocrystalline anisotropy effective field the derivation is very simple:
\begin{equation}
  - \frac{1}{2} \intd{\bigb{\kone (\mv \cdot \ev) \ev} \cdot \mv} =  - \frac{\kone}{2}\intd{(\mv \cdot \ev)^2} = \eca.
\end{equation}

For the exchange effective field we use \cref{eq:93} to obtain
\begin{equation}
  -\frac{1}{2} \intd{\mv \cdot \lap  \mv} = -\frac{1}{2} \intb{\mv \cdot \ddn{\mv}} + \frac{1}{2}  \intd{\grad \mv : \grad \mv}.
\end{equation}
Then applying Neumann or periodic boundary conditions on $\mv$ gives the result
\begin{equation}
  -\frac{1}{2} \intd{\mv \cdot \lap \mv} = \frac{1}{2} \intd{(\grad \mv)^2} = \eex.
\end{equation}
% Alternatively, with periodic boundary conditions, the boundary integral is zero due to opposite sides of the boundary having equal $\mv$ and $\dmdn$ except for the opposite sign of $\nv$ (as in \cref{eq:95}).

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "./main"
%%% End:


\documentclass[]{report}

\usepackage{amsmath}
\usepackage{bm}
\usepackage{graphicx}

\graphicspath{ {images/} }

\title{CSCI 567 HW \# 3}
\author{Mohmmad Suhail Ansari \\ USC ID: 8518586692\\e-mail: mohmmada@usc.edu}

\begin{document}

\maketitle

\paragraph{Sol. 1.1}
	Given,
	\[ E[\beta] = \frac{1}{n} \sum_{i = 1}^{N} {(y_i - x_i^T \beta)}^2 + \lambda {\|\beta\|}_2^2\]

	We can write from definition of X
	\[ \sum^N (y_i - x_i^T \beta)^2 = {\| (\textbf{y} - X \beta)\|}^2 = (\textbf{y} - X \beta)^T (\textbf{y} - X \beta)\]

	substituting in the first equation

	\[ E[\beta] = \frac{1}{n} (\textbf{y} - X \beta)^T (\textbf{y} - X \beta) + \lambda \beta^T \beta \]

	taking partial derivative w.r.t. $\beta$, we get

	\[ \frac{\partial{E[\beta]}}{\partial{\beta}} = \frac{2}{n} [-X^T \textbf{y} + X^T X \beta] + 2 \lambda \beta = 0\]
	\[ = -X^T \textbf{y} + X^T X \beta + n \lambda \beta \]
	\[ X^T \textbf{y} = (X^T X + n \lambda I_D) \beta \]
	\[ \hat{\beta}_\lambda = (X^T X + n \lambda I_D)^{-1} X^T \textbf{y} \]

	Since, $\epsilon$ is normally distributed, we have
	\[ y = \mathcal{N}(X^T \beta^*, \delta^2 I)\]
	and since $\hat{y} = X^T \hat{\beta}_\lambda$, we have
	\[ \hat{\beta}_\lambda = \mathcal{N}((X^T X + n \lambda I_D)^{-1} X^T X^T \beta^*, (X^T X + n \lambda I_D)^{-1} X^T \delta^2 (X^T X + n \lambda I_D)^{-1} )\]

\paragraph{Sol. 1.2}

	From 1.2, let $L = X^T X$, then if we replace it in the $\hat{\beta}_\lambda$, we get
	\[ \hat{\beta}_\lambda = (L + n \lambda I) X^T y \]
	\[ = (L(I + n\lambda L^{-1}))^{-1} L [L^{-1} X^T y] \]
	\[ = (I + n \lambda L^{-1})^{-1} L^{-1} L [(X^T X)^{-1} X^T y]\]
	and since, $(X^T X)^{-1} X^T y = \beta^*$, we get
	\[ \hat{\beta}_\lambda = (I + n\lambda L^{-1})^{-1} \beta^* \]

	Now,
	\[ E[ x^T \hat{\beta}_\lambda ] - x^T \beta^* = X^T (I + n\lambda L^{-1})^{-1} \beta^* - X^T \beta^* \]
	\[ = X^T (I + n\lambda (X^T X)^{-1})^{-1} - I) \beta^* \]

\paragraph{Sol. 1.3}

	N/A

\paragraph{Sol. 1.4}

	N/A

\paragraph{Sol. 2.1}
	using Mercer's theorem we can write for any function $f(\cdot)$
	\[ \int_{x, x'} k_3 (x, x') f(x) f(x') dx dx' \]
	\[ = \int_{x, x'} (a_1 k_1 (x, x') + a_2 k_2 (x, x') ) f(x) f(x') dx dx' \]
	\[ = \int_{x, x'} a_1 k_1 (x, x') f(x) f(x') dx dx' + \int_{x, x'} a_2 k_2 (x, x') f(x) f(x') dx dx' \]
	and since, we know that $k_1 (\cdot, \cdot)$ and $k_2 (\cdot, \cdot)$ are valid kernel functions, then
	\[ \int_{x, x'} k_3 (x, x') f(x) f(x') dx dx' \geq 0 \]
	and hence, $k_3 (\cdot, \cdot)$ is a valid kernel function.

\paragraph{Sol. 2.2}

	For a feature map $\phi$ we know that a valid kernel is defined as
	\[ k(x, x') = \langle \phi(x), \phi(x') \rangle \]
	so we if we use the feature map $\phi: x \rightarrow f(x)$, then
	\[ k_4(x, x')  = {\langle \phi(x), \phi(x') \rangle}_{\chi^\phi} \]
	\[ = f(x) f(x') \]
	because $f(x)$ is real valued and hence, $k_4(x, x') = f(x) f(x')$ is a valid kernel.

	%if $f(\cdot)$ is a real valued function then using Mercer's theorem we can write
	%\[ \int_{x, x'}  k_4 (x, x') f(x) f(x') dx dx' \]
	%\[ = \int_{x, x'} f(x) f(x') f(x) f(x') dx dx' \]
	%\[ = \int_{x, x'} (f(x))^2 (f(x'))^2 dx dx' \]
	%and since $(f(\cdot))^2 \geq 0$, therefore
	%\[ \int_{x, x'}  k_4 (x, x') f(x) f(x') dx dx' \geq 0 \]
	%and hence, $k_4$ is a valid kernel function.

\paragraph{Sol. 2.3}
	Let, $\phi_1$ and $\phi_2$ be feature map of $k_1$ and $k_2$ respectively and let $f_i (x)$ and $g_i (x)$ be $i^{th}$ feature value for
	$\phi_1$ and $\phi_2$ respectively, then by definition of $k_5$, we can write
	\[ k_5 (x, x') = k_1 (x, x') k_2 (x, x') \]
	\[ = {\phi_1 (x)}^T \phi_1 (x') {\phi_2 (x)}^T \phi(x') \]
	\[ = (\sum_{i} f_i (x) f_i (x')) (\sum_{j} g_j (x) g_j (x')) \]
	\[ = \sum_i \sum_j (f_i (x) g_j (x)) (f_j (x') g_j (x'))\]
	so now, we can define a feature map $\phi_3$ where its $i^{th}$ feature is defined as $h_{i, j}(x) = f_i (x) g_j (x)$. Which gives us
	\[ k_5 (x, x') = {\phi_3 (x)}^T \phi_3 (x')\]
	Also, $k_5$ is symmetric since both $k_1$ and $k_2$ are symmetric, i.e.
	\[ k_5 (x, x') = k_1 (x, x') k_2 (x, x') \]
	\[ = k_1 (x', x) k_2 (x', x) \]
	\[ = k_5 (x', x) \]
	therefore $k_5$ is a valid kernel function.

\paragraph{Sol. 3.1}
	From the text, we can write the objective function as
	\[ \epsilon(w) = \sum_n (y_i - w^T x_i)^2 + \lambda {\| w \|}_2^2 \]
	we know that
	\begin{align}
    	w^T x_i &= \begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_D
         \end{bmatrix} \begin{bmatrix}x_{i1} x_{i2} \cdots x_{iD} \end{bmatrix}
  	\end{align}
  	\[ = \sum_j^D w_j x_{ij} = \sum_j^D x_{ij} w_j \]
  	\[ = x_i^T w \]

  	Now, if $\textbf{y} = [y_1 \, y_2 \, y_3 \cdots \, y_N]$ and
  	\begin{align}
    	X &= \begin{bmatrix}
           x_{11} & x_{21} \cdots x_{N1}\\
           x_{12} & x_{22} \cdots x_{N2}\\
           x_{13} & x_{23} \cdots x_{N3}\\
           \vdots & \vdots  \\
           x_{1D} & x_{2D} \cdots x_{ND}
         \end{bmatrix}
  	\end{align}

  	we write
  	\[ \sum_n (y_i - w^T x_i)^2 = \| \textbf{y} - Xw \|_2^2 \]

  	then we have
  	\[ \epsilon(w) = \|\textbf{y} - Xw\|_2^2 + \lambda \| w\|_2^2 \]

	taking partial derivative w.r.t $w$, we get
	\[ \frac{\partial{\epsilon(w)}}{\partial{w}} = 2 X^T (\textbf{y} - Xw) + 2\lambda w = 0\]
	%%now, we can write $w^T x_i = x_i^T w$,
	%%\[ = -\sum_n (y_i x_i) + (\sum_n x_i^T x_i) w + \lambda w = 0 \]
	%%we can write $\sum_n (y_i x_i) = yX = X^T y$ and $\sum_n x_i^T x_i = X^T X$
	%%\[ = -X^T \textbf{y} + X^T X w + \lambda w = 0 \]
	%%\[ = -X^T \textbf{y} + (X^T X + \lambda I_D ) w = 0\]
	\begin{equation}
		 w^* = {(X^T X + \lambda I_D)}^{-1} X^T \textbf{y}
	\end{equation}

\paragraph{Sol. 3.2}
	we know
	\[ w^* =  {(X^T X + \lambda I_D)}^{-1} X^T \textbf{y} \]

	also,
	\[ (X^T X + \lambda I_D) X^T = X^T X X^T + \lambda X^T = X^T (X X^T + \lambda I_N)\]
	multiplying both sides with $(X^T X + \lambda I_D)^{-1}$ on left and $(X X^T + \lambda I_N)^{-1} \textbf{y}$ on right, we get
	\[ X^T (X X^T + \lambda I_N)^{-1} \textbf{y} = (X^T X + \lambda I_D)^{-1} X^T \textbf{y} \]

	using the above equation in $w^{*}$, we get
	\[ w^{*} = X^T (X X^T + \lambda I_N)^{-1} \textbf{y}\]

	Now, if we substitute $X$ with $\mathbf{\Phi} (x)$ (because dimensions of $XX^T$ and $\mathbf{\Phi}(x) {\mathbf{\Phi}(x)}^T$ is $N \times N$)
	therefore we can write
	\[ w^* = \mathbf{\Phi}^T {(\mathbf{\Phi} \mathbf{\Phi}^T + \lambda I_N)}^{-1} \textbf{y} \]

\paragraph{Sol. 3.3}
	\[ \hat{y} = w^{*T}\phi(x)\]
	\[ = {[\mathbf{\Phi}^T {(\mathbf{\Phi} \mathbf{\Phi}^T + \lambda I_N)}^{-1} \textbf{y}]}^{T} \phi(x)\]
	\[ = y^T {((\mathbf{\Phi} \mathbf{\Phi}^T + \lambda I_N)^T)}^{-1} \mathbf{\Phi} \phi(x) \]
	\[ = y^T {(\mathbf{\Phi}^T \mathbf{\Phi} + \lambda I_N)}^{-1} \mathbf{\Phi} \phi(x) \]

	we also have,
	\begin{align}
    	\mathbf{\Phi} \phi(x) &= \begin{bmatrix}
           \phi(x_1)^T \phi(x) \\
           \phi(x_2)^T \phi(x) \\
           \phi(x_3)^T \phi(x) \\
           \vdots  \\
           \phi(x_N)^T \phi(x)
         \end{bmatrix} = \kappa(x)
  	\end{align}
	substituting value of $K$ and $\kappa(x)$ in $\hat{y}$, we get
	\[ \hat{y} = \textbf{y}^T (K + \lambda I_N)^{-1} \kappa(x) \]

\paragraph{Sol. 3.4}
	The time complexity for both is $O(N^3)$, but in Ridge Regression there are two instances of matrix multiplication and one inversion, taking
	$O(N^3)$ whereas Kernel Ridge Regression only calculate matrix inverse once.

\paragraph{Sol. 4.1}
	As we can see from the graph below the positive examples are \textbf{not} linearly separable from negative examples.

\paragraph{Sol. 4.2}
	As observed from values of the transformed feature space a hyper plane in the dimension of $(x_1 x_2)$ feature separates the points, therefore we have
	\[ w = [0 \quad 0 \quad 0 \quad 1]\]

\paragraph{Sol. 4.3}
	Adding (0.5, 0.5) as a negative example will prevent the positive and negative examples to no longer be linearly separable.

\paragraph{Sol. 4.4}
	This feature transformation corresponds to the the $K(x, x')$ given as
	\[ K(x, x') = 1 + x^2 + {x'}^2 + (x x' )^2\]

\paragraph{Sol. 5} $\newline$

	\textbf{(1)} Since, $C \rightarrow \infty$, the penalty for misclassification is really high, i.e. the decision will perfectly divide the points. However, since the points are linearly inseparable the decision boundary is indeterminable (or at least it is not linear).
	\\

	\textbf{(2)}
	For $C \approx 0$, means the misclassification penalty is small and the decision boundary would mis-classify some points. But a linear can be found (shown in figure below)
	\\

	\textbf{(3)}
	The case with $C \approx 0$ will perform better than case with $C \rightarrow \infty$, since a large value of C will choose a smaller-margin hyperplane compared to small value of C, so small-margin hyperplane might not generalize the vectors enough, i.e. the accuracy for $C \rightarrow \infty$ will be high for training data, but it might perform well for test (unseen) data, whereas the $C \approx 0$ will generalize the vectors enough to perform well for test data.
	\\

	\textbf{(4)}
	Since, the data is already linearly inseparable and and for large $C$ there needs to be a decision boundary which should separate points with minimal misclassification so any point in or near the cluster or ``$-$'' points will still lead to same decision boundary (shown below).
	\\

	\textbf{(5)}
	A point far from the ``$-$'' cluster will change the decision boundary significantly since the decision boundary should not misclassify (large C) this point ans must include the new point on the correct side of the decision boundary.
	\\

	\begin{center}
		\includegraphics[width=\textwidth]{Problem-5}
	\end{center}

\paragraph{Sol. 6}
\subparagraph{1.a}
	For sample size of n = 10, we have the following plot
	\begin{center}
		\includegraphics[width=\textwidth]{fig_10_MSE}
	\end{center}
	and the bias and variance are

	\begin{center}
		\begin{tabular}{c|c|c}
		\hline
		 $g(x)$     &   $Var[\hat{y}]$ &   ${Bias}^2$ \\
		\hline
		 $g_0 (x)$ &      0           & 0.00835376 \\
		 $g_1 (x)$ &      0 		  & 0.0010114999 \\
		 $g_2 (x)$ &      0.0129696   & 0.0010114995 \\
		 $g_3 (x)$ &      0.0129735   & 0.0010114991 \\
		 $g_4 (x)$ &      0.0129809   & 0.0010114983 \\
		 $g_5 (x)$ &      0.0129812   & 0.0010114976 \\
		\hline
		\end{tabular}
	\end{center}

\subparagraph{1.b}
	For sample size of n = 100, we have the following plot
	\begin{center}
		\includegraphics[width=\textwidth]{fig_100_MSE}
	\end{center}

	and the bias and variance are

	\begin{center}
		\begin{tabular}{c|c|c}
			\hline
			 $g(x)$     &   $Var[\hat{y}]$ &   ${Bias}^2$ \\
			\hline
			 $g_0 (x)$ &      0           & 0.0100663   \\
			 $g_1 (x)$ &      0  		  & 0.00020746359 \\
			 $g_2 (x)$ &      0.0114071   & 0.00020746354 \\
			 $g_3 (x)$ &      0.011429    & 0.00020746349 \\
			 $g_4 (x)$ &      0.0114578   & 0.00020746346 \\
			 $g_5 (x)$ &      0.011458    & 0.00020746345 \\
			\hline
			\end{tabular}
	\end{center}

\subparagraph{1.c}
	If we plot the ${Bias}^2$ against $Var[\hat{y}]$ we see that with increasing complexity, the ${Bias}^2$ decreases and $Var[\hat{y}]$
	increases.

	For Sample Size (n) = 10
	\begin{center}
		\includegraphics[width=\textwidth]{fig_10_BV}
	\end{center}

	For Sample Size (n) = 100
	\begin{center}
		\includegraphics[width=\textwidth]{fig_100_BV}
	\end{center}

	We can also observe that with increase in sample size, both ${Bias}^2$ and $Var[\hat{y}]$ decreases.

\subparagraph{1.d}
	For each value of $\lambda$ we get the values for $Bias^2$ and $Var[\hat{y}]$ as
	\begin{center}
		\begin{tabular}{c|c|c}
			\hline
			           $\lambda$ &    $Var[\hat{y}]$ &          $Bias^2$ \\
			\hline
			 0.001 & 1.339555842497451 & 1.335753691359456 \\
			 0.003 & 1.339555198601041 & 1.335753691360410 \\
			 0.010 & 1.339547874313111 & 1.335753691371258 \\
			 0.030 & 1.339483487636583 & 1.335753691466629 \\
			 0.1   & 1.338751426233234 & 1.335753692551133 \\
			 0.3   & 1.332342280730491 & 1.335753702058702 \\
			 1.0   & 1.262651053579910 & 1.335753806797648 \\
			\hline
		\end{tabular}
	\end{center}

	We can see from the plots below that $Var[\hat{y}$ decreases with increasing $\lambda$, whereas $Bias^2$ increases with increasing $\lambda$.

	\begin{center}

		\includegraphics[width=\textwidth]{var_lambda}
		\[ Var[\hat{y}] \]
	\end{center}

	\begin{center}

		\includegraphics[width=\textwidth]{bias_lambda}
		\[ Bias^2 \]
	\end{center}

\subparagraph{Use linear SVM in LIBSVM}
	\begin{center}
		\begin{tabular}{l|c|c}
			\hline
			    C     & 3-Fold Cross Validation Accuracy  & Execution Time \\
			    \hline
			    $4^{-6}$     & 55.75\%     & 0m0.687s      \\
			    $4^{-5}$     & 55.75\%     & 0m0.654s      \\
			    $4^{-4}$     & 55.75\%     & 0m0.666s      \\
			    $4^{-3}$     & 71.55\%     & 0m0.711s      \\
			    $4^{-2}$     & 91.55\%     & 0m0.533s      \\
			    $4^{-1}$     & 91.8\%      & 0m0.338s      \\
			    $4^{2}$      & 96.35\%     & 0m0.204s      \\
			    $4^{0}$      & 94.05\%     & 0m0.239s      \\
			    $4^{1}$      & 95.5\%      & 0m0.206s      \\
			\hline
		\end{tabular}
	\end{center}
	Average Training Time : 0.4708s

\newpage

\subparagraph{Use Kernel SVM in LIBSVM}

	\begin{center}
		Polynomial Kernel
			\begin{tabular}{lr|c|c|c}
			\hline
			 C      &   degree & 3-Fold Cross Validation Accuracy   & Execution Time   \\
			\hline
			 $4^{-3}$ &        1 & 55.75\%                             & 0m0.727s         \\
			 $4^{-3}$ &        2 & 55.75\%                             & 0m0.697s         \\
			 $4^{-3}$ &        3 & 55.75\%                             & 0m0.717s         \\
			 $4^{-2}$ &        1 & 90.25\%                             & 0m0.581s         \\
			 $4^{-2}$ &        2 & 89\%                                & 0m0.688s         \\
			 $4^{-2}$ &        3 & 76.7\%                              & 0m0.801s         \\
			 $4^{-1}$ &        1 & 91.2\%                              & 0m0.360s         \\
			 $4^{-1}$ &        2 & 92.05\%                             & 0m0.441s         \\
			 $4^{-1}$ &        3 & 92.05\%                             & 0m0.513s         \\
			 $4^{0}$  &        1 & 92.75\%                             & 0m0.248s         \\
			 $4^{0}$  &        2 & 93.2\%                              & 0m0.292s         \\
			 $4^{0}$  &        3 & 92.7\%                              & 0m0.366s         \\
			 $4^{1}$  &        1 & 94.45\%                             & 0m0.200s         \\
			 $4^{1}$  &        2 & 94.95\%                             & 0m0.217s         \\
			 $4^{1}$  &        3 & 95.2\%                              & 0m0.233s         \\
			 $4^{2}$  &        1 & 94.5\%                              & 0m0.236s         \\
			 $4^{2}$  &        2 & 95.9\%                              & 0m0.191s         \\
			 $4^{2}$  &        3 & 96.15\%                             & 0m0.193s         \\
			 $4^{3}$  &        1 & 94.25\%                             & 0m0.209s         \\
			 $4^{3}$  &        2 & 96.65\%                             & 0m0.182s         \\
			 $4^{3}$  &        3 & 97\%                                & 0m0.195s         \\
			 $4^{4}$  &        1 & 94.45\%                             & 0m0.337s         \\
			 $4^{4}$  &        2 & 97\%                                & 0m0.188s         \\
			 $4^{4}$  &        3 & 97.05\%                             & 0m0.175s         \\
			 $4^{5}$  &        1 & 94.15\%                             & 0m0.544s         \\
			 $4^{5}$  &        2 & 96.75\%                             & 0m0.186s         \\
			 $4^{5}$  &        3 & 96.55\%                             & 0m0.203s         \\
			 $4^{6}$  &        1 & 94.2\%                              & 0m2.278s         \\
			 $4^{6}$  &        2 & 96.95\%                             & 0m0.240s         \\
			 $4^{6}$  &        3 & 96.7\%                              & 0m0.184s         \\
			 $4^{7}$  &        1 & 94.2\%                              & 0m5.259s         \\
			 $4^{7}$  &        2 & 96.7\%                              & 0m0.265s         \\
			 $4^{7}$  &        3 & 96.7\%                              & 0m0.188s         \\
			 \hline
			\end{tabular}
		\end{center}

		Average Training Time : 0.5555s

		\newpage
	\begin{center}
		RBF Kernel
			\begin{tabular}{l|l|c|c}
			\hline
			 C      & $\gamma$   & 3-Fold Cross Validation Accuracy   & Execution Time   \\
			\hline
			 $4^{-3}$ & $4^{-7}$   & 55.75\%                             & 0m0.717s         \\
			 $4^{-3}$ & $4^{-6}$   & 55.75\%                             & 0m0.760s         \\
			 $4^{-3}$ & $4^{-5}$   & 55.75\%                             & 0m0.773s         \\
			 $4^{-3}$ & $4^{-4}$   & 55.75\%                             & 0m0.724s         \\
			 $4^{-3}$ & $4^{-3}$   & 56\%                                & 0m0.800s         \\
			 $4^{-3}$ & $4^{-2}$   & 87.55\%                             & 0m0.658s         \\
			 $4^{-3}$ & $4^{-1}$   & 60.6\%                              & 0m0.704s         \\
			 $4^{-2}$ & $4^{-7}$   & 55.75\%                             & 0m0.676s         \\
			 $4^{-2}$ & $4^{-6}$   & 55.75\%                             & 0m0.749s         \\
			 $4^{-2}$ & $4^{-5}$   & 55.75\%                             & 0m0.883s         \\
			 $4^{-2}$ & $4^{-4}$   & 64.4\%                              & 0m0.738s         \\
			 $4^{-2}$ & $4^{-3}$   & 90.55\%                             & 0m0.862s         \\
			 $4^{-2}$ & $4^{-2}$   & 92.2\%                              & 0m0.415s         \\
			 $4^{-2}$ & $4^{-1}$   & 92.95\%                             & 0m0.595s         \\
			 $4^{-1}$ & $4^{-7}$   & 55.75\%                             & 0m0.673s         \\
			 $4^{-1}$ & $4^{-6}$   & 55.75\%                             & 0m0.739s         \\
			 $4^{-1}$ & $4^{-5}$   & 66.55\%                             & 0m0.741s         \\
			 $4^{-1}$ & $4^{-4}$   & 90.85\%                             & 0m0.890s         \\
			 $4^{-1}$ & $4^{-3}$   & 91.35\%                             & 0m0.370s         \\
			 $4^{-1}$ & $4^{-2}$   & 93.2\%                              & 0m0.292s         \\
			 $4^{-1}$ & $4^{-1}$   & 95.7\%                              & 0m0.391s         \\
			 $4^{0}$  & $4^{-7}$   & 55.75\%                             & 0m0.760s         \\
			 $4^{0}$  & $4^{-6}$   & 67.8\%                              & 0m0.697s         \\
			 $4^{0}$  & $4^{-5}$   & 91.05\%                             & 0m0.524s         \\
			 $4^{0}$  & $4^{-4}$   & 91.35\%                             & 0m0.588s         \\
			 $4^{0}$  & $4^{-3}$   & 93.55\%                             & 0m0.229s         \\
			 $4^{0}$  & $4^{-2}$   & 96.05\%                             & 0m0.207s         \\
			 $4^{0}$  & $4^{-1}$   & 97.15\%                             & 0m0.281s         \\
			 $4^{1}$  & $4^{-7}$   & 68.05\%                             & 0m0.836s         \\
			 $4^{1}$  & $4^{-6}$   & 90.9\%                              & 0m0.670s         \\
			 $4^{1}$  & $4^{-5}$   & 91.4\%                              & 0m0.391s         \\
			 $4^{1}$  & $4^{-4}$   & 93.3\%                              & 0m0.300s         \\
			 $4^{1}$  & $4^{-3}$   & 94.85\%                             & 0m0.185s         \\
			 $4^{1}$  & $4^{-2}$   & 96.8\%                              & 0m0.165s         \\
			 $4^{1}$  & $4^{-1}$   & 96.95\%                             & 0m0.263s         \\
			 $4^{2}$  & $4^{-7}$   & 90.9\%                              & 0m1.207s         \\
			 $4^{2}$  & $4^{-6}$   & 91.5\%                              & 0m0.537s         \\
			 $4^{2}$  & $4^{-5}$   & 93.35\%                             & 0m0.237s         \\
			 $4^{2}$  & $4^{-4}$   & 94.6\%                              & 0m0.205s         \\
			 $4^{2}$  & $4^{-3}$   & 96\%                                & 0m0.182s         \\
			 $4^{2}$  & $4^{-2}$   & 97.15\%                             & 0m0.163s         \\
			 $4^{2}$  & $4^{-1}$   & 96.7\%                              & 0m0.263s         \\
			 \hline
			\end{tabular}

			Continued... \newline

			\begin{tabular}{l|l|c|c}
			\hline
			 C      & $\gamma$   & 3-Fold Cross Validation Accuracy   & Execution Time   \\
			\hline
			 $4^{3}$  & $4^{-7}$   & 91.5\%                              & 0m0.429s         \\
			 $4^{3}$  & $4^{-6}$   & 93.45\%                             & 0m0.335s         \\
			 $4^{3}$  & $4^{-5}$   & 94.45\%                             & 0m0.278s         \\
			 $4^{3}$  & $4^{-4}$   & 94.65\%                             & 0m0.194s         \\
			 $4^{3}$  & $4^{-3}$   & 96.7\%                              & 0m0.168s         \\
			 $4^{3}$  & $4^{-2}$   & 97\%                                & 0m0.167s         \\
			 $4^{3}$  & $4^{-1}$   & 96.7\%                              & 0m0.249s         \\
			 $4^{4}$  & $4^{-7}$   & 93.45\%                             & 0m0.277s         \\
			 $4^{4}$  & $4^{-6}$   & 94.45\%                             & 0m0.266s         \\
			 $4^{4}$  & $4^{-5}$   & 94.5\%                              & 0m0.260s         \\
			 $4^{4}$  & $4^{-4}$   & 95.65\%                             & 0m0.196s         \\
			 $4^{4}$  & $4^{-3}$   & 97.3\%                              & 0m0.174s         \\
			 $4^{4}$  & $4^{-2}$   & 96.85\%                             & 0m0.180s         \\
			 $4^{4}$  & $4^{-1}$   & 96.7\%                              & 0m0.242s         \\
			 $4^{5}$  & $4^{-7}$   & 94.45\%                             & 0m0.236s         \\
			 $4^{5}$  & $4^{-6}$   & 94.5\%                              & 0m0.262s         \\
			 $4^{5}$  & $4^{-5}$   & 94.55\%                             & 0m0.297s         \\
			 $4^{5}$  & $4^{-4}$   & 96.7\%                              & 0m0.232s         \\
			 $4^{5}$  & $4^{-3}$   & 96.9\%                              & 0m0.203s         \\
			 $4^{5}$  & $4^{-2}$   & 96.85\%                             & 0m0.169s         \\
			 $4^{5}$  & $4^{-1}$   & 96.7\%                              & 0m0.249s         \\
			 $4^{6}$  & $4^{-7}$   & 94.45\%                             & 0m0.199s         \\
			 $4^{6}$  & $4^{-6}$   & 94.75\%                             & 0m0.191s         \\
			 $4^{6}$  & $4^{-5}$   & 95.65\%                             & 0m0.434s         \\
			 $4^{6}$  & $4^{-4}$   & 97.3\%                              & 0m0.311s         \\
			 $4^{6}$  & $4^{-3}$   & 97\%                                & 0m0.225s         \\
			 $4^{6}$  & $4^{-2}$   & 96.85\%                             & 0m0.166s         \\
			 $4^{6}$  & $4^{-1}$   & 96.7\%                              & 0m0.260s         \\
			 $4^{7}$  & $4^{-7}$   & 94.6\%                              & 0m0.264s         \\
			 $4^{7}$  & $4^{-6}$   & 94.45\%                             & 0m0.277s         \\
			 $4^{7}$  & $4^{-5}$   & 96.5\%                              & 0m0.484s         \\
			 $4^{7}$  & $4^{-4}$   & 96.85\%                             & 0m0.422s         \\
			 $4^{7}$  & $4^{-3}$   & 97\%                                & 0m0.225s         \\
			 $4^{7}$  & $4^{-2}$   & 96.85\%                             & 0m0.156s         \\
			 $4^{7}$  & $4^{-1}$   & 96.7\%                              & 0m0.238s         \\
			 \hline
			\end{tabular}

	\end{center}

	Average Training Time : 0.415s \newline

	We find that the best 3-Fold Cross Validation accuracy we can achieve it using an RBF Kernel with parameters $C = 4^6$ and $\gamma = 4^{-4}$.
	After running the libsvm with appropriate parameters we get the classification accuracy = 49.85\%.

\end{document}


\section{Rational and Periodic Functions}
\subsection{Rational Functions}
We want to study meromorphic functions in the Riemann sphere, which are simply the rational functions.
\begin{proposition}
    Every meromorphic function $f:\mathbb C_\infty\to\mathbb C_\infty$ on the Riemann sphere is a rational function, that is we can write
    $$f(z)=c\frac{(z-a_1)\cdots (z-a_m)}{(z-b_1)\cdots (z-b_n)}$$
    for $c,a_i,b_i\in\mathbb C$.
\end{proposition}
Given an $f$ in this form, we can assume WLOG that $a_i\neq b_j$ for all $i,j$.
\begin{proof}
    The case that $f$ is constant is trivial.
    Replacing $f\to 1/f$ necessary, then we can assume WLOG that $f(\infty)\neq\infty$.
    Let $b_1,\ldots,b_{n'}$ be poles of $f$ (there are finitely many as the Riemann sphere is compact).
    Then by assumption $b_i\neq\infty$ for all $i$.
    The Laurent series about $b_j$ then has the form
    $$f(z)=\sum_{l=-k_j}^\infty c_{j,l}(z-b_j)^l,c_{-k_j,j}\neq 0$$
    then $k_j$ is the order of the pole.
    Let
    $$Q_j(z)=\sum_{l=-k_j}^{-1} c_{j,l}(z-b_j)^l$$
    be the principal part of the series and let
    $$g(z)=f(z)-\sum_{j=1}^{n'}Q_j(z)$$
    Then $g$ is entire and has a removable singularity at $\infty$, hence it has to be constant, which means $f$ has to be rational.
\end{proof}
\begin{remark}
    In th proof, $f(\infty)\in\mathbb C$ means $m\le n$, and in this case we have $\deg f=\sum_jk_j=n$, therefore in general the degree of a rational function is $\max\{m,n\}$.
\end{remark}


\section{Optimization}

\subsection{Motivation}

\begin{frame}
  \frametitle{Motivation}

  \begin{itemize}
    \item Optimization is crucial for many solutions in pattern recognition, \\
      pattern analysis, machine learning, artificial intelligence, etc. \\[.25cm]
    \item Optimization has many faces:
      \begin{itemize}
        \item discrete optimization,
        \item combinatorial optimization,
        \item genetic algorithms,
        \item gradient descent,
        \item unconstrained and constrained optimization,
        \item linear programming,
        \item convex optimization, etc. \\[.25cm]
      \end{itemize}
    \item There is no lecture on pattern recognition without a \\
      refresher course on optimization techniques. \\[.25cm]
    \item Each researcher has his own favorite optimization algorithm.
  \end{itemize}
\end{frame}

\subsection{Unconstrained Optimization}

\begin{frame}
  \frametitle{Convexity}

  \begin{citeblock}{Definition}

    A function $f: \real^d \rightarrow \real$ is \structure{\emph{convex}} if the domain $\mathsf{dom}(f)$ of $f$ is a convex set \\
    and if $\forall \vec x, \vec y \in \mathsf{dom}(f)$, and $\theta$ with $0 \leq \theta \leq 1$, we have

    \begin{displaymath}
      f(\theta \vec x + (1-\theta) \vec y) \leq \theta f(\vec x) + (1-\theta) f(\vec y)
    \end{displaymath}
    \pause

    A function $f: \real^d \rightarrow \real$ is \structure{\emph{concave}} if $-f$ is convex.

  \end{citeblock}
  \pause
  \vspace{0.5cm}

  \structure{Geometric interpretation:} \\[.25cm]

  The line segment between $(\vec x, f(\vec x))$ and $(\vec y, f(\vec y))$ lies above the graph of $f$.
\end{frame}

\begin{frame}
  \frametitle{Unconstrained Optimization}

  Let us assume in the following that we have to compute the minimum \\
  of a convex function

  \begin{displaymath}
    f: \real^d \rightarrow \real
  \end{displaymath}

  that is twice differentiable. \\[.25cm] \pause

  The unconstrained optimization problem is just the solution of the\\
  minimization problem

  \begin{displaymath}
    \vec x^* = \argmin_{\vec x} f(\vec x)
  \end{displaymath}

  where $\vec x^*$ denotes the optimal point.
\end{frame}

\begin{frame}
  \frametitle{Unconstrained Optimization \cont}

  For this particular family of functions, a necessary and sufficient condition \\
  for the  minumum are the zero-crossings of the function's gradient:

  \begin{displaymath}
    \nabla f(\vec x^*) = \vec 0.
  \end{displaymath}
\end{frame}

\begin{frame}
  \frametitle{Unconstrained Optimization \cont}

  Most methods follow an \structure{iterative scheme}:

  \begin{eqnarray*}
    \mbox{initialization} & & \vec x ^{(0)}\\
    \mbox{iteration step} & & \vec x^{(k+1)}= g(\vec x^{(k)})
  \end{eqnarray*}

  where $g: \real^d\rightarrow \real^d$ is the update function. \\[0.5cm] \pause

  The iterations \structure{terminate}, if

  \begin{displaymath}
    \|\vec x^{(k+1)} - \vec x^{(k)}\| < \epsilon,
  \end{displaymath}

  i.\,e.\ no further significant change.
\end{frame}

\subsection{Descent Methods}

\begin{frame}

  \frametitle{Descent Methods}

  We now consider iteration schemes that produce a sequence of estimates \\
  according to the update function
%
  \begin{displaymath}
    \vec x^{(k+1)} = g(\vec x^{(k)}) = \vec x^{(k)} + t^{(k)}\Delta \vec x^{(k)} .
  \end{displaymath}
%
  where
%
  \begin{eqnarray*}
    \Delta \vec x^{(k)}\in \real ^d: & & \quad \mbox{is the \structure{search direction} in the $k$-th iteration}\\
    t^{(k)}\in \real:                & & \quad \mbox{denotes the \structure{step length} in the $k$-th iteration}
  \end{eqnarray*}
  \pause
%
  and where we expect
%
  \begin{displaymath}
    f( \vec x^{(k+1)} ) < f(\vec x^{(k)}) ~, \quad \mbox{i.\,e.}~~ \nabla f(\vec x^{(k)})^T \Delta \vec x^{(k)} < 0
  \end{displaymath}
%
  except ~~$ \vec x^{(k+1)}=\vec x^{(k)}=\vec x^*$.
\end{frame}

\begin{frame}
  \frametitle{Taylor Approximation}

  For many problems it is always good to know the \structure{second order Taylor approximation}:

  \begin{displaymath}
    f(\vec x + t\cdot \Delta\vec x)
    \approx f(\vec x) + t\cdot \nabla f(\vec x)^T \Delta \vec x +
      \frac{1}{2} t^2\cdot\Delta \vec x^T \nabla^2 f(\vec x)\Delta \vec x
  \end{displaymath}
\end{frame}

\begin{frame}
  \frametitle{Descent Methods \cont}

  \begin{algorithmic}
    \STATE \structure{Input:} function $f$, initial estimate $\vec x^{(0)}$
    \STATE Initialize: $k:=0$
    \REPEAT
      \STATE Select (or compute) descent direction
      \STATE Line search (1-D optimization): $$t^{(k)}= \argmin_{t \geq 0}f(\vec x^{(k)}+t\cdot \Delta\vec x^{(k)})$$
      \STATE Update:
        \begin{displaymath}
          \vec x^{(k+1)} = \vec x^{(k)} + t^{(k)}\Delta \vec x^{(k)} .
        \end{displaymath}
      \STATE $k:=k+1$
    \UNTIL{$\|\vec x^{(k)} - \vec x^{(k-1)}\| < \epsilon$}
    \STATE \structure{Output:} $\vec x^{(k)}$
  \end{algorithmic}
\end{frame}

\subsection{Backtracking Line Search}

\begin{frame}
  \frametitle{Line Search Methods}

  \begin{itemize}
    \item Multivariate optimization in its described form requires a \\
      proper line search method.\\[.25cm]
    \item Exact line search along the straight line $\{\vec x + t\Delta \vec x ~|~ t \geq 0\}$ has to solve
      \begin{displaymath}
        t^* = \argmin_{t\geq 0} f(\vec  x + t\Delta \vec x)
      \end{displaymath}
      and is rarely used. \\[.25cm]
    \item An overview of methods can be found in numerical recipes.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Line Search Methods \cont}

  \structure{Setting $t = 0.25$:}

  \begin{center}
    \resizebox{.7\linewidth}{!}{
        \input{\texfigdir/stepsize_example3.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Line Search Methods \cont}

  \structure{Setting $t=1.9$:}

  \begin{center}
    \resizebox{.7\linewidth}{!}{
      \alt<24->{
        \input{\texfigdir/stepsize_example1_24.pstex_t}
      }{\alt<23>{
        \input{\texfigdir/stepsize_example1_23.pstex_t}
      }{\alt<22>{
        \input{\texfigdir/stepsize_example1_22.pstex_t}
      }{\alt<21>{
        \input{\texfigdir/stepsize_example1_21.pstex_t}
      }{\alt<20>{
        \input{\texfigdir/stepsize_example1_20.pstex_t}
      }{\alt<19>{
        \input{\texfigdir/stepsize_example1_19.pstex_t}
      }{\alt<18>{
        \input{\texfigdir/stepsize_example1_18.pstex_t}
      }{\alt<17>{
        \input{\texfigdir/stepsize_example1_17.pstex_t}
      }{\alt<16>{
        \input{\texfigdir/stepsize_example1_16.pstex_t}
      }{\alt<15>{
        \input{\texfigdir/stepsize_example1_15.pstex_t}
      }{\alt<14>{
        \input{\texfigdir/stepsize_example1_14.pstex_t}
      }{\alt<13>{
        \input{\texfigdir/stepsize_example1_13.pstex_t}
      }{\alt<12>{
        \input{\texfigdir/stepsize_example1_12.pstex_t}
      }{\alt<11>{
        \input{\texfigdir/stepsize_example1_11.pstex_t}
      }{\alt<10>{
        \input{\texfigdir/stepsize_example1_10.pstex_t}
      }{\alt<9>{
        \input{\texfigdir/stepsize_example1_09.pstex_t}
      }{\alt<8>{
        \input{\texfigdir/stepsize_example1_08.pstex_t}
      }{\alt<7>{
        \input{\texfigdir/stepsize_example1_07.pstex_t}
      }{\alt<6>{
        \input{\texfigdir/stepsize_example1_06.pstex_t}
      }{\alt<5>{
        \input{\texfigdir/stepsize_example1_05.pstex_t}
      }{\alt<4>{
        \input{\texfigdir/stepsize_example1_04.pstex_t}
      }{\alt<3>{
        \input{\texfigdir/stepsize_example1_03.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/stepsize_example1_02.pstex_t}
      }{
        \input{\texfigdir/stepsize_example1_01.pstex_t}
      }}}}}}}}}}}}}}}}}}}}}}}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Line Search Methods \cont}

  \structure{Setting $t^{(k+1)}= \frac{1}{2} t^{(k)}$ and starting with $t^{(0)} = 0.5$:}

  \begin{center}
    \resizebox{.7\linewidth}{!}{
        \input{\texfigdir/stepsize_example2.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Backtracking Line Search}

  \begin{center}
    \resizebox{.7\linewidth}{!}{
      \alt<6->{
        \input{\texfigdir/armijo6.pstex_t}
      }{\alt<5>{
        \input{\texfigdir/armijo5.pstex_t}
      }{\alt<4>{
        \input{\texfigdir/armijo4.pstex_t}
      }{\alt<3>{
        \input{\texfigdir/armijo3.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/armijo2.pstex_t}
      }{
        \input{\texfigdir/armijo1.pstex_t}
      }}}}}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Backtracking Line Search \cont}

  The Armijo-Goldstein line search algorithm: \\[0.5cm]

  \begin{algorithmic}
    \STATE \structure{Input:} function $f$, search direction $\Delta\vec x$
    \STATE Initialize: $t:=1$
    \STATE Select: $\alpha\in [0,0.5]$ and $\beta\in [0,1]$.
    \WHILE{$f(\vec x + t\Delta \vec x) > f(\vec x) +  \alpha t \cdot \nabla f(\vec x)^T\Delta\vec x$}
      \STATE $t := \beta t$
    \ENDWHILE
    \STATE \structure{Output:} $t$
  \end{algorithmic}
\end{frame}

\subsection{Gradient Descent Methods}

\begin{frame}
  \frametitle{Gradient Descent Methods}

  A natural choice of the search direction is the \structure{negative gradient}:

  \begin{displaymath}
    \Delta \vec x^{(k)} = -\nabla f(\vec x^{(k)})
  \end{displaymath}
  \vspace{1cm}

  \structure{Rule of thumb:}\\[.15cm]

  The negative gradient is the steepest descent direction.
\end{frame}

\begin{frame}
  \frametitle{Gradient Descent Methods \cont}

  \begin{algorithmic}
    \STATE \structure{Input:} function $f$, initial estimate $\vec x^{(0)}$
    \STATE intialize: $k:=0$
    \REPEAT
      \STATE Set descent direction: $\Delta\vec x^{(k)}=- \nabla f(\vec x^{(k)}) $
      \STATE Line search (1-D optimization): $$t^{(k)}= \argmin_{t \geq 0}f(\vec x^{(k)}+t\cdot \Delta\vec x^{(k)})$$
      \STATE Update:
        \begin{displaymath}
          \vec x^{(k+1)} = \vec x^{(k)} + t^{(k)}\Delta \vec x^{(k)} .
        \end{displaymath}
      \STATE $k:= k+1$
    \UNTIL{$\|\vec x^{(k)} - \vec x^{(k-1)}\|_2 < \epsilon$}
    \STATE \structure{Output:} $\vec x^{(k)}$
  \end{algorithmic}
\end{frame}

\input{nextTime.tex}

\subsection{Steepest Descent Methods}

\subsubsection{Algorithm}

\begin{frame}
  \frametitle{Steepest Descent Methods}

  \structure{(Normalized) steepest descent, what does it mean?} \\[.25cm]

  We search for the unit vector that shows the largest decrease \\
  in the linear approximation of $f$:

  \begin{displaymath}
    \Delta \vec x = \argmin_{\vec u} \{\nabla f(\vec x)^T \vec u ; \ \| \vec u\|_p = 1\}
  \end{displaymath}
  \pause

  \structure{Conclusions:}

  \begin{itemize}
    \item The steepest descent direction depends on the chosen norm.
    \item The negative gradient is not necessarily the best choice for the \\
      search direction.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Steepest Descent Methods \cont}

  We consider now the first order Taylor approximation of $f(\vec x +\vec u)$ \\
  around the selected position $\vec x$:

  \begin{displaymath}
    f(\vec x+\vec u) \approx f(\vec x) + \nabla f(\vec x)^T \vec u.
  \end{displaymath}
  \pause

  \begin{itemize}
    \item Here $ \nabla f(\vec x)^T \vec u$ is the directional derivative at $\vec x$ in direction $\vec u$.
    \item The vector $\vec u$ denotes a descent direction if the inner product \\
      with the gradient vetor is negative, i.\,e.
      \begin{displaymath}
        \nabla f(\vec x)^T \vec u < 0~.
      \end{displaymath}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Steepest Descent Methods \cont}

  \vspace*{-.25cm}
  \begin{algorithmic}
    \STATE \structure{Input:} function $f$, initial estimate $\vec x^{(0)}$, norm $\|.\|$
    \STATE intialize: $k:=0$
    \REPEAT
      \STATE Compute highest descent direction: $$ \Delta \vec x ^{(k)}= \argmin_{\vec u} \{\nabla f(\vec x^{(k)})^T \vec u ; \ \| \vec u\| = 1\}$$
      \STATE Line search (1-D optimization): $$t^{(k)}= \argmin_{t \geq 0}f(\vec x^{(k)}+t\cdot \Delta\vec x^{(k)})$$
      \STATE Update:
        \begin{displaymath}
          \vec x^{(k+1)} = \vec x^{(k)} + t^{(k)}\Delta \vec x^{(k)} .
        \end{displaymath}
      \STATE $k:=k+1$
    \UNTIL{$\|\vec x^{(k)} - \vec x^{(k-1)}\| < \epsilon$}
    \STATE \structure{Output:} $\vec x^{(k)}$
  \end{algorithmic}
\end{frame}

\subsubsection{$L_2$-norm}

\begin{frame}
  \frametitle{$L_2$-Norm}

  The unit ball for the $L_2$-norm:

  \begin{figure}
    \resizebox{.3\linewidth}{!}{
      \alt<4->{
        \input{\texfigdir/unit_ball_gradient_L2_3.pstex_t}
      }{\alt<3>{
        \input{\texfigdir/unit_ball_gradient_L2_2.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/unit_ball_gradient_L2_1.pstex_t}
      }{
        \input{\texfigdir/unit_ball_gradient_L2_0.pstex_t}
      }}}
    }
  \end{figure}

  For the $L_2$-norm the steepest descent direction is the negative gradient:

  \begin{displaymath}
    \Delta \vec x
    = \argmin_{\vec u} \{\nabla f(\vec x)^T \vec u ; \ \| \vec u\|_2 = 1\}
    = -\nabla f(\vec x)
  \end{displaymath}
\end{frame}

\subsubsection{$L_1$-norm}

\begin{frame}
 \frametitle{$L_1$-Norm}

  The unit ball for the $L_1$-norm: \\[.25cm]

  \begin{columns}

    \column{.5\linewidth}
      \centering
      \resizebox{.9\linewidth}{!}{
        \alt<4->{
          \input{\texfigdir/unit_ball_gradient_L1_3.pstex_t}
        }{\alt<3>{
          \input{\texfigdir/unit_ball_gradient_L1_2.pstex_t}
        }{\alt<2>{
          \input{\texfigdir/unit_ball_gradient_L1_1.pstex_t}
        }{
          \input{\texfigdir/unit_ball_gradient_L1_0.pstex_t}
        }}}
      }

      \column{.5\linewidth}
        \centering
        \onslide<5->
        \resizebox{.9\linewidth}{!}{
          \input{\texfigdir/unit_ball_gradient_L1_4.pstex_t}
        }
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{$L_1$-Norm \cont}

  \begin{itemize}
    \item The steepest descent for the $L_1$-norm selects in each iteration \\
      the component of $\nabla f(\vec x)$ with maximum absolute value and then \\
      decreases or increases dependent on the sign of the selected component. \pause
    \item Let $i$ be the index of the gradient component with maximum absolute \\
      value, and let $\vec e_i\in\real^d$ denote the corresponding base vector.\\
      The steepest descent direction is given by:
      \begin{eqnarray*}
        \Delta \vec x &=& \argmin_{\vec u} \{\nabla f(\vec x)^T \vec u ; \ \| \vec u\|_1 = 1\} \\
                      &=& -\mbox{sgn}\left(\frac{\partial}{\partial x_i} f(\vec x)\right)\vec e_i
      \end{eqnarray*}
      \pause
    \item \structure{Note:} Steepest descent using the $L_1$-norm results in the {\em coordinate descent algorithm}.
  \end{itemize}
\end{frame}

\subsubsection{$L_\infty$-norm}

\begin{frame}
  \frametitle{$L_\infty$-Norm}

  The unit ball for the $L_\infty$-norm: \\[.25cm]

  \begin{columns}

    \column{.5\linewidth}
      \centering
      \resizebox{.9\linewidth}{!}{
        \alt<2->{
          \input{\texfigdir/unit_ball_gradient_Linfinity_2.pstex_t}
        }{
          \input{\texfigdir/unit_ball_gradient_Linfinity_1.pstex_t}
        }
      }

      \column{.5\linewidth}
        \centering
        \onslide<3->
        \resizebox{.9\linewidth}{!}{
          \input{\texfigdir/unit_ball_gradient_Linfinity_3.pstex_t}
        }
  \end{columns}
\end{frame}

\subsubsection{Quadratic $L_{\mat P}$-norm}

\begin{frame}
  \frametitle{$L_{\mat P}$-Norm}

  The unit ball for the $L_{\mat P}$-norm:

  \begin{figure}
    \resizebox{.45\linewidth}{!}{
      \alt<3>{
        \input{\texfigdir/unit_ball_gradient_LP_3.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/unit_ball_gradient_LP_2.pstex_t}
      }{
        \input{\texfigdir/unit_ball_gradient_LP_1.pstex_t}
      }}
    }
  \end{figure}
\end{frame}

\begin{frame}
  \frametitle{$L_{\mat P}$-Norm \cont}

  The \structure{steepest descent} for the $L_{\mat P}$-norm is given by:
%
  \begin{eqnarray*}
    \Delta \vec x
      &=& \argmin_{\vec u} \{\nabla f(\vec x)^T \vec u ; \ \| \vec u\|_{\mat P} = 1\} \\ \pause
      &=& \argmin_{\vec u} \{\nabla f(\vec x)^T \vec u ;\  (\vec u^T{\mat P}\vec u)^{\frac{1}{2}} = 1\} \\ \pause
      &=& \argmin_{\vec u} \{\nabla f(\vec x)^T \vec u ;\ \| {\mat P}^{\frac{1}{2}} \vec u\|_2 = 1\}
  \end{eqnarray*}
  \pause
%
  As we did in the LDA-transform, we introduce a transform to get \\
  \structure{spherical data}:
%
  \begin{displaymath}
    \vec u' = {\mat P}^{\frac{1}{2}} \vec u
  \end{displaymath}
  \pause
%
  and thus
%
  \begin{displaymath}
    f(\vec u) = f({\mat P}^{-\frac{1}{2}}\vec u') = f'(\vec u')
  \end{displaymath}
\end{frame}

\begin{frame}
  \frametitle{$L_{\mat P}$-Norm \cont}

  Instead of $f(\vec x)$ we now minimize $f'(\vec x')$ using the $L_2$-norm and \\
  back-transform the result:

  \begin{eqnarray*}
    \Delta \vec x'
      &=& \argmin_{\vec u} \{\nabla f'(\vec x')^T \vec u' ; \ \| \vec u'\|_{2} = 1\} \\[.15cm] \pause
      &=& -\nabla f'(\vec x') \\[.15cm] \pause
      &=& -\mat P^{-\frac{1}{2}}\nabla f({\mat P}^{-\frac{1}{2}}\vec x') \\[.15cm] \pause
      &=& - {\mat P}^{-\frac{1}{2}}\nabla f(\vec x)
  \end{eqnarray*}
\end{frame}

\begin{frame}
  \frametitle{$L_{\mat P}$-Norm \cont}

  Now we get for $\Delta \vec x$:

  \begin{eqnarray*}
    \Delta \vec x
      &=& {\mat P}^{-\frac{1}{2}} \Delta \vec x' \\[.15cm] \pause
      &=& {\mat P}^{-\frac{1}{2}} \left( - {\mat P}^{-\frac{1}{2}}\nabla f(\vec x) \right) \\[.15cm] \pause
      &=& - \mat{P}^{-1} \nabla f(\vec x).
  \end{eqnarray*}
  \pause

  \structure{Conclusion:} The steepest descent for the $L_{\mat P}$-norm is given by

  \begin{displaymath}
     \Delta \vec x = -\mat{P}^{-1} \nabla f(\vec x)~.
  \end{displaymath}
\end{frame}

\subsection{Newton's Method}

\begin{frame}
  \frametitle{Newton's Method}

  \structure{The idea:}

  \begin{itemize}
    \item Select a point.
    \item Compute the minimum of the second order Taylor approximation.
  \end{itemize}

  \begin{center}
    \resizebox{.7\linewidth}{!}{
      \alt<8->{
        \input{\texfigdir/newton-raphson8.pstex_t}
      }{\alt<7>{
        \input{\texfigdir/newton-raphson7.pstex_t}
      }{\alt<6>{
        \input{\texfigdir/newton-raphson6.pstex_t}
      }{\alt<5>{
        \input{\texfigdir/newton-raphson5.pstex_t}
      }{\alt<4>{
        \input{\texfigdir/newton-raphson4.pstex_t}
      }{\alt<3>{
        \input{\texfigdir/newton-raphson3.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/newton-raphson2.pstex_t}
      }{
        \input{\texfigdir/newton-raphson1.pstex_t}
      }}}}}}}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Newton's Method \cont}

  Second order Taylor approximation:
  \begin{displaymath}
    f(\vec x+\Delta \vec x)
    \approx f(\vec x) + \nabla f(\vec x)^T \Delta \vec x + \frac{1}{2} \Delta \vec x^T (\nabla^2 f(\vec x))\Delta \vec x
  \end{displaymath}
  \pause

  Now we select $\Delta \vec x$ such that
  \begin{displaymath}
    \nabla\{  f(\vec x) + \nabla f(\vec x)^T \Delta \vec x + \frac{1}{2}\Delta \vec x^T (\nabla^2 f(\vec x))\Delta \vec x\}
    = 0
  \end{displaymath}
  \pause

  Obviously the gradient is
  \begin{displaymath}
    \nabla f(\vec x)+ \nabla^2 f(\vec x)\Delta \vec x
    = 0
  \end{displaymath}
  \pause

  and thus
  \begin{displaymath}
    \Delta \vec x
    = -(\nabla^2 f(\vec x))^{-1} \nabla f(\vec x)
  \end{displaymath}
\end{frame}

\begin{frame}
  \frametitle{Newton's Method \cont}

  \structure{Conclusion:} \\[.25cm]

  Newton's method is an $\vec x$-dependent steepest descent method regarding \\
  the $L_{\mat P}$-norm, where $\mat P = \nabla^2 f(\vec x)$ is the Hessian.
\end{frame}

\begin{frame}
  \frametitle{Damped Newton's Method}

  \begin{algorithmic}
    \STATE \structure{Input:} function $f$, initial estimate $\vec x^{(0)}$
    \STATE intialize: $k:=0$
    \REPEAT
      \STATE Compute Newton step:     $$ \Delta \vec x^{(k)}= -\nabla^2 f(\vec x^{(k)})^{-1} \nabla f(\vec x^{(k)}) $$
      \STATE Line search (1-D optimization): $$t^{(k)}= \argmin_{t \geq 0}f(\vec x^{(k)}+t\cdot \Delta\vec x^{(k)})$$
      \STATE Update:
        \begin{displaymath}
          \vec x^{(k+1)} = \vec x^{(k)} + t^{(k)}\Delta \vec x^{(k)} .
        \end{displaymath}
    \STATE $k:=k+1$
   \UNTIL{$\|\vec x^{(k)} - \vec x^{(k-1)}\| < \epsilon$}
   \STATE \structure{Output:} $\vec x^{(k)}$
 \end{algorithmic}
\end{frame}

\subsection{Lessons Learned}

\begin{frame}
  \frametitle{Lessons Learned}

  \begin{itemize}
    \item Gradient descent is widely applied. \\[.25cm]
    \item Gradient descent and coordinate descent are special cases of steepest descent methods. \\[.25cm]
    \item Steepest descent method depends on the chosen norm.
  \end{itemize}
\end{frame}

\input{nextTime.tex}

\subsection{Further Readings}

\begin{frame}
  \frametitle{Further Readings}

  This chapter is basically copied from: \\[.25cm]

  \begin{itemize}
    \item S.~Boyd, L.~Vandenberghe: \\
      \structure{Convex Optimization}, \\
      Cambridge University Press, 2004. \\
      \point{\small \url{http://www.stanford.edu/~boyd/cvxbook/}} \\[.25cm]
    \item Jorge Nocedal, Stephen Wright: \\
      \structure{Numerical Optimization}, \\
      Springer, New York, 1999.
  \end{itemize}
\end{frame}

\subsection{Comprehensive Questions}

\begin{frame}
  \frametitle{Comprehensive Questions}

  \begin{itemize}
    \item What is the general formulation for an unconstrained optimization problem? \\[.5cm]
    \item Why do we need a line search in gradient descent approaches? \\[.5cm]
    \item What is the Armijo-Goldstein line search algorithm? \\[.5cm]
    \item What are the steepest descent directions if we apply the $L_\infty$, $L_1$, $L_2$, and $L_{\mat P}$ norm?
  \end{itemize}
\end{frame}


\section*{Exercise 2.1}
\enum{
\item
Since $\text{ran} X\subset\mathbb{N}$, then
\spl{
E[X]=\sum_{x=0}^{\infty}xP[X=x]=\sum_{x=0}^{\infty}\sum_{y=1}^{x}P[X=x].
}
We change the order of summation,
\spl{
    E[X]=\sum_{x=0}^{\infty}\sum_{y=1}^{x}P[X=x]=\sum_{x=0}^{\infty}\sum_{y=x+1}^{\infty}P[X=y]=\sum_{x=0}^{\infty}P[X>x].
}

\item
According to the result we obtain in 1), we perform the similar calculation to $(X,f_X)$ where $X=1,2,\hdots,N-r+1$.
\spl{
    E[X]=\sum_{x=0}^{N-r}P[X>x].
}
In fact, $P[X>x]$ is equal to the probability that the first $x$ balls are all black.

\spl{
    P[X>x]&=P[\text{The first x balls are all black}]\\
    &=\frac{N-r}{N}\frac{N-r-1}{N-1}\hdots\frac{N-r-x+1}{N-x+1}\\
    &=\frac{(N-r)!/(N-r-x)!}{N!/(N-x)!}=\frac{\binom{N-x}{r}}{\binom{N}{r}}\\
    E[X]&=\frac{1}{\binom{N}{r}}\sum_{x=0}^{r'}\binom{N-x}{N-r'}=\frac{1}{\binom{N}{r}}\sum_{x=0}^{r'}\binom{N-r'+x}{N-r'}\\
    &=\frac{\binom{N+1}{N-r'+1}}{\binom{N}{r}}=\frac{\binom{N+1}{r+1}}{\binom{N}{r}}=\frac{N+1}{r+1}.
}
}

\section*{Exercise 2.2}
For $p_0(t)$, when $t=0$ the probability of 0 failure is 1. For $p_x(t)\quad (x\geq1)$, when $t=0$ the probability of x failure is 0.
Hence, the initial conditions are as the follows.
\spl{
    p_0(0)&=1\\
    p_{x}(0)&=0,\quad x\geq1.
}

Since $p_0'=-\lambda p_0$, then $p_0(t)=c\cdot e^{-\lambda t}$.
Also we know that $p_0(0)=1$, thus c=1.
Hence, $p_0(t)=e^{-\lambda t}$.

Assume that $p_x(t)=(\lambda t)^xe^{-\lambda t}/x!$ for all $x\in\mathbb{N}$.

For $x=0$, $p_0(t)=e^{-lambda t}=(\lambda t)^0e^{-\lambda t}/0!$, which follows our assumption.

For $x=n\geq0$, we assume the statement is true. When $x=n+1$,

\spl{
    &p_{n+1}'+\lambda p_{n+1}=\lambda p_n,\\
    &(e^{\lambda t}p_x)'=\lambda e^{\lambda t}p_n,\\
    &p_{n+1}=\frac{\int\lambda e^{\lambda t}p_n dt}{e^{\lambda t}}.
}
\spl{
    p_{n+1}&=\frac{\lambda e^{\lambda t}p_n dt}{e^{\lambda t}}=\frac{\int\lambda e^{\lambda t}(\lambda t)^ne^{-\lambda t}/n! dt}{e^{\lambda t}}\\
    &=\frac{\lambda^{n+1}/n! \int t^n dt}{e^{\lambda t}}=\lambda^{n+1}/n!/(n+1)(t^{n+1}+C)e^{-\lambda t}\\
    &=(\frac{(\lambda t)^{n+1}+(\lambda)^{n+1}C}{(n+1)!})e^{-\lambda t}.
}

To satisfy that $p_{n+1}(0)=0$, $C$ has to be 0. Thus,
\spl{
    p_{n+1}(t)=\frac{(\lambda t)^{n+1}}{(n+1)!}e^{-\lambda t}.
}

Therefore, $p_x(t)=(\lambda t)^xe^{-\lambda t}/x!$ for all $x\in\mathbb{N}$.

\section*{Exercise 2.3}
\spl{
    f(x)=\binom{n}{x}p^x(1-p)^{n-x}=\binom{n}{x}(\frac{k}{n})^x(1-\frac{k}{n})^{n-x}=k^x\frac{\binom{n}{x}}{n^x}((1-\frac{1}{n/k})^{n/k})^k(1-\frac{k}{n})^{-x}
}

Since
\spl{
    &\lim_{n\to\infty}\frac{\binom{n}{x}}{n^x}=\lim_{n\to\infty}\frac{n!}{x!(n-x)!n^x}=\frac{1}{x!},\\
    &\lim_{n\to\infty}((1-\frac{1}{n/k})^{-n/k})^{-k}=e^{-k},\\
    &\lim_{n\to\infty}(1-\frac{k}{n})^{-x}=1.
}

then
\spl{
    \lim_{n\to\infty}f(x)=\frac{k^x}{x!}e^{-k}.
}

\section*{Exercise 2.4}
\enum{
\item
\spl{
    E[V]&=\int_{0}^{\infty}(\frac{2}{\pi})^{1/2}(m/kT)^{3/2}v^3e^{-\frac{m}{kT}v^2/2}dv\\
    &=(\frac{2}{\pi})^{1/2}(m/kT)^{3/2}\int_{0}^{\infty}v^3e^{-\frac{m}{kT}v^2/2}dv\quad(w=v^2)\\
    &=(\frac{2}{\pi})^{1/2}(m/kT)^{3/2}\int_{0}^{\infty}w^{3/2}e^{-\frac{m}{kT}w/2}\frac{1}{2}w^{-1/2}dw\\
    &=(\frac{1}{2\pi})^{1/2}(m/kT)^{3/2}\int_{0}^{\infty}we^{-\frac{m}{kT}w/2}dw\\
    &=(\frac{m}{2kT\pi})^{1/2}\int_{0}^{\infty}\frac{m}{kT}we^{-\frac{m}{kT}w/2}dw\quad(z=-\frac{m}{kT}w/2)\\
}

\spl{
    E[V]=(\frac{8kT}{m\pi})^{1/2}\int_{-\infty}^{0}ze^zdz=(\frac{8kT}{m\pi})^{1/2}.
}

Since $Var[V]=E[V^2]-E[V]^2$, then
\spl{
    E[V^2]&=(\frac{2}{\pi})^{1/2}(m/kT)^{3/2}\int_{0}^{\infty}v^4e^{-\frac{m}{kT}v^2/2}dv\\
    &=-(\frac{2m}{k\pi T})^{1/2}\int_{1}^{0}v^3de^{-\frac{m}{kT}v^2/2}\\
    &=-(\frac{2m}{k\pi T})^{1/2}(v^3e^{-\frac{m}{kT}v^2/2}\bigg|_0^\infty-\int_0^\infty 3v^2e^{-\frac{m}{kT}v^2/2}dv)\\
    &=3(\frac{2m}{k\pi T})^{1/2}\int_0^\infty v^2e^{-\frac{m}{kT}v^2/2}dv\\
    &=3(\frac{2kT}{m\pi})^{1/2}\int_0^\infty e^{-\frac{m}{kT}v^2/2}dv\\
    &=\frac{3kT}{m}.
}

\spl{
    Var[V]=E[V^2]-E[V]^2=\frac{3kT}{m}-\frac{8kT}{m\pi}=\frac{kT}{m}(3-\frac{8}{\pi}).
}

\item
\spl{
    E[E]=E[mV^2/2]=mE[V^2]/2=\frac{3kT}{2}.
}

\item
Since
\spl{
    E&=\varphi(v)=\frac{mv^2}{2},\\
    V&=\varphi^{-1}(\varepsilon)=(\frac{2\varepsilon}{m})^{1/2},
}
then
\spl{
    f_E(\varepsilon)&=f_V(\varphi^{-1}(\varepsilon))\bigg|\frac{d\varphi^{-1}(\varepsilon)}{d\varepsilon}\bigg|=(\frac{1}{2m\varepsilon})^{1/2}(\frac{2}{\pi})^{1/2}(\frac{m}{kT})^{3/2}\frac{2\varepsilon}{m}e^{-\frac{\varepsilon}{kT}}\\
    &=2(\frac{\varepsilon}{\pi k^3T^3})^{1/2}e^{-\frac{\varepsilon}{kT}}.
}

}

\section*{Exercise 2.5}
\spl{
    \Gamma(\frac{2n+1}{2})&=\int_0^\infty t^{(2n-1)/2}e^{-t}dt=(t^{(2n-1)/2}e^{-t})\bigg|_0^\infty+\frac{2n-1}{2}\int_0^\infty t^{(2n-3)/2}e^{-t}dt\\
    &=(\frac{2n-1}{2})\Gamma(\frac{2n-1}{2}).
}

Hence,
\spl{
    \Gamma(\frac{2n+1}{2})=\frac{(2n-1)(2n-3)\hdots(1)}{2\cdot2\hdots2}\Gamma(\frac{1}{2})=\frac{(2n-1)!!}{2^n}\Gamma(\frac{1}{2}).
}

Also we obtain
\spl{
    \Gamma(\frac{1}{2})&=\int_0^\infty t^{-1/2}e^{-t}dt\quad(t=w^2)\\
    &=\int_0^\infty w^{-1}e^{-w^2}2wdw=2\int_0^\infty e^{-w^2}dw=\sqrt{\pi}
}

Therefore,
\spl{
    \Gamma(\frac{2n+1}{2})=\frac{(2n-1)!!}{2^n}\sqrt{\pi}.
}

\section*{Exercise 2.6}
\enum{
\item
Denote that
\spl{
    Z=\frac{X-\mu}{\sigma}=\frac{X-6000}{100}
}
so that $Z\sim N(0,1)$.
Then,
\spl{
    &P[\text{A sample strength is less than 6250 kg/cm$^2$}]\\
    &=P[X<6250]=P[Z<2.50]=\Phi(2.50)=0.9938.
}

\item
\spl{
    P[5800\leq X\leq 5900]&=P[-2.00\leq Z\leq -1.00]\\
    &=\Phi(-1.00)-\Phi(-2.00)\\
    &=0.1587-0.0228\\
    &=0.1359.
}

\item
According to the standard normal distribution table, we know that
\spl{
    \Phi(-1.64)=0.0505,\quad \Phi(-1.65)=0.0495.
}

When $Z=-1.65$, $X=-1.65\times100+6000=5835$kg/cm$^2$.

Hence, the strength of 5835 kg/cm$^2$ exceeds 5\% samples, saying that is exceeded by 95\% samples.
}

\section*{Exercise 2.7}
\enum{
\item
The random variable is a map $X:\ S\to \mathbb{R}$ together with a function $f_X:\ \mathbb{R}\to\mathbb{R}$. Then, the two properties of a continuous random variable will be showed.

For $a<x<b$ where $b>a$, $f(x)=1/(b-a)>0$.
For other conditions, $f(x)=0>=0$.
Hence, $f(x)\geq0$.

Also,
\spl{
    \int_{-\infty}^{\infty}f(x)dx=\int_{a}^{b}1/(b-a)dx=(b-a)/(b-a)=1.
}

Hence, this is a density for a continuous random variable.

\item
The graph is as the following.
\begin{figure}[H]
    \centering
    \includegraphics[height=5.5cm]{images/a}
    \caption{Graph for Exercise 2.7 (2)}
\end{figure}

\item
\spl{
    P[X<=(a+b)/2]=\int_{-\infty}^{(a+b)/2}f(x)dx=\int_{a}^{(a+b)/2}1/(b-a)dx=(b-a)/2/(b-a)=\frac{1}{2}.
}

\item
Since [c,d] and [e,f] are the subintervals of [a,b], then
\spl{
    P[c\leq X\leq d]&=\int_{c}^{d}1/(b-a)dx=(d-c)/(b-a).\\
    P[e\leq X\leq f]&=\int_{e}^{f}1/(b-a)dx=(f-e)/(b-a).
}

Since $d-c=f-e$, then $P[c\leq X\leq d]=P[e\leq X\leq f]$.

\item
\[
    F(x)=\int_{-\infty}^{x}f(t)dt=
    \left\{\begin{aligned}
        &0,\quad x<a\\
        &\frac{x-a}{b-a},\quad a\leq x \leq b\\
        &1,\quad x>b\\
    \end{aligned}\right.
\]

\item
\spl{
    E[X]&=\int_{-\infty}^{\infty}xf(x)dx=\int_{a}^{b}\frac{x}{b-a}dx\\
    &=\frac{x^2}{2(b-a)}\bigg|_a^b=\frac{b^2-a^2}{2(b-a)}=\frac{a+b}{2}.
}
\spl{
    E[X^2]&=\int_{-\infty}^{\infty}x^2f(x)dx=\int_{a}^{b}\frac{x^2}{b-a}dx\\
    &=\frac{x^3}{3(b-a)}\bigg|_a^b=\frac{b^3-a^3}{3(b-a)}=\frac{a^2+ab+b^2}{3}.
}
Hence,
\spl{
    Var[X]&=E[X^2]-E[X]^2=\frac{a^2+ab+b^2}{3}-(\frac{a+b}{2})^2\\
    &=\frac{(b-a)^2}{12}.
}
}