% !TEX root = ./main.tex
\section{Implementing the evolutionary forces}

As hinted at in Fig.~\ref{fig01:moran}(A), the Moran process gives us a simple
recipe for how to encode the different evolutionary forces on the transition
rates $W^\pm(x)$. Recall that since the transition rates in allele frequency
space are the same as the transition rates in number of organisms space, we can
more easily conceptualize the forces in the latter. In other words, the rate to
transition from $x$ to $x + \Delta x$ is by construction equal to the rate to
transition from $n$ to $n + 1$; therefore we can define the transition rates in
the more convenient space of organism number. In the Moran process changes in
population composition happen when one of the organisms in the population chosen
randomly dies and it is immediately replaced by an organism with a different
allele (since replacement with the same allele does not change the population
composition). This means that the general form in which the number of organisms
with allele $A$ can change looks like
\begin{equation}
    W^+(x) = w^+(n) =
    \text{Rate of $A$ dying} \times
    \text{Probability of replacement by $a$},
\end{equation}
and
\begin{equation}
    W^-(x) = w^-(n) =
    \text{Rate of $a$ dying} \times
    \text{Probability of replacement by $A$},
\end{equation}
Having these general forms let us define the rates for different evolutionary
forces.

\subsection{Genetic drift}
\mrm{Further discussion of what genetic drift is?}

Given the intrinsic stochasticity of the Moran process, the easiest evolutionary
force to implement is that of genetic drift. For simplicity, we assume that both
allele types have the same death rate per organism, $\gamma$. For the case where
only genetic drift is changing the population composition, we assume that the
reproduction probability per organism is the same (equivalent to saying both
genotypes have the same fitness), so the probability of an organism that died
being replaced by an organism with the opposite allele is given by the relative
frequency of such allele. Mathematically this means that we can express the rate
with which the number of organisms with allele $A$ increases as
\begin{equation}
    W^+(x) = w^+(n) =
    \overbrace{\gamma \times (N-n)}^{\text{rate of $a$ dying}} \times
    \overbrace{\frac{n}{N}}^
    {\substack{\text{prob. of $A$} \\ \text{replacing}}},
\end{equation}
where the term $\gamma \times (N-n)$ defines the rate at which an organism with
an allele $a$ dies, and the term $n/N$ defines the probability of an organism
with allele $A$ reproducing. Equivalently the rate with which the number of
organisms with allele $a$ decreases takes the form
\begin{equation}
    W^+(x) = w^+(n) =
    \overbrace{\gamma n}^{\substack{\text{rate of $A$}\\ \text{dying}}}
    \overbrace{\frac{(N-n)}{N}}^
    {\substack{\text{prob. of $a$} \\ \text{replacing}}}.
\end{equation}
Given that both rates are equal, we can see that the first term in
Eq.~\ref{eq:pde_x_general} involving $W^+(x) - W^-(x)$ is zero. This means that
there are no deterministic (directional) forces when only genetic drift is
considered, just pure randomness. Substituting the sum of the rates into the
second term of Eq.~\ref{eq:pde_x_general} results in
\begin{equation}
\begin{aligned}
    \frac{\partial P(x, t)}{\partial t} &=\frac{1}{2 N^{2}}
    \frac{\partial^{2}}{\partial x^{2}}[2 \gamma(N-n) n P(x, t)], \\
    &=\frac{\partial^{2}}{\partial x^{2}}
    \left[\frac{1}{2 N^{2}} 2 \gamma\left(\frac{(N-n) n}{N} \right) P(x, t)\right],\\
    &=\frac{\gamma}{N} \frac{\partial^{2}}{\partial x^{2}}[x(1-x) P(x, t)],
\end{aligned}
\end{equation}
where for the last step we substitute the definition of $x\equiv n/N$. We
redefine the time scale to be in units of $\gamma^{-1}$, meaning that the time
units are measured in terms of the mean life expectancy of an organism. This
allows us to write
\begin{equation}
    \frac{\partial P(x, t)}{\partial t} =
    \frac{1}{N} \frac{\partial^{2}}{\partial x^{2}}[x(1-x) P(x, t)],
\end{equation}
the classic Kimura diffusion equation for genetic drift only.

\subsection{Genetic drift plus selection}

Natural selection is intrinsically associated with the concept of fitness. The
phrase ``survival of the fittest'' first used by Darwin guided and still guides
the way that biologists think about the evolution of many organisms. But despite
the fact that fitness is part of the daily jargon of many biologists, it is a
subtle and highly debated concept. After all, what defines the ability of an
organism to survive the challenges that surround them are completely context
dependent. Roughly speaking we can think of fitness as the ability of an
organism, or a population of organisms to survive and reproduce in the given
ecological niche they occupy. The concept of fitness only makes sense when there
is a competition between organisms. The intrinsic growth rate of a completely
homogeneous population is irrelevant since, as we mentioned before, a population
with a single allele growing is not evolving since there are no changes to the
population composition. The term ecology has to be included because fitness is a
result of the interplay between organisms with their environment including all
biotic and abiotic interactions.

It is common both in theory and in experiments to use the relative growth rates
of organisms, i.e. the speed at which they can reproduce and generate
offspring, as a proxy for fitness. This is a convenient approximation both
for experiments and for theory, but one should not lose track of the relevant
context dependence on the fitness. Just because redwoods have an average life
span of 500-700 years and a very low growth rate that doesn't mean they are not
fit. Having said that we will first begin with the simplest form of fitness,
i.e. frequency independent selection. The term frequency independence simply
refers to the assumption that the fitness of a particular allele does not
depend on the relative abundance of such allele. This assumption could break
down for cases such as some pathogenic bacteria that coordinate their attack
via cell-to-cell communication known as quorum sensing.

To implement the effect of different reproductive success for different alleles
we introduce parameters $f_A$ and $f_a$ as the fitness values for allele $A$ and
$a$, respectively. With these parameters in hand we must redefine the
probability of an organism reproducing to replace the one that dies in the Moran
process. The replacement probability for allele $A$ is now given by
\begin{equation}
    \text{Prob. of $A$ replacing} = \frac{f_A n}{f_A n + f_a (N - n)}.
\end{equation}
Likewise for allele $a$ we have
\begin{equation}
    \text{Prob. of $a$ replacing} = \frac{f_a (N - n)}{f_A n + f_a (N - n)}.
\end{equation}
Let us now assume that $f_A \approx (1 + s) f_a$ for a small $s$. This parameter
$s$ is the so-called selection coefficient, which in nature can be of the order
of $10^{-3}$ or less. With these assumptions we can simplify these substitution
probabilities to be
\begin{equation}
    \text{Prob. of $A$ replacing} \approx \frac{n}{N}(1 + s),
\end{equation}
and
\begin{equation}
    \text{Prob. of $a$ replacing} \approx \frac{(N - n)}{N},
\end{equation}
where, after canceling $f_a$ from the numerator and denominator, we assume that
$(1 + s)n + (N - n) \approx N$ since $s \ll 1$. With these updated replacement
probabilities we again compute the population change rates $W^\pm(x)$ as the
product of the rate of certain type of organism dying times the probability of
being replaced by the opposite allele. These rates take the form
\begin{equation}
    W^+(x) =
    \overbrace{\gamma \times (N - n)}^{\text{rate of $a$ dying}} \times
    \overbrace{\frac{n}{N} (1 + s)}^
    {\substack{\text{prob. of $A$ replacing}\\ \text{with fitness difference}}},
\end{equation}
and
\begin{equation}
    W^-(x) = \gamma n \frac{(N - n)}{N},
\end{equation}
With these rates we ca now compute the sum and difference required by
Eq.~\ref{eq:pde_x_general}. The difference of these two rates takes the form
\begin{equation}
    W^+(x) - W^-(x) = \gamma s \frac{n(N-n)}{N}.
\end{equation}
The sum results in
\begin{equation}
    W^+(x) + W^-(x) = \gamma (2 + s) \frac{n(N-n)}{N}.
\end{equation}
Substituting this into Eq.~\ref{eq:pde_x_general} results in
\begin{equation}
    \frac{\partial}{\partial t} P(x, t) =
    -\frac{1}{N}
    \frac{\partial}{\partial x}
    \left[\gamma s \left(\frac{(N-n) n}{N}\right) P(x, t)\right]
    +\frac{1}{2 N^{2}}
    \frac{\partial^{2}}{\partial x^{2}}
    \left[\gamma (2+s)\left(\frac{(N-n) n}{N}\right) P(x, t)\right].
\end{equation}
Simplifying terms and substituting the definition of the allele frequency gives
\begin{equation}
    \frac{\partial}{\partial t} P(x, t) =
    -\gamma_{S} \frac{\partial}{\partial x}[x(1-x) P(x, t)]
    +\frac{\gamma\left(1+\frac{s}{2}\right)}{N}
    \frac{\partial^{2}}{\partial x^{2}}[x(1-x) P(x, t)].
\end{equation}
To obtain the final form we again write the time scale in units of
$\gamma^{-1}$. Furthermore we use the simplification that $s \ll 1$, obtaining
the classic Kimura diffusion equation for selection and drift
\begin{equation}
    \frac{\partial}{\partial t} P(x, t) =
    -\frac{\partial}{\partial x}[s x(1-x) P(x, t)]
    +\frac{1}{N} \frac{\partial^{2}}{\partial x^{2}}[x(1-x) P(x, t)].
\end{equation}

\subsection{Genetic drift plus selection plus mutation}

One of the ingredients for evolution to take place is the constant appearance of
genetic variability. After all, the raw material for evolution to act on is the
appearance of new mutations in the population. While both genetic drift and
selection reduce population diversity, mutation creates more diversity. In the
case of our one-locus two-allele case it can even resurrect alleles that wen
extinct as one organism changes its genetic content. The implementation of this
third force changes the possibilities on how to change the composition of the
population. As depicted in Fig.~\ref{fig01:moran}(A), if mutation is taken into
account, there are two possible substitutions which would modify the allele
frequency: (1) The usual path in which an organism with the opposite allele to
the one that died reproduces and does not mutate when doing so, and (2) the
possibility of an organism of the same allele as the one that died reproducing,
but when doing so, it mutates to the opposite allele. For simplicity we will
assume that the mutation probability from $A$ to $a$, $\mu_{A\rightarrow a}$, is
the same as from $a$ to $A$, $\mu_{a\rightarrow A}$. For simplicity let us
define the transition rate $W^+(x)$ as
\begin{equation}
    W^+(x) = W^+_{(1)}(x) + W^+_{(2)}(x),
\end{equation}
where we break the rate into the two possible paths. The first path in which an
organism of the opposite allele to the one that died replaces it and does not
mutate when doing so takes the form
\begin{equation}
    W^{+}_{(1)}(x) =
    \overbrace{\gamma(N-n)}^{\text{rate of $a$ dying}}\times
    \overbrace{\frac{n}{N}(1+s)}^
    {\substack{\text{prob. of $A$ replacing}\\ \text{with fitness diff.}}}\times
    \overbrace{(1-\mu)}^
    {\substack{\text{prob. of not} \\ \text{mutating}}},
\end{equation}
where the evolutionary forces appear as a product of the rates and probabilities
of each of the steps taking place. For the second path in which the organism
that replaces the one that dies is of the same type, but when it reproduces
there is a mutation to the opposite allele, we have a rate of the form
\begin{equation}
    W^{+}_{(2)}(x) =
    \overbrace{\gamma(N-n)}^{\text{rate of $a$ dying}}\times
    \overbrace{\frac{(N - n)}{N}}^
    {\text{prob. of $a$ replacing}}\times
    \overbrace{\mu}^{\text{prob. of mutating}}.
\end{equation}
Putting these two rates together results in a transition rate $W^{+}(x)$ of the
form
\begin{equation}
    W^{+}(x) =
    \overbrace{\gamma(N-n) \frac{n}{N}(1+s) (1-\mu)}^{\text{path (1)}}
    + \overbrace{\gamma(N-n) \frac{(N-n)}{N} \mu}^{\text{path (2)}}.
\end{equation}
Equivalently, we can write the rate $W^{-}(x)$ as a decomposition of the two
possible paths that lead to population structure changes. Putting these two
rates back together results in
\begin{equation}
    W^{-}(x)=
    \overbrace{\gamma n \frac{(N-n)}{N}(1-\mu)}^{\text{path (1)}}
    +\overbrace{\gamma n \frac{n}{N}(1+s) \mu}^{\text{path (2)}}.
\end{equation}
Again we follow Eq.~\ref{eq:pde_x_general} and compute the sum and the
difference between these rates. After some algebra, we find that the difference
between these rates is of the form
\begin{equation}
    W^+(x) - W^-(x) =
    \frac{\gamma}{N}\left[n s\left(N-n-N\mu\right)+N{\mu}(N-2 n)\right].
\end{equation}
For the sum of the rates we find
\begin{equation}
    W^+(x) + W^-(x) =
    \frac{\gamma}{N}\left[N n(2-4 \mu+s-\mu s) +
    n^{2}(-2+4 \mu-s+2 \mu s)\right].
\end{equation}
Substituting these rates in Eq.~\ref{eq:pde_x_general} gives
\begin{equation}
\begin{split}
    \frac{\partial}{\partial t} P(x, t)=
    &-\frac{1}{N} \frac{\partial}{\partial x}
    \left[ \frac{\gamma}{N} \left(n s \left(N - n - N \mu \right)+
    N \mu (N - 2 n) \right) P(x, t)\right] \\
    & + \frac{1}{2 N^{2}} \frac{\partial^2}{\partial x^{2}}
    \left[\frac{\gamma}{N}\left(N n (2 - 4 \mu+s-\mu s)
    -n^{2}\left(2-4 \mu+s-2\mu s\right)\right) P(x, t)\right].
\end{split}
\end{equation}
Substituting the definition of the allele frequency results in
\begin{equation}
\begin{split}
    \frac{\partial}{\partial t} P(x, t)=
    &-\gamma \frac{\partial}{\partial x}[x s(1-x-\mu)+\mu(1-2 x) P(x, t)] \\
    &+\frac{\gamma}{2 N} \frac{\partial^{2}}{\partial x}
    \left[x(2-4 \mu+s-\mu s)-x^{2}(2-4 \mu+s-2 \mu s) P(x, t)\right].
\end{split}
\end{equation}
To get to the final equation we simply make use of the approximation that both
$s, \mu \ll 1$. Implementing this, and writing the time scale in units of
$\gamma^{-1}$ results in the classic diffusion theory equation with all three
forces implemented
\begin{equation}
    \frac{\partial}{\partial t} P(x, t) =
    -\frac{\partial}{\partial x}[s x(1-x) + \mu (1 - 2x) P(x, t)]
    +\frac{1}{N} \frac{\partial^{2}}{\partial x^{2}}[x(1-x) P(x, t)].
\end{equation}


\section{Dimension reduction}

% ===
\emph{PCA}
\enskip $\normalcolor
\to f\!\!: \mathbb{R}^d\to\mathbb{R}^k, k\!<\!d
\hfill (\lambda_1\!\geq\!...\!\geq\!\lambda_d\!\geq\!0)$

\textbf{centered:} $\mu = \mathbb{E}(X) = \frac{1}{n} \sum_{i=1}^n x_i = 0$\\
\textbf{empir. cov.:} $\Sigma = \frac{1}{n} \sum_{i=1}^n x_ix_i^\top = \sum_{i=1}^d \lambda_i v_iv_i^\top$

$(\hat W, \hat z_1, .., \hat z_n) = \arg\min \sum_{i=1}^n \norm{Wz_i - x_i}_2^2$

\textbf{Sol.:} \highlight*{$\hat z_i = \hat W^\top x_i$}, $\hat W = (v_1\vert..\vert v_k) \in \mathbb{R}^{d\times k}$, orth.

% ===
\emph{Kernel PCA \enskip
\normalfont\sffamily$\normalcolor\to$ non-linear, feature discov.}
\textbf{Ansatz:} {\small see KLR},\enskip
\textbf{Constraint:} $\norm{w}_2 \!\!=\! \alpha^{\!\top}\!K\alpha \!=\! 1$

\textbf{Kernel PC:} $\alpha^{(1)}\!,..,\alpha^{(k)}\!\in\!\mathbb{R}^n$, \enskip
$\alpha^{(i)} \!=\! \frac{1}{\sqrt{\lambda_i}}v_i$,\\
$K = \sum_{i=1}^n \lambda_i v_i v_i^\top$, $\lambda_1\!\geq\!..\!\geq\!\lambda_d\!\geq\!0$\\
\textbf{New point}: \highlight{$\hat{z}_i = \sum_{j=1}^n\alpha_j^{(i)}k(\hat{x}_i,x_j)$}

% ===
\emph{Autoencoders:}
Find identity fct.: $x \approx f(x;\theta)$\\
$f(x;\theta) = f\ped{decode}(f\ped{encode}(x;\theta\ped{enc.});\theta\ped{dec.})$


\section{Mutable Implementation}
We now give the implementation of the operators seen until now with a simple mutable state.

We define our state as that of the state monad (that is a statement that evaluates to a value of type $a$ in a state of type $s$ has the same type of its denotational semantics):
\begin{verbatim}
data ST s a = ST(s->(a,s))
\end{verbatim}

References will be based on the state since a reference must be easily convertible into statements, one for evaluating the reference and one for assigning it:
\begin{verbatim}
type Get s a = ST s a
type Set s a = a -> ST s ()
data Ref s a = Ref (Get s a) (Set s a)
\end{verbatim}

We now need to represent the state (our stack). The simplest implementation of a typed stack is based on heterogeneous lists. A heterogeneous list is build based on two type constructors:
\begin{verbatim}
data Nil = Nil
data Cons h tl = Cons h tl
\end{verbatim}

Since heterogeneous lists do not have a single type, we characterize all heterogeneous lists with an appropriate predicate:
\begin{verbatim}
class HList l
instance HList Nil
instance HList tl => HList (Cons h tl)
\end{verbatim}

We access heterogeneous lists by index. To ensure type safety we define type-level integers, encoded as Church Numerals:
\begin{verbatim}
data Z = Z
data S n = S n

class CNum n
instance CNum Z
instance CNum n => instance CNum (S n)
\end{verbatim}

We can now read the length of a heterogeneous list, as well as get the type of an arbitrary element of the list:

\begin{verbatim}
type family HLength l :: *
type instance HLength Nil = Z
type instance HLength (Cons h tl) = S (HLength tl)

type family HAt l n :: *
type instance HAt (Cons h tl) Z = h
type instance HAt (Cons h tl) (S n) = HAt tl n
\end{verbatim}

We will need a way to manipulate the values of a heterogeneous list. For this reason we define a lookup predicate:
\begin{verbatim}
class (HList l, CNum n) => HLookup l n where
	lookup :: l -> n -> HAt l n
	update :: l -> n -> HAt l n -> l

instance (HList tl) => HLookup (Cons h tl) Z where
	lookup (Cons h tl) _ = h
	update (Cons h tl) _ h’ = (Cons h’ tl)

instance (HList tl, CNum n) => HLookup (Cons h tl) (S n) where
	lookup (Cons _ tl) _ = lookup tl (undefined::n)
	update (Cons h tl) _ v’ = (Cons h’ (update tl (undefined::n) v’))
\end{verbatim}

Now we have all that we need to instance our stack, reference and state predicates.

We begin by instancing the $Stack$ predicate, since all heterogeneous lists are stacks and as such can be used:
\begin{verbatim}
instance Stack Nil where
	type Push Nil a = Cons a Nil
	push = Cons
	pop (Cons h tl) = tl

instance (Stack tl, s ~ Cons h tl) => Stack s where
	type Push s a = Cons a s
	push = Cons
	pop (Cons h tl) = tl
\end{verbatim}

We instance the $Monad$ class with the $ST$ type (as in the state monad):
\begin{verbatim}
instance Monad (ST s) where
	return x = ST(\s -> (x,s))
	(ST st) >>= k = ST(\s -> let (s’,res) = st s in k res s’)
\end{verbatim}

We also define a way to evaluate a statement and ignoring the resulting state:
\begin{verbatim}
runST :: ST s a -> s -> a
runST (ST st) s = snd (st s)
\end{verbatim}

Now that $Monad (ST s)$ is instanced we can instance the $RefSt$ predicate for our references and state:
\begin{verbatim}
instance (HList s, n~HLength s) => RefSt Ref ST s where
	eval (Ref get set) = get
	(Ref get set) := v = set v
	(Ref get set) *= f = do v <- get
							  set (f v)
	new a k =
				let r_new = Ref (ST (\s -> (lookup s (undefined::n), s)))
								  (\v -> ST(\s -> ((), update s (undefined::n) v)))
\end{verbatim}

Thanks to this last instance we can now give a first working example of usage of our references with mutable state:
\begin{verbatim}
ex1 :: ST Nil Int
ex1 = 10 ‘new’ (\(i :: Ref (New Nil Int) Int) ->
	  do i *= (+2)
		 i)

res1 :: Int
res1 = runST ex1 Nil
\end{verbatim}

The result, as expected, is $res1=12$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%EVALUATION STEPS FOR EXAMPLE1%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We complete the implementation of our system so far by adding records. We use as records heterogeneous lists to which we access via labels. A label is defined with a getter and a setter (similar to those found in the $Ref$ constructor) as:
\begin{verbatim}
data Label r a = Label (r->a) (r->a->r)
\end{verbatim}

We can instance the $Record$ predicate:
\begin{verbatim}
instance (Stack s, HList r) => Record r Ref ST s where
	type Label r a = Label r a
	Ref get set <= Label read write =
		Ref(do r <- get
				return read r)
			(\v’-> do r <- get
					   set (write r v’))
\end{verbatim}

To more easily manipulate records we define a function for building labels from $CNum$s:
\begin{verbatim}
labelAt :: (HList l, CNum n, HLookup l n) => l -> n -> Label l (HAt l n)
labelAt _ = Label (\l -> lookup l (undefined::n)) (\l -> update l (undefined::n))
\end{verbatim}

We can now give a second example that shows how records can be manipulated:
\begin{verbatim}
type Person = String ‘Cons’ String ‘Cons’ Int ‘Cons’ Nil
first :: Label Person String
first = labelAt Z
last :: Label Person String
last = labelAt (S Z)
age :: Label Person Int
age = labelAt (S S Z)

mk_person f l a = (f ‘Cons’ l ‘Cons’ a ‘Cons’ Nil)

ex2 :: ST Nil Person
ex2 = (mk_person “John” “Smith” 27) ‘new’ (\(p :: Ref (New Nil Person) Person) ->
			do (p <= last) *= (++ “ Jr.”)
			   (p <= age) := 25
			   pv <- eval p
			   return pv)
res2 :: Person
res2 = runST ex2
\end{verbatim}
The result is, as expected, $“John” ‘Cons’ “Smith Jr.” ‘Cons’ 25 ‘Cons’ Nil$.

We give one last example that does not work even though at a first glance we would expect it to. This example is used to introduce the next session:
\begin{verbatim}
ex3 :: ST Nil Unit
ex3 = 10 ‘new’ (\(i :: Ref (New Nil Int) Int) ->
	  “Hello” ‘new’ (\(s :: Ref (New (New Nil Int) String) String) ->
	  do i *= (+2)
		 s *= (++” World”)
		 return ())
\end{verbatim}

This example does not even compile because:
\begin{verbatim}
i *= (+2) :: ST (New Nil Int) ()
\end{verbatim}

while
\begin{verbatim}
s *= (++” World”) :: ST (New (New Nil Int) String) ()
\end{verbatim}

but the state monad cannot accept a state that varies between statements. It is of course worthy of notice that the above sample, though as it is does not compile, is definitely not nonsensical. Whenever we have a larger state such as
\begin{verbatim}
New (New Nil Int) String
\end{verbatim}

we expect to be able to work with references that expect a smaller state, such as
\begin{verbatim}
New Nil Int
\end{verbatim}

since all that is needed for them to work is contained in the larger state, and through appropriate conversion both reading and writing on the smaller state can be performed on the larger state. The notion we will use to fix this problem happens to be that of coercive subtyping.

Before Giuseppe's PhD/Monads/ObjectiveMonad/MonadicObjects prev/trunk/tex v2/mutable_instantiation.tex


\section{Method}\label{sec: method}

The method for calibration considers the frames of the accelerometer and gyroscope as two non-orthogonal and misaligned frames, as illustrated in Figure \ref{fig:axis}, and estimates a linear transformation from each sensor's frame to a common orthogonal one.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\linewidth]{figures/axis}
	\caption{Accelerometer and Gyroscope axis are non orthogonal and misaligned.}
	\label{fig:axis}
	\source{Reproduced from \cite{2014:Tedaldi}.}
\end{figure}

In other words, let $acc$ and $gyr$ be the accelerometer and gyroscope readings, the method estimates the scale $K_{3\times 3}$, the skew $S_{3\times 3}$ , and the bias $B_{3\times 1}$ for both sensors, and the calibrated measures $acc^*$ and $gyr^*$  are computed as follows:

\begin{equation}
\begin{aligned}
acc^* = K^aS^a(acc - B^a) \\
gyr^* = K^gS^g(gyr - B^g)
\end{aligned}
\end{equation}

\begin{important}
	\begin{enumerate}
		\item The $acc$ and $gyr$ readings given by the IMU are usually in their own scale. The conversion to S.I. is made intrinsically through the scale matrices $S^a$ and $S^g$.
		\item The scale  $K$ is a diagonal matrix (only the $3$ diagonal elements are non-zero),  $S^a$ is a triangle upper matrix ( only the $3$ elements on the upper triangle are non-zero), and $S^g$ is a triangle upper and lower matrix (all the $6$ non-diagonal elements are non-zero).
	\end{enumerate}
\end{important}

The first step is to calculate the calibration parameters for the accelerometer, which is done by optimizing:

\begin{equation}\label{eq: acc opt}
	\argmin_{K^a, S^a, B^a} (||g|| - ||K^aS^a(acc - B^a)||)^2
\end{equation}

Where $g$ is the local gravity, and the $acc$ measures used here belong to the static intervals of the data, which are automatically detected by the tool (more in Section \ref{sec: data collection}).

After the accelerometer is calibrated, the gyroscope is calibrated in two steps.

First, $B^g$ is estimated by averaging the gyroscope measures in the initial static interval (see Section \ref{sec: data collection}).
%
Second, $K^g$ and $S^g$ are estimated by optimizing:

\begin{equation}\label{eq: gyr opt}
	\argmin_{K^g, S^g} \left\|(\tilde{g}_{t_{i+1}}-\tilde{g}_{t_i}) - \int_{t_i}^{t_{i+1}} K^gS^g(gyr - B^g)dt\right\|
\end{equation}

Where $\tilde{g}_{t_i}$ is the gravity versor at time $t_i$, meaning that the term $||\tilde{g}_{t_{i+1}}-\tilde{g}_{t_i})||$ is the angular displacement between to consecutive static intervals, and the integral calculates the angular displacement from the gyroscope readings.

\textbf{Important:}
\begin{important}
	Errors in the accelerometer's calibration will propagate to the gyroscope's calibration giver the later depend on the earlier.
\end{important}


\subsection{Covariant derivative}

Essentially as we move across path, we are changing the basis.

We can look at how basis vector change as we translate

We can define as basis as:

\(e_i=\dfrac{\delta x}{\delta x_i}\)

How to measure transport

If we take a vector and move it around a curved surface and return it to the same point, it may not face the same way

Eg if you're on equator, move east, north, south to equator, you'll face diffrent direction

This is true on smaller movements of a curved surface

We can use this to measure curvature of a manifold without coordinates

\subsection{New}

covariant derivative. how does change in field compare to parallel transport from curent position?

\(\nabla_v (X)=\lim_{t\rightarrow 0}\dfrac{X(p+tv)-X(p)}{t}\)

We have point \(p\). We can compare how field in tangent space varies in direction of \(v\).

we don't define basis as each point, but rather how basis changes as you move along a curve


\paragraph{Two stage sampling procedure}
called Stein's sampling procedure:
\begin{enumerate}
  \item take a preliminary sample of small size $n_0$

  $x_1, \ldots, x_{n_0} \rightarrow X_0 \ \longrightarrow \ s^2_0 \rightarrow s_0$

  \item estimate how many more measurements you need:
  \begin{gather*}
  n = \lceil \left( \diststudentt^{[n_0-1]}_{1-\alpha/2} \frac{s_0}{d} \right) \rceil
  \end{gather*}
  if $n > n_0$ then take the second sample of size $n-n_0$

\end{enumerate}


\subsection{Dual Path Network (DPN)}
The general idea of DPN model \cite{chen2017dual} can be described as
\begin{equation}\label{eq:DPN}
\begin{cases}
f^{1,0} &=R_{\rm max}\circ \sigma \circ \theta^0(f), \\
\text{\bf For} &\ell = 1:J \\
\quad &\text{\bf For} \quad i = 1:\nu_\ell \\
&\tilde{f}^{\ell,i} = \tilde{\mathcal{H}}^{\ell,i} \left(  [f^{\ell,0}, \cdots, f^{\ell,i-1}]  \right),
\quad  \bar{f}^{\ell,i} = \bar{\mathcal{H}}^{\ell,i} \left([f^{\ell,0}, \cdots, f^{\ell,i-1}] \right),\\
&f^{\ell,i} =  \tilde f^{\ell,i} +  \bar f^{\ell,i} \\
\quad &\text{\bf EndFor} \\
&\tilde f^{\ell+1,0} =  \tilde R_\ell^{\ell+1} ( f^{\ell, \nu_\ell} ) ,  \quad \bar f^{\ell+1,0} =  \bar R_\ell^{\ell+1} ( f^{\ell, \nu_\ell} ) ,\\
&f^{\ell,0} =  \tilde f^{\ell,0} +  \bar f^{\ell,0} \\
\text{\bf End} &\\
H_0(f) &=  R_{\rm ave}(f^{L,\nu_\ell}). \\
\end{cases}
\end{equation}

Here, if we take
\begin{equation}
\tilde{\mathcal{H}}^{\ell,i} =  \bar{\mathcal{H}}^{\ell,i},
\end{equation}
with some special forms such as classical CNN or ResNet, this is
similar to the trick in AlexNet.

In DPN, the trick is to take different forms for $\tilde{\mathcal{H}}^{\ell,i} $ and $ \bar{\mathcal{H}}^{\ell,i}$.
For example, they take
\begin{equation}\label{eq:DPN-1}
\tilde{\mathcal{H}}^{\ell,i} \left(  [f^{\ell,0}, \cdots, f^{\ell,i-1}]  \right) = \sigma( f^{\ell,i-1} + \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i}(f^{\ell,i-1})),
\end{equation}
and
\begin{equation}\label{eq:DPN-2}
\bar{\mathcal{H}}^{\ell,i} \left(  [f^{\ell,0}, \cdots, f^{\ell,i-1}]  \right) = \sigma \left( \sum_{j=0}^{i-1} [\theta^{\ell,i}]_{j} f^{\ell,j} \right)
\end{equation}
in original DPN paper.
Here we need to mention that, equation \eqref{eq:DPN-1} mens that $\tilde f^{\ell,i}$ follows the
process of ResNet structure and equation \eqref{eq:DPN-2} mens that $\bar f^{\ell,i}$ follows the
process of DenseNet structure. That is to say, DPN is some special combination of these two models.

\paragraph{The connection to MgNet}
First, we can make a extended version for DPN as:
\begin{equation}\label{eq:DPN}
\begin{cases}
f^{1,0} &=R_{\rm max}\circ \sigma \circ \theta^0(f), \\
\text{\bf For} &\ell = 1:J \\
\quad &\text{\bf For} \quad i = 1:\nu_\ell \\
&\tilde{f}^{\ell,i} = \tilde{\mathcal{H}}^{\ell,i} \left(  f^{\ell,i-1} \right),
\quad  \bar{f}^{\ell,i} = \bar{\mathcal{H}}^{\ell,i} \left( f^{\ell,i-1} \right),\\
&f^{\ell,i} =  \red{[\tilde f^{\ell,i} ,\bar f^{\ell,i}]}\\
\quad &\text{\bf EndFor} \\
&\tilde f^{\ell+1,0} =  \tilde R_\ell^{\ell+1} ( f^{\ell, \nu_\ell} ) ,  \quad \bar f^{\ell+1,0} =  \bar R_\ell^{\ell+1} ( f^{\ell, \nu_\ell} ) ,\\
&f^{\ell,0} =  \tilde f^{\ell,0} +  \bar f^{\ell,0} \\
\text{\bf End} &\\
H_0(f) &=  R_{\rm ave}(f^{L,\nu_\ell}). \\
\end{cases}
\end{equation}
Then, we can take some special form of $ \tilde{\mathcal{H}}$,
$\bar{\mathcal{H}}^{\ell,i}$, $\tilde R_\ell^{\ell+1}$ and $ \bar R_\ell^{\ell+1}$
to connect our MgNet.
\begin{itemize}
\item recover $u^{\ell,i}$ for $i = 1:\nu_\ell$
\begin{equation}
\blue{\tilde f^{\ell,i}} = \tilde{\mathcal{H}}^{\ell,i} \left(  f^{\ell,i-1} \right) = \tilde{\mathcal{H}}^{\ell,i} \left(  [\tilde f^{\ell,i} ,\bar f^{\ell,i}] \right) =
\blue{\tilde f^{\ell,i-1} + B_{\ell,i}  ({\bar f^{\ell,i} -  A^{\ell} (\tilde f^{\ell,i-1})})},
\end{equation}
\item recover $f^{\ell}$
\begin{equation}
\blue{\bar f^{\ell,i}} = \bar{\mathcal{H}}^{\ell,i} \left(  f^{\ell,i-1} \right) = \bar{\mathcal{H}}^{\ell,i} \left(  [\tilde f^{\ell,i} ,\bar f^{\ell,i}] \right) =
\blue{\bar f^{\ell,i-1}} ,
\end{equation}
\item recover $u^{\ell,0}$
\begin{equation}
\blue{\tilde f^{\ell+1,0}} = \tilde f^{\ell+1,0} =  \tilde R_\ell^{\ell+1} ( f^{\ell, \nu_\ell} ) = \tilde f^{\ell+1,0} =
\tilde R_\ell^{\ell+1} ( [\tilde f^{\ell,\nu_\ell} ,\bar f^{\ell,\nu_\ell}]  )  = \blue{\Pi_\ell^{\ell+1} \tilde f^{\ell,\nu_\ell}},
\end{equation}
\item recover $f^{\ell+1}$
\begin{equation}
\begin{aligned}
\blue{\bar f^{\ell+1,0}} =  \bar R_\ell^{\ell+1} ( f^{\ell, \nu_\ell} )
&=  \bar R_\ell^{\ell+1} ( [\tilde f^{\ell,\nu_\ell} ,\bar f^{\ell,\nu_\ell}]  ),  \\
&= \blue{R^{\ell+1}_\ell( \bar f^{\ell,\nu_\ell} - A^\ell(\tilde f^{\ell,\nu_\ell})) + A^{\ell+1} (\Pi_\ell^{\ell+1} \tilde f^{\ell,\nu_\ell})},\\
&= \blue{R^{\ell+1}_\ell( \bar f^{\ell,\nu_\ell} - A^\ell(\tilde f^{\ell,\nu_\ell})) + A^{\ell+1} (\tilde f^{\ell+1,0})}. \\
\end{aligned}
\end{equation}
\end{itemize}
This means that, we can connect the MgNet and DPN with the next relationship:
\begin{equation}
u^{\ell,i} = \tilde f^{\ell,i}, \quad i = 0:\nu_\ell, \quad \ell = 1:J,
\end{equation}
and
\begin{equation}
f^{\ell} = \bar f^{\ell,i},  \quad i = 0:\nu_\ell, \quad \ell = 1:J.
\end{equation}

Thus to say, we may give a better explanation for DPN from the viewpoint of MgNet.
Furthermore, we give some further developments version of DPN by involving more
structure from MgNet like what we are doing for ResNet and iResNet.


\lab{Total Variation and Image Processing}{Total Variation and Image Processing}
\label{lab:tv_images}
\objective{Minimizing an energy functional is equivalent to solving the resulting Euler-Lagrange equations.  We introduce the method of steepest descent to solve these equations, and apply this technique to a denoising problem in image processing.}

% \begin{enumerate}
% \item Algorithm assumes smoothness.
% Other theorems/algorithms have been developed using convex analysis, due to the recognition that the image might not be best represented by a smooth function $u:[0,1]\times [0,1] \to \mathbb{R}$.
% \item Analyze efficiency.
% Code with cython/numexpr?
% \end{enumerate}

\section*{The Gradient Descent method}
Consider an energy functional $J[u]$, defined over a collection of admissible functions $u:\Omega \subset \mathbb{R}^n \to \mathbb{R}$, with the form
\[J[u] = \int_{\Omega} L(x,u,\nabla u) \, dx\]
where $L = L(x,u,\nabla u)$ is a function $\mathbb{R}^n \times \mathbb{R} \times \mathbb{R}^n \to \mathbb{R}$.
A standard result from the calculus of variations states that a minimizing function $u^*$ satisfies the Euler-Lagrange equation
\begin{align}
L_u-\sum_{i=1}^n\frac{\partial L_{u_{x_i}}}{\partial x_i}=L_u-\nabla\cdot L_{\nabla u}=L_u - {\rm div}\,(L_{\nabla u}) &= 0.    \label{EL:multivar_domain}
\end{align}
where $L_{\nabla u} = \nabla^\prime L = [L_{x_1},\hdots,L_{x_n}]^\intercal$.

This equation is typically an elliptic PDE, possessing boundary conditions associated with  restrictions on the class of admissible functions $u$.
To more easily compute $\eqref{EL:multivar_domain}$, we consider a related parabolic PDE,
\begin{align}
    \begin{split}
    &{ } u_t = -(L_y - {\rm div}\, L_{\nabla u}), \quad t > 0,\\
    &{ } u(x,0) = u_0(x), \quad t = 0.
    \end{split} \label{grad_desc_pde}
\end{align}
A steady state solution of \eqref{grad_desc_pde} does not depend on time, and thus solves the Euler-Lagrange equation.
It is often easier to evolve an initial guess using \eqref{grad_desc_pde}, and stop whenever its steady state is well-approximated, than to solve \eqref{EL:multivar_domain} directly.

\begin{example}
Consider the energy functional
\[ J[u] = \int_{\Omega} \|\nabla u\|^2 \, dx.\]
% where the class of admissible functions $u$ satisfy appropriate Dirichlet conditions on $\partial \Omega$.
The minimizing function $u^*$ satisfies the Euler-Lagrange equation
\[-{\rm div}\, \nabla u    = - \triangle u = 0.\]
The gradient descent flow is the well-known heat equation
\[u_t = \triangle u.\]
\end{example}

The Euler-Lagrange equation could equivalently be described as $\triangle u = 0$, leading to the PDE $u_t = -\triangle u$.
Since the backward heat equation is ill-posed, it would not be helpful in a search for the steady-state.

Let us take the time to make \eqref{grad_desc_pde} more rigorous.
We recall that
\begin{align*}
        \delta J(u;h) &= \left.\frac{d}{dt}J(u + \epsilon h)\right|_{\epsilon=0},\\
        &= \int_{\Omega} (L_y(u) - {\rm div}\, L_{\nabla u}(u)) h\, dx,\\
        &= \langle L_y(u) - {\rm div}\, L_{\nabla u}(u), h \rangle_{L^2(\Omega)},
\end{align*}
for each $u$ and each admissible perturbation $h$.
Then using the Cauchy-Schwarz inequality,
\begin{align*}
    |\delta J(u;h)| &\leq \|L_y(u) - {\rm div}\, L_{\nabla u}(u)\| \cdot \| h \|
\end{align*}
with equality iff $h =\alpha(L_y(u) - {\rm div}\, L_{\nabla u}(u))$ for some $\alpha \in \mathbb{R}$.
This implies that the ``direction''
$h = L_y(u) - {\rm div}\, L_{\nabla u}(u)$ is the direction of steepest ascent and
maximizes $\delta J(u;h)$.
Similarly,
\[h = -(L_y(u) - {\rm div}\, L_{\nabla u}(u))\]
 points in the direction of steepest descent, and the flow described by \eqref{grad_desc_pde} tends to move toward a state of lesser energy.

\subsection*{Minimizing the area of a surface of revolution}
The area of the surface obtained by revolving a curve $y(x)$ about the $x$-axis is
\[A[y] = \int_a^b 2 \pi y \sqrt{1 + (y')^2} \, dx.
\]
To minimize the functional $A$ over the collection of smooth curves with fixed end points $y(a) = y_a$, $y(b) = y_b$, we use the Euler-Lagrange equation
\begin{align}
    \begin{split}
    0 &= 1 - y \frac{y''}{1 + (y')^2} , \\
    &= 1 + (y')^2 - y y'',
    \end{split}\label{tv_images:SA_EL_equation}
\end{align}
with the gradient descent flow given by
\begin{align}
    \begin{split}
    &{ } u_t = -1 - (y')^2 + y y'',\quad t > 0,\, x \in (a,b), \\
    &{ } u(x,0) = g(x), \quad t = 0,\\
    &{ } u(a,t) = y_a, \quad u(b,t) = y_b.
    \end{split}\label{tv_images:SA_flow}
\end{align}

\subsection*{Numerical Implementation}
We will construct a numerical solution of \eqref{tv_images:SA_flow} using the conditions $y(-1) = 1$, $y(1) = 7$.
A simple solution can be found by using a second-order order discretization in space  with a simple forward Euler step in time. We create the grid and set our end states below.
\begin{lstlisting}
import numpy as np

a, b = -1, 1.
alpha, beta = 1., 7.
####  Define variables x_steps, final_T, time_steps  ####
delta_t, delta_x = final_T/time_steps, (b-a)/x_steps
x0 = np.linspace(a,b,x_steps+1)
\end{lstlisting}

Most numerical schemes have a stability condition that must be satisfied. Our discretization requires that $\frac{\triangle t}{(\triangle x)^2} \leq \frac{1}{2}$.
We continue by checking that this condition is satisfied, and use the straight line connecting the end points as initial data.

\begin{lstlisting}
# Check a stability condition for this numerical method
if delta_t/delta_x**2. > .5:
    print "stability condition fails"

u = np.empty((2,x_steps+1))
u[0]  = (beta - alpha)/(b-a)*(x0-a)  + alpha
u[1] = (beta - alpha)/(b-a)*(x0-a)  + alpha
\end{lstlisting}

Finally, we define the right hand side of our difference scheme, and time step until the scheme converges.
\begin{lstlisting}
def rhs(y):
    # Approximate first and second derivatives to second order accuracy.
    yp = (np.roll(y,-1) - np.roll(y,1))/(2.*delta_x)
    ypp = (np.roll(y,-1) - 2.*y + np.roll(y,1))/delta_x**2.
    # Find approximation for the next time step, using a first order Euler step
    y[1:-1] -= delta_t*(1. + yp[1:-1]**2. - 1.*y[1:-1]*ypp[1:-1])

# Time step until successive iterations are close
iteration = 0
while iteration < time_steps:
    rhs(u[1])
    if norm(np.abs((u[0] - u[1]))) < 1e-5: break
    u[0] = u[1]
    iteration+=1

print "Difference in iterations is ", norm(np.abs((u[0] - u[1])))
print "Final time = ", iteration*delta_t
\end{lstlisting}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{min_surface_area.pdf}
\caption{The solution of \eqref{tv_images:SA_EL_equation}, found using the gradient descent flow \eqref{tv_images:SA_flow}.}
\label{fig:tv_images:SA_image}
\end{figure}

\begin{problem}
Using $20$ $x$ steps, $250$ time steps, and a final time of $.2$, plot the solution that minimizes \eqref{tv_images:SA_flow}.
It should match figure \ref{fig:tv_images:SA_image}.

\end{problem}

\section*{Image Processing: Denoising}

A greyscale image can be represented by a scalar-valued function $u:\Omega \to \mathbb{R}$, $\Omega \subset \mathbb{R}^2$. The following code reads an image into an array of floating point numbers, adds some noise, and saves the noisy image.
\begin{lstlisting}
from numpy.random import random_integers, uniform, randn
import matplotlib.pyplot as plt
from matplotlib import cm
from imageio import imread, imwrite

imagename = 'baloons_resized_bw.jpg'
changed_pixels=40000
# Read the image file imagename into an array of numbers, IM
# Multiply by 1. / 255 to change the values so that they are floating point
# numbers ranging from 0 to 1.
IM = imread(imagename, as_gray=True) * (1. / 255)
IM_x, IM_y = IM.shape

for lost in xrange(changed_pixels):
    x_,y_ = random_integers(1,IM_x-2), random_integers(1,IM_y-2)
    val =  .1*randn() + .5
    IM[x_,y_] = max( min(val,1.), 0.)
imwrite("noised_"+imagename, IM)
\end{lstlisting}
A color image can be represented by three functions $u_1, u_2,$ and $u_3$. In this lab we will work with black and white images, but total variation techniques can easily be used on more general images.

\subsection*{A simple approach to image processing}
Here is a first attempt at denoising: given a noisy image $f$, we look for a denoised image $u$ minimizing the energy functional
\begin{align}
J[u] = \int_{\Omega} L(x,u,\nabla u) \, dx, \label{tv_images:diffusion}
\end{align}
where
\begin{align*}
L(x,u,\nabla u) &= \frac{1}{2}(u-f)^2 + \frac{\lambda}{2} | \nabla u|^2,\\
&= \frac{1}{2}(u-f)^2 + \frac{\lambda}{2} (u_x^2 + u_y^2)^2.
\end{align*}
This energy functional penalizes 1) images that are too different from the original noisy image, and 2) images that have large derivatives. The minimizing denoised image $u$ will balance these two different costs.

Solving for the original denoised image $u$ is a difficult inverse problem-some information is irretrievably lost when noise is introduced. However, a priori information can be used to guess at the structure of the original image.  For example, here $\lambda$ represents our best guess on how much noise was added to the image, and is known as a regularization parameter in inverse problem theory.

The Euler-Lagrange equation corresponding to \eqref{tv_images:diffusion} is
\begin{align*}
L_u - \text{div } L_{\nabla u} &= (u-f) - \lambda \triangle u,\\
&= 0.
\end{align*}
and the gradient descent flow is
\begin{align}
    \begin{split}
u_t &= -(u-f -\lambda \triangle u),\\
u(x,0) &= f(x).
    \end{split} \label{tv_images:diffusion_flow}
\end{align}

Let $u_{ij}^n$ represent our approximation to $u(x_i,y_j)$ at time $t_n$. We will approximate $u_t$ with a forward Euler difference, and $\triangle u$ with centered differences:
\begin{align*}
    u_t &\approx \frac{u_{ij}^{n+1}-u_{ij}^n}{\triangle t},\\
    u_{xx} &\approx \frac{u_{i+1,j}^{n}-2u_{ij}^n + u_{i-1,j}^n}{\triangle x^2}, \\
    u_{yy} &\approx \frac{u_{i,j+1}^{n}-2u_{ij}^n + u_{i,j-1}^n}{\triangle y^2}.
\end{align*}

\begin{problem}
Using $\triangle t = 1e{-3},$ $\lambda = 40,$ $\triangle x = 1,$ and $\triangle y = 1$, implement the numerical scheme mentioned above to obtain a solution $u$. (So $\Omega = [0,n_x]\times [0,n_y]$, where $n_x$ and $n_y$ represent the number of pixels in the $x$ and $y$ dimensions, respectively.) Take 250 steps in time. Compare your results with Figure \ref{fig:noise_compare_attempts}.

Hint: Use the function \li{np.roll} to compute the spatial derivatives. For example, the second derivative can be approximated at interior grid points using
\begin{lstlisting}
u_xx = np.roll(u,-1,axis=1) - 2*u + np.roll(u,1,axis=1)
\end{lstlisting}
\end{problem}

% diffusion_denoised_baloons_resized_bw.jpg
% \begin{lstlisting}
% delta_t = 1e-3
% lmbda = 40
% u = np.empty((2,IM_x,IM_y))
% u[1] = IM
%
% def laplace(z):
%     # Approximate first and second derivatives to second order accuracy.
%     z_xx = (np.roll(z,-1,axis=0) - 2.*z + np.roll(z,1,axis=0))#/delta_x**2.
%     z_yy = (np.roll(z,-1,axis=1) - 2.*z + np.roll(z,1,axis=1))#/delta_y**2.
%     # Find approximation for the next time step, using a first order Euler step
%     z[1:-1,1:-1] -= delta_t*(   (z[1:-1,1:-1]-IM[1:-1,1:-1])
%                                     -lmbda*(z_xx[1:-1,1:-1] + z_yy[1:-1,1:-1]))
%
% # Iterate towards a steady state solution of the gradient descent flow.
% iteration = 0
% while iteration < time_steps:
%     laplace(u[1])
%     if norm(np.abs((u[0] - u[1]))) < 1e-4: break
%     u[0] = u[1]
%     iteration+=1
%
% \end{lstlisting}
\begin{figure}
\begin{minipage}[b]{.47\linewidth}
\centering
\includegraphics[width=\textwidth]{balloons_resized_bw.jpg}
\caption*{Original image}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{lab_noised_baloons_resized_bw.jpg}
\caption*{Image with white noise}
\end{minipage}
\caption{Noise.}
\label{fig:noise_firstattempt}
\end{figure}

\begin{figure}
\begin{minipage}[b]{.47\linewidth}
\centering
\includegraphics[width=\textwidth]{diffusion_denoised_baloons_resized_bw.jpg}
\caption*{Initial diffusion-based approach}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{tv_denoised_baloons_resized_bw.jpg}
\caption*{Total variation based approach}
\end{minipage}
\caption{The solutions of \eqref{tv_images:diffusion_flow} and \eqref{tv_images:tv_flow}, found using a first order Euler step in time and centered differences in space.}
\label{fig:noise_compare_attempts}
\end{figure}

% \begin{figure}
% \centering
% \includegraphics[width=6cm]{diffusion_denoised_baloons_resized_bw.jpg}
% \caption{The solution of \eqref{tv_images:diffusion_flow}, found using a first order Euler step in time and centered differences in space.}
% \label{fig:diffusion_image_denoised}
% \end{figure}

\section*{Image Processing: Total Variation Method}
We represent an image by a function $u:[0,1]\times[0,1] \to \mathbb{R}$.
A $C^1$ function $u:\Omega \to \mathbb{R}$ has bounded total variation on $\Omega$ ($BV(\Omega)$) if $\int_{\Omega} |\nabla u| < \infty$; $u$ is said to have total variation $\int_{\Omega} |\nabla u|$.  Intuitively, the total variation of an image $u$ increases when noise is added.

The total variation approach was originally introduced by Ruding, Osher, and Fatemi\footnote{L. Rudin, S. Osher, and E. Fatemi, ``Nonlinear total variation based noise removal algorithms'', \emph{Physica D.}, 1992.}. It was formulated as follows: given a noisy image $f$, we look to find a denoised image $u$ minimizing
\begin{align}
\int_{\Omega} |\nabla u(x)|\, dx \label{tv_images:tv}
\end{align}
subject to the constraints
\begin{align}
    &{ } \int_{\Omega} u(x) \, dx = \int_{\Omega} f(x)\, dx, \label{tv_images:same_mean}\\
    &{ } \int_{\Omega} |u(x) - f(x)|^2\, dx = \sigma |\Omega|.\label{tv_images:aprior_variance}
\end{align}
Intuitively, \eqref{tv_images:tv} penalizes fast variations in $f$ - this functional together with the constraint \eqref{tv_images:same_mean} has a constant minimum of $u = \frac{1}{|\Omega|}\int_{\Omega} u(x) \, dx$. This is obviously not what we want, so we add a constraint \eqref{tv_images:aprior_variance} specifying how far $u(x)$ is required to differ from the noisy image $f$.  More precisely, \eqref{tv_images:same_mean} specifies that the noise in the image has zero mean, and \eqref{tv_images:aprior_variance} requires that a variable $\sigma$ be chosen a priori to represent the standard deviation of the noise.

Chambolle and Lions proved that the model introduced by Rudin, Osher, and Fatemi can be formulated equivalently as
\begin{align}
F[u] = \min_{u \in BV(\Omega)} \int_{\Omega} |\nabla u| + \frac{\lambda}{2}(u-f)^2 \, dx,
\end{align}
where $\lambda >0$ is a fixed regularization parameter\footnote{A. Chambelle and P.-L. Lions, ``Image recovery via total variation minimization and related problems", \emph{Numer. Math.}, 1997.}. Notice how this functional differs from \eqref{tv_images:diffusion}: $\int_{\Omega} |\nabla u|$ instead of $\int_{\Omega} |\nabla u|^2$. This turns out to cause a huge difference in the result.  Mathematically, there is a nice way to extend $F$ and the class of functions with bounded total variation to functions that are discontinuous across hyperplanes. The term $\int |\nabla|$ tends to preserve edges/boundaries of objects in an image.

% The Euler-Lagrange equation is
% \begin{align*}
% \lambda (u-f) - \frac{u_{xx}u_y^2 + u_{yy}u_x^2 - 2u_xu_yu_{xy}}{(u_x^2 + u_y^2)^{3/2}} &= 0.
% \end{align*}
The gradient descent flow is given by
\begin{align}
    \begin{split}
u_t &= -\lambda (u-f) + \frac{u_{xx}u_y^2 + u_{yy}u_x^2 - 2u_xu_yu_{xy}}{(u_x^2 + u_y^2)^{3/2}} ,\\
u(x,0) &= f(x).
\end{split} \label{tv_images:tv_flow}
\end{align}
Notice the singularity that occurs in the flow when $|\nabla u| = 0$. Numerically we will replace  $|\nabla u|^{3}$ in the denominator with $(\epsilon + |\nabla u|^{2})^{3/2}$, to remove the singularity.

\begin{problem}
Using $\triangle t = 1e-3, \lambda = 1, \triangle x = 1,$ and $ \triangle y = 1$, implement the numerical scheme mentioned above to obtain a solution $u$.  Take 200 steps in time. Compare your results with Figure \ref{fig:noise_compare_attempts}. How small should $\epsilon$ be?

Hint: To compute the spatial derivatives, consider the following:
\begin{lstlisting}
u_x = (np.roll(u,-1,axis=1) -  np.roll(u,1,axis=1))/2
u_xx = np.roll(u,-1,axis=1) - 2*u + np.roll(u,1,axis=1)
u_xy = (np.roll(u_x,-1,axis=0) - np.roll(u_x,1,axis=0))/2.
\end{lstlisting}
\end{problem}

% \begin{figure}
% \centering
% \includegraphics[width=6cm]{tv_denoised_baloons_resized_bw.jpg}
% \caption{The solution of \eqref{tv_images:diffusion_flow}, found using a first order Euler step in time and centered differences in space.}
% \label{fig:tv_image_denoised}
% \end{figure}


\documentclass{article}

\usepackage{mathrsfs,amssymb,amsmath,amsfonts}

\begin{document}

\section{If $\mathscr{H}$ is a Hilbert space, $\mathscr{A}=\mathscr{B}(\mathscr{H} )$ is a C*-algebra where for each A in $\mathscr{B}(\mathscr{H} )$, A*=the adjoint of A}

VIII.1.1 (pg 236)

$\langle Ah, k \rangle = \langle h, A^*k \rangle = \overline{ \langle A^*k,h \rangle}$

$\langle A^*k, h \rangle = \langle k, A^{**}h \rangle = \overline{\langle A^{**}h,k \rangle}$

The above are justified by the definition of the adjoint and inner product, for all values h and k in $\mathscr{H}$.

Taking the complex conjugate of the bottom row shows that $\langle Ah, k \rangle = \langle A^{**}h,k \rangle$, hence A=A**.

$\langle (AB)^*h, k \rangle = \langle h, (AB)k \rangle = \langle h, A(B(k)) \rangle = \langle A^* h, B k \rangle = \langle B^*A^*h,k \rangle$

Hence, $(AB)^* = B^*A^*$

$\langle (\alpha A + B)^*h,k \rangle = \langle h, (\alpha A + B) k \rangle = \langle h, \alpha A k + B k \rangle$

$\langle \bar{\alpha}A^*h+B^*h, k \rangle$ (kind of ran out of steam by this point)

\section{If X is a compact Hausdorff space, show that X is totally disconnected if and only if C(X) is the closed linear span of its projections ($\equiv$ hermitian idempotents)}

VIII.2.3 (pg 239)

Assume $X$ is totally disconnected. Consider a bounded function $f$. Ignoring net vs sequence issues (TODO: is this justified?), we must show that $f$ can be written as the convergence of $a_1 e_1 + a_2 e_2 + ... $ where $a_n \in \mathbb{C}$ and $e_n$ are hermitian idempotents.

In this context, a hermitian idempotent is one whose range is $\{0,1\}$, since those are the only complex numbers equal to their own square. Consider a series of function $e_{q_1}, e_{q_2}$, where for each ${q_i} \in X$, define

\begin{equation*}
e_{q_i}(x)= \left\{
        \begin{array}{ll}
            1 & \quad x = q_i \\
            0 & \quad x \ne q_i
        \end{array}
    \right.
\end{equation*}

$f$ can therefore be written as a countably infinite linear sum of such a series, thus is in the closed linear span of the projections.

Hm, I didn't use the fact that $X$ is totally disconnected so something is missing here.

\end{document}


\documentclass[a4paper,12pt]{article}
\usepackage[left=2.0cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\newenvironment{packed_enum}{
\begin{itemize}
  \setlength{\topsep}{0pt}
  \setlength{\itemsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\parsep}{0pt}
  \setlength{\partopsep}{0pt}
}{\end{itemize}}
\begin{document}
\section*{Simple model for $\textrm{R}_{0}$}
\begin{eqnarray*}
\textrm{j}(\textrm{t})&=&\int_{0}^{\infty}\textrm{A}(\tau)\textrm{j}(\textrm{t}-\tau)d\tau \\
\textrm{R}_{0}&=&\int_{0}^{\infty}\textrm{A}(\tau)d\tau
\end{eqnarray*}
Where
\begin{packed_enum}
\item j(t) is the number of new infections at time t
\item A($\tau$) is the rate an infected person infects a healthy person at time $\tau$ after infection
\item $\textrm{R}_{0}$ is the total number of new infections resulting from an infected person
\end{packed_enum}
If we assume the infection rate is constant for a number of consecutive days (w) following infection then
\begin{eqnarray*}
\textrm{j}(\textrm{t})&=&\textrm{A}(\textrm{t})\int_{0}^{\textrm{w}}\textrm{j}(\textrm{t}-\tau)d\tau \\
\textrm{R}_{0}(\textrm{t})&=&\textrm{w}\:\textrm{A}(\textrm{t})
\end{eqnarray*}
So
\begin{equation*}
\textrm{R}_{0}(\textrm{t})=\textrm{w}\:\frac{\textrm{j}(\textrm{t})}{\int_{0}^{\textrm{w}}\textrm{j}(\textrm{t}-\tau)d\tau}
\end{equation*}
Approximated as
\begin{eqnarray*}
\textrm{R}_{0}(\textrm{t})&=&\textrm{w}\:\frac{\textrm{j}(\textrm{t})}{\textrm{s}(\textrm{t})} \\
\textrm{s}(\textrm{t})&=&\sum_{\tau=0}^{\tau=\textrm{w}}\textrm{j}(\textrm{t}-\tau)
\end{eqnarray*}
\end{document}


\section{Control Flow Becomes Data Flow}
\label{controlflowtodataflow}

Conditional expressions negatively impact execution speed, even in
contemporary computer architectures, so compiler (and interpreter)
writers attempt to eliminate them, whenever that is feasible.
Boolean arrays facilitate the replacement of control flow by
data flow, frequently improving performance as a result of
eliminating conditionals.\cite{DBLP:conf/pldi/BerneckyS15}
For example, if we want to give everyone whose salary,
{\apl s}, is less than {\apl tiny},
a raise of {\apl r}, it can be done this way in parallel,
without resorting to scalar-oriented control flow constructs
such as {\tt if/then/else}:

\medskip
{\apl s\qlarrow\0s\qplus\0r\qtimes\0s\qlt\0tiny}
\medskip

\noindent This terse and expressive ability arises directly from
what Knuth calls {\em Iverson's convention for characteristic functions},
of treating Boolean true and false as the integers {\apl 1} and {\apl 0},
respectively.
Other examples include the verb {\apl mqs}, to mark the
quoted strings within a text vector, given in Section~\ref{reduceandscan},
and the (mostly) SIMD tokenizer for the
APEX APL compiler.~\cite{RBernecky:apex,RBernecky:tokenizer}


\section{Calculus with Power Series}\label{sec:CalculuswithPowerSeries}

We now know that some functions can be expressed as power series,
which look like infinite polynomials. Since it is easy to find derivatives
and integrals of polynomials, we might hope that we can take derivatives
and integrals of power series in an analogous way. In fact we can, as stated
in the following theorem, which we will not prove here.

\begin{theorem}{}{}
Suppose the power series $f(x)=\ds\sum_{n=0}^\infty a_n(x-a)^n$ has
radius of convergence $R$. Then
\begin{align*}
  f'(x)&=\sum_{n=0}^\infty na_n(x-a)^{n-1},	\\
  \int f(x)\,dx &= C+\sum_{n=0}^\infty {a_n\over n+1}(x-a)^{n+1},
\end{align*}
and these two series have radius of convergence $R$.
\end{theorem}

\begin{example}{}{}
Find a power series representation of $\ln|1-x|$.
\end{example}
\begin{solution}
Starting with the geometric series:
\begin{align*}
  {1\over 1-x} &= \sum_{n=0}^\infty x^n	\\
  \int{1\over 1-x}\,dx &= -\ln|1-x| = \sum_{n=0}^\infty {1\over n+1}x^{n+1}	\\
  \ln|1-x| &= \sum_{n=0}^\infty -{1\over n+1}x^{n+1}
\end{align*}
when $|x|<1$. The series does not converge when $x=1$ but does
converge when $x=-1$ or $1-x=2$. The interval of convergence is
$[-1,1)$, or $0<1-x\le2$.
We can use this series to express $\ln(a)$ as a series
when $0<a\le2$ by setting $x-1=a$. For example
$$
  \ln(3/2)=\ln\left(1-(-1/2)\right)=
  \sum_{n=0}^\infty (-1)^n{1\over n+1}{1\over 2^{n+1}}.
$$
We can use this in turn to approximate $\ln(3/2)$:
$$
  \ln(3/2)\approx {1\over 2}-{1\over 8}+{1\over 24}-{1\over 64}
  +{1\over 160}-{1\over 384}+{1\over 896}
  ={909\over 2240}\approx 0.406
.$$
Because this is an alternating series with decreasing terms,
we know that the true value is between $909/2240$ and
$909/2240-1/2048=29053/71680\approx .4053$, so $0.4053\leq\ln(3/2)\leq 0.406$.
\end{solution}

With a bit of arithmetic, we can approximate values outside of the interval of convergence:

\begin{example}{}{}
Find an approximation for $\ln(9/4)$.
\end{example}
\begin{solution}
We can use the approximation we just computed, plus some rules for logarithms:
$$\ln(9/4)=\ln((3/2)^2)=2\ln(3/2)\approx 0.812,$$
and using our bounds above,
$$0.8106\leq \ln(9/4)\leq 0.812.$$
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:CalculuswithPowerSeries}}

\begin{enumialphparenastyle}

\begin{ex}
Find a series representation for $\ln 2$.
\begin{sol}
the alternating harmonic series
\end{sol}
\end{ex}

\begin{ex}
Find a power series representation for $\ds 1/(1-x)^2$.
\begin{sol}
$\ds\sum_{n=0}^\infty (n+1)x^n$
\end{sol}
\end{ex}

\begin{ex}
Find a power series representation for $\ds 2/(1-x)^3$.
\begin{sol}
$\ds\sum_{n=0}^\infty (n+1)(n+2)x^n$
\end{sol}
\end{ex}

\begin{ex}
Find a power series representation for $\ds 1/(1-x)^3$.
What is the radius of convergence?
\begin{sol}
$\ds\sum_{n=0}^\infty {(n+1)(n+2)\over 2}x^n$, $R=1$
\end{sol}
\end{ex}

\begin{ex}
Find a power series representation for $\ds\int\ln(1-x)\,dx$.
\begin{sol}
$\ds C+\sum_{n=0}^\infty {-1\over (n+1)(n+2)}x^{n+2}$
\end{sol}
\end{ex}

\end{enumialphparenastyle}
\clearpage


\section*{Exercise 26.1-1}
\subsubsection*{Show that a maximum flow in $G'$ has the same value as a maximum flow in $G$}
Let $G$, $G'$, $x$, $u$ and $v$ be given as in the problem description.

What we want to argue here is that any flow in $G$ is also feasible in $G'$ and with the same flow value.
Then we conclude that $\abs{f_G^*} = \abs{f_{G'}^*}$ where $\abs{f_{G}^*}$ is the maximum flow value in the network $G$.

First observe that any flow $f(u,v)$ in $G$ can be replaced with a equivalent flow in $G'$ where $f(u,v) = f(u,x) = f(x,v)$ since $c(u,v) = c(u,x) = c(x,v)$.

Let $p_{f_G^*}$ be the path the maximum flow $f^*$ takes in $G$.

\textit{If $(u,v) \not\in p_{f_G^*}$}

In this case the same path is available in $G'$ which would yield the same flow value. If introducing the new vertex $x$ means that $p_{f_{G'}^*}$ is changed such that $\{(u,x),(x,v)\}\subset p_{f_{G'}^*}$, then by our first observation above, the same flow value is possible in $G$, and since $(u,v) \not\in p_{f_G^*}$ deviating cannot yield a higher flow value.

\textit{If $(u,v) \in p_{f_G^*}$}

In this case we have already observed that replacing the $f(u,v)$ with $f(u,x) = f(x,v)$ will yield the same flow value. Hence the maximum flow value will be the same.

If introducing the new vertex $x$ in $G'$ changes the flow value from $u$ to $v$, then this cannot yield a higher maximum flow value in $G'$. This is because of our observation above, as all flow from $u$ to $v$ are bounded by the same capacity constraint in both $G$ and $G'$, the same flow value is feasible in both networks. As $f^*_G$ is part of a maximum flow value, deviating from that cannot yield a higher flow value, as that goes against it being a maximum flow value in the beginning.

Hence we conclude that in all possible cases the maximum flow value will be the same, hence $\abs{f_G^*} = \abs{f_{G'}^*}$.


\documentclass{article}
\usepackage{amsmath}
\usepackage{geometry}
\usepackage[utf8]{inputenc}
\usepackage{enumitem}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1em}

\begin{document}

\section{Divisibility}
\subsection{Rules}
For integers $a,b,c,d$:
	\begin{enumerate}
		\item
		If $a \mid b$ and $ c \mid d$ then $ac \mid bd$
		\item
		If $a \mid b$ and $a \mid c$, then $a \mid b+c$
		\item
		\textbf{Euclid's algorithm.} \\
		$\gcd (a,b) = \gcd (a,b-a)$
		\item
		\textbf{Corollary of Euclid's algorithm.} \\
		$ax+by=n$ has solution $(x,y)$ in integers if and only if $gcd(a,b) \mid n$
	\end{enumerate}
\subsection{Problems}
\begin{enumerate}
	\item
	Prove that if $m-p \mid mn +pq$, then $m-p \mid mq +np$.

	\item
	Prove that $17 \mid 2a+3b  \iff 17 \mid 9a+5b$

	\item
	Prove that it is not possible to find positive integers $n$ and $m>1$, such that ${102^{2017}+103^{2017}=n^m}$

	\item
	Find all positive integers $d$, such that $d$ divides both: $n^2+1$ and $(n+1)^2+1$

	\item
	Find all prime numbers $p$ for which $p^2+2543$ has less than $16$ positive divisors.

	\item
	Prove that $\gcd (a^m-1,a^n-1)=a^{\gcd(m,n)}-1$

	\item % BW treening 2011-Q13
	Prove that any two non-equal integers in form $2^{2^n}+1$, where $n$ is a positive integer are coprime-

	\item % IMO shortlist 1984
	$a_1,a_2,\dots,a_{2n}$ are mutually distinct integers. Find all integers $x$ satisfying
	$$(x-a_1)\dots (x-a_{2n}) = (-1)^n(n!)^2$$

	\item %http://kodu.ut.ee/~zolki/math/bwsess04.pdf
	Let $a_0,\dots, a_n \geq -1$ be integers such that at least one of them is non-zero. It is known that $a_0 + 2a_1 +2^2a_2+ \dots + 2^na_n = 0$. Show that $a_0+a_1+\dots + a_n >0$.

\end{enumerate}

\newpage
\section{Congruences}
\subsection{Rules}
For integers $a,b,c,d,m,n$ and prime $p$.
\begin{enumerate}
	\item
	$a \equiv b \mod m \iff m \mid a-b $
	\item
	$a\equiv b \mod m$ and $ c\equiv d \mod m \implies a+c\equiv b+d \mod m$
	\item
	$a \equiv b \mod m \iff an \equiv bn \mod mn$
	\item
	$a \equiv b \mod m \implies an \equiv bn \mod m$
	\item
	\textbf{Fermat's little theorem.} \\
	$a^p \equiv a \mod p$
	\item
	\textbf{Wilson's theorem.} \\
	$(p-1)!\equiv -1 \mod p$

\end{enumerate}

\subsection{Problems}

\begin{enumerate}
	\item
	Prove that $2018!! + 2017!!$ is divisible by $2019$

	\item
	Find all primes $p$, such that:
	\begin{enumerate}
		\item  $p+4$, $p+14$ are primes;
		\item $8p^2+1$ is a prime;
		\item $p+10$, $p+1$ is a prime;
		\item $4p^2+1$, $6p^2+1$ are primes;
		\item $p^2-6$, $p^2+6$ are primes;
		\item $p^4-6$ is a prime;
		\item $p^3+6$, $p^3-6$ are primes;
		\item $p^2-2$, $2p^2-1$, $3p^2+4$ are primes;
		\item $2^p+1$, $2^p-1$ are primes;
		\item $p,q$, $p^q+q^p$ are primes.
	\end{enumerate}

	\item
	In a series of integers, the next element is obtained by concatenating the element's order number to previous element by using carry. The first elements of the series are therefore:
	$${1,12,123,1234,\dots,123456789,1234567900,12345679011} $$
	Find all elements in the series which are divisible by $7$.

	\item % http://kodu.ut.ee/~zolki/math/bwsess04.pdf
	Prove that there exist no integers $n>1$, such that $n \mid 3^n-2^n$.

	\item
	Find all positive integers $n$ which satisfy the following condition:
	For all integers $a$ and $b$ which are coprime with $n$
	$$ a \equiv b  \mod n \iff ab \equiv 1 \mod n$$

	\item
	Prove that it is not possible to separate $18$ consecutive integers into $2$ subsets with $9$ elements in such way that the product of each subset is equal.
\end{enumerate}

\end{document}


\chapter{Kinematics, and the calculus of infinitesimals}
\label{sec:derivative}

Kinematics is the mathematics that describes motions.

A \emph{frame} defines \emph{where} and \emph{when}.

\emph{Motion} is change of position.%
\footnote{\url{https://en.wikipedia.org/wiki/Motion_\%28physics\%29}}

An object \emph{moves} iff its position changes.

The \emph{speed} of an object is how fast it moves:
how far it moves in how much time.
\emph{Fast} means high speed,
going far in little time,
traveling much distance in little time.

\emph{Average speed} is distance traveled divided by time required.

\emph{Velocity} is the rate of change of position.
Speed is the magnitude of velocity.
\emph{Rate of change} is defined by \emph{derivative} (\S\ref{sec:derivative}).

\section{Describing motions using equations relating position and time}

We can use a position function.
Its type is \( \Real \to V \).

An example of an equation of motion is \( x(t) = 2 t e_1 \).
It describes an object that moves with constant velocity \(2\) towards the positive x-axis.
An \emph{equation of motion} is an equation that describes
the motion of an object by relating time and position.

Each equation of motion corresponds to a moving object.

To describe more objects, use more equations.

Let \(e\) be a linear basis.
Suppose that the position of an object at time \(t\) is
\(x(t) = e(x_1(t), \ldots, x_n(t))\).
Then the velocity at time \(t\) is \(v(t) = \der(x,t) = e(v_1(t), \ldots, v_n(t)) \).
Can we say that \(v_k(t) = \der(x_k,t)\)?

Moral of the story:
If we have a linear basis,
then doing calculus on the coordinates
is doing calculus on the vectors.

\section{Notating derivatives}

Let \(f : \Real \to \Real\).

\paragraph{\der}

We can describe the derivative by a function \(\der : (\Real \to \Real) \to (\Real \to \Real)\).
\[
    \der(f,x) = \StandardPart\parenthesize{\frac{f^*(x+\delta)-f^*(x)}{\delta}}
\]
where \(f^*\) is the natural extension of \(f\) to the hyperreals.%
\footnote{\url{https://en.wikipedia.org/wiki/Non-standard_calculus\#Definition_of_derivative}}
Then \(\der(f,x)\) is the slope of the tangent line of \(f\) at \(x\).

Strictly, \(\der(f)(x)\), not \(\der(f,x)\).

Thus \(f' = \der(f)\).

\(\dd{x}\) means an infinitesimal change in \(x\)?
What does that mean?
What is \(x\)?

\paragraph{Euler\textendash{}Arbogast D-notation}

% https://en.wikipedia.org/wiki/Notation_for_differentiation#Euler.27s_notation

\( D_x E \) is the derivative of expression \(E\) with respect to variable \(x\).
The variable \(x\) should occur free in \(E\).

Typing rule:
If \(x:\Real\) and \(E:\Real\), then \(D_x E:\Real\).

The notation \(E[x:=y]\) means \(E\) but with each free occurrence of \(x\) replaced with \(y\).

Then \( D_x E = \lim_{h \to 0} \frac{E[x := x+h] - E}{h} \).

Then \( D_x E = d(x \to E) \).

Example: \( D_x (4x^2) = d(x \to 4x^2) = 8x \).

Example: \( D_y (x + y) = d(y \to x + y) = x + 1 \).

Advantage: With \(D_x\) notation, we can refer to an input by name;
With \(d_k\) notation, we can refer to an input by index only.

\(D\) is a custom syntax, not an ordinary function.

% Multivariate differential calculus
% Vector calculus

\section{Relating velocities, tangent lines, and derivatives}

There are several ways of understanding \(f'(x)\) (the derivative of \(f\) at \(x\)):
\UnorderedList{
    \item rate of change of \(f\) at \(x\); instantaneous velocity
    \item slope of the tangent line of \(f\) at \(x\)
    \item best linear approximation (not discussed here)
}

\paragraph{Average velocity and the secant line}

Let there be an object.

Let \(x(t) : V^2\) be a vector that describes its position at time \(t : \Real\).

The \emph{average velocity} of that object in the time interval \([t,t+\Delta t]\) is
\[ \frac{x(t+\Delta t) - x(t)}{\Delta t}. \]

If at time \(t_1\) its position is \(x_1\)
and at time \(t_2\) its position is \(x_2\),
then its \emph{average velocity} in the time interval between \(t_1\) and \(t_2\)
is \((x_2 - x_1) / (t_2 - t_1)\).

A \emph{secant line of \(f\)} is a line that passes \((x_1,f(x_1))\) and \((x_2,f(x_2))\).
Think of average velocity.

\paragraph{Instantaneous velocity and the tangent line}

If the position of an object at time \(t\) is \(x(t)\),
then its \emph{instantaneous velocity} at time \(t\) is \(v(t) = (d(x))(t)\).
The velocity function is the derivative of the position function.

The term \emph{instantaneous velocity} is often shortened to just \emph{velocity}.

The unqualified \emph{velocity} means \emph{instantaneous velocity}.

A car's speedometer measures its instantaneous speed.

Derivative is about \emph{rate of change}:
how fast a function changes value,
how big is the change in output compared to the change in input.

Consider a function \(f : \Real \to \Real\).
If the input is \(x\), then the output is \(f(x)\).
If you change the input by \(\dd{x}\), the output changes by \(\dd{y}\).
Formally, \(f(x+\dd{x}) = f(x)+\dd{y}\).

A \emph{tangent line of \(f\) at \(x\)} is what the secant line converges to
if both \(x_1\) and \(x_2\) converge to \(x\).
Think of instantaneous velocity.

\paragraph{Understanding the derivative as the slope of the tangent line}

The \emph{derivative of \(f\) at \(x\)} is the slope of the tangent line of \(f\) at \(x\).
Reminder: The line \(y = mx + c\) has slope \(m\).

\section{Describing motions using implicit equations}

An example of \emph{implicit} equation is \( x(t) = - (d(d(x)))(t) \).
This is also an example of a \emph{differential equation} because it contains the derivative operator \(d\).
Differential equations are discussed in \S\ref{sec:diff-eqn}.

\section{Integrals}

We can think of an integral in several ways:
\UnorderedList{
\item area under a curve
\item slicing and summing
}

\paragraph{Integrating by slicing and summing}

\section{Calculating derivatives and integrals quickly}

\paragraph{Calculating derivatives}

Symbolic calculation is enabled by
the \emph{constant rule} \eqref{der-constant-rule},
the \emph{power rule} \eqref{der-power-rule},
the \emph{product rule} \eqref{der-product-rule},
and the \emph{chain rule} \eqref{der-chain-rule}.
\begin{align}
    d (x \to y) &= 0 \text{ if \( y \) is a constant} \label{der-constant-rule}
    \\ d (x \to x^p) &= p \cdot x^{p-1} \text{ if \( p \) is a non-zero real number} \label{der-power-rule}
    \\ d(f \cdot g) &= d(f) \cdot d(g) \label{der-product-rule}
    \\ d(f \circ g) &= d(g) \cdot (d(f) \circ g) \label{der-chain-rule}
\end{align}

The \(d\) operator is linear:
If \(c\) is a constant, then \(d(c \cdot f) = c \cdot d(f)\).
Also, \(d(f+g) = d(f) + d(g)\).

\paragraph{Calculating integrals}

With the \emph{fundamental theorem of calculus},
we can compute integrals using antiderivatives.
With this shortcut, we can skip the slicing and summing.

\Formula{
    \int_{[a,b]} f(x) \dd{x} \dd{y} \dd{z} = F(b) - F(a)
}

\section{Exercise}

Compute the derivative of each of these functions:
\( x \to 1 \), \( x \to x \), \( x \to 2x \), \( x \to x^2 \),
\( x \to 3e^x \), \(x \to e^x \cdot x \).

Using the power rule, show that \( d(x \to x^2) = 2x \).

Show that \( d(x \to x^3 + x^2) = 3x^2 + 2x \).


%% LyX 2.3.6.1 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english]{article}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage[a4paper]{geometry}
\geometry{verbose,tmargin=2cm,bmargin=2cm,lmargin=2cm,rmargin=2cm}
\setlength{\parskip}{\smallskipamount}
\setlength{\parindent}{0pt}
\usepackage{amssymb}
\usepackage{babel}
\begin{document}

\section{EKF in a nutshell}

Extended Kalman Filter (EKF) is an modification of the well known
Kalman Filter for the case of nonlinear dynamics. The Kalman Filter
itself is an optimal state estimation algorithm for linear dynamics
and measurement models which experience zero-mean Gaussian noise.
In the EKF context however, neither the dynamical model nor the measurement
model have to be linear, nonetheless, linearized dynamics and measurements
models are still required within the filter. One dynamical system
can be described as follows:

\begin{equation}
\dot{{\bf x}}=f(t,{\bf x})+{\bf w}
\end{equation}

in which the function $f:\mathbb{R}\times\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$
and ${\bf w}$ is a vector of $n$ normally distributed random variables
with zero mean and a covariance matrix ${\bf Q}$. The same system
can be written in discrete time as:

\begin{equation}
{\bf x}_{k+1}={\bf F}(t_{k},t_{k+1},{\bf x}){\bf x}+{\bf w}_{k}
\end{equation}

where ${\bf F}:\mathbb{R}\times\mathbb{R}\times\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\times n}$is
the state transition matrix. A first order approximation of the state
transition matrix is found below:

\begin{equation}
{\bf F}(t_{k},t_{k+1},{\bf x})={\bf I}+\frac{\partial f(t,{\bf x})}{\partial t}\cdot(t_{k}-t_{k+1})\label{eq:STM}
\end{equation}

The measurements are modeled as follows:

\begin{equation}
{\bf z}=h(t,{\bf x})+{\bf v}
\end{equation}

in which the function $h:\mathbb{R}\times\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$
and ${\bf v}$ is a vector of $m$ normally distributed random variables
with zero mean and a covariance matrix ${\bf R}$. One other important
matrix that needs to be available for the EKF algorithm is the $H(t,{\bf x})=\frac{\partial h(t,{\bf x})}{\partial t}$.

\section{Example details}

In the context of this example, the position and velocity of a vehicle
moving along a line are to be estimated. The linear acceleration of
the system is modeled follows:

\begin{equation}
\ddot{x}=-\sin(t)\label{eq:dynamics}
\end{equation}

letting $x_{1}=x$ and $x_{2}=\dot{x}$, the dynamical model can be
written as:

\begin{equation}
\left[\begin{array}{c}
\dot{x_{1}}\\
\dot{x_{2}}
\end{array}\right]=\left[\begin{array}{c}
x_{2}\\
-\sin(t)
\end{array}\right]
\end{equation}

The state transition matrix of the system is written according to
the approximation in (\ref{eq:STM}) as:

\begin{equation}
{\bf F}(t_{k},t_{k+1},{\bf x})=\left[\begin{array}{cc}
1 & t_{k}-t_{k+1}\\
0 & 1
\end{array}\right]
\end{equation}

The system is assumed to measure only the position of the vehicle,
hence:

\[
{\bf z}=x_{1}\quad\Rightarrow\quad H(t,{\bf x})=\left[\begin{array}{cc}
1 & 0\end{array}\right]
\]

The ground truth of the system is synthesized by analytically solving
(\ref{eq:dynamics}) and adding zero-mean Gaussian noise to the velocity
(which in-turn affects the position), while the measurements where
synthesized by adding zero-mean Gaussian noise to the position ground
truth.
\end{document}


\documentclass{subfile}

\begin{document}
	\section{IMO}\label{sec:imo}

		\begin{problem}[IMO $1995$, problem $2$]
			Let $a,b,c$ be real numbers such that $abc=1$. Prove that
				\begin{align*}
					\dfrac{1}{a^{3}(b+c)}+\dfrac{1}{b^{3}(c+a)}+\dfrac{1}{c^{3}(a+b)}
						& \geq\dfrac{3}{2}
				\end{align*}
		\end{problem}

		\begin{problem}[IMO $1999$, problem $2$]
			Let $n\geq2$ be an integer. Find the least constant $C$ such that
				\begin{align*}
					\sum\limits_{1\leq i < j\leq n}x_{i}x_{j}(x_{i}^{2}+x_{j}^{2})
						& \leq C\left(\sum\limits_{i=1}^{n}x_{i}\right)^{4}
				\end{align*}
			holds for all non-negative real numbers $x_{1},\ldots,x_{n}$. When does equality occur?
		\end{problem}

		\begin{problem}[IMO $2000$, problem $2$]
			Let $a,b,c$ be positive real numbers such that $abc=1$. Prove that
				\begin{align*}
					\left(a-1+\dfrac{1}{b}\right)\left(b-1+\dfrac{1}{c}\right)\left(c-1+\dfrac{1}{a}\right)
						& \leq1
				\end{align*}
		\end{problem}

		\begin{problem}[IMO $2001$, problem $2$]
			Let $a,b,c$ be positive real numbers. Prove that
				\begin{align*}
					\dfrac{a}{\sqrt{a^{2}+8bc}}+\dfrac{b}{\sqrt{b^{2}+8ca}}+\dfrac{c}{\sqrt{c^{2}+8ab}}
						& \geq1
				\end{align*}
		\end{problem}

		\begin{problem}[IMO $2003$, problem $5$]
			Let $n$ be a positive integer and $x_{1}\leq\ldots\leq x_{n}$ be real numbers. Prove that
				\begin{align*}
					\left(\sum\limits_{i,j=1}^{n}|x_{i}-x_{j}|\right)^{2}
						& \leq \dfrac{2(n^{2}-1)}{3}\sum\limits_{i,j=1}^{n}(x_{i}-x_{j})^{2}
				\end{align*}
			Show that equality holds if and only if $x_{1},\ldots,x_{n}$ forms an arithmetic sequence.
		\end{problem}

		\begin{problem}[IMO $2004$, problem $4$]
			Let $n\geq3$ be an integer. Let $t_{1},\ldots,t_{n}$ be positive real numbers such that
				\begin{align*}
					n^{2}+1
						& > (t_{1}+\ldots+t_{n})\left(\dfrac{1}{t_{1}}+\ldots+\dfrac{1}{t_{n}}\right)
				\end{align*}
			Show that $t_{i},t_{j},t_{k}$ are the sides of a triangle for all $1\leq i< j < k\leq n$.
		\end{problem}

		\begin{problem}[IMO $2005$, problem $3$]
			Let $x,y,z$ be real numbers such that $xyz\geq1$. Prove that
				\begin{align*}
					\dfrac{x^{5}-x^{2}}{x^{5}+y^{2}+z^{2}}+\dfrac{y^{5}-y^{2}}{x^{2}+y^{5}+z^{2}}+\dfrac{z^{5}-z^{2}}{x^{2}+y^{2}+z^{5}}
						& \geq0
				\end{align*}

				\begin{solution}
					We have already solved it in \autoref{prob:imo2005-3}.
				\end{solution}
		\end{problem}

		\begin{problem}[IMO $2006$, problem $3$]
			Determine the least real number $M$ such that the inequality
				\begin{align*}
					\left|ab\left(a^{2}-b^{2}\right)+bc\left(b^{2}-c^{2}\right)+ca\left(c^{2}-a^{2}\right)\right|
						& \leq M(a^{2}+b^{2}+c^{2})^{2}
				\end{align*}
			holds for all real numbers $a,b,c$.
		\end{problem}

		\begin{problem}[IMO $2008$, problem $2$]
			Let $x,y,z\neq1$ be real numbers such that $xyz=1$. Prove that
				\begin{align*}
					\dfrac{x^{2}}{(x-1)^{2}}+\dfrac{y^{2}}{(y-1)^{2}}+\dfrac{z^{2}}{(z-1)^{2}}
						& \geq1
				\end{align*}
			Also, prove that equality holds for infinitely many rational $x,y,z$ such that $xyz=1$ and $x,y,z\neq1$.
		\end{problem}

		\begin{problem}[IMO $2012$, problem $2$]
			Let $n\geq3$ be a positive integer and $a_2,\ldots,a_n$ be positive real numbers such that $a_1\cdots a_n=1$. Prove that
				\begin{align*}
					(1+a_2)^{2}\cdots(1+a_n)^{n}
						& \geq n^{n}
				\end{align*}
		\end{problem}

		\begin{problem}[IMO $2020$, problem $2$]
			Let $a,b,c,d$ be positive real numbers such that $a\geq b\geq c\geq d$ and $a+b+c+d=1$. Prove that
				\begin{align*}
					(a+2b+3c+4d)a^{a}b^{b}c^{c}d^{d}
						& < 1
				\end{align*}

				\begin{solution}
					Since $a+b+c+d=1$, by \nameref{thm:weightedpowermean} on $\omega=(a,b,c,d)$ and $\mathbf{a}=(a,b,c,d)$,
						\begin{align*}
							a\cdot a+b\cdot a+c\cdot c+d\cdot d
								& \geq a^{a}b^{b}c^{c}d^{d}
						\end{align*}
					So it is enough to prove that
						\begin{align*}
							(a+2b+3c+4d)(a^{2}+b^{2}+c^{2}+d^{2})
								& \leq (a+b+c+d)^{3}
						\end{align*}
					Expanding these we can easily see that the inequality has to follow. But we can prove it in a smarter way.
						\begin{align*}
							(a+b+c+d)^{3}
								& = (a+b+c+d)\left(a^{2}+b^{2}+c^{2}+d^{2}+2\sum ab\right)
						\end{align*}
					where the sum runs over all possible $\binom{4}{2}$ pairs. Then
						\begin{align*}
							(a+2b+3c+4d)(a^{2}+b^{2}+c^{2}+d^{2})
							& < (a+b+c+d)^{3}\\
							\iff (b+2c+3d)(a^{2}+b^{2}+c^{2}+d^{2})
								& < 2(a+b+c+d)\left(\sum ab\right)
						\end{align*}
					Now, using $a\geq b\geq c$,
						\begin{align*}
							a^{2}+b^{2}+c^{2}+d^{2}
								& \leq a(a+b+c+d)
						\end{align*}
					So, it is enough to show that
						\begin{align*}
							a(b+2c+3d)(a+b+c+d)
								& \leq  2(a+b+c+d)\left(\sum ab\right)\\
							\iff a(b+2c+3d)
								& \leq 2\sum ab\\
							\iff 3da
								& \leq ab+2ca\\
							\iff da+da+da
								& \leq ab+ca+ca
						\end{align*}
					This inequality obviously holds.
				\end{solution}

				\begin{remark}
					We could also use \nameref{thm:weightedjensen} after using the fact that $\log(x)$ is concave. The buffalo way works here as well. But the calculation is not going to be pretty if you go that way. This problem was highly criticized within some forums such as the Art of Problem Solving. It was the first inequality problem at the IMO since $2012$. A lot of people thought that the days of inequality at the IMO was over. But when this problem appeared at the IMO $2020$, many people complained and expressed their disappointment that the \textit{no inequality problem at the IMO} streak was finally broken with such a problem.
				\end{remark}
		\end{problem}

		\begin{problem}[IMO Shortlist $2015$, A1]
			Let $a,b,c$ be positive real numbers such that $\min\{ab,bc,ca\}\geq1$. Prove that
				\begin{align*}
					\sqrt[3]{(a^2+1)(b^2+1)(c^2+1)}
						& \leq\left(\dfrac{a+b+c}{3}\right)^2+1
				\end{align*}
		\end{problem}

		\begin{problem}[IMO Shortlist $2015$, A8]
			Determine the largest real number $a$ such that for all $n\geq1$ and for all real numbers $x_{0},\ldots,x_{n}$ satisfying
				\begin{align*}
					0
						& = x_{0}<x_{1}<\ldots<x_{n}
				\end{align*}
			we have
				\begin{align*}
					\dfrac{1}{x_{1}-x_{0}}+\dfrac{1}{x_{2}-x_{1}}+\ldots+\dfrac{1}{x_{n}-x_{n-1}}
						& \geq a\left(\dfrac{2}{x_{1}}+\dfrac{3}{x_{2}}+\ldots+\dfrac{n+1}{x_{n}}\right)
				\end{align*}
		\end{problem}

		\begin{problem}[IMO Shortlist $2018$, A7]
			Find the maximal value of
				\begin{align*}
					S
						& = \sqrt[3]{\dfrac{a}{b+7}}+\sqrt[3]{\dfrac{c}{d+7}}+\sqrt[3]{\dfrac{d}{a+7}}
				\end{align*}
			where $a,b,c,d$ are non-negative real numbers which satisfy $a+b+c+d=100$.
		\end{problem}
\end{document}


\subsection{Discrete and continous probability}

We know that:

\(\sum_yP(X\land Y)=P(X)\)

So for the continuous case

\(P(X)=\int_{-\infty }^{\infty }P(X\land Y)dy\)

This behaves like the probability for a single event, or multiple events with one fewer event if there were more than \(2\) events to start with.