\documentclass[10pt]{book}
\usepackage[german]{babel}
\usepackage[utf8]{inputenc}
\usepackage{fancyhdr}
\usepackage{amsmath}
\usepackage{calc}
\usepackage{listings}
\usepackage{hyperref}
\usepackage{tikz-network}
\usetikzlibrary{decorations.pathreplacing,calc}
\usepackage{multicol}

\newcommand{\ueberschrift}{How does a neural net, work?}

\title{\ueberschrift}
\author{Philipp Zettl}

\begin{document}
    \pagestyle{fancy}
    \maketitle

    At some point one can write a preamble to explain what a NN is, what to use it for
    and why it's actually better for us to use them then old numerical methods like the euler method to
    calculate the root(s) of a function

    \chapter{Neural Networks}
    Neural networks are a widely used technique to solve complex problems in a very simplified mathematical
    way. There are many different implementations of neural networks, but they all share the same core idea.

    \textbf{F}eed \textbf{f}orward \textbf{n}eural \textbf{n}etworks, a type or neural network, are a great entry point to start discovering
    neural networks. The following chapter will introduce the idea of general neural networks and use this motivation
    to discover the first type of neural networks.
    \section{Motivation}
    Unlike us humans computers have rather a hard time recognizing handwritten characters, digits or even full words.
    Humans taught themselves over millions of years of evolution how to recognize things.\\
    Since handwriting is mostly unique to a person one will have a hard time implementing an algorithm to detect digits from handwritten text,
    using an enormous amount of \lstinline{if/else} conditions will be the least problem, but figuring out how to provide a stable algorithmically solution
    to this problem seems impossible.
    Nowadays we can use neural networks to mock this behavior and create models to do that task for us.
    In the following chapter I will describe how to create such model as well as how to train it using real live
    examples.

    To train a NN one uses a so called \dq Training-Set\dq of data to \dq teach\dq the network what prediction to give for a given input. In some way we can assume
    achieving similar results using equal training data sets.\\
    But wait, how does a neural net, work?\\
    Now let us construct a basic neural network.
    \section{General NN}
    To define the structure of Neural Networks we need to define a bunch of
    technical terms. In biology we differentiate between different types of neurons.
    Perceptrons, a type of neuron, are basically binary decision makers having \(n\) input values and a single output value.
    We can illustrate them using the following notation.
    \begin{figure}[h]
        \begin{center}
            \begin{tikzpicture}
                \Vertex[x=0,y=0,label=$x_1$]{A}
                \Vertex[x=0,y=-1,label=$x_2$]{B}
                \Vertex[x=0,y=-2,label=$x_3$]{C}
                \Vertex[x=2,y=-1,label=$p_1$]{D}
                \Vertex[x=4,y=-1,label=$o_1$]{F}
                \Edge(A)(D)
                \Edge(B)(D)
                \Edge(C)(D)
                \Edge(D)(F)
            \end{tikzpicture}
        \end{center}
        \caption{Simple illustration of a single perceptron.\label{fig:Perceptron}}
    \end{figure}
    This model was introduced in 1958 by Frank Rosenblatt. He proposed a simple, yet complicate,
    rule to compute the output value of a perceptron. He introduced weights \(w_i\) to express the importance
    of an input value \(x_i\) to compute the output value \(o_i\).
    \begin{align}
        o_i =
        \left\{
            \begin{matrix}
                0 & \text{if } \sum_j w_ix_i \leq t\\
                1 & \text{if } \sum_j w_ix_i > t
            \end{matrix}
        \right.
        \label{eq:perceptron}
    \end{align}
    with \(t\) a threshold.
    Obviously a single perceptron is not even close to real human decision making and building
    a neural net out of a single perceptron seems simply too unflexible and stiff.
    But it works great to illustrate the idea behind a complex model without using too complicated notations,
    we can imagine a perceptron being a single element of a complex model to make a more detailed decision.\\
    Hence a NN in the from\\
    \begin{figure}[h]
        \begin{center}
            \begin{tikzpicture}
                \Vertex[x=-4,y=-0.5,Pseudo,label=$i_1$]{i1}
                \Vertex[x=-4,y=-1.5,Pseudo,label=$i_2$]{i2}

                \Vertex[x=-2,y=0]{A}
                \Vertex[x=-2,y=-1]{B}
                \Vertex[x=-2,y=-2]{C}

                \Vertex[x=0,y=1]{a}
                \Vertex[x=0,y=0]{b}
                \Vertex[x=0,y=-1]{c}
                \Vertex[x=0,y=-2]{d}
                \Vertex[x=0,y=-3]{e}

                \Vertex[x=2,y=-1]{D}
                \Vertex[x=4,y=-1,Pseudo,label=$o_1$]{F}

                \Edge(i1)(A)
                \Edge(i1)(B)
                \Edge(i1)(C)
                \Edge(i2)(A)
                \Edge(i2)(B)
                \Edge(i2)(C)

                \Edge(A)(a)
                \Edge(B)(a)
                \Edge(C)(a)

                \Edge(A)(b)
                \Edge(B)(b)
                \Edge(C)(b)

                \Edge(A)(c)
                \Edge(B)(c)
                \Edge(C)(c)

                \Edge(A)(d)
                \Edge(B)(d)
                \Edge(C)(d)

                \Edge(A)(e)
                \Edge(B)(e)
                \Edge(C)(e)

                \Edge(a)(D)
                \Edge(b)(D)
                \Edge(c)(D)
                \Edge(d)(D)
                \Edge(e)(D)
                \Edge(D)(F)
            \end{tikzpicture}
        \end{center}
        \caption{Simple illustration of a NN using multiple perceptrons.\label{fig:simpleNN}}
    \end{figure}\\
    will perform a more detailed analysis then (Figure \ref{fig:Perceptron}).\\
    A network consists of the input values \(i_j\), several columns of perceptrons – we will call them from now on layers – and the output \(o_j\).
    The above illustrated network contains 3 layers of perceptrons. The first layer is making three simple decisions out of the input data,
    then forwards these decisions to the second layer of perceptrons, which will make 5 decisions out of each of the previous 3 decisions by weighting their
    outputs. Before we can analyse the above displayed network in more detail we need to introduce a good notation, how to call a single perceptron.
    Say we have \(l > 0\) layers in our network and use \(n\) input values \(i_1, i_2, ..., i_n\) to predict \(m\) output values \(o_1, o_2, ..., o_m\).
    Then we call the \(i^{\text{th}}\) perceptron of the \(l^{\text{th}}\) layer
    \begin{align}
        p_i^l
    \end{align}

    Now let's take a more detailed look at our network and calculate the amount of decisions we actually perform.\\
    We have two input values \(i_1\) and \(i_2\), based on each we perform 3 decisions, so after processing the first layer \(l=1\) of perceptrons we made
    \begin{align}
        n_l\cdot \underbrace{n_{l-1}}_{=n} = 3\cdot 2 = 6
    \end{align}
    decisions. Moving on to the second layer we again count the number of perceptrons in the current layer \(n_2 = 5\) and
    multiply it with the amount of perceptrons in the previous layer
    \begin{align}
        \Rightarrow n_2 \cdot n_1 = 5 \cdot 3 = 15
    \end{align}
    And for the third layer \(l=3\) with one perceptron we perform \(5\) decision.\\
    We also need to include the last layer into the calculation with a single perceptron.
    Adding those values together \(6+15+5+1 = 27\), we actually perform 27 decisions for a single output.

    This makes us assume that increasing the amount of layers and perceptrons per layer will increase the complexity
    behind a decision and therefor a big neural network the way to compute complex and sophisticated decisions.

    In \eqref{eq:perceptron} we introduced a way how to calculate the output of a perceptron, we will change this notation
    a bit to achieve a way to calculate the outcome of a whole NN, not just a single perceptron of it.
    The previous notation of \(\sum_j w_jx_j > t\) can be written as a dot product \(w \cdot x\), where \(w\) and \(x\) are
    vectors whose components are the weights and input values respectively. The second change is to move the
    threshold from the right side of the inequality. But before we do that we rename it to its widely used name \dq Bias\dq, \(b = -t\)
    using these changes we can write a perceptron now using
    \begin{align}
        o_i =
        \left\{
            \begin{matrix}
                0 & \text{if } w\cdot x + b \leq 0\\
                1 & \text{if } w\cdot x + b > 0
            \end{matrix}
        \right.
        \label{eq:firstNN}
    \end{align}
    Think of the bias as how easy it is to get 1 out of the perceptron. Or in a more biological
    way, the bias is a measure of how easy it is to get the perceptron to \textit{fire}.\newline
    Previous descriptions of perceptrons were based on the idea that a perceptron implements a method of weighting
    input values to make decisions. A more CS way to use perceptrons would be to compute elementary logical functions
    such as \lstinline{AND, OR, NOR, XOR, ...} So we can for example build a network having the weights \(-2\) and the bias of \(3\), like this
    \begin{figure}[h]
        \begin{center}
            \begin{tikzpicture}
                \Vertex[x=0,y=0,label=$x_1$]{A}
                \Vertex[x=0,y=-2,label=$x_2$]{B}
                \Vertex[x=2,y=-1,label=$p_1$]{D}
                \Vertex[x=4,y=-1,label=$o_1$]{F}
                \Edge[label=$-2$](A)(D)
                \Edge[label=$-2$](B)(D)
                \Edge(D)(F)
            \end{tikzpicture}
        \end{center}
        \caption{Logical \lstinline{NAND} gate.\label{fig:NANDGate}}
    \end{figure}
    The above displayed NN \ref{fig:NANDGate} implements a so called \lstinline{NAND}-Gate.
    An input of \(x = \begin{pmatrix}0, 0\end{pmatrix}^T\) results in \((-2)\cdot 0 + (-2) \cdot 0 + 3 = 3 \Rightarrow o_1 = 1\).
    Similar calculations reveal that the inputs \(\begin{pmatrix}0, 1\end{pmatrix}^T, \begin{pmatrix}1, 0\end{pmatrix}^T\) produce the output
    \(1\), but \(\begin{pmatrix}1, 1\end{pmatrix}^T\) results in \((-2)\cdot 1 + (-2)\cdot 1 + 3 = -1\) therefor \(o_1 = 0\).
    To make it even more obvious lets take a look at a simple truth table from the first semester, some of you might even know that stuff since their born, anyway.
    \begin{center}
        \begin{tabular}{c|c|c|c}
            A & B & A\&B & !(A\&B)\\
            \hline
            1 & 0 & 0 & 1\\
            1 & 1 & 1 & 0\\
            0 & 0 & 0 & 1\\
            0 & 1 & 0 & 1\\
        \end{tabular}
    \end{center}
    as we can see the above mentioned results of our NN are equal to a regular \lstinline{NAND} operation.

    This \lstinline{NAND} example shows that we can use perceptrons to compute simple logical functions. In fact we can use
    perceptrons to compute \textbf{any} logical function. But the reason for this is not the flexibility or any other aspect of neural networks
    or perceptrons per se, this is due to the universal definition of logical operations.
    Since we can recreate these universal definitions it follows that a perceptron is a similar universal definition of an operation.
    This fact is actually quite disappointing because it seems it's merely possible achieve something more complex then combinations of \lstinline{NAND}
    gates and other logical operations. But on the other hand it might be reassuring to know that NNs are as powerful as modern computers.

    But don't take this introduction too serious, keep in mind we did not cover learning algorithms yet. Which open up
    a whole new world. We will use these learning algorithms to adjust weights and biases automatically within a network of
    artificial neurons. There are several methods of training all respond to external stimuli and do not require manual adjustments by
    a programmer. These algorithms allow us to use artificial neurons in a way which is radically different from
    logical gates. Instead of explicitly laying out a logical function our networks can simply learn to solve problems, which are way beyond
    simple conventional logical gates.

    \section{Activated Neurons}
    Writing a learning algorithm sounds scary on first glance, but how can one achieve such behavior for a neural network?
    Suppose we have a network of perceptrons as previously described which we want to use to solve a problem. For example, the inputs
    to the network might be raw pixels from a scanned, handwritten image of a digit. And we'd like our network to learn it's weights and biases so that the
    output correctly classifies the digit within the image. To illustrate how learning might work we make small adjustments to the weights (or biases) of the network.
    We want those changes in the weights to be small to cause only small changes in the corresponding output from the network. Those changes will be called
    \(\Delta w, \Delta b\) and \(\Delta o\). Suppose we input an image of a 9 and the network classifies this input as a 8, we then could figure out how to make a small
    change to the weights or biases so the network gets a bit closer to classify the image as a 9. We then repeat this process over and over again. The network is learning.

    Unfortunately our current definition of neural networks is not able to achieve this behavior using the perceptron model. Doing small adjustments on their weights and values
    will cause big changes in the networks output. This is due to the restriction in the binary output of perceptrons. Sometimes a change in a single weight flips a perceptrons output
    and the results in a complete different output of the network. So while the expected 9 might be classified correctly the behavior of the network in respect to other inputs
    might vary drastically. That makes it difficult to see how to gradually modify the weights and biases so the network achieves expected results.

    We simply overcome this issue with introducing a new type of neuron, the sigmoid neuron. Sigmoids are similar to perceptrons, with the small difference that small changes in their weights and biases cause only
    small changes in the networks output. That's a necessary feature to allow networks of sigmoids to learn. We depict sigmoids the same way as perceptrons.

    \begin{figure}[h]
        \begin{center}
            \begin{tikzpicture}
                \Vertex[x=0,y=0,label=$x_1$]{A}
                \Vertex[x=0,y=-1,label=$x_2$]{B}
                \Vertex[x=0,y=-2,label=$x_3$]{C}
                \Vertex[x=2,y=-1,label=$p_1$]{D}
                \Vertex[x=4,y=-1,label=$o_1$]{F}
                \Edge(A)(D)
                \Edge(B)(D)
                \Edge(C)(D)
                \Edge(D)(F)
            \end{tikzpicture}
        \end{center}
        \caption{Illustration of a Sigmoid Neuron.\label{fig:Sigmoid}}
    \end{figure}
    Just like the perceptron the sigmoid neuron gets \(n\) input values \(x_i, i=1, ...,n\) but instead of forcing those values to be binary,
    they can be any number between \(0\) and \(0\), so \(x_i \in [0, 1]\) as well as the perceptron the sigmoid neuron has a weight \(w\) for each input and overall biases \(b\). But it's output is
    again not binary. Instead it's \(\sigma(w \cdot x + b)\), where \(\sigma\) is the \textit{sigmoid} function (also called logistical function),
    \begin{multicols}{2}
        \includegraphics[width=.45\textwidth]{sigmoid.eps}

        \columnbreak
        \vspace*{.6cm}

        \begin{align}
            \sigma(z) = \frac{1}{1+e^{-z}}
        \end{align}

    \end{multicols}
    to be more precise the output of a sigmoid neuron is
    \begin{align}
        \frac{1}{1+e^{-w \cdot x - b}}
    \end{align}
    To understand the similarity between the previously discussed perceptron neuron and the sigmoid neuron, suppose
    \(z = w \cdot x + b\) is a large positive number. Then \(e^{-z} \approx 0\) so \(\sigma(z) \approx 1\). Or just
    \dq Is \(z\) large and positive, the output is approximately 1\dq, just like for the perceptron. So \(\sigma\) is a way
    to enforce values being \(\in [0, 1]\). One can achieve the behavior of perceptrons by redefining \(\sigma\) as in \eqref{eq:firstNN}.
    So Sigmoid Neurons are \dq just\dq smoothened perceptrons. But exactly this smoothening is the crucial fact which allows learning.
    This smoothness means that small changes in weights \(\Delta w\) and biases \(\Delta b\) will produce a small change in the output \(\Delta o\).
    Hence we achieve the same behavior of perceptrons with the additional effect of being able to predict the change in output which will occur while
    modifying a weight or bias within the network.
    In fact, calculus tells us how to calculate that change as
    \begin{align}
        \Delta o \approx \sum \limits_{j=1}^n \frac{\partial o}{\partial w_j} \Delta w_j + \frac{\partial o}{\partial b} \Delta b
    \end{align}
    where \(\frac{\partial o}{\partial w_j}\) denote partial derivatives of the output with respect to \(w_j\) and \(b\) respectively.
    This might sound overwhelming in the beginning and partial derivatives aren't that easy to understand. But what we can see from here
    is that \(\Delta o\) is a linear function, since it's a combination of the linear function \(\Delta w\) and \(\Delta b\). Therefor
    a small change in weights or biases will cause only a small change in the output. So while sigmoid neurons have much of the same quality
    as perceptrons, they make it easier to figure out which weights and biases to adjust in order to achieve a small change of the output
    into a desired direction.\newline

    \section{Summary}
    We've discussed two types of neurons in this chapter. The perceptron which is ideal to reconstruct logical
    functions and the sigmoid neuron, which has exactly the same features as the perceptron
    with the addition that we can perform small adjustments to modify the used weights and biases in order
    to \dq train \dq the model. This brings one big conclusion. When we talk about neurons we can assume these given values:
    \begin{enumerate}
        \item \(n:=\) Number input elements
        \item \(m:=\) Number output elements
        \item \(n_l:=\) Number desired \dq layers\dq of neurons
        \item Type of neurons in layer \(l\)
    \end{enumerate}
    Depending on the type of neuron, the behavior differs. But what is the actual difference between the sigmoid neuron and the perceptron?
    The answer is quite easy. The main and mostly only difference is their activation function.
    \begin{figure}[h]
        \includegraphics[width=\textwidth]{activation_functions.eps}
    \end{figure}
    Both clamp the values \(o \in [0,1]\), but the smoothness of the sigmoid function allows learning behavior.

    \subsection{Important things to remember}
    Imagine having a big network of perceptrons. Multiplying every weight and bias within this network with a constant \(c > 0\) will \underline{\textbf{not}} change the overall network output.

    \section{Network architecture}
    Over the next section we will introduce a neural network that can do a pretty good job classifying handwritten digits. It helps to first introduce some terminology
    that lets us name different parts of the network.
    Suppose we have the previously mentioned network

    \begin{figure}[h]
        \begin{center}
            \begin{tikzpicture}
                \Vertex[x=-4,y=-0.5,Pseudo,label=$i_1$]{i1}
                \Vertex[x=-4,y=-1.5,Pseudo,label=$i_2$]{i2}

                \Vertex[x=-4,y=-4,Pseudo]{B0}

                \Vertex[x=-2,y=0]{A}
                \Vertex[x=-2,y=-1]{B}
                \Vertex[x=-2,y=-2]{C}

                \Vertex[x=-2,y=-4,Pseudo]{B1}

                \Vertex[x=0,y=1]{a}
                \Vertex[x=0,y=0]{b}
                \Vertex[x=0,y=-1]{c}
                \Vertex[x=0,y=-2]{d}
                \Vertex[x=0,y=-3]{e}

                \Vertex[x=0,y=-4,Pseudo]{B2}

                \Vertex[x=2,y=-1]{D}
                \Vertex[x=2,y=-4,Pseudo]{B3}
                \Vertex[x=4,y=-1,Pseudo,label=$o_1$]{F}
                \Vertex[x=4,y=-4,Pseudo]{B4}

                \Edge(i1)(A)
                \Edge(i1)(B)
                \Edge(i1)(C)
                \Edge(i2)(A)
                \Edge(i2)(B)
                \Edge(i2)(C)

                \Edge(A)(a)
                \Edge(B)(a)
                \Edge(C)(a)

                \Edge(A)(b)
                \Edge(B)(b)
                \Edge(C)(b)

                \Edge(A)(c)
                \Edge(B)(c)
                \Edge(C)(c)

                \Edge(A)(d)
                \Edge(B)(d)
                \Edge(C)(d)

                \Edge(A)(e)
                \Edge(B)(e)
                \Edge(C)(e)

                \Edge(a)(D)
                \Edge(b)(D)
                \Edge(c)(D)
                \Edge(d)(D)
                \Edge(e)(D)
                \Edge(D)(F)
                \draw[-,decorate,decoration={brace,mirror}] (B0)-- node[below]{Input layer} (B0);
                \draw[-,decorate,decoration={brace,mirror}] (B1)-- node[below]{Hidden layers} (B3);
                \draw[-,decorate,decoration={brace,mirror}] (B4)-- node[below]{Output layer} (B4);
            \end{tikzpicture}
        \end{center}
        \caption{Simple illustration of a NN.\label{fig:NN}}
    \end{figure}
    The most left layer is called the input layer and the neurons within this layer are called \textit{input neurons}.
    The outermost right layer is called the output layer, and as you can guess it's neurons are called \textit{output neurons}, in this case a single neuron.
    The layers in between are called \textit{hidden} layers since their neurons are neither input nor output neurons. The term \dq hidden\dq might sound magical or
    deeply philosophical the first time you hear about it, but it's nothing else then another word for \dq not an input/output\dq.
    This might be somewhat confusion, and I just mention it for historical reason, some people call these networks
    MLPs, multiple layer perceptron networks, despite the fact that some of them hold sigmoid neurons, this should be mentioned
    so you're aware of that term.\newline
    \textbf{Important to note, I might call sigmoid neurons every now and then activated neurons, this is due to the nature of sigmoid neurons which use the sigmoid function as activation function.
    Whenever this activation function differs I will explicitly name this activation function. Other then that activated neurons are sigmoid neurons.}

    \subsection{Input and output layers}
    The design for input and output layers is straight forward. Depending on the problem we want to solve we
    figure out the number of input values. For the example of classifying handwritten digits we have a dataset of 64x64 px images
    therefor we have \(n = 64 \cdot 64 = 4096\) input parameters, which need to be clamped into \([0,1]\). To normalize the data we can use
    \begin{align}
        \tilde{x_i} = \frac{x_i - \min(x)}{\max(x)-\min(x)}
    \end{align}
    The output layer will contain \(m\) neurons, to simplify for now we want to classify if a picture shows a 9 or not, so a single neuron with values
    \(o \leq 0.5\) indicating it is \underline{not} a 9 and \(o > 0.5\) that the input image is a 9.

    \subsection{Hidden Layers}
    Albeit the design of input- and output layers is straight forward, it is kind of an art to architect hidden layers.
    To be more precise it is impossible to state a rule of thumb for hidden layer architectures. Instead researchers have found
    several heuristics to describe models which help solving different problems.
    As an example for the use of these heuristics they can be used to determine how to trade off hidden layers against the time required training the model.
    We'll meet several such design heuristics later.

    \subsection{Prospects}
    For now we learned about networks of layers of neurons where the output of one layer is used as the input of the next layer.
    These networks are called \textbf{Feed Forward Neural Networks} (FFNNs). The net works in a single operation forward feeding the input
    values to gain output values. No loops are involved. Actually it would be very hard to achieve loops with this design, so we don't allow them.
    However there are other designs of neural networks which allow these kind of designs.

    These models are called \textbf{Recurrent Neural Networks} (RNNs).
    The idea in these models is to have neurons which fire for some limited duration of time, before becoming quiescent. That firing can stimulate other neurons, which may fire a little while later, also for a limited duration. That causes still more neurons to fire, and so over time we get a cascade of neurons firing. Loops don't cause problems in such a model, since a neuron's output only affects its input at some later time, not instantaneously.
    Recurrent neural nets have been less influential than feedforward networks, in part because the learning algorithms for recurrent nets are (at least to date) less powerful. But recurrent networks are still extremely interesting. They're much closer in spirit to how our brains work than feedforward networks. And it's possible that recurrent networks can solve important problems which can only be solved with great difficulty by feedforward networks.
    However, to limit our scope, we're going to concentrate for now on the more widely-used feedforward networks.

    \chapter{NN to classifying handwritten digits}
    After this long and very dry introduction to neural networks, lets get our hands dirty and build something fun!
    In the following chapter we will implement the example we previously used to motivate the idea behind NNs. Classifying
    handwritten digits.
    \section{Problem description}
    We can split the problem of recognizing handwritten digits into two sub-problems.
    First we want to split an image of multiple
    handwritten digits into separate images, each containing a single digit.

    \begin{multicols}{2}
        So for example we want this number\\
        \begin{center}
            \includegraphics{digits-complete.eps}
        \end{center}
        \columnbreak

        To be split into 3 seperate images like\\
        \begin{center}
            \includegraphics{digits-split.eps}
        \end{center}
    \end{multicols}
    A human solves this quite easily, but we want to solve this using the computer.
    Once the image has been segmented, the program then needs to classify each digit individually.
    So for example we'd like the program to recognize the first digit in the image above to be 4.

    For now we focus on writing a program to solve the second problem. There are many approaches to solve
    the segmentation problem, but after we discussed the classification you will have a good idea how to tackle
    the segmentation. One method can be to generate many \dq trial\dq \; segmentations and use an individually trained
    classifier to score each segmentation. A trial gets a higher score when the classifier is more certain about the classification
    and a lower score whenever it is uncertain about the classification. The idea is that if the classifier has trouble classifying then
    the segmentation is not good enough. So instead of worrying about the first problem in the beginning, we assume for now that we have
    in fact already segmented training sets and can come back to the segmentation model later.

    To solve the more interesting problem, to namely classify handwritten digits we will use a three-layer neural network.
    It consists of one input layer having \(784\) input neurons (one for each pixel of a 28x28 px image), a hidden layer with \(15\) neurons and an output layer with \(10\) output neurons (one for each digit 0-9 ;-) )

    The input pixels are greyscale, with values of \(0.0\) representing white and \(1.0\) black. Values in between are different shades of grey.

    You might wonder why the output layer consists of 10 neurons and not the minimum amount required being 4. In fact using 4 neurons can be a lightweight solution
    here since we can encode the output as a binary value since \lstinline{111}\({}_2=7< n_o <13=\) \lstinline{1111}\({}_2\). The ultimate justification is empirical
    so let's try out both approaches later on and compare the results, but it will turn out that the network with 10 output values will achieve better results then the
    network using 4 output values. That leaves us with the big question: Does the number of output values influence the general behavior of the network?
    Is there maybe some heuristic which gives us a direction to chose the number of output values?
    And can this heuristic possibly tell us upfront which solution fits better?

    Many resources online and books start now to describe how NNs work by using a explicit example. Most of them start splitting the image of a
    digit into several parts and justify the way how the neurons \dq learn\dq their weights/biases by using these segments and saying \dq This weight is for the top arch of this digit...\dq but
    that's just a useless simplification. Imagine that in fact weights represent features of the images fed into the network during the training period. But the odds that you will chose the right weight
    which actually describes this particular feature is very low. Imagine weights and biases as feature parts of the fed data. This can be a line, a cloud of pixels anything within the data.
    But their explanation motivates the decision to chose 10 output neurons, by claiming
    \begin{quote}
        If the combination of hidden neurons is responsible to classify multiple digits it will get very hard for the output neuron to figure out
        which digit do chose from, since \lstinline{1000} and \lstinline{1001} aren't that different. But \lstinline{0000000001} and \lstinline{0000000010} do very much.
    \end{quote}
    This description works as an heuristic and is widely used. Although I disagree with the detailed description it is a nice simplification to remember.

    Note that in case you want to achieve the same results as the 10 output neuron model, you can add an additional layer to the end of your model having 4 output neurons.
    You can use the perceptron neurons for that and manually precalculate the weights for it.

    \section{Gradient decent}
    After we defined the design of our network, how can we train it?\\
    Hold your horses, first we need a dataset which we want to use to train our model. For this purpose we use
    the dataset provided by MNIST for \href{http://yann.lecun.com/exdb/mnist/}{handwritten digits}.
    The MNIST dataset comes in split into two parts, training with 60k images of handwritten digits by 25 individuals. The images are greyscaled and 28 by 28 pixel in size.
    The second part is a test set containing 10k images with the same features.
    We will use the test set to validate the quality of our network. To make it a good test set it has been acquired by
    250 individuals. This helps building confidence in the current state during the training phase.

    What we'd like is an algorithm which lets us find the weights and biases so that the output approximates \(o(x)\) for all training inputs \(x\). To quantify how well
    we're achieving this goal at the moment we define the \textit{cost}-function:
    \begin{align}
        C(w,b) = \frac{1}{2n}\sum \limits_x^n || o(x) - a ||^2
        \label{eq:cost}
    \end{align}
    \(w\) denotes the collection of all weights in the network, \(b\) all the biases, \(n\) the total number of training inputs, \(a\) is a vector of outputs from the network when \(x\) is input (so the current estimation)
    and the sum is over all training inputs \(x\). We call \(C\) the quadratic cost function or mean square error (MSE). Nice to see is that for \(o(x) \approx a\) we get \(C\approx 0\) and for big differences between \(o(x)\) and \(a\)
    we get big values for \(C\). This is nice! Whenever the model is good the cost is significantly lower then whenever we do a bad prediction.
    So the aim of training is to minimize \(C\), in other words we want the weights and biases that produce the least possible \(C\).
    We do that using the \textbf{gradient decent} algorithm.

    Why introduce the quadratic cost? After all, aren't we primarily interested in the number of images correctly classified by the network? Why not try to maximize that number directly, rather than minimizing a proxy measure like the quadratic cost? The problem with that is that the number of images correctly classified is not a smooth function of the weights and biases in the network. For the most part, making small changes to the weights and biases won't cause any change at all in the number of training images classified correctly. That makes it difficult to figure out how to change the weights and biases to get improved performance. If we instead use a smooth cost function like the quadratic cost it turns out to be easy to figure out how to make small changes in the weights and biases so as to get an improvement in the cost. That's why we focus first on minimizing the quadratic cost, and only after that will we examine the classification accuracy.

    It turns out that we can understand a tremendous amount by ignoring most of that structure, and just concentrating on the minimization aspect. So for now we're going to forget all about the specific form of the cost function, the connection to neural networks, and so on. Instead, we're going to imagine that we've simply been given a function of many variables and we want to minimize that function. We're going to develop a technique called gradient descent which can be used to solve such minimization problems. Then we'll come back to the specific function we want to minimize for neural networks.

    Gradient decent is an algorithm which you can imagine the following.
    You have a plane with minima and maxima in it, some places are higher some are lower, now you place a ball on that plane and gradient decent will find for you the minima of your plane. It uses
    derivatives of the function on which to calculate the extreme on to ensure convergence into the direction of the minima.

    So how can we apply it to neural networks? The idea is to use it to fit the models weights \(w_k\) and biases \(b_l\) during the training phase and therefor to minimize the cost function \(C\).
    Writing the gradient decent update rule in terms of components we get
    \begin{align}
        w_k' &= w_k - \eta \frac{\partial C}{\partial w_k}\\
        b_l' &= b_l - \eta \frac{\partial C}{\partial b_l}
        \label{eq:gradient-decent}
    \end{align}
    By repeatedly applying this update rule we can "roll down the hill", and hopefully find a minimum of the cost function. In other words, this is a rule which can be used to learn in a neural network.\newline
    There are several challenges applying the gradient decent rule. In following chapters we will take a look at some of them in depth.
    But for now I just want to mention one problem. To understand the problem we need to take a look at \eqref{eq:cost} again.
    To compute the proper gradient of \(C\), \(\nabla C\), we need to compute the gradients \(\nabla C_x\), separately the cost for each input \(x\). This will become a big problem
    for big training sets. Hence we need to use a different approach to calculate \(\nabla C\).

    The solution for this problem is simple, we use statistical tools to calculate the gradient, to be precise we use the stochastic gradient decent.
    The idea is to make a proper estimation of \(\nabla C\) by computing \(\nabla C_x\) for a small, random, subset of the training set.
    By averaging over the small subset we can achieve a pretty well result for the true gradient \(\nabla C\), which helps speeding up the computation of the gradient as well as the time spent to train the model.

    To give a proper explanation, stochastic gradient decent is using a small subset of the training set with \(m\) randomly chosen elements \(X_1, X_2, ..., X_m\), we refer to them as a mini-batch. For a sample size
    \(m\), which needs to be large enough, but not too large, we expect that the average value for \(\nabla C_{X_j}\) will be roughly equal to the over all average of \(\nabla C_x\), or
    \begin{align}
        \underbrace{\frac{\sum_{j=1}^m \nabla C_x}{m}}_{\text{Gradient of mini batch}}
        \approx \underbrace{\frac{\sum_x \nabla C_x}{n}}_{\text{Gradient of complete training set}}
        = \underbrace{\nabla C}_{\text{overall gradient}}
    \end{align}
    Hence we can use the gradient of the mini-batch to estimate the overall gradient!
    Applied to the cost calculation for weights and biases \eqref{eq:gradient-decent} we get

    \begin{align}
        w_k' &= w_k - \frac{\eta}{m} \sum_j \frac{\partial C_{X_j}}{\partial w_k}\\
        b_l' &= b_l - \frac{\eta}{m} \sum_j \frac{\partial C_{X_j}}{\partial b_l}
    \end{align}
    where the sums are over all the training examples \(X_j\) of the mini-batch. Then we pick out the next \(m\)
    elements for the next mini-batch. And so on, until we exhausted the whole dataset, which then completes an \textit{epoch}.
    At that point we start over with a new training epoch.\newline
    It's worth noting that the implementation of \eqref{eq:cost} might vary. We scaled the cost function using the factor \(\frac{1}{n}\) people sometimes
    omit this scaling and compute the sum of cost of the current mini-batch. In a similar way the equations \eqref{eq:gradient-decent} sometimes omit the term \(\frac{1}{m}\).
    Conceptually this makes little difference in the result, since it's equivalent of scaling the learning rate \(\eta\).

    \section{Implementation}
    Alright, lets get them hands dirty and implement a program which learns how to recognize handwritten digits using gradient decision and the MNIST dataset.

    Download the repository

    \begin{lstlisting}[basicstyle=\tiny]{language=bash}
git clone https://github.com/MichalDanielDobrzanski/DeepLearningPython35
    \end{lstlisting}
    which contains python 3 compatible code for a neural network as well as the training dataset by MNIST.
    Initially I stated that we split the training set into 60k training and 10k testing sets, but that's the official
    dataset form. We'll do it a bit different and keep the test set as is and split the training set into 50k training and 10k
    validation sets. We won't use the validation set for now, but later on it will come in handy.

    Check out the code in the repository, II will only explain certain features of it, so it's quite necessary that you understand how to read python code.
    The heavy lifting in form of mathematical operations is mostly lifted by the library \lstinline{numpy}. But no worries, I will explain implementation parts of the code whenever it's necessary and
    will provide the mathematical representation in those cases to enable the reader to implement the example in any programming language.

    So without any further ado, here goes nothing!

    For now the main focus is on \lstinline{network.py}. Let's run the statistic gradient decent training algorithm.
    In order to do so, include the \lstinline{network} and \lstinline{mnist_loader} scripts into  a python shell.
    Afterwards use \lstinline{lnist_loader.load_data_wrapper()} method to
\begin{lstlisting}{language=Python}
>>> import network
>>> import mnist_loader
>>> training_data, validation_data, test_data \
 = mnist_loader.load_data_wrapper()
>>> nn=network.Network([784,30,10])
>>> nn.SGD(training_data, 30, 10, 3.0, test_data=test_data)
\end{lstlisting}
%http://neuralnetworksanddeeplearning.com/chap1.html#MathJax-Span-788

    \section{Backpropagation}
    \subsection{Notation}

    Das Gewicht \(w\), welches die Verbindung zwischen dem \(k\)-ten Neuron des \(l-1\)-ten Layers und dem \(j\)-ten Neuron des \(l\)-ten Layers
    wird geschrieben als

\end{document}


\documentclass[12pt, fullpage,letterpaper]{article}

\usepackage[margin=1in]{geometry}
\usepackage{url}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{xspace}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{cancel}
%\usepackage{graphicx}
%\usepackage{subfig}

\newcommand{\semester}{Spring 2022}
\newcommand{\assignmentId}{4}
\newcommand{\releaseDate}{31 March, 2022}
\newcommand{\dueDate}{11:59pm, 15 April, 2022}
\newcommand\independent{\protect\mathpalette{\protect\independenT}{\perp}}
\def\independenT#1#2{\mathrel{\rlap{$#1#2$}\mkern2mu{#1#2}}}

\newcommand{\bx}{{\bf x}}
\newcommand{\bw}{{\bf w}}

\title{CS 6190: Probabilistic Machine Learning \semester}
\author{Homework \assignmentId}
\date{Handed out: \releaseDate\\
  Due: \dueDate}

\begin{document}
\maketitle

\input{emacscomm}
\footnotesize
	\begin{itemize}
		\item You are welcome to talk to other members of the class about
		the homework. I am more concerned that you understand the
		underlying concepts. However, you should write down your own
		solution. Please keep the class collaboration policy in mind.

		\item Feel free discuss the homework with the instructor or the TAs.

		\item Your written solutions should be brief and clear. You need to
		show your work, not just the final answer, but you do \emph{not}
		need to write it in gory detail. Your assignment should be {\bf no
			more than 10 pages}. Every extra page will cost a point.

		\item Handwritten solutions will not be accepted.

		\item The homework is due by \textbf{midnight of the due date}. Please submit
		the homework on Canvas.
	\end{itemize}

%GMM
%Bayes logistic regression, VB, Laplace, Laplace Hession,
%LDA

\section*{Practice  [100 points + 100 bonus]}
\label{sec:q1}
\begin{enumerate}
	\item~[20 points] Suppose we have a scalar distribution,
	\[
	p(z) \propto \exp(-z^2)\sigma(10z + 3).
	\]
	\begin{enumerate}
		\item~[3 points] Although the normalization constant is not analytical, we can use Gauss-Hermite quadrature to calculate an accurate approximation. Please base on the example in ``data/example-code/gmq\_example.py'', calculate the numerical approximation of the normalization constant, and report its value. With the normalization constant, please draw the density curve of $p(z)$, in the range $z \in [-5, 5]$.
		\item~[5 points] Implement the Laplace approximation of $p(z)$, and report the mean and variance of your Gaussian distribution. Draw the density of your Laplace approximation in the same plot as in (a).
		\item~[10 points] Use the local variational inference method and EM-style updates as we discussed in the class (for logistic regression) to implement the variational approximation to $p(z)$. Report the form of your approximate distribution, and draw its density in the same plot as above.
		\item~[2 points] By comparing the ``ground-truth'' (from (a)) and the approximations (from (b,c)), what do you observe and conclude?
	\end{enumerate}

	\item~[50 points] Let us work on a real-world dataset we have met before. Please download the data from the folder``data/bank-note''. The features and labels are listed in the file ``data-desc.txt''. The training data are stored in the file ``train.csv'', consisting of $872$ examples. The test data are stored in ``test.csv'', and comprise of $500$ examples. In both the training and testing datasets, feature values and labels are separated by commas. We assign the feature weight vector $\w$ a standard normal prior $\N(\0, \I)$.
	\begin{enumerate}
		\item~[7 points] Implement the standard Laplace approximation to the posterior distribution of the feature weight vector. Report your approximate posterior. Now, use Gauss-Hermite quadrature to implement the calculation of the predictive distribution. Please be careful: \textbf{you need to do a proper variable transformation in the integral before applying the Gauss-Hermite quadrature because you integrate  with a Gaussian like $\N(x|\mu, \sigma^2)$ rather than $\exp(-x^2)$!} Now we test the performance with two measures. First, we calculate the inner-product between the posterior mean of the weight vector and the feature vector of each test example, and throw the inner-product into the sigmoid function to calculate the probability that the test example is positive. If the probability is no less than 0.5, we classify the example to be positive (\ie 1) otherwise we classify the example to be negative (\ie 0). Report the prediction accuracy. Second, we calculate the average predictive likelihood of the test samples, namely we evaluate the predictive density value of each test sample based on the predictive distribution and then take an average.  Note that in Bayesian learning, the predictive likelihood includes all the information of the (approximate) posterior distribution, hence is more preferred in the evaluation.
		\item~[3 points] Implement Laplace approximation with the diagonal Hessian. Report the approximate posterior distribution of the feature weights, the prediction accuracy and average predictive likelihood.
		\item~[20 points] Implement variational logistic regression we introduced in the class. Use EM-style updates. Report the variational posterior of the feature weight vector you obtained (\ie a multivariate Gaussian). Report the prediction accuracy and average predictive likelihood.
		\item~[15 points] Implement variational logistic regression we introduced in the class.  But this time, you will use the fully factorized posterior,   $q(\w)= \prod_i q(w_i)$ where $w_i$ is $i$-th element in the weight vector $\w$. In the E step, please use the standard mean-field update to alternatively optimize each $q(w_i)$ given all the others fixed. Report your variational posterior (\ie diagonal Gaussian), the prediction accuracy and average predictive likelihood on the test data.

		\item~[5 points] Compare the results of the above four approximations. What do you observe and conclude?
	\end{enumerate}

	\item~[30 points] Gaussian Mixture Model (GMM). Please download the data ``data/faithful/faithful.txt''  from Canvas. Each row is a sample, including 2 features. Please normalize the features in each column to be in [-1, 1]. Specifically, denote the column by $\x$; then we compute for each $x_i \leftarrow (x_i - \mathrm{mean}(\x))/(\max(\x) - \min(\x))$.
	\begin{enumerate}
		\item~[20 points] Implement EM algorithm for GMM. Set the number of clusters to $2$. Initialize the cluster centers to be [-1, 1] and [1,-1], and the covariance matrix to be $0.1\cdot\I$ for both clusters. Run your EM algorithm for $100$ iterations. For iteration $1$, $2$, $5$, $100$, please draw the figures showing the corresponding cluster centers and memberships. Specifically, for each figure, first draw the scatter plots of the data points and your cluster centers. Each data point is assigned to the cluster that has a great posterior probability to include that data point. Please draw the cluster memberships with different colors.
		\item~[7 points] Now initialize the cluster centers to be [-1, -1] and [1, 1] and covariance matrix to be $0.5 \cdot \I$ for both clusters. Run your EM algorithm for $100$ iterations. Draw the figures showing the cluster centers and memberships for iteration $1$, $2$, $5$, $100$.
		\item~[3 points] Compare the results in (a) and (b), what do you observe and conclude?
	\end{enumerate}

	\item~[100 points][\textbf{Bonus}] Latent Dirichlet Allocation (LDA). Please download the pre-processed corpus from ``data/lda''. From ``ap.txt'', you can see the original Associated Press corpus. ``ap.dat'' are the processed data which you will work on. Each row in ``ap.dat'' represents a document. In total, we have $2,246$ documents. The first number in each row is the number of words in the document. The following are a list of  \textbf{word-id}:\textbf{count} items. Word-id starts with 0. The corresponding word list is given in ``vocab.txt''. The first row correspond to Word-id 0, second, Word-id 1, and continue.

	\begin{enumerate}
		\item~[70 points] Implement the mean-field variational EM algorithm for LDA inference as we discussed in the class. Following the orignal LDA paper ( \url{http://www.cs.columbia.edu/~blei/papers/BleiNgJordan2003.pdf}) to implement the perplexity calculation on test documents (Sec. 7.1). Please randomly select $10\%$ documents as the test set, and run your LDA inference algorithm on the remaining $90\%$ documents. Vary the number of topics from \{5, 10, 15, 20, 50, 100, 200\}. Run your algorithm until convergence or $500$ iterations have achieved. Draw a figure to show how the perplexity vary along with the growth of the topic numbers. What do you observe and conclude?
		\item~[30 points] Set the number of topics to $20$ and run your variational inference algorithm. Examine the top 15 words (\ie with the largest probability) in each learned topic distribution. List a few topics which you think is semantically meaningful and explain why.
	\end{enumerate}
\end{enumerate}

\end{document}


\documentclass{article}
\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{February 3, 2014}
\maketitle
\section*{last time}
\section*{lesson 7}
\subsection*{step 1}
solve pde for fundamental solutions
\begin{align*}
  u&=T(t)X(x)\\
  \frac{T'(t)}{\alpha^2T(t)}&=\frac{X'(x)}{X(x)}=\text{separation constant}\\
  \intertext{phyysical considerationsindicatethat solutionsdeca to 0. simplify by assigning sep constant $\leq0$}\\
  \frac{T'(t)}{\alpha^2T(t)}&=\frac{X''(x)}{X(x)}=-\lambda^2\leq0 &T(x)X(x)=e^{-\alpha^2\lambda^2t}\\
\end{align*}
\subsection*{step 2}
solve pde and bc. solve $X''(x)+\lambda^2X(x)=0 with X(0)=0$.

\subsubsection*{case $\lambda=0$}
\begin{align*}
  X''(x)&=0\\
  X&=ax+b\\
  X(0)&=0 \text{ gives} b=0\\
  X'(1)+hX^{(1)}&=0\\
  \intertext{only trivial solution$X=0$}
  X&=ax
  \intertext{ohter BC gives $a(h+1)=0$ give $a=0$}
\end{align*}
\subsubsection*{case $\lambda>0$}
\begin{align*}
  X(x)&=a\sin(\lambda x)+b\cos(\lambda x)\\
  X(0)&=0\to b=0\\
  &=a\sin(\lambda x)\\
  X'(1)+hX(1)&=0\\
  \lambda\cos(\lambda)+h\sin(\lambda)&=0\\
  X'(x)&=a\lambda\cos(\lambda x)\\
  \intertext{this is on page 52}
  a\lambda\cos(\lambda)+ha\sin(\lambda&=0\\
  a(\lambda\cos(\lambda)+h\sin(\lambda))&=0\\
\end{align*}
question: can $\cos(\lambda)=0$? no! then $\sin(\lambda)=0$. but $\cos^2+\sin^2=1$. Sothe equation becomes $\tan\lambda)=-\frac{\lambda}{h}$. And from the graph we see that there are infinitely many positively roots $0<\lambda<\lambda_2<\lambda_3<\dots$.
\begin{align*}
  \frac{\pi}{2}&<\lambda_1<\pi\\
  \frac{3\pi}{2}&<\lambda_2<2\pi\\
  n\pi-\pi/2&<y_n<n\pi\\
  \lambda_n-(n\pi-\pi/2)&\to0 \text{ as } n\to\infty
\end{align*}
so back to the case:
\begin{align*}
  X(x)=a_n\sin(\lambda_nx)\\
  \intertext{where $\lambda_n$ is positive root to $\tan(\lambda)=-\lambda/h$}
  \intertext{find solutions}
  u_n&=e^{-\alpha^2{\lambda_n}^2t}a_n\sin(\lambda_nx)\text{ with }\tan(\lambda_n)=-\lambda_n/h\\
  \text{so }&\sum\limits_{n=1}^\infty{a_ne^{-\alpha^2{\lambda_n}^2t}\sin(\lambda_nx)}
\end{align*}
\subsection*{step 3}
find the coefficients $a_n$: want $\sum\limits_{n=1}^\infty{a_n\sin(\lambda_nx)}=\phi(x)$ on $0<x<1$.

recall in lesson 5: $\sum\limits_{n=1}^\infty{a_sin(n\pi x)=\phi(x)}\left\{\begin{aligned}u(0,t)=0\\u(1,t)=0\end{aligned}\right\}$.
Used othagonality $\int_0^1{\sin(m\pi x)\sin(n\pi x)\,\mathrm{d}x}$ for $m\neq n$
\begin{align*}
  \int_0^1{\sin(m\pi x)\sin(n\pi x\,\mathrm{d}x}&=0, m\neq n\\
  \int_0^1{(\sin(n\pi x))^2\,\mathrm{d}x}&=\frac{1}{2}, m=n\\
  &\left\{\begin{aligned}
    \cos(a+b)&=\cos(a)\cos(b)-\sin(a)\sin(b)\\
    \cos(a-b)&=\cos(a)\cos(b)+\sin(a)\sin(b)\\
    \sin(a)\sin(b)&=\frac{1}{2}(-\cos(a+b)+\cos(a-b))
  \end{aligned}\right.
\end{align*}
\subsubsection*{sturn-liouville theory}
\begin{align*}
  X_n(x)&=\sin(\lambda_n x)\\
  \text{claim }&\int_0^1{X_m(x)X_n(x)\mathrm{d}x}\text{ for } m\neq n\\
  \intertext{use $X_n(x)$ solve $X''(x)+\lambda^2X(x), X(0)=0, X'(1)+hX(1)=0$}
  \text{set }& I=\int_0^1{X_m(x)X_n(x)\mathrm{d}x}\\
  \lambda_n^2I&=\int_0^1{X_m(x)\left(\lambda_n^2X_n\right)\mathrm{d}x}=
\end{align*}

\end{document}


\documentclass[a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{bbold}
\usepackage{csquotes}
\usepackage[framemethod=TikZ]{mdframed}
\usepackage[pdfborder={0 0 0}]{hyperref}
\usepackage{xcolor}

\makeatletter
\def\moverlay{\mathpalette\mov@rlay}
\def\mov@rlay#1#2{\leavevmode\vtop{%
   \baselineskip\z@skip \lineskiplimit-\maxdimen
   \ialign{\hfil$\m@th#1##$\hfil\cr#2\crcr}}}
\newcommand{\charfusion}[3][\mathord]{
    #1{\ifx#1\mathop\vphantom{#2}\fi
        \mathpalette\mov@rlay{#2\cr#3}
      }
    \ifx#1\mathop\expandafter\displaylimits\fi}
\makeatother

\newcommand\dotcup{\charfusion[\mathbin]{\cup}{\cdot}}
\newcommand\bigdotcup{\charfusion[\mathop]{\bigcup}{\cdot}}
\newcommand\p{\mathcal{P}}

\newcommand\op[2][P]{\mathbb{#1}\left(#2\right)}
\newcommand\os[2][P]{\mathbb{#1}\left[#2\right]}
\newcommand\cond[3][P]{\mathbb{#1}\left(#2 \left|\:#3\right)\right.}
\newcommand\cons[3][P]{\mathbb{#1}\left[#2 \left|\:#3\right]\right.}
\newcommand\kld[3][D]{\mathbb{#1}\left(#2 \left\|\:#3\right)\right.}

\newcommand\fall{\:\forall\:}
\newcommand\ex{\:\exists\:}
\newcommand\dt{\,\text{d}}
\newcommand\set[1]{\left\{#1\right\}}
\newcommand\key[1]{\textit{#1}}
\newcommand\card[1]{\left|#1\right|}
\newcommand\ind{\mathbb{1}}
\newcommand\converges{\rightarrow}
\newcommand\Perr{\mathbb{P}_{\text{err}}}

\DeclareMathOperator{\Capacity}{Cap}
\DeclareMathOperator{\Ex}{\mathbb{E}}
\DeclareMathOperator{\Va}{\mathbb{V}}

\theoremstyle{definition}
\newtheorem{thm}{Theorem}

\parindent0pt
\setlength\parskip{8pt}

\makeatletter
\DeclareRobustCommand{\em}{%
  \@nomath\em \if b\expandafter\@car\f@series\@nil
  \normalfont \else \bfseries \fi}
\makeatother

\title{
  Discrete Stochastics and Information Theory  \\
  \small{Elaboration on Prof. Wöss' course}
}
\author{Lukas Prokop}

\begin{document}
\maketitle
\tableofcontents
\clearpage

% questions of high  importance: 29
% question of medium importance: 18
% questions of low   importance: 3

\section{Definitions and axioms of probability theory}

\subsection[Basic definitions]{Explain the basic definitions of probability theory. What is $\Omega$? What is $\sigma$?}
%
\begin{description}
  \item[Event space $\Omega$] \hfill{} \\
    Defines the event space. A coin toss results in head or tail. So we could define an event space of $\{h, t\}$. Or we could consider sequences of coin tosses: $\{\text{hhh}, \text{hht}, \text{hth}, \text{thh}, \text{htt}, \text{tth}, \text{tht}, \text{ttt}\}$. $\Omega$ contains all possible outcomes. Events are (reasonable) subsets.

  \item[Subset $\mathcal{A}$] \hfill{} \\
    Defines a subset of possible outcomes of $\Omega$ with $\mathcal{A} \subset \op{\Omega}$.
    An example is a throw of the dice with result \enquote{even number} ($\mathcal{A} = \set{2, 4, 6}$) or \enquote{number 6} ($\mathcal{A} = \set{6}$).

  \item[$\sigma$ algebra on a set $\mathcal{A}$] \hfill{} \\
    A set closed under complement, union and intersection. The following axioms hold:
    \begin{enumerate}
      \item $\varnothing \in \mathcal{A}$
      \item $A \in \mathcal{A} \Rightarrow A^C \in \mathcal{A}$
      \item If $A_n \in \mathcal{A}$ with $n = 1, 2, \ldots$ then $\bigcup_{n=1}^\infty A_n \in \mathcal{A}$
    \end{enumerate}

  \item[Probability measure] \hfill{} \\
    Assigns a real number between 0 and 1 (inclusive) to every event.
    $P: \mathcal{A} \rightarrow [0, 1]$.
    It satisfies $P(\varnothing) = 0$ and $P(\Omega) = 1$ where $P$ with a set parameter means \enquote{one of}.

  \item[Probability space $(\Omega, \mathcal{A}, P)$] \hfill{} \\
    Defines a probability space $(\Omega, \mathcal{A}, P)$ consisting of an event space, the possible outcomes $\mathcal{A}$ and some probability measure.
\end{description}

\subsection{What does $\sigma$ additivity mean?}

\[
  A_n \in \mathcal{A} \land A_n \cap A_m = \varnothing \quad\fall n \in \mathbb{N}^+, n \neq m
\] \[
  \Rightarrow
  \op{\bigcup_{n=1}^\infty A_n} = \sum_{n=1}^\infty \op{A_n}
\]

\subsection{What does the \enquote{Law of continuity} tell?}

The law can be derived from the definition of $\sigma$ algebras:
\begin{align*}
  A_n \in \mathcal{A}, A_1 \subset A_2 \subset A_3 \subset \ldots & \Rightarrow \lim_{n\rightarrow\infty} \os A_n = \op{\bigcup_{n=1}^\infty A_n} \\
  B_n \in \mathcal{B}, B_1 \supset B_2 \supset B_3 \supset \ldots & \Rightarrow \lim_{n\rightarrow\infty} \os B_n = \op{\bigcap_{n=1}^\infty B_n}
\end{align*}

\subsection{What is a random variable?}

A random variable $X$ is a function associating an event with a real value. It maps the probability space to real world values.
\[ X: \Omega \rightarrow \mathbb{R} \]

\subsection{What is distribution, density, expected value, absolute convergence, continuity and monotonicity?}
%
A \key{probability distribution} assigns a probability to each subset of possible outcomes of a random experiment. It can be defined by a probability mass function, probability density function, cumulative distribution function, survival function, hazard function, characteristic function or a rule to create a new random variable with a known joint probability distribution.

The \key{probability density function} (PDF) of a continuous random variable $X: \Omega \rightarrow \mathbb{R}$ is a function that describes the relative likelihood for this random variable to take on a given value. So there exists some $f: \mathbb{R} \rightarrow [0, \infty)$ such that
\[
  \os{X \in B} = \p^X(B) = \int_B f(x) \dt x
\]

Let $X$ be a random variable defined on a probability space ($\Omega, \Sigma, \p$) then the \key{expected value} is defined as
\[
  \mathbb{E}(X) = \int_{\Omega} X \dt \p
\]

Let $P^X$ operate on ($\mathbb{R}, \mathcal{B}$) which is the smallest $\sigma$ algebra containing all intervals. $P^X(B)$ is the probability that $B$ occurs with respect to a random variable $X$ defined as
\[
  P^X(B) = \os{X \in \mathcal{B}} = \op{\set{\omega \in \Omega: X(\omega) \in \mathcal{B}}}
\]

Convergence is a property of a sequence of random variables. \key{Absolute convergence} means that even the sum of absolute values of the series converges as well.

A \key{continuous probability distribution} associates a probability to a continuous range of values. Only continuous probability distributions have a probability density function. The PDF can be \key{monotonic} meaning that it is a function between ordered sets that preserves the given order.

\subsection{What is conditional probability and independence of events?}
%
Conditional probability is defined as division of joint probability divided by marginal probability:
\begin{align}
  \cond AB = \begin{cases}
    \frac{\op{A \cap B}}{\op{B}}  & \op{B} > 0 \\
    0                                 & \op{B} = 0
  \end{cases}
\end{align}
$\cond AB$ semantically asks \enquote{assuming $B$ to happen with certainty, what is the probability that $A$ will happen}.
$A$ and $B$ are independent if and only if $\cond AB = \op{A}$. Or in general $A_1, A_2, \ldots, A_n \in \mathcal{A}$ is independent if $\forall\: 1 \leq i_1 < i_2 < \ldots < i_k \leq n$

\[
  \op{A_{i_1} \cap A_{i_2} \cap \ldots \cap A_{i_k}}
  = \op{A_{i_1}} \op{A_{i_2}} \cdot \ldots \cdot \op{A_{i_k}}
\]

\subsection{What is marginal/joint distribution? What's their relation?}
%
Let $A$ and $B$ be two events occuring with \key{marginal} probability $\op{A}$ and $\op{B}$. The \key{joint probability} is intuitively defined as probability that $A$ \emph{and} $B$ will happen. Formally $\op{A, B}$, defined as the intersection $\op{A \cap B}$.

\begin{align*}
  \text{marginal probability: }   && p_{X,Y}(x, y) &= \os{X=x, Y=y} \\
  \text{joint probability: }      && p_X(x_1, x_2, \ldots, x_n) &= \os{X = (x_1, x_2, \ldots, x_n)}
\end{align*}

Then the following relation holds:
\[
  \cond AB = \begin{cases}
    \frac{\op{A \cap B}}{\op{B}}  & \text{if } \op{B} > 0 \\
    0 & \text{if } \op{B} = 0
  \end{cases}
\]

Bayes' Theorem is given by
\[
  \cond AB = \frac{\cond BA \cdot \op A}{\op B}
\]

\subsection{Prove $\mathbb{E}(X\cdot Y) = \mathbb{E}(X) \cdot \mathbb{E}(Y)$ for independent variables in the discrete case}

\begin{align*}
  \Ex(X)         &= \sum_x x \cdot p_X(x) \\
  \Ex(Y)         &= \sum_y y \cdot p_Y(y) \\
  \Ex(X \cdot Y) &= \sum_{x,y} xy \cdot p_{X,Y}(x,y) \\
                 &= \sum_x \sum_y xy \cdot p_X(x) \cdot p_Y(y) \\
                 &= \left(\sum_x x \cdot p_X(x)\right) \cdot \left(\sum_y y \cdot p_Y(y)\right) \\
                 &= \Ex(X) \cdot \Ex(Y)
\end{align*}

\subsection{How is covariance and variance defined?}

The expected value $\Ex(X)$ of a random variable $X$ is also known as mean $\mu$.

Covariance is a measure of how much two random variables change together. If the greater values of one variable primarily correspond with the greater values of the other variable (and vice versa for small values), the covariance is positive. If they are linearly disproportional, it is negative.

\[ \sigma(X, Y) = \Ex((X - \Ex(X))(Y - \Ex(Y))) = \Ex(XY) - \Ex(X) \Ex(Y) \]

Variance is a special case of covariance with only one parameter. Variance is the covariance of a variable with itself: $\sigma^2(X) = \sigma(X, X)$. It measures how far a set of numbers is spread out. A variance of zero indicates that all the values are identical. Variance is always non-negative.

\[ \sigma^2(X) = \Va(X) = \Ex((X - \Ex(X))^2) \]
\begin{align*}
  \text{discrete:}   && \Va(X) &= \sum_{i=1}^n (p_i \cdot x_i^2) - \mu^2 \\
  \text{continuous:} && \Va(X) &= \int x^2 p(x) \dt x - \mu^2
\end{align*}

Covariance is symmetrical: $\sigma(X, Y) = \sigma(Y, X)$. If $X$ and $Y$ are independent, then $\sigma(X, Y) = 0$.

\subsection{Derive Bienaymé's equation for $\Va(\overline{X_n})$}

Bienaymé's equation states that
%
\[
  \Va\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n \Va(X_i)
\]
%
\dots which makes variance preferrable in many applications. Furthermore it implies that
%
\begin{align*}
  \overline{X_n}                 &= \frac{X_1 + \ldots + X_n}{n} \\
%  \Ex(\overline{X_n})            &= \frac{\Ex(X_1) + \ldots + \Ex(X_n)}{n} = \mu \\
  \Va\left(\overline{X_n}\right) &= \Va\left(\frac1n \sum_{i=1}^n X_i\right) \\
                                 &= \frac1{n^2} \sum_{i=1}^n \Va(X_i) \\
                                 &= \frac1{n^2} n \Va(X) \\
                                 &= \frac1n \Va(X) \\
                                 &= \frac{\sigma^2}{n}
\end{align*}

\subsection{Distinguish between convergence \enquote{almost surely} and \enquote{in probability}}

Let $\left(X_n\right)_{n \in \mathbb{N}}$ be a sequence of random variables.

\begin{align*}
  \lim_{n \rightarrow \infty} X_n = X \text{ almost surely if }
      & \mathbb{P}[\exists \lim X_n \land \lim X_n = X] = 1 \\
  \lim_{n \rightarrow \infty} X_n = X \text{ in probability if }
      & \forall\: a > 0, \lim_{n \rightarrow \infty} \mathbb{P}[\left|X_n - X\right| \geq a] = 0
\end{align*}

Intuitively almost surely means convergence is given until infinity. In probability means $n$ converges until a fixed constant (not necessarily infinity).

\subsection{In our notation, what is the difference between $x$, $X$, $X_n$ and $\mathcal{X}$?}
% source: 10

\begin{description}
  \item[$x$] \hfill{} \\
    A specific value (realization) of a random value. For example we iterate over all values of the discrete domain. The $x$ is typically used as iterator $x \in X$.

  \item[$X$] \hfill{} \\
    $X$ denotes a random variable used in an equation. Often $Y$ accompanies $X$ as second random variable to discuss joint probability.

  \item[$X_n$] \hfill{} \\
    $X_n$ denotes a sequence of random variables.

  \item[$\mathcal{X}$] \hfill{} \\
    $\mathcal{X}$ is used with two different meanings:
    \begin{itemize}
      \item In the context of Markov chains or information channels, we used $\mathcal{X}$ as the given input values for the Markov process.
      \item In the context of probability theory, $\mathcal{X}$ denotes the domain of a random variable $X$ or the sequence $X_n$ (if they all use the same domain).
      \item In the context of codes, $\mathcal{X}$ denotes the alphabet before encoding.
    \end{itemize}
\end{description}

\subsection{Define and prove Markov's inequality}

\begin{mdframed}
  Let $X \geq 0$ be a random variable with $0 < \Ex(X) < \infty$, then
  \[ \os{X > a \cdot \Ex(X)} \leq \frac1a \quad\forall\: a > 0 \]
\end{mdframed}

\begin{align*}
  A                     &= [X \geq a \Ex(X)] \\
                        &= \set{w \in \Omega: X(w) \geq a \cdot \Ex(X)} \\
  X                     &\geq X \ind_A \\
                        &\geq \underbrace{a \Ex(X)}_{\text{constant } c} \cdot \ind_A \Leftrightarrow \\
  X(w)                  &\geq X(w) \ind_A(w) \\
                        &\geq c \cdot \ind_A(w) \Rightarrow \\
  \Ex(X)                &\geq \Ex(c \cdot \ind_A) \\
                        &= a\Ex(X) \os A \Rightarrow \\
  \frac{\Ex(X)}{\Ex(X)} &\geq a \os A \Rightarrow \\
  \os A                 &\leq \frac 1a
\end{align*}

\subsection{Define and prove Chebyshev's inequality}

\begin{mdframed}
  Let $X$ be a random variable with finite $\Ex(X)$ and $\Va(X)$, then $\forall\: a > 0$
  \[ \os{\card{X - \Ex(X)} \geq a} \leq \frac{\Va(X)}{a^2} \]
\end{mdframed}

\begin{align*}
  Y      &= (X - \Ex(X))^2 \geq 0 \\
  \Ex(Y) &= \Va(X) \\
  \os{\card{X - \Ex(X)} \geq a}
         &= \os{Y \geq a^2} \\
         &= \os{Y \geq \frac{a^2}{\Va(X)} \Ex(Y)} \\
         &\leq \frac{1}{\frac{a^2}{\Va(X)}}  && \text{[apply Markov inequality]} \\
         & = \frac{\Va(X)}{a^2}
\end{align*}

\subsection{Distinguish between the Weak and Strong Law of Large Numbers}

The Weak Law of Large Numbers derives from Chebyshev's inequality and states,

\begin{mdframed}
  Given $X_n$ as sequence of iid random variables, $\mu = \Ex(X_n)$ and $\sigma^2 = \Va(X_n) < \infty$, then
  \[ \overline{X_n} \converges \mu \text{ in probability} \]
\end{mdframed}

The Strong Law of Large Numbers states,

\begin{mdframed}
  Given $X_n$ as sequence of iid random variables then there exists a finite random variable $Y$ such that
  \[ \overline{X_n} \converges Y \text{ almost surely} \]
\end{mdframed}

The Strong Law of Large Numbers can be equivalently defined as $\Ex(X_n)$ being finite and in this case $Y = \Ex(X_n)$ almost surely.

\subsection{Prove the Weak Law of Large Numbers}
% source: Patrick

\begin{align*}
  \os{\card{\overline{X_n} - \mu} \geq a} &\leq \frac{\Va(\overline{X_n})}{a^2}  && \text{[Chebyshev inequality must hold]} \\
    &= \frac{\sigma^2}{na^2} \\
    &\converges 0
\end{align*}

\section{Entropy}

\subsection{Give Hartley's definition of information value}
%
\begin{quote}
  The answer to a question that can assume the two values \enquote{yes} and \enquote{no} (without taking into account the meaning of the question) contains one unit of information.
\end{quote}

Let $U_N$ be a set of $N$ elements of uniform probability; hence the information amount to identify an element is
\[ \op[H]{U_N} = \log_2(N) \in \mathbb{R} \]

It holds that
\begin{itemize}
  \item $\op[H]{U_2} = 1$
  \item $\op[H]{U_N} \leq \op[H]{U_{N+1}}$
  \item $\op[H]{U_{M\cdot N}} = \op[H]{U_M} + \op[H]{U_N}$
\end{itemize}

\subsection{Derive Shannon's entropy definition from Hartley's information value}

As can be seen by the third corollary of Hartley, we can group information into (not necessarily equally large) groups of values:

\[ U_N = U_{N_1} \dotcup U_{N_2} \dotcup \ldots \dotcup U_{N_n} \]

If we want to identify the value in those groups, we need to ask two questions:
\begin{enumerate}
  \item In which group?
  \item Which element in the group?
\end{enumerate}

If the value can be found in group $k$, then $\log_2{N_k}$ questions are needed to identify the value.
The average number of questions for the second question is:

\[ H_2 = \sum_{k=1}^n \frac{N_k}{N} \log_2{N_k} \]

What about $H_1$?

\begin{align*}
  \op[H]{U_N} &= H_1 + H_2 \\
  \log_2{N}     &= H_1 + \sum_{k=1}^n \frac{N_k}{N} \log_2{N_k} \\
  H_1           &= \log_2{N} - \sum_{k=1}^n \frac{N_k}{N} \log_2{N_k} \\
                &= \sum_{k=1}^n \frac{N_k}{N} \log_2{N} - \sum_{k=1}^n \frac{N_k}{N} \log_2{N_k} \\
                &= -\sum_{k=1}^n \frac{N_k}{N} \log_2{\frac{N_k}{N}} \\
                &= -\sum_{k=1}^n p_k \log_2{p_k} && \text{with } p_k = \frac{N_k}{N}
\end{align*}

\subsection{Give Shannon's definition of \key{entropy} and provide examples for small/large entropy}

\key{Entropy} is a measure of uncertainty of a random variable. Let $X$ be a discrete random variable with alphabet $\mathcal{X}$ and probability mass function $p_X(x) = \os{X = x}$ for $x \in \mathcal{X}$. The entropy $\op[H]{X}$ of a discrete random variable $X$ is defined by
\[ \op[H]{X} = -\sum_{x \in \mathcal{X}} p_X(x) \log_2{p_X(x)} = -\Ex(\log_2 p_X(x)) \]
$\op[H]{X} \geq 0$ holds for all $X$. Furthermore we define
\[
  0 \log 0 = 0              \quad \forall\, b \geq 0 \qquad
  a \log \frac a0 = \infty  \quad \forall\, a > 0
\]

For example a uniform distribution of two discrete values gives
\[ \op[H]{X} = - \left(\frac12 \log_2{\frac12} + \frac12 \log_2{\frac12} \right) = 1 \]
Uniform distribution always provides a \key{large} value of entropy. Certain events provide a \key{small} value. For example consider two events, where only event $A$ always happens:
\[ \op[H]{X} = - \left(0\cdot\log_2{0} + 1\cdot\log_2{1}\right) = \left(1\cdot0\right) = 0 \]

The following holds:
\begin{itemize}
  \item $\op[H]{X}$ depends only on the probabilities; not on the specific interpretation.
        \[ f: \mathcal{X} \rightarrow \mathcal{X}' \land X' = f(X) \Rightarrow \op[H]{X'} = \op[H]{X} \]
  \item $p_0 \mapsto \op[H]{p_0, 1 - p_0}$ is continuous. The maximum is attained at $\frac12$:
        \[ \op[H]{p_0, 1 - p_0} = -p_0 \log_2(p_0) - (1 - p_0) \log_2(1 - p_0) \]
        \[
           -\frac12 \log_2\left(\frac12\right)
           - \left(1 - \frac12\right) \log_2 \left(1 - \frac12\right)
           = +\frac12 - \left(-\frac12\right)
           = 1
        \]
  \item For some fixed $n$,
        \[ \op[H]{p_1, \ldots, p_n} \leq \op[H]{\frac1n, \ldots, \frac1n} = \log_2(n) \]
\end{itemize}

\subsection{What is joint entropy? What is the entropy of conditional distribution aka. conditional entropy?}

Given a pair of discrete random variable ($X, Y$). The joint entropy of $X$ and $Y$ is defined as:
\begin{align*}
  \op[H]{X, Y} &= -\sum_{x \in \mathcal{X}} \sum_{y \in \mathcal{Y}} p(x, y) \log_2 p(x, y) \\
                 &= -\Ex\left(\log_2{p(X, Y)}\right)
\end{align*}

Entropy of conditional distribution:
\begin{align*}
  \cond[H]YX &= \sum_{x \in \mathcal X} p_X(x) \cond[H]{Y}{X=x} \\
             &= \sum_{x \in \mathcal X} \op{X=x} \left(-\sum_{y \in \mathcal Y} \cond{Y=y}{X=x} \log_2 \cond{Y=y}{X=x} \right) \\
             &= -\sum_{x \in \mathcal X} \sum_{y \in \mathcal Y} \op{X=x} \cond{Y=y}{X=x} \log_2 \cond{Y=y}{X=x}
                    & \left[\op x \cdot \cond yx = \op{y,x}\right] \\
             &= -\sum_{x \in \mathcal X} \sum_{y \in \mathcal Y} \op{Y=y,X=x} \log_2 \frac{\op{Y=y,X=x}}{\op{Y=y}} \\
             &= \sum_{x \in \mathcal X} \sum_{y \in \mathcal Y} \op{Y=y,X=x} \log_2 \frac{\op{Y=y}}{\op{Y=y,X=x}} \\
             &= \sum_{x \in \mathcal X} \sum_{y \in \mathcal Y} p_{X,Y}(x,y) \log_2 \frac{p_Y(y)}{p_{X,Y}(x,y)}
\end{align*}

\subsection{Prove that $\cond[H] YX = \op[H]{X, Y} - \op[H]{X}$}

\begin{align*}
  \cond[H] YX &= -\sum_x \sum_y p_X(x) p_{X,Y}(y\,|\,x) \log_2 p_{X,Y}(y,x) \\
            &= -\sum_x \sum_y p_{X,Y}(x,y) \left[\log_2 p_{X,Y}(x,y) - \log_2 p_X(x)\right] \\
            &= \op[H]{X,Y} + \sum_x \log_2 p_X(x) \sum_y p_{X,Y}(x,y) \\
            &= \op[H]{X,Y} - \op[H]{X}
\end{align*}

\subsection{What is the Kullback-Leibler distance? Are the parameters symmetrical?}

Let $p$ and $q$ be two probability distributions on the finite set $\mathcal{X}$.
$X$ is a random variable with distance probability $p$.
The \key{relative entropy} or \key{Kullback-Leibler distance} of $p$ with respect to $q$ is
\[ \kld{p}{q}  = \sum_x p(x) \log_2 \frac{p(x)}{q(x)} = \Ex\left(\log_2 \frac{p(x)}{q(x)}\right) \]

\begin{itemize}
  \item Be aware that for $p(x) > 0$ and $q(x) = 0$: $\kld pq  = \infty$.
  \item $\kld pq $ is always non-negative.
  \item From $\kld pq = 0$ it follows that $p(x) = q(x) \: \forall x \in \mathcal{X}$.
  \item $\kld pq \neq \kld qp$
  \item Intuitively it is a measure of the information lost when $q$ is used to approximate $p$.
\end{itemize}

\subsection{What is mutual information? What is conditional mutual information? What is the chain rule?}

Let $X$ and $Y$ be a pair of random variables. \key{Conditional mutual information} is defined as,

\begin{align*}
  \op[I]{X, Y} &= \kld{p_{(X,Y)}}{p_X \bigotimes p_Y} \\
               &= \sum_{x,y} p(x,y) \log_2 \frac{p(x,y)}{p_X(x)\cdot p_Y(y)} \\
               &= \op[H]{X} - \cond[H] XY = \op[H]{Y} - \cond[H] YX
\end{align*}

Intuitively, mutual information measures the information that X and Y share. It measures how much knowing one of these variables reduces uncertainty about the other.

Specifically it holds that $\op[I]{X, X} = \op[H]{X}$. The \key{chain rule} states that
\[
  \op[H]{X_1, \dotsc, X_n} = \sum_{k=1}^n \cond[H]{X_k}{X_{k-1}, \dotsc, X_{1}}
\]

The chain rule of mutual information is defined as,
\[
  \op[I]{(X_1, \dotsc, X_k); Y} = \sum_{k=1}^n \cond[I]{X_k; Y}{X_{k-1},\dotsc,X_{1}}
\]

\key{Conditional mutual information} of $X$ and $Y$ given $Z$:

\begin{align*}
  \cond[I]{X;Y}{Z} &= \sum_z p_Z(z) \cond[I]{X;Y}{Z=z} \\
    &= \begin{cases}
      \cond[H]YZ - \cond[H]Y{X,Z} \\
      \cond[H]XZ - \cond[H]X{Y,Z}
    \end{cases}
\end{align*}

\subsection{What is Jensen's inequality?}
%
Given a continuous, convex function $f: I \rightarrow \mathbb{R}$ (with $I$ as open interval) and $X$ as random variable,
\[  \Ex(f(X)) \geq f(\Ex(X))  \]

\subsection{What is information inequality?}

Let $p(x)$ and $q(x)$ be a pair of probability distributions with $x \in \mathcal{X}$, then $\kld{p}{q} \geq 0$.

Equality holds if and only if $p(x) = q(x) \quad\fall x$.

Corollaries:
\begin{itemize}
  \item $\op[I]{X;Y} \geq 0$
  \item $\cond[H]XY \leq \op[H]{X}$
  \item $\cond[I]{X;Y}{Z} \geq 0$
  \item $\op[H]{X_1, \dotsc, X_n} \leq \sum_{k=1}^n \op[H]{X_k}$
\end{itemize}

\subsection{What is the log-sum inequality?}

Given $a_1, \dotsc, a_n$ and $b_1, \dotsc, b_n \geq 0$, then
\[
  \sum_{i=1}^n a_i \log{\frac{a_i}{b_i}}
    \geq \left(\sum_{i=1}^n a_i\right) \log{\frac{\sum_{i=1}^n a_i}{\sum_{i=1}^n b_i}}
\]

Given $p^{(1)}(\cdot), p^{(2)}(\cdot)$ and $q^{(1)}(\cdot), q^{(2)}(\cdot)$
as probability densities on $\mathcal{X}$ with $0 \leq \lambda \leq 1$
\begin{multline*}
  \Rightarrow
    \kld{\lambda \cdot p^{(1)} + (1 - \lambda) p^{(2)}}%
    {\lambda \cdot q^{(1)} + (1 - \lambda) \cdot q^{(2)}} \\
  \leq
    \lambda \kld{p^{(1)}}{q^{(1)}} +
    (1 - \lambda) \kld{p^{(2)}}{q^{(2)}}
\end{multline*}

\subsection{What is a Markov chain?}
%
A \key{Markov chain} is a random process satisfying the Markov property which means transitions happen memoryless.
Given a state space, transitions can happen between any two states with associated probabilities. Transition probabilities are stored in the so-called \enquote{stochastic matrix}.

Hence a Markov chain is a sequence of random variables $X_1, X_2, X_3, \dotsc$ with the Markov property, namely that, given the present state, the future and past states are independent. Formally $\cond{X_{n+1}}{X_1 = x_1, X_2 = x_2, \dotsc, X_n = x_n} = \cond{X_{n+1} = x}{X_n = x_n}$.

A Markov chain is ergodic iff every transition probability is greater zero. A Markov triple is a Markov chain of three states.

\subsection{Define Data Processing inequality}

Let $X \rightarrow Y \rightarrow Z$ be a Markov triple \emph{in that order} meaning $X$ and $Z$ are independent but conditional on $Y$. Hence
\[ \cons{X=x, Z=z}{Y=y} = \cons{X=x}{Y=y} \cdot \cons{Z=z}{Y=y} \]

It holds that
\[  \op[I]{X;Y} \geq \op[I]{X;Z}  \]

The information that $Y$ knows about $X$ cannot be increased by manipulating (processing) $Y$ deterministically or at random.

\subsection{Prove Data Processing inequality}

The chain rule tells
\[  \op[I]{(Y, Z); X} = \op[I]{Y;X} + \cond[I]{Z;X}{Y}  \]

So,
\begin{align*}
  \op[I]{X;Y,Z} &= \op[I]{X;Y,Z} \\
  \op[I]{X;Y} + \underbrace{\cond[I]{X;Z}{Y}}_{\geq0} &= \op[I]{X;Z} + \underbrace{\cond[I]{X;Y}{Z}}_{=0} \\
  \op[I]{X;Y} &\geq \op[I]{X;Z}
\end{align*}

\subsection{Define and prove Fano's inequality}
% source: 10, Felix

\begin{mdframed}
  Define a Markov triple $X \rightarrow Y \rightarrow \hat X$
  with $\mathbb{P}_{\text{err}}$ as the error that $X \neq \hat X$.
  \[
    H(\Perr) + \Perr \log{\card{\mathcal{X}}}
      \geq \cond[H]{X}{\hat X}
      \geq \cond[H]XY
  \]

  Equivalently
  \[ 1 + \Perr \log{\card{\mathcal{X}}} \geq \cond[H]XY \]
  \[ \Perr \geq \frac{\cond[H]XY - 1}{\log_2\card{\mathcal{X}}} \]
\end{mdframed}

\[
  E = \begin{cases}
    1 & \text{if } \hat X \neq X \\
    0 & \text{if } \hat X = X
  \end{cases}
\]

\begin{align*}
  \cond[H]{E,X}{\hat X} &= \cond[H]{E,X}{\hat X} \\
  \underbrace{\cond[H]{E}{\hat X}}_{\leq \op[H]{\Perr}} + \underbrace{\cond[H]{X}{E,\hat X}}_{\leq \Perr \log\card{\mathcal{X}}}  &= \cond[H]{X}{\hat X} + \underbrace{\cond[H]{E}{X,\hat X}}_{=0} \\
\end{align*}

$\cond[H]{X}{E,\hat X}$ is bounded by
\begin{align*}
  \cond[H]{X}{E,\hat X} &= \os{E = 0}\cond[H]{X}{\hat X, E = 0} + \os{E = 1}\cond[H]{X}{\hat X, E = 1} \\
                        &\leq (1 - \Perr) \cdot 0 + \Perr \log \card{\mathcal{X}}
\end{align*}

Hence, we obtain Fano's inequality. Intuitively it relates the probability of error in guessing the random variable $X$ to its conditional entropy $\cond[H]XY$. Given $X$ as a function of $Y$. Iff $X$ can estimate $Y$ with zero probability of error, then $\cond[H]XY = 0$. Fano's inequality uses this fact: Estimate $X$ with a low probability of error if the conditional entropy $\cond[H]XY$ is small.

\section{Asymptotic entropy}

\subsection{What does iid mean?}

iid is a property of a set of random variables. A set of random variables can be \enquote{independent, identically distributed} meaning that they all utilize the same probability distribution, but are independent in every possible form.

\subsection{What is a stochastic process?}

A stochastic process (in discrete time) is a sequence of random variables
\[  \left(X_n\right)_{n\geq1} \]

\subsection{What is asymptotic entropy?}

Given $X_n$ as discrete random variables and its values in some finite/countable set $\mathcal{X}$.
The \key{asymptotic entropy} or \key{asymptotic rate} $h$ of the stochastic process is defined as
\[ h = \lim_{n \rightarrow \infty} \frac1n \op[H]{X_1,\dotsc,X_n} \]

Furthermore
\[ h' = \lim_{n \rightarrow \infty} \cond[H]{X_n}{X_{n-1},\dotsc,X_1} \]
if the limit exists. If $h'$ exists, then $h$ exists and $h = h'$.

Convergence is interpreted as
\[
  \lim_{n\rightarrow\infty} a_n = a
    \Leftrightarrow
      \fall \varepsilon > 0
      \ex N_\varepsilon:
      \fall n > N_\varepsilon:
      \card{a_n - a} < \varepsilon
\]

\subsection{What is a stationary distribution?}

A distribution $\left(X_n\right)_{n\geq1}$ is stationary if $\fall n, l \in \mathbb{N}:$ $(X_1,\dotsc,X_n)$ and $(X_{l+1},\dotsc,X_{l+n})$ have the same joint distribution.

\[ \fall x_1, \dotsc, x_n \in \mathcal{X}: \os{X_1=x_1,\dotsc,X_n=x_n} = \os{X_{l+1}=x_1,\dotsc,X_{l+n}=x_n} \]

If $\left(X_n\right)_{n\geq1}$ is stationary, $h'$ exists.

\subsection{What is time homogeneity?}

A Markov chain is called time-homogeneous if
\[ p_n(y\,|\,x) = p_m(y\,|\,x) \qquad \fall m,n \]

Hence, transition probability do not depend on time.

\subsection{Prove that at least one stationary distribution exists for every stochastic matrix.}

Formally, a distribution at time $n$ is given as
\[
  \os{X_n=y}
    = \sum_{x\in\mathcal{X}} \cons{X_n=y}{X_0=x} \cdot \os{X_0=x}
    = \sum_x p_{x,y}^{(n)} \mu_x
    = \left(\mu P^n\right)_x
\]

Now for an arbitrary stochastic matrix $P = \left(p_{x,y}\right)_{x,y \in \mathcal{X}}$ at least one stationary distribution $\nu$ exists.

Consider an initial distribution $\mu = \mu_0$.
\[ \left( \mu + \mu P + \mu P^2 + \dotsb + \mu P^{n-1} \right) \frac1n = \mu_n \]
$\mu_n$ is a probability vector on $\mathcal{X}$ for various $n$. A subsequence $\left(\mu_{n_k}\right)_{k \in \mathbb{N}}$ exists which converges to some vector $\nu$.

\begin{align*}
  \sum_{x \in \mathcal{X}} \nu_X &= \sum_{x \in \mathcal{X}} \lim \mu_{n_k} \\
                                 &= \lim \sum_{x \in \mathcal{X}} \mu_{n_k} && \text{[because $\mathcal{X}$ is finite]} \\
                                 &= 1 \\
  \mu_n^P - \mu_n &= \frac1n \left(\mu P + \mu P^2 + \dotsb + \mu P^n\right) \\
                  &= \frac1n \left(\mu + \mu P + \dotsb + \mu P^{n+1}\right) \\
                  &= \frac1n \underbrace{\left(\mu - \mu P^n\right)}_{\text{bounded by } 1} \\
                  &\converges 0
\end{align*}
\[
 \begin{array}{ccccc}
   \mu_{n_k} P & - & \mu_{n_k}  & \rightarrow & 0 \\
   \downarrow  &   & \downarrow &             & \\
   \nu P       & - & \nu        & =           & 0
 \end{array}
\]

\subsection{What does irreducibility of $(\mathcal{X}, P)$ mean?}

$(\mathcal{X}, P)$ is called irreducible\footnote{for finite $\mathcal{X}$ it is called ergodic} if $\fall x,y \in \mathcal{X}: \exists n = n_{x,y}: p_{x,y}^n > 0$. Graph theoretically irreducibility means strongly connectedness.

If $(\mathcal{X}, P)$ is finite and irreducible then there is a unique stationary distribution $\mu$ and
\[
  \nu_x = \frac{1}{\Ex_X(t^x)}
\]
where $\Ex_X(t^x)$ denotes the expected return time to $x$ with $t_{(w)}^x = \inf\set{n \geq 1: X_n^{(w)} = x}$.
$t^x$ is almost surely finite: $\os{t^x < \infty} = 1$ and $\Ex_X(t^x) < \infty$.

\subsection{How can we compute the stationary distribution for a given irreducible distribution?}

\subsection{Discuss a random walk.}

\subsection{Under which circumstances does a unique stationary initial distribution exist? Prove it.}

A unique stationary initial distribution $\nu$ exists, if $(\mathcal{X}, P)$ is irreducible.

Proof: missing
% TODO: lecture of 2014.05.07

\section{Asymptotic equipartition}

\subsection{What is asymptotic equipartition?}

Suppose that $(X_n)_{n \geq 1}$ is an $\mathcal{X}$-valued stochastic process with entropy rate $h$. Then the process is said to have asymptotic equipartition property (AEP), if
\[ -\frac1n \log_2 p_n(X_1, \dotsc, X_n) \converges h \text{ almost surely} \]
(Sometimes convergence in probability suffices)

\subsection{Under which conditions does the asymptotic equipartition property hold? What is it good for?}

\[ -\frac1n \log_2 p_n(X_1, \dotsc, X_n) \converges h \text{ almost surely} \]
must hold. Furthermore the AEP is good for data compression.

\subsection{What is a typical set?}

The typical set is defined as
\[
  A_\varepsilon^{(n)}
    = \left\{
      (x_1, \dotsc, x_n) \in \mathcal{X}^n:
      \left|-\frac1n \log_2 p_n(x_1,\dotsc,x_n) - h\right| < \varepsilon
    \right\}
\]
where $\varepsilon > 0$. The following theorems hold for typical sets
\begin{itemize}
  \item $2^{-n(h+\varepsilon)} < p_n(x_1,\dotsc,x_n) < 2^{-n(h - \varepsilon)} \quad\fall (x_1,\dotsc,x_n) \in A_\varepsilon^{(n)}$
  \item $\os{(X_1,\dotsc,X_n) \in A_\varepsilon^{(n)}} > 1 - \varepsilon \quad \fall n \geq N_\varepsilon$
  \item $(1 - \varepsilon) 2^{n(h-\varepsilon)} \leq \card{A_\varepsilon^{(n)}} \leq 2^{n(h + \varepsilon)}$
\end{itemize}
Those theorems follow from the definition of typical sets.

\section{Coding and compression}

\subsection{How is expected and average code length defined?}

\begin{description}
  \item[\key{Expected code length}] \hfill{} \\
    $\sum_{(x_1,\dotsc,x_n)} l(C(x_1,\dotsc,x_n)) p_n(x_1,\dotsc,x_n)$
  \item[\key{Average code length}] \hfill{} \\
    $\frac1n \Ex\left(l(x_1,\dotsc,x_n)\right) \leq h + \varepsilon' \qquad\fall n > N_\varepsilon$
\end{description}

\subsection{What does the Ergodic theorem state?}

For finite irreducible Markov chains $(\mathcal{X}, P)$ it holds that for any $f: \mathcal{X} \rightarrow \mathbb{R}$
\begin{align*}
  \frac1n\left(f(X_0) + f(X_1) + \dotsb + f(X_{n-1})\right)
    &= \sum_{x \in \mathcal{X}} f(X) \nu_x \text{ almost surely}   && \text{ [discrete case]} \\
    &= \int_\mathcal{X} f \dt\nu                                   && \text{ [continuous case]} \\
\end{align*}

\section{Codes}

\subsection{Define the properties non-singularity, unique decodability and prefix-freedom.}

\begin{description}
  \item[\key{nonsingular}]
    $C: \mathcal{X} \rightarrow \Sigma^+$ is injective
  \item[\key{uniquely decodable}]
    $C: \mathcal{X}^* \rightarrow \Sigma^*$ is injective
  \item[\key{instantaneous or prefix-free}]
    $\fall x,y \in \mathcal{X}: C(x)$ is not a prefix of $C(y)$
\end{description}

\subsection{What is the Theorem Kraft Inequality?}

Let $D = \card{\Sigma}$ and $C: \mathcal{X} \rightarrow \Sigma$ is a prefix-free code. It holds that
\[
  \sum_{x \in \mathcal{X}} D^{-l(C(x))} \leq 1
\]

Conversely if $l_X \in \mathbb{N}$ ($x \in \mathcal{X}$) are such that $\sum_{x \in \mathcal{X}} D^{-l_x} \leq 1$
then there is a prefix-free code $C$ with $C(L(x)) = L_x \fall x \in \mathcal{X}$.

\subsection{Define a general integer optimization problem.}

\subsection{What are Lagrange multipliers and what are they used for?}

\subsection{Which theorem regarding code length holds for every prefix-free code?}

For every prefix-free code $C: \mathcal{X} \rightarrow \Sigma^+$ the expected code length satisfies $L_C \geq H_D(X)$.
If equality holds, then $p(x) = D^{-l(C(x))}$. Remember that $D = \card{\Sigma}$.

It follows that if $p$ is D-adic\footnote{meaning $p(x) \in \set{D^{-n}: n \in \mathbb{N}}$} then the minimum value $H_D(X)$ is attained. We can set $l_X = -\log_D p(x)$ satisfy Kraft equality.

Otherwise minimise $\log_D{B} + \frac{1}{\log_2 D} \kld pr$.

\subsection{Which theorem holds for optimal code lengths?}

Optimal code lengths $l_x^*$ with $x \in \mathcal{X}$ are such that the expected code length satisfies $H_D(X) \leq L^* < H_D(X) + 1$.

\subsection{How does the Huffman algorithm work?}

\subsection{Prove that Huffman codes are optimal.}

\subsection{Which codes are canonical? Are Huffman codes canonical?}

Canonical codes satisfy the following properties:

\begin{enumerate}
  \item if $p(x) > p(y)$ then $l(C(x)) \leq l(C(y))$
  \item If $v,w \in C(\mathcal{X})$ are longest code words, then $l(v) = l(w)$
  \item $C$ can be modified into another optimal prefix-free code $C'$. $C'$ satisfies:
    Let $v, w$ be the least likely symbols
    \begin{itemize}
      \item $v$ and $w$ are siblings in the coding tree
      \item $\fall z \in \mathcal{X} \setminus \set{x,y}: p(z) \geq p(x) \geq p(y)$
    \end{itemize}
\end{enumerate}

Huffman codes are canonical.

\section{Information channels}

\subsection{Define discrete channels. What distinguishes a memoryless channel and the $n$-th extension of $\mathcal{C}$ without feedback?}

Input alphabet $\mathcal X$ \\
Output alphabet $\mathcal Y$ \\
Probability transition matrix $\cond yx$

The channel is \key{memoryless} iff the probability distribution of the output depends only on the current input and is conditionally independent of previous channel inputs or outputs.

%A \key{discrete memoryless channel} $\mathcal{C} = (\mathcal{X}, P, Y)$ with $P = (p(y\,|\,x))_{x \in \mathcal{X}}$ with $p(y\,|\,x)$ is the probability that the outcome is $y$ given that the input is $x$.

The \key{n-th extension of $\mathcal{C}$ without feedback} is the channel
\[ \mathcal{C}^n = (\mathcal{X}^n, P_n, Y^n) \]

Without feedback means than input symbol are conditionally independent of previous output values.
In this case the transition function satisfies
\[ \cond{y^n}{x^n} = \prod_{i=1}^n \cond{y_i}{x_i} \]

\subsection{How is capacity of a channel defined?}

\[ \Capacity(\mathcal C) = \max_{p(x)}\set{\op[I]{X;Y}: p_X \in M(\mathcal{X})} \]
where $M(\mathcal{X}) = \set{p(\cdot) \text{ probability distributions on } \mathcal{X}}$
given $P = (p(y\,|\,x))_{x\in\mathcal{X}}$ and $p_X$. This defines \key{information channel capacity}.

From Shannon's second theorem it follows that information channel capacity equals to operational channel capacity.

\subsection{Give examples for channels and their capacity}

\begin{itemize}
  \item Noise-less binary channel
  \item Channel with non-overlapping outputs
  \item Noisy typewriter
  \item Binary symmetric channel
  \item Binary erasure channel
\end{itemize}

\paragraph{A noise-less binary channel}
transmits both bits always without error.
If we want to compute the capacity
\begin{align*}
  C &= \max\left\{\op[I]{X;Y}\right\} \\
    &= \op[H]{X} - \cond[H]XY \\
    &= -2\left(\frac12 \log_2 \frac12\right) + \left(2 \cdot 0 \log_2 0 + 2 \cdot 1 \log_2 1\right) \\
    &= 1 + 0 = 1
\end{align*}

\paragraph{A noisy typewriter} transmits a letter of the latin alphabet correctly with probability $\frac12$. With probability $\frac12$ the next letter of the alphabet will be transmitted instead.

\[ \Capacity(\mathcal C) = \max\left\{\op[I]{X;Y}\right\} = \op[H]{X} - \cond[H]{X}{Y} \]

\begin{align*}
  \cond[H]{X}{Y} &= \sum_{y \in \mathcal Y} \sum_{x \in \mathcal X} \op{X=x,Y=y} \log_2 \frac{\op{Y=y}}{\cond{X=x}{Y=y}} \\
                 &= \sum_{y \in \mathcal Y} \sum_{x \in \mathcal X} \frac1{2\cdot 26} \log_2 \frac{\frac{1}{26}}{\frac{1}{52}} \\
                 &= \sum_{y \in \mathcal Y} \sum_{x \in \mathcal X} \frac{1}{2\cdot 26} \log_2 2 \\
                 &= 26 \cdot 2 \cdot \frac{1}{2\cdot 26} \cdot 1 \\
                 &= 1
\end{align*}

or

\begin{align*}
  \cond[H]{X}{Y} &= \sum_{y \in \mathcal Y} \op{Y=y} \cond[H]{X}{Y=y} \\
                 &= \sum_{y \in \mathcal Y} \frac1{26} \cdot 1 \\
                 &= 26 \cdot \frac1{26} \\
                 &= 1
\end{align*}

with

\begin{align*}
  H(X) &= -\sum_{x \in \mathcal X} \op{X=x} \log_2 \op{X=x} \\
       &= -\sum_{x \in \mathcal X} \frac{1}{26} \log_2 \frac{1}{26} \\
       &= -26 \cdot \frac{1}{26} \log_2 \frac{1}{26} \\
       &= \log_2 \frac1{26}
\end{align*}

\paragraph{A binary symmetric channel} transmits a bit correctly
with probability $(1-p)$. The wrong bit is sent with probability $p$.

\begin{align*}
  \Capacity(\mathcal{C}) &= \op[H]{X} - \sum_{y \in \mathcal Y} \op{Y=y} \cond[H]{X}{Y=y} \\
                         &= \op[H]{\frac12, \frac12} - \op[H]{p, 1-p} \\
                         &= 1 - \op[H]{p}
\end{align*}

\paragraph{A binary erasure channel} transmits a bit correctly with probability
$1 - \alpha$. With probability $\alpha$ a bit will be transmitted as detectable
error bit.

\begin{align*}
  \Capacity(\mathcal C) &= \max_{p(x)}\left(\op[H]{X} - \sum_{y \in \mathcal Y} \op{Y=y} \cond[H]{X}{Y=y}\right) \\
                        &= \max_{p(x)}\left(\op[H]{X} - \alpha \op[H]{X}\right) \\
                        &= \max_{\op[H]{X}}\left(\op[H]{X} - \alpha \op[H]{X}\right) \\
                        &= \left(1 - \alpha \cdot 1\right) \\
                        &= 1 - \alpha
\end{align*}

\subsection{When is $P$ of an information channel called (weakly) symmetric? Which associated theorem exists?}

$P = (p(y\,|\,x))_{x \in \mathcal{X}, y \in \mathcal{Y}}$ is called \key{symmetric} iff
\begin{enumerate}
  \item rows are permutations of each other
  \item columns are permutations of each other
\end{enumerate}

$P = (p(y\,|\,x))_{x \in \mathcal{X}, y \in \mathcal{Y}}$ is called \key{weakly symmetric} if
\begin{enumerate}
  \item rows are permutations of each other
  \item all columns yield the same sum
    \[ \exists c: \sum_x p(y\,|\,x) = c \qquad \fall y \in Y \]
\end{enumerate}

The theorem states,
\begin{quote}
  If $P$ is weakly symmetric, then $\Capacity(\mathcal{C}) = \log_2\card{Y} - \op[H]{p(\cdot\,|\,x)}$ and the maximum is achieved when $X$ is uniform on $\mathcal{X}$.
\end{quote}

\subsection{Prove $\Capacity(\mathcal{C}^n) = n \cdot \Capacity(\mathcal{C})$}

\subsection{What is a $(M, n)$ code? What is the average and maximum error of it?}

An $(M, n)$ code for the channel $C = (\mathcal{X}, P, Y)$ consists of the following:
\begin{enumerate}
  \item An input set $W$ with $\card{W} = M$ or wlog. $W = \set{1,\dotsc,M}$
  \item The codebook $B$ (set of codewords)
  \item A decoding function $g: Y^n \rightarrow \hat W$
\end{enumerate}

The average error is defined as
\begin{align*}
  \lambda_w^{(n)} &= \cons{g(Y^{(n)}) \neq w}{X^{(n)} = x^{(n)}(w)} \\
                  &= \sum_{y \in Y} p(y \,|\, x(w))
\end{align*}

The maximum probability of the error is $\lambda^{(n)} = \max_w \lambda_w^{(n)}$.

\subsection{What is the rate of a code?}

\[ R = \frac{\log_2 M}{n} \]

%A rate $R$ is said to be \key{achievable} if there exists a sequence of $\left(\lceil 2^{nR} \rceil, n\right)$ codes such that the maximal probability of error $\lambda^{(n)}$ converges towards $0$ as $n \to \infty$.

\subsection{What is an achievable rate? What is an achievable capacity?}

A rate $R > 0$ is achievable if there is a sequence of $(M_n, n)$ codes such that
\[ R_n = \frac{\log_2 M_n}{n} \converges R \]
and
\[ \lambda^{(n)} \converges 0 \]

The achievable capacity $R^*$ of a channel is the supremum of all achievable rates:
\[
  \fall \varepsilon > 0 \qquad
  \exists (M,n) \text{ code with rate } R > R^* - \varepsilon \land \lambda^{(n)} < \varepsilon
\]

\subsection{What is the set of jointly typical sequences?}

Intuitively, we decode a channel output $Y^n$ as the $i$-th index iff the codeword $X^n(i)$ is in the jointly typical set of the received signal $Y^n$.

The set $A_\varepsilon^{(n)}$ of \key{jointly typical} sequences $\set{(x^n, y^n)}$ with respect to the distribution $\op{x,y}$ is the set of $n$-sequences with empirical entropies $\varepsilon$-close to the true entropies:
\[
	A_\varepsilon^{(n)} = \set{
		(x^n, y^n) \in \mathcal X^n \times \mathcal Y^n:
		\card{-\frac1{n} \log_2 \op{x^n} - \op[H]{X}} < \varepsilon
	}
\]

The following statements hold for jointly typical sequences:
\begin{enumerate}
  \item $\op{(X^{(n)}, Y^{(n)}) \in A_\varepsilon^{(n)}} \to 1$ as $n \to \infty$
  \item $(1 - \varepsilon) 2^{n(\op[H]{X,Y} - \varepsilon)} \leq \card{A_n^\varepsilon} \leq \varepsilon 2^{n(\op[H]{X,Y} - \varepsilon)}$
  \item $(X'_1, \ldots, X'_n), (Y'_1, \ldots, Y'_n)$ are independent. Then
    \[
		\op{(X'_1, \ldots, X'_n, Y'_1, \ldots, Y'_n) \in A_\varepsilon^{(n)}} =
		\begin{cases}
		  \leq 2^{-n(\op[I]{X,Y} - 3\varepsilon)} & \text{else} \\
		  \geq (1 - \varepsilon) 2^{-n (\op[I]{X,Y} + 3\varepsilon)} & n \geq N_\varepsilon
		\end{cases}
    \]
\end{enumerate}

\subsection{Define Shannon's 2nd theorem and give an overview over the proof.}

Also called ``Channel coding theorem''.

\begin{quote}
  For a discrete memoryless channel, all rates below capacity C are achievable.
  Specifically for every rate $R < C$, there exists a sequence of $(2^{nR}, n)$ codes with maximum probability of error $\lambda^n \to 0$.

  Conversely any sequence of $(2^{nR}, n)$ codes with $\lambda^{(n)} \to 0$ must have $R \leq C$.
\end{quote}

\[ R^* = \Capacity(\mathcal{C}) \]

\section{List of theorems}

See also
\begin{itemize}
  \item Markov inequality
  \item Chebyshev inequality
  \item Weak and Strong Law of Large Numbers
\end{itemize}

\begin{thm}
  If $X,Y$ are independent random variables with finite $\Ex(X)$ and $\Ex(Y)$, then
  \[ \Ex(X\cdot Y) = \Ex(X) \cdot \Ex(Y) \]
\end{thm}

\begin{thm}
  If $X_n \rightarrow X$ almost surely then $X_n \rightarrow X$ in probability.
\end{thm}

\begin{thm}
  $X_n \rightarrow X$ almost surely $\Leftrightarrow$ $U_k \rightarrow 0$ in probability with
  $ U_k = \sup\set{\card{X_n - X}: n \geq k} $
\end{thm}

\begin{thm}
  If $\lim X_n = X$ almost surely and $\lim X_n = X'$ almost surely, then $X = X'$ almost surely.
  If $\lim X_n = X$ in probability and $\lim X_n = X'$ in probability, then $X = X'$ almost surely.
\end{thm}

\begin{thm}
  The function $N \mapsto H(U_N) = \log_2{N}$ is the \emph{unique} function satisfying the associated 3~axioms to Hartley formula:
  \begin{itemize}
    \itemsep0pt
    \item $H(U_2) = 1$
    \item $H(U_N) \leq H(U_{N+1})$ (\enquote{monotonicity})
    \item $H(U_{M\cdot N}) = H(U_M) + H(U_N)$
  \end{itemize}
  It is also the \emph{unique} function satisfying
  \begin{itemize}
    \itemsep0pt
    \item $H(U_{N+1}) - H(U_N) \xrightarrow{N \rightarrow \infty} 0$
  \end{itemize}
\end{thm}

\begin{thm}
  \begin{align*}
    & \cond[H] YX = \op[H]{X,Y} - \op[H]{X} \\
    \Leftrightarrow{ } & \op[H]{X,Y} = \op[H]{X} + \cond[H] YX
  \end{align*}
\end{thm}

\end{document}


\subsection{Recap of Hidden Markov Models (HMMs)}

We don’t see state

Each state produces a visible output. this output is drawn from a distribution for each state.

We observe a sequence of outputs, not states.