%%%%%%%%%%%%%%%%%%%%%%%%
%
%   Thesis template by Youssif Al-Nashif
%
%   May 2020
%
%%%%%%%%%%%%%%%%%%%%%%%%

\section{NHTSA Special Crash Investigations}
\hspace*{0.3cm} For the NHTSA Special Crash Investigation dataset, a hierarchical clustering was performed on the resulting graph kernel computed from the skip-gram graphs. Since this dataset was small at $N=48$, it served as a good subject to study the interactions between modifying the skip window width, kernel hyper parameters, and number of clusters. This study on these interactions informs decisions made on which hyper parameters to use in final analysis of the NHTSA data as well as the reddit thread data. \\

For the study on these interactions, hyper-parameters were varied and the value of cluster within sum of squares was tracked. Specifically, the values in Figure 4.1 were tested. In Figure 4.1, we see some promising values occurring at more prominent ``elbow" points in the plots. These points will be further analyzed and compared. The hyper parameters for these points are:\\

 \begin{table}
\centering
\begin{tabular}{c|c|c|c}
Point&Skip-gram k& Graph Kernel sigma&No. of Clusters \\
\hline
A&3&1000&5\\
B&2&1100&3\\
C&3&1100&5\\
D&3&800&6\\
E&3&900&7
\end{tabular}
\caption{Hyper-parameter for Variation Study in Figure 4.1}
\end{table}

\begin{figure}
\includegraphics[width=6in]{Content/Images/hclust_variation.png}
\caption{Hierarchical Clustering Variation (NHTSA) for differing hyper-parameters.}
\end{figure}

These hyper-parameter sets, and their corresponding graph kernel matrix, are used in a hierarchical clustering analysis. The results from the analysis perform well, and display prominent clusters, often of comparable sizes. In Figure 4.2, we see the 5 dendrograms produced from hyper-parameter sets A, B, C, D, and E. Upon some examination, one will notice that documents often appear in the same clusters together, regardless of the hyper-parameter sets. This is most encouraging, as it indicates that these clusters are not a result of user-chosen parameters, but that the documents are indeed similar. \\

\begin{figure}
\includegraphics[width=6in]{Content/Images/5cluster.png}
\caption{Five hyper-parameter sets and their corresponding dendrograms.}
\end{figure}

To compare how often this co-membership in groups was appearing, a matrix of co-membership was created. In Figure 4.3, we see that each document has a set of other documents with which it is always clustered with, regardless of hyper-parameter configurations. These small document sets are the document sets which we can expect to have the most in common with one another.\\

\begin{figure}
\includegraphics[width=6in]{Content/Images/comembers5.png}
\caption{Co-membership of documents across hyper-parameter sets A, B, C, D, and E.}
\end{figure}

We can inspect the results by checking some of the co-members' text. For example, document two (which was an ambulance crash in Angola, Delaware), has only 3 other co-members. Document two's co-members are: document 11, document 30, and document 38. Document 11 was an ambulance crash in Sheridan, Indiana. Document 30 was an ambulance crash in Pasadena, California. Document 38 was an ambulance crash in Ocilla, Georgia.\\

Now, we can examine some information about these four incidents that may explain their clustering. For example, all of these incidents had fatalities, three of them had roll over events, and these four were all front end collisions to the ambulance where they struck another vehicle, or object, head on. While this is just one example of the clustering methods identifying similarities in the text, we can examine similarities across the dataset through use of term-frequency/inverse document frequency.\\

\begin{equation}
TF-IDF = \frac{\frac{\text{Term Frequency in Document}}{\text{Total Words in Document}}}{\log_2(\frac{\text{Total Documents in Corpus}}{Documents with Term})}
\end{equation}

\newpage

\begin{figure}
\includegraphics[width=6in]{Content/Images/nhtsa_tf_idf.png}
\caption{Term-Frequency/Inverse Document-Frequency for NHTSA reports, using hyper parameter set A.}
\end{figure}

In Figure 4.4, we see that words that appear frequently, and are unique to that cluster appear to be vehicle manufacturers, medical terms, and words that describe a crash situation (e.g. guardrail, tree, interstate). We can use these to get an idea of the crash situation. Alternatively, we can also examine portions of the skip-gram graph produced, which was used for the graph kernel calculation. \\

We can put together some of the ideas and topics discussed in the cluster. For example, in Figure 4.5 we see that ``Front Left Passenger", ``Rear Facing Seat", and ``Patient Compartment" are all skip-grams which appeared frequently in this cluster.\\

\begin{figure}
\includegraphics[width=6in]{Content/Images/graph_k5_2.png}
\caption{Skip-grams which occurred more than 50 times in hierarchical clustering with hyper-parameter set A, in cluster 3.}
\end{figure}

\section{reddit Threads}

\hspace*{0.3cm} For the reddit threads dataset, analysis was focused on the ability of the clusterings to organize threads based on their query words. The words used to query the \texttt{r/MentalHealth} subreddit (\texttt{https://www.reddit.com/r/mentalhealth}) all returned threads which were then clustered into new groups. Some of these clusters expressed strong preference for posts that corresponded with specific key words. \\

This data was collected with the \texttt{\{redditExtractoR\}} package \cite{rivera2015package}, and then we augmented and cleaned with \texttt{\{tidytext\}} \cite{silge2016tidytext} and other \texttt{\{tidyverse\}} tools \cite{wickham2019welcome}. These tools made it easy to extract the data from reddit, tidy the data, and perform text processing tasks such as tokenization and parsing for punctuation or numbers. Once the data was in a clean format, it followed the script map to get to a format where it could be analyzed in the same fashion as the NHTSA data was. Since this data was a large set, it utilized the scripts set up with \texttt{\{furrr\}} to speed up computation to under 20 minutes \cite{bengtsson2020unifying}.\\

To assess how many clusters to use, an ``elbow plot" was formed similar to those in Figure 4.1, see Figure 4.6. For the reddit threads, the hyper parameters from the NHTSA analysis were used, and the consistently lowest metric parameter set was chosen; skip-grams of $k=3$ and kernel parameter $\sigma = 1200$ were used in the reddit clustering analysis. In Figure 4.6, we see that 4 clusters is the optimal value, where additional clusters provides diminishing results. \\

\newpage

\begin{figure}
\includegraphics[width=6in]{Content/Images/reddit_wss.png}
\caption{Within sum of squares as a function of number of clusters}
\end{figure}

After computing the kernel and using these parameters, four clusters are produced. Examining the dendrogram, in Figure 4.7, we see that we have 2 very distinct groups, which then break into two smaller groups. These groups are very clear and provide confidence in the clustering results. Along side the dendrogram, there is a heat map of what proportion of documents (or threads) in the cluster came from a query word. In Figure 4.7, we see that clusters 3 and 4 collected posts from the ``angry" query word. Additionally cluster 1 displays a strong amount of its posts coming from the ``sad" query word. \\

\begin{figure}
\includegraphics[width=6in]{Content/Images/k4_reddit_clusters.png}
\caption{Left - Proportion of cluster's threads coming from each query word. Right - Dendrogram for reddit thread analysis, with 4 clusters.}
\end{figure}

The prominence of some of these query words in the clusters indicates that the tone or language used by the thread authors was picked out by the clustering methods. This is of particular interest, as the goal of these methods was to perform unsupervised text clustering which utilized more of the rich intricacies of language.\\

\begin{figure}
\includegraphics[width=6in]{Content/Images/reddit_network.png}
\caption{10 Reddit posts and their skip-grams which occurred 8 or more times.}
\end{figure}

\section{Comparative Analysis}

\hspace*{0.5cm} Looking at the results from both hierarchical clusterings on the graph kernels it is apparent that these methods work much better on larger documents. The performance of the graph kernels being applied to the skip-gram graph and then clustering was much better in the NHTSA dataset than the reddit threads dataset. By measuring within sum of squares, the NHTSA data set performed much better, as its WSS to number of observations ratio was 2.08 at its lowest, whereas the reddit dataset had a WSS to number of observation ratio of 9.\\

As for computational time, the NHTSA dataset took less than 2 minutes to compute, whereas the reddit dataset took just under 20 minutes to compute with the parallel processing. However, the reddit dataset displayed more prominent clusters, which was also indicative of success.\\

Comparing the elbow plots for both the NHTSA and the reddit threads datasets, it is clear that the optimal number of clusters for these methods will be dependent upon the dataset being analyzed. The results from the NHTSA study that indicated that skip-grams with $k=3$ did seem to apply to the reddit dataset as well. These hyper parameters will need tuning for any new application.\\

Overall, the largest difference between the two analyses was that although the reddit threads displayed strong, prominent clusters, the NHTSA dataset was being computed in reasonable amount of time. Since the application of these methods are scaling better for increasing document size instead of increasing document lists, there is a case to be made that aggregating reddit threads into larger documents may be more effective, but to keep the observational unit tied to a reddit thread, this was not completed.\\

These methods could be applied in any text mining application. Since the methods were able to be parallelized during their most computationally expensive portions of code, the methods could scale with sufficient hardware. In a high performance computing environment, these methods could either be applied in batch/scheduled computations or in a stream/real time application (with some modification). The readiness for HPC environments is important as it allows the code to scale and be used in much larger analytic applications.

\section{Contributions to the Field}

\hspace*{0.5cm} Through this work, several new contributions have been made to the field. First, a specific study on the use of graph kernels as preprocessing for hierarchical clustering; the combination of these ideas has not been well documented within the literature reviewed here. Second, the analysis of such differing datasets will bring new perspective to the topic as it relates to the performance of such methods on different types of text data. Thirdly, the workflow developed here was made open for others to use, edit, and analyze for future applications. All of the scripts, notebooks, supplementary materials, and data being made open encourages further development by the community. Lastly, the results of the analysis will go on to inform their respective subfields about the exciting applications of these methods. In the case of NHTSA, edge cases that can be summarized through these methods can lead to more representative simulations being made that can lead to better trained autonomous vehicles. For the reddit mental health data, the work completed here can be used for semi-supervised applications to classify discussion in these type of subreddits. The classification can inform mental health practitioners about the state of mental health within such communities at scale. They can then use those results to drive mental health awareness campaigns, better target what the community is feeling about treatment, or perhaps identify new language or a new subgroup of people dealing with a specific mental health issue. These contributions will support furthering research in this field, as well as the fields from which the data is of interest to.


\subsection{Tensors}

A tensor is an element of a tensor product.


\input{permve-ntnu-latex-assignment.tex}

\usepackage{float}
\usepackage{tabularx}

\title{
\normalfont \normalsize
\textsc{Norwegian University of Science and Technology\\IT3105 -- Artificial Intelligence Programming}
\horrule{0.5pt} \\[0.4cm]
\huge Module 6:\\ Deep Learning for Game Playing\\
\horrule{2pt} \\[0.5cm]
}

\author{Per Magnus Veierland\\permve@stud.ntnu.no}

\date{\normalsize\today}

\newacro{ANN}{Artificial Neural Network}
\newacro{SGD}{Stochastic Gradient Descent}

\begin{document}

\fancyfoot[C]{}
\maketitle

\newpage
\fancyfoot[C]{\thepage~of~\pageref{LastPage}} % Page numbering for right footer
\setcounter{page}{1}

\section*{Knowledge Representation}

The goal of the module is to find a way to train an \ac{ANN} with supervised learning such that it can beat a random player at the game \textsc{2048}. The assignment states the difficulty of this task clearly, and heavily suggests preprocessing the board state to expose features in a form which is easier to learn by the network.

Based on this, two preprocessing schemes has been developed to transform a board state into features which are used as inputs to the \ac{ANN}. To construct the training data for the two schemes, an evaluation function is needed to decide which move to choose for each network input when generating training data.

During prototyping, a small reference algorithm was written in Python to find a set of working features upon which to base a heuristic for satisfactory play. An important factor when doing supervised learning is that the answer in training examples must correspond well to the inputs presented in the training example. It was found when building the reference implementation that greedily selecting the move which maximizes the number of open cells will result in a mean score of $\approx 230$, which significantly beats the mean random play score of $\approx 110$ (see Table~\ref{table:results}). This same heuristic is used to produce the correct move for both approaches.

The first knowledge engineering approach is the most basic of the two and involves feeding the number of possible merges for each row and each column in the current state as inputs to the network; resulting in 8 network inputs. Intuitively this should work well, as the information of how many merges are possible in each direction should be sufficient to correctly chose the optimal move according to the heuristic of maximizing the number of free cells.

The second knowledge engineering approach is more complex. An obvious solution would be to feed the board state directly into the network. However, this representation is complex and involves a range of different values for each cell. In an effort to simplify the board input, while allowing the network to learn when cells can be merged, a reduction is performed on the input values. For a given board state, the number of distinct non-zero values in the state is first counted. If a row in the board state has the values \texttt{[0 2 4 2]}, then the row has two distinct non-zero values. After counting the number of non-zero values, each non-zero value on the board is reassigned to be its equal to its index in the sorted list of unique non-zero values. This reduction is meant to make the network blind to the magnitude of values in a board state and only treat values according to mergeability. The number of network inputs for the second approach is 16, corresponding to the reduced board cell values.

An important detail when producing training examples was to leave out examples where the heuristic was not able to distinguish one move as better than any other move. Unless the heuristic clearly knows that one move is the right move for the given state, it is assumed to not be beneficial to insist that the training examples follows the same ``random'' move that the example generator ends up making.

\begin{table}
\centering
{\small
\begin{tabular}{ccccccc}
\toprule
Player    & Dimensions                          & Hidden $f$    & Output $f$       & $\varepsilon$ [\%] & Mean score & $\sigma~\text{score}$ \\
\midrule
Random    & N/A                                 & N/A           & N/A              & N/A                & 107.26     &  54.4479              \\
Reference & N/A                                 & N/A           & N/A              & N/A                & 231.87     & 125.5831              \\
Network A & $8 \times 4$                        & \textsc{ReLu} & \textsc{Softmax} & 0.0000             & 250.56     & 129.0481              \\
Network B & $16 \times 512 \times 512 \times 4$ & \textsc{ReLu} & \textsc{Softmax} & 0.2563             & 221.31     & 117.8580              \\
\bottomrule
\end{tabular}
}
\caption{Statistics comparing the two chosen networks and the reference implementation. All networks use cross-entropy cost functions and are trained with learning~rate~0.08. Network~A is trained with minibatch size 40 and network~B with minibatch size 50. All scores are based on highest tile present at end of game; averaged over 1000 games. $\varepsilon~=~\text{Training set error}$. $\sigma~=~\text{standard deviation}$.}
\label{table:results}
\end{table}

\section*{Network Design}

The network inputs chosen for both representations has the same range in each representation. By using the \textit{rectifier} activation function, scaling the inputs is not required as the activation function cannot be saturated, and because all values will be close to zero already. The \textit{softmax} function is used for the output nodes to rank the most probable move.

The first network using the basic knowledge representation scheme does not require any hidden layer. It is able to train to 0\% error within the first couple of epochs. It is however necessary to provide a large enough number of training examples. With training examples based on 100 games played a training error of 0\% was not achieved. However, increasing the number of training examples to 1000 games was sufficient to complete optimal training. It is clear that the first knowledge representation scheme requires little ``intelligence'' from the network. This is intuitive, as it should be straightforward to select an optimal move to maximize the number of free cells based on the number of possible merges for each row and column.

The second network requires a much larger topology. Achieving good training errors required large hidden layers, and the final network uses two hidden layers of 512 nodes each. It is possible to train a network to achieve a training error of 0 since the preprocessing used for the first network is a simple algorithm. However this will likely require more training data and possibly even larger topology.

Both networks use \textit{softmax} output nodes and the cross-entropy loss function as these yielded good results for the MNIST dataset in module~5.

Running each trained network for 1000 games and running a \textit{Welch}-test produces a $p$-value of 0.0. It is clear that network~A with the much simpler input and topology beats network~B with the more complex input and topology. It is also clear that the choices and reduction algorithm used to produce feature input to the more complex network produces good results which could be improved further with the same input.

\section*{Play Analysis}

Both network configurations and their associated preprocessing stages are able to approximate the heuristic function well by achieving a low training error. When the training error is 0, bad moves will either occur because the training set is not large enough for the \ac{ANN} to capture the information necessary to model the heuristic function -- or because the heuristic function simply made an evaluation with poor results.

During observed play bad moves were only seen which were caused by the simplistic heuristic function chosen.

\begin{figure}[!h]
\centering
\begin{tabularx}{\textwidth}{cXc}
\includegraphics[scale=0.35]{bad_1} & ~ & \includegraphics[scale=0.35]{bad_2} \\
\end{tabularx}
\caption{\textit{Example of poor gameplay:} Instead of moving up to gather the 4-tile, the 8-tile and the 16-tile to the right -- while keeping the 2-tiles to the upper left gathered -- the \ac{ANN} instead choses to greedily merge two 2-tiles while risking worsening the board state by permitting tiles to spawn at the top of the board.}
\label{fig:N1}
\end{figure}
\begin{figure}[!h]
\centering
\begin{tabularx}{\textwidth}{cXc}
\includegraphics[scale=0.35]{good_1} & ~ & \includegraphics[scale=0.35]{good_2} \\
\end{tabularx}
\caption{\textit{Example of good gameplay:} In this scenario the board state offers the possibility to merge four tile pairs. The greedy heuristic which the \ac{ANN} models recognizes this possibility and performs four simultaneous merge while at the same time lining up two new merges.}
\label{fig:N1}
\end{figure}

\end{document}


As early as 1964, Goffman~\cite{goffman64OnRelevanceAsAMeasure}, a
mathematical information science pioneer \cite{harmon08RememberingWG},
notes that the relevance of documents in a list has to depend on the
documents preceding it.  More recently, work on
MMR~\cite{carbonell98MMR} was one of the first to formalize
diversification as a mathematical optimization criterion; MMR has
proved one of the most popular diversity approaches.  We note that the
results as derived in the last section formally motivate much of MMR and now
we discuss further connections between optimizing expected $n$-call@$k$
and other diversification approaches proposed in the literature.

\subsection{Greedy Submodular Optimization}

%% SHENGBO: This section is almost entirely about Exp-1-call@k not n-call
%%          since all of this text predates SIGIR 2012.  Do any of these
%%          comparisons extend to n-call?  Not all do, but you should
%%          make any connections where possible -- i.e., a max is allowed.
%%          I modified the last item on rank-based objectives to
%%          mention n-call.

\cite{agrawal09diversifying} proposes a set-based objective function
to answer ambiguous web queries in a setting where there exists a
predefined taxonomy of information, and that both queries and
documents may belong to more than one category according to this
taxonomy. The proposed set-based objective function aims at maximizing
the probability that the average user finds at least one useful
resulting document retrieved within the top $k$
results. Mathematically, this objective function
is defined below:
%% SHENGBO - IA-Select is the greedy algorithm, not the objective
%
%% SHENGBO - don't define new notation, c is really equivalent to topics
%%           t and you should use notation for S_k, S_k^*, s_i, s_i^* defined previously.
\begin{align}
	P(S_k|\vec{q}) = \sum_{t\in T} P(t|\vec{q}) \left( 1 - \prod_{s_i\in S}(1-V(r_i| \vec{q}, s_i))\right)
\label{eq:diversifykObjectiveFunction}
\end{align}
where $V(r_i| \vec{q}, s_i)$ broadly defines the
likelihood that a document $s_i$ satisfies the query $\vec{q}$ given the
topics of the query $\vec{q}$ and the document $s_i$, i.e., $t$ and $t_i$, respectively\footnote{We
have slightly adapted notation in the above equation from the original
for consistency with our previous definitions in this article.}.
%% SHENGBO - learning has nothing to do with this paper - LDA in the
%%           PLMMR paper was inappropriately applied (it is an admixture model,
%%           not a mixture model) and we have not proposed any learning in
%%           approach in this work -- indeed we suggested non-learning similarity
%%           metrics could be converted to topic probabilities to match MMR.
%, and note also that the taxonomy of $c$ given the query
%and document is not learnt by some unsupervised model, but
%hand-crafted.

The authors point out that one particular advantage of this set-based objective
function is that it is submodular, hence permitting a $\left( 1 - \frac{1}{e} \right)$
approximation if optimized greedily as shown in~\cite{Nemhauser:MathProg1978}.

Referring back to our results shown in Equation~\ref{eq:partial_simp}~and~\eqref{eq:diversifykObjectiveFunction} from 1-call@$k$, we note that the objective function in Equation \eqref{eq:diversifykObjectiveFunction} is related with ours, but it is unclear if Equation \eqref{eq:diversifykObjectiveFunction} can be derived from the 1-call@$k$ objective, suggesting potential improvements in \cite{agrawal09diversifying}.
%writing out the mathematical expectation in terms of the sum over all
%possible topics (e.g., taxomony in \cite{agrawal09diversifying}) the
%weighted relevance where weights are the topic distributions below
%%% SHENGBO: the following is not correct, look at Eq (7)
%%%          you need P(t_k=t|s_k) and i={1..k-1}
%%%
%%%          This means P(c|q) should be replaced with P(t|\vec{q}) * P(t_k=t|s_k)
%%%          if deriving from 1-call@k... it's not clear whether *their* result
%%%          can be derived from an objective and model and this suggests their
%%%          approach may be improved.
%\begin{align*}
%    \ExpOneCall(S_k,\vec{q}) & = \mathbb{E} \left[\left. \bigvee_{i=1}^{k}r_i=1 \right| s_{1},\dots, s_{k},\vec{q} \right], \\
%    												 & = \sum_{t\in T} P(t|\vec{q}) \left( 1 - \prod_{i=1}^{k}(1-p(t_i = t| q, s_i))\right)
%\end{align*}
%Clearly our proposed objective is equivalent to the objective
%Equation~\eqref{eq:diversifykObjectiveFunction} proposed
%in \cite{agrawal09diversifying} when one replaces the likelihood
%function $V(s|\vec{q}, c)$ by $p(t_i = t| \vec{q}, s_i)$.

More recently, \cite{Vargas:SIGIR2012} introduce several interesting formal probabilistic
relevance models to instantiate $V(r_i| \vec{q}, s_i)$, which are more
appopriate in modeling the relevance in a probabilistic
framework. Furthermore, \cite{Vallet:SIGIR2012} propose to introduce a
user as an explicit random variable in state of the art
diversification methods, thus developing a generalized framework for
personalized diversification.

Another recent interesting instantiation of $V(r_i|\vec{q},s_i)$ is proposed
in \cite{Zuccon:ECIR2012} motivated by the facilitation location
problem~\cite{Gonzalez:Handbook2007} taken from Operations Research:
for a set of customer ``locations" $D$, one aims at choosing a subset
$S_k$ in $D$ to open $k$ ``facilities" that optimize a graph-theoretic
objective that depends on the cost of opening a facility at each
location and also the distance between each pair of locations. However
all of the described methods do not derive their objective
functions from the expected 1-call@$k$ objective as we have achieved.

\subsection{Portfolio Theory}
\cite{wang09PortfolioTheory} motivates
diversification in set-based information retrieval by a
risk-minimizing portfolio selection approach. Viewing a result set as
an investment portfolio with the objective to maximize return while
minimizing risk, the derived result of~\cite{wang09PortfolioTheory}
mimics both MMR and Exp-$1$-call@$k$ in that the similarity term may
be viewed as \emph{expected portfolio payoff} (relevance) and the
diversity term may be viewed as \emph{expected portfolio risk}, which
increases as the correlations between documents in the result set
increase.
%% SHENGBO -- you used to have math here for Wang09 which you've omitted.
%%            It's not very clear to say a sum here if you never show it!
%%            (There was not room in CIKM to display this.)
%%
%%            Also I suggest dropping the mention of Shi -- it is not
%%            clear and it interrupts the flow of discussion about wang
%%            and the lead-in to the next section.
%Note that diversification based on portfolio theory is
%extended in \cite{Shi:SIGIR2012} by introducing latent factors for
%collaborative filtering tasks.

One major difference in the
framework~\cite{wang09PortfolioTheory} is that rather than computing
the diversity term via a max (MMR) or product (Exp-$1$-call@$k$) the
portfolio theory derivation uses a summation:
\begin{align}
s_{k}^{*} = \argmax_{s_{k}} \left[ \Sim^{\text{BM25}}(s_k,\vec{q}) \hspace{-.7mm} - \hspace{-.7mm} \lambda \sum_{i=1}^{k-1} \omega_{i} \Sim^{\text{TFIDF}}(s_i,s_k) \right]
\label{eq:wang09PortfolioTheory}
\end{align}
Here, $\lambda$ is a manually tuned weight parameter, $\omega_1,\ldots,\omega_k$
are the weights of each position in the result set,
$\Sim^{\text{BM25}}(s_k,\vec{q})$ is the BM25~\cite{bm25} probabilistic
relevance score of document $s_k$ w.r.t.\ query $\vec{q}$ and
$\Sim^{\text{TFIDF}}(s_i,s_k)$ is a TFIDF~\cite{salton83Introduction}
similarity metric between two documents $s_i$ and $s_k$ represented as term
frequency vectors. We examine the implications of using the summation in Equation~\eqref{eq:wang09PortfolioTheory} next.

\subsection{Set Covering}
Yue and Joachims~\cite{yue081224Predicting} propose a set covering
approach for training SVMs to predict diverse result sets for
information retrieval. In their work, they equate subtopics with
words and build a loss function for SVM training that penalizes result
sets according to the sum of weights of query-relevant words
\emph{not} covered by the result set. The proposed loss function is called the weighted subtopic loss $\text{WSL}(S_k,
\vec{q})$, which is the weighted percentage of distinct
subtopics in the topic set of query $\vec{q}$ \emph{not} covered by the
retrieved set $S_k$. Denoted by the set of topics for query
$\vec{q}$ as $T_{\vec{q}}$ and denote the set of topics in $T_{\vec{q}}$ covered
by $S_k$ as $T_{S_{k}}$, Yue and Joachims define this loss function below:
\begin{align}
    \text{WSL}(S_k, \vec{q}) = \sum_{i \in T_{\vec{q}} \setminus T_{S_{k}}} \frac{n_{i}}{\sum_{i \in T'} n_{i}} \label{eq:wsl}.
\end{align}
This loss function function penalizes a result set $S_k$ by the topics
$T_{\vec{q}} \setminus T_{S_{k}}$ according to the sum of their weights, where topic $i$ is weighted according to the count $n_i$ of documents in $D$
covered by topic $i$ (normalized by $\sum_i n_i$).

While the approach by Yue and Joachims~\cite{yue081224Predicting} provides a ``hard'' set-covering view of diversity, we note that an expansion of
$\tilde{P}(t | S_{k-1}^*)$ used in the diversity term
of~\eqref{eq:1call} provides a ``soft'' latent set-covering
interpretation; that is, $s_k$ is chosen so as to best cover (in a
probabilistic sense) the latent topic space not already covered by $\{
s_1^*,\ldots,s_{k-1}^* \}$.  Formally, expanding the product in
$\tilde{P}(t | S_{k-1}^*) = \prod_{i=1}^{k-1} \left(1 -
P(t_{i}=t|s_{i}^{*})\right)$, collecting terms and writing it as a
series, we arrive at a form that reflects the inclusion-exclusion
principle applied to the calculation of probability that topic $t$ is
covered by $\{ s_1^*,\ldots,s_{k-1}^* \}$:
\begin{align}
& \prod_{i=1}^{k-1} \left(1 - P(t_{i}=t|s_{i}^{*})\right) \nonumber \\
& = 1 - \left[ \sum_{i=1}^{k-1} P(t_{i}= t|s_{i}^{*}) - \sum_{i=1}^{k-1}\sum_{j=1}^{k-1}P(t_{i}= t|s_{i}^{*})P(t_{j}= t|s_{j}^{*}) + \dots - (-1)^{k-1}\prod_{i=1}^{k-1}P(t_{i}=t|s_{i}^{*})\right] \label{eq:setcover}
\end{align}

This result has a natural interpretation: the first summation term
determines the coverage of topic $t$ by each document $s_i$ ($1 \leq i
\leq k-1$) currently in the result set, the second double summation
term corrects the first term by removing the joint probability mass
from all pairs of documents that was double counted, and so on
according to the principle of inclusion-exclusion.
\eqref{eq:setcover} not only provides a probabilistic set covering
view of Exp-$1$-call@$k$, but it also suggests that a portfolio
approach to diversity using only the first summation would overcount
each document's contribution to the diversity metric according to this
set covering perspective.

The inclusion-exclusion principle calculation provided by the second
term in Equation~\ref{eq:setcover} is illustrated in~Figure~\ref{fig:inclusionExclusionPrinciple}. In words, this term
is calculating the total topic probability coverage of $t$ by all
selected items $\{ s_1^*,\ldots,s_{k-1}^* \}$ by properly applying the
inclusion-exclusion principle to ensure that overlapping probability
coverage is not double counted. Then referring back to
Equation~\ref{eq:partial_simp}, we note that $s_k$ is chosen by
maximizing a weighted sum over topics, where each topic weight is
determined by its relevance to the query $\vec{q}$, the item $s_k$,
and penalized (i.e., due to the $1 - $) by the topic coverage of $t$
by the set $\{ s_1^*,\ldots,s_{k-1}^* \}$ to naturally encourage
diversity.  We note that this is a soft probabilistic version of the
``in or out'' topic coverage approach of the weighted subtopic loss function $\text{WSL}(S_k,
\vec{q})$ (Equation~\eqref{eq:wsl}).
%% SHENGBO: WSL is not defined!!!

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[t!]
\begin{center}
\centerline{\includegraphics[scale = 0.4]{inclusionExclusionPrinciple}}
\caption[Inclusion-exclusion principle.]{Inclusion-exclusion principle. The sets represent candidate
items $s$ for a query, and the area covered by each set is the
``information" covered by that item for query topic $t$. Numbers on different areas
indicates the number of sets that share these areas. }
\label{fig:inclusionExclusionPrinciple}
\end{center}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Subtopic Relevance Models}

We use a subtopic relevance model that is a simplified version of the
model in~\cite{plmmr} with fewer dependence assumptions.  In other
work, Zhai {\it et al}~\cite{zhai03Beyond} present an empirical risk
minimization view of dependent document retrieval from a subtopic
perspective, where they derive a formalization of the
\emph{greedy} selection step that is similar to MMR and to a lesser
extent, Exp-$1$-call@$k$.

\subsection{Set-based Relevance Objectives}

Chen and Karger~\cite{chen06Less}, whose derivation we extended,
directly optimize $1$-call@$k$, but their intention is not to
formalize MMR and instead use na\"{i}ve Bayes to directly evaluate
\eqref{eq.ncall}.  Agrawal et al~\cite{agrawal09diversifying}
and Santos et al (xQuad)~\cite{santos2010xquad} both specify set-based
diversity metrics \emph{very} similar to Exp-$1$-call@$k$ but do not provide
formal derivations as we have done in this work.

\subsection{Ranking Based Objectives}

Finally, returning to our introductory motivation, Wang and
Zhu~\cite{wangzhu10} have shown that natural forms of result set
diversification arise via the optimization of average
precision~\cite{ap} and reciprocal rank~\cite{mrr}.  Both of these
methods share the view of directly optimizing a
\emph{ranking-based} objective, whereas this paper proposes a novel
derivation from the alternate view of optimizing a \emph{set-based}
objective w.r.t.\ a subtopic model of relevance.  However, even though
Exp-$n$-call@$k$ is a set-based objective, an indirect consequence of
(and motivation for) greedily optimizing it is that documents added
earlier yield a greater increase in objective than those added later;
this yields a natural rank ordering on the greedy Exp-$n$-call@$k$
result set.

%% SHENGBO: I removed the table because you are referring to learning and
%%          a lot of traits that were relevant to PLMMR in SIGIR 2010 that
%%          are no longer relevant now.  Either you need to do an entirely
%%          new analysis relevant to this paper and its objectives, or just
%%          drop it as I did.
%%
%%          Old content is checked in under related_work_unused.tex


% Created 2020-07-13 lun 12:51
% Intended LaTeX compiler: pdflatex
\documentclass[letterpaper,fleqn]{scrartcl}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{khpreamble}
\usepackage{tabularx}
\usepackage{geometry}
\usepackage{pgfplots}
\pgfplotsset{compat=1.13}
\geometry{top=20mm, bottom=20mm, left=24mm, right=18mm}
\author{Kjartan Halvorsen}
\date{}
\title{Polynomial design (RST) exercise}
\hypersetup{
 pdfauthor={Kjartan Halvorsen},
 pdftitle={Polynomial design (RST) exercise},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 26.3 (Org mode 9.3.6)},
 pdflang={English}}
\begin{document}

\maketitle

\section*{Determine the order of the controller}
\label{sec:org6e78179}
\begin{center}
\includegraphics[width=0.7\linewidth]{../../figures/2dof-block-explicit}
\end{center}
In each of the cases determine the order of the feedback controller \(F_{b}(z)=\frac{S(z)}{R(z)}\) and write out the \(R(z)\) and \(S(z)\) polynomials. Determine also the order of the observer polynomial \(A_o(z)\). You don't have to solve for the controller coefficients.

\subsection*{Case 1}
\label{sec:orgb815efe}
Plant is \(H(z) = \frac{b_0z+b_1}{z^3  + a_1z^2 + a_2z}\),  desired response to reference signal \(H_c(z) = \frac{0.2^2}{z(z-0.8)(z-0.8)}\), observer poles in the origin.

\vspace*{27mm}

\subsection*{Case 2}
\label{sec:org6948edd}
Plant is \(H(z) = \frac{b_0z+b_1}{z^3  + a_1z^2 + a_2z}\),  desired response to reference signal \(H_c(z) = \frac{0.2^2}{(z-0.8)^3}\), observer poles in the origin and integral action in the feedback controller (incremental controller).

\vspace*{27mm}

\subsection*{Case 3}
\label{sec:orgef90a48}
Plant is \(H(z) = \frac{b_0z+b_1}{z^2  + a_1z + a_2}\) and there is a delay of 2 sampling periods in the feedback path. The desired response to reference signal \(H_c(z) = \frac{0.2^2}{(z-0.8)(z-0.8)}\), observer poles in the origin and integral action in the feedback controller (incremental controller).
\end{document}


\documentclass{report}

\title{AICC II \\ Prof. Bixio Rimoldi}
\author{Benjamin Bovey}
\date{Semester of Spring 2019}

%math packages
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{amssymb}
\usepackage{amsthm}

\usepackage{thmtools}
%\usepackage{thmbox}
%\usepackage{shadethm}

\usepackage[dvipsnames]{xcolor}

%utility packages
\usepackage{cancel}
\usepackage{soul}

%margins
\usepackage{geometry}
\geometry{left=2cm, right=2cm, top = 2cm, bottom=2cm}

\newcommand\important[1]{\noindent {\underline{\textsc{#1}}} \ }
\DeclareMathOperator{\supp}{supp}

\declaretheorem[name=Theorem, style=plain, shaded={rulecolor=Lavender, rulewidth=2pt, bgcolor={rgb}{1,1,1}}]{thm}

\declaretheorem[name=Definition, sibling=thm, style=definition, shaded={rulecolor={blue!30}, rulewidth=2pt, bgcolor={rgb}{1,1,1}}]{defn}
\declaretheorem[name=Example, sibling=thm, style=definition]{exmp}
\declaretheorem[name=Summary, numbered=no, style=definition]{summary}

\declaretheorem[name=Remark, numbered=no, style=remark]{remark}

%\theoremstyle{plain}
%\newtheorem{thm}{Theorem}
%
%\theoremstyle{definition}
%\newtheorem{exmp}[thm]{Example}
%\newtheorem{defn}[thm]{Definition}
%\newtheorem*{summary}{Summary}
%
%\theoremstyle{remark}
%\newtheorem*{remark}{Remark}
%\newtheorem*{question}{Question}
%\newtheorem*{answer}{Answer}

% the next 3 lines are for removing section numbering. You may want to comment them out.
\makeatletter
\renewcommand{\@seccntformat}[1]{}
\makeatother

\begin{document}
\maketitle

\section{19th February 2019}
As opposed to the first AICC course, where we were mostly presented with tools, we will now see more applications of these tools for communication and computation. Mainly, we will see 3 applications in the first part of the semester:
\begin{itemize}
	\item \textbf{Source coding} (compressing information)
	\item \textbf{Cryptography} (authentication / privacy / integrity of information)
	\item \textbf{Channel Coding} (dealing with noise and loss of information / protecting the information from natural damages)
\end{itemize}
What these three have in common is the idea of storing and communicating information. The notion of entropy, which will come up quite often, will also be important.

\subsection{Basic probability review}
\textbf{Special case first}: \underline{finite} sample space $\Omega$ and \underline{uniform distribution}. $\Omega = \{\omega_1, \omega_2, \dots, \omega_n\}$. Events: $E \subseteq \Omega$. Then:
\begin{equation}
	P(E) = \dfrac{|E|}{|\Omega|} \qquad \text{(uniform distribution)}
\end{equation}

\begin{defn}[conditional probability]
Let $E, F$ be two events. Then, the probability that event $E$ occurs knowing that $F$ has occured:
\begin{equation}
	P(E|F) = \dfrac{|E \cap F|}{|F|}
\end{equation}
\end{defn}
Intuitively, you restrict the sample space to $F$ only, because you \emph{know} that $F$ has happened: this translates to the division by the cardinality of $F$ instead of the cardinality of $\Omega$. The intersection of $E$ and $F$ follows from the fact that the sample space is restricted to $F$: if there exists elements that are in $E$ but not in $F$, they are outside of the new sample space $F$, which means that these elements CANNOT occur in conjunction with $F$. Therefore, we take the intersection of $E$ and $F$ to assure that these elements are not taken into account in the computation.

\begin{thm}[Law of total probability]
Let $E$ and $F$ be two events in $\Omega$, and let $F^C$ denote the complement of $F$. Then:
\begin{equation}
	P(E) = P(E|F)P(F) + P(E|F^C)P(F^C)
\end{equation}
This follows quite directly from the fact that $E = (E \cap F) \cup (E \cap F^C)$, so $P(E) = P(E \cap F) + P(E \cap F^C)$ \dots
\end{thm}

\begin{remark}[divide and conquer] You can sometimes create a partition of your sample space, in a way that allows you to better apply the numbers you are given (p.27-28). This method is called \emph{divide and conquer}.
\end{remark}

\begin{thm}[Bayes]
Bayes' theorem allows you to compute $p(F|E)$, given that you know $p(E|F), p(E)$ and $p(F)$:
\begin{equation}
	p(F|E) = \dfrac{p(E|F)p(F)}{p(E)}
\end{equation}
\end{thm}

\begin{remark}[application] This is useful, in real scenarios, when either one of $p(E|F)$ and $p(F|E)$ is easily observable, but the other isn't. For example, policemen may observe how many people are driving drunk knowing that they have had an accident (they just test the driver after the accident), but they cannot observe how many people are having an accident knowing that they are driving drunk (they can not really test drivers, and then let them drive drunk just to check if they have an accident or not).
\end{remark}

\begin{defn}
A \textbf{random variable} $X$ is a function $X: \Omega \to \mathbb{R}$. It is attached a \emph{probability distribution function} $p_X(x)$, which represents the probability that $X$ will take on the value $x$, that is, that the following event $E$ occurs:
\begin{equation}
	E = \{\omega \in \Omega : X(\omega) = x\}
\end{equation}
Hence,
\begin{equation}
	p_X(x) = p(E) = \sum_{\omega \in E} p(\omega)
\end{equation}
The set of all possible values of $X$ is sometimes called the \emph{alphabet} of $X$, written with more curly letters like $\mathcal A$.
\end{defn}

\begin{defn}[two random variables]
	Let $X: \Omega \to \mathbb{R}$ and $Y: \Omega \to \mathbb{R}$ be two random variables. \\
	The probability of the event $E = \{\omega \in \Omega : X(\omega) = x \land Y(\omega) = y\}$, or, more shortly written, $\{X = x \land Y = y\}$, is
	\begin{equation}
		p_{X, Y}(x, y) = \sum_{\omega \in E} p(\omega)
	\end{equation}
	We can compute $p_X$ (or $p_Y$, similarly) from $p_{X, Y}$:
	\begin{equation}
		p_X(x) = \sum_y p_{X, Y}(x, y)
	\end{equation}
\end{defn}
In one sense, we "fix in place" the value of $x$ and "scroll through" all possible values of $y$, and add their probabilities up. Here, $p_X$ is called the \textbf{marginal distribution} of $p_{X, Y}(x, y)$ with respect to $x$.

\section{21st February 2019}

\begin{defn}[expected value]
The \textbf{expected value}, or \textbf{mean} of a random variable $X: \Omega \to \mathbb{R}$, can be computed as
\begin{equation}
	E[X] = \sum_{x \in \mathcal A(X)} x \cdot p_X(x) \quad \text{(requires $p_X$)},
\end{equation}
or as
\begin{equation}
	E[X] = \sum_{\omega \in \Omega} X(\omega) \cdot p(\omega).
\end{equation}
\end{defn}
One could say that the first way is ``calculating over the codomain'', and the second way is ``calculating over the domain'' (of $X$).

\begin{remark} The expected value is a linear operation. Let $X_1, X_2, \dots, X_n$ be random variables from $\Omega$ to $\mathbb{R}$, and let $\lambda_1, \lambda_2, \dots, \lambda_n$ be real numbers. Then
\begin{equation}
	E\underbrace{\Bigr[\sum_{i=1}^n \lambda_i X_i\Bigl]}_{\text{random variable}} = \sum_{i=1}^n \lambda_i E[X_i]
\end{equation}
\end{remark}

\subsection{Extending notions from events to random variables}
The notion of independent events extends to random variables. Recall that two events $E$ and $F$ are independent iff $p(E|F) = p(E)$, which is equivalent to saying that $p(E \cap F) = p(E)p(F)$.

Similarly, two random variables are independent iff the value taken by one does not influence the value taken by the other.

\begin{defn}[independent random variables]
	We say that two random variables $X, Y: \Omega \to \mathbb{R}$ are \textbf{independent} iff
	\begin{equation}
		p_{X, Y}(x,y) = p_X(x)p_Y(y)
	\end{equation}
	More generally, $n$ random variables are independent iff
	\begin{equation}
		p_{S_1, \ldots, S_n} = \prod_{i=1}^n p_{S_i}
	\end{equation}
\end{defn}

\noindent From there, we can also extend the notion of conditional probability to random variables.
\begin{defn}
We define the \textbf{conditional probability of two random variables} $X, Y: \Omega \to \mathbb{R}$ as
\begin{equation}
	p(X=x | Y=y) = \dfrac{p(X=x \land Y=y)}{p(Y=y)},
\end{equation}
or, with simpler notation,
\begin{equation}
	p_{X|Y} = \dfrac{p_{X, Y}(x, y)}{p_Y(y)}
\end{equation}
\end{defn}

\begin{remark} The following statements are all equivalent to the statement ``$X$ and $Y$ are independent'':
\begin{align}
	p_{X, Y}(x, y) = p_X(x) \\
	\label{eq:condprobtip1}
	p_{X|Y}(x|y) = p_X(x) \\
	\label{eq:condprobtip2}
	p_{Y|X}(y|x) = p_Y(y)
\end{align}
\end{remark}

\begin{remark}[useful trick] If you are asked to check the independence of $X$ and $Y$, you don't have to check the equality of \eqref{eq:condprobtip1} or \eqref{eq:condprobtip2}. You just have to find the expression for the left-hand side function, and see if it depends on the other variable ($\implies$ they are NOT independent), or if it is just a function of one variable ($\implies$ they are independent).
\end{remark}

\begin{thm}[consequence of the independence of random variables] In all cases, $E[X+Y] = E[X] + E[Y]$. However, if $X$ and $Y$ are \emph{independent}, we also have that
\begin{equation}
	E[XY] = E[X]E[Y]
\end{equation}
\end{thm}

\subsection{Source \& Entropy}
The main object that will interest us when studying source coding is the source itself. The question of the definition of a source took mathematicians and computer scientists a while to answer. We can loosely model a source as a black box that outputs \emph{information}: we are not really interested in its inner mechanisms, but rather in the information that comes out of it. This information could take many forms, for example sequences of symbols: since we are considering sources from a computer science point of view, we will be interested in sources that shite out sequences of numbers. \\
 The notion of \emph{entropy} comes into the frame when we realize that a symbol that can be \emph{predicted} before it comes out of the source provides no new information. For example, if the sequence of numbers coming out of the source is \texttt{1, 1, 5, 5, 3, 3, 19, 19, 5, ...}, we quickly realize that we do not need to store the second number of each pair, as it brings no new information to the table. \\
 An important observation that we can make at this point (it was initially made by Hartley in 1929) is that this link between information and entropy can be modeled very elegantly by random variables! If we think about it, the core idea of a random variable is that it gives you a value that you cannot certainly predict until you actually do the experiment that it models and observe its outcome (hence, in fact, the name of \emph{random} variable). If we choose to use this model, a source can be viewed as outputting a sequence of random variables, where each random variable represents one (or more, as we'll see later) symbol(s) in the sequence of symbols. \\
Let us now consider a source outputting a sequence of random variables, call them $S_1, S_2, S_3, \dots, S_n$. Another fundamental question we may ask ourselves is: \ul{how much information is actually conveyed by each individual symbol?} A partial answer was given by Hartley, that is, that this must depend on the \textbf{alphabet} of the random variable.
\begin{defn}
	The \textbf{alphabet} of a random variable is the codomain of the random variable, that is, the set of all values that the random variable may take.
\end{defn}
Indeed, the bigger the alphabet, the more information it can carry, as there are more possibilities for each symbol, and therefore less predictability. With basic combinatorics, we can see that there are $|\mathcal A|^n$ possible length-$n$ sequences $(s_1, s_2, \dots, s_n)$. Therefore, the amount of information carried by $S_i$ is $\log_b |\mathcal A|$ \footnote{We will see later that the value of $b$ determines the unit of information used. Most often, it is 2, which means that the bit is the unit.}.
\begin{exmp} Imagine this is the sequence of good days (1) and bad days (0) in London during a year:
\begin{equation*}
	(s_1, s_2, \dots, s_{365}) = (0, 1, 1, 0, 1, 0, 0, 0, 1, 0, \dots, 0, 0, 1).
\end{equation*}
 This sequence of numbers is very unpredictable, as there is no constant pattern underlying it. It would therefore be hard to find a better way of storing this information (without losing any) than just storing all the 365 bits individually. When this happens, we will see that what we shall soon define as the \textbf{entropy} of this random variable is very high. \par
Now imagine that in San Diego, the sequence looks like this:
\begin{equation*}
	(\overbrace{0, 0, 0, \dots, 0}^{\text{24 zeros}}, 1, \overbrace{0, 0, 0, \dots, 0, 0}^{\text{340 zeros}}).
\end{equation*}
 This is a very predictable sequence: we could, for example, just store $(24, 1, 340)$ rather than storing all 365 bits. This means we can shrink down how we represent this information without losing any information! Here, the amount of information is much lower than in the case of London, and the entropy is very low.
\end{exmp}

\subsection{Entropy redefined by Shannon}
Shannon, in 1948, gave a new formula for the amount of information carried by a random variable $S$. He found out that the amount of information \emph{is} in fact the entropy itself\footnote{Like we saw with the example of London and San Diego weather, when the entropy is very low (which means that the source is very predictable), we can shrink down the information to store it easier. What that really means is that we can cut out unnecessary bits that do not bring any new information. This allows us to observe that when entropy is low, the actual amount of information is low. Similar observations can be made with high entropy and high amounts of information.}, and gave this formula for the entropy $H(S)$:
\begin{equation}
	H(S) = -\sum_{\mathclap{s \in \supp (p_S)}} p_S(s) \cdot \log_b \bigl( p_S(s) \bigr)
\end{equation}

\begin{remark} We may observe some things:
	\begin{itemize}
		\item The rather heavy notation $s \in \supp (S)$ is needed because $\log(0)$ is undefined. However, if we accept the common convention $0 \cdot \log(0) = 0$, then we can simplify the notation to this: \begin{equation*}H(S) = - \sum_{s \in \mathcal A} p_S(s) \log_b \bigl(p_S(s)\bigr); \end{equation*}
		\item when $b=2$ then the unit is the bit. By default, $H(S) = H_2(S)$;
		\item we may rewrite the formula as
		\begin{equation*}
		H(S) = \sum\limits_{s \in \mathcal A} p_S(s) \underbrace{\Bigr(-\log_b \bigl(p_S(s)\bigl) \Bigl)}_{\text{rand. var. $X$}} = E[X],
		\end{equation*}
		because this is the expression of the expected value of a random variable $X$.
		\item Since the sources that we will study are most often sequences of random variables, it is important to know that \ul{entropy can extend to any number of random variables}.
	\end{itemize}
\end{remark}

\begin{exmp}
	Let us try and give an intuitive example of entropy applied to a sequence of random variables. Let $S_1, S_2, \ldots, S_n$ be a sequence of coin flips. Then $S_i \in \{0, 1\} = \mathcal A$, and $P_S(s) = \frac12$. \\
Intuitively, we are flipping $n$ coins, and it should take $n$ bits to describe the result (a length-$n$ bitstring).
\begin{align*}
	&\underbrace{P_{S_1, S_2, \ldots, S_n}(s_1, s_2, \ldots, s_n)}_{\text{abbreviate as } P(s_1, \ldots, s_n)} = \prod_{i=1}^n P(s_i) = \left(\frac12 \right)^n \\
	&H(S_1, S_2, \ldots, S_n) = \log\left(|\mathcal A|^n \right) = n \log 2 = n
\end{align*}

Recall that when we write $P(s_1, s_2, \ldots, s_n)$, we mean the probability of getting the sequence $(s_1, s_2, \ldots, s_n) \in \mathcal A^n$. The cardinality of the alphabet is $|\mathcal A^n| = |\mathcal A|^n$.
\end{exmp}

\section{26th February 2019}
We saw last time that a source can mathematically be modeled as one or more random variables, each being described by its probability mass function. We saw that the entropy is a number which represents the ultimate amount of bits (not necessarily, but generally, binary) needed to represent a random variable (and therefore a source).

\begin{defn}[binary entropy function] When the random variable $S \in \{0, 1\} = \mathcal A$ with $p_S(0) = P$ represents a Bernouilli trial, that is, its alphabet is of size 2, we may compute the entropy as
\begin{align*}
	H(S) &= - \sum_{s \in \mathcal A} p_S(0) \log_b \bigl(p_S(0)\bigr) \\
	&= \underbrace{-P \log P - (1-P) \log(1-P)}_{h(P) \text{ function of } P}
\end{align*}
This function $h(P)$ is called the \textbf{binary entropy function}.
\end{defn}

Many results in information theory are the consequence of the following inequality \ref{thm:it-inequality}.
\begin{thm}[IT inequality]\label{thm:it-inequality}
Let $r > 0$. Then
\begin{equation}
	\log_b(r) \leq (r-1) \log_b (e) % WHAT IS e
\end{equation}
with the equality iff r = 1.
\end{thm}

\begin{thm}[Entropy bounds]
Let $s \in \mathcal A$. Then
\begin{equation}
	0 \leq H(s) \leq \log_b |\mathcal A|
\end{equation}
with the first inequality holding iff $S=\text{const.}$, and the second inequality holding iff $p_S(s)=\frac{1}{|\mathcal A|}$.
\end{thm}

\begin{exmp}
	Let $S$ be your 4-digit lock number. Then $S = \{0, 1, \ldots, 9999\}$. Let's say you choose your lock number at random: then $H(S) = \log 10^4$. However, if your grandma \ul{always} chooses $0000$, then $S$ is a constant, and $H(S) = 0$. A random (least predictable) choice has the most entropy possible ($\log_b |\mathcal A|$), and a constant (most predictable) choice has the least entropy (zero). This makes sense, and it also means that the random lock number carries the most information, and the constant one carries the least information.
\end{exmp}

\noindent Let's apply this to sources.

\subsection{Source coding}
The setup we have is the following: let's say we have a source emitting $S_i \in \mathcal A$ towards an encoder with an \emph{encoding map} (a function $\mathcal A \to \mathcal C$) called $\Gamma$. We're going to map each element of the alphabet into a codeword, for example, each letter of the word \texttt{dinner}. For example: d$\to 000$, i$\to 010$, \dots \\
The encoder is specified by
\begin{itemize}
	\item an input alphabet $\mathcal A$, which is the alphabet of the source
	\item an output alphabet $\mathcal C$
	\item the encoding map $\Gamma : \mathcal A \to \mathcal C$
\end{itemize}
The code is a set of codewords, which are the output of the $\Gamma$ map. The $\Gamma$ map is always one-to-one and onto (bijective), but this doesn't mean that we can necessarily go back from the code words to the words, since we usually concatenate the output.

\begin{exmp}
 Let $\mathcal A = \{\texttt H, \texttt E, \texttt L, \texttt O\}$ and $\mathcal C = \{01, 10, 0, 11\}$ ($\Gamma$ maps them in the written order). Then $\Gamma : \mathcal A \to \mathcal C$ encodes the word \texttt{HELLO} to the bitstring \texttt{01100011}. The conversion is easy in this direction, but when trying to decode the message, we run into a difficulty: the bitstring could either be interpreted as \texttt{01,10,0,0,11}, which would give back the correct message \texttt{HELLO}, or as \texttt{0,11,0,0,0,11}, which would give the incorrect message \texttt{LOLLLO}.
\end{exmp}

\begin{defn}
	We say that a code is \textbf{uniquely decodable} if each concatenation of codewords has a unique parsing into codewords, that is, if we can be sure of getting the correct message when decoding a sequence of codewords. The kinds of encodings that give uniquely decodable codes are the ones that are most interesting in information encoding.
\end{defn}

\begin{exmp}\label{exmp:first-codes}
Let's have a look at a few different $\Gamma$ mappings, and check whether they are uniquely decodable or not:
\begin{center}
	\begin{tabular}{l | l l l l}
		$\mathcal A$ & $\Gamma_O$ & $\Gamma_A$ & $\Gamma_B$ & $\Gamma_C$ \\ \hline
		\texttt a & 00 & 0 & 0 & 0 \\
		\texttt b & 01 & 01 & 10 & 01 \\
		\texttt c & 10 & 10 & 110 & 011 \\
		\texttt d & 11 & 11 & 1110 & 0111
	\end{tabular}
\end{center}
The code $\Gamma_O$ \textbf{is} uniquely decodable because of its constant codeword length: we will always group symbols two by two, which means that there is a single possible decoding. \par
 \noindent The code $\Gamma_A$ \textbf{is not} uniquely decodable: for example, if we have the sequence \texttt{0110}, we could either decode it as \texttt{01,10} which would mean ``\texttt{bc}'', or as \texttt{0,11,0} which would mean ``\texttt{ada}''. \par
 \noindent The code $\Gamma_B$ \textbf{is} uniquely decodable, since 0 acts as a delimiter between codewords (suffix). \par
 \noindent The code $\Gamma_C$ \textbf{is} uniquely decodable, since 0 acts as a delimiter between codewords (prefix).
\end{exmp}

 Example \ref{exmp:first-codes} showed us two cases of codes that have either a suffix or a prefix, and that are uniquely decodable. In fact, using prefixes and suffixes are a way to guarantee that a code is uniquely decodable; however, prefixes do come with a problem which we shall now discover.

\begin{defn}
	A code is said to be \textbf{prefix-free} if no codeword is the prefix of a longer codeword. Prefix-free codes are preferred in encoding information. They are also called \textbf{instantaneous codes}, since they allow instantaneous decoding.
\end{defn}

 Indeed, prefixes may seem like a good idea since they guarantee uniquely-decodable codes. The issue, however, with codes that are not prefix-free, is they are not ``instantaneously'' decodable: if you have only received part of the sequence of codewords, you cannot decode it, since there may be ambiguities. This is illustrated by the next example \ref{exmp:prefixes-are-not-so-great}.

\begin{exmp}\label{exmp:prefixes-are-not-so-great} Here's an example of why prefix codes are not the best in terms of decoding. The following code is uniquely decodable, but it uses a prefix:
\begin{center}
	\begin{tabular}{l | l}
		$\mathcal A$ & $\Gamma$ \\ \hline
		\texttt a & 0 \\
		\texttt b & 00001
	\end{tabular}
\end{center}
If the decoder receives the sequence \texttt{00}, it cannot instantaneously determine whether this is the start of a \texttt{b} or two concatenated \texttt{a}, and so it has to wait until it has the full string of bits to be able guarantee correct decoding. \par
 In real life, this sort of problem shows up, for example, when streaming video or audio from the Internet, where it may cause unwanted delays.
\end{exmp}

Theorem \ref{thm:kraft-mcmillan-1} comes in very handy when trying to determine whether a code is uniquely decodable or not.

\begin{thm}[Kraft-McMillan 1]\label{thm:kraft-mcmillan-1}
	If a $D$-ary code is uniquely decodable, then its codeword lengths $l_1, l_2, \ldots, l_M$ satisfy the following inequality:
\begin{equation}
	D^{-l_1} + \dots + D^{-l_M} \leq 1 \quad \text{(Kraft's inequality)}
\end{equation}
\begin{remark}
Kraft's sum is only about the lengths of the codewords!
\end{remark}
\end{thm}
Do be wary that this theorem is an ``if-then'' theorem, which means that the converse may not be true! Such a case is illustrated by example \ref{exmp:kraft-mcmillan-1-converse}.

\begin{exmp}\label{exmp:kraft-mcmillan-1-converse}
Let's look at the following code:
\begin{center}
	\begin{tabular}{l | r}
	$\mathcal A$ & $\mathcal C$ \\ \hline
	\texttt a & \texttt{01} \\
	\texttt b & \texttt{0101}
	\end{tabular}
\end{center}
	This is a 2-ary (binary) code with lengths 2 and 4, so Kraft's sum gives
	\begin{equation*}
		2^{-2} + 2^{-4} = \frac14 + \frac{1}{16} = \frac{5}{16} \leq 1,
	\end{equation*}
	and Kraft's inequality is satisfied, however the code is clearly \textbf{not} uniquely decodable.
\end{exmp}

\section{28th February 2019}

These following properties are what we aim for with any encoding map $\Gamma : \mathcal A \to \mathcal C$, in order to optimize transmission and decoding speed:
\begin{itemize}
	\item $\mathcal C$ be uniquely decodable
	\item $\mathcal C$ be prefix-free
	\item $\mathcal C$ have its average codeword length be as small as possible
\end{itemize}

\begin{thm}[Kraft-McMillan 2]
If $l_1, \ldots, l_m$ satisfy Kraft's inequality for some positive integer $D$, then there exists a $D$-ary prefix-free code that has codeword  lengths $l_1, \ldots, l_m$.
\end{thm}

This second part of the Kraft-McMillan theorem guarantees that \emph{any} uniquely decodable code can be substituted by a prefix-free code of the same codeword lengths.
\begin{summary}[Kraft-McMillan] The theorem is in 2 parts:
	\begin{enumerate}
		\item If a $D$-ary code is uniquely decodable, then its codeword lenghts $l_1, \ldots, l_M$ satisfy Kraft's inequality
		\begin{equation*}
		D^{-l_1} + \dots + D^{-l_M} \leq 1.
		\end{equation*}
		\item If the positive integers $l_1, \ldots, l_M$ satisfy Kraft's inequality for some integer $D$, then there exists a $D$-ary \ul{prefix-free} code that has those codeword lengths.
	\end{enumerate}
\end{summary}

\begin{defn}
We define the \textbf{average length} $L(S, \Gamma)$ as
	\begin{equation}
		L(S, \Gamma) = \sum_{S \in \mathcal A} p_S(s) \underbrace{l \bigl(\Gamma(s)\bigr)}_{{\mathclap{\text{shorthand } l(s)}}}.
	\end{equation}
	Sometimes we write
	\begin{equation}
		L(S, \Gamma) = \sum_i p_i l_i.
	\end{equation}
This is rather intuitively defined, as the expected length should according to common sense indeed depend on the length of each codeword, and on its probability of appearing in the code. \\
The units of the average length are \textbf{code symbols}. When $D = 2$, the units are \textbf{bits}.
\end{defn}

An interesting question we may now ask ourselves is the following: is there a lower bound to the average length for uniquely decodable codes?

\begin{thm}[lower bound on average length]
	Let $\Gamma$ be the encoding map of a $D$-ary code for the source $S$. If the $D$-ary code is uniquely decodable, then
	\begin{equation}
		H_D(S) \leq L(S, \Gamma).
	\end{equation}
	\textbf{The entropy is a lower bound to the average length}.
\end{thm}

\begin{remark}
	A key observation we may make is that the definition of the average length is somewhat similar to that of the entropy:
	\begin{align*}
		&L(S, \Gamma) = \sum_{s \in \mathcal A} p(s) l(\Gamma(s)) \\
		&H_D(S) = \sum_{s \in \mathcal A} p(s) \log_D \left(\frac{1}{p_S(s)}\right)
	\end{align*}
In fact, the definitions are identical iff $l(\Gamma(s)) = \log_D \left(\frac{1}{p_S(s)}\right)$. Unfortunately this equality is often not possible (the $\log$ is often not an integer). But what if we chose $l(\Gamma(s)) = \left\lceil \log_D \left(\frac{1}{p_S(s)}\right) \right\rceil$? Is it a valid choice for a prefix-free code (is Kraft's inequality satisfied)?
\end{remark}

\begin{defn}[Shannon-Fano code]
	A code $\mathcal C$ for which $l_i = \left\lceil -\log_D \bigl(P_S(s) \bigr) \right\rceil$ is called a \textbf{Shannon-Fano code}. Visually, it is constructed by going from the top down when creating the code tree.
\end{defn}
\begin{remark}
	The Shannon-Fano code is not always optimal, in the sense that its average codeword length is not always the best. This means that it is a lot less used than the Huffman code which we will see later, and which always gives optimal codes.
\end{remark}

\begin{defn}
	We say that a probability distribution is \textbf{diadic} iff
	\begin{equation}
		p_i = D^{-l_i}
	\end{equation}
\end{defn}

\begin{thm}
	It is possible to make the codewords lengths $l_i$ equal the entropy iff the code is diadic.
\end{thm}

CLEAR UP THE THING ABOUT THE $-\log_D(p_i)$ THAT I DONT UNDERSTAND

\begin{remark}
	Most probability distributions are not diadic. Then, $-\log_D(p_i)$ is not an integer, and we can't make the length equal that number. \par
	The only cases where a Shannon-Fano code is optimal is when the probability distribution is diadic (CHECK IF THIS IS CORRECT).
\end{remark}

\begin{defn}[Huffman code]
 Visually, the Huffman code on an alphabet is constructed by going from the bottom up when creating the code tree, and grouping together the least probable symbols.
\end{defn}

\begin{exmp} (COMPLETE WITH THE PROBABILITY DISTRIBUTION) The Huffman code on $\mathcal A = \{\texttt a, \texttt b, \texttt c, \texttt d\}$ is:
	\begin{center}
	\begin{tabular}{r | l}
		$\mathcal A$ & $\Gamma_H$ \\ \hline
		\texttt a & \texttt{000} \\
		\texttt b & \texttt{001} \\
		\texttt c & \texttt{01} \\
		\texttt d & \texttt{1}
	\end{tabular}
	\end{center}
 We can compute the expected length and the entropy (in bits):
	\begin{align*}
		&L(S, \Gamma_H) = 0.15 + 0. 15 + 0.2 + 0.8 = 1.3 \\
		&H_2(S) = \ldots = 1.022
	\end{align*}
\end{exmp}

\section{5th March 2019}

\begin{thm}
	The average length can also be computed by adding together the probabilities of all nodes on the code tree, except for the last leaves:
	\begin{equation}
		\underbrace{\sum_{\substack{i \in \\ \text{terminal} \\ \text{leaves}}}}_{L(S, \Gamma)} = \quad \sum_{\mathclap{\substack{j \in \\ \text{intermediate} \\ \text{nodes}}}} q_j.
	\end{equation}
\end{thm}

\begin{thm}[optimality of Huffman codes]
	Let $\Gamma_H$ be an encoder of a $D$-ary Huffman code for $S$, and let $\Gamma$ be another $D$-ary uniquely decodable encoder for $S$. Then
	\begin{equation}
		L(S, \Gamma_H) \leq L(S, \Gamma).
	\end{equation}
	Basically, the Huffman code on $S$ is the optimal code on $S$.
\end{thm}

\subsection{}

\begin{defn}[IID source]
	A source is said to be \textbf{IID} (Independant and Identically Distributed) iff all random variables are mutually independent and have the same probability distribution.
\end{defn}
Most sources are not IDD.

\begin{exmp}
	A sequence of coin flips is an IID source.
\end{exmp}

\begin{exmp}
	Let $S_1, S_2 \in \{1, 2, \ldots, 6\}$ represent dice throws. They are independant and uniformly distributed. Let $(L_1, L_2)$ be the first and second digit of the sum $S_1 + S_2$. We can compute $P_{L_1 | L_2} (1|1) = \frac{P_{L_1 | L_2}(1, 1)}{P_{L_1}(1)}$. We know that the event $(L_1, L_2) = (1, 1)$ is the same as the event $S_1 + S_2 = 11$, which is the same as saying $(S_1, S_2) \in \{(5, 6), (6, 5)\}$, which has probability $2/36$. Then the conditional probability that we wanted to compute is
	\begin{align*}
		\frac{2/36}{3/36 + 2/36 + 1/36} = \frac13
	\end{align*}
\end{exmp}

\begin{defn}[conditional entropy]
	Let $p_X$ be the probability distribution of a random variable $X$. We already know how to compute the entropy $H(x)$ from this probability distribution. Then $p_{X|Y=y}$, which is also a probability distribution, allows us to compute the \textbf{conditional entropy} $H(X|Y=y)$:
	\begin{equation}
		H_b(X|Y=y) = - \sum_{x \in X} p_{X|Y}(x|y) \log_b \bigl(p_{X|Y}(x|y)\bigr)
	\end{equation}
\end{defn}

\begin{exmp}[continuation]
COMPLETE THIS FROM THE NOTES :DDDDDDDDDDDDDDD I AM GOING INSANE AND 2 HOURS LEFT :DDDDDDDDDDD
\end{exmp}

\section{7th March 2019}

IS THE ENTROPY EQUAL TO THE AVERAGE LENGTH IFF IT IS A HUFFMAN CODE?

\begin{thm}
	Let $X$ and $Y$ be two random variables. Then
	\begin{equation}
		H(X|Y) \leq H(X),
	\end{equation}
	with the equality iff $X$ and $Y$ are independent.
\end{thm}

\begin{thm}[chain rule of entropy]
	Let $S_1, \ldots, S_n$ be random variables. Then
	\begin{equation}
		\boxed{H(S_1, \ldots, S_n) = \sum_{i=1}^n H(S_1 | S_1, \ldots, S_{i-1})}
	\end{equation}
\end{thm}

 For this to make a bit more sense, recall that $P_{X, Y}(x, y) = P_X(x)P_{Y|X}(y|x)$ (this follows directly from the definition of conditional probability). More generally,
 \begin{equation*}
 P_{S_1, \ldots, S_n}(s_1, \ldots, s_n) = \prod_{i=1}^n P_{S_i | S_1, \ldots, S_{i-1}}(s_i | s_1, \ldots, s_{i-1}).
 \end{equation*}
 The chain rule of entropy is very similar. \par
 This equality will help us in proving many theorems.

 \begin{exmp}[continuation]
 	$H(l_1) = 0.65$ bits, $H(l_2) = 3.2188$ bits, $H(l_1, l_2) = 3.744$ bits, $H(l_2 | l_1) = H(l_1, l_2) - H(l_1) = 2.624$ bits (as obtained before).
 \end{exmp}

 \begin{exmp}[continuation]
 	We saw that $(S_1, S_2)$ determine $(l_1, l_2)$. Then what is $H(l_1, l_2 | S_1, S_2)$? We can know that $H(l_1, l_2 | (S_1, S_2) = (s_1, s_2)) = 0$. This is because $(S_1, S_2)$ fully determine $(l_1, l_2)$ (they are NOT independent). As such, when we know that $(S_1, S_2) = (s_1, s_2)$, we know exactly and can \emph{predict} the only possible digits $(l_1, l_2)$ of the sum of $s_1$ and $s_2$, which means that there is no entropy here. \\
 	Suppose we know $H(l_1, l_2)$ and $H(S_1, S_2)$. Then we can compute $H(S_1, S_2 | l_1, l_2)$:
 	\begin{align*}
 		&H(S_1, S_2, l_1, l_2) \overbrace{=}^{\substack{\text{chain} \\ \text{rule}}} \begin{cases} H(S_1, S_2) + \cancel{H(l_1, l_2 | S_1, S_2)} \\
 													  H(l_1, l_2) + H(S_1, S_2 | l_1, l_2)
 													  \end{cases} \\
		&\implies H(S_1, S_2, | l_1, l_2) = H(S_1, S_2) - H(l_1, l_2), \quad \text{which we both know by supposition.}
 	\end{align*}
 \end{exmp}

 \begin{exmp}
 	Let's say we have a random variable $X$ which takes a value in $\{0, +1, -1, +2, \ldots, +13, -13\}$. Suppose $X$ is uniformly distributed. Any weighing strategy (not yet determined) is an encoding $\Gamma: \mathcal A \to \mathcal C$ ($|\mathcal C| = 3$, so this is a $3$-ary code). $\Gamma$ is a bijective function $x \leftrightarrow s_1 s_2\ldots s_L$. So $H_3(x) = H_3(S_1, S_2, \ldots S_L)$.
 \end{exmp}
NOTES: a negative number means "light" on the balance, a positive number means it is "heavy". At each step of the guessing process, he makes sure that the new balance is independent of previous knowledge, and that it is evenly distributed (even chances of going on each side). Actually the balls are billiard.
 One coould be heavier or lighter, ut not the 0 ball. You have a balance for balls. How many times to weigh to determine if one bakk us fakse and if yes, which one and settle if it is heavier or lighter.

\section{12th March 2019}
So far, we have assumed that we have a random variable over the alphabet $\mathcal A$, an encoding function $\Gamma$ which is a map $\mathcal A \to \mathcal C$, from the alphabet to the code (which was assumed to be uniquely decodable), and a source outputting a sequence of $n$ random variables. We saw that if this source is IID, then the total length divided by $n$ will tend to $L(S, \Gamma)$ as $n \to \infty$. We saw that the entropy is a lower bound to the average length. \par
We will now drop the assumption that the source is IID: this is very common, for example, when you try to compress voice, or video, or any kind of "natural" information.

\begin{exmp}
	We reuse the example of the coin flip source: $S_i \in \{H, T\}$. $S_1, S_2, \ldots, S_n$ are IID. Therefore
	\begin{equation*}
	p_{S_1, S_2, \ldots, S_n}(s_1, s_2, \ldots, s_n) = \prod_{i=1}^n p_{S_i}(s_i) = \bigl(\frac12\bigr)^n
	\end{equation*}
\end{exmp}

\begin{defn}
	The source $\mathcal S = S_1 S_2 \ldots S_n$ is said to be \textbf{regular} iff
	\begin{enumerate}
		\item $H(\mathcal S) = \lim_{i \to \infty} H(S_i)$ (the entropy per symbol) exists, and
		\item $H^*(\mathcal S) = \lim_{i \to \infty} H(S_i | S_1, \ldots, S_{i-1})$ (the entropy rate) exists.
	\end{enumerate}
\end{defn}

\begin{exmp}[cont.]
	The coin flip source is regular:
	\begin{equation*}
		H(\mathcal S) = 1 = H^*(\mathcal S)
	\end{equation*}
\end{exmp}

\label{exmp:sunnyrainymarkov}\begin{exmp}[Sunny-Rainy source]
	$S_i \in \{S, R\}$ represents the weather on day $i$. $S_i$ is uniformly distributed. If the weather one way one day, it will stay the same tomorrow with probability $q$, and change with probability $1-q$.
	\begin{equation*}
		p_{S_1, \ldots, S_n} (s_1, \ldots, s_n) =  p_{S_1}(s_1) \cdot \prod_{i = 2}^n p_{S_i | S_{i-1}}(s_i | s_{i-1}).
	\end{equation*}
	For example, $p_{S_1, S_2, S_3, S_4} (RRSR) = \frac12 \cdot q \cdot (1-q) \cdot (1-q)$. More generally, $p_{S_1, \ldots, S_n}(s_1, \ldots, s_n) = \frac12 \cdot (1-q)^c \cdot q^{n-1-c}$, where $c$ is the number of transitions. \par
	We may also check whether the source is regular or not. $H(S_i) = H(S_1) = 1$, therefore the entropy per symbol exists and equals $1$. Let us also compute $H(S_i | S_{i-1})$. We know that $H(S_i | S_{i-1} = R) = h(q)$ and $H(S_i | S_{i-1} = S) = h(1-q) = h(q)$ (where $h$ is the binary entropy function, that is, the entropy function adapted to binary bernouilli trials, which is intuitively symmetrical with regards to $1/2$). Finally $H(S_i | S_{i-1}) = h(q) = H(S_1 | S_{i-1}, \ldots, S_1)$ (the outcomes before the previous day do not have any effect). Therefore the entropy rate exists, and is equal to $h(q)$.
\end{exmp}

\begin{defn}
	A Markov chain is the simplest kind of source which has some memory: each random variable depends only on the state of the one immediately preceding it.
\end{defn}
\begin{remark}
	The source in example \ref{exmp:sunnyrainymarkov} is a Markov chain.
\end{remark}

\begin{defn}
	A source $S_1,  S_2 \ldots, S_i \in \mathcal A$, is \textbf{stationary} if for every $n, k$ positive integers, the statistic of $(S_1, \ldots, S_n)$ is the same as the statistic of $(S_{k+1}, \ldots, S_{k+n})$. Formally,
	\begin{equation}
		p_{S_1, \ldots, S_n}(s_1, \ldots, s_n) = P_{S_{k+1}, \ldots, S_{k+n}}(s_1, \ldots, s_n).
	\end{equation}
\end{defn}

\begin{thm}
	A stationary source is always regular.
\end{thm}

\begin{thm}
	For a stationary source $\mathcal S$,
	\begin{equation}
		H^*(\mathcal S) \leq H(\mathcal S),
	\end{equation}
	with equality iff the symbols are independent.
\end{thm}

\section{14th March 2019}
	What if, instead of encoding codewords one at a time, we encoded concatenations of codewords?

\begin{equation}
	\boxed{H_D(S) \leq L(S, \Gamma_H) \leq L(S, \Gamma_{SF}) < H_D(S) + 1}
\end{equation}
This inequation is not directly related but muchho importanto in source coding. We can tho adapt it to block-encoding (idk how it's called):
\begin{equation}
	H_D(S_1 \ldots S_n) \leq L(\mathcal S, \Gamma_H) \leq L(\mathcal S, \Gamma_{SF}) < H_D(S_1 \ldots S_n) + 1
\end{equation}
If we divide everything by $n$ to get the average codeword length per symbol:
\begin{equation}
	\frac{H_D(S1 \ldots S_n)}{n} \leq \frac{L(\mathcal S, \Gamma_H)}{n} < \frac{H_D(S_1 \ldots S_n)}{n} + \frac1n
\end{equation}
The $\frac1n$ goes to 0 as $n$ grows large. \par
Our goals: study the behavior of $\frac{H_D(S_1 \ldots S_n)}{n}$ as $n$ grows large and try to relate it to $H_D^*(\mathcal S)$ (entropy rate).

\begin{exmp}
	Consider a monkey source $S$ that randomly picks one letter at a time from a French book. $H(S) = 3.95$ bit2	s, so a Huffman code $\Gamma_H$ approaches $L(S, \Gamma_H) \approx 4$ bits/letter when encoding French monkey text. \par
	However, a Lempel-Ziv code (used by various compression programs) approaches 1 bit per letter when compressing French text. This is due to the fact that letters in a French text are not completely random, therefore the entropy goes down by quite a bit, and beat the Huffman code. \par
	This means that as $n$ grows large, $\frac{H_D(S_1 \ldots S_n)}{n} \to 1$. This, rather than $H(S)$, is the important quantity for us. \par
	If $S_1 \ldots S_n$ were IID, then
	\begin{equation*}
		\frac{H_D(S_1 \ldots S_n)}{n} = \frac{H_D(S_1) \cdot H_D(S_2) \ldots H_D(S_n)}{n} = H(S_1) \quad \forall 1 \leq i \leq n
	\end{equation*}
	Imagine you start with the text from a book, and take the alphabet of that book (all symbols that appear in it). You put these in a table and assign an integer to each (the position in the table), in binary. You then start looking for groups of 2 letters, and if you find some, you add them to the dictionary, so $n$ increases (it has a limit). Same thing for 3 symbols, or groups that appear often.
\end{exmp}

\begin{thm}[Cesàro mean]
	Consider a source of real-valued numbers $a_1 \ldots a_n$. If $\lim_{n \to \infty} a_n = A$, then if $c_n = \frac{a_1 + a_n}{n}$, we have that $\lim_{n \to \infty} c_n = A$.
\end{thm}

\begin{thm} Let $S$ be a source. Then
	\begin{enumerate}
		\item if $S$ is stationary, then $S$ is regular,
		\item $\frac{H_D(S_1 \ldots S_n)}{n}$ is non-increasing in $n$,
		\item $\lim_{n \to \infty} \frac{H_D(S_1 \ldots S_n)}{n} = H_D^*(S)$.
	\end{enumerate}
\end{thm}

\begin{summary}
	Let $S$ be a stationary source outputting an infinite sequence of symbols $S_1 \ldots S_n$. By encoding blocks of $n$ keywords at a time using a $D$-ary code, the average codeword length per symbol approaches $H^*(S)$ as $n$ grows large. No uniquely decodable $D$-ary code can do better than $H^*(S)$.
\end{summary}

\begin{exmp}[The 20 Questions Game]
	Was a very popular game on UK and US TV. You ask questions where the answers are yes or no. Equivalent to a binary search (akinator). \par
	Let $X$ be a random variable. Question: how many YES/NO questions do we need to find the realization of $X$, how should we ask the questions? \par
	The idea is to build a binary code $\Gamma$ for $X$. Once $\Gamma$ is fixed, identify the realization of $X (= \bar X)$, which is equivalent to finding $\Gamma(\bar X)$. The $i$-th question should reveal the $i$-th bit of $\Gamma(\bar X)$. The average number of questions should be the average codeword length of $\Gamma$. If $\Gamma$ is $\Gamma_H$, we cannot do better in terms of the average number of questions. \par
	Let us consider a random variable $X$ with $\mathcal A = \{a, b, c, d, e\}$, $p(a) = 0.1, p(b) = 0.1, p(c) = 0.2, p(d) = 0.2, p(e) = 0.4$. After its construction, the Huffman code is $a = 000, b = 001, c = 010, d = 011, e = 1$. Let's ask some questions from the top of the tree:
	\begin{enumerate}
		\item is $X = e$ ? NO $\Rightarrow$ the first bit of $X$ is 0.
		\item is $X \in \{c, d\}$ ? NO $\Rightarrow$ the second bit of $X$ is 0.
		\item is $X = b$ ? YES $\Rightarrow$ the third bit of $X$ is 1.
	\end{enumerate}
	Therefore, $X = 001 = b$. \par
	Let us now discuss the optimality of this strategy. We have seen that a binary code (prefix-free) implies a question strategy, and that $\Gamma_H$ gave the best strategy, since the average number of questions is equal to the average codeword length of $\Gamma$. \par
	Question: does a questioning strategy (YES/NO) as above always lead to a prefix-free code? \par
	We start with the root: let $X \in \mathcal X$ be a random variable. Split $\mathcal X$ into $\mathcal A$ and $\mathcal A^c$ (complement) and suppose that $X \in \mathcal A$. When we ask the question, we will know in which of the two it is. Suppose that the answer is yes: then $X \in \mathcal A$. Continue by splitting $\mathcal A$ into $\mathcal B$ and $\mathcal B^c$ and repeat the process. By construction, we obtain a prefix-free code.
\end{exmp}

ENCODING OF INTEGERS: we saw that the Lempel-Ziv code needed to encode integers. Let's think about some binary prefix-free codes for positive integers. \par
The first one is the "natural" way, that is, encoding each number into its binary representation. However, this is not good, because it is not prefix-free, and has $l(n) = \lfloor \log(n) \rfloor + 1$. \par
 The next try (elias code 1) would be adding $l(n) - 1$ zeros in front of each codeword $n$. This is prefix-free, but it is pretty long: $l(n) = 2 \lfloor \log(n) \rfloor + 1$. \par
 The next try (elias code 2) would be replacing the leading zeros and the following one by $c_1(l(n)$ ($c_1 =$ elias code 1). Then we get a ITS PI DAY PEOPLE \par
3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117\\ 0679821480132823066470

\begin{summary}[Source Coding]
\begin{itemize}
	\item If a $D$-ary code is UD for source $S$,
	\begin{equation*}
		H_D(S) \leq L(S, \Gamma) < H_D(S) + 1
	\end{equation*}
	The first inequality comes from Kraft's sum and the IT inequality. \\
	This also applies to $\bar S = S_1S_2\ldots S_n$:
	\begin{align*}
		&H_D(\bar S) \leq L(\bar S, \Gamma) < H_D(\bar S) + 1 \\
		&\iff \\
		&\frac{H_D(\bar S)}{n} \leq \frac{L(\bar S, \Gamma)}{n} < \frac{H_D(\bar S)}{n} + \frac1n
	\end{align*}
	(we divided by $n$ to get the entropy per symbol). The second and last terms (without $\frac1n$, which tends to 0 as $n \to \infty$) tend to $H^*(\bar S)$ if the source is stationary.
	\item For encoding integers, we saw the Elias code (UD), which encodes $n \in \mathbb N$ with $\approx \log_2 n + 1$ bits.
	\item Among other things, we have left out what's called "universal source coding" (example in the notes), for example Lempel-Ziv and Elias-Willens.
\end{itemize}
\end{summary}

\chapter{Cryptography}
\section{19th March 2019}
The two main goals of Cryptography are \textbf{authenticity} and \textbf{privacy}. Authenticity means we should be assured that we are indeed sharing information with the correct person/computer. Privacy means that no other party should be able to see the shared information. \\
 Until just about when the internet was invented, criptography was basically only used by generals and diplomats. The advent of public Internet made cryptography into a branch/tool that was used universally and constantly by every person and/or machine on the Internet.

\subsection{Setup for privacy}
The idea is the following: We have two "private spaces", one for Alice and one for Bob. Alice wants to send some plain text, call it $t$, to Bob. It goes through an encryption device, which requires a key $k_A$, and which is located in Alice's "private space". The encrypted message, $c$, also called cryptogram or ciphertext. is then sent to Bob's private space, but has to go through an unprotected zone called the public channel. \\
 The public channel is being eavesdropped by Eve. We cannot block Eve from accessing $c$, therefore our only way of guaranteing privacy is by making $c$ undecryptable by Eve, or rather, only decryptable by Bob.

\section{Rudiments of Number Theory}
The RSA encryption system relies on number theory. We want to work with a finite set of numbers, since this will make everything much easier on a computer. \par

Actually, on second thought, fuck cryptography. I'll meet you again when we're doing channel coding.

\end{document}


    \filetitle{datrange}{Numerically safe way to create a date range}{dates/datrange}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
Rng = datrange(Start,End)
Rng = datrange(Start,End,Step)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{Start} {[} numeric {]} - Start date of the range.
\item
  \texttt{End} {[} numeric {]} - End date of the range.
\item
  \texttt{Step} {[} numeric {]} - Step size in the number of base
  periods; if omitted, \texttt{Step = 1}.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{Rng} {[} numeric {]} - Date vector
  \texttt{Start : Step : End}.
\end{itemize}

\paragraph{Description}\label{description}

Most of the time, using a colon operator to create a date range works
fine,

\begin{verbatim}
Start : Step : End
\end{verbatim}

Under some rare circumstances, the colon operator may bump into round
error difficulties as IRIS serial date numbers are non-integer values.
In that case, the function \texttt{datrange} provides a safe workaround:

\begin{verbatim}
datrange(Start,End,Step)
\end{verbatim}

is equivalent (but numerically safer) to

\begin{verbatim}
Start : Step : End
\end{verbatim}

\paragraph{Example}\label{example}

The date ranges created in this example are identical, and no numerical
inaccuracies exist:

\begin{verbatim}
r1 = qq(2000,1) : qq(2010,4);
r2 = datrange(qq(2000,1),qq(2010,4));
format long
r1 - r2
\end{verbatim}