%\chapter{An Introduction of Machine Learning}
\section{What is a learning algorithm?}

        A \textbf{machine learning algorithm} is an algorithm that is able to learn from data. But what does "learning" mean? Mitchell (1997) provides a definition "A computer program is said to learn from experience $E$ with respect to some class of tasks	$T$	and performance measure	$P$, if its performance at tasks in	$T$, as measured by $P$, improves with experience $E$." Learning is not the task, but the means of attaining the ability to perform the task. The result of the learning is a model that is used to perform tasks.

        The machine learning algorithm grew out of work in \textbf{Artificial Intelligence (AI)} and take advantage of new capability of computing resources. \textbf{Deep learning} is a subset of machine learning.

    \subsection{Types of learning algorithm}

         Machine learning algorithms can be broadly categorized as \textbf{unsupervised} or \textbf{supervised} or \textbf{Reinforcement learning algorithms
         }	by what kind of experience they are allowed to have during the learning process.
    \begin{itemize}
        \item Unsupervised learning algorithms learn from a dataset containing many features, then learn useful properties of the structure of this dataset. E.g. K-means clustering
        \item Supervised learning algorithms learn from a dataset containing features, but each example is also associated with a label or target. E.g. classification , regression method to predict housing price
        \item Reinforcement learning algorithms interact with an environment, so there is feedback loop between the learning system and its experience. But it's out of the scope of this note.
    \end{itemize}
    Here we offer a simple example to illustrate the difference between the supervised and unsupervised learning algorithm.

     Suppose we have a set of points in $\mathbb{R}^2: \{(x_1,y_1),(x_2,y_2),\dots,(x_n,y_n)\}$.
     \begin{itemize}
         \item Ther's no more information about the points except their coordinates. We want to split them into several subsets (which is 'clustering') based on their coordinates. That's an unsupervised algorithm.
         \item The points are painted red,green or blue. We want to learn from the data about the reason why the point is red,green or blue. If given a new point, we can label its color according the points before. That's an example of classification, which is one kind of supervised learning algorithm.
     \end{itemize}
     \begin{figure}[htbp]
         \centering
         \includegraphics[width = 1.0 \textwidth]{Cluster.png}
         \caption{An example of clustering}
     \end{figure}
    \subsection{Maximum Likelihood Estimation}
    As the process of machine learning is to estimate a set of parameters �� based on observations of examples X, \textbf{maximum likelihood} is often considered the preferred estimator to use for machine learning.

    \begin{itemize}
        \item Consider a set of $m$ examples $\mathbb X=\{\bm x^{(1)},...,\bm x^{(m)}\}$ drawn independently from the true but unknown data generating distribution $p_{data}(\bm x)$.
        \item Let $p_{model}(\bm x;\bm \theta)$ be a parametric family of probability distributions over the same space indexed by $\bm \theta$.
        \item The maximum likelihood estimator for $\bm \theta$ is then defined as
            \begin{equation*}
                \begin{split}
                    \bm \theta_{ML} &= \arg \max_{\bm \theta}p_{model}(\mathbb X;\bm \theta) \\
                    &=\arg \max_{\bm \theta}\prod_{\bm x \in \mathbb X} p_{model}(\bm x;\bm \theta)\\
                    &=\arg \max_{\bm \theta}\sum_{\bm x \in \mathbb X} \log p_{model}(\bm x;\bm \theta)\\
                \end{split}
            \end{equation*}
    \end{itemize}

    \subsection{Input data}
    Previously observed data in the experience can be split to three categories:
    \begin{itemize}
        \item Training data: used to train the model
        \item Validation data: used to adjust the model
        \item Test data: used to evaluate the generalization of the model
    \end{itemize}
    Training data and validation data are usually splitted from a large observed data set. The ratio between training data and validation data is commenly 4:1 or 9:1.
    \subsection{Capacity and Performance}

    The capacity of the learning algorithms can be under fitting, appropriate capacity and overfitting. Underfitting occurs when the model is not able to obtain a sufficiently low error value on the training set. Overfitting occurs when the gap between the training error and test error is too large.

    \begin{figure}[htbp]
        \centering
        \includegraphics[width = 1.0 \textwidth]{fitCap.png}
        \caption{We fit three models to this example training set. The training data was
        generated synthetically, by randomly sampling $x$ values and choosing $y$ deterministically
        by evaluating a quadratic function. (Left)A linear function fit to the data suffers from underfitting-it cannot capture the curvature that is present in the data. (Center) A quadratic function fit to the data generalizes well to unseen points. It does not suffer from a significant amount of overfitting or underfitting. (Right)A polynomial of degree 9 fit to the data suffers from overfitting. Here we used the Moore-Penrose pseudoinverse to solve the underdetermined normal equations. The solution passes through all of the training points exactly, but we have not been lucky enough for it to extract the correct structure. It now has a deep valley in between two training points that does not appear in the true underlying function. It also increases sharply on the left side of the data, while the true function decreases in this area.}
    \end{figure}
    The factors determining the performance of the machine learning algorithms are:
    \begin{itemize}
        \item Test error (final error of the generated model) The error is calculated by a cost function based on the test data set
        \item The gap between training and validation error
    \end{itemize}
    The central challenge in machine learning is that we must perform well on new, previously unseen inputs, not just those on which our model was trained. So Machine learning algorithms indirectly minimize the test error. Regularization is the modification of machine learning algorithms to reduce validation error but not the training error in order to avoid overfitting.
    \begin{figure}[htbp]
        \centering
        \includegraphics[width = 1.0 \textwidth]{ErrorwithCap.png}
        \caption{Typical relationship between capacity and error. Training and test error behave differently. At the left end of the graph, training error and generalization error are both high. This is the underfitting regime. As we increase capacity, training error decreases, but the gap between training and generalization error increases. Eventually, the size of this gap outweighs the decrease in training error, and we enter the overfitting regime, where capacity is too large, above the optimal capacity.}
    \end{figure}
    \section{Regularization and generalization}
    \subsection{Regularization}
    Regularization is a process of introducing additional information in order to prevent \textbf{overfitting}. Empirical learning of models (learning from a finite data set) is always an underdetermined problem, because in general we are trying to infer a function of any $x$ given only some examples $ x_{1},x_{2},...x_{n}$.

    A \textbf{regularization term} (or \textbf{regularizer}) $R(f)$ is added to a loss function:

    \begin{equation}
        \min _{f}\sum _{i=1}^{n}L(f({\hat {x}}_{i}),{\hat {y}}_{i})+\lambda R(f)
    \end{equation}

    where $L$ is an underlying loss function that describes the cost of predicting $ f(x)$ when the label is $y$ (for classification problem); and $\lambda$  is a parameter which controls the importance of the regularization term. $ R(f)$ is typically chosen to impose a penalty on the complexity of $f$. Concrete notions of complexity used include restrictions for smoothness and bounds on the vector space norm.

    \subsection{Generalization error}
    In supervised learning applications in machine learning and statistical learning theory, \textbf{generalization error} (also known as the \textbf{out-of-sample error}) is a measure of how accurately an algorithm is able to predict outcome values for previously unseen data. Because learning algorithms are evaluated on finite samples, the evaluation of a learning algorithm may be sensitive to sampling error. As a result, measurements of prediction error on the current data may not provide much information about predictive ability on new data. Generalization error can be minimized by \textbf{avoiding overfitting} in the learning algorithm.

    The generalization error, $I[f_{n}]$ of a particular function $f_{n}$ over all possible values of $x$ and $y$ is:

    \begin{equation}
        I[f_{n}]=\int _{X\times Y}V(f_{n}(x),y)\rho (x,y)dxdy
    \end{equation}
    Since $I[f_{n}]$ cannot be computed for an unknown probability distribution, the generalization error cannot be computed explicitly. Instead, the aim of many problems in statistical learning theory is to bound or characterize the generalization error in probability:
    \begin{equation}
        P_{G}=P(I[f_{n}]-I_{S}[f_{n}]\leq \epsilon )\geq 1-\delta _{n}
    \end{equation}
    while
    \begin{equation}
        I_{S}[f_{n}]={\frac {1}{n}}\sum _{i=1}^{n}V(f_{n}(x_{i}),y_{i})
    \end{equation}
    is the empirical error computed from the sample points.

    \section{Types of problems}
    \subsection{Classification problem}
    In machine learning and statistics, classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known. An example would be assigning a given email into "spam" or "non-spam" classes or assigning a diagnosis to a given patient as described by observed characteristics of the patient (gender, blood pressure, presence or absence of certain symptoms, etc.).
     \begin{figure}[htbp]
         \centering
         \includegraphics[width = 1.0 \textwidth]{Classification.png}
         \caption{An example of classification}
     \end{figure}
    \subsection{Regression problem}
    In statistical modeling, regression is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables (or 'predictors'). More specifically, regression analysis helps one understand how the typical value of the dependent variable (or 'criterion variable') changes when any one of the independent variables is varied, while the other independent variables are held fixed.  In all cases, the estimation target is a function of the independent variables called the \textbf{regression function}.
    \begin{figure}[htbp]
        \centering
        \includegraphics[width = 1.0 \textwidth]{Regression.png}
        \caption{An example of multivariate regression}
    \end{figure}

\input{6DL/PageRank.tex}


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}

\title{MATH 542 Homework 4}
\author{Saket Choudhary\\skchoudh@usc.edu}

\begin{document}
\maketitle
\section*{1a. Problem 1}

\begin{align*}
E[(X-a)(X-a)']&= E[(X-a)(X'-a')]\\
&= E[XX'-Xa'-aX'+aa']\\
&= E[XX']-E[Xa']-E[aX']+E[aa']\\
&= E[XX']-E[X]a'-aE[X']+E[aa']\\
&= (Var[X]+E[X]E[X]')-E[X]a'-aE[X']+aa'\\
&= Var[X]+ E[X]E[X]'-E[X]a'-aE[X]'+aa'\ \text{ since } E[X']=E[X]'\\
&= Var[X] + (E[X]-a)(E[X]'-a')\\
&= Var[X] + (E[X]-a)(E[X]-a)'
\end{align*}

$$Var[X] = \sum = (\sigma_{ij})$$

$$||X-a||_{1\times 1}^2 = (X-a)'_{1 \times r}(X-a)_{1 \times r}$$

And hence(replace $X$ with $X'$ and $a$ with $a'$):

\begin{align*}
E[||X-a||^2] &= E[(X-a)'(X-a)]\\
&= E[X'X]-E[X']a-a'E[X]+a'a\\
&= \sum_i E[X_i^2]-E[X']a-a'E[X]+a'a\\
&= \sum_i (Var[X_i]+E[X_i]^2)-E[X']a-a'E[X]+a'a\\
&= \sum_i Var[X_i]+ E[X']E[X]-E[X']a-a'E[X]+a'a\ \text{since } \sum_i E[X_i]^2 = E[X'X]\\
&=\sum_i Var[X_i]+ E[X]'E[X]-E[X]'a-a'E[X]+a'a\ \text{since } \sum_i E[X_i]^2 = E[X'X]\\
&= \sum_i Var[X_i] + (E[X]-a)'(E[X]-a)\\
&= \sum_i \sigma_i + ||E[X]-a||^2
\end{align*}

\section*{1a. Problem 2}
Fact: $X-a-E[X-a]=X-E[X]$
\begin{align*}
Cov[X-a,Y-b] &= E[(X-a-E[X-a])(Y-b-E[Y-b])'])\\
&= E[(X-E[X])(Y-E[Y])']\\
&= Cov[X,Y]
\end{align*}

\section*{1a. Problem 3}
$Y_i = X_i-X_{i-1}$

$Cov[Y_i,Y_j]=0$ for $i \neq j$

Consider the vector $(Y_1, Y_2, Y_3, \dots, Y_n)' = (X_1,X_2-X_1,X_3-X_2, \dots, X_n-X_{n-1})'$

We make use of $Var(AX) = AVar(X)A'$.

To find $A$, consider the vectors $(Y_1, Y_2, Y_3, \dots, Y_n)' = (X_1,X_2-X_1,X_3-X_2, \dots, X_n-X_{n-1})'$

\begin{align*}
\begin{pmatrix}
Y_1\\
Y_2\\
Y_3\\
\vdots\\
Y_n
\end{pmatrix} &= \begin{pmatrix}
X_1\\
X_2-X_1\\
X_3-X_2\\
\vdots\\
X_n-X_{n-1}
\end{pmatrix}\\
&= \begin{pmatrix}1 & 0 & 0 & \dots & 0 & 0\\
-1 & 1 & 0 & \dots & 0 & 0\\
0 & -1 & 1 & \dots & 0 & 0\\
\vdots & \vdots & \vdots & \vdots & \vdots & \vdots\\
0 & 0 & 0 & \dots & -1 & 1
\end{pmatrix} \times \begin{pmatrix}
X_1\\
X_2\\
X_3\\
\vdots\\
X_n
\end{pmatrix}
\end{align*}

Hence $A=\begin{pmatrix}1 & 0 & 0 & \dots & 0 & 0\\
-1 & 1 & 0 & \dots & 0 & 0\\
0 & -1 & 1 & \dots & 0 & 0\\
\vdots \\
0 & 0 & 0 & \dots & -1 & 1
\end{pmatrix}$

Now using $Var(Y) = AVar(X)A'$ we get

$$Var(X) = A^{-1}Var(Y)A'^{-1}$$

$Var(Y) = I_{n\times n}$
and hence $Var(X) = A^{-1}A'^{-1} = BB^T$ where $B=A^{-1}$

\newpage
\section*{Problem 4}

$X_{i+1} = \rho X_i$

Consider:

\begin{align*}
\begin{pmatrix}
X_1\\
X_2\\
X_3\\
\vdots\\
X_n
\end{pmatrix} &= \begin{pmatrix}
X_1\\
\rho X_1\\
\rho X_2\\
\vdots\\
\rho X_{n-1}
\end{pmatrix}\\
&= \begin{pmatrix}
1\\
\rho\\
\rho^2\\
\vdots\\
\rho^{n-1}
\end{pmatrix}X_1
\end{align*}

Let $A=\begin{pmatrix}1 & \rho & \rho^2 & \dots & \rho^{n-1}\end{pmatrix}'$
and hence variance $Var[X] = AVar(X_1)A' = \sigma^2AA'$

$Var[X] = \begin{pmatrix}
1 & \rho & \rho^2 & \dots & \rho^{n-1}\\
\rho & \rho^2 & \rho^3 & \dots & \rho^n\\
\rho^2 & \rho^3 & \rho^4 & \dots & \rho^{n+1}\\
\vdots & \vdots & \vdots & & \vdots\\
\rho^n & \rho^{n+1} & \rho^{n+2} & \dots & \rho^{2n-2}\\
\end{pmatrix}$

\section*{1b. Problem 1}

\begin{align*}
X_1^2+2X_1X_2-4X_2X_3+X_3^2 &= (X_1+X_2)X_1 + (X_1-2X_3)X_2 + (-2X_2+X_3)X_3\\
&= \begin{pmatrix}
X_1 & X_2 & X_2
\end{pmatrix}\begin{pmatrix}
1 & 1 & 0\\
1 & 0 & -2\\
0 & -2 & 1\\
\end{pmatrix}\begin{pmatrix}
X_1\\
X_2\\
X_3
\end{pmatrix}
\end{align*}

$$X=\begin{pmatrix}X_1 & X_2 & X_3\end{pmatrix}'$$

$$A = \begin{pmatrix}
1 & 1 & 0\\
1 & 0 & -2\\
0 & -2 & 1\\
\end{pmatrix}$$

$$A\sum = AVar[X] = \sigma^2\begin{pmatrix}
1 & 1 & \frac{1}{4}\\
1 & -\frac{1}{2} & -2\\
0 & -\frac{7}{4} & \frac{1}{2}\\
\end{pmatrix}$$

Thus, $$E[X'AX] = tr(A\sum) + \mu'A\mu = \sigma^2 + \mu'A\mu$$

\section*{1b. Problem 2}

\begin{align*}
\sum_i(X_i-\bar{X})^2 &= \sum_i (X_i^2-2X_i\bar{X}+\bar{X}^2)\\
&= \sum_i X_i^2 -2\sum_i X_i \bar{X} + \bar{X}^2\\
&= \sum_i X_i^2 -n\bar{X}^2
\end{align*}

Now $\sum_iX_i^2 = X'X$ and $\bar{X} = \frac{1}{n}\sum_i X_i = \frac{1}{n} \mathbf{1'}X = \frac{1}{n}X'\mathbf{1} $

Hence,

\begin{align*}
\sum_i(X_i-\bar{X})^2 &= \sum_i X_i^2-n\bar{X}^2\\
&= X'X - n\frac{1}{n^2}(X'\mathbf{11'}X)\\
&= X'X-\frac{1}{n}(X'\mathbf{11'}X)\\
&= X'(\mathbf{I}-\frac{1}{n}\mathbf{11'})X
\end{align*}

Let $A=(\mathbf{I}-\frac{1}{n}\mathbf{11'})$

Now using $E[X'AX]= tr(A\sum) + \mu'A\mu$ we have:

\begin{align*}A\sum &= \begin{pmatrix}1-1/n & -1/n & -1/n & \dots & -1/n\\
-1/n & 1-1/n & -1/n & \dots & -1/n\\
\vdots\\
-1/n & -1/n & -1/n & \dots & 1-1/n
\end{pmatrix}\times diag(\sigma_1^2, \sigma_2^2, \dots, \sigma_n^2)\\
&= (1-\frac{1}{n})\sum_i \sigma_i^2
\end{align*}

Also, $$\mu'A = \begin{pmatrix} \mu & \mu & \dots & \mu \end{pmatrix} \times \begin{pmatrix}1-1/n & -1/n & -1/n & \dots & -1/n\\
-1/n & 1-1/n & -1/n & \dots & -1/n\\
\vdots\\
-1/n & -1/n & -1/n & \dots & 1-1/n
\end{pmatrix} = 0$$

and hence $\mu'A\mu=0$

\begin{align*}
E[\sum_i(X_i-\bar{X})^2] &= (1-\frac{1}{n})\sum_i \sigma_i^2  \\
E[\frac{1}{n(n-1)}\sum_i(X_i-\bar{X})^2]  &= \frac{1}{n^2} \sum_i \sigma_i^2
\end{align*}

Finally,

\begin{align*}
var(\bar{X}) &= Var(\frac{1}{n}\sum_i X_i)\\
&= \frac{1}{n^2} \sum_i Var(X_i)\\ \text{ since $X_i$ are mutually independent}\\
&= \frac{1}{n^2}\sum_i \sigma_i^2\\
&= E[\frac{1}{n(n-1)}\sum_i(X_i-\bar{X})^2]
\end{align*}

\section*{1b. Problem 3}

Given: $\bar{X_w} = \sum_i w_i X_i $ and $\sum w_i =1 $

\begin{align*}
Var(\bar{X_w}) &= Var(\sum w_i X_i)\\
&= \sum w_i^2Var(X_i)\ \text{ since $X_i$ are mutually independent}\\
&= \sum w_i^2 \sigma_i^2
\end{align*}

Now consider,

$$
\text{minimize} \sum w_i^2\sigma_i^2 \text{ subject to } \sum_i w_i =1
$$

We consider the following lagrange formulation:

$$min_w f(\mathbf{w}) = \sum_i w_i^2\sigma_i^2 +\lambda(\sum_iw_i-1)$$

Now to find optimal $\lambda$, we solve $\frac{\partial f(\mathbf{w})}{\partial w_i} = 0$

\begin{align*}
\frac{\partial f(\mathbf)}{\partial w_i} &= 2w_i\sigma_i^2+\lambda =0\\
\implies w_i &= -\frac{\lambda}{2\sigma_i^2}
\end{align*}

Thus, $w_i = -\frac{\lambda}{2\sigma_i^2}$ or $w_i \propto \frac{1}{\sigma_i^2}$

Using $\sum_i w_i=1$  we get:
\begin{eqnarray*}
\sum_i w_i = 1\\
\sum_i \frac{\lambda}{-2\sigma_i^2} =1\\
\implies \lambda = \frac{-2}{\sum_i 1/\sigma_i^2 }\\
\implies w_i = \frac{1}{\sigma_i^2 \sum_i(1/\sigma_i^2)}\\
\implies f_{min}(w) = \sum_i (\frac{1}{\sigma_i^2 \sum_i(1/\sigma_i^2)}) \sigma_i^2\\
v_{min} = \frac{1}{\sum_i(1/\sigma_i^2)}
\end{eqnarray*}

\subsection*{Part b}

\begin{align*}
\sum_i w_i (X_i-\bar{X_w})^2 &= \sum_i w_i(X_i^2-2X_i\bar{X_w}+\bar{X_w}^2)\\
&= \sum_i w_iX_i^2-2\bar{X_w}\sum_iw_iX_i+\bar{X_w^2}\\
&= \sum_i w_iX_i^2 -2\bar{X_w}^2+\bar{X_w}^2\\
&= \sum_i w_iX_i^2 -\bar{X_w}^2
\end{align*}

Now, we rewrite $\sum_i w_iX_i^2 = X'\Lambda X$ where $\Lambda = diag(w_1, w_2, \dots, w_n)$

and $\bar{X_w} = \sum_i w_iX_i = X'w = w'X$

\begin{align*}
\bar{X_w}^2 &= (\sum_i w_iX_i)^2\\
&=X'ww'X
\end{align*}

and hence $\sum_i w_i (X_i-\bar{X_w})^2=X'(\Lambda-ww')X$

Define $A=\Lambda-ww' = \begin{pmatrix}
w_1-w_1^2 & w_1w_2 & w_1w_3 & \dots & w_1w_n\\
w_2w_1 & w_2-w_2^2 & w_2w_3 & \dots & w_2w_n\\
\vdots & \vdots & \vdots & \vdots \\
w_nw_1 & w_nw_2 & w_nw_3 & \dots w_n-w_n^2\\
\end{pmatrix}$

so that
$$E[\sum_i w_i (X_i-\bar{X_w})^2] = E[X'AX]= tr(A\sum) + \mu'A\mu$$

$$tr(A\sum) = \sum_i(w_i-w_i^2)\sigma_i^2 = \sum_i w_i\sigma_i^2-w_i(w_i\sigma_i^2) = \sum_i (a-aw_i) = na-a$$

and the $(1,1)$ element of the matrix $\mu'A$ is given by:

\begin{align*}
\mu'A_1 &= \mu (w_1-w_1^2 - w_1\sum_{i=2}w_i)\\ &= w_1-w_1^2-w_1(1-w_1)\\
&= 0\end{align*}

and hence it's essentially a zero matrix(other elements are zero similarly)

Also, $$v_{min} = \frac{1}{\sum_i 1/\sigma_i^2} = \frac{1}{\sum_i \frac{w_i}{a}} = a$$

Thus,

\begin{align*}
E[S_w^2] &= \frac{1}{n-1} tr(A\sum) + \mu'A\mu\\
&= \frac{1}{n-1} (na-a) + 0\\
&=a\\
&= v_{min}
\end{align*}

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{tikz}
\setlength{\parindent}{0pt}

\newtheorem*{theorem}{Theorem}
\newtheorem*{definition}{Definition}
\newtheorem*{lemma}{Lemma}
\newtheorem*{corollary}{Corollary}
\newtheorem{example}{Example}
\newtheorem*{trick}{Trick}
\newtheorem*{question}{Question}

\title{Lecture 3: Matrices}
\author{}
\date{}

\begin{document}

\maketitle

\section{Matrices}

A motivation for bringing in matrices is better expressing linear relations
between variables.

\begin{example}
  The coordinates of a point in two different Cartesian coordinate systems have
  linear relations. Suppose in the first coordinate system, the point
  $P = (x_1, x_2, x_3)$, and in the second one, the point $P = (u_1, u_2, u_3)$.
  The relations between them can be described by linear equations, such as
  \[
    \left\{ \begin{array}{ll}
    u_1 = 2x_1 + x_2 + 5x_3 \\
    u_2 = x_1 + 3x_2 + 2x_3 \\
    u_3 = x_1 + 2x_2 + x_3
    \end{array} \right.
  \]
  The reason why their relation is linear is that each unit vector in the
  second coordinate system can be decomposed into three vectors along the
  directions of the unit vectors in the first coordinate system, i.e. each unit
  vector in the second coordinate system has a unique coordinate in the first
  system. Since $x_1$, $x_2$, $x_3$ are just scalars in three directions of the
  first coordinate system, the coordinates of the unit vectors of the second
  system can also be expressed with $x_1$, $x_2$, $x_3$, a linear combination
  of these three scalars. Then the coordinate of the point in the second
  coordinate system is also a linear combination of the unit vectors in the
  system, so the coordinate of the point in the second coordinate system is a
  linear combination of the coordinate of the point in the first one.

  Then we can actually use matrix product to express the linear system:
  \[
    \begin{bmatrix}
      u_1 \\
      u_2 \\
      u_3 \\
    \end{bmatrix}
    =
    \begin{bmatrix}
      2 & 1 & 5 \\
      1 & 3 & 2 \\
      1 & 2 & 1 \\
    \end{bmatrix}
    \begin{bmatrix}
      x_1 \\
      x_2 \\
      x_3 \\
    \end{bmatrix}
  \]
  \[
    U = AX
  \]
  Such a matrix $A = \begin{bmatrix}
                       2 & 1 & 5 \\
                       1 & 3 & 2 \\
                       1 & 2 & 1 \\
                     \end{bmatrix}$ is called a tranformation matrix, and such
  operation $AX$ is called matrix product.
\end{example}

\subsection{Definition of matrix product}

Entries in a matrix product $AB$ are the dot products of the corresponding rows
in the matrix $A$ and the corresponding columns in the matrix $B$ respectively.

Therefore, in order to apply the matrix product to the matrices $A$ and $B$ in
this way $AB$,
\[
  \textnormal{the number of columns in A} = \textnormal{the number of rows in B}
\]
In other words, the width of $A$ must equal the height of $B$.

\subsection{Property of matrix product}

\subsubsection{Associative property}

The matrix product satisfies the associative property.

\[
  (AB)X = A(BX)
\]

Therefore, when there are multiple tranformation matrices applied on a vector
such as $ABX$, it means the tranformation matrix $B$ is applied to $X$ first,
then the tranformation matrix $A$ is applied to their result.

\subsubsection{Commutative property}

The matrix product doesn't satisfy the commutative property.

\[
  AB \neq BA
\]

Not only in some cases $AB$ is valid while $BA$ doesn't make sense, but also
when both $AB$ and $BA$ are valid, their results can be different.

\subsection{Identity Matrix}

The identity matrix $I$ is defined as the matrix where
\[
  X = IX
\]

Deriving from the definition, we know that all identity matrices are square
matrices, and identity matrices always have the elements of the main diagonal
(the list of entries $A_{i,j}$ where $i = j$) as 1, and other elements outside
of the main diagonal as 0.

\[
  I_{4 \times 4} =
  \begin{bmatrix}
    1 & 0 & 0 & 0 \\
    0 & 1 & 0 & 0 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1 \\
  \end{bmatrix}
\]

\begin{example}
  Find the transformation matrix to make a plane rotated by $90^{\circ}$
  counterclockwise.

  Since the transformation matrix represents a rotation of a plane, i.e.
  transforming each 2D point on the plane into another 2D point, the size of
  the transformation matrix is $2 \times 2$:
  \[
    R =
    \begin{bmatrix}
      r_1 & r_2 \\
      r_3 & r_4 \\
    \end{bmatrix}
  \]
  Then we can use some points and their transformed results to figure out the
  elements:
  \[
    A =
    \begin{bmatrix}
      1 \\
      0 \\
    \end{bmatrix}
    A' =
    \begin{bmatrix}
      0 \\
      1 \\
    \end{bmatrix}
  \]
  \[
    B =
    \begin{bmatrix}
      0 \\
      1 \\
    \end{bmatrix}
    B' =
    \begin{bmatrix}
      -1 \\
      0 \\
    \end{bmatrix}
  \]
  After calculation,
  \[
    R =
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
  \]
  We can verify the result by multiply itself by 4 times, which in effect
  doesn't rotate the plane at all, hence the result should be the identity
  matrix.
  \[
    RRRR =
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix} =
    \begin{bmatrix}
      1 & 0 \\
      0 & 1 \\
    \end{bmatrix}
  \]
\end{example}

\section{Inverse Matrix}

\subsection{Definition of inverse matrix}

The inverse matrix of $A$, usually denoted as $A^{-1}$, is defined as
\begin{gather*}
  A^{-1}A = I \\
  AA^{-1} = I \\
\end{gather*}

Deriving from the definition, only square matrices have inverse matrix. The
matrices which have its corresponding inverse matrix are said to be invertible.

Invert matrices can help solve matrix equations:
\begin{gather*}
  AX = B \\
  A^{-1}AX = A^{-1}B \\
  X = A^{-1}B \\
\end{gather*}

\subsection{How to invert a matrix}

The formula of inverse matrices is:
\[
  A^{-1} = \frac{1}{det(A)}adj(A)
\]
$adj(A)$ is the adjugate matrix of $A$, or the classical adjoint matrix of $A$.

\bigskip

Steps to invert a matrix:

1. Calculate the minor $M$ of $A$. The entry on the position $(i,j)$ of the
minor, called the $(i,j)$-minor of $A$ and denoted $M_{i,j}$, is the determinant
of the matrices that results from deleting the row $i$ and column $j$ of $A$.

2. Calculate the cofactor matrix $C$ of $A$. The entry on the position $(i,j)$
of the cofactor matrix is the result of multiplying $M_{i,j}$ by $(-1)^{i+j}$.

3. Calculate the adjugate matrix $adj(A)$ of $A$ by transposing the cofactor
matrix of $A$. Transposing means switching rows and columns correspondingly.

4. Calculate the inverse matrix of $A$ by dividing the adjugate matrix of $A$ by the determinant of $A$.

\end{document}


% !Mode:: "TeX:UTF-8"
% !TEX program  = xelatex
In this section, we will give a proper and accurate definition of geometric progression, from which we will derive the summation formula for the geometric progression in both finite and infinite cases.

\subsection{Definition}
\begin{defnbox}{Geometric Progression\cite{Weisstein2019gp}}{gp}
    A geometric sequence is a sequence $\{a_k\}, k=0, 1, \ldots$, such that each term is given by a multiple $r$ of the previous one. Another equivalent definition is that a sequence is geometric iff\footnote{if and only if} it has a zero series bias. If the multiplier is $r$, then the $k$th term is given by
    \[
        a_k = ra_{k-1} = r^2a_{k-2} = \cdots = a_0r^k.
    \]

    Taking $a_0=1$ gives the simple special case
    \[
        a_k = r^k.
    \]
\end{defnbox}

Accordingly, we can define the arithmetic progression in advance to facilitate future introductions.
\begin{defnbox}{Arithmetic Progression\cite{Weisstein2019ap}}{ap}
    An arithmetic progression, also known as an arithmetic sequence, is a sequence of $n$ numbers $\{a_0+kd\}^{n-1}_{k=0}$ such that the differences between successive terms is a constant $d$. That is,
    \[
        a_k = a_{k-1}+d = a_{k-2}+d = \cdots = a_0+kd.
    \]

    Taking $a_0=0$ gives the simple special case
    \[
        a_k = kd.
    \]
\end{defnbox}

\subsection{Derivation}
\subsubsection{Summation Formula For the Geometric Progression}
According to the definition of the geometric progression, we have,
\begin{equation}\label{E:gp-1}
    S_n = \sum_{k=1}^{n} a_k = \sum_{k=1}^{n} a_1 r^{k-1}.
\end{equation}

Multiply both sides of the equation by $r$, then we have,
\begin{equation}\label{E:gp-2}
    rS_n = \sum_{k=1}^{n} a_1 r^{k} = \sum_{k=1}^{n} a_{k+1} = \sum_{k=2}^{n+1} a_k.
\end{equation}

Subtract equation~\eqref{E:gp-2} from equation~\eqref{E:gp-1}, we have,
\begin{equation}\label{E:gp-3}
    (1-r) S_n = a_1 - a_{n+1} = a_1(1-r^n).
\end{equation}

Therefore, we need a classification discussion, if the $r$ equals to 1, both sides of the equation cannot divide $1-r$ simultaneously, but we can derivate the formula from equation~\eqref{E:gp-1}, $S_n=na_1$.

Above all, we derivate the summation formula for the geometric progression,
\begin{equation}\label{E:gp-4}
    S_n = \begin{cases}
              na_1 & \text{if } r=1, \\
              \frac{a_1(1-r^n)}{1-r} & \text{if } r\neq 1.
          \end{cases}
\end{equation}

Moreover, if we use the form of the limit, we can simplify the formula,
\begin{equation}\label{E:gp-5}
    S_n = \lim_{q\to r} \frac{a_1(1-q^n)}{1-q}.
\end{equation}

Next we consider what happens when $n\to\infty$. $1-q$ is finite, the convergence of $S_n$ is equivalent to convergence of $\lim_{q\to r}1-q^n$. Therefore,  $S$ convergence if and only if $|r|<1$.
\begin{equation}\label{E:gp-6}
    S = \lim_{n\to\infty} S_n = \lim_{\substack{n\to\infty \\ q\to r}} \frac{a_1(1-q^n)}{1-q} = \lim_{n\to\infty}\frac{a_1(1-r^n)}{1-r} = \frac{a_1}{1-r}.
\end{equation}

In summary, we get the summation formula for the geometric progression,
\[
    S_n = \begin{cases}
              na_1 & \text{if } r=1, \\
              \frac{a_1(1-r^n)}{1-r} & \text{if } r\neq 1.
          \end{cases}
\]
and $S$ convergence if and only if $|r|<1$,
\[
    S = \lim_{n\to\infty} = \frac{a_1}{1-r}.
\]

\subsubsection{Summation Formula For the Arithmetic progression}
The summation formula for the arithmetic progression is slightly different from the geometric progression, and it relates back to a famous mathematician, Gauss. The main idea of the formula comes from
\[
    1+2+\cdots+n = \frac{n(n+1)}{2}.
\]

The proof part is left to readers as a practice question, we just provide clues below,
\begin{equation}\label{E:ap-1}
    \begin{aligned}
        S_{n+1} &= \sum_{k=0}^n a_k \\
            &= \sum_{k=0}^n (a_0+kd) \\
            &= \sum_{k=0}^n a_0 + d\sum_{k=0}^n k.
    \end{aligned}
\end{equation}


\documentclass[12pt]{mmalatex}
\usepackage{examples}

\begin{document}

\section*{Displaying long expressions}

This example uses a simple (though contrived) example of a Taylor series expansion of $1/(1+x)$ to demonstrate the problems that can arise when displaying very long expressions.

\begin{minipage}[t]{0.47\textwidth}
\begin{mathematica}
   f[x_] = 1/(1+x)
   ans = f[x]                       (* mma (ans.511,ans) *)
   ans = Series[f[x],{x, 0, 10}]    (* mma (ans.512,ans) *)
   ans = Series[f[x],{x, 0, 20}]    (* mma (ans.513,ans) *)
   ans = Series[f[x],{x, 0, 23}]    (* mma (ans.514,ans) *)
\end{mathematica}
\end{minipage}
\hskip 0.5cm
\begin{minipage}[t]{0.53\textwidth}
\begin{latex}
   \begin{dgroup*}[spread={5pt}]
      \begin{dmath*} f(x) = \Mma*{ans.511} \end{dmath*}
      \begin{dmath*}    {}= \Mma*{ans.512} \end{dmath*}
      \begin{dmath*}    {}= \Mma*{ans.513} \end{dmath*}
      \begin{dmath*}    {}= \Mma*{ans.514} \end{dmath*}
      \begin{dmath*}    {}= \Mma*[\hskip 2cm]{ans.514} \end{dmath*}
   \end{dgroup*}
\end{latex}
\end{minipage}

\vspace{18pt}

The first four lines of the following output were set using {\tt\small\verb|\Mma*|}
while the final line used {\tt\small\verb|\Mma*[\hskip=2cm]|}. The last pair of lines displays
the output for the same tag {\tt\small ans.514} and clearly the formatting of the second
last line is not ideal as the text has overlapped the tag. This was corrected in the final
line by using the option argument {\tt\small\verb|[\hskip=2cm]|} in the call to {\tt\small\verb|\Mma*|}.

\begin{dgroup*}[spread={5pt}]
   \begin{dmath*} f(x) = \Mma*{ans.511} \end{dmath*}
   \begin{dmath*}    {}= \Mma*{ans.512} \end{dmath*}
   \begin{dmath*}    {}= \Mma*{ans.513} \end{dmath*}
   \begin{dmath*}    {}= \Mma*{ans.514} \end{dmath*}
   \begin{dmath*}    {}= \Mma*[\hskip 2cm]{ans.514} \end{dmath*}% LCB: do we need extra space for the tag?
\end{dgroup*}

\end{document}


\subsubsection{The Definition of Entropy}
Now that we know what micro and macro states are, we can define entropy using statistics! In this case, entropy is defined as
\begin{equation}
    \label{eqn:(41)}
    S=k_{b}\ln{\Omega}
\end{equation}

where $\Omega$ is the number of possible microstates within the current macrostate of the system. \\
With a closed system of constant volume, we can characterize macrostates using the temperature of the system, and microstates as the exact distribution of kinetic energies amongst the particles in the system. Using this we can start to see all the parallels between the two different ways of defining entropy.
\begin{enumerate}
    \item At $T=0\textrm{K}$ every particle must have no kinetic energy. So, there's only one possible microstate, and by equation \ref{eqn:(41)} above $S=0$. This is exactly what we'd expect from the third law of thermodynamics, at absolute zero temperature, we obtain that entropy has a constant value\footnote{Note that the third law of thermodynamics does not say a system reaches zero entropy as it approaches zero temperature, as there do exist certain systems where the minimum energy (i.e. low temperature) states are not unique. Therefore, there still can be more than one microstate, and hence nonzero (but still constant) entropy. You don't need to know about these in any kind of detail.}.
    \item If you have two systems (a and b), the amount of microstates in a system including both of them is $\Omega_{a}\Omega_{b}$. Therefore, the total entropy is...
    \begin{align*}
        S_{total}&=k_{b}\ln{\Omega_{a}\Omega_{b}} \\
        &=k_{b}\ln{\Omega_{a}}+k_{b}\ln{\Omega_{b}} \\
        &=S_{1}+S_{2}
    \end{align*}
    Since the total entropy is the sum of the entropy of both systems, entropy is an extensive quantity!
    \item In our example for microstates/macrostates above, the macrostate with the highest number of microstates was the one where each side had three particles (if you don't believe me count them). This analogy applies to two systems that can exchange energy, they have the maximum number of microstates (and therefore entropy) when they share the energy equally (i.e. have the same temperature).
\end{enumerate}
To close off our discussion of macrostates, microstates, and the statistical definition of entropy, let us consider another concrete example, in the form of the entropy of rolling two dice. Therein, we can consider the macrostate as the sum of the two dice rolls, and the microstates as the particular values we got on each dice (here, we imagine that the dice are unique, in that I can tell them apart from each other). For example, if I rolled the first dice to be 6, and the second dice to be 1, then the microstate could be described with $(6,1)$ and this microstate belongs to the macrostate of $7$. A complete description of all of the macrostates and microstates (as well as the entropy of the macrostate, as defined in this section) is given in the table below, though you may want to work this out for yourself first for practice.

\begin{center}
 \begin{tabular}{|c c c|}
 \hline
 \textbf{Macrostates} & \textbf{Microstates} & \textbf{Entropy}\\
 \hline\hline
 2 & (1,1) & $k_b\ln(1) = 0$\\
 \hline
 3 & (1,2),(2,1) & $k_b\ln(2)$ \\
 \hline
 4 & (1,3),(2,2),(3,1) & $k_b\ln(3)$ \\
 \hline
 5 & (1,4),(2,3),(3,2),(4,1) & $k_b\ln(4)$ \\
 \hline
 6 & (1,5),(2,4),(3,3),(4,2),(5,1) & $k_b\ln(5)$ \\
 \hline
 7 & (1,6),(2,5),(3,4),(4,3),(5,2),(6,1) & $k_b\ln(6)$\\
 \hline
 8 & (2,6),(3,5),(4,4),(5,3),(6,2) & $k_b\ln(5)$\\
 \hline
 9 & (3,6),(4,5),(5,4),(6,3) & $k_b\ln(4)$\\
 \hline
 10 & (4,6),(5,5),(6,4) & $k_b\ln(3)$\\
 \hline
 11 & (5,6),(6,5) & $k_b\ln(2)$\\
 \hline
 12 & (6,6) & $k_b\ln(1) = 0$\\
 \hline
\end{tabular}
\end{center}
This example also demonstrates an assumption I implicitly made up until this point: all possible microstates are equally likely to appear (in the last example, unless you had loaded dice, every dice roll would have been equally likely!). This is a very fundamental assumption, so much so that it is called the \textbf{Fundamental Assumption of Statistical Mechanics}. Just to restate it for a more general case (as not every system in the universe is a bunch of dice), it states that every microstate of a system is equally likely and the system on average spends the same amount of time in each of them. We will use this in the next section to conclude our discussion of entropy.


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}

\title{MATH 542 Homework 9}
\author{Saket Choudhary\\skchoudh@usc.edu}

\begin{document}
\maketitle
\subsection*{Problem 1}
Since $\mathbf{X}$ is full rank, the least sqare estimate of $\beta$ for the linear model $\mathbf{Y}=\mathbf{y}\beta+\epsilon$ then $\hat{\beta} = \mathbf{(X'X)^{-1}X'y}$ also this is an unbiased estimator of $\beta$ that is $E[\hat{\beta}]=\beta$
\begin{align*}
a'\beta &= a'E[\hat{\beta}]\\
&= a'E[\mathbf{(X'X)^{-1}X'y}]\\
&= E[a'\mathbf{(X'X)^{-1}X'y}]\\
&= E[b'\mathbf{y}]
\end{align*}
Hence when $\mathbf{X}$ is full rank, we have can take $b'=a'\mathbf{(X'X)^{-1}X'}$ making every $a'\beta$ estimable

For each individual $\beta_i$ to be estimable, we set $a'=\begin{pmatrix}0 & 0 & \dots & 0 & 1 & 0 \dots 0 \end{pmatrix}$
setting $a_i=1$ remaining zero.

\subsection*{Problem 2}
Since each $a_i'\beta$ is estimable: $a_i'\beta = E[b_i'Y]$
Consider $\lambda_i \in \mathcal{R}$
\begin{align*}
\lambda_1a_1'\beta + \lambda_2a_2'\beta + \dots + \lambda_na_n'\beta &= \lambda_1 E[b_1'Y] +  \lambda_2 E[b_2'Y] + \dots \lambda_n E[b_n'Y]\\
&= \sum_i E[\lambda_ib_i'Y]\\
&= \sum E[c'Y] \text{ where }c'= \mathbf{\lambda I b'}
\end{align*}
where $\lambda =  \begin{pmatrix}\lambda_1 & \lambda_2 &  \dots & \lambda_n \end{pmatrix}$ and $b' = \begin{pmatrix} b_1 & b_2 & \dots b_n \end{pmatrix}$

Hence linear combination of $a_i'\beta$ is also estimable.

\subsection*{Problem 3}
\begin{align*}
X &= \begin{pmatrix}
1 & 1 & 3\\
1 & 1 & -2\\
1 & 1 & 0\\
1 & 0 & 0\\
\end{pmatrix}
\beta & = \begin{pmatrix}
\beta_1\\
\beta_2\\
\beta_3
\end{pmatrix}\\
\beta_1 & = \begin{pmatrix}1 & 1 & 0\end{pmatrix}\mathbf{\beta}\\
\beta_1-\beta_2 &= \begin{pmatrix}1 &-1 & 0\end{pmatrix}\mathbf{\beta}\\
5\beta_1+3\beta_2+9\beta_3 &= \begin{pmatrix}5 & 3& 9\end{pmatrix}\mathbf{\beta}\\
\end{align*}

X has full column rank and hence we can make use of the theorem we proved in Problem 1 to say that all three cases $(a,b,c)$ are indeed estimable, that is since $X$ is full rank, every $a'\beta$ is estimable (One particular case of the corollary proved in Problem 1 here is the case $a$ where $a'=\begin{pmatrix}1 & 0 & 0\end{pmatrix}$
\subsection*{Problem 4}
\subsubsection*{Problem 4.a}
Define $\mathbf{y}=\begin{pmatrix}Y_1 & Y_2 &  Y_3\end{pmatrix}'$, $\mathbf{\tau} = \begin{pmatrix} \tau_1 & \tau_2 & \tau_3 \end{pmatrix}$, $\mathbf{\epsilon} = \begin{pmatrix}\epsilon_1 & \epsilon_2 & \epsilon_3\end{pmatrix}$

Thus, $\mathbf{y} = \begin{pmatrix}1 & 1& 1\\
1 & 0 & 1\\
0 & 1 & 0
\end{pmatrix}\mathbf{\tau} + \mathbf{\epsilon}$

Rank of design matrix is 2

\subsubsection*{Problem 4.b}

$E[b'Y] = a'\beta = b'X\beta$ iff $a'=b'X$ or $a=Xb'$
%which is equivalent to saying that $a$ belongs to row space of $X$
$\tau_2$ is estimable:
\begin{align*}
\tau_2  &= a'\mathbf{\beta}= \begin{pmatrix}0 & 1 & 0\end{pmatrix}\mathbf{\tau}\\
&= EY_3\\
&= E[\begin{pmatrix}0 & 0 & 1\end{pmatrix}]
\end{align*}

$\tau_1 = \begin{pmatrix}1 & 0 & 0 \end{pmatrix}\mathbf{\tau}$ so we need to find $b$ such that $\begin{pmatrix}1 & 0 & 0 \end{pmatrix}' = X'b$ It is clear to see no b does not exist.

$\tau_2 =  \begin{pmatrix}0 & 1 & 0\end{pmatrix}\mathbf{\tau}$ is estimable because $b=\begin{pmatrix}0 & 1 & 0 \end{pmatrix}$

$\tau_3$ is not estimable because of symmetry with $\tau_1$

\subsubsection*{Problem 4.c}

$\tau_1-2\tau_2+\tau_3 = \begin{pmatrix}1 &-2 &1 \end{pmatrix}\mathbf{\tau} = \begin{pmatrix}1 &1 &0\\ 1 &0 &1\\ 0 & 1 & 0\end{pmatrix} \begin{pmatrix}0\\1\\-2\end{pmatrix}$ and hence it is estimable

\subsubsection*{Problem 4.d}

A possible unbiasd estimator of $\tau_1-2\tau_2+\tau_3 = E[Y_2-2Y_3]$ i.e $Y_2-2Y_3$ whihc is not necessarily BLUE.

For BLUE, we simply take the OLS estimate of $\beta$ as $\hat{\beta} = \mathbf{X'X}^{-1}\mathbf{X'y}$ and from Gauss-Markov model BLUE follows.

Generalized inverse(using $R$) $\mathbf{X'X}^{-1} = \begin{pmatrix} \frac{1}{6} & \frac{-1}{6} & \frac{1}{6}\\
\frac{-1}{6} & \frac{2}{3} & \frac{-1}{6}\\
\frac{1}{6} & \frac{-1}{6} & \frac{1}{6} \\
\end{pmatrix}$

so $\hat{\beta} =\mathbf{X'X}^{-1}\mathbf{X'y} = \begin{pmatrix} \frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\frac{1}{3} & \frac{-1}{3} & \frac{2}{3}\\
\frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\end{pmatrix}\mathbf{y}$

and hence $\tau_1-2\tau_2+\tau_3 = \begin{pmatrix} 1 &-2 &1\end{pmatrix}\begin{pmatrix} \frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\frac{1}{3} & \frac{-1}{3} & \frac{2}{3}\\
\frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\end{pmatrix}\mathbf{y} = \frac{-Y_1+4Y_2-5Y_3}{3}  $

Thus BLUE of  $\tau_1-2\tau_2+\tau_3$ : $ \frac{-Y_1+4Y_2-5Y_3}{3}$
\end{document}


\section{Numpy} % (fold)
\label{sec:numpy}
Generate matrices $A$, with random Gaussian entries, $B$, a Toeplitz matrix, where
$A \in \R^{n\times m}$ and $B \in \R^{m \times m}$, for n = 200, m = 500.

\begin{questions}

\titledquestion{Matrix operations}
\label{sub:mat_ops}

Calculate $A + A$, $A A^\top, A^\top A$ and $AB$.
Write a function that computes $A(B - \lambda I)$ for any $\lambda$.

\titledquestion{Solving a linear system}
\label{sub:linear_system}

Generate a vector $b$ with $m$ entries and solve $Bx = b$.

\titledquestion{Norms} % (fold)
\label{sub:norms}

Compute the Frobenius norm of $A$: $\|A\|_F$ and the infinity norm of $B$:
$\|B\|_{\infty}$. Also find the largest and smallest singular values of $B$.

% titledquestion norms (end)

\titledquestion{Power iteration}
\label{sub:power}

Generate a matrix $Z$, $n \times n$, with Gaussian entries, and use the power iteration to
find the largest eigenvalue and corresponding eigenvector of $Z$.
How many iterations are needed till convergence?

Optional: use the \texttt{time.clock()} method to compare computation time when varying $n$.

\titledquestion{Singular values} % (fold)
\label{sub:svd}

Generate an $n \times n$ matrix, denoted by $C$, where each entry is $1$ with probability $p$ and
$0$ otherwise.
Use the linear algebra library of Scipy to compute the singular values of $C$.
What can you say about the relationship between $n$, $p$ and the largest singular value?

% titledquestion svd (end)

\titledquestion{Nearest neighbor} % (fold)
\label{sub:nearest_neighbor}

Write a function that takes a value $z$ and an array $A$ and finds
the element in $A$ that is closest to $z$. The function should
return the closest value, not index.

Hint: Use the built-in functionality of Numpy rather than writing code to find this
value manually. In particular, use brackets and \texttt{argmin}.

% titledquestion nearest_neighbor (end)

\end{questions}
% section numpy (end)


\chapter{eq}

\begin{equation}
T_{motes}\ = N_{slots}*N_{motes}
\end{equation}

\begin{tabular}{lllp{10cm}}
where, &   &   &   \\
%\hline
  & $T_{motes}$  & = & the maximum number of motes that can be accommodated in the network \\
%\hline
  & $N_{slots}$ & = & the maximum number of transmission slots in a transmission period \\
%\hline
  & $N_{motes}$ & = & the maximum number of motes in each transmission slot of the \textit{\textbf{root}} mote \\
\end{tabular}

\begin{center}
$N_{motes}\ =\ \dfrac{internal\ message\ buffer\ size}{message\ length}$
\end{center}

$N_{slots}\ =\ \dfrac{transmission\ interval}{transmission\ slot\ duration}$

$B_{charged}\ =\ T_{sunlight}*I$

\begin{tabular}{lllp{10cm}}
where, &   &   &   \\
%\hline
  & $B_{charged}$  & = & the amount of battery charged $(milliamps)$ \\
%\hline
  & $T_{sunlight}$ & = &  the time for which sunlight is present $(hours)$\\
%\hline
  & $I$ & = & the amount of current used to charge the battery $(milliamps)$\\
\end{tabular}


\input ../6001mac

\begin{document}

\psetheader{Sample Problem Set}{Streams And Series}
\begin{center}
{\bf Streams}
\end{center}

\section{Part 1: Tutorial exercises}

Prepare the following exercises for oral presentation in tutorial:

\paragraph{Tutorial exercise 1:}
Describe the streams produced by the following definitions.  Assume
that {\tt integers} is the stream of non-negative integers (starting
from 1):

\beginlisp
(define A (cons-stream 1 (scale-stream 2 A)))
\null
(define (mul-streams a b)
  (cons-stream
   (* (stream-car a) (stream-car b))
   (mul-streams (stream-cdr a)
                (stream-cdr b))))
\null
(define B (cons-stream 1 (mul-stream B integers)))
\endlisp

\paragraph{Tutorial exercise 2}

Given a stream {\tt s} the following procedure returns the stream of
all pairs of elements from {\tt s}:

\beginlisp
(define (stream-pairs s)
  (if (stream-null? s)
      the-empty-stream
      (stream-append
       (stream-map
        (lambda (sn) (list (stream-car s) sn))
        (stream-cdr s))
       (stream-pairs (stream-cdr s)))))
\null
(define (stream-append s1 s2)
  (if (stream-null? s1)
      s2
      (cons-stream (stream-car s1)
                   (stream-append (stream-cdr s1) s2))))
\endlisp

\noindent
(a) Suppose that {\tt integers} is the (finite) stream 1, 2, 3, 4, 5.
What is {\tt (stream-pairs s)}?  (b) Give the clearest explanation
that you can of how {\tt stream-pairs} works.  (c) Suppose that {\tt
s} is the stream of positive integers.  What are the first few
elements of {\tt (stream-pairs s)}?  Can you suggest a modification of
{\tt stream-pairs} that would be more appropriate in dealing with
infinite streams?

\section{Part 2: Laboratory---Using streams to represent power series}

We described in lecture a few weeks ago how to represent polynomials
as lists of terms.  In a similar way, we can work with {\it power
series}, such as
\begin{eqnarray*}
e^{x} &=&
1+x+\frac{x^{2}}{2}+\frac{x^{3}}{3\cdot2}+\frac{x^{4}}{4\cdot 3\cdot
2}+\cdots \cr
\cos x &=& 1-\frac{x^{2}}{2}+\frac{x^{4}}{4\cdot 3\cdot 2}-\cdots \cr
\sin x &=& x-\frac{x^{3}}{3\cdot 2}+\frac{x^{5}}{5\cdot 4\cdot 3\cdot 2}- \cdots
\end{eqnarray*}
represented as streams of infinitely many terms.  That is, the power
series

\[ a_0 + a_1 x + a_2 x^2 + a_3 x^3 + \cdots \]

\noindent
will be represented as the infinite stream whose elements are $a_0,
a_1, a_2, a_3, \ldots$.\footnote{In this representation, all streams are
infinite: a finite polynomial will be represented as a stream with an
infinite number of trailing zeroes.}

Why would we want such a method?  Well, let's separate the idea of a
series representation from the idea of evaluating a function.  For
example, suppose we let $f(x) = \sin x$.  We can separate the idea of
evaluating $f$, e.g., $f(0) = 0, f(.1) = 0.0998334$, from the means we
use to compute the value of $f$.  This is where the series
representation is used, as a way of storing information sufficient to
determine values of the function.  In particular, by substituting a
value for $x$ into the series, and computing more and more terms in
the sum, we get better and better estimates of the value of the
function for that argument.  This is shown in the table, where $\sin
{1 \over 10}$ is considered.

\begin{tabular}{lllll}
Coefficient&$x^n$&term&sum&value\\
\hline
0& 1& 0 & 0& 0\\
\\
1& ${1 \over 10}$& ${1 \over 10}$ & ${1 \over 10}$& .1\\
\\
0&${  1 \over 100}$& 0 &  ${1 \over 10}$& .1\\
\\
- ${1 \over 6}$& ${1 \over 1000}$& - ${1 \over 6000}$ & ${599 \over 6000}$& .099833333333\\
\\
0& ${1 \over 10000}$& 0 & ${599 \over 6000}$& .099833333333\\
\\
${1 \over 120}$& ${1 \over 100000}$& ${1 \over 12000000}$ & ${1198001 \over
12000000}$& .09983341666\\
\\
\end{tabular}

The first column shows the terms from the series representation for
sine.  This is the infinite series with which we will be dealing.  The
second column shows values for the associated powers of ${1 \over 10}$.
The third column is the product of the first two, and represents the
next term in the series evaluation.  The fourth column represents the
sum of the terms to that point, and the last column is the decimal
approximation to the sum.

With this representation of functions as streams of coefficients,
series operations such as addition and scaling (multiplying by a
constant) are identical to the basic stream operations.  We provide
series operations, though, in order to implement a complete power
series data abstraction:

\beginlisp
(define (add-streams s1 s2)
  (cond ((stream-null? s1) s2)
        ((stream-null? s2) s1)
        (else
         (cons-stream (+ (stream-car s1) (stream-car s2))
                      (add-streams (stream-cdr s1)
                                   (stream-cdr s2))))))
\null
(define (scale-stream c stream)
  (stream-map (lambda (x) (* x c)) stream))
\null
(define add-series add-streams)
\null
(define scale-series scale-stream)
\null
(define (negate-series s)
  (scale-series -1 s))
\null
(define (subtract-series s1 s2)
  (add-series s1 (negate-series s2)))
\endlisp

\noindent You can use the following procedure to examine the series you will
generate in this problem set:

\beginlisp
(define (show-series s nterms)
  (if (= nterms 0)
      'done
      (begin (write-line (stream-car s))
             (show-series (stream-cdr s) (- nterms 1)))))
\endlisp

\noindent
You can also examine an individual coefficient (of $x^n$) in a series
using {\tt series-coeff}:

\beginlisp
(define (series-coeff s n)
  (stream-ref s n))
\endlisp

We also provide two ways to construct series.  {\tt Coeffs->series}
takes an list of initial coefficients and pads it with zeroes to
produce a power series.  For example, {\tt (coeff->series '(1 3 4))}
produces the power series $1+3x+4x^2+0x^3+0x^4+\ldots{}$.

\beginlisp
(define (coeffs->series list-of-coeffs)
  (define zeros (cons-stream 0 zeros))
  (define (iter list)
    (if (null? list)
        zeros
        (cons-stream (car list)
                     (iter (cdr list)))))
  (iter list-of-coeffs))
\endlisp

{\tt Proc->series} takes as argument a procedure $p$ of one numeric
argument and returns the series
\[  p(0) + p(1)x + p(2)x^2 + p(3)x^3 + \cdots  \]
The definition requires the stream {\tt non-neg-integers} to be the stream of
non-negative integers: $0,1,2,3,\ldots{}$\,.

\beginlisp
(define (proc->series proc)
  (stream-map proc non-neg-integers))
\endlisp

\medskip

\noindent
{\bf Note:} Loading the code for this problem set will change
Scheme's basic arithmetic operations {\tt +}, {\tt -}, {\tt *}, and
{\tt /} so that they will work with rational numbers.  For instance,
{\tt (/ 3 4)} will produce 3/4 rather than .75.  You'll find this
useful in doing the exercises below.

\paragraph{Lab exercise 1:}
Load the code for problem set 9.  To get some initial practice with
streams, write and turn in definitions for each of the following:

\begin{itemize}

\item {\tt ones}: the infinite stream of 1's.

\item {\tt non-neg-integers}:  the stream of integers, $1, 2, 3, 4,
\ldots$

\item {\tt alt-ones}:  the stream $1, -1, 1, -1, \ldots$

\item {\it zeros}:  the infinite stream of 0's.  Do this using {\tt alt-ones}.

\end{itemize}

Now, show how to define the series:
\begin{eqnarray*}
S_1 &=& 1 + x + x^2 + x^3 + \cdots \cr
S_2 &=& 1 + 2x + 3x^2 + 4x^3 + \cdots
\end{eqnarray*}
Turn in your definitions and a couple of coefficient printouts to
demonstrate that they work.

\paragraph{Lab exercise 2:}

Multiplying two series is a lot like multiplying two multi-digit numbers,
but starting with the left-most digit, instead of the right-most.

For example:

\begin{verbatim}

                11111
             x  12321
            _________

            11111
             22222
              33333
               22222
                11111
            ---------
            136898631

\end{verbatim}

Now imagine that there can be an infinite number of digits, i.e., each of
these is a (possibly infinite) series.  (Remember that because each "digit"
is in fact a term in the series, it can become arbitrarily large, without
carrying, as in ordinary multiplication.)

Using this idea, complete the definition of the following procedure,
which multiplies two series:
\beginlisp
(define (mul-series s1 s2)
  (cons-stream $\langle E_1 \rangle$
               (add-series $\langle E_2 \rangle$
                           $\langle E_3 \rangle$)))
\endlisp
To test your procedure, demonstrate that the product of $S_1$ (from
exercise 1) and $S_1$ is $S_2$.  What is the coefficient of
$x^{10}$ in the product of $S_2$ and $S_2$?  Turn in your definition of
{\tt mul-series}.  (Optional: Give a general formula for the
coefficient of $x^n$ in the product of $S_2$ and $S_2$.)

\subsection*{Inverting a power series}

Let $S$ be a power series whose constant term is 1.  We'll call such a
power series a ``unit power series.''  Suppose we want to find the {\em
inverse} of $S$, namely, the power series $X$ such that $S\cdot X= 1$.
To see how to do this,
write $S=1+S_R$ where $S_R$ is the rest of $S$ after the constant
term.  Then we want to solve the equation $S \cdot X = 1$ for $S$ and
we can do this as follows:

\begin{eqnarray*}
S \cdot X &=& 1 \cr
(1+S_R)\cdot X &=& 1 \cr
X + S_R \cdot X &=& 1 \cr
X &=& 1 - S_R \cdot X
\end{eqnarray*}

In other words, $X$ is the power series whose constant term is 1 and
whose rest is given by the negative of $S_R$ times $X$.

\paragraph{Lab exercise 3:} Use this idea to write a procedure {\tt
invert-unit-series} that computes $1/S$ for a unit power series $S$.
To test your procedure, invert the series $S_1$ (from exercise 1) and
show that you get the series $1-x$.  (Convince yourself that this is
the correct answer.)  Turn in a listing of your procedure.  This is a
very short procedure, but it is very clever.  In fact, to someone
looking at it for the first time, it may seem that it can't
work---that it must go into an infinite loop.  Write a few sentences
of explanation explaining why the procedure does in fact work, and
does not go into a loop.

\paragraph{Lab exercise 4:} Use your answer from exercise 3 to produce
a procedure {\tt div-series} that divides two power series.  {\tt
Div-series} should work for any two series, provided that the
denominator series begins with a non-zero constant term.  (If the
denominator has a zero constant term, then {\tt div-series} should
signal an error.)  Turn in a listing of your procedure along with
three or four well-chosen test cases (and demonstrate why the answers
given by your division are indeed the correct answers).

\paragraph{Lab exercise 5:}  Now suppose that we want to integrate a
series representation.  By this, we mean that we want to perform
symbolic integration, thus, for example,
given a series
\[ a_0 + a_1 x + a_2 x^2 + a_3 x^3 + \cdots \]
we want to return the integral of the series (except for the constant term)
\[ a_0 x + \frac{1}{2}a_1 x^2 + \frac{1}{3}a_2 x^3 + \frac{1}{4}a_3 x^4 + \cdots \]

Define a procedure {\tt
integrate-series-tail} that will do this.   Note that all you need to
do is transform the series

\[ a_0 \ \ \ \ \  a_1 \ \ \ \ \  a_2  \ \ \ \ \  a_3  \ \ \ \ \   a_4  \ \ \ \ \   a_5  \ \ \ \ \  \cdots \]

into the series

\[ a_0 \ \ \ \ \  {a_1 \over 2} \ \ \ \ \  {a_2 \over 3}  \ \ \ \ \
{a_3 \over 4}  \ \ \ \ \
{a_4 \over 5}  \ \ \ \ \
{a_5 \over 6}  \ \ \ \ \  \cdots \]

Note that this means that the procedure generates the coefficients of
a series starting with the first order coefficient, not that the
zeroth order coefficient is 0.

Turn in a listing of your procedure and demonstrate that it works by
computing {\tt integrate-series-tail} of the series $S_2$ from
exercise 1.

\paragraph{Lab exercise 6:} Demonstrate that you can generate the
series for $e^x$ as

\beginlisp
(define exp-series
  (cons-stream 1 (integrate-series-tail exp-series)))
\endlisp

\noindent Explain the reasoning behind this definition.  Show how to generate
the series for sine and cosine, in a similar way, as a pair of mutually
recursive definitions.  It may help to recall that the integral
\[\int \sin x =  - \cos x\]
and that the integral
\[\int \cos x = \sin x\]

\paragraph{Lab exercise 7:} Louis Reasoner is unhappy with the idea
of using {\tt integrate-series-tail} separately.  ``After all,'' he
says, ``if we know what the constant term of the integral is supposed
to be, we should just be able to incorporate that into a procedure.''
Louis consequently writes the following procedure, using {\tt
integrate-series-tail}:

\beginlisp
(define (integrate-series series constant-term)
  (cons-stream constant-term (integrate-series-tail series)))
\endlisp

\noindent
He would prefer to define the exponential series as

\beginlisp
(define exp-series
  (integrate-series exp-series 1))
\endlisp

\noindent
Write a two or three sentence clear explanation of why this won't
work, while the definition in exercise 6 does work.

\paragraph{Lab exercise 8:} Write a procedure that produces the
derivative of a power series.  Turn in a definition of your procedure
and some examples demonstrating that it works.

\paragraph{Lab exercise 9:} Generate the power series for tangent,
and secant.  List the first ten or so coefficients of each
series.  Demonstrate that the derivative of the tangent is the square
of the secant.

\paragraph{Lab exercise 10:}  We can also generate power series for
inverse trigonometric functions.  For example:
\[\tan^{-1}(x) = \int_0^x {dz \over {1 + z^2}}\]
Use this equation, plus methods that you have already created, to
generate a power series for arctan.
Note that $1+z^2$ can be viewed as a finite series.
Turn in your definition, and a
printout of the first few coefficients of the series.

\paragraph{Lab exercise 11:}  One very useful feature of a power
series representation for a function is that one can use the initial
terms in the series to get approximations to the function.  For
example, suppose we have

\[f(x) = a_0 + a_1x + a_2 x^2 + a_3 x^3 + \ldots\]

We have represented this by the series of coefficients:

\[\left\{a_0\ \ a_1\ \ a_2 \ \ a_3 \ \ \ldots\right\}\]

Now suppose that we want to approximate the value of the function $f$
at some point $x_0$.  We could successively improve this
approximation\footnote{actually there are some technical issues
about whether the argument is in the radius of convergence of the
series, but we'll ignore that issue here} by considering the following

\begin{eqnarray} f(x_0) \approx a_0\\ f(x_0) \approx a_0 + a_1 x_0\\
f(x_0) \approx a_0 + a_1 x_0 + a_2 x_0^2\\
\end{eqnarray}

Notice that each of these expressions (1), (2), (3) could also be captured in a
stream representation, with first term $a_0$, second term $a_0 + a_1
x_0$ and so on.

Implement this idea by defining a procedure {\tt approximate} which
takes as arguments a value $x_0$ and a series representation of a
function $f$, and which returns a stream of successive approximations
to the value of the function at that point $f(x_0)$.  Turn in a
listing
of your code, as well as examples of using it to approximate some
functions.
Note that to be very careful, this is not a series representation but
a stream one, so you may want to think carefully about which
representations to use.

{\bf Optional additional exercises}

he {\em Bernoulli numbers} $B_n$ are
defined by the coefficients in the following power series:%
%
\footnote{The Bernoulli numbers arise in a wide variety of
applications involving power series and approximations.  They were
introduced by Jacob Bernoulli in 1713.}%
%
\[
\frac{x}{e^x-1} = B_0 + B_1 x + \frac{B_2 x^2}{2!} + \frac{B_3
x^3}{3!} + \cdots = \sum_{k \geq 0} \frac{B_k x^k}{k!}
\]
Write a procedure that takes $n$ as an argument and produces the $n$th
Bernoulli number.  (Check: All the odd Bernoulli numbers are zero
except for $B_1=-1/2$.)  Generate a table of the first 10 (non-zero)
Bernoulli numbers.  Note: Since $e^x-1$ has a zero constant term, you
can't just use {\tt div-series} to divide $x$ by this directly.

It turns out that the coefficient of
$x^{2n-1}$ in the series expansion of $\tan x$ is given by
\[
\frac{(-1)^{n-1} 2^{2n} (2^{2n} - 1) B_{2n}}{(2n)!}
\]
Use your series expansion of tangent to verify this for a few of
values of $n$.

\end{document}