\chapter{Results} \label{sec:Results}
\section{Identification Problem's Results} \label{sec:Identification Problem's Results}
\subsection{Optimization Results} \label{sec:IdProbRes - Optimization}
Running the 3-layered network with the GPU ``set'' with different learning rates on different seeds for 10000 epochs showed that the model was constantly giving lower loss values (\Cref{eqn:iden_problem}) with learning rate = $10^{-3}$.

We observed that higher learning rates ($>10^{-2}$) could only decrease the loss value to a limit, after which the updates in the weights caused the loss function to oscillate and increase. On the other side of $10^{-3}$, lower learning rates took too long to train. This phenomena is exemplified in \Cref{fig:Test Loss Plots of Different Learning Rates to Find Weights}, which shows plots of different runs with the same seed but different learning rates. Runtime for the model on 10000 epochs was 1232.56 seconds (average over 20 runs = 5 seeds $\times$ 4 learning rates), and models with learning rates less than $10^{-3}$ did not perform better than those with learning rate = $10^{-3}$ on \textit{any} run. Although there was constant, gradual decrease in test losses for learning rates less than $10^{-3}$, we argue that models with those specifications may be computationally expensive and temporally inconvenient to train.
\begin{figure}[!htbp]
    \centering
    \includegraphics[width=16cm, height=10cm]{test_losses_diff_lr_plot}
    \caption[Test Loss Plots of Different Learning Rates to Find Weights]{Test Loss Plots of Different Learning Rates to Find Weights: Learning rates lower than $10^{-3}$ couldn't reach the loss value attained by the model at learning rate = $10^{-3}$ even after 10000 epochs.}
    \label{fig:Test Loss Plots of Different Learning Rates to Find Weights}
\end{figure}
\begin{figure}[!htbp]
    \centering
    \begin{minipage}{.48\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        ybar,
        width=.98\textwidth,
        enlarge x limits=0.15,
        legend style={at={(0.02,0.98)},
            anchor=north west,legend columns=1},
        ylabel={Loss Value},
        symbolic x coords={2-layer,3-layer,4-layer, Historical},
        xtick=data,
        nodes near coords,
        nodes near coords align={vertical},
        every node near coord/.append style={/pgf/number format/fixed},
        xlabel={Models},
        ]
        \addplot coordinates {(2-layer,0.26) (3-layer,0.12) (4-layer,0.23) (Historical, 0.36)};% test set
        \addplot coordinates {(3-layer,0.03) (4-layer,0.06)};% training set
        \legend{Test Set,Training Set}
        \end{axis}
        \end{tikzpicture}
        \caption[Comparison of Loss Values from Different Models of the Identification Problem]{Comparison of Loss Values from Different Models of the Identification Problem: Loss values for the training set are inevitably lower than that for the test set, which is the basis for comparison.}
        \label{fig:Comparison of Loss Values from Different Models of the Identification Problem}
    \end{minipage}\hfill
    \begin{minipage}{.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{weights_map_plot_3}
        \caption[Predicted Probabilities of Agents Visiting Each Location Plotted on a Map (Latitude, Longitude) Representing Tompkins and Cortland Counties, NY]{Predicted Probabilities of Agents Visiting Each Location Plotted on a Map (Latitude, Longitude) Representing Tompkins and Cortland Counties, NY: Dark dots represent high prediction of visits. This can be compared to the plots for the 2-layered network and other models \cite[Figure~3]{Xue2016Avi2}.}
        \label{fig:Predicted Probabilities of Agents Visiting Each Location Plotted on a Map}
    \end{minipage}
\end{figure}

\begin{table}[!htbp]
    \centering
    \caption[Loss Values Calculated for Different Models for Identification Problem]{Loss Values Calculated for Different Models for Identification Problem: For both the 3- and the 4-layered models, learning rate = $10^{-3}$ outperformed other learning rates. Consequently, that learning rate is used in comparison with other models in \Cref{fig:Comparison of Loss Values from Different Models of the Identification Problem}.}
    \label{tab:Loss Values Calculated for Different Models for Identification Problem}
    \begin{tabular}{c | c | c}
        \hline
        \multirow{2}{*}{\textbf{Learning Rate}} & \multicolumn{2}{c}{\textbf{Average Test Loss Values}} \\
        & 3-layered & 4-layered\\
        \hline
        $10^{-2}$ & 0.168 $\pm$ 0.068 & 0.494 $\pm$ 0.083\\
        $10^{-3}$ & \textbf{0.119 $\pm$ 0.016} & \textbf{0.228 $\pm$ 0.048}\\
        $10^{-4}$ & 0.151 $\pm$ 0.040 & 0.237 $\pm$ 0.067\\
        $10^{-5}$ & 0.212 $\pm$ 0.040 & 0.320 $\pm$ 0.067\\
        \hline
    \end{tabular}
\end{table}

Observing that the average test loss values of learning rate = $10^{-3}$ is the lowest, we compare its results with the previous study's 2-layered network, historical data \cite[Table~1]{Xue2016Avi2}, and the 4-layered network with learning rate = $10^{-3}$ (see \Cref{sec:Identification Problem-Optimizing the Original Dataset}). As depicted in \Cref{fig:Comparison of Loss Values from Different Models of the Identification Problem}, our 3-layered neural network outperformed the 2-layered model by \textbf{0.14~units}~in~loss~value, and also produced better results (0.12~units less) than the 4-layered model.

We also generated the predicted probabilities of the agents visiting each location ($\matr{P}\matr{x}$) during time units in the Test Set, and plotted it onto maps marked by the locations' latitudes and longitudes. \Cref{fig:Predicted Probabilities of Agents Visiting Each Location Plotted on a Map} shows such a plot generated by the 3-layered network, where each dot represents a location.
\begin{figure}[!htbp]
    \centering
    \begin{subfigure}{.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{weights_train_test_loss_3_plot}
        \caption{Plot for 3-layered Model}
        \label{fig:Plot for 3-layered Model}
    \end{subfigure}
    \begin{subfigure}{.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{weights_train_test_loss_4_plot}
        \caption{Plot for 4-layered Model // todo}
        \label{fig:Plot for 4-layered Model}
    \end{subfigure}
    \caption[Train and Test Loss Values' Plots of Different Models]{Train and Test Loss Values' Plots of Different Models: Both networks learn the set of weights quickly, as displayed in the steep descent in loss values before $\approx 1000$ epochs. This quick learning is due to the choice of \textsc{Gradient-Descent}($\cdot$) function --- Adam's algorithm \cite{Adam}. Other algorithms like SGD \cite{SGD} and Adagrad \cite{Adagrad} learn relatively slowly.}
    \label{fig:Train & Test Loss Values' Plots of Different Models}
\end{figure}

To check if the model was overfitting at learning rate = $10^{-3}$, we plotted loss values at the end of each epoch for all tests. Although there remained a difference of 0.09 loss units in the values of training and testing set, the 3-layered model was not drastically overfitting as an average \textit{end} difference of $0.0876 \pm 0.0159$ loss units persisted for many epochs, instead of increasing and tuning more to the training set.

Even though one would expect overfitting with more tuned parameters, the 4-layered model also  produced an average \textit{end} difference of $0.01677 \pm 0.0473$ loss units, which persisted during the run. \Cref{fig:Train & Test Loss Values' Plots of Different Models} shows the results of a randomly selected experiment with plots of loss values at each epoch for the both networks.

\subsection{GPU Speedup Results} \label{sec:IdProbRes - GPU}
Running on batches of sizes $J = \{11,37,63,90,116,145,174,203,232\}$ on a randomly generated dataset for 1000 epochs with both GPU and CPU ``set'' separately, we obtained information on full execution runtimes (combining training and testing runtimes). The average results (over 3 seeds for each batch) are plotted in \Cref{fig:Execution Times of Different Batch-Sizes with GPU and CPU ``set'' Separately}, which show promising GPU Speedup figures for bigger dataset batches; the GPU Speedup varied \textbf{from~0.94~to~10.32} on all tested batch-sizes.
\begin{figure}[!htbp]
    \centering
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        width=.9\textwidth,
        height=8cm,
        xtick=data,
        xlabel={Batch-Size $J$},
        ylabel={Time/s},
        enlarge x limits=0.05,
        enlarge y limits=0.15,
        y tick label style={/pgf/number format/1000 sep=},
        extra y tick style={grid=major, tick label style={xshift=-1cm}},
        legend style={at={(0.02,.98)},
            anchor=north west},
        nodes near coords,
        every node near coord/.append style={yshift=-0.7cm,/pgf/number format/.cd,fixed zerofill,precision=2},
        ]
        \addplot+ [mark=*,
        nodes near coords=\raisebox{0.8cm}{\pgfmathprintnumber\pgfplotspointmeta},error bars/.cd, y dir=both,y explicit] coordinates {
            (11,95.32) +- (1.62,1.62)
            (37,202.10) +- (4.48,4.48)
            (63,391.45) +- (2.82,2.82)
            (90,654.24) +- (0.41,0.41)
            (116,999.53) +- (1.98,1.98)
            (145,1248.27) +- (5.50,5.50)
            (174,1748.61) +- (1.53,1.53)
            (203,2279.90) +- (1.18,1.18)
            (232,2777.15) +- (36.84,36.84)
        };  % cpu
        \addplot+ [mark=*,error bars/.cd, y dir=both,y explicit] coordinates {
            (11,100.77) +- (0.74,0.74)
            (37,101.09) +- (0.38,0.38)
            (63,101.55) +- (0.76,0.76)
            (90,100.62) +- (0.33,0.33)
            (116,109.77) +- (0.07,0.07)
            (145,137.35) +- (0.06,0.06)
            (174,169.53) +- (0.16,0.16)
            (203,224.14) +- (0.14,0.14)
            (232,269.20) +- (0.11,0.11)
        };  % gpu
        \legend{CPU ``set'',GPU ``set''}
        \end{axis}
        \end{tikzpicture}
        \caption{Time Taken by the Full Model: GPUs can sustain calculations without increase in runtime for small datasets (upto $J=116$ in this case). This may be because of underutilization of GPU resources for small datasets (fewer threads needed). On the other hand, full utilization results in a steady increase in runtime, as seen in the upward slope trend after $J=116$.}
    \end{subfigure}\vspace*{1em}
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
            \begin{axis}[
            xbar,
            width=.9\textwidth,
            height=8cm,
            ytick=data,
            ylabel={Batch-Size $J$},
            xlabel={GPU Speedup},
            enlarge y limits=0.15,
            nodes near coords,
            every node near coord/.append style={/pgf/number format/.cd,fixed zerofill,precision=2},
            ]
            \addplot+ coordinates {
                (0.94,11)
                (2.00,37)
                (3.85,63)
                (6.50,90)
                (9.10,116)
                (9.09,145)
                (10.31,174)
                (10.17,203)
                (10.32,232)
            };
            \legend{}
            \end{axis}
        \end{tikzpicture}
        \caption{GPU Speedup for the Identification Problem: We see the GPU performing operations much faster for bigger datasets. The absence of drastic speedup for smaller datasets highlights the inferiority of GPUs for tiny models, which is often caused due to dominating transfer times.}
    \end{subfigure}
    \caption[Finding Weights --- Execution Times of Different Batch-Sizes $J$ with GPU and CPU ``set'' Separately]{Finding Weights --- Execution Times of Different Batch-Sizes $J$ with GPU and CPU ``set'' Separately: The GPU delivers faster computation than CPU even as the datasets' sizes grow --- the speedup is $0.94-10.32$ depending on the tested dataset's size. Also, the error bars are indiscernible because they are too small ($< 1\%$).}
    \label{fig:Execution Times of Different Batch-Sizes with GPU and CPU ``set'' Separately}
\end{figure}

\section{Pricing Problem's Results} \label{sec:Pricing Problem's Results}
\subsection{Optimization Results} \label{sec:PriProbRes - Optimization}
Taking the approach mentioned in \Cref{sec:Pricing Problem-Optimizing the Original Dataset}, we obtained consistent loss values for each set of weights (even with differently seeded rewards). The best performing set of weights was set-2, using which the average loss value for the Pricing Problem hovered around $0.0079\%$. Next, running differently seeded rewards with different learning rates on set-2 of weights, we obtained the lowest loss value of $\textbf{0.0069\%}$.
\begin{table}[!htbp]
    \centering
    \caption[Loss Values Calculated from Different Sets of Rewards]{Loss Values Calculated from Different Sets of Rewards: The values are small because the loss function $Z_P(\vect{r})$ (\Cref{eqn:pricing_problem}) is averaged over the number of locations.}
    \label{tab:Loss Values Calculated from Different Sets of Rewards}
    \begin{tabular}{|c|c|}
        \hline
        \textbf{Rewards Obtained From} & \textbf{Best Loss Values (In \%)}\\
        \hline
        Model's Prediction & 0.0069\\
        Random Initialization & 0.0331\\
        Proportional Distribution & 0.0235\\
        \hline
    \end{tabular}
\end{table}

Compared to the proportional reward distribution (loss values calculated using set-2 of weights), our model's set of rewards produced loss value $\approx$ \textbf{3 times} lower. \Cref{tab:Loss Values Calculated from Different Sets of Rewards} lists the best loss values obtained on each type of reward allocation (model's predicted, random and proportional --- \Cref{sec:Pricing Problem-Optimizing the Original Dataset}).

\subsection{GPU Speedup Results} \label{sec:PriProbRes - GPU}
After running on different batch-sizes $J = \{11, 37, 63, 90, 116, 145, 174, 203, 232\}$, we \textbf{did not observe drastic GPU speedup} for the full model. \Cref{fig:Finding Rewards - Time Taken by the Full Model} shows the Speedup trend: GPU Speedup for the full model was a meager \textbf{1.12 $-$ 1.60}.
\begin{figure}[!htbp]
    \centering
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
            \begin{axis}[
            width=.9\textwidth,
            height=8cm,
            xtick=data,
            xlabel={Batch-Size $J$},
            ylabel={Time/s},
            enlarge y limits=0.15,
            enlarge x limits=0.05,
            y tick label style={/pgf/number format/1000 sep=},
            extra y tick style={grid=major, tick label style={xshift=-1cm}},
            legend style={at={(0.02,.98)},
                anchor=north west},
            nodes near coords,
            every node near coord/.append style={yshift=-0.7cm,/pgf/number format/.cd,fixed zerofill,precision=2},
            ]
            \addplot+ [mark=*,
            nodes near coords=\raisebox{0.8cm}{\pgfmathprintnumber\pgfplotspointmeta},error bars/.cd, y dir=both,y explicit
            ] coordinates {
                (11,9.95) +- (0.12,0.12)
                (37,41.70) +- (0.18,0.18)
                (63,83.69) +- (1.60,1.60)
                (90,144.59) +- (1.37,1.37)
                (116,234.69) +- (2.09,2.09)
                (145,299.91) +- (2.46,2.46)
                (174,390.79) +- (3.49,3.49)
                (203,510.15) +- (4.29,4.29)
                (232,651.64) +- (5.25,5.25)
            };  % cpu
            \addplot+ [mark=*,error bars/.cd, y dir=both,y explicit] coordinates {
                (11,8.79) +- (0.07,0.07)
                (37,29.43) +- (0.28,0.28)
                (63,55.59) +- (0.20,0.20)
                (90,94.10) +- (0.85,0.85)
                (116,146.63) +- (0.42,0.42)
                (145,223.77) +- (2.57,2.57)
                (174,323.67) +- (9.67,9.67)
                (203,432.95) +- (6.76,6.76)
                (232,581.69) +- (15.66,15.66)
            };  % gpu
            \legend{CPU ``set'',GPU ``set''}
            \end{axis}
        \end{tikzpicture}
        \caption{Time Taken by the Full Model: GPU Speedup over all batch-sizes is only \textbf{1.12 $-$ 1.60}.}
        \label{fig:Finding Rewards - Time Taken by the Full Model}
    \end{subfigure}\vspace*{1em}
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
            \begin{axis}[
            width=.9\textwidth,
            height=8cm,
            xtick=data,
            enlarge y limits=0.15,
            enlarge x limits=0.05,
            xlabel={Batch-Size $J$},
            ylabel={Time/s},
            y tick label style={/pgf/number format/1000 sep=},
            legend style={at={(0.02,.98)},
                anchor=north west},
            nodes near coords,
            every node near coord/.append style={yshift=-0.7cm,/pgf/number format/.cd,fixed zerofill,precision=2},
            ]
            \addplot+ [mark=*,
            nodes near coords=\raisebox{0.8cm}{\pgfmathprintnumber\pgfplotspointmeta},error bars/.cd, y dir=both,y explicit
            ] coordinates {
                (11,9.14) +- (0.03,0.03)
                (37,40.15) +- (0.18,0.18)
                (63,81.03) +- (1.56,1.56)
                (90,140.40) +- (1.37,1.37)
                (116,228.46) +- (2.14,2.14)
                (145,289.98) +- (2.38,2.38)
                (174,379.11) +- (3.44,3.44)
                (203,494.61) +- (4.16,4.16)
                (232,631.07) +- (5.32,5.32)
            };  % cpu
            \addplot+ [mark=*,error bars/.cd, y dir=both,y explicit] coordinates {
                (11,6.50) +- (0.07,0.07)
                (37,26.82) +- (0.28,0.28)
                (63,52.63) +- (0.20,0.20)
                (90,90.71) +- (0.80,0.80)
                (116,142.91) +- (0.42,0.42)
                (145,219.35) +- (2.72,2.72)
                (174,318.93) +- (9.69,9.69)
                (203,427.58) +- (6.75,6.75)
                (232,574.45) +- (16.02,16.02)
            };  % gpu
            \legend{CPU ``set'',GPU ``set''}
            \end{axis}
        \end{tikzpicture}
        \caption[Time taken by the LP]{Time taken by the LP: GPU Speedup over all batch-sizes varies \textbf{1.10 $-$ 1.60}. As discussed in \Cref{sec:PriProbRes - GPU,app:GPU Speedup in LP Computation}, we shouldn't have witnessed this speedup as both configurations' LPs were computed on the CPU. Hence, we expected the GPU Speedup value for LP to be $\approx$ 1. }
        \label{fig:Finding Rewards - Time taken by the LP}
    \end{subfigure}
    \caption{Finding Rewards --- Execution Times of Different Batch-Sizes $J$ with GPU and CPU ``set'' Separately}
    \label{fig:Finding Rewards - Execution Times of Different Batch-Sizes J with GPU and CPU ``set'' Separately}
\end{figure}

As the the low GPU Speedup was disappointing, we looked for operations that were causing the program to slow down on the GPU. Since the 2-layered network was quite small, we suspected the LP problem (\Cref{eqn:lp_math_constrain_rewards,eqn:lp_code_constrain_rewards}) to influence the runtimes. Thus, we recorded execution times for both the neural network and the LP separately. As we suspected, the LP did impact the runtime more than the neural networks did, and accounted for $\approx 94\%$ of the total model runtime (\Cref{fig:Finding Rewards - Execution Times of Different Batch-Sizes J with GPU and CPU ``set'' Separately}). However, the GPU Speedup in LP runtime was unusual.

\subsubsection{Strange GPU Speedup in LP Computation}
The Speedup is exceptional as we intentionally transferred the needed matrices/tensors to the CPU for solving the LP using Simplex. Since SciPy's Optimize Module uses a single CPU core and does not utilize the GPU, we expected similar runtimes for both configurations. Our efforts to determine the reasons for this speedup are collected in \Cref{app:GPU Speedup in LP Computation} instead of digressing here. As stated later in this section, we disregard the old GPU Speedup results for finding rewards and use the more suited ones.\\

Since \Cref{alg:Solving the Pricing Problem} performs operations sequentially (Calculate loss $\rightarrow$ Gradient-Descent and Update $\vect{r}$ $\rightarrow$ Constrain using LP), we suggest these possibilities:
\begin{enumerate}
    \item The 2-layered network is parallelized but \texttt{scipy.optimize}'s LP solver is not. Since both sub-problems are solved by independent frameworks, we could not thread the whole algorithm without implementing OpenMP \cite{OpenMP} back-end manually and allowing multiple copies of the script to run simultaneously. Currently, the LP problem acts as a shared resource and requires all neural network's threads to synchronize. This might cause time lag as the LP waits for all elements/matrices to become fully available.
    \item The simultaneously threads downgrade the LP solver's performance. We do not endorse this possibility because the CPU resources are independent when using multi-threading over CPU's cores. We checked if the same script running simultaneously on another core hampered the performance of the original copy, but did not witness any reduction.
\end{enumerate}

We found that the LP in CPU ``set'' was slower than normal because of latency in thread synchronization\footnote{Pytorch uses OpenMP \cite{PTDocs,OpenMP} when the CPU is ``set'', which threads the 2-layered network.} (\Cref{app:Parallelism Doesn't Always Help}). While one might contend the lack of this behavior in GPU ``set'', we reiterate that we intentionally impose a synchronization barrier when transferring the matrices to the CPU after the network has been executed, adding the time elapsed in the network's time logs. This is not case with CPU ``set'' because the network remains in execution in other cores while the LP has started. Therefore, when we restrict the script's access to a single CPU core, we do not witness any GPU Speedup in LP runtime. This only affects CPU ``set'' performance because the GPU ``set'' configuration independently executes the network on the GPU.
\begin{figure}[!htbp]
    \centering
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        width=.9\textwidth,
        height=8cm,
        xtick=data,
        xlabel={Batch-Size $J$},
        ylabel={Time/s},
        enlarge y limits=0.15,
        enlarge x limits=0.05,
        y tick label style={/pgf/number format/1000 sep=},
        extra y tick style={grid=major, tick label style={xshift=-1cm}},
        legend style={at={(0.02,.98)},
            anchor=north west},
        nodes near coords,
        every node near coord/.append style={yshift=-0.7cm,/pgf/number format/.cd,fixed zerofill,precision=2},
        ]
        \addplot+ [mark=*,
        nodes near coords=\raisebox{0.8cm}{\pgfmathprintnumber\pgfplotspointmeta},error bars/.cd, y dir=both,y explicit
        ] coordinates {
            (11,7.03) +- (0.03,0.03)
            (37,28.25) +- (0.28,0.28)
            (63,54.18) +- (0.29,0.29)
            (90,93.34) +- (0.39,0.39)
            (116,147.15) +- (0.42,0.42)
            (145,225.92) +- (1.74,1.74)
            (174,322.61) +- (4.21,4.21)
            (203,438.17) +- (1.56,1.56)
            (232,583.95) +- (5.08,5.08)
        };  % cpu
        \addplot+ [mark=*,error bars/.cd, y dir=both,y explicit] coordinates {
            (11,8.52) +- (0.02,0.02)
            (37,29.64) +- (0.16,0.16)
            (63,55.25) +- (0.05,0.05)
            (90,94.02) +- (0.87,0.87)
            (116,146.52) +- (0.50,0.50)
            (145,218.42) +- (0.81,0.81)
            (174,317.33) +- (3.98,3.98)
            (203,430.61) +- (2.55,2.55)
            (232,573.49) +- (4.69,4.69)
        };  % gpu
        \legend{CPU ``set'',GPU ``set''}
        \end{axis}
        \end{tikzpicture}
        \caption{Time Taken by the Full Model: GPU Speedup over all batch-sizes is only \textbf{0.82 $-$ 1.03} --- heavily impacted by LP runtimes.}
        \label{fig:Restricted Finding Rewards - Time Taken by the Full Model}
    \end{subfigure}\vspace*{1em}
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        width=.9\textwidth,
        height=8cm,
        xtick=data,
        enlarge y limits=0.15,
        enlarge x limits=0.05,
        xlabel={Batch-Size $J$},
        ylabel={Time/s},
        y tick label style={/pgf/number format/1000 sep=},
        legend style={at={(0.02,.98)},
            anchor=north west},
        nodes near coords,
        every node near coord/.append style={yshift=-0.7cm,/pgf/number format/.cd,fixed zerofill,precision=2}
        ]
        \addplot+ [mark=*,nodes near coords=\raisebox{0.8cm}{\pgfmathprintnumber\pgfplotspointmeta},error bars/.cd, y dir=both,y explicit
        ] coordinates {
            (11,6.42) +- (0.03,0.03)
            (37,27.16) +- (0.28,0.28)
            (63,52.34) +- (0.28,0.28)
            (90,90.18) +- (0.35,0.35)
            (116,142.46) +- (0.42,0.42)
            (145,218.63) +- (1.72,1.72)
            (174,313.16) +- (4.22,4.22)
            (203,425.66) +- (1.56,1.56)
            (232,567.86) +- (5.10,5.10)
        };  % cpu
        \addplot+ [mark=*,error bars/.cd, y dir=both,y explicit] coordinates {
            (11,6.37) +- (0.01,0.01)
            (37,27.15) +- (0.16,0.16)
            (63,52.42) +- (0.06,0.06)
            (90,90.81) +- (0.87,0.87)
            (116,142.90) +- (0.51,0.51)
            (145,214.30) +- (0.81,0.81)
            (174,312.71) +- (3.98,3.98)
            (203,425.38) +- (2.58,2.58)
            (232,567.77) +- (4.71,4.71)
        };  % gpu
        \legend{CPU ``set'',GPU ``set''}
        \end{axis}
        \end{tikzpicture}
        \caption[Time taken by the LP]{Time taken by the LP: GPU Speedup over all batch-sizes is $\approx$ \textbf{1}. Expectedly, there is no GPU Speedup in LP runtimes.}
        \label{fig:Restricted Finding Rewards - Time taken by the LP}
    \end{subfigure}
\end{figure}

\begin{figure}[!htbp]
    \centering
    \ContinuedFloat
    \begin{subfigure}{\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        width=.9\textwidth,
        height=9cm,
        xtick=data,
        enlarge y limits=0.15,
        enlarge x limits=0.05,
        xlabel={Batch-Size $J$},
        ylabel={Time/s},
        legend style={at={(0.02,.98)},
            anchor=north west},
        nodes near coords,
        every node near coord/.append style={yshift=-1.25cm,/pgf/number format/.cd,fixed zerofill,precision=2}
        ]
        \addplot+ [mark=*,
        nodes near coords=\raisebox{1.8cm}{\pgfmathprintnumber\pgfplotspointmeta},error bars/.cd, y dir=both,y explicit] coordinates {
            (11,0.61) +- (0.01,0.01)
            (37,1.09) +- (0.003,0.003)
            (63,1.84) +- (0.01,0.01)
            (90,3.16) +- (0.04,0.04)
            (116,4.69) +- (0.004,0.004)
            (145,7.28) +- (0.02,0.02)
            (174,9.45) +- (0.02,0.02)
            (203,12.51) +- (0.06,0.06)
            (232,16.08) +- (0.04,0.04)
        };  % cpu
        \addplot+ [mark=*, error bars/.cd, y dir=both,y explicit] coordinates {
            (11,2.15) +- (0.01,0.01)
            (37,2.49) +- (0.006,0.006)
            (63,2.84) +- (0.02,0.02)
            (90,3.21) +- (0.01,0.01)
            (116,3.62) +- (0.005,0.005)
            (145,4.11) +- (0.02,0.02)
            (174,4.62) +- (0.02,0.02)
            (203,5.23) +- (0.05,0.05)
            (232,5.72) +- (0.04,0.04)
        };  % gpu
        \legend{CPU ``set'',GPU ``set''}
        \end{axis}
        \end{tikzpicture}
        \caption{Time taken by the Neural Network: GPU Speedup over all tested batch-sizes varies between \textbf{0.28 $-$ 2.81} --- as transfer time between RAM and GPU's internal memory dominates for smaller datasets. However, as batch-size increases, computation time dominates over transfer time --- GPU ``set'' performs better than CPU ``set''.}
        \label{fig:Restricted Finding Rewards - Time taken by the Neural Network}
    \end{subfigure}\vspace*{1em}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        xbar,
        width=0.9\textwidth,
        height=8cm,
        ytick=data,
        ylabel={Batch-Size $T$},
        xlabel={GPU Speedup},
        ytick align=inside,
        xmin=0,
        xmax=1.5,
        enlarge y limits=0.15,
        nodes near coords,
        every node near coord/.append style={/pgf/number format/.cd,fixed zerofill,precision=2},
        ]
        \addplot+ coordinates {
            (0.82,11)
            (0.95,37)
            (0.98,63)
            (0.99,90)
            (1.00,116)
            (1.03,145)
            (1.02,174)
            (1.02,203)
            (1.02,232)
        };
        \end{axis}
        \end{tikzpicture}
        \caption{Speedup for the Full Model}
        \label{fig:Restricted Finding Rewards - Speedup for the Full Model}
    \end{subfigure}\hfill
    \begin{subfigure}{0.3\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        xbar,
        width=\textwidth,
        height=8cm,
        ytick=data,
        yticklabels={,,},
        xlabel={GPU Speedup},
        ytick align=inside,
        xmin=0,
        xmax=1.5,
        enlarge y limits=0.15,
        nodes near coords,
        every node near coord/.append style={/pgf/number format/.cd,fixed zerofill,precision=2},
        ]
        \addplot+ coordinates {
            (1.01,11)
            (1.00,37)
            (1.00,63)
            (0.99,90)
            (1.00,116)
            (1.02,145)
            (1.00,174)
            (1.00,203)
            (1.00,232)
        };
        \end{axis}
        \end{tikzpicture}
        \caption{Speedup for the LP}
        \label{fig:Restricted Finding Rewards - Speedup for LP}
    \end{subfigure}\hfill
    \begin{subfigure}{0.37\textwidth}
        \centering
        \begin{tikzpicture}
        \begin{axis}[
        xbar,
        width=\textwidth,
        height=8cm,
        ytick=data,
        yticklabels={,,},
        xlabel={GPU Speedup},
        ytick align=inside,
        xmin=0,
        xmax=3.75,
        enlarge y limits=0.15,
        nodes near coords,
        every node near coord/.append style={/pgf/number format/.cd,fixed zerofill,precision=2},
        ]
        \addplot+ coordinates {
            (0.28,11)
            (0.44,37)
            (0.65,63)
            (0.98,90)
            (1.30,116)
            (1.77,145)
            (2.04,174)
            (2.39,203)
            (2.81,232)
        };
        \end{axis}
        \end{tikzpicture}
        \caption{Speedup for the Neural Network}
        \label{fig:Restricted Finding Rewards - Speedup for the Neural Network}
    \end{subfigure}
    \caption[Finding Rewards After Restricing Script's Access to CPU Cores --- Execution Times of Different Batch-Sizes $J$ with GPU and CPU ``set'' Separately]{Finding Rewards After Restricing Script's Access to CPU Cores --- Execution Times of Different Batch-Sizes $J$ with GPU and CPU ``set'' Separately: Scaling is strongly hampered by the LP solver. Comparing the contributions of LP and Neural Network to total runtime on GPU ``set'', LP accounts for \textbf{94.42 $\pm$ 5.00}\% of the total time.}
    \label{fig:Restricted Finding Rewards - Execution Times of Different Batch-Sizes J with GPU and CPUa ``set'' Separately}
\end{figure}

\subsubsection{Final Results}
The results after restricting the script's access to a single CPU core are plotted in \Cref{fig:Restricted Finding Rewards - Execution Times of Different Batch-Sizes J with GPU and CPUa ``set'' Separately}. We prefer using the results after restricting access as it gives the better performance for CPU ``set''. The time elapsed for the Neural Network includes time taken to transfer tensors to and from the GPU, which results in overhead --- as seen in higher runtimes for smaller datasets in \Cref{fig:Restricted Finding Rewards - Time taken by the Neural Network,fig:Restricted Finding Rewards - Speedup for the Neural Network}. However, as the batch-size increases, we see computation time dominating over the transfer time, resulting in higher CPU ``set'' runtimes.

Moreover, the LP subproblem highly impacts the full model, accounting for more than $94\%$ of the total runtime. Therefore, even though the Neural Network gets sped up at bigger datasets, its GPU Speedup is not reflected in the full model.


\documentclass[a4paper]{article}

\def\npart{II}

\def\ntitle{Automata and Formal Languages}
\def\nlecturer{M.\ Chiodo}

\def\nterm{Michaelmas}
\def\nyear{2017}

\input{header}

\begin{document}

\input{titlepage}

\begin{theorem}\leavevmode
  \begin{enumerate}
\item Let \(I \subseteq \N\) be a (possibly infinite) r.e. set and \(I' =C(I) \subseteq I\) then \(\bigcup_{n\in I}w_n\) is r.e.
\item Let \(j \subset \N\) be finite and \(J' = C(J) \subset J\) then \(\bigcap_{n\in J'}w_n\) is r.e.
  \end{enumerate}
\end{theorem}

In both cases, we can construct indices for the r.e. set so de

\begin{proof}\leavevmode
  \begin{enumerate}
  \item Set \(A = \bigcup_{n\in T'}w_n\). For any \(x\in \N\), start a diag process to compute if \emph{any} \(f_{n,1}(x)\). This terminates if and only if \(x\in w_n\) for some \(n\in I'\), if and only if \(x \in A\).

  This describes an algorithsm to enumerate \(A\), i.e. to compute \(\phi_A\). By Church (2), \(\phi_A\) is PR, so \(A\) is r.e. Moreover, by Church (3), we can compute \(N\) with \(f_{N,1} = \phi_A\), i.e. \(w_N = A\).
\item Set \(B = \cap w_n\). For \(x\in w_1\), start a diag process to compute if ACC \(f_{n,1}(x)\) down (over all \(n\in J')\). This terminates if and only if \(x\in w_n\) for all \(n\in J'\), if and only if \(x\in A\).
  This desribes an algorithm to compute \(\phi_B\). By Church, \(\phi_B\) is PR so \(B\) is r.e.
  \end{enumerate}
\end{proof}

\subsection{Universality \& Undecidability}

In 1936 Turing defined his version of an abstract theory of finite computation, now known as Turing machines. In the same paper he explored the idea of ``universal computing'' and ``undecidabilities''.

\begin{theorem}
  There exists a PR function \(\N^3\to \N\) such that if \(n\) codes a RM and \(m\) codes a \(k\)-tuple and \(f_{n,k}((m)_1,\dots,(m)_k) down = r\)
  such a PR function is said to be \emph{universal}, as it can simulate \emph{any} RM and \emph{any} input.
\end{theorem}

\begin{proof}
  Here is a description of an algorithm to compute \(u\):
  \begin{enumerate}
  \item on input \((n, k, m)\), check if \(n\) is a code (if not, infinite loop).
  \item decode \(m\) as a \(k\)-tuple \(((m)_1, \dots, (m)_k)\) (if \(m=0\) or exists \(p>k\) such that \(p|m\), infinite loop).
  \item Then \(P_n\) as an input
  \end{enumerate}
\end{proof}
\end{document}


\chapter{Experiments and Results}

The main aim of this section is to assess the use of static features when involved in the characterization of Android malware and benignware samples used to train detection tools. Thus, through
the use of several machine learning classifiers, it is studied if this combination with stacking methods is feasible and, if
so, to assess its performance when applied to build a machine learning aided Android malware
detection tools.

\section{1st Stage training experiments} \label{1ststagetrain}

For the 1st stage training step of our model, the selected ensemble classifiers were run with the Scikit-learn library for Python\cite{scikitlearn}.
They include AdaBoost, XGBoost, Catboost, Bagging Classifier, K-nearest neighbors, Neural network, Random Forest.
All these ensemble classifiers were tested with the previously mentioned dataset through a 10-folds cross validation with the accuracy scoring method shown below.

$$Accuracy = (TP+TN)/(TP+TN+FP+FN)$$
$$where: TP = \text{True positive}; FP = \text{False positive};$$$$TN = \text{True negative}; FN = \text{False negative}$$

For a better evaluation of the capacity of these algorithms to extract
and generalize conclusions, several experiments were conducted to judge the individual performance of all models.
The results are shown in Table \ref{table:1ststage} and Figure \ref{fig:1ststage}

\begin{table}[htbp]
    \centering
    \caption{Experiments results on 1st training stage}
    \label{table:1ststage}
    \begin{adjustbox}{width=\textwidth}

    \begin{tabular}{|cccccccc|}
        \hline

        \textbf{Model} & \textbf{Activities CV} & \textbf{API calls CV} & \textbf{Opcodes CV} & \textbf{Permissions CV} & \textbf{Receivers CV} & \textbf{Services CV} & \textbf{System CV} \\ \hline
        LightGtextbf & 0.5094 & 0.8783 & 0.8519 & 0.7979 & 0.8486 & 0.5094 & 0.7866 \\ \hline
        Bagging & 0.5108 & \textbf{0.8817} & 0.8544 & 0.8094 & 0.8577 & 0.5099 & 0.7986 \\ \hline
        CatBoost & 0.5103 & 0.8762 & 0.8499 & 0.8048 & 0.8525 & 0.5031 & 0.7912 \\ \hline
        Neural Network & 0.5124 & 0.8518 & 0.8189 & 0.7841 & 0.8429 & 0.5092 & 0.7692 \\ \hline
        XGBoost & 0.5099 & 0.8740 & 0.8469 & 0.8008 & 0.8388 & 0.5094 & 0.7809 \\ \hline
        Random Forest & 0.5098 & 0.8238 & 0.8348 & 0.6143 & 0.7946 & 0.5064 & 0.7526 \\ \hline
        K-Neighbors & 0.5050 & 0.8466 & 0.8279 & 0.7817 & 0.8440 & 0.5116 & 0.7743 \\ \hline

        \end{tabular}

    \end{adjustbox}
    \end{table}

    \begin{figure}[htbp]
        \centering
        \includegraphics[width=\textwidth]{./Figure/1ststage.png}
        \caption{1st Stage models CV accuracy score average}
        \label{fig:1ststage}
      \end{figure}

 In particular, API
calls compose the most appropriate representation to build a machine learning classification
tool, exhibiting a 88.17\verb+%+ accuracy with a Bagging classifier. In general,
this algorithm obtains the best results overall.
% It is remarkable the fact that a combination of API calls with other also competitive features,
% such as opcodes or receivers, which have proved to be powerful at building detection mechanisms, does not lead to better values.

It is remarkable the fact that not only API calls have good results but also Opcodes and Receivers have proved to be powerful at building detection mechanisms as we can see in Figure \ref{fig:1ststage}.

Other conclusions can be made after the analysis of these results. In the case of services and activities, it can be seen how these features do not allow to discern the nature of the
sample, since they provide very high-level information unable to describe specific behaviors.

\section{2nd Stage training experiments}

In the 2nd stage for training, we used the predictions results of the 1st stage as input to training our 4 selected classifiers. Due to we have only 49 inputs to each model, we selected 4 algorithms that are good to handle with small datasets. It is Neural Network, Logistic Regression Classifier, Linear SVC, Random Forest.
All these classifiers were tested with the previously mentioned dataset through a 10-folds cross validation with accuracy scoring method as section \ref{1ststagetrain}.
The results are shown in the Table \ref{table:2ndstage} and Figure \ref{fig:2ndstage}.

\begin{table}[htbp]
    \centering
    \caption{Experiments results on 2nd training stage}
    \label{table:2ndstage}

        \begin{tabular}{|cc|}
            \hline

            \textbf{Model} & \textbf{CV score} \\ \hline
            \textbf{LinearSVC} & \textbf{0.9356} \\ \hline
            \textbf{RandomForestClassifier} & \textbf{0.9339} \\ \hline
            \textbf{Neural Network} & \textbf{0.9334} \\ \hline
            \textbf{LogisticRegression} & \textbf{0.9312} \\ \hline
            1st Stage best model & 0.8817 \\ \hline

            \end{tabular}
    \end{table}

    \begin{figure}[htbp]
        \centering
        \includegraphics[scale=0.7]{./Figure/2ndstage.png}
        \caption{2nd Stage models CV accuracy score}
        \label{fig:2ndstage}
      \end{figure}

From these results from the 2nd stage, we could see that our all models score are better than the best score from the 1st stage. Moreover, comparing to the best model at this stage we have a difference of about 5.39\verb+%+ from the 1st stage best model. This fact could prove that our Stacking method is a useful and very powerful method to improve the performance of malware detection systems or models.

\section{3rd Stage training experiments}

In the 3rd stage, we use the 4 outputs from the 2nd stage to do our final prediction.
Also, we tested the same algorithms of the 2nd stage to find out the best model. The validation method and dataset are the same as the previous stages.
The results are shown in Table \ref{table:3rdstage} and Figure \ref{fig:3rdstage}.

\begin{table}[htbp]
    \centering
    \caption{Experiments results on 3rd training stage}
    \label{table:3rdstage}

        \begin{tabular}{|cc|}
            \hline

            Model & CV score \\ \hline
            LinearSVC & 0.9213 \\ \hline
            RandomForestClassifier & 0.9209 \\ \hline
            Neural Network & 0.9205 \\ \hline
            LogisticRegression & 0.9203 \\ \hline
            \textbf{1st Stage best model} & \textbf{0.9356} \\ \hline

            \end{tabular}
    \end{table}

    \begin{figure}[htbp]
        \centering
        \includegraphics[scale=0.6]{./Figure/3rdstage.png}
        \caption{3rd Stage models CV accuracy score}
        \label{fig:3rdstage}
      \end{figure}

From these results, we can see that the 3rd stage models could not improve the accuracy score from the 2nd stage best model score. And the difference in the score of our 3rd stage models is only 0.1\verb+%+. It may happen because the size of our input for the 3rd stage was very small, only 4 inputs. So we have a conclusion that stacking many steps in small data may cause poor performance in classification models.

\section{Comparing to other techniques}

In this section, we compare the results of our experiments to other static analysis techniques to detect Android malware. We can see in Table \ref{table:comparison} related works results with different datasets.

\begin{table}[htbp]
    \centering
    \caption{Comparison of android malware static analysis techiniques}
    \label{table:comparison}
    \begin{adjustbox}{width=\textwidth}
    \begin{tabular}{|l|m{40mm}|m{40mm}|l|}
        \hline

        Paper & Input & Dataset & Accuracy score \\ \hline
        Samaneh et al. \cite{samaneh}& Intents, Permissions & 114 apps (Malware 57, Benign-ware 57) & 98\% \\ \hline
        Manzhi et al.\cite{manzhi} & Resource file, Manifest file, executable class dex files. & 1,100 apps (Malware 550, Benign-ware 550) & 98.1\% \\ \hline
        Zi Wang et al.\cite{ziwang} & Permissions, API function calls & 6334 apps (Malware 4000, Benign-ware 2334) & 93\% \\ \hline
        Zarni et al.\cite{zarni} & Permissions & 500 apps & Around 91\% \\ \hline
        Xiong et al.\cite{xiong} & Permissions & 540 apps (Malware 298, Benign-ware 342) & More than 90\% \\ \hline
        \textbf{Our proposed system} & \textbf{APK}(Activities, API function calls, Opcodes, Permissions, Receivers, Services, System) & \textbf{20,000 apps (Malware 10,000, Benign-ware 10,000)} & \textbf{94\%} \\ \hline

        \end{tabular}
    \end{adjustbox}
    \end{table}


\documentclass[]{report}
\usepackage[letterpaper,left=1in,right=1in,top=1.5in,headheight=1in,headsep=.3in]{geometry}
\usepackage{amsmath}
\title{TestTitle}
\date{15/03/2021}
\begin{document}
\chapter{TestChapter}
\section{TestSection}
\chapter{EmptyChapter}
\chapter{NestedChapter}
\section{TopSection}
\subsection{TestSubSection}
\subsubsection{TestSubSubSection}
\begin{equation}
    \frac{\text{\textbf{Test} rm} + 2}{x^2}
    \frac{x^2}{x+1}
\end{equation}
\subsubsection{SecondSubSub}
\section{Section With spaces}
\end{document}


% $Id: SolverUtilities.tex,v 1.1 2008/10/09 16:28:29 dconway Exp $
\chapter{\label{chapter:SolverUtil}SolverUtilities}
\chapauthor{Darrel J. Conway}{Thinking Systems, Inc.}

GMAT's Solver classes may require one or more of several routine utility functions in order to
proceed towards the dictated solution.  These functions include calculation of gradients and
Jacobians, Hessians, and systematic searches in a specified direction (aka ``Line searches'') for
minima, and other routines common to the solver algorithms.  For each of these utilities, users can
provide their own implementation -- analytic or numerical -- or use an algorithm implemented in the
class.

This chapter describes the interfaces for these solver utilities and the implementation of the
algorithms internal to GMAT.  It also provides information about how user specified implementations
are used in GMAT.

\section{Overview}

\section{Gradient Calculations}

\section{Jacobian Calculations}

\section{Hessian Calculations}

\section{Line Searches}


\chapter{Horizontally and virtically implicit}
\label{chap:hivi}
{\bf \Large
\begin{tabular}{ccc}
\hline
  Corresponding author & : & Seiya Nishizawa\\
\hline
\end{tabular}
}

\section{Equations}
The governing equation is the followings:
\begin{align}
  \frac{\partial G^{\frac{1}{2}}\rho'}{\partial t} &=
  -\frac{\partial G^{\frac{1}{2}}\rho u}{\partial x^*}
  -\frac{\partial G^{\frac{1}{2}}\rho v}{\partial y^*}
  -\frac{\partial J_{33}G^{\frac{1}{2}}\rho w}{\partial \xi}
  + G^{\frac{1}{2}}S_\rho, \\
  \frac{\partial G^{\frac{1}{2}}\rho u}{\partial t}
  &= -\frac{\partial G^{\frac{1}{2}}p'}{\partial x^*} + G^{\frac{1}{2}}S_{\rho u}, \\
  \frac{\partial G^{\frac{1}{2}}\rho v}{\partial t}
  &= -\frac{\partial G^{\frac{1}{2}}p'}{\partial y^*} + G^{\frac{1}{2}}S_{\rho v}, \\
  \frac{\partial G^{\frac{1}{2}}\rho w}{\partial t}
  &= -\frac{\partial J_{33}G^{\frac{1}{2}}p'}{\partial \xi} -G^{\frac{1}{2}}\rho' g + G^{\frac{1}{2}}S_{\rho w}, \\
  \frac{\partial G^{\frac{1}{2}}\rho\theta}{\partial t} &=
  -\frac{\partial G^{\frac{1}{2}}u \rho \theta}{\partial x^*}
  -\frac{\partial G^{\frac{1}{2}}v \rho \theta}{\partial y^*}
  -\frac{\partial J_{33}G^{\frac{1}{2}}w \rho \theta}{\partial \xi}
  + G^{\frac{1}{2}}S_{\rho\theta}, \\
  p &= P_{00}\left(\frac{R\rho\theta}{P_{00}}\right)^{c_p/c_v},
\end{align}
where
\begin{align}
  G^{\frac{1}{2}}S_\rho &=
  - \frac{\partial J_{13}G^{\frac{1}{2}}\rho u + J_{23}G^{\frac{1}{2}}\rho v}{\partial \xi}, \\
  G^{\frac{1}{2}}S_{\rho u}
  &= - \frac{\partial G^{\frac{1}{2}}u\rho u}{\partial x^*}
     - \frac{\partial G^{\frac{1}{2}}v\rho u}{\partial y^*}
     - \frac{\partial}{\partial \xi}(J_{13}G^{\frac{1}{2}}u\rho u + J_{23}G^{\frac{1}{2}}v\rho u + J_{33}G^{\frac{1}{2}}w\rho u) \nonumber\\&
     - \frac{\partial}{\partial \xi}(J_{13}G^{\frac{1}{2}}p'), \\
  G^{\frac{1}{2}}S_{\rho v}
  &= - \frac{\partial G^{\frac{1}{2}}u\rho v}{\partial x^*}
     - \frac{\partial G^{\frac{1}{2}}v\rho v}{\partial y^*}
     - \frac{\partial}{\partial \xi}(J_{13}G^{\frac{1}{2}}u\rho v + J_{23}G^{\frac{1}{2}}v\rho v + J_{33}G^{\frac{1}{2}}w\rho v), \nonumber\\&
     - \frac{\partial}{\partial \xi}(J_{23}G^{\frac{1}{2}}p'), \\
  G^{\frac{1}{2}}S_{\rho w}
  &= - \frac{\partial G^{\frac{1}{2}}u\rho w}{\partial x^*}
     - \frac{\partial G^{\frac{1}{2}}v\rho w}{\partial y^*}
     - \frac{\partial}{\partial \xi}(J_{13}G^{\frac{1}{2}}u\rho w + J_{23}G^{\frac{1}{2}}v\rho w + J_{33}G^{\frac{1}{2}}w\rho w), \\
  G^{\frac{1}{2}}S_{\rho\theta} &=
  - \frac{\partial J_{13}G^{\frac{1}{2}}u\rho\theta+J_{23}G^{\frac{1}{2}}v\rho\theta}{\partial \xi}.
\end{align}
Prime describes deviation from a reference state, and the reference state depends only $z$ and satisfies in hydrostatic barance:
\begin{align}
  p' &= p - \bar{p}, \\
  \rho' &= \rho - \bar{\rho}, \\
  \frac{d \bar{p}(z)}{d z} &= -\bar{\rho}(z) g.
\end{align}

\section{Descritization}

For the temporal discritization, backward temporal integrations are employed for the terms related to acoustic wave.
\begin{align}
  \frac{\rho'^{n+1}-\rho'^n}{\Delta t} &=
  -G^{-\frac{1}{2}}\frac{\partial}{\partial x^*}\{G^{\frac{1}{2}}(\rho u)^{n+1}\}
  -G^{-\frac{1}{2}}\frac{\partial}{\partial y^*}\{G^{\frac{1}{2}}(\rho v)^{n+1}\} \nonumber\\ & \;\;\;
  -G^{-\frac{1}{2}}\frac{\partial}{\partial \xi}\{J_{33}G^{\frac{1}{2}}(\rho w)^{n+1}\}
  + S_\rho^n, \label{eq: hivi rho} \\
  \frac{(\rho u)^{n+1}-(\rho u)^n}{\Delta t}
  &= -G^{-\frac{1}{2}}\frac{\partial}{\partial x^*}(G^{\frac{1}{2}}p'^{n+1}) + S_{\rho u}^n, \\
  \frac{(\rho v)^{n+1}-(\rho v)^n}{\Delta t}
  &= -G^{-\frac{1}{2}}\frac{\partial}{\partial y^*}(G^{\frac{1}{2}}p'^{n+1}) + S_{\rho v}^n, \\
  \frac{(\rho w)^{n+1}-(\rho w)^n}{\Delta t}
  &= -G^{-\frac{1}{2}}\frac{\partial}{\partial \xi}(J_{33}G^{\frac{1}{2}}p'^{n+1}) -g\rho'^{n+1} + S_{\rho w}^n, \\
  \frac{p'^{n+1} - p'^n}{\Delta t}
  &= P_{00}\left(\frac{R}{P_{00}}\right)^{\kappa^n} \kappa^n {(\rho\theta)^n}^{\kappa^n-1} \frac{\partial \rho\theta}{\partial t}
   = \kappa^n \frac{p^n}{(\rho\theta)^n}\frac{\partial \rho\theta}{\partial t}, \\
  \frac{\partial \rho\theta}{\partial t} &=
  -G^{-\frac{1}{2}}\frac{\partial}{\partial x^*}\{G^{\frac{1}{2}}\theta^n (\rho u)^{n+1}\}
  -G^{-\frac{1}{2}}\frac{\partial}{\partial y^*}\{G^{\frac{1}{2}}\theta^n (\rho v)^{n+1}\}  \nonumber\\ & \;\;\;
  -G^{-\frac{1}{2}}\frac{\partial}{\partial \xi}\{J_{33}G^{\frac{1}{2}}\theta^n (\rho w)^{n+1}\}
  + S_{\rho\theta}^n,
\end{align}
where $\kappa = c_p/c_v$.
Note that the potential temperature at previous step, $\theta^n$, is used.

In order to obtain Helmholtz equation, a linearlized equation for the density is used instead of Eq. \ref{eq: hivi rho}.
\begin{align}
  \rho'^{n+1} &\sim
  \rho'^n + \frac{1}{\kappa^n} \frac{\rho^n}{p^n} \left\{p'^{n+1}-p'^n\right\}.
\end{align}
Here we assume that the potential temperature does not change during a temporal step due to acoustic wave.

Eliminating $(\rho u)^{n+1}, (\rho v)^{n+1} (\rho w)^{n+1}, $, and $\rho'^{n+1}$, the Helmholtz equation for $p'^{n+1}$ is obtained.
\begin{align}
  & \frac{\partial}{\partial x^*}\left(\theta^n\frac{\partial G^{\frac{1}{2}}p'^{n+1}}{\partial x^*}\right)
  + \frac{\partial}{\partial y^*}\left(\theta^n\frac{\partial G^{\frac{1}{2}}p'^{n+1}}{\partial y^*}\right)
  + \frac{\partial}{\partial \xi}\left(J_{33}\theta^n\frac{\partial J_{33}G^{\frac{1}{2}}p'^{n+1}}{\partial \xi}\right) \nonumber\\&
  + g \frac{\partial}{\partial \xi} \left(\frac{J_{33}G^{\frac{1}{2}}\theta^np'^{n+1}}{C_s^{2n}}\right)
  - \frac{G^{\frac{1}{2}}\theta^np'^{n+1}}{\Delta t^2 C_s^{2n}} \nonumber\\
  &=
  \frac{1}{\Delta t}\left[
      \frac{\partial G^{\frac{1}{2}}\theta^n\left\{(\rho u)^n + \Delta t S_{\rho u}^n\right\}}{\partial x^*}
    + \frac{\partial G^{\frac{1}{2}}\theta^n\left\{(\rho v)^n + \Delta t S_{\rho v}^n\right\}}{\partial y^*}
    + \frac{\partial J_{33}G^{\frac{1}{2}}\theta^n\left\{(\rho w)^n + \Delta t S_{\rho w}^n\right\}}{\partial \xi}
    \right] \nonumber\\&\;\;\;
  + g \frac{\partial}{\partial \xi}\left\{\frac{J_{33}G^{\frac{1}{2}}\theta^np'^n}{C_s^{2n}}-J_{33}G^{\frac{1}{2}}(\rho'\theta)^n\right\}
  + \frac{G^{\frac{1}{2}} S_{\rho\theta}}{\Delta t}
  - \frac{G^{\frac{1}{2}}\theta^np'^n}{\Delta t^2 C_s^{2n}},
\end{align}
where
\begin{align}
  C_s^2 &= \kappa\frac{p}{\rho}.
\end{align}

Spatial differentials are descritized.
\begin{align}
  & \frac{1}{\Delta x_i}\left(
   \hat{\theta}_{i+1/2}\frac{(G^{\frac{1}{2}}p'^{n+1})_{i+1}-G^{\frac{1}{2}}p'^{n+1}}{\Delta x_{i+1/2}}
  -\hat{\theta}_{i-1/2}\frac{G^{\frac{1}{2}}p'^{n+1}-(G^{\frac{1}{2}}p'^{n+1})_{i-1}}{\Delta x_{i-1/2}}
  \right) \nonumber\\&
  + \frac{1}{\Delta y_j}\left(
   \hat{\theta}_{j+1/2}\frac{(G^{\frac{1}{2}}p'^{n+1})_{j+1}-G^{\frac{1}{2}}p'^{n+1}}{\partial y_{j+1/2}}
  -\hat{\theta}_{j-1/2}\frac{G^{\frac{1}{2}}p'^{n+1}-(G^{\frac{1}{2}}p'^{n+1})_{j-1}}{\partial y_{j-1/2}}
 \right) \nonumber\\&
  + \frac{1}{\Delta z_k}\left\{
   (J_{33})_{k+1/2}\hat{\theta}_{k+1/2}\frac{J_{33}G^{\frac{1}{2}}p'^{n+1}_{k+1}-J_{33}G^{\frac{1}{2}}p'^{n+1}}{\Delta z_{k+1/2}}
  -(J_{33})_{k-1/2}\hat{\theta}_{k-1/2}\frac{J_{33}G^{\frac{1}{2}}p'^{n+1}-J_{33}G^{\frac{1}{2}}p'^{n+1}_{k-1}}{\Delta z_{k-1/2}}
  \right) \nonumber\\&
  + g \frac{1}{\Delta_{k+1/2}+\Delta z_{k-1/2}} \left\{
   \frac{J_{33}G^{\frac{1}{2}}\theta p'^{n+1}_{k+1}}{C^2_{s k+1}}
  -\frac{J_{33}G^{\frac{1}{2}}\theta p'^{n+1}_{k-1}}{C^2_{s k-1}} \right\}
  - \frac{G^{\frac{1}{2}}\theta p'^{n+1}}{\Delta t^2 C^2_s} \nonumber\\
  &=
  \frac{1}{\Delta t}\left[
      \frac{
         G^{\frac{1}{2}}_{i+1/2}\hat{\theta}_{i+1/2}\left\{(\rho u)_{i+1/2} + \Delta t (S_{\rho u})_{i+1/2}\right\}
        -G^{\frac{1}{2}}_{i-1/2}\hat{\theta}_{i-1/2}\left\{(\rho u)_{i-1/2} + \Delta t (S_{\rho u})_{i-1/2}\right\}
      }{\Delta x_i} \right.\nonumber\\&
    + \frac{
        G^{\frac{1}{2}}_{j+1/2}\hat{\theta}_{j+1/2}\left\{(\rho v)_{j+1/2} + \Delta t (S_{\rho v})_{j+1/2}\right\}
       -G^{\frac{1}{2}}_{j-1/2}\hat{\theta}_{j-1/2}\left\{(\rho v)_{j-1/2} + \Delta t 'S_{\rho v})_{j-1/2}\right\}
      }{\Delta y_j} \nonumber\\& \left.
    + \frac{
        J_{33}G^{\frac{1}{2}}\hat{\theta}_{k+1/2}\left\{(\rho w)_{k+1/2} + \Delta t (S_{\rho w})_{k+1/2}\right\}
       -J_{33}G^{\frac{1}{2}}\hat{\theta}_{k-1/2}\left\{(\rho w)_{k-1/2} + \Delta t (S_{\rho w})_{k-1/2}\right\}
      }{\Delta z_k}
    \right] \nonumber\\&\;\;\;
  + g \frac{1}{\Delta z_{k+1/2}+\Delta z_{k-1/2}}\left\{
    \frac{J_{33}G^{\frac{1}{2}}\theta p'^n_{k+1}}{C^2_{s k+1}}
   -\frac{J_{33}G^{\frac{1}{2}}\theta p'^n_{k-1}}{C^2_{s k-1}}
   - J_{33}G^{\frac{1}{2}} \{(\rho'\theta)_{k+1} - (\rho'\theta)_{k-1}\}
  \right\} \nonumber\\&
  + \frac{G^{\frac{1}{2}} S_{\rho\theta}}{\Delta t}
  - \frac{G^{\frac{1}{2}}\theta p'^n}{\Delta t^2 C^2_s}.
\end{align}


\section{Measurement Method}

The experimental setup consists of a signal source, two piezoelectric
transducers $S_1$ and $S_2$, and oscilloscope arranged as shown in Figure~
\ref{app}.

\begin{figure}[H]
    \centering
    \includegraphics[width=12cm]{fig/es1}
    \caption{Experimental setup}\label{app}
\end{figure}

\subsubsection{Resonance method}

The elements $S_1$ and $S_2$ are the wave source and the receiver (also
reflector), respectively, placed a distance $L$ from each other. If they are
arranged parallel to each other, the sound wave is reflected. If

\begin{equation}
    L=n\frac{\lambda}{2},
    \label{equ_res}
\end{equation}

where $n=1,2,\cdots,$ \emph{i.e.} the distance is a multiple of
half-wavelength, standing waves will form, and maximum output power will be
observed in the oscillograph (Figure 2). The distance between two successive
maxima ($L_{i+1} - L_i$) is always $\lambda/2$. After the position
corresponding to each maximum is measured, it is easy to find the wavelength
and then the speed of sound by using Eq.~\ref{vlf}. The frequency $f$ is
displayed directly on the signal generator.

\begin{figure}[H]
    \centering
    \includegraphics[width=12cm]{fig/r}
    \caption{Relation between the signal voltage and the distance}\label{vL}
\end{figure}

\subsubsection{Phase-comparison method}

If the phase of the wave at two points on the wave propagation direction is
equal, then the distance between these points L has to be a multiple of the
wavelength, \emph{i.e.}

\[
    L=n\lambda,
\]

where $n=1,2,\cdots$. The experimental setup for the phase comparison method
is the same as in the previous method (Figure \ref{app}). Lissajous figures
are used to identify the values of L. Lissajous figures (or Lissajous
curves) are trajectories of a particle that moves in a plane so that
\emph{i.e.} it moves in a harmonic motion independently along two
perpendicular directions (for example the axes x and y of a Cartesian
coordinate system), so that $\textbf{r}(t) = (A_x cos(\omega_x t + \phi_x
),\ A_y cos(\omega_y t + \phi_y ))$. When the two superimposed harmonic
motions have identical frequency $\omega x = \omega y$ and phase difference
$|\phi x - \phi y | = n\pi$, where $n = 0, 1, 2,\cdots$, the Lissajous
figure will show as a straight line. For other values of the phase
difference the figures will have an elliptical shape.\\

\subsubsection{Time-difference method}

When an ultrasonic pulse signal emitted by $S_1$ arrives at $S_2$, it is
received and returned back to the processor. By contrasting the original
signal with the received one, one can measure the time needed for the sound
to travel from $S_1$ to $S_2$ over a distance of L. When the values of $L$
and $t$ are known, the phase speed of sound can be found from Eq.
\ref{vlt}.

\subsubsection{Successive difference method}

The successive difference method is an effective method to increase the
accuracy of the average value calculated from a series of measurement data.
In this experiment, the usual method of calculating the average value,
illustrated by the formula

\begin{equation}\label{equ_suc}
\frac{\bar{\lambda}}{2}
=\frac{[(L_1-L_0)+(L_2-L_1)+\cdots+(L_n-L_{n-1})]}{n}
=\frac{L_n-L_0}{n},
\end{equation}

will be modified, because as Eq.~\ref{equ_suc} shows, the average value of
the wavelength is determined only by the first and the last value, $L_0$ and
$L_n$.

A modification of the formula by rearranging terms as
\begin{equation}\label{equ_suc}
    n\frac{\bar{\lambda}}{2}
    =\frac{\Sigma_{i=1}^{n}(L_{n+i}-L_i)}{n},
\end{equation}
produces more accurate results, as each value contributes to the final
result.


%% Submissions for peer-review must enable line-numbering
%% using the lineno option in the \documentclass command.
%%
%% Preprints and camera-ready submissions do not need
%% line numbers, and should have this option removed.
%%
%% Please note that the line numbering option requires
%% version 1.1 or newer of the wlpeerj.cls file.

\documentclass[fleqn,10pt,numbers]{wlpeerj} % for journal submissions
% \documentclass[fleqn,10pt]{wlpeerj} % for preprint submissions

\renewcommand\thesection{S\arabic{section}}
\renewcommand\thesubsection{\thesection.\arabic{subsection}}

%\usepackage[numbers]{natbib}

\usepackage{lmodern}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[scaled=0.8]{DejaVuSansMono}

\usepackage{hyperref}
\usepackage{graphicx}
\usepackage[all]{xy}
\usepackage{amsmath}
\usepackage{caption}
\graphicspath{ {images/} }

% Makes quote characters in monospace font not be curly
\usepackage{upquote}

\usepackage{amsmath}
\usepackage{url}

\hypersetup{
  colorlinks=true,
  urlcolor=blue,
  citecolor=black
}

% for nice units
\usepackage{siunitx}

% for images: png, pdf, etc
\usepackage{graphicx}

% for nice table formatting, i.e., /toprule, /midrule, etc
\usepackage{booktabs}

% to allow for \verb++ declarations in captions.
\usepackage{cprotect}

% to allow usage of \mathbb symbols
\usepackage{amssymb}

\usepackage{longtable}

\usepackage{listings}

\begin{document}

\flushbottom
\thispagestyle{empty}%
\vskip-36pt%
{\raggedright\sffamily\bfseries\fontsize{20}{25}\selectfont SymPy: symbolic computing in Python\par}%
\vskip10pt
{\raggedright\sffamily\fontsize{12}{16}\selectfont  Supplementary material\par}
\vskip25pt%

The supplementary material takes a deeper look at certain topics in SymPy
which there was not enough room to discuss in the paper.
Section~\ref{suppsec:Gruntz} discusses the Gruntz algorithm, used to
calculate limits in SymPy.  Sections~\ref{suppsec:Series}--\ref{suppsec:Ten}
discuss in depth some selected submodules.  Section~\ref{suppsec:numsimpl}
discusses numerical simplification.  Section~\ref{suppsec:examples} provides
additional examples for topics discussed in the main paper.  In section~\ref{suppsec:sympy-gamma} the SymPy Gamma
project is discussed.  Finally, section~\ref{suppsec:comp-mma}
has a
brief comparison of SymPy with Wolfram Mathematica.

As in the paper, all examples in the supplement assume that the following
has been run:
\begin{verbatim}
>>> from sympy import *
>>> x, y, z = symbols('x y z')
\end{verbatim}

\section{Limits: The Gruntz Algorithm}
\label{suppsec:Gruntz}
\input{gruntz.tex}

% Series module (Formal Power Series, Fourier Series)
\section{Series}
\label{suppsec:Series}
\input{series.tex}

% Logic module
\section{Logic}
\label{suppsec:Logic}
\input{logic}

\section{Diophantine Equations}
\label{suppsec:Dioph}
\input{diophantine}

% Sets
\section{Sets}
\label{suppsec:Sets}
\input{sets}

\section{Statistics}
\label{suppsec:Stats}
\input{stats}

\section{Category Theory}
\label{suppsec:Cat}
\input{categories}

\section{Tensors}
\label{suppsec:Ten}
\input{tensors}

\section{Numerical Simplification}
\label{suppsec:numsimpl}
\input{nsimplify}

\section{Examples}
\label{suppsec:examples}
\input{examples}

\section{SymPy Gamma}
\label{suppsec:sympy-gamma}
\input{gamma}

\section{Comparison with Mathematica}
\label{suppsec:comp-mma}
\input{comparison_with_mma}

\phantomsection
\bibliography{paper}

\end{document}


% !TEX options=--shell-escape
\documentclass [12pt]{article}

\usepackage {amsmath}
\usepackage {amsthm}
\usepackage {amssymb}
\usepackage {graphicx}
\usepackage {float}
\usepackage {multirow}
\usepackage {xcolor}
\usepackage {algorithmic}
\usepackage [ruled,vlined,commentsnumbered,titlenotnumbered]{algorithm2e} \usepackage {array}
\usepackage {booktabs}
\usepackage {url}
\usepackage {parskip}
\usepackage [margin=1in]{geometry}
\usepackage [T1]{fontenc}
\usepackage {cmbright}
\usepackage [many]{tcolorbox}
\usepackage [colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage {enumitem}
\usepackage {xparse}
\usepackage {verbatim}
\usepackage{listings}
\usepackage{xcolor}
\usepackage[cache=false]{minted}
\usepackage{mdframed}
\newtheorem{theorem}{Theorem}

\BeforeBeginEnvironment{minted}{\begin{mdframed}}
\AfterEndEnvironment{minted}{\end{mdframed}}

\DeclareTColorBox {Solution}{}{breakable, title={Solution}}
\DeclareTColorBox {Solution*}{}{breakable, title={Solution (provided)}}
\DeclareTColorBox {Instruction}{}{boxrule=0pt, boxsep=0pt, left=0.5em, right=0.5em, top=0.5em, bottom=0.5em, arc=0pt, toprule=1pt, bottomrule=1pt}
\DeclareDocumentCommand {\Expecting }{+m}{\textbf {[We are expecting:} #1\textbf {]}}
\DeclareDocumentCommand {\Points }{m}{\textbf {(#1 pt.)}}

\begin {document}

{\LARGE \textbf {COMP 285 (NC A\&T, Spr `22)}\hfill \textbf {Homework 2} }
\vspace {1em}
\begin {Instruction}

\paragraph {Due.} Tuesday, February 1st, 2022 @ 11:59 PM!
\end {Instruction}

\vspace {1em}
\begin {Instruction} \paragraph {Homework Expectations:} Please see the \href{https://www.comp285.ml/homework/#general-homework-information}{Homework} part of the Course Website (\href{https://www.comp285.ml/policies/#collaboration-policy-and-honor-code}{comp285.ml/policies}) for guidance on what we are looking for in homework solutions. We will grade according to these standards, and you should cite all sources you used outside course material.

\paragraph {What we expect:} Make sure to look at the ``\textbf {We are expecting}'' blocks below each problem to see what we will be grading for in each problem! \end {Instruction}

\vspace {1em}
\begin {Instruction}

\paragraph {Exercises} The following questions are exercises. We encourage you to work with a group and discuss solutions to make sure you understand the material.

\paragraph {Points} This assignment is graded out of 100 points. However, you can get up to 120 points if you complete everything. These are not bonus points, but rather points to help make-up any parts you miss.

\end {Instruction}

\begin{centering}
\section*{Formal Fun with Recursion and Big-Oh}
\end{centering}

\begin{Instruction}

\paragraph{Written Problems} The following questions are to be submitted in written/typed form to gradescope.

\end{Instruction}

\section{Exercise: Getting Basic with Big-Oh}
\Points{20}
Dr. Wang is is preparing for a review lecture for the students who want to get a bit more confortable with big-Oh notation. given how studious you've been, she asks for your help on a few questions.

\subsection{Can you identify big-Oh?}
\Points{10} Write the functions below that are $O(n^2)$?

You don't need to provde a prove or explanation, but you probably want to convince yourself you're right.

\begin{enumerate}
    \item $g_1(n) = 4n + 10$
    \item $g_2(n) = 4n^2 + 10$
    \item $g_3(n) = 4n^3 + 10$
    \item $g_4(n) = n \log n$
    \item $g_5(n) = \sin(n) + 5$
    \item $g_6(n) = 4^n$
    \item $g_7(n) = (285^3)^4$
\end{enumerate}

\Expecting{A list of which functions are $O(n^2)$. No explanation is required.}

\subsection{Formal Definition of Big-Oh}
\Points{10} Serena is having a great time on COMP 285! She wants to prove, from the definition of big-Oh, that $f(n) = O(g(n))$, where $f (n) = 6n$ and $g(n) = n^2 - 3n$.\footnote{For this problem, it might help you to plot the functions you're trying on Wolfram Alpha. For example, \href{https://www.wolframalpha.com/input/?i=plot+5n+and+n\%5E2-3n+from+0+to+10}{here} is the plot of $6n$ and $n^2 - 3n$.}

As a reminder, here is the formal definition of big-Oh is:
\begin{align*}
f(n) &= O(g(n)) \\
&\iff \\
\exists c, n_0 \text{ such that } &\forall n > n_0, f(n) \leq c \cdot g(n)
\end{align*}

\subsubsection{First proof...}
\Points{5} Serena claims she has a proof in the definition above where $c = 1$. Give a possible proof using $c=1$.

\subsubsection{Second proof...}
\Points{5} Serena realizes that she'll never get inputs to her algorithm that are smaller than $4$. As such, she says she wants $n_0 = 4$. Give a possible proof using $n_0 = 4$.

\Expecting{For both parts, a short but complete proof. That is, write out the formal definition of big-Oh plugging in the values for $f(n), g(n), n_0$ and $c$ as given in the problem statement and that you chose.}

\pagebreak
\section{Interview Practice: Comparing Big-Oh, Big-Omega, and Big-Theta}
\Points{10} In industry, it's quite common that you have to compare the running times of different algorithms you're implementing \footnote{If you'd like specific examples, come to my student hours!}. In this exercise, we'll practice a few common running times (I've personally seen all of these show-up on interviews) and compare them.

For each blank below, indicate whether $A_i$ is in $O$, $\Omega$, or $\Theta$ of $B_i$
. More than one space per row can be valid.\footnote{One way to get a sense for which function is bigger is to plot them. For example, \href{https://www.wolframalpha.com/input/?i=plot+log_2+n+and+n+from+0+to+10}{this} plots $\log_2 n$ and $n$ from $0$ to $10$.}

\Expecting{All spaces in the below table should have an 'x' if $A_i$ is big-[COLUMN] of $B_i$}.

\begin{table}[h!]
\centering
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{A} & \textbf{B} & $O$ & $\Omega$ & $\Theta$ \\ \hline
    $\log n$       &      $n$      &  &  &  \\ \hline
     $2n^3$      &  $2^n$         &  &  &  \\ \hline
       $\log_{10} n$    &  $\log_2 n$          &  &  &  \\ \hline
    $n^{0.1}$    &  $(0.1)^n$          &  &  &  \\ \hline
    $n^2$    &  $4^{\log_2 n}$          &  &  &  \\ \hline
\end{tabular}
\end{table}

\pagebreak
\section{Exercise: Practice with Recurrence Relations}
\Points{10} After settling down after her fourth move (joining us in Greensboro), Sloane wants to practice some \textbf{recurrence relations} with you. To refresh your memory, she jots down the \textbf{Master Theorem} below:

\begin{theorem}[Master Theorem]
Let $T(n) = a \cdot T\left(\frac{n}{b}\right) + O(n^d)$ be a recurrence where $a, b > 1$, Then,
\begin{align*}
T(n) &= \begin{cases}
    O(n^d \log n) & \text{if } a = b^d \\
    O(n^d) & \text{if } a < b^d \\
    O(n^{\log_b a}) & \text{if } a > b^d
\end{cases}
\end{align*}

For each of the below, provide the big-Oh running time:
\begin{enumerate}[label=(\alph*)]
    \item \Points{4} $T(n) = 18T\left(\frac{n}{9} \right) + O(\sqrt{n})$
    \item \Points{2} $T(n) = 5T\left(\frac{n}{4} \right) + O(n^2)$
    \item \Points{3} $T(n) = 16T\left(\frac{n}{2} \right) + O(n^4)$
    \item \Points{1} $T(n) = 4T\left(\frac{n}{3} \right) + O(n)$
\end{enumerate}
\end{theorem}

\Expecting{Use Master Theorem to calculate the tigheter upper bound for each of these recurrence relations in big-Oh notation.}

\pagebreak
\section{Interview Practice: Modified MergeSort}
\Points{10} In most interviews, you won't be asked to reproduce algorithms we've learned in class. Instead, you'll be asked to implement modifications of these algorithms. In this exercise, we'll see how changing the size of the subproblems affects MergeSort.

Your friend Malia gives you a new version of \texttt{MergeSort} that she coded after class. She claims this runs asymtotically better than the version we coded.

This is her code, where \texttt{merge} is left unchanged.

\vspace{2em}
\begin{minted}[linenos=true]{cpp}
// MergeSort algorithm!
std::vector<int> mergeSortThirds(const std::vector<int>& input) {
    const std::size_t n = input.size();
    if (n <= 1) {
        return input;
    }
    const std::vector<int> left = mergeSortThirds(
        {input.begin(), input.begin() + n / 3});
    const std::vector<int> middle = mergeSortThirds(
        {input.begin() + n / 3, input.begin() + 2*n / 3})
    const std::vector<int> right = mergeSortThirds(
        {input.begin() + 2*n / 3, input.end()});
    const std::vector<int> temp = merge(L, M);
    return merge(temp, right);
}
\end{minted}

Write down the recurrence relation and runtime for this version of \texttt{MergeSort}.

\Expecting{A recurrence relation for the runtime, as well as the closed-form big-Oh runtime.\footnote{Once you have the recurrence relation, you can probably use the Master Theorem.}}

\pagebreak
\section{Industry Practice: What's the big-Oh of code?}
\Points{20} In industry, you'll often times be expected to analyze existing code. In this question, you'll get a bit of practice looking at unfamiliar C++ code and identifying the running time.

\subsection{A loop that doubles each time!}
\Points{5} Your friends Victor and Dezmon are working on their new internship. They show you the below code. What is the big-Oh running time? Please justify your answer.

\vspace{2em}
\begin{minted}[linenos=true]{cpp}
void doSomething(const std::vector<int> input) {
    for (int i = 1; i < input.size(); i *= 2) {
        std::cout << input[i] << std::endl;
    }
}
\end{minted}

\Expecting{You should provide a big-Oh running time $O(\cdots)$ and a short sentence explaining why this is the running time.}

\subsection{Counting down by 10...}
\Points{5} Your friends Liam and Miah are working on their new jobs at Foogle. They show you the below code. What is the big-Oh running time? Please justify your answer.

\vspace{2em}
\begin{minted}[linenos=true]{cpp}
void doSomethingElse(const std::vector<int> input) {
    int z = input.size() - 1;
    std::cout << z << std::endl;
    while(z >= 10) {
        std::cout << input[z] << std::endl;
        z /= 10;
    }
}
\end{minted}

\Expecting{You should provide a big-Oh running time $O(\cdots)$ and a short sentence explaining why this is the running time.}

\pagebreak
\subsection{Fun with Recursion}
\Points{10} Your friends Ryan and Jordan are working with the NSA. They show you the below code. What is the big-Oh running time? Please justify your answer.

\vspace{2em}
\begin{minted}[linenos=true]{cpp}
std::string doSomethingSecret(const std::vector<int> input) {
    if (input.size() % 7 == 0) {
        return "Bzzzt!";
    }
    // Remove the last element in-place. This is O(1).
    input.pop_back();
    return doSomethingSecret(input);
}
\end{minted}

\Expecting{You should provide a big-Oh running time $O(\cdots)$ and a short sentence explaining why this is the running time.}

\pagebreak
\begin{Instruction}

\paragraph{Coding Problems} The following questions are to be submitted as a ".zip" file on Gradescope. For complete submissions instructions, please see HW1.

\end{Instruction}

\section{Coding}
\Points{50} After completing the written portion of the assignment, you should submit to \href{https://www.gradescope.com/courses/350304}{Gradescope}.

You can get your starter code for the coding portion \href{https://replit.com/team/COMP285/HW2-Code}{here}.

Note there are \textbf{two} coding questions. Each part is worth 20 points for correcness. Your code will also be reviewed for style (5 points) and documentation (5 ponts), for a total of 50 points for the coding portion.

Please reference the \texttt{README.md} included in your starter code for detailed instructions.

\section*{Submitting the Assignment}

This assignment is a combination of written and programming questions. Both portions of the assignment should be submitted through \href{https://www.gradescope.com/courses/350304}{Gradescope}.

The "Homework 2: Fun with Recursion and Big-Oh" assignment is the written portion, for which you should submit a \textbf{typed} response to the non-coding questions (questions 1-5). Each response should clearly be marked with its corresponding number. You are free to use the provided templates, print the questions and write your answers, or to simply type your responses on a blank document (whatever works for you).

The "Homework 2: Coding" is the programming portion of the assignment. For this portion, download the ".zip" file from replit and upload this ".zip" file as your answer to \href{https://www.gradescope.com/courses/350304}{Gradescope}. You can upload the assignment as many times as you want.

To summarize, do the following:
\begin{enumerate}
    \item Submit your \textbf{typed} responses to the non-coding portion of the assignment on Gradescope for "Homework 2: Fun with Recursion and Big-Oh"
    \item Submit your \textbf{code files} as a ".zip" to Gradescope for "Homework 2: Coding".
    \item Fill out this \textbf{Google Form} (\href{https://forms.gle/ELVUbA9Tawrr4oFU8}{https://forms.gle/ELVUbA9Tawrr4oFU8}) to help me track your thoughts on the homework.
\end{enumerate}

\end {document}


\subsection{Hyper-elliptic curves}


\subsection{Spectral theorem for infinite-dimensional vector spaces}


\documentclass[notes]{beamer}          % print frame + notes
%\documentclass[notes=only]{beamer}     % only notes
%\documentclass{beamer}                 % only frames

\usecolortheme{beaver}

% Some commonly used packages
% (copied mainly from the Utrecht University theme: https://www.overleaf.com/project/5c900fa3bd9930036341116a)
\usepackage{ragged2e}  % `\justifying` text
\usepackage{booktabs}  % Tables
\usepackage{tabularx}
\usepackage{tikz}      % Diagrams
\usetikzlibrary{calc, shapes, backgrounds}
\usepackage{amsmath, amssymb, amsfonts, amsthm}
\usepackage{url}       % `\url`s
\usepackage{listings}  % Code listings
\usepackage{comment}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage{subfig}
\usepackage{amsmath}

% Mainly math commands
\newcommand{\vect}[1]{\bm{#1}}
\usepackage{amsfonts}% to get the \mathbb alphabet
\newcommand{\field}[1]{\mathbb{#1}}
\newcommand{\C}{\field{C}}
\newcommand{\R}{\field{R}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\argmin}{\operatornamewithlimits{argmin}}
\providecommand{\abs}[1]{\lvert#1\rvert}
\providecommand{\norm}[1]{\lVert#1\rVert}

% A variable used to exclude slides from the lecture version
\newif\iffull
%\fullfalse
\fulltrue

% Bibliography
\usepackage[uniquename=init,giveninits=true,maxcitenames=1,style=authortitle-comp]{biblatex}
\bibliography{references}

%Information to be included in the title page:
\title{Linear models}
\author{Federica Eduati}
\institute{Eindhoven University of Technology

Department of Biomedical Engineering}
\date{2019}

\begin{document}

\frame{\titlepage}

\begin{frame}
\frametitle{Learning goals}

At the end of this lecture you will:
\begin{itemize}
    \item Have a good understanding of linear models for regression, including methods for subset selection and shrinkage.
    \item Have a good understanding of logistic regression as linear method for classification.
\end{itemize}

\vspace{5mm}

Materials:
\begin{itemize}
    \item Chapters 3 and 4 from \cite{elements}
\end{itemize}

\end{frame}

\begin{frame}{Overview}
Topics covered in this lecture:
    \tableofcontents
\end{frame}

\section{Linear models for regression}

\begin{frame}
\frametitle{Introduction to linear models}
Linear models: a linear combination of the \textit{inputs} (also known as \textit{predictors}, \textit{features} or \textit{independent variables}) is used to predict one or more \textit{outputs} (also known as \textit{responses} or \textit{dependent variables}).

\vspace{5mm}

The prediction task is defined as:
\begin{itemize}
    \item \textit{Regression}: when we predict \textit{quantitative outputs}.
    \item \textit{Classification}: when we predict \textit{qualitative outputs} (also referred to as \textit{categorical or discrete variables}).
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Linear regression model}
Given a vector of inputs $X^T = (X_1, X_2, \dots, X_p)$, where $p$ is the number of features, a linear regression model has the form:

\begin{equation*}
f(X)=\beta_0 + \sum_{j=1}^p X_j\beta_j
\end{equation*}
where $\beta_j$ are the unknown parameters or coefficients.

\vspace{5mm}

Note that the model remains linear in the parameters even if the the variables $X_j$ are polynomial (e.g. $X_2 = X_1^2$) or derive from interactions between variables (e.g. $X_3 = X_1 \cdot X_2$).

\end{frame}

\begin{frame}
\frametitle{Estimation via least squares}
Parameters $b_j$ can be estimated from a set of training data $(x_1,y_1) \dots (x_N,y_N)$, where each $x_i=(x_{i1}, x_{i2}, \dots, x_{ip})^T$ is a vector of feature measurements for the $i$th case.

\vspace{5mm}

With the \textit{least squares} estimation methods, the coefficients $\beta=(\beta_0, \beta_1, \dots, \beta_p)^T$ are selected by minimizing the residual sum of squares:

\begin{equation*}
RSS(\beta) = \sum_{i=1}^N (y_i - f(x_i))^2 = \sum_{i=1}^N (y_i - \beta_0 -  \sum_{j=1}^p x_{ij}\beta_j)^2
\end{equation*}
\end{frame}

\begin{frame}
\frametitle{Example of univariate linear regression}
$\hat{y} = \beta_0 + \beta_1 x$, where output variable $Y$ represents the sensitivity to a certain drug and input variable $X$ the expression of a gene.

\begin{center}
\includegraphics[height=6cm]{figures/week_3/Univariate_linear_regression_0.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Example of univariate linear regression}
Least square fitting: linear function of $X$ that minimises the sum of squared residuals.

\begin{center}
\includegraphics[height=6cm]{figures/week_3/Univariate_linear_regression_1.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Example of univariate linear regression}
Least square fitting: linear function of $X$ that minimises the sum of squared residuals. $RSS = \sum_{i=1}^N (y_i - \beta_0 - \beta_1 x_i)^2$

\begin{center}
\includegraphics[height=6cm]{figures/week_3/Univariate_linear_regression_2.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Genomics of Drug Sensitivity in Cancer (GDSC) project}
We will use as example data from the GDSC project (https://www.cancerrxgene.org/) \cite{GDSC}.

\begin{center}
\includegraphics[height=6cm]{figures/week_3/GDSC_study_description.png}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Genomics of Drug Sensitivity in Cancer (GDSC) project}
Cancer patients tends to respond very differently to drugs. A big challenge in cancer research
is to find ways to assign the optimal treatment to each patient.

\begin{center}
\includegraphics[height=2.5cm]{figures/week_3/precision_oncology.png}
\end{center}

The aim of the study is to find biomarkers of drug sensitivity.

\begin{center}
\includegraphics[height=2.5cm]{figures/week_3/GDSC_study_aim.png}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Our dataset}

We will consider data for 148 cell lines from four cancer types.

\begin{center}
\begin{tabular}{ |c|c| }
 \hline
 ID & Cancer type \\
 \hline
 COAD/READ & Colorectal adenocarcinoma \\
 NB & Neuroblastoma \\
 KIRC & Kidney renal clear cell carcinoma \\
 BRCA & Breast carcinoma \\
 \hline
\end{tabular}
\end{center}

We can interpret the prediction of drug sensitivity as a linear regression problem using:

\begin{itemize}
    \item Output variable: Sensitivity to YM155 (Sepantronium bromide), as natural log of the fitted IC50.
    \item Predictors: expression of 244 genes (the ones with higher variance), as RMA normalised expression.
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Feature scaling}

Idea: make sure features are on a similar scale.

\begin{equation*}
x_i = \frac{x_i-\mu_i}{\sigma_i}
\end{equation*}

where $\mu_i$ and $\sigma_i$ are respectively the average and the standard deviation of all the values of feature $i$.

\vspace{5mm}

This makes the estimated coefficients comparable and speeds up the optimization algorithm.

\vspace{5mm}

If we standardize the predictors, the solution for $\hat{\beta_0}$ is $\overline{y}$, therefore we fit a model without the intercept.

\end{frame}

\section{Subset selection}

\begin{frame}
\frametitle{Model complexity}

Especially when we have a large number of features ($p$ large compared to $N$), least squares estimates can suffer from:

\begin{itemize}
    \item Low \textit{prediction accuracy}: high model complexity gives low bias but high variance, setting some coefficients to zero can reduce the variance of predictions (at the price of increasing the bias).
    \item Poor \textit{interpretability}: we might want to identify which predictors are really useful to explain the output.
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Model complexity and overfitting}

\begin{center}
\includegraphics[height=5.5cm]{figures/week_3/Bias_variance_complexity.pdf}
\end{center}

Complex models tend to overfit the training data and perform poorly on the test data.

\end{frame}

\begin{frame}
\frametitle{Subset selection}

With \textit{subset selection} we want to retain only a subset of the predictors and eliminate the rest from the model.

\vspace{5mm}

We can try a lot of different models, each containing a different subset of the predictors, and check which model is the best, e.g. based on \textit{Akaike information criterion (AIC)} or \textit{Bayesian information criterion (BIC)}.

\vspace{5mm}

Problem: there are a total of $2^p$ models that contain subsets of p predictors!!

\end{frame}

\begin{frame}
\frametitle{Subset selection}

Three classical approaches to step-wise \textit{subset selection}:

\begin{itemize}
    \item \textit{Forward selection}. Start from the null model (i.e. only intercept) and sequentially add to the model the variable that gives lowest RSS (i.e. higest improvement of the fit).
    \item \textit{Backward selection}. Start with the full model (i.e. all variables) and sequentially remove the predictor with the largest p-value (i.e. lower impact on the fit).
    \item \textit{Mixed selection}. Alternation of forward and backward steps to add variables that improves RSS while maintaining the p-value below a certain threshold.
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Subset selection}

\textit{Subset selection} tends to:

\begin{itemize}
    \item improve \textit{interpretability}: only the most relevant predictors useful to explain the output are selected.
    \item still suffer from  low \textit{prediction accuracy}: high variance (discrete process in retaining and discarting predictors) often does not reduce prediction error.
\end{itemize}

\end{frame}

\section{Shrinkage methods}

\begin{frame}
\frametitle{Improving least square estimates with regularization}
Least square estimates generally provide all non-zero coefficient and, if $p>N$, solutions are not unique (i.e. multiple solutions with same minimum, often overfitting the data).

\vspace{5mm}

Need to constrain or regularize the estimation process.

\vspace{5mm}

Idea: shrink regression coefficients by imposing a penalty on their size.

\end{frame}

\begin{frame}
\frametitle{Ridge regression}
\textit{Ridge regression} penalizes the sum of squares of the coefficients (L2 regularization).

\begin{align*}
\hat{\beta}^{ridge} &= \argmin_{\beta} \left\{ \sum_{i=1}^N (y_i - \beta_0 -  \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^p \beta_j^2 \right\} \\
& =  \argmin_{\beta} \left\{ RSS + \lambda \sum_{j=1}^p \beta_j^2 \right\}
\end{align*}

\end{frame}

\begin{frame}
\frametitle{Ridge regression: the meaning of $\lambda$}

\begin{equation*}
    \hat{\beta}^{ridge} =  \argmin_{\beta} \left\{ RSS + \lambda \sum_{j=1}^p \beta_j^2 \right\}
\end{equation*}

$\lambda \geq 0$ is a tuning parameter.

\begin{itemize}
    \item $\lambda = 0$ corresponds to the least square estimates.
    \item for larger values of $\lambda$, coefficients $\beta_1, \dots, \beta_p$ will be shrinked towards zero
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{Ridge regression: the meaning of $\lambda$}

\begin{center}

\includegraphics[height=5.5cm]{figures/week_3/Ridge_regression_coefficinets.pdf}
\end{center}

Profile of the Ridge regression coefficients for the GDSC example.

\end{frame}

\begin{frame}
\frametitle{The effect of regularization on bias and variance}
In matrix form we can rewrite the RSS as:

\begin{equation*}
    RSS(\lambda) = (\mathbf{y} - \mathbf{X}\beta)^T (\mathbf{y} - \mathbf{X}\beta) + \lambda\beta^T\beta
\end{equation*}

And the Ridge regression solution becomes:

\begin{equation*}
    \hat{\beta}^{ridge} = (\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}
\end{equation*}

where $\mathbf{I}$ is the $p \times p$ identity matrix. This is linear in $\mathbf{y}$ and in closed form.

Assuming that $\mathbf{y} = \mathbf{X}\beta + \mathbf{e}$, where $\mathbf{e}$ is a random error on the output variable, the Ridge regression solution can be written as:

\begin{align*}
    \hat{\beta}^{ridge} &= (\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T(\mathbf{X}\beta + \mathbf{e}) \\
    &= \underbrace{(\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{X}\beta}_{\textrm{bias}} + \underbrace{(\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{e}}_{\textrm{variance}}
\end{align*}

\end{frame}

\begin{frame}
\frametitle{The effect of regularization on bias and variance}
Effect of $\lambda$ on the bias-variance trade off in a simulated example ($y_i = x_{i1}\beta_1 + x_{i2}\beta_2 + e_i$, $i=1,\dots, 20$). The model is simulated 50k times, each time the noise $e_i$ is randomly generated.

\begin{center}
\includegraphics[height=5.5cm]{figures/week_3/Ridge_bias_variance_insilico.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Lasso regression}
\textit{Lasso regression} penalizes the sum of the absolute value of the coefficients (L1 regularization).

\begin{align*}
\hat{\beta}^{lasso} &= \argmin_{\beta} \left\{ \sum_{i=1}^N (y_i - \beta_0 -  \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^p | \beta_j | \right\} \\
& =  \argmin_{\beta} \left\{ RSS + \lambda \sum_{j=1}^p | \beta_j | \right\}
\end{align*}

\end{frame}

\begin{frame}
\frametitle{Lasso regression: the meaning of $\lambda$}

\begin{equation*}
    \hat{\beta}^{lasso} =  \argmin_{\beta} \left\{ RSS + \lambda \sum_{j=1}^p | \beta_j | \right\}
\end{equation*}

$\lambda \geq 0$ is a tuning parameter.

\begin{itemize}
    \item $\lambda = 0$ corresponds to the least square estimates
    \item larger values of $\lambda$, will make some of the coefficients $\beta_1, \dots, \beta_p$ to be exactly zero, performing a continuous subset selection.
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{Lasso regression: the meaning of $\lambda$}

\begin{center}
\includegraphics[height=5.5cm]{figures/week_3/Lasso_regression_coefficinets.pdf}
\end{center}

Profile of the Lasso regression coefficients for the GDSC example.

\end{frame}

\begin{frame}
\frametitle{Comparing Lasso and Ridge regression}

An equivalent way or writing the Ridge and Lasso problems is:

\begin{equation*}
    \argmin_{\beta} \sum_{i=1}^N (y_i - \beta_0 -  \sum_{j=1}^p x_{ij}\beta_j)^2
\end{equation*}

subject to:
\begin{itemize}
    \item $\sum_{j=1}^p \beta_j^2 < t$ for Ridge regression
    \item $\sum_{j=1}^p | \beta_j | < t$ for Lasso regression
\end{itemize}

\vspace{5mm}

It is possible to show that there is a one-to-one correspondence between the parameter $\lambda$ and $t$.

\end{frame}

\begin{frame}
\frametitle{Comparing Lasso and Ridge regression}

\begin{center}
\includegraphics[height=7cm]{figures/week_3/Comparison_lasso_ridge.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{How to select optimal $\lambda$}

We need a method to select the tuning parameter $\lambda$ for both Lasso and Ridge regression.

\vspace{5mm}

We want the model that provides the best predictions on the test set. In general this is very sensitive to the data used for training.
\end{frame}

\begin{frame}
\frametitle{How to select optimal $\lambda$}

Two data partitions - GDSC example (90\% training, 10\% test).

\begin{center}
\includegraphics[height=7cm]{figures/week_3/Lasso_MSE_train_test.pdf}
\end{center}
\end{frame}

\begin{frame}
\frametitle{How to select optimal $\lambda$}

We can use cross-validation following these steps:
\begin{enumerate}
  \item Choose a grid of $\lambda$.
  \item Optimize the model for each training set and compute Mean-Squared Error (MSE) for each validation set.
  \item Choose the model with smallest cross-validation error.
  \item Re-fit using all the available observations and the selected tuning parameter.
\end{enumerate}

\end{frame}

\begin{frame}
\frametitle{$\lambda$ selection: GDSC example 10-fold cross validation}

\begin{center}
\includegraphics[height=5cm]{figures/week_3/Lasso_MSE_crossvalidation.pdf}
\end{center}

\vspace{-2mm}

\begin{itemize}
    \item $\lambda_{min}$: value of $\lambda$ that gives the minimum MSE
    \item $\lambda_{min+1se}$: largest value of $\lambda$ such that error is within 1 standard error of the minimum
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Lasso for feature selection}
GDSC example 10-fold cross validation

\begin{center}
\includegraphics[height=5cm]{figures/week_3/Lasso_feature_selection.pdf}
\end{center}

$\lambda_{min+1se}$ gives more sparse solutions.

\end{frame}

\begin{frame}
\frametitle{Lasso for feature selection}

Selected features can be sensitive to the partition of the cross-validation.

\vspace{5mm}

To select only robust features we can use \textit{bootstrap}: feature selection procedure is repeated M times, using every time a different bootstrapped dataset (i.e. sampling N samples with replacement).

\end{frame}

\begin{frame}
\frametitle{Lasso for feature selection: using bootstrap}
GDSC example: 100 times bootstrap (with 10-fold cross-validation)

\begin{center}
\includegraphics[height=7cm]{figures/week_3/Lasso_feature_selection_bootstrap.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Collinearity}
Two or more predictor variables can be closely related to each other and show a high correlation.

\vspace{5mm}

This is common in genomic data, because genes can be redundant or have related functions.

\begin{center}
\includegraphics[height=5cm]{figures/week_3/collinearity.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Collinearity}
The effect of \textit{collinear variables} can be difficult to separate in the regression context.

\vspace{5mm}

\textit{Collinearity} in general reduces the accuracy of the estimates of the coefficients of the correlated variables.

\vspace{5mm}

When using Lasso, \textit{collinearity} can result in an arbitrary selection of one or the other correlated feature, depending on the data used for training.
\end{frame}

\begin{frame}
\frametitle{Elastic Net regression}

Elastic Net provides a compromise between Ridge and Lasso, by selecting variables like Lasso and shrinking together correlated variables like Elastic Net.

\begin{equation*}
    \hat{\beta}^{elastic net} =  \argmin_{\beta} \left\{ RSS + \lambda \sum_{j=1}^p ((\alpha \beta_j^2 + (1-\alpha)| \beta_j |) \right\}
\end{equation*}

Where $\alpha$ is a tunable parameter. This corresponds to:

\begin{itemize}
    \item Ridge for $\alpha=0$
    \item Lasso for $\alpha=1$
\end{itemize}

\end{frame}

\begin{frame}
\frametitle{Elastic Net regression}

\begin{center}
\includegraphics[height=5.5cm]{figures/week_3/ElasticNet_regression_coefficinets.pdf}
\end{center}

Profile of the Elastic net (with $\alpha=0.5$) regression coefficients for the GDSC example.

\end{frame}

\section{Classification}

\begin{frame}
\frametitle{Classification}
In a clinical setting, instead of predicting the continuous IC50 for each patient, we might be interested in classifying patients in sensitive or resistant to a specific drug.

\vspace{5mm}

Using the cell lines from the GDSC study this corresponds to divide them in sensitive and resistant to the drug.

\begin{center}
\includegraphics[height=2.7cm]{figures/week_3/GDSC_classification_overview.pdf}
\end{center}

Cell lines are separated in sensitive and resistant based on the average response across all cell lines in the panel.

\end{frame}

\begin{frame}
\frametitle{Classification}

For binary classification we will consider a positive and a negative class:

\begin{itemize}
    \item 0 ("negative class"): sensitive
    \item 1 ("positive class"): resistant
\end{itemize}

\begin{center}
\includegraphics[height=5cm]{figures/week_3/GDSC_classification_sensitivity.pdf}
\end{center}
\end{frame}

\begin{frame}
\frametitle{Example of classification with one variable}

Using the same example used for univariate linear regression, we consider the ability of gene ABCB1 to classify cell lines in sensitive and resistant.

\begin{center}
\includegraphics[height=4cm]{figures/week_3/GDSC_one_variable_example_logistic_regression_1.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{Example of classification with one variable}

We can try to fit a linear regression to a binary response, e.g.using 0.5 as threshold for prediction.

\begin{center}
\includegraphics[height=4.5cm]{figures/week_3/GDSC_one_variable_example_logistic_regression_2.pdf}
\end{center}

Poor interpretability, with predicted values also outside [0, 1].

\end{frame}

\begin{frame}
\frametitle{Example of classification with one variable}

With logistic regression we want to model the probability (between 0 and 1) that $Y$ belongs to a particular class.

\begin{equation*}
Pr(response = resistant|ABCB1expression)
\end{equation*}

\begin{center}
\includegraphics[height=4.5cm]{figures/week_3/GDSC_one_variable_example_logistic_regression_3.pdf}
\end{center}

\end{frame}

\section{Logistic regression}
\begin{frame}
\frametitle{Logistic regression}

\begin{equation*}
p(X) = Pr(G = 1|X)
\end{equation*}

With \textit{logistic regression} this probability is modeled as a \textit{logistic function}:

\begin{equation*}
p(X) = \frac{e^{\beta_0 + \beta_1X}}{1+e^{\beta_0 + \beta_1X}}
\end{equation*}

This corresponds to:

\begin{equation*}
\log \left(\frac{p(X)}{1-p(X)}\right)  = \beta_0 + \beta_1X
\end{equation*}

The left-hand side is called the log-odds or logit transformation and it is linear in X.

\end{frame}

\begin{frame}
\frametitle{Logistic regression: multi-class classification}

When having $K$ classes, the posterior probability is modeled as:

\begin{align*}
\log \frac{Pr(G=1 | X=x)}{Pr(G=K | X=x)} &= \beta_{10} + \beta_1^T x \\
\log \frac{Pr(G=2 | X=x)}{Pr(G=K | X=x)} &= \beta_{20} + \beta_2^T x \\
 & \vdots \\
\log \frac{Pr(G=K-1 | X=x)}{Pr(G=K | X=x)} &= \beta_{(K-1)0} + \beta_{K-1}^T x \\
\end{align*}

Using $K$-1 log-odds satisfies the constraint that probabilities sum to one. The choice of the last class to be used at denominator is arbitrary and does not affect the result.

\end{frame}

\begin{frame}
\frametitle{Logistic regression: multi-class classification}

It is easy to see that probabilities sum to one by rewriting the model as:

\begin{align*}
Pr(G=k | X=x) &= \frac{e^{\beta_{k0} + \beta_k^T x}}{1+\sum_{l=1}^{K-1} e^{\beta_{l0} + \beta_l^T x}}, k=1, \dots, K-1 \\
Pr(G=K | X=x) &= \frac{1}{1+\sum_{l=1}^{K-1} e^{\beta_{l0} + \beta_l^T x}}
\end{align*}

An alternative notation for the probabilities is $P(G=k|X=x) = p_k(x;\theta)$, to emphasize the dependence on the entire parameter set $\theta=  \{ \beta_{10}, \beta_1^T, \dots, \beta_{(K-1)0}, \beta_{K-1}^T \}$.

\end{frame}

\begin{frame}
\frametitle{Logistic regression: parameters estimation}

The parameters for logistic regression models are generally estimated using \textit{maximum-likelihood}.

\vspace{5mm}

For $N$ observations the log-likelihood to be maximized is:

\begin{equation*}
    \ell(\theta) = \sum_{i=1}^N \log p_{g_i}(X_i; \theta)
\end{equation*}

where $p_k(x_i; \theta) = Pr(G=k|X=x_i; \theta)$.

\end{frame}

\begin{frame}
\frametitle{Logistic regression: parameters estimation}

For the two classes case the log-likelihood can be written

\begin{align*}
    \ell(\beta) &= \sum_{i=1}^N \{ y_i \log p(x_i;\beta) + (1-y_i) \log (1-p(x_i;\beta)) \} \\
    &= \sum_{i=1}^N \{ y_i \beta^T x_i - \log (1+e^{\beta^T x_i}) \}
\end{align*}

where $\beta=\{ \beta_{10}, \beta_1\}$ and vector of inputs $x_i$ includes the constant term 1 for the intercept.

\end{frame}

\begin{frame}
\frametitle{Multiple logistic regression}

When we have multiple predictors

\begin{equation*}
    \log \left( \frac{p(X)}{1-p(X)} \right) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p
\end{equation*}

where $X=(X_1, \dots, X_p)$ are the $p$ predictors. Or with the equivalent form

\begin{equation*}
    p(X) = \frac{e^{\beta_0 + \beta_1 X_1 + \dots + \beta_p X_p}}{1+e^{\beta_0 + \beta_1 X_1 + \dots + \beta_p X_p}}
\end{equation*}

\end{frame}

\begin{frame}
\frametitle{L1 regularized logistic regression}

Similarly to what we have seen for linear regression, regularization can be used for variable selection and shrinkage also with logistic regression.

\vspace{5mm}

When applying L1 penalty we want to maximise a penalized version of the log-likelihood

\begin{equation*}
    \max_{\beta_0, \beta} \left\{ \sum_{i=1}^N [ y_i(\beta_0 + \beta^T x_i) - \log (1+e^{(\beta_0 + \beta^T x_i))}] -\lambda \sum_{j=1}^p | \beta_j| \right\}
\end{equation*}

Note that the intercept term is not penalized. Similarly to Lasso, the predictors should be standardized in order for the coefficients to be comparable.

\end{frame}

\begin{frame}
\frametitle{Example cross-validation}

Also in this case cross-validation can be used to select the tuning parameter.

\begin{center}
\includegraphics[height=4.5cm]{figures/week_3/GDSC_logistic_regression_cross_validation.pdf}
\end{center}

\end{frame}

\begin{frame}
\frametitle{References}
\printbibliography
\end{frame}

\end{document}