\chapter{Results}
\label{sec:results}
We will now first investigate the performance of our models compared to our chosen reference models on the $17$ \gls{tcga} datasets considered in our benchmark. In addition to the absolute performance as measured by Harrell's concordance index (Equation \ref{eq:concordance}), we also calculated the rank of each model (by mean concordance, lower rank implies higher mean concordance) on each cancer to get an overview of how the models compared to each other in relative terms. We also compared the computation times of all models. Further, we tested for a statistically significant difference in performance between our newly proposed supervised autoencoders and all non-integrative baseline models (that is, \gls{rsf}, \gls{cox} and \gls{lasso}).

Afterward, we tried to understand our models better by applying a few different techniques. Note that due to the sheer number of models we have in total (six), we only investigated these topics for the non-residual version of our best model since the differences overall were minor (data not shown). In addition, since it is impossible to show results for every single cancer when investigating a latent space, we restricted ourselves to two cancers. We visualized the information contained in the latent space for the best test split of \gls{blca} and \gls{kirp} to understand better what kind of information is learned and carried through the model.

Lastly, we show the results of our Lasso surrogate models. We only fitted the surrogate models on our best overall model variant, as this would be done in a clinical setting. For the surrogate results, we show the performance of the resulting surrogates in terms of approximating our architectures (as measured by train and test $R^2$) and in terms of test concordance. Furthermore, we investigated the sparsity levels of the surrogate models and compared them to (non-surrogate) Lasso. Finally, we checked the performance of arbitrary survival models fitted using the feature set discovered by the surrogate models to deconstruct to which extent multi-omics integration can be reduced to feature selection.

\section{Performance on \gls{tcga} datasets}
\label{sec:tcga-performance}
We start this section by comparing the performances of our three proposed architectures to each other before evaluating the effect of the residual variants and lastly putting the overall performance of our models into perspective by comparing them to the reference models.

\begin{table}
\caption{Test concordance of all methods across the 10 test splits of all 17 considered TCGA cancers rounded to three decimals. Best value bolded by column.}
\label{tab:tcga-mo-performance-overall}
\centering
\begin{tabular}[t]{cccc}
\toprule
Model & Mean & Median & SD\\
\midrule
SHAE (ours) & 0.661 & 0.656 & 0.111\\
SHAE residual (ours) & 0.660 & \textbf{0.657} & 0.113\\
MSAE (ours) & 0.627 & 0.613 & 0.111\\
MSAE residual (ours) & 0.636 & 0.627 & 0.114\\
CSAE (ours) & 0.620 & 0.612 & \textbf{0.100}\\
\addlinespace
CSAE residual (ours) & 0.621 & 0.613 & \textbf{0.100}\\
BF & 0.649 & 0.637 & 0.118\\
RB favoring & \textbf{0.663} & 0.648 & 0.113\\
RSF & 0.595 & 0.578 & 0.125\\
Lasso & 0.614 & 0.590 & 0.120\\
\addlinespace
Clin. Cox PH & 0.622 & 0.626 & 0.112\\
\bottomrule
\end{tabular}
\end{table}

\subsection{Performance of \gls{shae}, \gls{msae} and \gls{csae} compared}
Overall, the \glsxtrfull{shae} performed by far the best out of our three proposed architectures, outperforming the \glsxtrfull{msae} and the \glsxtrfull{csae} in terms of both overall concordance across all cancers (Figure \ref{fig:overall-tcga-performance} and Table \ref{tab:tcga-mo-performance-overall}) and mean rank per cancer (Figure \ref{fig:tcga-ranks}).

We found that even on a per-cancer level, neither \gls{msae} nor \gls{csae} were able to outperform \gls{shae} consistently, except on very rare occasions such as \gls{lgg} (at least for \gls{msae}) (Figure \ref{fig:overall-tcga-performance-by-cancer} and Table \ref{tab:tcga-mean-performance}). These results were consistent with those of \citet{simidjievski2019variational} who similarly found that their hierarchical autoencoder architecture worked best for multi-omics integration on a different task.

\subsection{Performance of residual architecture variants}
The residual variants of \gls{shae} and \gls{csae} performed virtually identical to their non-residual version in terms of both overall concordance (Figure \ref{fig:overall-tcga-performance} and Table \ref{tab:tcga-mo-performance-overall}) and rank (Figure \ref{fig:tcga-ranks}).\footnote{Although the ranks sometimes suggests that there was a difference between models, this should be interpreted with caution, as even a tiny performance delta will show up in terms of higher rank. Thus, Figure \ref{fig:tcga-ranks} should always be investigated in conjunction with Figure \ref{fig:overall-tcga-performance} or Table \ref{tab:tcga-mo-performance-overall}.}For \gls{msae} on the other hand, there was a noticeable difference in terms of both overall concordance (Figure \ref{fig:overall-tcga-performance} and Table \ref{tab:tcga-mo-performance-overall}) and rank (Figure \ref{fig:tcga-ranks}). We will not muse on potential reasons for this and leave this instead to the Discussion in Chapter \ref{sec:discussion}.

Somewhat interestingly, we did not see a one-to-one correspondence between \gls{cox} performing especially well and the residual variants of our architectures outperforming the non-residual variants. On \gls{ucec} for example, even though \gls{cox} performed quite well compared to the other models, \gls{shae} residual underperformed \gls{shae} (Figure \ref{fig:overall-tcga-performance-by-cancer} and Table \ref{tab:tcga-mean-performance}).\footnote{If \gls{cox} performs especially well on a cancer, we would naturally expect the skipping of clinical variables to improve concordance more than on cancers where clinical data has limited predictive value.} There were also examples to the contrary, but the pattern was less strong than expected.

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_tcga_overall_multi_omics_performance.png} \caption{Test concordance of all methods across the 10 test splits of all 17 considered TCGA cancers.}\label{fig:overall-tcga-performance}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_tcga_mean_ranks.png} \caption{Mean test concordance rank per cancer across the 17 considered \gls{tcga} cancers (lower is better).}\label{fig:tcga-ranks}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_tcga_overall_performance_by_cancer.png} \caption{Test concordance of all methods across the 10 test splits of each of the 17 considered TCGA cancers.}\label{fig:overall-tcga-performance-by-cancer}
\end{figure}

\subsection{Performance of the supervised autoencoders relative to the non-multi-omics reference models}
We tested for an overall statistically significant outperformance of each of our proposed architectures relative to each non-multi-omics baseline method as described in Chapter \ref{sec:testing}. We found that none of our models was able to statistically significantly outperform \gls{cox} after multiple testing correction (Table \ref{tab:stat-sig}). This may have partially been due to the number of tests performed (a total of $6 * 3 = 18$ tests were performed).
Both \gls{shae} and \gls{shae} residual statistically significantly outperformed both \gls{lasso} and \gls{rsf} after multiple testing correction, validating that \gls{shae} performed its job of multi-omics integration. Furthermore, we saw a clear gain in performance when comparing \gls{shae} and \gls{shae} residual to  \gls{cox}, even though this difference was not statistically significant after correction (Table \ref{tab:tcga-mo-performance-overall} and Figure \ref{fig:overall-tcga-performance}).

With the other two architectures, \gls{msae} and \gls{csae}, the situation was a bit more nuanced. For \gls{csae}, neither the regular variant nor the residual version was able to statistically significantly outperform any of the non-multi-omics baseline methods after correction. For \gls{msae}, the situation was very similar, except that \gls{msae} residual statistically significantly outperformed \gls{rsf} after correction at the $10\%$ level (Table \ref{tab:stat-sig}).

There was no major pattern visible by specific cancers except for the fact that we generally saw a split between cancers on which a dedicated multi-omics integration model (such as \gls{shae}) seemed to help, for example, on \gls{blca} or \gls{hnsc} and those for which it did not make a difference (\emph{\emph{e.g.,}} \gls{sarc}).

\subsection{Performance of the supervised autoencoders compared to \glsxtrfull{bf} and its variants}

\begin{table}
\caption{Mean test concordance of all methods across the 10 test splits of each of the 17 considered TCGA cancers rounded to three decimals. Best value bolded by column.}
\centering
\label{tab:tcga-mean-performance}
\resizebox{\linewidth}{!}{
\begin{tabular}[t]{cccccccccccccccccc}
\toprule
Model & BLCA & BRCA & COAD & ESCA & HNSC & KIRC & KIRP & LGG & LIHC & LUAD & LUSC & OV & PAAD & SARC & SKCM & STAD & UCEC\\
\midrule
SHAE (ours) & \textbf{0.655} & 0.636 & 0.711 & \textbf{0.648} & 0.639 & 0.765 & 0.853 & 0.800 & 0.565 & 0.634 & 0.597 & 0.600 & 0.538 & 0.638 & 0.639 & \textbf{0.559} & 0.751\\
SHAE residual (ours) & 0.653 & 0.644 & 0.724 & 0.641 & \textbf{0.641} & 0.766 & \textbf{0.854} & 0.815 & 0.565 & 0.638 & 0.588 & 0.602 & 0.535 & 0.615 & 0.645 & 0.556 & 0.739\\
MSAE (ours) & 0.622 & 0.624 & 0.652 & 0.596 & 0.591 & 0.681 & 0.827 & 0.821 & 0.557 & 0.598 & 0.558 & 0.564 & 0.526 & 0.628 & 0.643 & 0.519 & 0.648\\
MSAE residual (ours) & 0.624 & 0.642 & 0.684 & 0.569 & 0.629 & 0.700 & 0.829 & 0.821 & 0.562 & 0.604 & 0.591 & 0.584 & 0.519 & 0.612 & \textbf{0.649} & 0.521 & 0.671\\
CSAE (ours) & 0.601 & 0.596 & 0.655 & 0.626 & 0.562 & 0.622 & 0.760 & 0.809 & 0.522 & 0.607 & 0.589 & \textbf{0.627} & 0.509 & 0.610 & 0.626 & 0.554 & 0.668\\
\addlinespace
CSAE residual (ours) & 0.600 & 0.599 & 0.660 & 0.631 & 0.568 & 0.628 & 0.761 & 0.808 & 0.517 & 0.612 & 0.592 & 0.619 & 0.496 & 0.622 & 0.626 & 0.553 & 0.664\\
BF & 0.624 & 0.626 & 0.696 & 0.525 & 0.602 & 0.779 & 0.852 & 0.840 & 0.566 & 0.651 & 0.573 & 0.584 & 0.539 & \textbf{0.678} & 0.620 & 0.541 & 0.731\\
RB favoring & 0.633 & \textbf{0.655} & 0.717 & 0.552 & 0.634 & \textbf{0.784} & 0.853 & \textbf{0.851} & \textbf{0.603} & \textbf{0.655} & 0.602 & 0.613 & 0.535 & 0.660 & 0.649 & 0.557 & 0.724\\
RSF & 0.532 & 0.591 & 0.594 & 0.500 & 0.494 & 0.668 & 0.834 & 0.837 & 0.594 & 0.551 & 0.471 & 0.489 & \textbf{0.555} & 0.644 & 0.569 & 0.521 & 0.667\\
Lasso & 0.593 & 0.556 & 0.667 & 0.505 & 0.540 & 0.778 & 0.812 & 0.847 & 0.575 & 0.574 & 0.497 & 0.535 & 0.553 & 0.631 & 0.596 & 0.518 & 0.658\\
\addlinespace
Clin. Cox PH & 0.642 & 0.629 & \textbf{0.731} & 0.575 & 0.594 & 0.760 & 0.590 & 0.759 & 0.542 & 0.641 & \textbf{0.614} & 0.610 & 0.533 & 0.453 & 0.629 & 0.528 & \textbf{0.752}\\
\bottomrule
\end{tabular}
}
\end{table}

\begin{table}
\caption{P-Values of testing the non-inferiority of each baseline method (rounded to four digits). $H_0$: The baseline method performed as well as our respective model it is being compared to. Bonferroni-Holm \citep{holm1979simple} used for multiple testing correction.}
\label{tab:stat-sig}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}[t]{cccc}
\toprule
Model & Comparison model & P-Value (before correction) & P-Value (after correction)\\
\midrule
SHAE (ours) & RSF & 0.0002 & 0.0041\\
SHAE (ours) & Lasso & 0.0007 & 0.0098\\
SHAE (ours) & Clin. Cox PH & 0.0265 & 0.2911\\
SHAE residual (ours) & RSF & 0.0003 & 0.0045\\
SHAE residual (ours) & Lasso & 0.0006 & 0.0095\\
\addlinespace
SHAE residual (ours) & Clin. Cox PH & 0.0243 & 0.2911\\
MSAE (ours) & RSF & 0.0079 & 0.1027\\
MSAE (ours) & Lasso & 0.1274 & 0.8919\\
MSAE (ours) & Clin. Cox PH & 0.4215 & 1.0000\\
MSAE residual (ours) & RSF & 0.0040 & 0.0560\\
\addlinespace
MSAE residual (ours) & Lasso & 0.0352 & 0.3516\\
MSAE residual (ours) & Clin. Cox PH & 0.2446 & 1.0000\\
CSAE (ours) & RSF & 0.0742 & 0.6360\\
CSAE (ours) & Lasso & 0.3491 & 1.0000\\
CSAE (ours) & Clin. Cox PH & 0.5475 & 1.0000\\
\addlinespace
CSAE residual (ours) & RSF & 0.0707 & 0.6360\\
CSAE residual (ours) & Lasso & 0.3314 & 1.0000\\
CSAE residual (ours) & Clin. Cox PH & 0.5302 & 1.0000\\
\bottomrule
\end{tabular}
}
\end{table}

When comparing our newly suggested architectures to \gls{bf} and \gls{rbf}, we found that \gls{shae} and \gls{shae} residual alone were able to perform on par with them. In particular, both of them performed better than \gls{bf} in terms of overall concordance (Figure \ref{fig:overall-tcga-performance} and Table \ref{tab:tcga-mo-performance-overall}) and rank (Figure \ref{fig:tcga-ranks}). \gls{rbf} performed very similar to \gls{shae} and its residual variant in terms of overall concordance across all cancers (Figure \ref{fig:overall-tcga-performance} and Table \ref{tab:tcga-mo-performance-overall}), but was able to capture the best performance in terms of mean concordance rank per dataset (Figure \ref{fig:tcga-ranks}). When investigating the performance per individual cancers, we found that for most cancers, the performance of \gls{shae} (residual), \gls{bf} and \gls{rbf} was qualitatively similar (Table \ref{tab:tcga-mean-performance}). There were however a few examples on which one of the models four clearly outperformed the others: On Esophageal carcinoma (ESCA), \gls{shae} (and in fact even \gls{msae} and \gls{csae}) \gls{shae} strongly outperformed not just \gls{bf} and \gls{rbf} but in fact all other non-autoencoder models. A few similar existed for \gls{bf} and \gls{rbf}: On \gls{kirc} and \gls{lgg} we clearly saw \gls{bf} and \gls{rbf} outperform all of the neural models.

\gls{msae}, \gls{csae} and their respective residual versions did not perform well when compared to either \gls{bf} or \gls{rbf}. All of them were clearly outperformed by \gls{bf} and especially \gls{rbf} in terms of overall concordance (Figure \ref{fig:overall-tcga-performance} and Table \ref{tab:tcga-mo-performance-overall}) and rank (Figure \ref{fig:tcga-ranks}). Only \gls{msae} residual performed somewhat close to \gls{bf} in terms of overall concordance but was still inferior in terms of rank. When investigating the performances by individual cancers, we found that \gls{msae} and \gls{csae} underperformed \gls{bf} and \gls{rbf} on virtually all cancers, save for \gls{esca}, on which all supervised autoencoders performed especially well (Table \ref{tab:tcga-mean-performance} and Figure \ref{fig:overall-tcga-performance-by-cancer}).

Overall, we found that we could classify the $17$ considered \gls{tcga} cancers roughly into three different categories:

\begin{enumerate}
    \item Cancers on which (proper) multi-omics integration models outperformed non-integrative multi-omics models but not \gls{cox}. Examples of this included \gls{blca} and \gls{coad}.
    \item Cancers on which non-integrative multi-omics models performed as well as multi-omics integration models but \gls{cox} underperformed. Examples for this group included \gls{lgg} and \gls{kirp}.
    \item Cancers on which all three groups of models performed roughly equally. Examples for this one included \gls{paad} and \gls{lihc}.
\end{enumerate}

We found that \gls{shae} and \gls{shae} residual belonged to the same group as \gls{bf} and \gls{rbf} in that they behaved all in a similar manner of well-working multi-omics methods. \gls{msae} and \gls{csae} were a bit harder to pin down in that they sometimes performed similar to \gls{shae}, \gls{rbf} and \gls{bf} (\emph{e.g.,} on \gls{blca}) but sometimes also strongly underperformed them (\emph{e.g.,} on \gls{kirc}). In that sense, we would put \gls{msae} and \gls{csae} in a separate category of multi-omics integration methods which do not work consistently. They could perhaps be grouped with some of the less well-working variants of \gls{bf} introduced in the study of \citet{hornung2019block}.

\subsection{Computation times}
Lastly, before we move on, we briefly look at the computation times of all respective models. Overall, \gls{bf} and \gls{rbf} were by far the slowest models, which is consistent with the timings done in the study by \citet{herrmann2021large} (Figure \ref{fig:computation-times}). Interestingly, we found that \gls{rbf} was quite a bit faster than \gls{bf} which could be due to the fact the clinical block does not have to be considered in sampling but is instead included mandatorily at every split.

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_computation_times.png} \caption{Mean computation time in seconds across the 10 test splits of each of the 17 considered \gls{tcga} cancers (lower is better).}\label{fig:computation-times}
\end{figure}

The supervised autoencoder architectures were the next slowest, with \gls{csae} and \gls{csae} residual (they had virtually identical computation times) the slowest, likely due to the high dimensionality of their second level latent space. \gls{shae} and \gls{msae} came next, although it should be noted that for \gls{msae}, we only had to tune one parameter instead of two (since \gls{msae} does not use \gls{sgl}) which naturally made it faster than the other two architectures. Interestingly, for \gls{msae}, the difference between its residual version and the regular version was quite noticeable, which was not the case for \gls{shae} and \gls{csae}. Of course, \gls{rsf}, \gls{lasso} and \gls{cox} were the fastest models across all datasets, with \gls{cox} always being the fastest, followed by \gls{rsf} and \gls{lasso}. These fast computation times of all non-integrative models came at the price of decreased concordance (Figure \ref{sec:tcga-performance}), except for \gls{cox} which performed quite well overall, despite its virtually instant computation times.

The ordering of methods in terms of computation times stayed constant across all datasets (except for one negligible switch between \gls{lasso} and \gls{rsf}), suggesting that all methods scaled similarly across datasets of different sizes. We note that \gls{shae} (residual) performed approximately equal to \gls{rbf} while achieving mean computation times across splits which were on average a factor of four times lower. Although \citet{hornung2019block} argued in their study that the speed of \gls{bf} and its variants could be improved by performing feature selection on the higher dimensional blocks (since they contain redundant information), the same argument holds for \gls{shae}, thus likely keeping overall computation times in a similar range.

Since our overall best models were \gls{shae} and \gls{shae} residual, we restrict the following investigations in the main text to \gls{shae}.

\section{Learned representations}
We now investigate the learned representations of our best model, \gls{shae} on two exemplary cancers.\footnote{Since \gls{shae} and \gls{shae} residual were so similar, we only investigated the non-residual variant here and added a short section on differences between the residual and non-residual variant at the end of this chapter.} Specifically, we picked \gls{blca} and \gls{kirp} since they exemplify two of the main variants of cancers we identified in the previous section.  We argue that studying these two cancers gives a representative picture of the subset of cancers considered here. Specifically, on \gls{blca}, \gls{cox} performed as well or better than the multi-omics models (Figure \ref{fig:overall-tcga-performance-by-cancer} and Table \ref{tab:tcga-mo-performance-overall}), suggesting that there was limited additional information contained in the molecular omics groups. On \gls{kirp}, on the other hand, all multi-omics models (even the non-integrative ones) strongly outperformed \gls{cox}, suggesting that there was considerable additional information contained in the molecular data. When this added additional information, we also tied the learned latent representations back to group-wise permutation importance (Algorithm \ref{alg:fi-group}) to validate whether the learned representation matched the feature importances calculated through permutation.

Throughout this section, we show results only for the best split of \gls{shae} (in terms of test concordance) since the data for other splits did not differ in a meaningful way (data not shown). Tables S1-S3 contain results for the top correlated features with each latent space dimension for the best split of all cancers and the train- and test-permutation importances for the best split of every cancer (for \gls{shae} only).

\subsection{\gls{blca}}
\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_shae_heatmap_blca.png} \caption{Correlation heatmap of all features which had the highest correlation with at least one of the 64 latent space dimensions of \gls{shae} on the best test split of \glsxtrfull{blca}.}\label{fig:BLCA-SHAE-heatmap}
\end{figure}

The highest correlated features for each latent dimension for the best split of bladder cancer included only features belonging to methylation, clinical, and gene expression (Figure \ref{fig:BLCA-SHAE-heatmap}). This does not necessarily imply that the other variable groups, such as mutation, were disregarded by \gls{shae}; since we only looked at the top correlated feature for each dimension, this did not capture all information present in latent space.

Nevertheless, the features with the highest correlations included multiple with proven consequences for survival. Almost trivially, all clinical features contained in the top correlated features are relevant for survival in bladder cancer. The \citet{ncipathologicstage2021} defines pathologic stage as the "stage of cancer (amount or spread of cancer in the body) that is based on how different from normal the cells in samples of tissue look under a microscope", which is highly relevant to survival. Diagnosis subtype, that is, whether a bladder cancer was papillary or non-papillary, is also highly clinically relevant, where the \citet{ncipapillary2021} defines a papillary tumor as a "tumor that looks like long, thin 'finger-like' growths." Non-papillary tumors have been shown to imply an increased risk of progression and may act as an independent prognostic factor for disease-free survival \citep{andius2007prognostic}. The female gender has also been shown to be associated with worse survival rates in \gls{blca} \citep{mungan2000gender, dobruch2016gender}, which may not be fully explained by more "by the more frequent diagnosis of higher stages at first presentation among women" \citep{mungan2000gender}. Interestingly, \citet{datta2006gender} found that when adjusting for socioeconomic factors, five-year survival rates were not significantly different for men and women in pathologic stages one to three but stayed significantly different for patients in stage four. This additionally validates the fact that our model's latent space contained not just gender but also pathologic stage (Figure \ref{fig:BLCA-SHAE-heatmap}).

The remaining features primarily contained in the latent space were all strongly associated with gender (Figure \ref{fig:BLCA-SHAE-heatmap}). All captured methylation features overlap genes on the $X$ chromosome. The single gene expression feature captured among the highest correlated features with the latent space, CYorf15B, has been shown to differ significantly between male and female cancer patients \citep{tabernero2007patient}. Some of these features have also been shown to potentially have independent prognostic value, however: For example, L1CAM (overlapped by cg10926623) methylation has been shown to have an inverse relationship with L1CAM gene expression in cancers \citep{notaro2016evaluating}, which is associated with bad prognosis \citep{schirmer2013epigenetic}. Future work is needed to disentangle the directionality of these relationships, \emph{i.e.,} whether the methylation and gene expression features contained in our model's latent space were present \emph{only} due to a strong correlation with gender or whether they were genuinely independent prognostic factors.

The group-wise permutation importance for \gls{blca} revealed no additional insights, similarly suggesting that \gls{shae} was relying virtually exclusively on clinical data to predict survival (Figure \ref{fig:SHAE-BLCA-permutation-importance}).

\gls{shae} also captured pathway activations in its latent space (Figure \ref{fig:BLCA-SHAE-pathways}). Out of the $50$ pathways in the hallmarks of cancers gene sets, $36$ were enriched in at least one of the latent dimensions of \gls{shae} in the best split of \gls{blca}. This behavior of strong enrichment was coherent with \citet{ching2018cox}, and \citet{kim2020improved}, who both performed GSEA on \emph{KEGG} pathways and found that most pathways were enriched in at least one of their top hidden nodes.\footnote{Both \citet{ching2018cox} and \citet{kim2020improved} only looked at a certain number of hidden nodes which had maximal variance among all nodes, which they deemed the "top" nodes.}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_shae_pathways_blca.png} \caption{Enriched hallmarks of cancer pathways for each of the 64 latent dimension of  \gls{shae} on the best split of \glsxtrfull{blca}.}\label{fig:BLCA-SHAE-pathways}
\end{figure}

\subsection{\glsxtrfull{kirp}}
\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_shae_heatmap_kirp.png} \caption{Correlation heatmap of all features which had the highest correlation with at least one of the 64 latent space dimensions of \gls{shae} on the best test split of \glsxtrfull{kirp}.}\label{fig:KIRP-SHAE-heatmap}
\end{figure}

Even though \gls{cox} performed much worse on \gls{kirp} compared to \gls{blca}, we saw a similar picture when investigating the latent space of \gls{kirp} on the best split for \gls{shae} (Figure \ref{fig:KIRP-SHAE-heatmap}). In particular, the top correlated features with the latent space contained various clinical features like age, gender, and laterality. Similar to bladder cancer, gender is a vital prognostic factor in \gls{kirp}, with men not only being diagnosed more frequently but also having worse outcomes overall \citep{mancini2020gender, rampersaud2014effect}. In addition, \citet{rampersaud2014effect} showed that not only is age an independent prognostic factor for women (but not men), but also that survival differences between men and women were no longer statistically significant for patients older than $59$. This validates the fact that \gls{shae} picked up on clinical features in our study since both for \gls{blca} and \gls{kirp}, direct clinical connections (between gender and stage for \gls{blca} and between age and gender for \gls{kirp}) were apparent in the top correlated features of the models latent space. Similar to the latent space for \gls{blca}, the one for \gls{kirp} again contained mainly methylation features that overlap genes contained on the $X$ chromosome. Another clinical feature with prognostic value included laterality, which \citet{roychoudhuri2006cancer} showed to be relevant for cancers of virtually every paired organ (including the kidneys).

There were only a few other features that were among the highest correlated with any of the latent space dimensions, despite \gls{cox}'s relatively poor performance on \gls{kirp} (compared to models having access to multi-omics data). These included multiple potentially cancer-associated features, all from the gene expression group: GPATCH2, a gene newly discovered only a bit over a decade ago, has been shown to be highly expressed in breast cancer patient samples \citep{lin2009involvement}. It was also demonstrated that downregulation of GPATCH2 expression resulted in decreased growth of breast cancer cells \citep{lin2009involvement}. The SHMT1 gene plays a role in various human cancers such as lung \citep{paone2014shmt1}, liver \citep{dou2019shmt1}, and ovarian \citep{gupta2017serine}, although no involvement in kidney cancer has been reported thus far. Lastly, the KHSRP gene has similarly been found to play a role in cancer invasion. In particular low expression levels were shown to be associated with better prognosis in lung cancer and other cancers \citep{yan2019rna, briata2016diverse}.

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_shae_permutation_importance_kirp.png} \caption{Group permutation feature importance for each input group of \gls{shae} on the best split of \glsxtrfull{kirp} using 10 permutations.}\label{fig:SHAE-KIRP-permutation-importance}
\end{figure}

The permutation importance for kidney cancer told a more nuanced picture than the latent space's heatmap. The train permutation importance suggested that mutation was the most crucial feature group. Still, when comparing this to the test permutation importance (which we care about in the end, the predictions on the test set, that is), we quickly saw that the situation was more complicated (Figure \ref{fig:SHAE-KIRP-permutation-importance}). Specifically, clinical was still one of the most critical input groups (which matches the many clinical features in Figure \ref{fig:KIRP-SHAE-heatmap}), but gene expression, methylation, miRNA, and especially \gls{rppa} also seemed to play a prominent role in the prediction of survival in \gls{kirp}. We did not see any features from \gls{mirna} or \gls{rppa} reflected in the top correlated feature set with the latent space (Figure \ref{fig:KIRP-SHAE-heatmap}), even though these also had reasonably high group permutation importance (Figure \ref{fig:SHAE-KIRP-permutation-importance}). This result drives home the point that while plotting the top correlated features in a heatmap can give an interesting first overview of the latent space learned by the model, this kind of plot is in no way a perfect representation of all features used by the model. At the same time, the permutation importance showed that \gls{shae} was relying on most (although not quite all, \gls{cnv} and mutation had a test permutation importance close to one) input omics groups on \gls{kirp} and was thus presumably able to achieve good performance because of this.

\gls{shae} similarly captured pathway activations for \gls{kirp}. Of course, some of the activations themselves were different (because \gls{kirp} is another cancer, and thus the specific pathways which are essential would be different) (Figure \ref{fig:KIRP-SHAE-pathways}).

The latent spaces of \gls{shae} residual looked virtually identical to that of the non-residual variant on both \gls{blca} (Figure \ref{fig:BLCA-SHAE-residual-heatmap}) and \gls{kirp} (Figure \ref{fig:KIRP-SHAE-residual-heatmap}). This may come as somewhat of a surprise as the goal of the residual models was that clinical data would not have to be "carried through" the model and into the latent space (or only insofar as it was useful to learn connections to other features). We will discuss this issue further in Chapter \ref{sec:discussion}.

Overall, within this section, we showed that \gls{shae} learned distinct clinical (and, at times, molecular) features relevant for cancer survival. In addition, we showed that \gls{shae} captured pathway activity, even if gene expression was not among the feature groups most highly correlated with its latent space.

\section{Surrogate models}
\label{sec:surrogate-results}
Recall that to judge the clinical applicability of our best architecture, the \glsxtrfull{shae}, we fitted surrogate Lasso linear regression models on clinical and gene expression only. We then compared their performance to all other models trained on clinical and gene expression only. Note that we fitted the Lasso with no penalties on the clinical variables (using the \emph{relative\_penalties} argument in \emph{python-glmnet} \citep{glmnet, py-glmnet}). We did not fit any surrogate models for \gls{msae} or \gls{csae}, since \gls{shae} was shown to perform much better on \gls{tcga} in Chapter \ref{sec:tcga-performance}.

\begin{table}
\caption{Test concordance of all methods trained on clinical and gene expression only across the 10 test splits of all 17 considered TCGA cancers rounded to three decimals. Clin. Cox PH trained on clinical only. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only. Best value bolded by column.}
\label{tab:overall-perform-surrogates}
\centering
\begin{tabular}[t]{cccc}
\toprule
Model & Mean & Median & SD\\
\midrule
SHAE (ours) & 0.643 & 0.636 & 0.116\\
SHAE residual (ours) & 0.641 & 0.635 & 0.113\\
SHAE surrogate (ours) & 0.657 & \textbf{0.652} & \textbf{0.107}\\
SHAE residual surrogate (ours) & 0.660 & 0.649 & 0.109\\
BF & \textbf{0.663} & 0.651 & 0.113\\
\addlinespace
RB favoring & 0.652 & 0.633 & 0.113\\
RSF & 0.610 & 0.598 & 0.117\\
Lasso & 0.623 & 0.610 & 0.119\\
Clin. Cox PH & 0.622 & 0.626 & 0.112\\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_surrogate_performance.png} \caption{Test concordance of all methods trained on clinical and gene expression only across the 10 test splits of all 17 considered TCGA cancers. Clin. Cox PH trained on clinical only. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only.}\label{fig:surrogate-performance}
\end{figure}

The performance of our surrogate models exceeded that of any other model trained on clinical and gene expression except \gls{bf} (Figure \ref{fig:surrogate-performance} and Table \ref{tab:overall-perform-surrogates}). However, \gls{bf} beat both of the surrogate models in terms of mean concordance, while it achieved roughly the same median concordance. All multi-omics models and surrogate models beat \gls{lasso}, \gls{rsf} and \gls{cox} when trained on clinical and gene expression only (Figure \ref{fig:surrogate-performance} and Table \ref{tab:overall-perform-surrogates}).

Interestingly, \gls{bf} actually performed \emph{better} with only clinical and gene expression relative to the \gls{bf} model with multi-omics data (Table \ref{tab:tcga-mo-performance-overall} and Table \ref{tab:overall-perform-surrogates}). To be specific, \gls{bf} achieved an approximately $0.014$ higher mean as well as $0.014$ higher median concordance across all \gls{tcga} cancers when trained on clinical and gene expression only, relative to the full multi-omics model. With that, it would have been about as good as the best models (\gls{shae} and \gls{rbf}) on the multi-omics data. This result is in line with what \citet{hornung2019block} found in their study. \citet{hornung2019block} showed that \gls{bf} performed better when trained on clinical and gene expression only (relative to multi-omics) and stated that this might be due to the limited information contained in groups beyond clinical and gene expression. That said, it is not always straightforward which input variable groups are important for each cancer, and so expecting researchers or clinicians to choose which groups to use \emph{a priori} might be challenging.

\begin{table}
\caption{Mean test concordance of all methods trained on clinical and gene expression only across the 10 test splits of each of the 17 considered TCGA cancers rounded to three decimals. Clin. Cox PH trained on clinical only. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only. Best value bolded by column.}
\label{tab:surrogate-performance-by-cancer}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}[t]{cccccccccccccccccc}
\toprule
Model & BLCA & BRCA & COAD & ESCA & HNSC & KIRC & KIRP & LGG & LIHC & LUAD & LUSC & OV & PAAD & SARC & SKCM & STAD & UCEC\\
\midrule
SHAE (ours) & 0.652 & 0.616 & 0.722 & 0.498 & 0.589 & 0.771 & 0.752 & 0.842 & 0.575 & 0.646 & \textbf{0.628} & 0.593 & 0.536 & 0.594 & 0.636 & 0.546 & 0.735\\
SHAE residual (ours) & 0.653 & 0.613 & 0.729 & 0.515 & 0.599 & 0.769 & 0.749 & 0.838 & 0.556 & 0.645 & 0.627 & 0.602 & 0.536 & 0.558 & 0.631 & 0.549 & 0.727\\
SHAE surrogate (ours) & 0.650 & 0.630 & \textbf{0.746} & 0.588 & 0.628 & 0.773 & 0.820 & 0.770 & 0.604 & 0.640 & 0.592 & 0.629 & 0.543 & 0.605 & \textbf{0.657} & \textbf{0.568} & 0.733\\
SHAE residual surrogate (ours) & \textbf{0.655} & 0.638 & 0.744 & \textbf{0.595} & \textbf{0.645} & 0.781 & 0.814 & 0.795 & 0.610 & 0.644 & 0.583 & \textbf{0.639} & 0.521 & 0.608 & 0.647 & 0.565 & 0.730\\
BF & 0.637 & \textbf{0.644} & 0.708 & 0.564 & 0.568 & 0.784 & 0.825 & 0.844 & 0.615 & \textbf{0.679} & 0.572 & 0.611 & \textbf{0.599} & 0.658 & 0.640 & 0.560 & \textbf{0.757}\\
\addlinespace
RB favoring & 0.635 & 0.631 & 0.710 & 0.569 & 0.581 & \textbf{0.789} & \textbf{0.829} & \textbf{0.847} & \textbf{0.622} & 0.621 & 0.544 & 0.575 & 0.583 & 0.664 & 0.623 & 0.554 & 0.713\\
RSF & 0.572 & 0.595 & 0.621 & 0.584 & 0.463 & 0.694 & 0.810 & 0.832 & 0.609 & 0.545 & 0.495 & 0.513 & 0.579 & \textbf{0.666} & 0.588 & 0.541 & 0.662\\
Lasso & 0.584 & 0.589 & 0.662 & 0.470 & 0.597 & 0.784 & 0.822 & 0.830 & 0.568 & 0.592 & 0.527 & 0.552 & 0.563 & 0.655 & 0.606 & 0.545 & 0.645\\
Clin. Cox PH & 0.642 & 0.629 & 0.731 & 0.575 & 0.594 & 0.760 & 0.590 & 0.759 & 0.542 & 0.641 & 0.614 & 0.610 & 0.533 & 0.453 & 0.629 & 0.528 & 0.752\\
\bottomrule
\end{tabular}
}
\end{table}

Since we fitted our surrogate models by leaving the clinical variables unpenalized, we also compared the performance in terms of concordance to a \gls{lasso} model fit on clinical and gene expression, which did not penalize the clinical variables (that is, not a surrogate model, but a Lasso model directly predicting survival). Our surrogate models clearly outperformed Lasso variant which left clinical variables unpenalized (Figure \ref{fig:surrogate-performance-favor-clinical}). Lasso with unpenalized clinical variables did, however, perform slightly better than Lasso with penalized clinical variables, albeit with higher variance (Figure \ref{fig:surrogate-performance-favor-clinical}). Lasso with unpenalized clinical variables still performed slightly worse than \gls{cox}, again with higher variance.

Thus, our surrogate models did not merely perform well due to leaving the clinical variables unpenalized, as they easily outperformed Lasso with unpenalized clinical variables. The inferior performance of Lasso with unpenalized clinical variables seemed to be primarily due to a few cancers on which it virtually failed completely (for example, \gls{sarc}, \gls{esca} and \gls{lusc}) (Figure \ref{fig:surrogate-performance-favor-clinical-by-cancer}).

\begin{figure}[h]
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_surrogate_performance_favor_lasso.png} \caption{Test concordance of all methods trained on clinical and gene expression only across the 10 test splits of all 17 considered TCGA cancers. Clin. Cox PH trained on clinical only. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only. Lasso unpenalized clinical trained on clinical and gene expression only to predict survival while leaving clinical variables unpenalized. Surrogate models also left clinical variabes unpenalized.}\label{fig:surrogate-performance-favor-clinical}
\end{figure}

Moving on to the surrogate performance of our models, that is how well the surrogate models performed in reproducing the predictions of the \gls{shae} and \gls{shae} residual models in terms of $R^2$. The training $R^2$ of our surrogate models for most cancers was generally quite good, with an overall median train $R^2$ across all cancers and splits of $0.909$ for the \gls{shae} surrogate, and $0.934$ for the \gls{shae} residual surrogate. However, closer inspection revealed that there was some more nuance. While for most cancers, the surrogate models did perform well in terms of training $R^2$, there were a few cancers where it did quite badly (Figure \ref{fig:surrogate-train-r2}). These cancers were more-or-less consistent across both surrogate models and included \gls{esca}, \gls{lgg} (for the non-residual surrogate) and \gls{lusc} (although to a lesser extent). While \gls{lusc} and \gls{paad} were still acceptable with at least a median train $R^2$ above $0.5$, the train $R^2$ for \gls{esca} was below $0.5$ for both models.

\begin{figure}[h]
    \centering
        \includegraphics[width=15.7463cm,height=10.49753cm]{./content/figures/fig_plot_surrogate_train_r2.png} \caption{Train $R^2$ of the \gls{shae} (residual) surrogate models across the 10 test splits of each of the 17 considered \gls{tcga} cancers.}\label{fig:surrogate-train-r2}
\end{figure}

The test $R^2$ was more mixed overall, although we did see a fairly direct correspondence between high train $R^2$ and high test $R^2$, for example on \gls{blca}, \gls{brca} and \gls{coad} (Figure \ref{fig:surrogate-test-r2}). Even though the surrogate models had quite high train $R^2$ overall, on some cancers they also strongly overfit the training data, as evidenced by particularly low test $R^2$ (relative to the corresponding train $R^2$). Some examples for this included \gls{hnsc} and \gls{ov}, for both of which the surrogate models achieved a train $R^2$ around $0.9$, while only achieving a test $R^2$ below $0.5$. This suggests that while the models effectively reproduced the predictions of \gls{shae} and \gls{shae} residual on the training set for almost all cancers, they struggled to do so on the test set for additional cancers (beyond the set which already had low train $R^2$). Some of the cancers which already had fairly low train $R^2$ predictably turned out with even worse test $R^2$, which was, for example, the case for \gls{esca} (both surrogates had a median test $R^2$ below $0$).

\begin{figure}[h]
    \centering
        \includegraphics[width=15.7463cm,height=10.49753cm]{./content/figures/fig_plot_surrogate_test_r2.png} \caption{Test $R^2$ of the \gls{shae} (residual) surrogate models across the 10 test splits of each of the 17 considered \gls{tcga} cancers.}\label{fig:surrogate-test-r2}
\end{figure}

The fact that most cancers on which the surrogate models achieved a median test $R^2$ of below $0.5$ had a sample size of below $200$ suggests that this issue of low test $R^2$ might be at least partially due to low sample size (Figure \ref{fig:surrogate-test-r2} and Table \ref{tab:tcga-overview}). We saw overall high test $R^2$ values for cancers such as \gls{blca} and \gls{brca}, both of which contained over $300$ samples (Table \ref{tab:tcga-overview}). Additionally, various other cancers (\emph{e.g.,} \gls{skcm}, \gls{stad} and most others) had acceptably high test $R^2$, suggesting that the surrogate models were indeed replicating the predictions of the full \gls{shae} models well, despite only having had access to two input groups (clinical and gene expression).

Interestingly, low test $R^2$ did not necessarily correspond to lower performance relative to the full \gls{shae} model (Figure \ref{fig:surrogate-multi-omics-shae-comparison}). For example, even though the surrogate models clearly overfitted to the training set and thus had quite low test $R^2$ on \gls{ov}, they slightly outperformed the original multi-omics \gls{shae} models in terms of test concordance. The same held for \gls{lihc}. This, in turn, validated that low test $R^2$ does not necessarily imply bad performance on the target task. There were, however, also examples, such as \gls{esca} and \gls{kirp}, where the surrogate models fell short in terms of both test $R^2$ and test concordance.

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_surrogate_performance_compared_to_full_model.png} \caption{Test concordance of all methods trained on clinical and gene expression only across the 10 test splits of all 17 considered TCGA cancers. Clin. Cox PH trained on clinical only. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only. \gls{shae} (residual) multi-omics models trained on full multi-omics data.}\label{fig:surrogate-multi-omics-shae-comparison}
\end{figure}

Overall, our surrogate models were very sparse, with the median number of variables selected being $35$ and $37$ across all cancers for the \gls{shae} surrogate and the \gls{shae} residual surrogate, respectively (Figure \ref{fig:overall-surrogate-sparsity}). Lasso and Lasso with unpenalized clinical variables selected a median number of $34$ and $33$ variables, respectively. Still, they had much higher means and variances in the number of features selected (Figure \ref{fig:overall-surrogate-sparsity}).\footnote{Recall that since clinical variables were unpenalized, they were always selected for both surrogate models and the Lasso model with unpenalized clinical variables.}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_surrogate_overall_sparsity_compared.png} \caption{Overall model sparsity of the surrogate models and the \gls{lasso}. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only. \gls{lasso} trained on clinical and gene expression only to predict survival.}\label{fig:overall-surrogate-sparsity}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_shae_residual_surrogate_sparsity.png} \caption{Model sparsity of the \gls{shae} residual surrogate model across the 10 test splits of each of the 17 \gls{tcga} cancers when trained on clinical and gene expression to predict the predictions $\hat \varphi(\tilde C)$ of \gls{shae} residual.}\label{fig:shae-surrogate-sparsity-by-cancer}
\end{figure}

Overall, we found that the number of features selected by the surrogate models was fairly stable across splits (that is, there was generally a similar number of variables selected across splits) (Figure \ref{fig:shae-surrogate-sparsity-by-cancer}). There were no great differences between the overall coefficient structure between the surrogates for \gls{shae} and \gls{shae} residual, hence we relegated the plot for \gls{shae} to the Supplement (Figure \ref{fig:shae-residual-surrogate-sparsity-by-cancer}). The Lasso more often picked quite different numbers of coefficients between splits, with some splits for a particular cancer having less than ten features while the next had over a hundred selected (Figure \ref{fig:lasso-sparsity-by-cancer}). In addition, it was readily apparent that the \gls{lasso} model picked considerably fewer clinical features compared to both \gls{shae} surrogate and \gls{shae} residual surrogate since the surrogate models did not penalize clinical variables. The Lasso model which did not penalize clinical variables looked similar to the Lasso in that it had considerable jumps in between the number of variables selected across splits (\ref{fig:lasso-sparsity-by-cancer-unpenalized}). Of course, the Lasso model with unpenalized clinical variables chose all clinical variables for all cancers and thus picked more clinical variables overall (relative to the Lasso model with penalized clinical variables).

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_lasso_sparsity.png} \caption{Model sparsity of the \gls{lasso} model across the the 10 test splits of each of the 17 \gls{tcga} cancers when trained on clinical and gene expression to predict survival.}\label{fig:lasso-sparsity-by-cancer}
\end{figure}

Seeing that our surrogate models were able to maintain high performance comparable to black-box multi-omics methods while enabling high sparsity, we were interested in explaining this good performance. As mentioned in Chapter \ref{sec:arch-search}, there are essentially two main possible contributor patterns to the excellent performance achieved by our surrogate models:

\begin{enumerate}
    \item The sparsity pattern, that is, the features selected by our surrogate model.
    \item The magnitude of the coefficients (\emph{i.e.,} weights) learned by our surrogate models.
\end{enumerate}

Most likely is that both of these factors play a role. Nevertheless, we wanted to establish how much the performance came down to the features selected instead of the specific weights learned.

\begin{figure}[h]
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_feature_selection_performance.png} \caption{
        Test concordance of the \gls{shae} residual surrogate model, \gls{cox} and a \gls{rsf} and Ridge regularized Cox PH model across the 10 test splits of all 17 considered TCGA cancers. Clin. Cox PH trained on clinical only. Surrogate trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} residual using clinical and gene expression data only. \gls{rsf} and Ridge regularized Cox PH trained using the same feature set as the \gls{shae} residual model but to predict survival directly.}\label{fig:feature-selection-performance}
\end{figure}

We found that if we fitted a Ridge regularized Cox PH model as well as a \gls{rsf} model with only the features selected by our \gls{shae} residual surrogate model, we were able to outperform \gls{cox}, but not quite reach the same performance as the surrogate model (Figure \ref{fig:feature-selection-performance}). The fact that \gls{rsf} and Ridge regularized Cox PH with feature selection were able to outperform \gls{cox} was primarily due to a few cancers on which \gls{cox} vastly underperformed (presumably due to lack of access to gene expression data) (Figure \ref{fig:feature-selection-performance-by-cancer}). This was the case for example on \gls{kirp}, \gls{sarc} and \gls{lgg}. On most other cancers, the feature selected \gls{rsf} and Ridge models in fact underperform \gls{cox}, while the surrogate model continued to outperform \gls{cox} (Figure \ref{fig:feature-selection-performance-cancers-excluded}).

Thus, using the same features as our surrogate model clearly helped the performance of the \gls{rsf} and Ridge regularized Cox PH models (recall that before feature selection, both \gls{rsf} and \gls{lasso} underperformed \gls{cox}). Specifically, the Ridge with feature selection had median and mean test concordances across all cancers of $0.647$ and $0.635$ while the results for \gls{lasso} fitted on all clinical and gene expression features were $0.623$ and $0.610$ respectively. For \gls{rsf}, the values for the feature selected model were $0.636$ and $0.625$ while the model fitted on all features from the clinical and gene expression groups achieved scores of $0.610$ and $0.598$. However, the feature-selected models did still not perform \emph{on par} with the surrogate model.\footnote{We admit that the comparison between \gls{lasso} and the Ridge model with feature selection is limited since they use different forms of regularization. We did not, however, want to fit a Lasso on the features selected by the surrogate model to ensure that the model used all features (the \gls{lasso} might have set additional coefficients to zero).} This implies that the magnitude of the weights learned by the surrogate model may have played an additional role in their good performance. We will further discuss this result and possible meanings and implications within the Discussion in Chapter \ref{sec:discuss-arch-search}.

Having presented our three main groups of results, we now discuss some of the takeaways of our study before concluding the thesis with a summary.


\graphicspath{ {./img/intro/} }

\chapter{Introduction}

This set of lecture notes for the course {\bf Introduction to the Finite
Element Method} is part of the learning resources used in the flipped
classroom\footnote{``Inverting the classroom means that events that have
traditionally taken place inside the classroom now take place outside the
classroom and vice versa.'' \citep{lage2000}.} version of the subject
adopted at Universidad EAFIT. In the flipped class approach, the students cover
most of the theoretical aspects through independent home study, while the class
time is invested in various learning activities conducted with support from the
instructor and strongly based on peer-to-peer interaction. In this context, by
collecting and summarizing fundamental numerical, theoretical and computational
aspects required in the formulation of finite element algorithms, the lecture
notes are intended to serve as a self-study guide. The notes are not as
rigorous as material existing in published literature and are certainly not
close in quality to excellent textbooks available in the subject.

This introductory version of the course covers fundamental theoretical and
computational aspects required in the formulation of finite element methods as
a numerical solution technique of boundary value problems. Although the studied
algorithms are general the course evolves around the model of the linearized
theory of elasticity. The fundamental theoretical framework is mainly covered
in the lecture notes and some referenced complementary material. The various
theoretical topics are then associated with in-class learning activities
involving some degree of computational work. Most of these activities are given
in the form of Jupyter Notebooks.\footnote{Jupyter notebooks are open-source
web applications used to create and share documents that contain live code,
equations, visualizations, and narrative text.} In addition to the lecture
notes and notebooks, the authors have also developed {\bf
SolidsPy}\footnote{Check the documentation in
\url{https://solidspy.readthedocs.io} \cite{solidspy}.} which
is a complete Python-based finite element code to conduct stress analysis over
arbitrary two-dimensional elastic domains. The code, which follows a modular
structure facilitates the realization of learning activities in the form of (i)
full stress analysis simulations (ii) implementation of intermediate steps
aimed at covering the various steps in the FE algorithm and (iii) extensions to
incorporate additional kinematic models. In the rest of this introductory
chapter, we present a simple problem of a mechanical spring-mass system that
resembles the final form of a finite element algorithm and which serves to
justify why we cover the material in the specific order proposed here. Then we
describe the topics covered in the notes and in the final part we indicate how
to use the material.

\section*{Introductory problem}
\subsection*{A simple discrete system}
The simple case of a mechanical spring-mass system considered next resembles most of the algorithmic aspects of a finite element code. This system serves as a nice motivational example since the problem is already discrete thus avoiding the discussion of mathematical complexities. As will be presented throughout the course one of the goals of the finite element algorithm, when applied to a continuous system represented in terms of a boundary value problem, is to reformulate it as a discrete system fully analogous to the simple problem.

The system consists of an assemblage of masses joined by different springs submitted to time varying loads. For later purposes it is convenient to associate a spring with the concept of a finite element and a mass with a nodal point in a finite element algorithm.

The system may be like the one shown in \cref{fig:bathe} where the masses are
represented by ``cars'' connected by springs (or finite elements) of different
stiffness coefficients.

\begin{figure}[h]
\centering
\includegraphics[width=8cm]{spring_system.pdf}
\caption{Typical assemblage of springs and masses.}
\label{fig:bathe}
\end{figure}

Considering a typical spring (or finite element), \cref{fig:springel}

\begin{figure}[h]
\centering
\includegraphics[width=6cm]{springel.pdf}
\caption{Typical spring element.}
\label{fig:springel}
\end{figure}

one finds that under a relative displacement $\delta u = u_1 - u_2$  the spring develops a force :

\[f_1 = K(u_1 - u_2)\]

while equilibrium of the spring requires

\[f_1 + f_2 = 0.\]

The force-displacement and equilibrium equations can be combined into:

\begin{equation}
    \begin{Bmatrix}
        f_1\\
        f_2
    \end{Bmatrix} =
    K\begin{bmatrix}
          1.0 & -1.0\\
        - 1.0 & 1.0
    \end{bmatrix}
    \begin{Bmatrix}
        u_1\\
        u_2
    \end{Bmatrix}.
    \label{eq:Kspring}
\end{equation}

\Cref{eq:Kspring} resembles a first fundamental aspect displacement based finite element methods which is the fact that the problem is described in terms of displacements of specific points (or nodes) and where internal element forces are found from these nodal displacements via a constitutive relationship. In the case of a continuous problem an analogous force-displacement relationship is established using interpolation theory together with equilibrium statements.

On the other hand, the equilibrium equation for a typical mass $m_j$ with displacement $u_j$  (see \cref{fig:dclmass}) and assumed to be attached to springs $i$ and $i+1$ reads
\begin{equation}
f_2^i + f_1^{i + 1} + m_j \dv{V_j}{t} = P_j.
\label{eq:equilmass}
\end{equation}

This equation can be written in terms of displacements after expressing the involved forces $f_2^i$ and $f_1^{i + 1}$ using terms from equations like \ref{eq:Kspring}

\[(K^i + K^{i + 1}) u_j - K^i u_{j - 1} - K^{i + 1} u_{j + 1} + m_j\dv{V_j}{t} = P_j .\]

\begin{figure}[H]
\centering
\includegraphics[width=6cm]{dcl_mass.pdf}
\caption{Free body diagram for a typical mass connected to springs $i$ and $i+1$.}
\label{fig:dclmass}
\end{figure}

Writing the elemental equilibrium equations for the springs in terms of the corresponding mass displacements $u_{j - 1}$, $u_j$ and $u_{j + 1}$ and generalizing the coefficients notation we get:

\[\left\{ {\begin{array}{*{20}{c}}
{f_1^i}\\
{f_2^i}
\end{array}} \right\} = \left[ {\begin{array}{*{20}{c}}
{k_{11}^i}&{k_{12}^i}\\
{k_{21}^i}&{k_{22}^i}
\end{array}} \right]\left\{ {\begin{array}{*{20}{c}}
{{u_{j - 1}}}\\
{{u_j}}
\end{array}} \right\}\]

and

\[\left\{ {\begin{array}{*{20}{c}}
{f_1^{i + 1}}\\
{f_2^{i + 1}}
\end{array}} \right\} = \left[ {\begin{array}{*{20}{c}}
{k_{11}^{i + 1}}&{k_{12}^{i + 1}}\\
{k_{21}^{i + 1}}&{k_{22}^{i + 1}}
\end{array}} \right]\left\{ {\begin{array}{*{20}{c}}
{{u_j}}\\
{{u_{j + 1}}}
\end{array}} \right\}\]

which gives for the equilibrium equation of the $m_j$ mass:

\[k_{21}^i{u_{j - 1}} + (k_{22}^i + k_{11}^{i + 1}){u_j} + k_{12}^{i + 1}{u_{j + 1}} + {m_j}\frac{{d{V_j}}}{{dt}} = {P_j}.\]

If one the other hand we also consider the contributions from the springs $K^i$ and $K^{i+1}$ to the equilibrium of masses $m_{j-1}$ and $m_{j+1}$ respectively we have the following matrix block:

\[\left[ {\begin{array}{*{20}{c}}
{}&{}&{}&{}\\
{}&{k_{11}^i}&{k_{12}^i}&{}\\
{}&{k_{21}^i}&{k_{22}^i + k_{11}^{i + 1}}&{k_{12}^{i + 1}}\\
{}&{}&{k_{21}^{i + 1}}&{k_{22}^{i + 1}}
\end{array}} \right].\]

Consideration of the complete system of masses leads to a system of linear equations of the general form

Considering now the complete system of masses and springs leads to a system of linear equations of the form
\begin{equation}
\left[ {{K_G}} \right]\left\{ {{U_G}} \right\} + \left[ M \right]\left\{ {{A_G}} \right\} = \left\{ {{F_G}} \right\}.
\label{eq:global}
\end{equation}
where each equation represents the equilibrium of a given mass.

The process of forming these global coefficient matrices by adding the contribution from different elements (springs) to the equilibrium equations of the different masses is known as element assembly and this can be achieved in a systematic way by establishing the connection between the global and local degrees of freedom. This can be accomplished through an operator storing in each row the global degrees of freedom corresponding to each element. For instance, \cref{fig:IBC} shows elements $K^i$ and $K^{i+1}$ and the global degrees of freedom corresponding to masses $m_{j-1}$, $m_j$ and $m_{j+1}$. The corresponding entries of the $DME$ operator for these elements are given by:

\[DME = \left[ {\begin{array}{*{20}{c}}
{}&{}\\
{j - 1}&j\\
j&{j + 1}\\
{}&{}
\end{array}} \right]\]

\begin{figure}[H]
\centering
\includegraphics[width=12cm]{ibc}
\caption{Global degrees of freedom connected to the spring elements $K^i$ and $K^{i+1}$ respectively.}
\label{fig:IBC}
\end{figure}

and the assembly process from the contribution of these elements to the global coefficient matrix for elements $i$ and $i+1$ proceeds as:

\[\begin{array}{l}
{K_{j - 1,j - 1}} \leftarrow {K_{j - 1,j - 1}} + k_{11}^i\\
{K_{j - 1,j}} \leftarrow {K_{j - 1,j}} + k_{12}^i\\
{K_{j,j - 1}} \leftarrow {K_{j,j - 1}} + k_{21}^i\\
{K_{j,j}} \leftarrow {K_{j,j}} + k_{22}^i
\end{array}\]

and

\[\begin{array}{l}
{K_{j,j}} \leftarrow {K_{j,j}} + k_{11}^{i + 1}\\
{K_{j,j + 1}} \leftarrow {K_{j,j + 1}} + k_{12}^{i + 1}\\
{K_{j + 1,j}} \leftarrow {K_{j + 1,j}} + k_{21}^{i + 1}\\
{K_{j + 1,j + 1}} \leftarrow {K_{j + 1,j + 1}} + k_{22}^{i + 1}
\end{array}\]

The system given by \cref{eq:global} and assembled with the aid of the $DME$ operator can be solved for the global displacements $U_G$ and later use these displacements to compute the element forces.

This algorithmic strategy of assembling the contribution from all the elements to formulate the global equilibrium equations of the system is typical of finite element methods. However in the case of a BVP the continuous system must be converted first into a discrete system through different numerical and mathematical methods. A broad description of the method is described in \cref{algo:springs}.

\begin{algorithm}[H]
    \SetAlgoLined
    \KwData{Problem parameters; NUMNP, NUMEL, NMATP}
    \KwResult{Displacements and spring forces}
    Create $DM$E operator\;
    Assemble $K^G$, $F^G$\;
    \While{$j \leq 1, NUMEL$}{
        $K^G \leftarrow K^G+K^i$\\
        $F^G \leftarrow F^G+F^i$\\
    }
    Solve $[K^G]U=F^G$\\
    Find internal forces
    \caption{Springs Algorithm.}
    \label{algo:springs}
\end{algorithm}

The full implementation of the mass-springs system is given in an accompanying notebook in the course REPO.
\newpage
\section*{Contents of the course}
The course is divided in three parts as follows:

\begin{itemize}
\item Part 1 covers classical numerical methods such as interpolation theory and numerical integration within the context of finite element methods. These methods are shown to be fundamental in the conversion of the continuous system into a discrete system analogous to the discussed mass-springs system. The numerical methods are studied from its theoretical and computational aspects and the suggested learning activities are described in notebooks 1 through 6.

\item Part 2 presents the boundary value problem corresponding to the model of the linearized theory of elasticity and in particular its description through formulations which are suitable for a solution in terms of finite elements algorithms. The boundary value problem is covered in notebook 7.

\item Part 3 concentrates on the formulation of the elasticity boundary value
problem in a finite element algorithm. The algorithm and several related
aspects are covered in notebooks 8 through 11, while notebook 12 contains a
brief reference to the course finite element code SolidsPy.
\end{itemize}

The main set of accompanying notebooks also includes NB-0 which makes an introduction to the use of notebooks and a quick reference to data flow structures in Python. The set of lecture notes also include an appendix section presenting some more mathematical aspects of the method and additional tools that may result useful depending on the student's abilities.

\paragraph*{How to follow the course?\footnote{This set of lecture notes and
additional course material has been developed as part of the sabatical period
of the first author and with the collaboration of the second author Nicolas
Guarin-Zapata. The development of the notebooks and design of the learning
activities in these notebooks has been developed thanks to the advisory of
Camilo Vieira.}}

The course and its different resources have been created to be used in a flipped class environment or as a self study material. In the formal course, as thought in the graduate program at Universidad EAFIT, it is recommended to follow the proposed sequence of activities starting with the section covering numerical methods as described previously.This same sequence is also recommended for independent learners with no previous knowledge on fundamental numerical methods like interpolation theory and numerical integration. More advanced students, with previous backgrounds on mathematical analysis and numerical methods can test their actual abilities by developing the activities proposed on NB-4 and NB-6 and then moving directly into Chapter 4 discussing the boundary value problem.


%!TEX root =  ../main.tex
\subsection{Chapter Review}
This chapter was about functions, and how they model so much of reality.  Functions can be
described numerically (tables), algebraically (formula), graphically (Cartesian plane curves), or
verbally (technical language).  Some functions we have seen before or will see in this course
are linear, quadratic, exponential, power, rational, logarithmic, and periodic.
You need to have memorized the simplest form of each type of equation (Tab.~\ref{tab:generalequations}).

\columntable[-0.75in]{2.8in}{
    \textbf{Linear:}\\  $y=m\cdot{}b$

    \medskip

    \textbf{Inversely Proportional:}\\
    $y=\frac{k}{x}$

    \medskip

    \textbf{Power:}\\
    $y=a\cdot{}x^b$

    \medskip

    \textbf{Quadratic:}\\
    $y=ax^2+bx+c$

    \medskip

    \textbf{Exponential:}\\
    $y=a\cdot{}b^x$

    \medskip

    \textbf{Logarithmic:}\\
    $y=a+b\ln{x}$

    \medskip

    \textbf{Periodic:}\\
    $y=a\cdot\sin(b(x-c))+d$
    }{General equations\label{tab:generalequations}}

You should review the graphs of each of these.  In fact, you should attempt to engage all
these forms in all four ways.

Functions are written in function notation, which looks annoyingly like multiplication.
Functions can have operations done on them, as numbers do, only they have two place where
operators may be applied: inside (before) and outside (afterwards).  Outside effects the output
(y) and is reasonable.  Inside effects x and is contrarian!  Adding translates (moves) the graph.
Multiplying stretches (dilates) the graph.  Negatives `flip' the graph.  Graphs that are the same when flipped
left/right are called even, like even powers of x.  Graphs that look the same when they are
flipped left/right \emph{and} up/down are called odd, like odd powers of x.  The calculator can help us turn
numerical data into function notation via its many regression functions.

Some questions you should be prepared to address are:
What are functions?  What are some of the basic types of functions?  What are the four ways
we describe functions in this chapter?  How do we move between the various descriptions?
What happens to $f(x)$ as we vary four constants, like this: $a\cdot{}f(b(x+c))+d$?
What are some of the differences between our models and reality?  What have previous classes
shown you to be the nature of the mathematic task?  What is technical (vs ordinary or poetic)
language?  What role does technical language play in your life, now and presumably in the future?

\begin{figure}
\begin{centering}
\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=5cm, semithick]
  \tikzstyle{every state}=[fill=red,draw=none,text=white]

  \node[state,minimum size=2cm] (A)                    {Graph};
  \node[state,minimum size=2cm]         (B) [above right of=A] {Data};
  \node[state,minimum size=2cm]         (D) [above left of=B] {Equation};
  \node[state,minimum size=2cm]         (C) [above left of=A] {Verbal};

  \path (A) edge [bend right=5]              node[below,sloped] {``read''} (B)
            edge [bend right=5] node[above,sloped] {``describe''} (C)
            edge [bend right=5,dotted]  node[below,sloped]  {``find''} (D)
            edge [loop below]	node {``change''} (A)
           (B) edge[bend right=5] node [above,sloped] {``plot''} (A)
           edge [loop right] node {``convert''} (B)
           edge [bend right=5]  node[above,sloped] {``regress''} (D)
           edge [bend right=20] node [above,sloped] {``characterize''} (C)
           (C) edge[bend right=5] node[below,sloped] {``sketch''} (A)
           edge [bend right=5] node[below,sloped] {``build''} (D)
           edge [bend right=20] node[below,sloped] {``estimate''} (B)
           edge [loop left] node {``paraphrase''} (C)
           (D) edge [loop above] node {``algebra''} (D)
           edge [bend right=5] node[below, sloped] {``tabulate''} (B)
           edge [bend right=5] node[above,sloped] {``explain''} (C)
           edge [bend right=5,dotted] node[below,sloped] {``graph''} (A);
\end{tikzpicture}
\caption{Some possible words to describe moving among the four representations\label{fig:16relationships}}
\end{centering}
\end{figure}

Figure~\ref{fig:16relationships} is not the final word(s) on relating the four representations,
but it is supposed to get you thinking.  We often manipulate symbols and change one
equation type into another, but we can do the same with graphs.  Paper does not
allow it, but couldn't we build an object that changes color over time, in order to
represent a function?  Data could be changed from absolute $x$ and $y$ to
``number of steps since last'' and ``percent change.''  Try to come up with a simple
model of a natural phenomenon and see how many relationships you can
fulfill.

\subsection{Chapter Test}
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch01/01testA.pdf}}
\newpage
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch01/01testB.pdf}}


\documentclass{article}

%%%%%% Include Packages %%%%%%
\usepackage{sectsty}
\usepackage{amsmath,amsfonts,amsthm,amssymb}
\usepackage{fancyhdr}
\usepackage{lastpage}
\usepackage{setspace}
\usepackage{graphicx}
\usepackage{array}

%%%%%% Formatting Modifications %%%%%%

\usepackage[margin=2.5cm]{geometry} %% Set margins
\sectionfont{\sectionrule{0pt}{0pt}{-8pt}{0.8pt}} %% Underscore section headers
\setstretch{1.2} %% Set 1.2 spacing

%%%%%% Set Homework Variables %%%%%%

\newcommand{\hwkNum}{11}
\newcommand{\hwkAuthors}{Ben Drucker}

%%%%%% Set Header/Footer %%%%%%

\pagestyle{fancy}
\lhead{\hwkAuthors}
\rhead{Homework \#\hwkNum}
\rfoot{\textit{\footnotesize{\thepage /\pageref{LastPage}}}}
\cfoot{}
\renewcommand\headrulewidth{0.4pt}
\renewcommand\footrulewidth{0.4pt}

%%%%%% Document %%%%%%

\begin{document}

\title{Homework \#\hwkNum}
\author{\hwkAuthors}
\date{}

\maketitle

%%%%%% Begin Content %%%%%%

\section*{7.3}
	\subsection*{32}
		CI: $\overline{X} \pm t \frac{S}{\sqrt{n}} = 1584 \pm t_{19, .005} \frac{607}{\sqrt{20}} = 1584 \pm 2.861 \frac{607}{\sqrt{20}} = (1195.68, 1972.32)$
	\subsection*{34}
		\subsubsection*{a)}
			$8.48-1.771 \frac{.79}{\sqrt{14}} = 8.11.$ With 95\% confidence, the true mean of all joints is in the interval (8.11, $\infty$). For an infinite number of sample confidence intervals, 95\% will include the true mean.  A normal distribution is assumed.
		\subsubsection*{b)}
			$8.48 - 1.771 * .79 \sqrt{1+\frac{1}{14}} = 7.03$. If we calculate this bound for an infinite number of samples, 95\% will give the lower bound for future values of a joint.
\section*{7 Supplement}
	\subsection*{50}
		$\overline{x} = \frac{229.764+233.502}{2} 231.63; t_{.025, 5-1} = 2.78\\
		233.502 - 229.764 = 3.74; 2*2.78 \frac{s}{\sqrt{5}} =s \rightarrow s = \frac{\sqrt{5}*3.74}{2*2.78} = 1.51\\
		t_{.005,4} = 4.604 \\
		231.63 \pm 4.604 \frac{1.51}{\sqrt{5}} = 213.63 \pm 3.1$

	\subsection*{60}
		$(z_Y+z_{\alpha - Y} \frac{s}{\sqrt{n}}) \rightarrow \min(z_Y+z_{\alpha - Y} \frac{s}{\sqrt{n}}) \rightarrow \min[\Phi^{-1}(1-Y) + \Phi^{-1} (1-\alpha + Y).$
		\\ Setting the derivative equal to 0:
		\\ $ \frac{1}{\Phi(1-Y)} = \frac{1}{\Phi(1-\alpha+Y)} \Rightarrow Y = \frac{\alpha}{2}$
\section*{8.1}
	\subsection*{2}
		\subsubsection*{a)}
			Yes.
		\subsubsection*{b)}
			No, because $H_0$ is not an equality claim.
		\subsubsection*{c)}
			No, because $H_a$ is the equality claim instead of $H_0$.
		\subsubsection*{d)}
			No. $\mu_1 -\mu_2$ should appear in $H_a$.
		\subsubsection*{e)}
			No because $S^2$ is a statistic and shouldn't be into a hypothesis.
		\subsubsection*{f)}
			No, both $H_0$ and $H_a$ can't be equality claims.
		\subsubsection*{g)}
			Yes
		\subsubsection*{h)}
			Yes
	\subsection*{4}
		Versus $H_a: \mu < 5$. In this case, the type I error should be avoided at all costs whereas the type II error is not as serious.
	\subsection*{6}
		$H_0: \mu  = 40; H_a: \mu \not = 40. \mu \not = 40$ is interesting for the manufacturer in either direction. A type I error would be rejected a fuse where $\mu$ is actually 40. A type 2 error would be letting a fuse through where $\mu$ is 40.

%%%%%% End Content %%%%%%
\end{document}

Spring 2013/STAT W1211 - Statistics/Homework/Homework 11/Homework 11.tex


\documentclass[a4paper,12pt]{article}

 \addtolength{\oddsidemargin}{-.5in}
\addtolength{\evensidemargin}{-.5in}
\addtolength{\textwidth}{1.0in}
\addtolength{\topmargin}{-.5in}
\addtolength{\textheight}{1.0in}

\usepackage{cite}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{xcolor}
\usepackage{color}
\usepackage{bbding}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}

\begin{document}

\title{Group 4: The Kaczmarz Algorithm}
\author{Wei Deng, Nicole Eikmeier, Nate Veldt, and Xiaokai Yuan}
\maketitle{}

\section{Introduction}
Project Objective:

Given a large sparse linear system $Ax = f$, of order $n = 10^6$ that can be effectively reduced to a banded matrix using the reordering scheme ``reverse Cuthill-McKee", use the method of Row Projection, accelerated via the Conjugate Algorithm to yield a parallel solver. Compare the robustness and parallel scalability/speed with preconditioned Krylov subspace methods preconditioned via approximate LU-factorization.

\vspace{.2in}

The Kaczmarz Algorithm is a row projection method which is equivalent to solving $AA^Ty = f$ using the Gauss-Siedel iteration, with $x = A^Ty$. In Kaczmarz we consider each equation as a hyperplane:
$$S_i=\{x:A_ix-b_i=0\}$$ for $i=1,2,...$, where $A_i$ and $b_i$ are $i$th row of matrix $A$ and vector $b$. So the problem is transformed into finding the coordinates of the point of intersection of these hyperplanes.  See figure \ref{fig:ClassKacz} for a visualization of the problem.

\begin{figure}[htbp]
\begin{centering}
%\begin{minipage}[b]{0.8\linewidth}
\includegraphics[width=5in]{Images/Classical_Kaczmarz}
\caption{2-Partition Case of the Kaczmarz Algorithm}
\label{fig:ClassKacz}
%\end{minipage}
\end{centering}
\end{figure}

From this basic idea we use symmetrization and acceleration via the CG method to improve the procedure. The details will be given carefully in the following sections.

\section{Implementation}

%------------------------------------------------
%------------------------------------------------
% REVERSE CUTHILL-MCKEE
%------------------------------------------------
%------------------------------------------------

\subsection{Reverse Cuthill-McKee and the Woodbury Formula}
The first step in our implementation is to transform the sparse matrix $A$ to a banded form using Reverse Cuthill-McKee. We can perform a symmetric permutation with matrix $P$ so that $P^TAP$ is banded. In figure \ref{fig:v3}, we see the result from Matlab of Reverse Cuthill Mckee on the stomach data set. The band size is nice and small.

%\begin{figure}[ht]
%\centering
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/SparseA.jpg}
%\caption{$A$ sparse, non-symmetric}
%\end{minipage}
%\quad
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/Arcm.jpg}
%\caption{$P^TAP$ (banded)}
%\label{fig:banded}
%\end{minipage}
%\end{figure}

\begin{figure}[ht]
\centering
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/SparseA_v5.jpg}
\caption{$A$ sparse, non-symmetric}
\label{fig:v5}
\end{minipage}
\quad
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/Arcm_v3.jpg}
\caption{$P^TAP$ (banded)}
\label{fig:v3}
\end{minipage}
\end{figure}

For some problems the bandwidth could be too large after using Reverse Cuthill Mckee. See for example figure \ref{fig:approxbanded}, where we see the result of RCM on the lns$\_$131 data set, again in Matlab.  If this is the case, instead we could choose $P$ so that $P^TAP$ is narrow banded plus a low rank matrix. See figure \ref{fig:woodbury}.

\begin{figure}[ht]
\centering
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/SparseA_v4.jpg}
\caption{$A$ is sparse and non-symmetric}
\label{sparseA}
\end{minipage}
\quad
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/ApproxBanded_v2.jpg}
\caption{$P^TAP$ narrow band+low rank}
\label{fig:approxbanded}
\end{minipage}
\end{figure}

%\begin{figure}[ht]
%\centering
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/SparseA.jpg}
%\caption{$A$ is sparse and non-symmetric}
%\end{minipage}
%\quad
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/ApproxBanded.jpg}
%\caption{$P^TAP$ narrow band + low rank}
%\end{minipage}
%\label{fig:bandedplus}
%\end{figure}

\begin{figure}[htbp] %  figure placement: here, top, bottom, or page
   \centering
   \includegraphics[trim = 5mm 30mm 0mm 40mm, clip, width=4in]{Images/Slide1.jpg}
    \caption{Breaking up $A$ into a banded matrix plus a low rank matrix}
    \label{fig:woodbury}
\end{figure}

In this case we use the Woodbury Formula to solve $Ax = b$. Recall first that $ Ax = b$  implies $x = A^{-1} b$. Then,
%\begin{block}{Woodbury Formula}
\begin{align*}
A^{-1} &= (B - USV^T)^{-1}\\
	  &= B^{-1} - B^{-1}UTV^TB^{-1}
\end{align*}
where $T = (V^TB^{-1}U - S^{-1})^{-1}$. And
%\end{block}
\begin{align*}
x &= A^{-1}b\\
  &= {\bf B^{-1} b} - B^{-1}UTV^T {\bf B^{-1} b} \\
   &= {\bf a} - B^{-1}UT(V^T {\bf a}) \\
   &= {\bf a} - B^{-1}UT {\bf c}  \hspace{.5cm} \mbox{( solve ($V^TB^{-1}U-S^{-1}) \textbf{d} = \textbf{c}$)}\\
   &= {\bf a} - B^{-1}U\textbf{d}\\
   &= {\bf a} - B^{-1}\textbf{h}
\end{align*}

The Woodbury formula is useful because all systems involving $B$ are relatively easy to solve. We did not implement the Woodbury formula in our experiments, due to time contraints. This means that we transformed $A$ to a completely banded matrix, no matter how large the band size. We expect there would be improvements in our final results if the Woodbury formula was implemented.

In table \ref{tab:Bandsize}, the band size after permutation is shown. Matlab performed the best across the board, but all three permutations did well for our largest test case. You can see also that for the bayer01 matrix, our band size is very large, and so we did not get reasonable results in our parallel Kaczmarz implementation.

\begin{table}
\begin{center}
\begin{tabular}{| l | l | c  c  c |}
\hline

    matrix           & size         &    Matlab & PETSc     &  rcm.cpp        \\
 \hline
 lns                   & 131             &     32        &       \color{red} $\times$ 111         & \color{red} $\times$ 113  \\
 Jac2-db            & 21,982     &         545       &     \color{red} $\times$            &  \color{red} $\times$  \\
 bayer01           & 57,735      &     \color{red} $\times$ 18,322        &     \color{red} $\times$     &  \color{red} $\times$   \\
 venkat25         & 62,424      &     1,515        &    1,515    & 1,495                   \\
 stomach          & 213,360    &      1,133      & 2,216 & 2,239                    \\
 atmosmodd     & 1,270,432 &    7,772       & 7,772&  7,772                   \\
 \hline

\end{tabular}
\caption{Bandsize after RCM using various codes}
\label{tab:Bandsize}
\end{center}

\end{table}

%------------------------------------------------
%------------------------------------------------
%KACZMARZ
% NEEDS SOME RE-WRITING
%------------------------------------------------
%------------------------------------------------
%
\subsection{The Kaczmarz method}

Once we have a banded matrix, $A$, we move on to the Kaczmarz algorithm, which was introduced earlier. Suppose we partition $A$ into two pieces $$A = \begin{pmatrix} A_1^T \\ A_2^T \end{pmatrix}  $$ and we also partition the right hand size vector $f^T = (f_1^T, f_2^T)$.

In the classical Kaczmarz method at each iteration we compute:
$$x_k=x_k+\overrightarrow{n_k}=x_k+\frac{r_k^i}{||A_i||^2}A_i^T$$
where  $\overrightarrow{n_k}=\bigtriangleup{x_k}cos\theta=\frac{\langle\,A_i{,}\,\bigtriangleup{x_k}\rangle}{||A_i||^2}A_i^T=\frac{b_i-\langle\,A_i{,}\,x_k\rangle}{||A_i||^2}A_i^T=\frac{r_k^i}{||A_i||^2}A_i^T$.
$$
\left\{
\begin{array}{ll}
                  AA^Ty = f\\
                  x = A^Ty\\
 \end{array}
 \right.
 \implies AA^T = \begin{pmatrix} A_1^T \\ A_2^T \end{pmatrix} \begin{pmatrix} A_1, A_2 \end{pmatrix}
  $$

  From Gauss-Seidel, use $x^{k+1} = \begin{pmatrix} A_1, A_2 \end{pmatrix} \begin{pmatrix} y_1^{k+1} \\ y_2^{k+1} \end{pmatrix} $.

  $$ \begin{pmatrix} A_1^TA_1 & 0 \\ A_2^T A_1 & A_2^T A_2 \end{pmatrix} \begin{pmatrix} y_1^{k+1} \\ y_2^{k+1} \end{pmatrix} =  \begin{pmatrix} 0 & -A_1A_2 \\ 0 & 0\end{pmatrix} + \begin{pmatrix} y_1^{k+1} \\ y_2^{k+1} \end{pmatrix}\begin{pmatrix} f_1 \\ f_2 \end{pmatrix}$$

  Replacing some parameters with projection operator $P_i$, we can get:
  $$x^{k+1} = A_1y_1^{k+1} + A_2y_2^{k+1} = Qx_k +b$$ where $Q = (I-P_2)(I-P_1)$. Similarly, for an m-partition, $$x^{k+1} = Q_u x^k + f_u = (I-P_m)(I-P_{m-1}) \ldots (I-P_1)x^k + f_u.$$

  However, the spectral radius of $Q_u$ may interfere with the convergence speed, and the distribution of eigenvalues could also influence the performance. To handle this, we can symmetric $Q_u$ to get an accelerated iteration:
  $$ x^{k+1} = Q(\omega)x^k + T(\omega) f$$ where $Q(\omega) = (I-\omega P_1)(I-\omega P_2) \ldots (I-\omega P_m) \ldots (I-\omega P_2) (I-\omega P_1)$, and $T(\omega) = A^T(D+\omega L)^{-T}D(D+\omega L)^{-1},$ where $A^TA = L+ D+L^T$ is the splitting into block lower, block upper, and block diagonal pieces of $A^TA$. Since $(I-Q)$ is symmetric positive definite, the conjugate gradient method is suitable to accelerate the basic scheme.

 Once we have a banded matrix, we considered using various partitions of the rows. A very simple example is shown in figure \ref{fig:permutation}. A permutation like this gives several benefits. The first is that we can have an outer level of parallelism for each projection. Also, when we split the matrix this way we create several small independent least squares problems, which reduces the time.

 \begin{figure}[htbp]
\begin{minipage}[b]{1\linewidth}
\centering
\includegraphics[width=4in]{Images/permutation}
\end{minipage}
\caption{Re-ordering the banded matrix}
\label{fig:permutation}
\end{figure}

 In Figure \ref{quality} we consider different types of partitions, both in the inner and outer levels. We found that have more independent blocks resulted in faster implementation of Kaczmarz. We also found that more partitions, i.e. taking 4 partitions: $$A = \begin{pmatrix} A_1^T  \\A_2 ^T \\ A_3^T \\ A_4^T \end{pmatrix} $$ instead of 2 partitions is slower.

  Furthermore, it has been proven that the optimal value of $\omega$ is 1.0 if we use two partitions, and $Q(1)$ has the minimal spectral radius. For these reasons we decided to use 2 outer partitions for our implementation. In this case the problem can be simplified as follows: $$(I-Q)x = Tf$$

\begin{figure}[ht]
\begin{centering}
%\begin{minipage}[b]{1\linewidth}
\includegraphics[width=6in]{Images/quality.jpg}
%\end{minipage}
\end{centering}
\caption{Speed for different partition types.}
\label{quality}
\end{figure}

%\begin{figure}[ht]
%\centering
%\begin{minipage}[b]{1.0\linewidth}
%\includegraphics[width=4.8in]{Images/permute.jpg}
%\caption{$A$ is partitioned into 2 parts with several independent submatrices}
%\end{minipage}
%\end{figure}

%------------------------------------------------
%------------------------------------------------
% SYMMETRIZE KACZMARZ
% NEEDS SOME HELP
%------------------------------------------------
%------------------------------------------------
\subsection{Calculation of $c$}
To transform the system $Ax = f$ into solving the symmetric positive definite system $(I-Q)x = c$, we need to get the new right hand side vector $c$. Recall that we use $m=2$ partitions in our work. We let

 $$A = \begin{pmatrix} A_1^T \\ A_2^T \end{pmatrix}  $$ and we also partition the right hand size vector $$f^T = (f_1^T, f_2^T).$$

 For two partitions, if we expand $c = Tf$ to be:

 $$\textbf{Tf} = \begin{bmatrix} (I + (I-P_2)(I-P_1))(A_1^T)^+ && (I-P_1)(A_2^T)^+ \end{bmatrix} \begin{bmatrix} f_1  \\ f_2 \end{bmatrix}= c $$

 We see that in order to perform this we need to solve two pseudo inverse problems:

 $$\hat{f}_i = (A_i^T)^+f_i$$

 and whenever we see a product of the form $(I-P_i)u = v$ we are really performing a least squares operation involving the block $A_i$. We will discuss this least squares problem in more detail later.

Computing $c$ can be accomplished by solving a saddle point problem. In our implementation we spent some of our efforts trying to compute $c$, but due to time constraints and difficulties with other parts of our algorithm we were not able to ever correctly incorporate a proper computation of $c$ in our work. For this reason, for our code we generated a right hand side vector $c$ by setting $u$ to be the all ones vector, and then performing $c := (I-Q)\cdot u$ via a sequence of least squares operations. Then we were able to use our conjugate gradient framework to solve the system $(I-Q)x = c.$

In general for Kaczmarz we expect the time taken in running the CG scheme and calling the least squares function to be the bottleneck in our computations. So even though we were unable to properly generate the right hand side vector from a starting vector $f$, we chose to instead focus on the CG scheme and least squares framework in our implementation.

\subsection{Least Squares Computations}
Recall that the new system we are dealing with is $(I-Q)x = c$, where $c = Tf$, and $$Q=(I-P_{1})(I-P_{2})(I-P_{1})$$
 with $P_{i}=A_{i}(A_{i}^{T}A_{i})^{-1}A_{i}^{T}$.

It is not stable to form $P_{i}$ directly since it will square the condition number of $A_{i}$, and at the same time will cost time in solving a system with the matrix $(A^{T}_{i}A_{i})$. Instead, given a vector $u$ we note that $v=(I-P_{j})u \Leftrightarrow \min\limits_{v}||u-A_{j}w||_{2}$. We therefore solve $\min\limits_{v}||u-A_{j}w||_{2}$ to obtain the vector $v$.

%In PETSc, we use the CG method on the normal equations: $A_{j}^{T}A_{j}w=A_{j}^{T}u$. The parallel step is:
%        \begin{equation*}
%            \begin{split}
%                v&=\min\limits_{v}||u-A_{j}w||_{2}\Leftrightarrow \min\limits_{v_{i}}||u_{i}-A_{j,i}w_{i}||_{2},\\
%                v&^{T}=[v_{1}^{T},v_{2}^{T},...,v_{k}^{T}],\\
%                w&^{T}=[w_{1}^{T},w_{2}^{T},...,w_{k}^{T}].
%            \end{split}
%        \end{equation*}

Because of the way we have split our matrix, when we solve a least squares problem $(I-P_i)u = v$ we in theory can do so independently because $A_i$ is a matrix with many independent blocks. For proper parallel implementation, we would perform the following steps:

\begin{enumerate}
\item At the beginning of the algorithm, send different blocks of $A_i$ (for each sub matrix $i$) to different processors.
\item In each least squares computation, independently perform least squares operations on the blocks, and then gather the final result after the simultaneous, independent least squares operations.
\end{enumerate}

\subsubsection*{QR Factorizations}
One very effective way to compute the least squares problem in parallel is to use the QR factorization via Given's rotations on each independent block of each sub matrix $A_i$. The computation and storage of the QR factorization of the entire matrix $Q$ would be extremely expensive. However, because $A_i$ is made up of independent diagonal blocks that are all very small in size, we can perform QR factorizations of each of these blocks with great success.

There are two clear benefits of using the QR factorization on the independent sub blocks of $A_i$. First of all, we can find this factorization simultaneously for each block of $A_i$ at the very beginning of the algorithm, then in all future calls to our least squares function we can use the factorization without needing to recompute it. Secondly, the storage required for the factorization of the blocks is very small in comparison with the overall system.

\subsubsection*{Difficulties}

Unfortunately in our project we were not able to make full use of the parallelism that is theoretically possible by the Kaczmarz framework. Our difficulty was that splitting up a parallel matrix $A_i$ into independent pieces is extremely difficult in PETSc. This causes a number of issues in our final code, including the following:

\begin{itemize}
\item We cannot be sure that the blocks are perfectly split up among different processors, since PETSc assigns rows automatically by itself without our control.
\item Even though we have multiple processors at work on a least squares problem, the least squares computation is still performed globally on the entire sub matrix $A_i,$ rather than simultaneously on independent blocks of $A_i$.
\item We are unable to factorize pieces of $A_i$ ahead of time to save time in later calls to our sequential least squares algorithm
\end{itemize}

With a better knowledge of PETSc these problems could be remedied, but time constraints rendered it impossible to master the software enough so that we could reach the full potential of the Kaczmarz framework.

\subsubsection{Attempt to Improve Speed by LSQR}

In the end, in order to solve the least squares problem, we implemented LSQR. LSQR is an iterative method for solving $Ax = b$ where $A$ is large, sparse, and overdetermined. It is equivalent at each iteration to CG.

\begin{algorithm}
\caption{LSQR}
\begin{algorithmic}[1]
%\Require{$P_1$ an initiator matrix; $n$, where we want to use $P_n$}
\State{$\beta_{1}u_{1}=b, \alpha_{1}v_{1}=A^{T}u_{1}, w_{1}=v_{1}, x_{0}=0, \bar{\phi_{1}}=\beta_{1},\bar{\rho_{1}}=\alpha_{1}$}
\For{$i = 1,2,3,\cdots$}
\State{Bidiagonalization:}
\State{$\beta_{i+1}u_{i+1}=Av_{i}-\alpha_{i}u_{i}$}
\State{$\alpha_{i+1}v_{i+1}=A^{T}u_{i+1}-\beta_{i+1}v_{i}$}
\State{Orthogonal Transformation:}
\State{$\rho_{i}=\sqrt{(\bar{\rho_{i}^2}+\beta_{i+1}^2)}$}
\State{$c_{i}=\bar{\rho_{i}}/\rho_{i},$}
\State{$s_{i}=\beta_{i+1}/\rho_{i},$}
\State{$\theta_{i+1}=s_{i}\alpha_{i+1},$}
\State{ $\bar{\rho_{i+1}}=-c_{i}\alpha_{i+1},$}
\State{$\phi_{i}=c_{i}\bar{\phi_{i}},$}
\State{$\bar{\phi_{i+1}}=s_{i}\bar{\phi_{i}}.$}
\State{Update x:}
\State{$x_{i}=x_{i-1}+(\phi_{i}/\rho_{i})w_{i},$}
\State{$w_{i+1}=v_{i+1}-(\theta_{i+1}/\rho_{i})w_{i}.$}
\EndFor

\end{algorithmic}
\end{algorithm}

    \begin{figure}[htbp]
    \begin{center}
        %\begin{minipage}[b]{0.8\linewidth}
            \includegraphics[width=3in]{Images/LSQR}
        %\end{minipage}
        \caption{Convergence of LSQR}
        \label{fig:LSQR}
        \end{center}
    \end{figure}

We had hoped that using this method and truncating the number of iterations used in the least squares calculations would allow us to obtain good overall results for our Kaczmarz algorithm. Note the plot of the convergence of the LSQR method in Figure \ref{fig:LSQR}. We can see that the algorithm mostly converges in the first several iterations, so we hoped that by truncating the number of steps taken we would make the least squares computation faster without harming the accuracy too much. We know that Krylov subspace methods are robust against poor matrix-vector products, so the hope was that our outer CG scheme would still progress towards an accurate overall solution of $(I-Q)x = c.$ . However, our final results are still far from satisfactory in terms of accuracy if we truncate the computations too much.

One thing we tried was to increase the maximum number of inner LSQR iterations used as the number of outer CG scheme iterations increased. We did this to try to find a balance between speed and accuracy. However, to achieve any type reasonable accuracy for our larger test cases, we did eventually need to make the iterative least squares method run for many iterations (over 100 for each least squares computation). This leads to very slow runtimes. We provide our results in a later section.

 %------------------------------------------------
%------------------------------------------------
% ACCELERATION VIA CG
%MAYBE NEEDS SOME HELP
%------------------------------------------------
%------------------------------------------------

 \subsection{Acceleration via the CG Method}
As stated previously, since $(I-Q)$ is symmetric positive definite, we accelerate solving $(I-Q)x = Tf$  using the conjugate gradient method. This is the framework of our entire algorithm. The outline follows:

\vspace{.2in}
 \noindent \textbf{Step 1} : $x_0 = c$ \\
\hspace{.5in} Compute $r_0 = Tf - \boxed{(I-Q)c} = Qc$\\
\hspace{.5in} Set $p_0 = r_0, i = 0$ \\
\textbf{Step 2}:  Compute: \\
\hspace{.5in} $\alpha_i = (r_i, r_i)/(p_i,\boxed{(I-Q)p_i})$ \\
\hspace{.5in} $ x_{i+1} = x_i + \alpha_i p_i $ \\
\hspace{.5in} $\beta_i = (r_{i+1}, r_{i+1})/(r_i, r_i) $ \\
\hspace{.5in} $p_{i+1} = r_{i+1} + \beta_i p _i $ \\
\textbf{Step 3}: If convergence criterion is satisfied, terminate the iterations; else set $i = i+1$ and return to Step 2.

 We can take a convergence criterion as : $$ \frac{ ||r_i||}{||r_0||} \leq \epsilon$$

The boxed out portions in our framework are very important. Remember that whenever we see: $$(I - Q) u$$ for any vector $u$, we really mean to solve multiple least squares problems:
$$(I-Q) u = (I-P_1)(I-P_2)(I-P_1) u $$
$$(I-P_i) u \Rightarrow \text{min}_{v} ||u-A_i w|| $$

 %------------------------------------------------
%------------------------------------------------
% LAST SECTION: RESULTS
%------------------------------------------------
%------------------------------------------------

\section{Results and Conclusions}

\subsection{Results}

We will now discuss the results of our parallel solver. Because of our slow least squares computations, we did not reach a very low tolerance for any of our test cases. For this reason we chose to simply truncate our CG loop at 100 iterations and report the runtime and the tolerance attained by this point. The results are given in table $\ref{tab:badresults}$.

\begin{table}
\begin{center}
\begin{tabular}{| l | l | c  c | c  c |}
\hline
                         &                & Our Parallel & Solver & Krylov & ILU\\
                          \hline
    matrix           & size         &  tol achieved & time (s)     &  num-its & time (s)        \\
 \hline
 lns                   & 131             &     $7.5E{-03}$   &         5.6  & \color{red} $\times$  &  \color{red} $\times$\\
 Jac2-db            & 21,982     &   $2.0E{-04}$     &    283    &  \color{red} $\times$  & \color{red} $\times$ \\
 venkat25         & 62,424      & $2.5E{-03}$      &     874  & 374                   & 14.17 \\
 stomach          & 213,360    &   $2.1E{-06}$    & 194 & 16                   & 2.21 \\
 atmosmodd     & 1,270,432 &  $3.0E{-03}$     &7100 &  266                   & 130.72 \\
 \hline

\end{tabular}
\end{center}
\caption{Runtimes of our algorithm (100 iterations of CG) in comparison with GMRES}
\label{tab:badresults}
\end{table}

To find a benchmark of how our method performed, we compared with a Krylov Subspace method. The implementation of this consisted of first using the MC64 software to re-order the matrix. This software finds a permutation which brings non-zeros to the diagonal. Once this permutation is used, then we called a Krylov Subspace method with ILU pre-conditioner in PETSc. The Krylov subspace method used is GMRES.

Note that the ILU preconditioner fails for some of our data. The reason for this could be that the eigenvalues of these matrices are distributed too much across the positive and negative spectrum. The ILU preconditioner is not well equipped to handle these so called ``highly indefinite'' matrices. Table \ref{tab:MC64} shows which of our data failed with this method.
\begin{table}
\begin{center}
\begin{tabular}{| l | l | c |}
\hline
matrix             & size        & Converged? \\
\hline
lns                  & 131         &    \color{red} $\times$  \\
std1-Jac2-db  & 21,982    & \color{red}  $\times$  \\
venkat25        & 62,424     & \color{green} \Checkmark   \\
stomach         & 213,360   &  \color{green}  \Checkmark  \\
atmosmodd    & 1,270,432 &  \color{green} \Checkmark  \\
\hline
\end{tabular}
\end{center}
\caption{ILU preconditioner}
\label{tab:MC64}
\end{table}

We note that the one success of our algorithm was its ability to make progress in solving the first few test cases where we failed to converge for GMRES. We see that the framework of our algorithm is good for solving these difficult systems, though runtimes are terrible for reasons already addressed.

\subsection{Conclusions}
In conclusion, it was a mistake to use PETSc. PETSc is better for high level implementation, not for optimization and efficiency. PETSc was also not well suited specifically for grabbing the exact sub-matrices that we needed in order to call the parallel Least Squares.

In the end we tried to salvage our project by implementing LSQR, but even with this attempt we did not obtain the desired outcome. If we had left PETSc out of our project we know our results would have been even better.

\nocite{KamathSameh1988,GallopoulosPhilippeSameh2016,Paige,HSL}

\newpage

\bibliography{Ref}
\bibliographystyle{plain}

\end{document}

Parallel Solver for a Large Sparse Linear System/slides/Group4_Final_Report_Complete.tex