In the sequel, we detail the application of scPCA to a number of simulated and publicly available datasets, comparing our proposal to several competing techniques. An additional analysis of protein expression data is presented in Section \ref{sup_mice}. \edit{Details on hyperparameters used by all dimensionality reduction methods are presented in Sections \ref{sup_sim}, \ref{sup_dengue}, and \ref{sup_aml}.}

\subsection{Simulated scRNA-seq Data}\label{sim_scRNA-seq}

The scPCA technique was tested on a simulated scRNA-seq dataset generated with the \textit{Splat} framework from the \texttt{Splatter} \texttt{R} package  \citep{Zappia2017}. \textit{Splat} simulates a scRNA-seq count matrix by way of a gamma-Poisson hierarchical model. This simulation framework mimics real scRNA-seq data by including hyperparameters to control the number of over- and under-expressed genes (using multiplicative factors for mean expression levels), zero inflation, batch effects, and other technical and biological factors relevant for scRNA-seq data.

A simple dataset of 300 cells and 500 genes was simulated such that the cells were approximately evenly distributed among three biological groups: two groups making up a target dataset and a third group corresponding to a background dataset. 5\% of the genes are differentially expressed between the background dataset and each of the two target datasets but not between the two target datasets, 10\% of the genes are differentially expressed between the background dataset and the first target dataset but not the second target dataset, and 10\% of the genes are differentially expressed between the background dataset and the second target dataset but not the first target dataset. There is overlap between these three sets of genes and, in particular, a total of 98 genes are differentially expressed between the two target datasets. Based on these levels of differential expression, cells are more dissimilar between the two target datasets than between either of the target datasets and the background dataset. Therefore, the samples comprising the background dataset can be viewed as a set of controls for use by cPCA and scPCA. Additionally, a large batch effect was included to confound the biological variation between groups, effectively dividing each biological group into two subgroups of \edit{near} equal size (Figure~\ref{fig:sim}A).

\begin{figure*}[!htbp]
  \centering
  \includegraphics[width = \textwidth]{figures/sim_results}
  \caption{
  {\em Simulated scRNA-seq data.}
  \textbf{A} Plot of the first two principal components of the complete simulated dataset (i.e., the combination of the target and background datasets). The batch effect and the biological signal are responsible for approximately identical amounts of variance. \textbf{B} Two-dimensional representations of the target dataset by t-SNE, UMAP, cPCA, scPCA, and ZINB-WaVE, with accompanying average silhouette widths quantifying the strengths of the batch effect and the biological signal. Only scPCA fully removes the batch effect in two dimensions when batches are not adjusted for explicitly. \textbf{C} A gene-by-gene comparison of the \edit{standardized absolute loadings} in the first loading vectors of cPCA and scPCA, in decreasing order with respect to the values produced by cPCA.}
  \label{fig:sim}
\end{figure*}

PCA, t-SNE, UMAP, cPCA, and scPCA were applied to the log-transformed and column-centered target data\edit{, and SIMLR to the raw target data}, to determine whether these methods could identify the biological signal of interest, i.e., the two groups in the target dataset (Figure~\ref{fig:sim}B. Note that PCA was not included due to the similarity of results to Figure~\ref{fig:sim}A \edit{and SIMLR's embedding is included in the appendix in Figure~\ref{fig:simulated_SIMLR}}). Also note that cPCA was not performed in the traditional manner of \citet{Abid2018}, but with automatic hyperparameter selection as described in Sections \ref{hyp_tune} \edit{and \ref{algo1}}. The number of \textit{a priori} specified clusters for the cPCA and scPCA methods was set to 2, and the column-centered background data were used in their contrastive steps. While PCA, t-SNE, UMAP, \edit{SIMLR,} and cPCA were incapable of completely eliminating the batch effect in their two-dimensional representations, scPCA successfully removed the unwanted variation while producing the tightest clusters, as indicated by the average silhouette widths (see also Figure~\ref{fig:ave_sil_widths}), and generating sparse, interpretable loadings.

To compare the loadings produced by cPCA and scPCA, each of their loading vectors were standardized as follows. The $i^{\text{th}}$ entry of the $j^{\text{th}}$ standardized loading vector is given by
$
\frac{|V_{ij}| - \text{min}_i |V_{ij}|}{\text{max}_i |V_{ij}| - \text{min}_i |V_{ij}|}
$,
where $\mathbf{V}$ is a $p \times k$ loading matrix.
Juxtaposing the \edit{standardized absolute loadings} of the first loading vectors produced by cPCA and scPCA, each of which linearly separate the target dataset's groups, we find the scPCA loadings to be, as expected, much sparser (see Figure~\ref{fig:sim}C). In fact, only 20 genes have non-zero values in scPCA's first loading vector compared to 500 in cPCA; moreover, these 20 genes correspond to those which have the largest absolute entries in cPCA's first loading vector. Furthermore, these genes are among the most differentially expressed in the target dataset, based on the values of their multiplicative differential expression factors (Figure~\ref{fig:sim_de_genes}).

scPCA's results were also compared to the two leading latent factors found by ZINB-WaVE, a method of choice for dimensionality reduction for scRNA-seq data, under conditions in which the batch factor is viewed as known and unknown (Figure~\ref{fig:sim}B). In both cases, ZINB-WaVE was applied to the count matrix of the simulated target dataset with no gene-level covariates. When the batch factor was treated as unknown, no cell-level covariates were included in the model; however, when we treated the batch factor as known, a binary cell-level covariate was added to indicate each sample's batch membership. When the batch effect is not explicitly regressed out in the ZINB-WaVE model, we find the results to be virtually identical to those of PCA. Even when the batch effect is included in the model, the clusters of the biological groups are elongated and less dense than those produced by scPCA, and the first latent factor does not linearly separate the groups.

\subsection{Dengue Microarray Data}\label{dengue_data}

\citet{Kwissa2014} used gene expression microarrays to analyze the whole-blood transcriptome of 47 dengue patients hospitalized at the Siriraj Hospital in Bangkok and 9 local healthy controls. Of the affected patients, 18 were classified as having acute dengue fever (DF), 10 as having acute dengue hemorrhagic fever (DHF), and 19 as convalescent at least four weeks after discharge.

As part of data pre-processing, all but the 500 most variable genes were filtered out. The target dataset consists of the log-transformed microarray expression measures of 47 patients with some form of dengue, while the background dataset consist of the log-transformed microarray expression measures of the control samples. PCA, cPCA, scPCA, t-SNE, and UMAP were then applied to the column-centered target data matrix with the goal of discerning three unique clusters (Figure~\ref{fig:dengue}A), one for each sub-class of dengue (DF, DHF, and convalescent). cPCA and scPCA took as additional input the column-centered background data matrix and specified three clusters \textit{a priori}. t-SNE's embedding was found to be similar to UMAP's and is therefore only included in the supplementary materials (Figure~\ref{fig:dengue_tsne}).

Of the four dimensionality reduction methods, only cPCA and scPCA successfully fully separated the convalescent patients from those with DF and DHF in two dimensions. scPCA's low-dimensional representation was virtually identical to that of cPCA, producing very similar average silhouette widths among classes, though only a tenth of the genes have non-zero values in the first and second columns of the scPCA loading matrix, and the most important genes identified by each methods' first loading vector differ substantially (Figure~\ref{fig:dengue}B). The genes found by scPCA include CD38, HLA-DQB1, and RSAD2 (Viperin), which have been previously associated to the susceptibility to, protection against, or presence of dengue \citep{Castaneda2016,Cardozo2014,Fitzgerald2011}. For a full list of these genes, refer to Table \ref{tab:dengue_1} and Table \ref{tab:dengue_2}. \edit{A gene set enrichment analysis (GSEA) was also performed with the genes contained in these tables to identify the most significant biological processes in which they play a role; details and results are presented in Table \ref{tab:gsea_dengue}.}

No method successfully distinguished between the three sub-classes of dengue. In fact, previous research suggests that the transcriptomes of patients with DF and DHF are virtually indistinct \citep{Kwissa2014}. Instead, \citet{Kwissa2014} found that DF and DHF patients may form distinct clusters based on viral load and concentration of the DENV NS-1 antigen in their plasma. This may explain the sub-clusters within the DF and DHF cases found by UMAP. Though the number of pre-specified clusters for each algorithm was set to three, cPCA's and scPCA's projections onto two dimensions contain two clusters. To test the sensitivity of these methods to this tuning parameter, both methods were reapplied to the data with varying numbers of pre-specified clusters. Each of cPCA's iteration produced virtually identical embeddings (Figure~\ref{fig:dengue_cpca_centers}). However, scPCA's produced identical results to those of PCA when the number of clusters was set to four or higher (Figure~\ref{fig:dengue_scpca_centers}). This may provide an empirical approach to selecting the appropriate number of clusters for scPCA, i.e., selecting the largest value before which the quality of the embedding deteriorates.

\FloatBarrier
\begin{figure*}[!htbp]
  \centering
  \includegraphics[width = \textwidth]{figures/dengue_results}
  \caption{
  {\em Dengue microarray data.}
  \textbf{A} Two-dimensional representations of the target dataset by PCA, UMAP, cPCA, and scPCA, with accompanying average silhouette widths quantifying the strengths of the biological signal.  cPCA and scPCA are the only methods that fully separate the convalescent patients from those with DF and DHF. The second PC of the PCA plot is dominated by some batch effect, and the low-dimensional representation produced by UMAP also appears to be affected by some source of unwanted variation. \textbf{B} The \edit{standardized absolute loadings} in the two leading loading vectors of scPCA are much sparser than those of cPCA, though their two-dimensional embeddings are virtually identical. The genes are in decreasing order of cPCA's \edit{standardized absolute loadings}, demonstrating that the genes with non-zero \edit{loadings} in scPCA generally correspond to the genes with the largest \edit{absolute loadings} in cPCA. This is much more apparent for the second loading vector where the distribution of cPCA's absolute loadings has a thin tail, attributing increased importance to a small subset of genes.}
  \label{fig:dengue}
\end{figure*}

\subsection{Leukemia Patient scRNA-seq Data}\label{leukemia_data}

Finally, we tested scPCA on scRNA-seq data from the cryopreserved bone marrow mononuclear cell (BMMC) samples of two acute myeloid leukemia (AML) patients (Patient 035: 4,501 cells; Patient 027: 7,898 cells), before and after undergoing allogeneic hematopoietic stem cell transplant treatment \citep{Zheng2017}. The BMMCs of two healthy individuals from the same publicly available dataset (Healthy 1: 1,985 cells; Healthy 2: 2,472 cells) were used to generate a control dataset. Following pre-processing, all but the 1,000 most variable genes measured across all 16,856 cells were removed. The scRNA-seq data from the AML patients were then split into separate target datasets since \citet{Zheng2017} found evidence of distinct subpopulation membership following transplantation. Data belonging to the healthy controls were combined to create the background dataset. PCA, t-SNE, UMAP, ZINB-WaVE, \edit{SIMLR,} cPCA, and scPCA were applied to the target datasets to explore differences in the AML patients' BMMCs's transcriptome engendered by the treatment (Figures~\ref{fig:comp_leuk_pat1},~\ref{fig:comp_leuk_pat2},\edit{~\ref{fig:SIMLR_aml_035}, and ~\ref{fig:SIMLR_aml_027}}).

Of the seven dimensionality reduction methods applied to Patient 035's data (Figures~\ref{fig:comp_leuk_pat1}A \edit{ and ~\ref{fig:SIMLR_aml_035}}), cPCA and scPCA best capture the biologically meaningful information relating to treatment status. Each produces linearly separable clusters corresponding to pre- and post-treatment cells; scPCA's projection yields a tighter cluster of pre-transplant cells when compared to that produced by cPCA, and the opposite is true regarding the clusters of post-transplant cells. Additionally, scPCA's projection required considerably less information, even though its results are analogous to cPCA's: 176 genes and 17 genes have non-zero entries in, respectively, the first and second columns of the loading matrix produced by scPCA (Figure~\ref{fig:comp_leuk_pat1}B). In general, the leading loading vectors of cPCA and scPCA place an increased importance on the same genes. \edit{Genes with non-zero \edit{loadings} in scPCA's first and second loading vectors were also subjected to a gene set enrichment analysis to uncover their roles in biological processes; details and results are presented in Table \ref{tab:gsea_aml035}}. Regarding the other methods' results, PCA, t-SNE, \edit{and SIMLR} fail to separate the pre- and post-transplant cells, and UMAP's and ZINB-WaVE's embeddings resemble a trajectory more closely than they do a set of clusters.

Similarly to Patient 035's results, the two-dimensional embeddings of Patient 027's data produced by PCA, t-SNE, UMAP, \edit{SIMLR,} and ZINB-WaVE do not contain distinct clusters of pre- and post-transplant BMMCs \ref{fig:comp_leuk_pat2}; however, cPCA and scPCA generate low-dimensional representations of the data in which samples are clustered based on treatment status. Although cPCA's representation produces denser\edit{, more distinct} groupings, the first two columns of scPCA's loading matrix contain non-zero values in only \edit{five} genes, STMN1, \edit{CA1, LDHA,} PDLIM1\edit{, and C1QBP. The first four genes have been linked to leukemia \citep{Machado-Neto2014,Mentese2017,Wang2014,Holleman2004}, and the last gene, C1QBP, is responsible for a protein that plays a critical role in tumor metabolism \citep{Fogal2010}.}

\FloatBarrier
\begin{figure*}[!htbp]
  \centering
  \includegraphics[width = \textwidth]{figures/aml035_results}
  \caption{{\em AML Patient 035 scRNA-seq data.}
  \textbf{A} The two-dimensional embeddings of the patient's BMMCs produced by PCA, ZINB-WaVE, t-SNE, UMAP, cPCA, and scPCA, with accompanying average silhouette widths quantifying the strengths of the biological signal. cPCA and scPCA produce representations of the data in which the pre- and post-transplant cells form discernible clusters. Based on visual inspection and average silhouette width, scPCA's grouping of pre-transplant cells is denser than that of cPCA's and the opposite is true of the post-transplant cells' cluster. \textbf{B} scPCA's embedding is much sparser, increasing interpretability of the exploratory analysis.}
  \label{fig:comp_leuk_pat1}
\end{figure*}


\subsection{Up-Scaling Computational Efficiency}

The CDM model for this case requires about two orders of magnitude less computational effort than the DEM model (Table \ref{tab:computation}). The CDM simulation uses a comparable number of continuum elements ($29,866$) as in the DEM simulation ($25,898$) for comparison and adequate convergence. The CDM model efficiency can be improved by applying a Selectively Refined Mesh (SRM) where only the areas with stress concentrations and large stress gradients have a strongly refined mesh. With the SRM, a converged CDM solution is achievable with only $3,577$ elements leading to another order of magnitude reduction in computational effort.

\begin{table}[!htbp]
\centering
\caption{Comparison of Computational Time for the DNS}
\label{tab:computation}
\begin{tabular}{@{}ccccc@{}}
\toprule
\textbf{Simulation} & \textbf{No. Continuum} & \textbf{Processor} & \textbf{Slope Failure} & \textbf{Computational} \\
\textbf{Type} & \textbf{Elements} & \textbf{Clock Speed} & \textbf{Load} & \textbf{Time} \\ \midrule
DEM                      & $25,898$                         & $2.20 GHz$                    & $11.2 MPa$                  & $46.5 hr$                  \\
CDM                      & $29,866$                         & $1.80 GHz$                    & $11.5 MPa$                  & $0.65 hr$                  \\
CDM - SRM                      & $3,577$                         & $1.80 GHz$                    & $11.5 MPa$                  & $0.013 hr$                  \\ \bottomrule
\end{tabular}
\end{table}

The DEM simulation was run serially on a $2.2GHz$ CPU while the CDM simulation was run serially on a $1.8GHz$ CPU. Despite the CDM model having more continuum elements than the DEM model, and the CDM model running on a slower CPU, a decrease in computational time of the DEM simulation from $46.5 hr$ to $0.65 hr$ was observed. Running the CDM model with a SRM reduces the total computational time to $0.013 hr$, or eight minutes instead of two days. This large increase in computational efficiency with marginal decrease in model accuracy can be immensely useful for large scale geomechanical problems in NFR.


\documentclass[../notes.tex]{subfiles}

\pagestyle{main}
\renewcommand{\chaptermark}[1]{\markboth{\chaptername\ \thechapter\ (#1)}{}}
\setcounter{chapter}{18}

\begin{document}

\chapter{The First Law of Thermodynamics}
\section{The First Law and Heat Capacity}
\begin{itemize}
    \item \marginnote{8/20:}By the conservation of energy, $\Delta E_\text{int} = Q+W$. This implies the following.
    \item \textbf{First Law of Thermodynamics}: The following formula, which is analogous to conservation of energy.
    \begin{equation*}
        \dd{E_\text{int}} = \dd{Q}-p\dd{V}
    \end{equation*}
    \item The work done \emph{by} the gas is the area under the curve defined by the path between two pressure-volume conditions.
    \begin{figure}[h!]
        \centering
        \begin{subfigure}[b]{0.33\linewidth}
            \centering
            \begin{tikzpicture}[
                every node/.append style={black},
                pics/point/.style args={#1:#2}{code={
                    \node [circle,fill=pix,inner sep=1.5pt,label={#1:$#2$}] {};
                }}
            ]
                \footnotesize
                \path (0.6,0.1) -- ++(0,-0.2) node[below]{${\color{white}V_a}$};
                \fill [orz] (0.6,0) -- plot[domain=0.6:2.5,smooth] (\x,{1/\x}) -- (2.5,0);

                \draw [stealth-stealth] (0,2) node[above]{$p$} -- (0,0) -- (3,0) node[right]{$V$};

                \draw [orx,thick,postaction={decorate},decoration={
                    markings,
                    mark=at position 0.33 with \arrow{>},
                    mark=at position 0.67 with \arrow{>}
                }] plot[domain=0.6:2.5] (\x,{1/\x});
                \draw [orx,thick,dashed]
                    (0.6,0) -- ++(0,{1/0.6}) pic{point=left:a}
                    (2.5,0) -- ++(0,{1/2.5}) pic{point=right:b}
                ;

                \node at (1.55,0.25) {$\int p\dd{V}$};
            \end{tikzpicture}
            \caption{Isothermic process.}
            \label{fig:workIntegrala}
        \end{subfigure}
        \begin{subfigure}[b]{0.32\linewidth}
            \centering
            \begin{tikzpicture}[
                every node/.append style={black},
                pics/point/.style args={#1:#2}{code={
                    \node [circle,fill=pix,inner sep=1.5pt,label={#1:$#2$}] {};
                }}
            ]
                \footnotesize
                \fill [orz] (0.6,0) rectangle (2.5,0.8);

                \draw [stealth-stealth]
                    (0.1,0.8) -- ++(-0.2,0) node[left]{$p_0$}
                    (0.6,0.1) -- ++(0,-0.2) node[below]{$V_a$}
                    (2.5,0.1) -- ++(0,-0.2) node[below]{$V_b$}
                    (0,2) node[above]{$p$} -- (0,0) -- (3,0) node[right]{$V$}
                ;

                \draw [orx,thick,postaction={decorate},decoration={
                    markings,
                    mark=at position 0.33 with \arrow{>},
                    mark=at position 0.67 with \arrow{>}
                }] (0.6,0.8) -- (2.5,0.8);
                \draw [orx,thick,dashed]
                    (0.6,0) -- ++(0,0.8) pic{point=left:a}
                    (2.5,0) -- ++(0,0.8) pic{point=right:b}
                ;

                \node at (1.55,0.35) {$p_0(V_b-V_a)$};
            \end{tikzpicture}
            \caption{Isobaric process.}
            \label{fig:workIntegralb}
        \end{subfigure}
        \begin{subfigure}[b]{0.32\linewidth}
            \centering
            \begin{tikzpicture}[
                every node/.append style={black},
                pics/point/.style args={#1:#2}{code={
                    \node [circle,fill=pix,inner sep=1.5pt,label={#1:$#2$}] {};
                }}
            ]
                \footnotesize
                \path (0.6,0.1) -- ++(0,-0.2) node[below]{${\color{white}V_a}$};
                \draw [stealth-stealth] (0,2) node[above]{$p$} -- (0,0) -- (3,0) node[right]{$V$};

                \draw [orx,thick,postaction={decorate},decoration={
                    markings,
                    mark=at position 0.33 with \arrow{>},
                    mark=at position 0.67 with \arrow{>}
                }] (0.6,1.7) pic{point=left:a} -- node[right]{$W=0$} (0.6,0.4) pic{point=left:b};
            \end{tikzpicture}
            \caption{Isochoric process.}
            \label{fig:workIntegralc}
        \end{subfigure}
        \caption{Work as a geometric integral of a pressure/volume graph.}
        \label{fig:workIntegral}
    \end{figure}
    \begin{itemize}
        \item Thus, for an isothermic process, $W=-\int p\dd{V}$.
        \item For an isobaric process, $W=-P(V_b-V_a)$.
        \item For an isochoric process, $W=0$.
    \end{itemize}
    \item Two processes of getting from condition $a$ to condition $b$:
    \begin{figure}[H]
        \centering
        \begin{tikzpicture}[
            scale=1.8,
            every node/.append style={black},
            pics/point/.style args={#1:#2}{code={
                \node [circle,fill=pix,inner sep=1.5pt,label={#1:$#2$}] {};
            }}
        ]
            \footnotesize
            \draw [stealth-stealth] (0,2) node[above]{$p$} -- (0,0) -- (3,0) node[right]{$V$};

            \draw [blx,thick,dashed] plot[domain=1:2.5] (\x,{1.67/\x});
            \draw [blx,thick,dashed] plot[domain=0.6:2.5] (\x,{1/\x});
            \draw [blx,thick,dashed] plot[domain=0.3:2.5] (\x,{0.5/\x});
            \draw [blx,thick,dashed] plot[domain=0.12:2.5] (\x,{0.2/\x});

            \draw [orx,very thick,postaction={decorate},decoration={
                markings,
                mark=at position 0.13 with \arrow{>},
                mark=at position 0.6 with \arrow{>},
                mark=at position 0.4 with {\node[above right=4pt,circle,draw,thin,inner sep=1pt]{2};}
            }] (0.4,{0.5/0.4}) -- plot[domain=0.8:2] (\x,{1/\x});
            \draw [orx,very thick,postaction={decorate},decoration={
                markings,
                mark=at position 0.25 with \arrow{>},
                mark=at position 0.75 with \arrow{>},
                mark=at position 0.4 with {\node[below left=-2pt,circle,draw,thin,inner sep=1pt]{1};}
            }] (0.4,{0.5/0.4}) pic{point=left:a} plot[domain=0.4:1] (\x,{0.5/\x}) -- (2,0.5) pic{point=above right:b};
        \end{tikzpicture}
        \caption{Alternate paths.}
        \label{fig:alternatePaths}
    \end{figure}
    \begin{itemize}
        \item We know from the first law that $\Delta E_\text{int}$ is the same for both properties.
        \item However, from the above, we know that $W$ is not the same for both (different areas under the curve).
        \item Thus, by the first law, heat cannot be the same (it must be offset to compensate).
    \end{itemize}
    \item \textbf{Molar heat capacity} (of a gas at constant volume): Defined by the following.
    \begin{equation*}
        C_V = \frac{1}{n}\dv{Q_V}{T}
    \end{equation*}
    \begin{itemize}
        \item Tells you how much heat you have to add to raise the temperature a certain amount.
    \end{itemize}
    \item Additionally, since $\dd{W}=0$ at constant volume, $\dd{E_\text{int}}=\dd{Q_V}$.
    \begin{itemize}
        \item Thus, $C_V=1/n\cdot\dv*{E_\text{int}}{T}$.
        \item Consequently, since $E_\text{int}=\frac{3}{2}nRT$, we have that $\dv*{E_\text{int}}{T}=\frac{3}{2}nR$.
        \item Therefore, at constant volume, $C_V=\frac{3}{2}R$.
    \end{itemize}
    \item \textbf{Molar heat capacity} (of a gas at constant pressure): Defined by the following.
    \begin{equation*}
        C_p = \frac{1}{n}\dv{Q_p}{T}
    \end{equation*}
    \begin{itemize}
        \item For the same $\dd{T}$, $\dd{Q_p}\neq\dd{Q_V}$.
    \end{itemize}
    \item We know that $C_p>C_V$.
    \begin{itemize}
        \item This is principally because of the difference between the equations $\dd{E_\text{int}}=\dd{Q}$, pertaining to an isochoric process, and $\dd{E_\text{int}}+\dd{W}=\dd{Q}$, pertaining to an isobaric process.
        \item In the former, we can see that the change in internal energy (which is directly related to the temperature for an ideal gas) is directly reflected by the amount of heat flowing into the system.
        \item However, in the latter, we raise the temperature by the same amount, so internal energy increases by the same amount. However, the gas must expand to maintain its constant pressure. Thus, some work is done. Thus, more heat must flow into the system to account for the work done \emph{and} maintain the same temperature.
    \end{itemize}
    \item For some $\dd{T}$ (i.e., some $\dd{E_\text{int}}$).
    \begin{itemize}
        \item Heat added at constant volume: $\dd{E_\text{int}}=\dd{Q_V}=nC_V\dd{T}$.
        \item Heat added at constant pressure: $\dd{E_\text{int}}=\dd{Q_p}-p\dd{V}=nC_p\dd{T}-p\dd{V}$.
        \item The amount of heat you add in both cases will be different, but the change in internal energy will be the same.
        \item Thus,
        \begin{align*}
            nC_V\dd{T} &= nC_p\dd{T}-p\dd{V}\\
            C_p-C_V &= \frac{p}{n}\dv{V}{T}\\
            &= \frac{p}{n}\cdot\frac{nR}{p}\\
            &= R\\
            C_p &= R+\frac{3}{2}R\\
            &= \frac{5}{2}R
        \end{align*}
    \end{itemize}
\end{itemize}

\section{Storing Energy in Bonds}
\begin{itemize}
    \item We know that
    \begin{align*}
        \frac{3}{2}kT &= \overline{KE}_\text{molecule}\\
        &= \frac{1}{2}\mu\bar{v}^2\\
        &= \frac{1}{2}\mu\bar{v}_x^2+\frac{1}{2}\mu\bar{v}_y^2+\frac{1}{2}\mu\bar{v}_z^2
    \end{align*}
    \item \textbf{Equipartition of energy theorem}: Every \textbf{degree of freedom} has associated with it, on average, $\frac{1}{2}kT$ of energy.
    \begin{itemize}
        \item Can be derived, but we're not expected to know this.
    \end{itemize}
    \item \textbf{Degree of freedom}: Any direction that can be expressed as the time derivative of a coordinate, squared.
    \item For example, translational kinetic energy has three degrees of freedom: $v_x^2$, $v_y^2$, and $v_z^2$.
    \begin{itemize}
        \item Notice that since each has $\frac{1}{2}kT$ of associated energy, the total translational kinetic energy has $\frac{3}{2}kT$ of energy, as desired.
    \end{itemize}
    \item Rotational kinetic energy has three degrees of freedom:
    \begin{align*}
        K_\text{rot} &= \frac{1}{2}I\omega^2\\
        &= \frac{1}{2}I_x\omega_x^2+\frac{1}{2}I_y\omega_y^2+\frac{1}{2}I_z\omega_z^2
    \end{align*}
    \begin{itemize}
        \item For a diatomic molecule, it's symmetry implies that $K_\text{rot}$ actually equals just $\frac{1}{2}I_y\omega_y^2+\frac{1}{2}I_z\omega_z^2$, if its axis is aligned with the $x$-axis.
    \end{itemize}
    \item Thus, adding up these five degrees of freedom, we have that for each diatomic molecule, $E_\text{int}=\frac{5}{2}kT$.
    \begin{itemize}
        \item This implies that for a mole of diatomic molecules, $E_\text{int}=\frac{5}{2}RT$, making $C_V=\frac{5}{2}R$ and $C_P=\frac{7}{2}R$.
    \end{itemize}
    \item Factoring in vibrational energy, if we let $x$ be the separation between atoms, we have $x^2$ and $v_x^2$, corresponding to potential- and kinetic-energy degrees of freedom.
    \begin{itemize}
        \item This yields $C_V=\frac{7}{2}R$ and $C_P=\frac{9}{2}R$.
    \end{itemize}
    \item For hydrogen, we have changes in the molar heat capacity as a function of temperature.
    \begin{itemize}
        \item This can be explained by the increase in degrees of freedom at higher temperatures (first only translational, then rotational gets added in, then vibrational).
        \item Why? Quantum physics --- more excited states become available at higher temperatures.
    \end{itemize}
    \item If $D$ is the number of degrees of freedom of a molecule, then
    \begin{equation*}
        C_V = D\cdot\frac{1}{2}R
    \end{equation*}
    \begin{itemize}
        \item On the other hand, it is always true that $C_P=C_V+R$.
        \item Similarly, $E_\text{int}=D\cdot\frac{1}{2}nRT$.
    \end{itemize}
\end{itemize}

\section{Thermal Expansion}
\begin{itemize}
    \item Consider a potential energy vs. bond stretch graph.
    \begin{itemize}
        \item Since you can't push atoms too close together, the graph will be asymmetric.
        \item Indeed, at higher energies, the bond will \emph{stretch} more, but it will not shrink that much more.
        \item Thus, the average separation increases, and the gas expands.
    \end{itemize}
    \item \textbf{Coefficient of linear expansion}: The following quantity, where $L$ is the length of something that's expanding.
    \begin{equation*}
        \alpha = \frac{1}{L}\dv{L}{T}
    \end{equation*}
    \begin{itemize}
        \item This formula essentially tells us how much something expands per unit increase in temperature: $\dd{L}=\alpha L\dd{T}$.
    \end{itemize}
    \item If we have a circular metal disk with a hole, raising its temperature will enlarge the whole thing (both the outer radius and inner radius increase).
\end{itemize}

\section{Adiabatic Processes}
\begin{itemize}
    \item \marginnote{8/23:}Extra Gazes office hours Tuesday 3:30-5:00 PM CT.
    \item Your grade is calculated two ways, and Gazes takes the higher of the two:
    \begin{itemize}
        \item HW (15\%), Lab (15\%), Quiz + midterm (35\%), Final (35\%).
        \item HW (15\%), Lab (15\%), Quiz + midterm (20\%), Final (50\%).
    \end{itemize}
    \item Analyzing an adiabatic process.
    \begin{itemize}
        \item $\dd{Q}=0$.
        \item Thus, since $\dd{E_\text{int}}=-p\dd{V}$, $\dd{E_\text{int}}=nC_V\dd{T}$, and $pV=nRT$, we have that
        \begin{align*}
            -\frac{nRT}{V}\dd{V} &= nC_V\dd{T}\\
            \frac{\dd{T}}{T}+\frac{R}{C_V}\frac{\dd{V}}{V} &= 0
        \end{align*}
        \item But since $R=C_p-C_V$, we have that
        \begin{equation*}
            \frac{R}{C_V} = \frac{C_p-C_V}{C_V} = \frac{C_p}{C_V}-1 = \gamma-1
        \end{equation*}
        where $\gamma$ is the \textbf{ratio of specific heats}.
        \item We now integrate the above equation.
        \begin{align*}
            \int\frac{\dd{T}}{T}+\int(\gamma-1)\frac{\dd{V}}{V} &= \int 0\\
            \ln T+(\gamma-1)\ln V &= C\\
            \ln(TV^{\gamma-1}) &= C\\
            TV^{\gamma-1} &= \e[C]
        \end{align*}
        \item Therefore,
        \begin{equation*}
            TV^{\gamma-1} = \text{constant}
        \end{equation*}
        \item It follows if we multiply the above by $pV/T$ that $pV^\gamma=\text{constant}$ (this equation defines the slopes of adiabatic processes on a $pV$-graph [see the $Q=0$ lines in Figures \ref{fig:ottoCycle} and \ref{fig:carnotCycle} for examples]).
    \end{itemize}
    \item \textbf{Ratio of specific heats}: The quotient of the molar heat capacity of a gas at constant pressure and the molar heat capacity of that same gas at constant volume. \emph{Denoted by} $\bm{\gamma}$.
    \begin{itemize}
        \item For example, $\gamma=5/3$ for a monoatomic gas, $\gamma=7/5$ for a diatomic gas with rotation, and $\gamma=9/7$ for a diatomic gas with rotation and vibration.
        \item Note that it is always true that $\gamma>1$.
    \end{itemize}
\end{itemize}

\end{document}


\section{Summary and discussion}
\label{sec:discussion}

In this section, we shall address the burning question: which waveform analysis method should my experiment use?  We surveyed 8 methods, heuristics~(figure~\ref{fig:method}), deconvolution~(figure~\ref{fig:deconv}), neuron network~(figure~\ref{fig:cnn-performance}) and regressions(figures~\ref{fig:dcf}--\ref{fig:fbmp-performance}), from the simplest to the most dedicated.  To make a choice, we shall investigate the waveform reconstruction precisions under different light intensities $\mu$.

\subsection{Performance}

First, we constrain the candidates by time consumption, algorithm category and $D_\mathrm{w}$.  Figure~\ref{fig:chargesummary} shows the $D_w$ summary of all 8 methods with the same sample as figure~\ref{fig:method}.
\begin{figure}[H]
    \centering
    \resizebox{\textwidth}{!}{\input{figures/summarycharge.pgf}}
    \caption{\label{fig:chargesummary} $D_\mathrm{w}$ for all the methods with the same dataset as figure~\ref{fig:method}. The error bars are 5 to 95 percentiles.  The methods colored with dark blue are discussed in section~\ref{sec:discussion}. }
\end{figure}
The $D_\mathrm{w}$ performance of Waveform shifting, peak finding and Fourier deconvolution are suboptimal.  MCMC is too time consuming and the principle is not too different from FBMP.  We therefore leave out those in the following discussion.

Table~\ref{tab:efficiency} gives the efficiency of four performant methods on our simulation datasets. The CNN on GPU is the fastest. The number of iteration for LucyDDM and gradient descent, as well as $t'_i$ sampling frequency of FBMP can be decreased to trade for speed. The time consumed by FBMP contains the initialization time of a LucyDDM pre-conditioner.

\begin{table}[H]
    \centering
    \caption{\label{tab:efficiency} Reconstruction Efficiency.}
    \begin{tabular}{cc}
        \hline
        & Performance/$10^{5}$Waveform \\
        \hline
        CNN & 187.8s (GPU)\tablefootnote{one graphics card of NVIDIA\textsuperscript{\textregistered} A100.} \\
        Direct Charge Fitting & 1560.1s (CPU)\tablefootnote{100 CPU cores of AMD EYPC\texttrademark\ 7742} \\
        LucyDDM & 540.0s (CPU) \\
        FBMP & 3557.4s (CPU) \\
        \hline
    \end{tabular}
\end{table}

% Figure of Merits

The $D_\mathrm{w}$ and RSS dependence on $\mu$ of four remaining methods, LucyDDM, Fitting, CNN and FBMP, are plotted in figure~\ref{fig:wdistsummary} and \ref{fig:rsssummary}.  When $\mu$ increases $D_\mathrm{w}$ approach each other, while RSS diverges.  The $D_\mathrm{w}$ behavior is observed in section~\ref{sec:cnn} that with large $N_\mathrm{PE}$ the overall PE times dominate.  It is harder to balance $D_\mathrm{w}$ and RSS with large $N_\mathrm{PE}$.  Notice that in qualitative discussion, large $N_\mathrm{PE}$, large light intensity $\mu$ and large pile-ups are used interchangably.
\begin{figure}[H]
  \begin{subfigure}[b]{\textwidth}
    \resizebox{\textwidth}{!}{\input{figures/vs-wdist.pgf}}
    \caption{\label{fig:wdistsummary}Dependence of Wasserstein distance by four methods on light intensity.}
  \end{subfigure}

  \vspace{0.5em}
  \begin{subfigure}[b]{\textwidth}
    \resizebox{\textwidth}{!}{\input{figures/vs-rss.pgf}}
    \caption{\label{fig:rsssummary}Dependence of residual sum of squares by four methods on light intensity.}
  \end{subfigure}
  \caption{\label{fig:summary}The dependence of $D_\mathrm{w}$~\subref{fig:wdistsummary} and RSS~\subref{fig:rsssummary} on light intensity $\mu$ for typical Cherenkov (left) and scintillation (right) configurations.  With more pile-ups, $D_\mathrm{w}$ tends to converge while RSS diverges.  The pile-up effect is more significant for Cherenkov case because the time scale of light curve is narrower.  The error bars are 5--95 percentiles.  Method abbreviations are the same as figure~\ref{fig:chargesummary}. }
\end{figure}

CNN and FBMP are the kings of $D_\mathrm{w}$ and RSS.  Because their loss functions are chosen accordingly.  It is therefore informative to study the posterior charge distribution which is not related to the loss function of any method.

\subsection{Charge fidelity and sparsity}

All the discussed mothods output $\hat{q}_i$ as the inferred charge of the PE at $t_i'$.  Evident in figure~\ref{fig:recchargehist}, FBMP considers and almost retains the true charge distribution.  It is the only method modeling PE correctly.

In contrast, the distributions of LucyDDM, Fitting, and CNN are severely distorted.  During the optimization process of $D_\mathrm{w}$ or RSS, the number of $q_i$ is a constant. Many $\hat{q}_i$ are inferred to be fragmented values.  In this view, retaining charge distribution is a manifestation of sparsity.  FBMP has the best sparsity because it chooses a PE configuration $\bm{z}$ before fitting $\hat{q}_i$.  CNN somehow learns the sparsity better than Fitting, although the latter in theory has self-regulated sparsity.  It is interesting to notice, but we are ignorant of the mechanism.

\begin{figure}[H]
  \centering
  \resizebox{0.6\textwidth}{!}{\input{figures/recchargehist.pgf}}
  \caption{\label{fig:recchargehist} $\hat{q}_i$ distributions on the same waveform dataset as figure~\ref{fig:method}.  Method abbreviations are defined in figure~\ref{fig:chargesummary}. ChargePDF is the charge distribution introduced for simulation in section~\ref{subsec:spe}. The cut-off near 0 in LucyDDM is an artifect of thresholding in eq.~\eqref{eq:fdconv2}.}
\end{figure}

For large $N_\mathrm{PE}$ waveforms the sparsity condition is by definition lost.  The equivalence of charge fidelity and sparity implies that FBMP breaks for large $N_\mathrm{PE}$ cases where PMTs operates in current mode, as we shall see in the next sections.

\subsection{Time resolution}
\label{subsec:timeresolution}

Like figure~\ref{fig:summary}, we show the dependence on $\mu$ of bias~(figure~\ref{fig:biasmethods}) and resolution~(figure~\ref{fig:deltamethods}) for different time estimators in two typical experimental setups.  Comparing with figure~\ref{fig:reso-diff}, In figure~\ref{fig:deltamethods}, all 4 methods (LucyDDM, Fitting, CNN, FBMP) provide better time resolution than only using the first PE during construction.

\begin{figure}[H]
  \begin{subfigure}[b]{\textwidth}
    \centering
    \resizebox{\textwidth}{!}{\input{figures/vs-bias.pgf}}
    \caption{\label{fig:biasmethods} Time reconstruction bias of methods}
  \end{subfigure}
  \begin{subfigure}[b]{\textwidth}
    \centering
    \resizebox{\textwidth}{!}{\input{figures/vs-deltamethodsdiv.pgf}}
    \caption{\label{fig:deltamethods} Time reconstruction ratio of resolution $\sigma/\sigma_{\mathrm{ALL}}$ of methods}
  \end{subfigure}
  \caption{ALL is the $\hat{t}_\mathrm{ALL}$ estimator defined in eq.~\eqref{eq:2}. LucyDDM, Fitting, CNN use eq.~\eqref{eq:pseudo}.  FBMP has its own natural $\hat{t}_0$ estimator in eq.~\eqref{eq:bayesianinter}. Error bars are 5--95 percentiles.}
\end{figure}

From figure~\ref{fig:biasmethods}, we can see that the $t_0$ estimation biases for the four methods are all similar to those based on all PEs. The considerable bias for the datasets with $\tau_l=\SI{20}{ns}$, and $\sigma_l=\SI{5}{ns}$ when $\mu$ is small is intrinsic.

% Time bin

The reconstruction results of hit time, $\bm{\hat{t}}$, are discrete values (time bin edges in a DAQ window). Obtaining continuous values of $\bm{\hat{t}}$ may result in better analysis results. The results in the study show that when using refined time bins, the time resolution, the sparsity of reconstructed PEs, and all figures of merits (Wasserstein distance and RSS) are better; however, a lost efficiency.

\subsection{Intensity resolution}
\label{subsec:chargereconstruction}

Integration estimation (footnoted as $\mathrm{int}$) with $\hat{\mu}=\int w(t)\mathrm{d}t$ is deteriorated by the randomization process, such as charge distribution and Gaussian white noise in section~\ref{subsec:spe}.

Considering that the energy response is a multiplier in the detector and the existing $\hat{\mu}$ bias, which we discuss below, the charge resolution is defined on $\log{\hat{\mu}}$. A perfect $\mu$ estimation of a waveform is that $\hat{\mu}_\mathrm{ALL}=N_\mathrm{PE}$. The corresponding resolution $\sigma_{\log\mu}$ is the lower limit of charge reconstruction.

Figure~\ref{fig:biasmu} shows the bias of $\log\hat{\mu}$ of all four methods, where $\mathrm{FBMP}$ corresponds to $\hat{\mu}$ defined in equation~\eqref{eq:bayesianinter}, and for the other three methods (LucyDDM, Fitting, CNN), $\hat{\mu}$ is the sum of $\hat{\bm{q}}$. As $\log\hat{\mu}$ is unfavorably low when $\hat{\mu}$ is near 0 due to the lower long tail of charge distribution, the bias of integration is less than zero if $\mu$ is small. A detailed explanation for the bias of FBMP is in appendix~\ref{sec:mubias}, and briefly it comes from the degeneracy of reconstruction results and the non-ergodic approximation of FBMP.

\begin{figure}[H]
  \begin{subfigure}[b]{\textwidth}
    \centering
    \resizebox{\textwidth}{!}{\input{figures/vs-biasmut.pgf}}
    \caption{\label{fig:biasmu} intensity reconstruction bias of methods}
  \end{subfigure}
  \begin{subfigure}[b]{\textwidth}
    \centering
    \resizebox{\textwidth}{!}{\input{figures/vs-deltamethodsdivmu.pgf}}
    \caption{\label{fig:deltamu} intensity reconstruction ratio of resolution $\sigma/\sigma_{\log\mu}$ of methods}
  \end{subfigure}
  \caption{intensity reconstruction performance}
\end{figure}

Figure~\ref{fig:deltamu} shows the charge resolution of both integration and the four methods. While estimating the PE number $\hat{\bm{z}}$ in each time bin $\hat{\bm{t}}$ with model selection, $\hat{\mu}$ of FBMP is not critically affected by the charge distribution, which gives a better charge resolution especially when $\mu$ is small.

In figure~\ref{fig:deltamethods}, FBMP shows convincing resolution as expected. The other three methods (LucyDDM, Fitting, CNN) shows charge resolution between integration.

% Pedestal & Hardware

We do not evaluate the pedestal of waveforms in this work, which is a potential future work to do. Additionally, widely equipped PMTs bring the data storage pressure. A voltage data pre-process hardware can solve this problem with waveform analysis, equivalent to or even more effective than data compression, which is also a prospective work.


%
% This is Chapter 3 file (chap3.tex)
%
\chapter{Non-linear Plasma Dynamics}\label{chap:chap3}

    \section{Introduction to Turbulence}\label{sec:intr3}

        In laminar flow, different layers of fluid move smoothly without much mixing between layers,
        and the characteristic quantities/parameters - like velocity, pressure, or density - vary
        smoothly in a predictable way. A system where these quantities fluctuate in a chaotic
        fashion is called turbulent and the phenomena is called turbulence\index{turbulence}. Because of the chaotic
        nature of the fluctuations, unlike the laminar flow, prediction of the exact state of the
        system is essentially impossible. This makes a system extremely complicated to study, so
        much so that Sir Horace Lamb once remarked \citep{Goldstein1969}:

        \begin{quote}
             I'm an old man now, and when I die and go to heaven there are two matters on which I
             hope for enlightenment. One is quantum electrodynamics, and the other is turbulent
             motion of fluids. And about the former I'm rather optimistic.
            %\footnote{To be honest as an Indian, I find the idea of a $19^{th}$ century British's
            %assumption that they would go to heaven rather optimistic.}
        \end{quote}

        Turbulence is extremely complicated and the fact that it is ubiquitous in nature (and in
        most man made processes involving fluids) makes it
        unavoidable.%\footnote{Not that we would have avoided it anyway.}.

        Whether a given fluid system will develop turbulence largely depends on its viscosity, which
        is the liquid equivalent of friction and is a measure of how easy it is to for the liquid to
        flow \citep{Chapman1916, Jeans1905} and its Reynolds number
        \citep{Reynolds1883,Reynolds1886,Matthaeus1980}. A small Reynolds number\index{Reynolds number} means that the
        system is laminar whereas a high value Reynolds number implies turbulent flow. For a neutral
        fluid it is defined as:
        \begin{align}
            R_{\rm e} = L \frac{u}{\nu} \label{eq:rnld}
        \end{align}
        where $L$ is the characteristic length of the system, $u$ is the mean flow velocity and
        $\nu$ is the kinematic viscosity. For a similar amount of force or external pressure, a
        highly viscous fluid or one with low $R_{\rm e}$ can maintain laminar flow for much longer
        duration than a fluid with low viscosity or high $R_{\rm e}$. Presence of viscosity in a
        fluid leads to interaction between different layers or scales and results in energy transfer
        from larger to smaller scales through eddies which eventually reaches the smallest scale and
        dissipate as heat \citep{Kolmogorov1941a, Kolmogorov1941} in a process called energy cascade
        in turbulence. In a weakly collisional and magnetized plasma, the presence of charged
        particle and magnetic field complicates the process. For a system like solar wind, the
        situation is further complicated because of the relatively similar size of the system and
        the mean free path\footnote{Mean free path is defined as the average distance travelled by
        particles between two successive collisions.}\citep[Appendix 2]{Echim2010} both of which are
        of the order of 1\,au \citep[Table 1]{Verscharen2019} and thus one cannot use the classic
        methodology developed by \citet{Enskog1917} and \citet{Chapman1918}.

        Turbulence cascade has far reaching consequences for both neutral fluids and plasmas. It
        provides a pathway for the dissipation or transfer of energy from large scales, where they
        can be introduced, to smaller scales. In the next section (\Cref{sec:inter3b}) we will look
        at some of the consequences of turbulence in space plasmas. We discuss only those which are
        relevant to this thesis. In \Cref{sec:nlts} we discuss the linear and non-linear time scales
        associated with their respective phenomena.

    \section{Consequences of Turbulence in Space Plasmas} \label{sec:inter3b}

        In-situ observations and theoretical interpretations have established the ubiquitous
        presence of turbulence in space plasmas \citep[and references
        therein]{Matthaeus2011,Matthaeus2021}. In this section we discuss three of the major
        consequences that arise because of turbulence in space plasmas. Though these three are not
        the only consequences of turbulence, these were selected because of their relevance to this
        thesis, as we will see in \Cref{chap:chap5,chap:chap6,chap:chap7}.

        \subsection{Heating of plasma} \label{sec:hop}

            In space plasmas, under the assumption that the magnetic field changes slowly (slower
            than the ion gyrotropic time scale), the magnetic moment ($\mu$) of the particle is
            conserved \citep{Baumjohann1996,Verscharen2019}. Thus, we can write:
            \begin{align}
                \frac{d \mu}{dt} & = 0 \label{eq:mu_0}
            \end{align}
            where, $\mu = m_{\rm p}\,w_{\rm \perp p}^2/(2B)$, $m_{\rm p}$ is the proton mass,
            $w_{\rm \perp p}^2$ is the perpendicular thermal velocity and $B$ is the magnitude of
            magnetic field. Writing \Cref{eq:mu_0} in terms of the proton-perpendicular temperature
            using \Cref{eq:temp}, we have:
            \begin{align}
                \frac{d}{dt} \left(\frac{k_{\rm B} T_{\rm \perp p}}{B}\right) & = 0 \label{eq:mu_1}
            \end{align}
            or:
            \begin{align}
                T_{\rm \perp p} & \propto B \label{eq:mu_2}
            \end{align}
            In a similar vein for the parallel direction, under the assumption of no dissipation, we
            have:
            \begin{align}
                \frac{d}{dt} \left(\frac{k_{\rm B} T_{\rm \parallel p} B^2}{n_{\rm p}^2}\right) & = 0 \label{eq:mu_3}
            \end{align}
            or:
            \begin{align}
                T_{\rm \parallel p} & \propto \left(\frac{n_{\rm p}}{B}\right)^2 \label{eq:mu_4}
            \end{align}
            These two conservation laws (\Cref{eq:mu_2,eq:mu_4}) for the \textit{double-adiabatic
            invariants} are also called \textit{Chew–Goldberger–Low or CGL invariants} (for a bit
            more detailed discussion and derivation of CGL invariants, see see \Cref{apdx:B}).

            In the inner heliosphere, the magnitude of magnetic field (B) varies with solar distance
            as $B \propto r^{-1.5}$ \citep{Hellinger2013,Hanneson2020}, and the proton density
            varies as $n_{\rm p} \propto r^{-1.9}$ \citep{Hellinger2013}. If the CGL invariants were
            actively being conserved, the radial dependence for the perpendicular and parallel
            temperatures would be:
            \begin{align}
                %\begin{split}
                    T_{\rm \perp p} & \propto r^{-1.5} \label{eq:tperp_trend}\\
                    T_{\rm \parallel p} & \propto r^{-0.8} \label{eq:tpar_trend}
                %\end{split}
            \end{align}
            However, in-situ observations in the inner as well as outer heliosphere show a much
            flatter curve than those predicted by \Cref{eq:tperp_trend,eq:tpar_trend}. Based on
            Helios 1 and Helios 2 data, \citet{Hellinger2013} reported the value of exponents to be
            $-0.58$ and $-0.59$ for perpendicular and parallel temperatures respectively and $-0.58$
            for the scalar temperature for $r \in [0.3, 1]\,\rm{au}$.

            Flatter than expected temperature curves imply the existence of some mechanism which
            continues to heat the solar wind beyond the corona in both the parallel and
            perpendicular directions. Indeed, several studies
            \citep{ColemanJr1968,Verma1995,SorrisoValvo2007,MacBride2008} predict around
            $1000\,\rm{kJ/kg/sec}$ is being added as internal energy to the plasma at 1\,au.
            %\citet{Bandyopadhyay2020a} observed almost 2 order of magnitude higher heating rate
            %closer to the sun at $r \approx 0.2\,\rm{au}$.
            The dissipation of this energy at least partially accounts for the flatter radial trend
            in solar-wind temperature than that predicted by the double adiabatic expansion
            assumption.

        \subsection{Anisotropy} \label{sec:aniso}

            In \Cref{sec:instab2} we discussed the fact that because of anisotropy\index{anisotropy}, VDFs have excess
            free energy that results in development of microkinetic instabilities, though we did not
            discuss the origin of such anisotropies. As we saw in \Cref{sec:hop}, turbulence results
            in the transfer of heat from larger to smaller scales. However, in presence of an
            external background magnetic field the rate at which transfer occurs is not identical in
            each direction. Because of an uneven transfer along the parallel and perpendicular
            direction relative to the average magnetic field (inhibition along the direction of
            magnetic field), there is an imbalance between the amount of heating in different
            directions, resulting in anisotropy \citep{Shebalin1983,Oughton1994}.

        \subsection{Intermittency\index{Intermittency}}\label{sec:intmt}

            The solar wind at 1\,au exhibits localized structures that have been studied since the
            pioneering work of \citet{Burlaga1968}, \citet{Hudson1970}, \citet{Tsurutani1979}, and
            more recently by \citet{Ness2001}, \citet{Neugebauer2006}, \citet{ErdHoS2008}. Several
            studies have found evidence that plasma turbulence generates these structures
            dynamically \citep{Matthaeus1986, Veltri1999, Osman2013}. The structures are
            inhomogeneous and highly intermittent \citep{Osman2011, Osman2013,Greco2008}.
            Intermittency or burstiness in measured properties of turbulence is typically associated
            with the dynamical formation of coherent structures in space. These arise as a direct
            consequence of discontinuities in the magnetic field
            \citep{Greco2008,Greco2009,Vasquez2007}.

            One method for identifying a discontinuity in a time series of magnetic-field (or any
            other field in general) data is Partial Variance of Increments (PVI) \citep{Greco2008}.
            PVI is a powerful and reliable tool for identifying and locating such regions and it is
            unbiased towards any special structure since it cares only about the discontinuities in
            the magnetic field. This also manifests as a shortcoming of the technique since one
            cannot use it to study different kinds of discontinuities like radial or tangential
            discontinuities. \citet{Greco2008} defines PVI\index{PVI} as:
            \begin{align}
                \mathcal{I}(t, \delta t) & = \frac{|\Delta \mathbf{B}(t, \delta t)|}{
                    \sqrt{\langle |\Delta \mathbf{B}(t, \delta t)|^2 \rangle}} \label{eq:pvi}
            \end{align}
            where, $\Delta \mathbf{B}(t, \delta t) = \mathbf{B}(t+\delta t) - \mathbf{B}(t)$, is the
            vector increment in magnetic field at any given time $t$ and a time lag of $\delta t$.
            $\langle ...  \rangle$ is the ensemble average over a period of time, and $\mathcal{I}$
            is the normalized PVI. For studying local structures induced by turbulence, $\delta t$
            is typically chosen to be, assuming the validity of Taylor's hypothesis
            \citep{Taylor1938} which was found to be valid for inner heliosphere
            \citep{Chasapis2021}, of the order of $d_{\rm i}$.

    \section{Linear and Non-linear Time Scales} \label{sec:nlts}

        Since turbulence is not the only process that governs the dynamics, we must compare its
        characteristic timescale with other with those of other relevant processes. As we saw in
        \Cref{sec:instab2}, linear instabilities grow at growth rates of $\gamma_{\max}$. Inverse of
        $\gamma_{\max}$ gives us a linear time scale\index{time scale!linear} associated with such microinstabilities.
        \begin{align}
            \tau_{\rm lin} & = \frac{2\,\pi}{\left(\gamma_{\max}/\Omega_{\rm cp}\right)} \label{eq:lt}
        \end{align}
        Here we have scaled time scale with the proton cyclotron frequency ($\Omega_{\rm{cp}}$) to
        get a dimensionless timescale. This gives us an idea of timescales required by such linear
        processes to affect the local plasma.

        In a similar vein, one can compute nonlinear frequency associated with turbulence at any
        position \textbf{r} for a lag length scale of $\ell$ as follows\footnote{Ideally, velocity
        and not the magnetic field should be used for computing $\omega_{\rm nl}$. However, neither
        of the spacecraft data we used has enough resolution for such computation. We thus fall back
        to using magnetic field under the assumption of Alfv\'enic fluctuations.}:
        \begin{align}
            \omega_{\rm nl} \sim \delta b_\ell/\ell \label{eq:omnl}
        \end{align}
        where $\delta b_\ell$ is the change in the longitudinal magnetic field:
        \begin{align}
            \delta b_{\ell} = \left \lvert\hat{\boldsymbol{\ell}}
            \mathbf{\cdot} \left[\mathbf{b} (\mathbf{r} + \boldsymbol{\ell}) - \mathbf{b}
            (\mathbf{r})\right]\right\lvert \label{eq:db}
        \end{align}
        where \textbf{b} is the total magnetic field expressed in local Alfv\'en speed units
        ($\mathbf{b} = \mathbf{B}/\sqrt{\mu_\circ n_{\rm p} m_{\rm p}}$). Thus nonlinear time scale\index{time scale!nonlinear}
        has the expression:
        \begin{align}
            \tau_{\rm nl} & = \frac{2\,\pi}{\left(\omega_{\rm nl}/\Omega_{\rm cp}\right)} \label{eq:nlt}
        \end{align}
        These two processes under certain conditions might compete with each other and depending on
        the value of other kinetic or turbulent parameters one or the other may dominate. A
        simplistic understanding of this competition would imply that if one time scale is
        significantly smaller than the other, then the processes associated with former time scale
        will dominate the dynamics of the plasma. However, as we will see in \Cref{chap:chap7} the
        situation is a bit more complicated than that.


\typeout{new file: Old_Species_RadCal_Chapter.tex}

\chapter{The Original RadCal Species}
\label{chap:old_species}

The characteristics of the original RadCal species, \textit{i.e.} the species present in the 1993 RadCal version, are presented in this chapter. These species are gas phase $\rm H_2O$, $\rm CO_2$, CO, and $\rm CH_4$. The data pertaining to these species have not been changed since the 1993 RadCal version.

\section{Water vapor: $\rm H_2O$}

The water molecule is an asymmetric top molecule with the oxygen atom in the middle. The bound length is 0.958 \AA{} and the bond angle is 104.45\textdegree. Because of its large permanent electric dipole moment ($6.16 \times 10^{-30}$ Coulomb-meters) in its equilibrium configuration, it has strong rotational bands. In addition, it three moments of inertia differ greatly: the rotational constants are A~=~27.877~cm$^{-1}$, B~=~14.512~cm$^{-1}$, and C~=~9.285~cm$^{-1}$ (values from Ref.~\cite{NIST2013}). Recall that the rotational constant relates to the molecule principal moment of inertia with:
\begin{equation}
 A = \dfrac{h}{8 \pi^2 c I_A},
\end{equation}
where $I_A$ is one of the principal moment of inertia, $h$ is the Planck constant ($6.626 \times 10^{-34}$~J$\cdot$s), and $c$ is the speed of light in vacuum, $c$ = $299,792,458$~m/s.

Because the three moments of inertia are greatly different from each other and are small, they give rise to a widespread and apparently disorderly array of rotation lines. This makes the infrared spectrum of $\rm H_2O$ very complex. $\rm H_2O$ has three fundamentals vibration modes, which are reported in Table \ref{table::fund_H2O}.

\begin{table}[ht]
    \centering
    \caption{Observed wavenumbers associated with fundamental vibration modes of $\rm H_2O$.}
    \label{table::fund_H2O}
    \begin{tabular}{|c|c|c|} \hline
    Band & Upper State & H$^{16}$OH  \\
    \hline
    $\nu_1$  &  100 & 3657.05 \\
    $\nu_2$  &  010 & 1594.75 \\
    $\nu_3$  &  001 & 3755.93 \\
    \hline
   \end{tabular}
\end{table}

The important bands of water vapor fall into a number of distinct classes: rotation band from 0 to 900~cm$^{-1}$, $\nu_2$ bands from 900 to 2400~cm$^{-1}$ (6.3 $\rm \mu m$ region), $\nu_1$, $\nu_3$, and 2$\nu_2$ bands from 2800 to 4400~cm$^{-1}$ (2.7 $\rm \mu m$ region), 6 distinguishable groups of lines between 4500 and 11,000~cm$^{-1}$: $\Omega$, $\Psi$, $\Phi$, $\tau^c$, $\sigma^c$, $\rho^c$; these are combined vibrational modes and overtones.

The strongest bands at atmospheric temperature are the rotational (0 -- 900~cm$^{-1}$) and the 6.3~$\rm \mu m$ band. In fire application, the shift of the dominant region of the Planck distribution toward higher wavenumber renders the 2.7~$\mu$m very important for high temperature emission purposes.

In Radcal, the $\rm H_2O$ mean absorption coefficient $\bar{\kappa}$ has been tabulated for the wavenumbers between 50 to 9300~cm$^{-1}$, and for the following temperatures: 300~K, 600~K, 1000~K, 1500~K, 2000~K, 2500~K. Figure~\ref{fig:H2O_300-2500K} displays the spectral mean absorption coefficient, in units of $\rm atm^{-1}.cm^{-1}$, at these different temperatures. The data originate from Ludwig \textit{et al.}~\cite{Ludwig1973}. Experimental data have been fitted using the statistical Goody narrow band model.

\begin{figure}[p]
\begin{tabular*}{\textwidth}{l@{\extracolsep{\fill}}r}
\includegraphics[height=2.5in]{Figures/H2O_300K.png} &
\includegraphics[height=2.5in]{Figures/H2O_600K.png} \\
\includegraphics[height=2.5in]{Figures/H2O_1000K.png} &
\includegraphics[height=2.5in]{Figures/H2O_1500K.png} \\
\includegraphics[height=2.5in]{Figures/H2O_2000K.png} &
\includegraphics[height=2.5in]{Figures/H2O_2500K.png}
\end{tabular*}
\caption{Spectral variations of the $\rm H_2O$ mean absorption coefficient $\bar{\kappa}_{\om}$, in $\rm atm^{-1}.cm^{-1}$, at various temperatures as tabulated in RadCal.\label{fig:H2O_300-2500K}}
\end{figure}

To calculate the band overlap parameter $\beta$, the Lorentz HWHM is considered constant over the spectrum and is calculated using Eq.~\ref{eq:gamma_L}, while the effective line spacing is given by \cite{Ludwig1973}:
\begin{equation}
 d = \exp\left(0.7941 \sin\left(0.036 \omega- 8.043\right) + 2.294 + 0.3004 \times 10^{-2} T - 0.366 \times 10^{-6} T^2\right),
\end{equation}
where $T$ is the local temperature.

\clearpage

\section{Carbon dioxide: $\rm CO_2$}

Carbon dioxide is a linear, symmetric molecule. Its bond length is 115.98~pm in ground state, and its rotational constant is 0.3906~cm$^{-1}$. Because of the symmetry, $\rm CO_2$ has no permanent dipole moment. It has four vibrational modes, but only two fundamental IR vibration modes. Five distinct bands are included in RadCal (four are calculated, one is tabulated), see Table \ref{Table::CO2}. All the bands are modeled after the Goody model.

\begin{table}[h!]
    \centering
    \caption{Spectral bands of $\rm CO_2$ included in RadCal.}
    \label{Table::CO2}
    \begin{tabular}{|c|c|c|c|l|}
      \hline
      Band \# & \multicolumn{2}{|l|}{Bounds (cm$\rm ^{-1}$) } & Method & Notes: \\
      \cline{1-5}
      1 &  500 & 880  & tabulated&  15 $\mu$m region - strong (for atmospheric application)\\
      2 &  880 & 1100 & modeled  & 10 $\mu$m region\\
      3 & 1975 & 2475 & modeled  & 4.9 and 4.3 $\mu$m region - very strong\\
      4 & 3050 & 3800 & modeled  & 2.7 $\mu$m region - important for fire application\\
      5 & 4550 & 5275 & modeled  & 2.0 $\mu$m region\\
      \hline
    \end{tabular}
\end{table}

\begin{figure}[p]
\begin{tabular*}{\textwidth}{l@{\extracolsep{\fill}}r}
\includegraphics[height=2.5in]{Figures/CO2_300K.png} &
\includegraphics[height=2.5in]{Figures/CO2_600K.png} \\
\includegraphics[height=2.5in]{Figures/CO2_1200K.png} &
\includegraphics[height=2.5in]{Figures/CO2_1500K.png} \\
\includegraphics[height=2.5in]{Figures/CO2_1800K.png} &
\includegraphics[height=2.5in]{Figures/CO2_2400K.png}
\end{tabular*}
\caption{Spectral variations of the $\rm CO_2$ mean absorption coefficient $\bar{\kappa}_{\om}$, in $\rm atm^{-1}.cm^{-1}$, at various temperatures as tabulated in RadCal.\label{fig:CO2_300-2500K}}
\end{figure}

The strongest band in the $\rm CO_2$ spectrum is Band~3. At 300~K, it has an integrated band intensity (see Section~\ref{sec:integrated_intensity} for a definition of this quantity) of 2963~atm$^{-1}$cm$^{-2}$. The tabulated data (mean absorption coefficient $\bar{\kappa}_{\om}$) were obtained from experiments for the following temperatures: 300, 600, 1200, 1500, 1800, 2400~K. The modeled bands use the harmonic oscillator approximation. All the bands uses the Goody statistical narrow band model. Figure~\ref{fig:CO2_300-2500K} to displays the mean absorption coefficient at 300, 600, 1200, 1500, 1800, 2400~K, as tabulated (Band 1) and modeled in RadCal (Bands 2 to 5).

\clearpage

\section{Carbon monoxide: $\rm CO$}

Carbon Monoxide is a diatomic molecule and as such, it has only one fundamental vibrational mode. RadCal includes one distinct band, the 1600--2400 band. It corresponds to the stretching of the triple bond $\rm C \equiv O$. This band is modeled and it uses the model from Malkmus and Thompson \cite{Malkmus1962}. It is based on the anharmonic oscillator model and uses band integrated intensity measurements. Figure~\ref{fig:CO_300-2500K} displays the spectral mean absorption coefficient of CO at 300~K and 2500~K.

\begin{figure}[ht]
\begin{tabular*}{\textwidth}{l@{\extracolsep{\fill}}r}
\includegraphics[height=2.5in]{Figures/CO_300K.png} &
\includegraphics[height=2.5in]{Figures/CO_2500K.png}
\end{tabular*}
\caption{Spectral variations of the CO mean absorption coefficient $\bar{\kappa}_{\om}$, in $\rm atm^{-1}.cm^{-1}$, at 300~K and 2500~K as tabulated in RadCal.\label{fig:CO_300-2500K}}
\end{figure}

The first overtone (centered at $ \om\approx 4260\;\rm {cm^{-1}}$) is not accounted for as its integrated band intensity is negligible at standard temperature and pressure. See Table \ref{Table::CO}. At 295~K, the integrated band intensity of Band~1 is $260\;\rm {atm^{-1}cm^{-2}}$. The statistical narrow band model associated with $\rm CO$ is the Goody model. Recommended temperatures of use range from 295~K to 2500~K.

\begin{table}[ht]
      \centering
      \caption{Spectral bands of $\rm CO$ included in RadCal.}
      \label{Table::CO}
    \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    Band \# & \multicolumn{2}{|l|}{Bounds (cm$\rm ^{-1}$) } & Method & Assignment & $\alpha(T=296 \; {\rm K}) \; (\rm {atm^{-1} cm^{-2}})$   \\
    \cline{1-6}
    1 & 1600 & 2400 & modeled &  $\rm C\equiv O$ stretching & 260  \\
    \hline
   \end{tabular}
\end{table}

\clearpage

\section{Methane: $\rm CH_4$}

Methane is a spherical top molecule of tetrahedral shape with the carbon atom occupying the center of the tetrahedron. It belongs to the point group $T_d$. The methane IR spectrum is the result of the vibration-rotation modes of the $\rm C-H$ groups. It has nine vibrational modes, but due to its symmetry, this translates into only two distinct IR active fundamental vibration frequencies. In RadCal, the methane IR spectrum is divided into three distinct bands, which include the fundamentals plus degenerate (overtone) vibration modes. Bands 1 and 2 are calculated using tabulated mean absorption coefficient data which were obtained by Brosmer and Tien, \cite{Brosmer1985} for temperatures of 290, 600, 850~K. Band 3 is modeled, using a just-overlapping line model; the integrated intensities are computed in the harmonic-oscillator, rigid-rotator approximation. See Penner and Gray, Ref.~\cite{Gray1965}, for more details. In RadCal, the Elsasser model is used to calculate the collision optical depth. Table \ref{Table::CH4} tabulates the different characteristics of the $\rm CH_4$ bands.

\begin{table}[ht]
      \centering
      \caption{Spectral bands of $\rm CH_4$ included in RadCal.}
      \label{Table::CH4}
    \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    Band \# & \multicolumn{2}{|l|}{Bounds (cm$\rm ^{-1}$) } & Method & Assignment & $\alpha(T=296 \; {\rm K}) \; (\rm {atm^{-1} cm^{-2}})$   \\
    \cline{1-6}
    1 & 1150 & 1600 & tabulated &  $\rm C-H$ Bend    & 237  \\
    2 & 2700 & 3250 & tabulated &  $\rm C-H$ Stretch & 212  \\
    3 & 3400 & 5000 & modeled   &  $\rm C-H$ Stretch &  35  \\
    \hline
   \end{tabular}
\end{table}
The strongest bands are Bands~1 and 2 which at standard temperature and pressure have an integrated band intensity of $237\;\rm {atm^{-1}cm^{-2}}$ and $212\;\rm {atm^{-1}cm^{-2}}$, respectively. Figure~\ref{fig:CH4_290-850K} displays the spectral mean absorption coefficient $\bar{\kappa}$ for $\rm CH_4$ at 290~K, 600~K, and 850~K, respectively.

\begin{figure}[ht]
\begin{tabular*}{\textwidth}{l@{\extracolsep{\fill}}r}
\includegraphics[height=2.5in]{Figures/CH4_290K.png} &
\includegraphics[height=2.5in]{Figures/CH4_600K.png} \\
\multicolumn{2}{c}{\includegraphics[height=2.5in]{Figures/CH4_850K.png}}
\end{tabular*}
\caption{Spectral variations of the $\rm CH_4$ mean absorption coefficient $\bar{\kappa}_{\om}$, in $\rm atm^{-1}.cm^{-1}$, at various temperatures as tabulated in RadCal.\label{fig:CH4_290-850K}}
\end{figure}

\clearpage

\section{Soot}

The soot spectral absorption coefficient is calculated using the below formulation:
\be\label{eq:soot_kappa}
\bar{\kappa} = 7.0 \om f_v
\ee
where $f_v$ is the soot volume fraction and is dimensionless. The narrow band transmissivity of soot is calculated as:
\be\label{eq::soot_optical_depth}
\bar{\tau}(0 \rightarrow s) = \exp\left(-\bar{\kappa} s \right).
\ee
From Eq.~\ref{eq:soot_kappa}, it can be seen that particulate matters tend to absorb soot in the high wavenumber part of the spectrum. Figure~\ref{fig:SOOT_300K} plots the spectral transmissivity of a homogeneous layer of soot at 300~K with a soot volume fraction value of 5~ppm ($5\times 10^{-6}$) and a thickness of 20~cm.

\begin{figure}[ht]
\begin{center}
 \includegraphics[height=2.5in]{Figures/SOOT_300K.png}
\end{center}
\caption{Predicted spectral transmissivity of a homogeneous layer of soot at 300~K, of thickness 20~cm, and with a soot volume fraction of 5~ppm ($f_v = 5\times 10^{-6}$).\label{fig:SOOT_300K}}
\end{figure}


% !TEX root=../main.tex
\section{Experimental Setup}
\label{sec:ocnnexperiment-setup}

In this section, we show the empirical effectiveness of OC-NN formulation over the state-of-the-art methods on real-world data. Although our method is applicable in any context where autoencoders may be used for feature representation, e.g., speech.  Our primary focus will be on non-trivial high dimensional images.
%\vspace{-0.3 cm}
\subsection{Methods compared}
\label{sec:methods_compared}
We compare our proposed one-class neural networks (OC-NN) with the following state-of-the-art methods for anomaly detection:
\let\labelitemi\labelitemii
\begin{itemize}{}
    \item \textbf{OC-SVM -SVDD} as per formulation in ~\cite{scholkopf2002support}
    \item \textbf{Isolation Forest} as per formulation in ~\cite{liu2008isolation}.
    \item \textbf{Kernel Density Estimation (KDE)} as per formulation in ~\cite{parzen1962estimation}.
    \item \textbf{Deep Convolutional Autoencoder (DCAE)} as per formulation in ~\cite{masci2011stacked}.
    \item \textbf{AnoGAN} as per formulation in ~\cite{radford2015unsupervised}.
    \item \textbf{Soft-Bound and One Class Deep SVDD} as per formulation in ~\cite{pmlrv80ruff18a}.
    \item \textbf{Robust Convolutional Autoencoder (RCAE)} as per formulation in ~\cite{chalapathy2017robust}.
    \item \textbf{One-class neural networks (OC-NN) \footnote{\url{https://github.com/raghavchalapathy/oc-nn}}}, our proposed model as per Equation \ref{eqn:oc-nn}.
\end{itemize}
We used Keras~\cite{chollet2015keras} and  TensorFlow~\cite{abadi2016tensorflow} for the implementation of OC-NN, DCAE, RCAE \footnote{\url{https://github.com/raghavchalapathy/rcae}}
For OC-SVM\footnote{\url{http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html}} and Isolation Forest\footnote{\url{http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html}}, we used publicly available implementations.

\begin{table}[!t]
    \centering
    \renewcommand{\arraystretch}{1.25}
    \setlength{\tabcolsep}{6pt}
    %\scalebox{0.85}{
    \begin{tabular}{@{}llll@{}}
        \toprule
        \toprule
        Dataset & \# instances & \# anomalies & \# features \\
        \toprule
        {\tt Synthetic}       & 190   & 10                                        & 512 \\
        {\tt MNIST}           & single class   & 1\%  ( from all class)            & 784 \\
        {\tt CIFAR$-$10}      & single class    & 10\% ( from all class)           &3072 \\
        {\tt GTSRB }           & 1050 (stop signs )     & 100 (boundary attack)  &3072 \\
        \bottomrule
    \end{tabular}
    %}
    %\vspace{0.1 cm}
    \caption{Summary of datasets used in experiments.}
    \label{tbl:datasets}
    \vspace{-\baselineskip}
\end{table}

% \vspace{-0.5 cm}
\subsection{Datasets}
%%\vspace{-0.2 cm}
We compare all methods on synthetic and four real-world datasets as summarized below :
\begin{itemize}
    \item {\tt Synthetic Data}, consisting of 190 normal data points 10 anomalous points drawn from normal distribution with dimension $512$.
    \item {\tt MNIST}, consisting of 60000 $28\times28$ grayscale images of handwritten digits  in 10 classes, with 10000  test images~\cite{lecun2010mnist}.
    \item {\tt GTSRB }, are $32\times32$ colour images comprising of  adversarial boundary attack on stop signs boards~\cite{stallkamp2011german}.
    \item {\tt CIFAR$-$10} consisting of 60000 $32\times32$ colour images in 10 classes, with 6000 images per class~\cite{krizhevsky2009learning}.
\end{itemize}

For each dataset, we perform further processing to create a well-posed anomaly detection task, as described in the next section.
\vspace{-0.3 cm}

\subsection{Evaluation of Models}
\label{sec:evaluationOfModels}
% Shallow methods compared
\subsubsection{{Baseline Model Parameters:}}
\label{sec:baselinemodel_parameters.}
The proposed OC-NN method is compared with several state-of-the-art baseline models as illustrated in Table~\ref{tab:ocnn_results}. The model parameters of shallow baseline methods are used as per implementation in~\cite{pmlrv80ruff18a}. Shallow Baselines (i) Kernel OC-SVM/SVDD with
Gaussian kernel. We select the inverse length scale $\gamma$ from
$\gamma$  $\in$ {$2^{ - 10}$ , $2^{ - 9}$, . . . , $2^{ - 1}$ via grid search using the performance on a small holdout set (10) \% of randomly drawn test samples). We run all experiments for $\nu = 0.1$ and report the better result. (ii) Kernel density estimation (KDE). We select the bandwidth \textit{h} of the Gaussian kernel from \textit{h} $ \in {2^{0.5} , 2 ^1, . . . , 2^5}$ via 5-fold cross-validation using  the log-likelihood score. (iii) For the  Isolation Forest (IF) we set the number of trees to \textit{t} = 100 and the sub-sampling size to $\Psi = 256$, as recommended in the original work~\cite{pmlrv80ruff18a}.
\vspace{-0.4cm}
% Deep Anomaly detection models compared
\subsubsection{ {Deep Baseline Models:}}
We compare OC$-$NN models to  four deep approaches described Section ~\ref{sec:methods_compared}.
We choose to train DCAE using the Mean sqaure error (MSE) loss since our experiments are on image data. For the DCAE encoder, we employ the same network architectures as we use for Deep SVDD, RCAE, and OC-NN models. The decoder is then constructed symmetrically, where we substitute max-pooling with upsampling. For AnoGAN we follow the implementation as per ~\cite{radford2015unsupervised} and set
the latent space dimensionality to $256$.
For we Deep SVDD, follow the implementation as per ~\cite{pmlrv80ruff18a} and employ a  two phase learning rate schedule (searching + fine tuning) with initial learning rate $\eta$ = $10^{ - 4}$  and subsequently $\eta$ = $10^{ - 5}$. For DCAE we train 250 + 100 epochs, for Deep SVDD 150 + 100. Leaky ReLU activations are used with leakiness $\alpha=0.1$. For RCAE we train the autoencoder using the robust loss and follow the parameter settings as per formulation in ~\cite{chalapathy2017robust}.

%OC-NN Model Architecture proposed
\subsubsection{ {One-class neural Networks (OC-NN):}}
\label{model_architecture}
In OC-NN technique firstly, a deep autoencoder is trained to obtain the representative features of the input as illustrated in Figure~\ref{fig:model-architecture}(a). Then, the encoder layers of this pre-trained autoencoder is copied and fed as input to the feed-forward network with one hidden layer as shown in Figure~\ref{fig:model-architecture}(b). The summary of feed forward network architecture's used for various datasets is presented in Table~\ref{tbl:feed-forward-OC-NN}. The weights of encoder network are not frozen (but trained) while we learn feed-forward network weights, following the algorithm summarized in Section~\ref{sec:algorithm}. A feed-forward neural network consisting of single hidden layer, with  linear activation functions produced the best results, as per Equation~\ref{eqn:oc-nn}.  The optimal value of parameter $\nu$ $\in$ ${[0, 1]}$ which is equivalent to the percentage of anomalies for each data set, is set according to respective outlier proportions.

\vspace{-0.2cm}
% Model architectures tested
% \begin{figure*}
% \subfloat[Autoencoder .\label{sfig:autoencoder-for-representation-learning}]{%
%   \includegraphics[width=.40\linewidth]{images/transferLearningAE}%
% }\hspace{0.5cm}
% \subfloat[one-class neural networks. \label{sfig:oc-nn-model-architecture}]{%
%   \includegraphics[width=.40\linewidth]{images/oneClassNN_model}%
% }
% \caption{Model architecture of Autoencoder  and the proposed one-class neural networks (OC-NN).}
% \label{fig:model-architecture}
% \end{figure*}
%  End of the Figure

\begin{figure}[!t]
     \begin{subfigure}[b]{1\textwidth}
   \centering
   {\includegraphics[scale=0.50]%{catswithrotatedcats}}
{images/transferLearningAE}}
\caption{Autoencoder.}
        \end{subfigure}%
        \hfill
        \vspace{2mm}
     \begin{subfigure}[b]{1\textwidth}
\centering
   {\includegraphics[scale=0.50]{images/oneClassNN_model}}
 \caption{One-class neural networks.}
        \end{subfigure}%
    \caption{
     Model architecture of Autoencoder  and the proposed one-class neural networks (OC-NN).
    }
    \label{fig:model-architecture}
\end{figure}

%Table containing the feed forward network architectures used in experiments
\begin{table}[!t]
    \centering
    \renewcommand{\arraystretch}{1.25}
    \setlength{\tabcolsep}{6pt}
    %\scalebox{0.85}{
    \begin{tabular}{@{}llll@{}}
        \toprule
        \toprule
         Dataset& \#    Input (features) & \# hidden layer( or output) \# optional layer  \\
        \toprule
        {\tt Synthetic }      & 512     & 128   & 1 \\
        {\tt MNIST}           & 32      & 32    & None \\
        {\tt CIFAR$-$10}      & 128     & 32    & None \\
        {\tt GTSRB }          & 128     & 32    & 16 \\
        \bottomrule
    \end{tabular}
    %}
    %\vspace{0.1 cm}
    \caption{Summary of best performing feed-forward network architecture's used in OC-NN model for experiments.}
    \label{tbl:feed-forward-OC-NN}
    \vspace{-\baselineskip}
\end{table}
% end of the table

% \begin{figure}
%     \centering
%     \includegraphics[scale=0.38]{images/s1}
%     \caption{Decision Score Histogram of anomalous vs normal data points, {\tt synthetic} dataset.}
%     \label{fig:synthetic-histogram}
% \end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%
%
%   Thesis template by Youssif Al-Nashif
%
%   May 2020
%
%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Introduction}
\label{introduction}

% Big Ideas.
\hspace*{0.3cm} Text mining is the process of extracting insight from text documents using computational and statistical methods. A common task in text mining is document clustering, where the natural language of the documents is reformatted and used to group documents into clusters of similar topics or text content. These type of tasks are known as unsupervised clustering. Unsupervised clustering is the use of algorithms to take unlabeled data and estimate potential clusters, or groupings, of the observations of concern; supervised learning methods, like support vector machines or logistic regression, use labeled data to ``learn" how to discern between groups of data. Since the data are guiding the definition of clusters, or categories, this is an unsupervised approach. Popular methods to cluster text include k-means, hierarchical methods, and topic models. Most of these methods utilize term frequency-inverse document frequency measures  or bag-of-words methods to do clustering, though these methods often divide words and thus may lose meaning, or context, surrounding the word of concern. In an effort to cluster documents while still preserving the relationship between words, text can be modeled with graphs which better preserve the context surrounding a word. Text can be represented with bigram graphs, where the edge between two vertices is the result of two words appearing adjacent in the text. This idea can be extended to more general n-grams, and further extended to skip-n-grams, resulting in even richer representations of the text. The similarity of two of graphical representations can then be assessed using modern methods called graph kernels. Graph kernels produce a measure of similarity between graphs, thus allowing for further machine learning to take place. With a measure of similarity between two graphs, we can do an assortment of clustering methods.\\

The chosen graph representation of the observational unit of text is an extension of bigram graphs. The use of skip-grams here is a way to connect ideas between words that would have normally attained meaning through the ``context" of the sentence. By connecting words that appear close to one another while not being immediately adjacent, we are creating ``skip-grams"-- an extension of the bigram where the window for which words are considered adjacent is widened. A graphic in Figure 1.1 demonstrates this.\\

\begin{figure}
 \includegraphics[width=6in]{Content/Images/Concept_map.png}
 \caption{A map of concepts used and how the methods used here work together. Example is a skip-gram graph of $k=2$.}
 \end{figure}

\hspace*{0.3cm} Many methods used in text mining use bag-of-words techniques, but these methods do not always preserve the context and relationships between words. The graph representations for text, used here, aim to preserve the relationship and context between words. The idea of bigrams (pairs of two words) can be extended to ``skip-grams”; skip-grams are words which appear within some width $w$ of each other. This leads to more connections, and a richer graph representation of the text. This skip-gram method is similar to that used in the popular machine learning package Word2Vec, which uses the “continuous bag of words” or ``continuous skip-grams”. In this paper, a graph representation of the text is constructed, using skip-gram methods, for each document in the datasets. \\

The graph representations are compared with a graph kernel, which is a measure of the similarity of two graphs. In this study, the graph kernel of choice is a ``edge histogram kernel”, because the kernel utilizes the labels of the vertices to assess the similarity of two graphs– not just the topology. Additionally, this is a computationally cheaper than other graph kernels. For each graph, the kernel produces a measure of similarity to the other $n-1$ graphs in the dataset. We can use machine learning methods which work nicely with kernels, e.g. support vector machine (supervised) or hierarchical clustering (unsupervised). The hierarchical clustering methods will be applied here for two datasets, and the results compared.\\

Alternatively, we could use this similarity measure between two graphs, to assess how similar other graphs in the dataset may be. That is, we can compare the similarity of graphs $B$ and $C$ through their similarity to $A$. To do this, a KDE (Kernel Density Estimation) to estimate a kernel density curve, the curve is then used to partition the rest of the dataset into clusters. The KDE bandwidth is modified to produce more or less local minima/maxima that can be then used to identify potential clusters. Basic calculus can be used to locate to inflection points that will serve as intervals for which a cluster will be defined. These methods were developed, but were not included in the final analysis; they are available in the appendices and in a GitHub repository. \\

To test these methods, two different text data sets will be considered. First, reddit comments from subreddits pertaining to mental health. Using the \texttt{\{redditExtractoR\}} package \cite{rivera2015package}, the comments are collected if they match criteria set by the query. From the subreddit \texttt{r/mentalhealth}, posts are collected if they have chosen keywords and at least 10 comments. The keywords considered are ``anxious", ``anxiety", ``depressed", ``depression", ``mental", ``illness", ``scared", ``afraid", ``sad", ``emotion", ``anger", ``angry", ``upset", ``suicide", ``abuse", ``emotional", ``help", and ``addiction". These words do not represent all words which define or indicate mental health topics may be present; these words were chosen by the researcher arbitrarily. The result of the queries is thousands of comments. Mental health continues to be a growing focus at the national level, and the discussion on reddit is often candid and open, due to reddit's reputation as an anonymous website. The second data set being considered in the study, is NHTSA report data. These reports are form the Special Crash Investigation (SCI) reports from the NHTSA (National Highway Traffic Safety Administration). The pdf documents that are considered in this study are those that involved ambulance(s) in the incidents highlighted in the reports. These events are considered ``edge cases" and could prove useful to those working in autonomous vehicles. \\

These datasets were chosen because they are two very different styles of writing. The reddit posts are often in contemporary and casual use of language, possibly including ``netspeak", while the NHTSA reports feature a technical writing style with plain language, absent of hyperbole or sarcasm. These datasets show two very different sides of text data, and comparisons will be made to how each dataset responds to the methods outlined above. \\

Throughout the analysis, all the code used and datasets were documented for future use. The code is well documented in the form of explanatory code notebooks (RMarkdown), code comments, and supplementary markdown files. Datasets, scripts, notebooks, and supplementary files are all made available through a public GitHub repository. Part of the primary focus while developing these methods was to produce them a manner which follows good practices for reproducibility and open source software development. The scripts can all be used in a manner that allows a workflow from raw text to clustered documents; the workflow is documented in the form of a ``script map" or a process flow diagram, so new users can understand the process more clearly.