\section{\src{comp_caldyn_horiz}}

\subsection{Description}

Kernel \src{comp_caldyn_horiz} is taken from the original subroutine
\src{compute_caldyn_horiz} in \DYNAMICO.
%
This subroutine is originally defined in module \src{caldyn_gcm_mod}.
%
This module defines subroutine \src{caldyn}, which is the main
subroutines for dynamics part of the model, and several sub-subroutines
for various terms in the governing equation, such as potential
vorticity, geopotential, etc.
%
This subroutine calculates several horizontal terms, including mass
flux, Bernouilli term, etc.

\subsection{Discretization and code}

\autoref{l:definition_comp_caldyn_horiz} shows the definition part of this subroutne,
and \autoref{f:pad_comp_caldyn_horiz_1}, \ref{f:pad_comp_caldyn_horiz_2}, \ref{f:pad_comp_caldyn_horiz_3}
show the PAD of this.

\begin{LstF90}[%
caption={Definition part of \src{compute_caldyn_horiz}},%
label={l:definition_comp_caldyn_horiz}%
]
  SUBROUTINE compute_caldyn_horiz(u,rhodz,qu,theta,pk,geopot, hflux,convm, dtheta_rhodz, du)
  USE icosa
  USE disvert_mod
  USE exner_mod
  USE trace
  USE omp_para
  IMPLICIT NONE
    REAL(rstd),INTENT(IN)  :: u(iim*3*jjm,llm)    ! prognostic "velocity"
    REAL(rstd),INTENT(IN)  :: rhodz(iim*jjm,llm)
    REAL(rstd),INTENT(IN)  :: qu(iim*3*jjm,llm)
    REAL(rstd),INTENT(IN)  :: theta(iim*jjm,llm)  ! potential temperature
    REAL(rstd),INTENT(INOUT) :: pk(iim*jjm,llm) ! Exner function
    REAL(rstd),INTENT(IN)  :: geopot(iim*jjm,llm+1)    ! geopotential

    REAL(rstd),INTENT(INOUT) :: hflux(iim*3*jjm,llm) ! hflux in kg/s
    REAL(rstd),INTENT(INOUT) :: convm(iim*jjm,llm)  ! mass flux convergence
    REAL(rstd),INTENT(INOUT) :: dtheta_rhodz(iim*jjm,llm)
    REAL(rstd),INTENT(INOUT) :: du(iim*3*jjm,llm)

    REAL(rstd) :: cor_NT(iim*jjm,llm)  ! NT coriolis force u.(du/dPhi)
    REAL(rstd) :: urel(3*iim*jjm,llm)  ! relative velocity
    REAL(rstd) :: Ftheta(3*iim*jjm,llm) ! theta flux
    REAL(rstd) :: berni(iim*jjm,llm)  ! Bernoulli function

    INTEGER :: i,j,ij,l
    REAL(rstd) :: ww,uu
\end{LstF90}

Where \src{u}, \src{rhodz}, \src{qu}, \src{geopot} are wind velocity on the edge,
mass, potential vorticity on the edge, and geopotential, respectively.
\src{pk}, \src{hflux}, \src{convm}, \src{dtheta_rhodz}, \src{du} are Exner function,
horizontal mass flux on the edge, mass flux convergence,
time derivative of the mass-weighted potential temperature,
and time derivative of wind velocity on the edge.
%
Local arrays \src{cor_NT}, \src{urel}, \src{Ftheta}, \src{berni} are
Coriolis's force, relative velocity, potential temperature flux and
Bernoulli function, respectively.
%
All of these arrays are two dimensional.
%
First dimension is for horizontal index, and the size depends on the point
where the variable is defined, since \DYNAMICO adopts C-grid.
%
Second dimension is for vertical index, and the size is \src{llm},
except \src{llm+1} for \src{geopot} that is defined on the half level in
vertical, while others are defined on the full level.

This subroutine is relatively long, and is able to be split by three
sections.
%
In the first section (\autoref{f:pad_comp_caldyn_horiz_1}), there is one $l$-loop
and two $ij$-loop in it.
%
The first one calculates mass flux \src{hflux} and potential
temperature flux \src{Ftheta} at the edge of each control volume.
%
The second loop calculates convergence of mass flux \src{convm}
and convergence of potential temperature flux \src{dtheta_rhodz}.

\begin{figure}[tbp]
\centering
\includegraphics[scale=.6]{figs/caldyn_horiz_sec1.pdf}
 \caption{PAD of \src{compute_caldyn_horiz(1)}}\label{f:pad_comp_caldyn_horiz_1}
\end{figure}

The second section (\autoref{f:pad_comp_caldyn_horiz_2}) calculates potential
vorticity contribution to \src{du} based on the TRiSK scheme
\citep{Ringler:2010:UAE:1749635.1750220}.
%
Here \src{wee} is interpolating weight, prepared in module
\src{geometry} in original \DYNAMICO.
%
Note that since \src{caldyn_conserv} is set as
\src{energy} in this kernel program, second choice of CASE is selected.

\begin{figure}[tbp]
\centering
\includegraphics[scale=.4]{figs/caldyn_horiz_sec2.pdf}
 \caption{PAD of \src{compute_caldyn_horiz(2)}}\label{f:pad_comp_caldyn_horiz_2}
\end{figure}

The last section (\autoref{f:pad_comp_caldyn_horiz_3}) calculates Bernoulli term
first, then adds gradients of it and Extern functions to \src{du}.
%
Here Bernoulli term is sum of kinetic energy and geopotential.
%
Note that \src{boussinesq} is set as \src{.true.} in this kernel
package, Exner function \src{pk} is calculated in advance to calculate
Bernoulli term \src{berni}.

\begin{figure}[tbp]
\centering
\includegraphics[scale=.5]{figs/caldyn_horiz_sec3.pdf}
 \caption{PAD of \src{compute_caldyn_horiz(3)}}\label{f:pad_comp_caldyn_horiz_3}
\end{figure}

\clearpage

\subsection{Input data and result}

Input data file is prepared and you can download from official server using
\file{data/download.sh} script.
%
This data file is created by original \DYNAMICO\footnotemark with
Held-Suarez case parameter set included in the original source archive.
%
\footnotetext{with slight modification by AICS.}
%
Max/min/sum of input/output data of the kernel subroutine are output as
a log.
%
Below is an example of \src{$IAB_SYS=Ubuntu-gnu-ompi} case.

\begin{LstLog}
 [KERNEL] comp_caldyn_horiz
 *** Start  initialize
                iim, jjm, llm:    23    25    19
             ij_begin, ij_end:    48   528
     ij_begin_ext, ij_end_ext:    24   552
             ll_begin, ll_end:     1    19
        t_right, t_rup, t_lup:     1    23    22
     t_left, t_ldown, t_rdown:    -1   -23   -22
        u_right, u_rup, u_lup:     0  1173   575
     u_left, u_ldown, u_rdown:    -1  1150   553
           z_rup, z_up, z_lup:   598     0   597
     z_ldown, z_down, z_rdown:   -23   575   -22
               caldyn_conserv:     1
                   boussinesq:     F
                            g:     9.80000000
 +check[pk_prev         ] max=  1.0014594722514462E+03,min=  0.0000000000000000E+00,sum=  6.9872296819747351E+06
 +check[hflux_prev      ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[convm_prev      ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[dtheta_rhodz_pre] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[du_prev         ] max=  3.4399140149818440E-03,min= -3.0658810374294527E-03,sum=  5.5109794763703335E-01
 +check[le              ] max=  1.3457165724385556E+05,min=  0.0000000000000000E+00,sum=  1.6031419146648201E+08
 +check[Ai              ] max=  3.4618288017294556E+10,min=  0.0000000000000000E+00,sum=  1.7746401564746273E+13
 +check[de              ] max=  4.5171816240714993E+06,min=  0.0000000000000000E+00,sum=  4.7785815753077912E+08
 +check[Av              ] max=  4.1228713627140027E+11,min=  0.0000000000000000E+00,sum=  3.2428753277257527E+13
 +check[Wee             ] max=  5.9893054722291683E-01,min= -5.8540209553487599E-01,sum=  2.4695023837951297E+01
 *** Finish initialize
 *** Start kernel
 ### check point iteration:           1
 ### Input ###
 +check[u               ] max=  4.1278968179782127E-01,min= -4.1278968179782127E-01,sum=  1.6791131703073393E+01
 +check[rhodz           ] max=  1.2306877011993038E+03,min=  0.0000000000000000E+00,sum=  5.3979591836733194E+06
 +check[qu              ] max=  1.0339537867296609E-06,min= -8.4408169682701225E-07,sum=  3.9419811615778674E-04
 +check[theta           ] max=  8.0139914420291746E+02,min=  0.0000000000000000E+00,sum=  3.8582633571973117E+06
 +check[geopot          ] max=  3.8250620498369227E+05,min=  0.0000000000000000E+00,sum=  1.1718001851963627E+09
 +check[pk_prev         ] max=  1.0014594722514462E+03,min=  0.0000000000000000E+00,sum=  6.9872296819747351E+06
 +check[hflux_prev      ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[convm_prev      ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[dtheta_rhodz_pre] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[du_prev         ] max=  3.4399140149818440E-03,min= -3.0658810374294527E-03,sum=  5.5109794763703335E-01
 ### Output ###
 +check[pk              ] max=  1.0014594722514462E+03,min=  0.0000000000000000E+00,sum=  6.9872296819747351E+06
 +check[hflux           ] max=  3.1805763161244854E+07,min= -2.8604204589892026E+07,sum=  2.4131331333014986E+08
 +check[convm           ] max=  1.0361970643226587E-03,min= -1.0359249303947807E-04,sum= -1.5233533963107249E-01
 +check[dtheta_rhodz    ] max=  3.2251351666935379E-01,min= -3.3676276308628725E-02,sum= -5.3720539414185993E+01
 +check[du              ] max=  3.4404317002518906E-03,min= -3.0804630348046005E-03,sum=  5.5048589972605033E-01
 ### final iteration:        1000
 ### Validation : grid-by-grid diff ###
 +check[pk              ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[hflux           ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[convm           ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[dtheta_rhodz    ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 +check[du              ] max=  0.0000000000000000E+00,min=  0.0000000000000000E+00,sum=  0.0000000000000000E+00
 *** Finish kernel
\end{LstLog}

Check the lines below \src{``Validation : grid-by-grid diff''} line,
that shows difference between calculated output array and
pre-calculated reference array.
These should be zero or enough small to be acceptable.
%
There are sample output log files in \file{reference/}
in each kernel program directory, for reference purpose.

\subsection{Sample of performance result}

Here's an example of the performance result part of the log output.
Below is an example executed with the machine environment described in \autoref{s:measuring_env}.
%
Note that in this program kernel part is iterated 1000 times.

\begin{LstLog}
 *** Computational Time Report
 *** ID=001 : MAIN_comp_caldyn_horiz           T=     0.876 N=   1000
\end{LstLog}


\chapter{Selected Configuration Interaction}
\label{chap:sCI}
A direct path towards improving the accuracy of a QMC calculation is
through a better trial wavefunction.  While using a multireference
wavefunction can be straightforward in theory, in actual practice
methods such as CASSCF are not always intuitive and often require
being an expert in either the method or the code generating the
wavefunction.  An alternative is to use a Selected Configuration of
Interaction method (selected CI) such as CIPSI (Configuration
Interaction using a Perturbative Selection done Iteratively). This
provides a direct route to systematically improving the wavefunction.

\section{Theoretical Background}

The principle behind selected CI is rather simple and was first published in 1955 by R.K. Nesbet\cite{Nesbet1955}.
The first calculations on atoms were performed by Diner, Malrieu and Claverie\cite{Diner1967} in 1967, and it become computationally viable for larger molecules in 2013 by Caffarel \textit{et al.}\cite{Caffarel2013}
%  \textbf{To Paul: I do not recall who added this section (maybe me?)
% but it is word for word the paper by caffarel in ref
% \cite{Caffarel2013}. It needs either to be removed or strongly
% rewritten. see bellow for attempt to simplify in my own words; }\\

As described by Caffarel et al in Ref.~\cite{Caffarel2013},
multi-determinantal expansions of the ground-state wavefunction
$\Psi_T$ are written as a linear combination of Slater determinants
\begin{equation}
\sum_k c_k \sum_q d_{k,q}D_{k,q\uparrow } (r^{\uparrow})D_{k,q\downarrow}(r^{\downarrow}) %$\ket{D_i}$
\end{equation}
  where each determinant corresponds to a given occupation by the $N_{\alpha}$ and $N_{\beta}$ electrons of $N=N_{\alpha}+N_{\beta}$ orbitals among a set of M spin-orbitals $\{\phi_1,.,\phi_M\}$ (restricted case). When no symmetries are
considered, the maximum number of such determinants is
\begin{eqnarray}
\label{eqn:Det-Permutations}
\left(
\begin{array}{c} M \hspace{1.5mm} \\ N_{\alpha}  \end{array}
\right).
\left(
\begin{array}{c} M \hspace{1.5mm} \\ N_{\beta}  \end{array}
\right).
\end{eqnarray}
a number that grows factorially with M and N. The best representation of the exact wavefunction in the determinantal basis is the full configuration interaction (FCI) wave function written as
\begin{equation}
\ket{\Psi_0}=\sum_{i} c_{i} \ket{D_i}
\end{equation}
where $c_i$ are the ground-state coefficients obtained by
diagonalizing the matrix, $H_{ij}=\bra{D_i}H\ket{D_j}$, within the
full orthonormalized set $\bra{D_i}\ket{D_j}=\delta_{ij}$, of
determinants $\ket{D_i}$. CIPSI provides a convenient method to build up to this full wavefunction with a single criteria.

A CIPSI wavefunction is built iteratively starting from a reference
wavefunction, usually Hartree-Fock or CASSCF, by adding all single and
double excitations and then iteratively selecting relevant
determinants according to some criteria. Detailed iterative steps can
be found in the reference by Caffarel \textit{et al.} and references
within\cite{Caffarel2013, Scemama2016,Scemama2018,Garniron2017-2} but
are summarized below:

\begin{itemize}
\item Step 1: Define a reference wavefunction:

    \begin{equation}
     \begin{gathered}
       \begin{aligned}
         \ket{\Psi}&=\sum_{i\in D} c_i\ket{i} \,         \,
         &E_{var}&= \frac{\bra{\Psi}\hat{H}\ket{\Psi}}{\bra{\Psi}\ket{\Psi}} \\[12pt]
       \end{aligned} \\[12pt]
     \end{gathered}
   \end{equation}

 \item Step 2: Generate external determinants $\ket{\alpha}$:\\
New determinants are added by generating all single and double excitations from determinants $i \in D$ such as:\\
\begin{equation}
\bra{\Psi_0^{(n)}}H\ket{D_{i_c}}\neq 0
\end{equation}

\item Step 3: Evaluate second order perturbative contribution to each determinant $\ket{\alpha}$:
\begin{equation}
\Delta E=\frac{\bra{\Psi}\hat{H}\ket{\alpha}\bra{\alpha}\hat{H}\ket{\Psi}}{E_{var}-\bra{\alpha}\hat{H}\ket{\alpha}}
\end{equation}

\item Step 4: Select the determinants with the largest contributions and add them to the Hamiltonian.
\item Step 5: Diagonalize the Hamiltonian within the new added determinants and update the wavefunction and the the value of $E_{var}$.
\item Step 6: Iterate until reaching convergence.\\
\end{itemize}
Repeating this process leads to a multi-reference trial wavefunction of high quality that can be used in QMC.

\begin{equation}
\Psi_T(r)=e^{J(r)}\sum_k c_k \sum_q d_{k,q}D_{k,q\uparrow } (r^{\uparrow})D_{k,q\downarrow}(r^{\downarrow})
\end{equation}
The linear coefficients $c_k$ are then optimized with the presence of the Jastrow function.

It is important to note that:
\begin{itemize}
\item When all determinants $\ket{\alpha}$ are selected, the full configuration interaction result is obtained.\\
\item CIPSI can be seen as a deterministic counter part of FCIQMC. \\
\item In practice, any wavefunction method can be made multireference with CIPSI. For instance, a multireference Coupled Cluster (MRCC) with CIPSI is implemented in QP.\cite{Garniron2017-1}\\
\item At any time, with CIPSI selection, $E_{PT_2}=\sum_\alpha \Delta E_\alpha$ estimates the distance to the FCI solution.
\end{itemize}

\subsection{CIPSI wavefunction interface}
\label{sec:cipsi}

\begin{figure}
\begin{center}
\includegraphics[trim = 0mm 0mm 0mm 0mm, clip,width=0.3\columnwidth]{./figures/Reactant.jpg}
\end{center}
\caption{$C_2O_2H_3N$ molecule.}
\protect{\label{fig:C2O2H3N}}
\end{figure}
The CIPSI method
%\cite{XXXrecentCIPSIpaper}
is implemented in the \textit{Quantum Package} (QP) code\cite{QP} developed by the Caffarel group. Once the trial wavefunction is generated, QP is able to produce output readable by the QMCPACK converter as described in section \ref{sec:convert4qmc}.\\
QP can be installed with multiple plugins for different levels of theory in quantum chemistry. When installing the "QMC" plugin, QP can save the wavefunction in a format readable by the QMCPACK converter. \\

In the following we use the $C_2O_2H_3N$ molecule (Fig \ref{fig:C2O2H3N}) as an example of how to run a multireference calculation with CIPSI as a trial wavefunction for \qmcpack. The choice of this molecule is motivated by its multireference nature. While the molecule remains small enough for CCSD(T) calculations with aug-cc-pVTZ basis set, the D1 diagnostic shows a very high value for  $C_2O_2H_3N$, suggesting a multireference character.  Therefore, an accurate reference for the system is not available and it becomes difficult to trust the quality of a single-determinant wavefunction, even when using the DFT-B3LYP exchange and correlation functional. Therefore, in the following, we show an example of how to systematically improve the nodal surface by increasing the number of determinants in the trial wavefunction.\\

The following steps show how to run from Hartree-Fock to selected CI using QP, convert the wavefunction to a QMCPACK trial wavefunction and finally how to analyze the result.

\begin{itemize}
\item Step 1: Generate the QP input file:\\
QP takes for input an XYZ file containing the geometry of the molecule such as:

\begin{center}
\begin{tabular}{ l c c c }
8\\
C2O2H3N\\
C &       1.067070 &  -0.370798 &   0.020324\\
C &      -1.115770 &  -0.239135 &   0.081860\\
O &      -0.537581 &   1.047619 &  -0.091020\\
N &       0.879629 &   0.882518 &   0.046830\\
H &      -1.525096 &  -0.354103 &   1.092299\\
H &      -1.868807 &  -0.416543 &  -0.683862\\
H &       2.035229 &  -0.841662 &   0.053363\\
O &      -0.025736 &  -1.160835 &  -0.084319
\end{tabular}
\end{center}

The input file is generated through the following command line:\\

\begin{shade}
qp_create_ezfio_from_xyz C2O2H3N.xyz -b cc-pvtz
\end{shade}

This means that we will be simulating the molecule in all-electrons within the cc-pVTZ basis set. Other options are of course possible such as using ECPs, different spin multiplicities etc. For more details, to the Quantum Package tutorial \url{https://github.com/LCPQ/quantum_package/wiki/Tutorial}.\\
A directory called \textit{C2O2H3N.ezfio} is created and contains all the relevant data to run the SCF Hartree-Fock calculation. Note that due to the large size of molecular orbitals (220), it is preferable to run QP in parallel. QP parallelization is based on a Master/Slave process allowing a master node to manage the work load between multiple MPI processes through the LibZMQ library. In practice one submits the run to one master node, then submits as many nodes as necessary to speed up the calculations. If a "slave" node dies before the end of its task, the master node will resubmit the workload to another available node. If more nodes are added at any time during the simulation, the master node will use them to reduce the time to solution.\\
\item Step 2: Running Hartree-Fock:\\
To save the integrals on disk and avoid recomputing them later, edit the ezfio directory with the following command:\\
\begin{shade}
qp_edit C2O2H3N.ezfio
\end{shade}

This will generate a temporary file showing all the contents of the simulation and opens an editor to allow modification of their values. Look for \textit{disk\_access\_ao\_integrals} and modify its value from \textit{None} to \textit{Write}\\

To run a simulation with QP, use the binary \textit{qp\_run} with the desired level of theory, in this case Hartree-Fock (SCF). \\
\begin{shade}
mpirun -np 1 qp_run SCF C2O2H3N.ezfio &> C2O2H3N-SCF.out
\end{shade}

If run in serial, the evaluation of the integrals and the Hamiltonian diagonalization would take a substential amount of computer time. It is recommended to add a few more \textit{slave-nodes} to help speed up the calculation.\\

\begin{shade}
mpirun -np 20 qp_run -slave qp_ao_ints C2O2H3N.ezfio &> C2O2H3N-SCF-Slave.out
\end{shade}
The total Hartree-Fock energy of the system in cc-pVTZ is \textit{$E_{HF}=-283.0992$}Ha.\\
\item Step 2: Freeze Core electrons:\\
In order to avoid making excitation from the core electrons, freeze the core electrons and only do the excitations from the valence electrons.\\
\begin{shade}
qp_set_frozen_core.py C2O2H3N.ezfio
\end{shade}
This will will automatically freeze the orbitals from 1 to 5, and leave the remaining active. \\
\item Step 3: Atomic orbitals to Molecular Orbital transformation\\
This step, transforming the atomic orbitals to molecular orbitals, is the most costly, especially given that its implementation in Quantum Package is serial. It is recommended to do it in a separate run and on one node.\\
\begin{shade}
qp_run four_idx_transform C2O2H3N.ezfio
\end{shade}

The MO integrals are now saved on disk and unless the orbitals are changed, they will not be recomputed.\\
\item Step 4: CIPSI \\
At this point the wavefunction is ready for the selected CI. By default QP has 2 convergence criteria. The number of determinants (set by default to 1M) or the value of PT2 (set by default to $1.10^{-4}$Ha). For this molecule, the total number of determinants in the FCI space is $2.07e+88$ determinants. While his number is completely out of range of what is possible to compute, we will set the limit of determinants in QP to 5M determinants and see if the nodal surface of the wavefunction is converged enough for the DMC. It is important to remember at this point that the main value of CIPSI compared to other selected CI method, is that the value of PT2 is evaluated directly at each step giving a good estimated of the error to the FCI energy. This allows us to conclude that when the E+PT2 energy is converged, the nodal surface is probably also converged.\\
Similar to the SCF runs, FCI runs have to be submitted in parallel with a \textit{Master/Slave} process:\\

\begin{shade}
mpirun -np 1 qp_run fci_zmq C2O2H3N.ezfio &> C2O2H3N-FCI.out
mpirun -np 199 qp_run -slave selection_davidson_slave C2O2H3N.ezfio\\
&> C2O2H3N-FCI-Slave.out
\end{shade}

\item Step 5 (optional): Natural orbitals \\
While this step is optional, it is important to note that using natural orbitals instead of Hartree Fock orbitals will always improve the quality of the wavefunction and improve the quality of the nodal surface by reducing the number of needed determinants for the same accuracy. When a full convergence to the FCI limit is attainable, this step will not lead to any change in the energy but will only reduce the total number of determinants. However, if a full convergence is not possible, this step can increase significantly the accuracy of the calculation at the same number of determinants. \\

\begin{shade}
qp_run save_natorb C2O2H3N.ezfio
\end{shade}
\hfill

At this point, the orbitals are modified, a new AO$\rightarrow$MO transformation is required and Steps 3 and 4 need to be run again.\\

\item Step 6: Analysis of the CIPSI results\\
\begin{figure}
\begin{center}
\includegraphics[trim = 2mm 2mm 2mm 2mm, clip,width=0.95\columnwidth]{./figures/CIPSI.jpg}
\end{center}
\caption{Evolution of the variational energy and the Energy + PT2 as a function of the number of determinants for the $C_2O_2H_3N$ molecule.}
\protect{\label{fig:CIPSI}}
\end{figure}
Figure \ref{fig:CIPSI} shows the evolution of the variational energy and the energy corrected with PT2 as a function of the number of determinants up to 4M determinants. While it is clear that the raw variational energy is far from being converged, the Energy + PT2 appears converged around 0.4M determinants.\\

\begin{table}[t]
\centering
\caption{Energies of $C_2O_2H_3N$ using orbitals from Hartree-Fock, natural orbitals, 0.4M and 4M determinants}
\label{TAB:CIPSI}
\begin{tabular}{l|c|c}
\hline \hline
Method & N\_det & Energy\\
\hline
Hartree-Fock &    1    & -281.6729\\
Natural Orbitals & 1 & -281.6735\\
E\_Variational &  438,753 & -282.2951 \\
E\_Variational &  4,068,271   & -282.4882 \\
E+PT2 & 438,753& -282.6809 \\
E+PT2 & 4,068,271 & -282.6805  \\ \hline \hline
\end{tabular}
\end{table}

\item Step 7: Truncation of the number of determinants.\\ While using
  all the 4M determinants from CIPSI always guarantees that all
  important determinants are kept in the wavefunction, practically,
  such a large number of determinants would make any QMC calculation
  prohibitively expensive, as the cost of evaluating a determinant in
  DMC grows as $\sqrt[]{N_{det}}$ where $N_{det}$ is the number of
  determinants in the trial wavefunction. To truncate the number of
  determinants, we follow the method described by Scemama
  \textit{et. al}~\cite{Scemama2018} where the wavefunction is
  truncated by removing independently spin-up and spin-down
  determinants whose contribution to the norm of the wavefunction is
  below a user-defined threshold, $\epsilon$. For this step, we choose
  to truncate the determinants whose coefficients are below,
  $1.10^{-3}$, $1.10^{-4}$, $1.10^{-5}$ and $1.10^{-6}$, translating
  to 239, 44539, 541380 and 908128 determinants, respectively.

To  truncate the determinants in QP, edit the ezfio file as follows:

\begin{shade}
qp_edit C2O2H3N.ezfio
\end{shade}

then look for \textit{ci\_threshold} and modify the value according to the desired threshold. Use the following run to truncate the determinants:

\begin{shade}
qp_run truncate_wf_spin C2O2H3N.ezfio
\end{shade}

\item Step 7: Save the wavefunction for \qmcpack \\
The wavefunction in QP is now ready to be converted to \qmcpack format.
Save the wavefunction into \qmcpack format and then convert the wavefunction using the \textit{convert4qmc} tool\\

\begin{shade}
qp_run save_for_qmcpack C2O2H3N.ezfio &> C2O2H3N.dump
convert4qmc -QP C2O2H3N.dump -addCusp -production
\end{shade}

Since we are running all-electron calculations, orbitals in QMC need
to be corrected for the electron-nuclearcusp condition..  This is done
by adding the option \textit{-addCusp} to \textit{convert4qmc}, which
adds a tag forcing \qmcpack to run the correction or read them from a
file if pre-computed. When running multiple DMC runs with different
truncation thresholds, only the number of determinants is varied and
the orbitals remain unchanged from one calculation to another and the
cusp correction needs only be run once.

\item Step 7: Running \qmcpack \\
At this point, running a multideterminant DMC becomes identical to running a regular DMC with \qmcpack;
After correcting the orbitals for the cusp, optimize the Jastrow functions and then run the DMC.
It is however important to mention a few remarks;\\

(1) \qmcpack allows reoptimization of the coefficients of the
determinants during the Jastrow optimization step. While this has
proven to lower the energy significantly when the number of
determinants is below 10k, a large number of determinants from CIPSI
is often too large to optimize conveniently. Keeping the coefficients
of the determinants from CIPSI unoptimized is an alternative strategy.\\

(2) The large determinant expansion and the Jastrows are both trying
to recover the missing correlations from the system. When optimizing
the Jastrows, we recommend to first optimize J1 and J2 without the J3,
and then with the added J3. Trying to initially optimze J1, J2 and J3
at the same time may lead to numerical instabilities.\\

(3) The parameters of the Jastrow function will need to be optimized
for each truncation scheme and usually cannot be reused effeciently
from one truncation scheme to another.

\item Step 8: Analyzing DMC results from \qmcpack \\

From Table \ref{TAB:CIPSI-DMC}, we can see that increasing the number
of determinants from 0.5M to almost 1M determinant keeps the energy
within error bars and does not improve the quality of the nodal
surface. We can conclude that the DMC energy is converged at 0.54M
determinants. It is important to note that this number of determinants
also corresponds to the convergence of E+PT2 in CIPSI calculations,
confirming for this case that the convergence of the nodal surface can
follow the convergence of E+PT2 instead of the more difficult
variational energy.

\begin{table}[t]
\centering
\caption{DMC Energies and CIPSI(E+PT2) of $C_2O_2H_3N$ in function of the number of determinants in the trial wavefunction.}
\label{TAB:CIPSI-DMC}
\begin{tabular}{l|c|c}
\hline
N\_det & DMC& CISPI\\
\hline
1 & -283.0696 (6)&-283.0063\\
239 & -283.0730 (9)&-282.9063\\
44,539 & -283.078 (1)&-282.7339\\
541,380 & -283.088 (1)&-282.6772\\
908,128& -283.089  (1)&-282.6775\end{tabular}
\end{table}

\end{itemize}

As mentioned in previous sections, DMC is variational relative to the
exact nodal surface. A nodal surface is "better" if it lowers the DMC
energy. To assess the quality of the nodal surface from CIPSI, we
compare these DMC results to other single determinant calculations
from multiple nodal surfaces and theories. Figure \ref{fig:CIPSI-DMC}
shows the energy of the $C_2O_2H_3N$ molecule as a function of
different single-determinant (SD) trial wavefunctions with an
aug-cc-pVTZ basis set, including Hartree-Fock (HF), DFT-PBE and hybrid
functionals B3LYP and PBE0. The last 4 points in the plot show the
systematic improvement of the nodal surface as a function of the
number of determinants.

When the DMC-CIPSI energy is converged with respect to the number of
determinants, its nodal surface is still lower than the best SD-DMC
(B3LYP) by 6(1)mHa. When compared to CCSD(T) with the same basis set,
$E_{CCSD(T)}$ is 4mHa higher than DMC-CIPSI and 2mHa lower than
DMC-B3LYP. While 6 (1) mHa can seem very small, it is however
important to remember that CCSD(T) cannot describe correctly
multireference systems and therefore it is impossible to assess the
correctness of the SD-DMC result, making CIPSI-DMC calculations an
ideal benchmark tool for multireference systems.

\begin{figure}
\begin{center}
\includegraphics[trim = 2mm 2mm 2mm 2mm, clip,width=0.9
\columnwidth]{./figures/DMC-Multidet.jpg}
\end{center}
\caption{DMC energy of $C_2O_2H_3N$ molecule as a function of different single determinant trial wavefunctions with aug-ccp-VTZ basis set using nodal surfaces from Hartree-Fock (HF), DFT-PBE and DFT with hybrid functionals PBE0 and P3LYP. The CIPSI trial wavefunction contains as indicated 239, 44539, 514380 and 908128 determinants (D). }
\protect{\label{fig:CIPSI-DMC}}

\end{figure}


% this file is called up by thesis.tex
% content in this file will be fed into the main document

% --------- begin appendices --------- %

\begin{appendices}

%: ----------------------- paths to graphics ------------------------

\graphicspath{{9_appendix/images/}}

%: ----------------------- contents from here ------------------------

% --------- ngram analysis --------- %

\chapter{n-gram frequency analysis}
\label{appendix:ngramanalysis}

During our analysis of the Public BI benchmark we noticed several \verb|VARCHAR| columns containing strings that are concatenations of data from different distributions. These columns can be stored more efficiently in separate subcolumns, in order to allow independent compression of their constituent parts. We approached this problem with the \nameref{subsec:pd:charsetsplit} pattern detector (\ref{subsec:pd:charsetsplit}) and obtained good results, but we did not cover all the variations of such columns. Table~\ref{tab:appendix:ngramanalysis:table1} shows two samples of data from columns that have this property but we did not cover.

\input{9_appendix/appendix-table_1.tex}

Both columns are composed of two parts: one with high repetition factor---which can benefit from compression---and one rather unique---which cannot. They are separated by a delimiter character: \verb|'@'|, respectively \verb|'-'|. While the \nameref{subsec:pd:charsetsplit} could cover these columns with a delimiter character set (e.g. \verb|[@-]|), there are also cases with no explicit separator character but other structures instead (e.g. fixed number of characters from the start of each string).

To cover all cases in a more generic way, we tried to develop a different approach based on n-gram frequencies. We represented each string in a different way, such that we can identify its constituent substrings more easily, as follows: 1) we computed the frequencies of all 3-grams based on their number of occurrences on the entire column; 2) we replaced each character in each string with the frequency of the 3-gram starting at the position of the character. The results are presented in Figures \ref{fig:appendix:ngramanalysis:ngram_1} and \ref{fig:appendix:ngramanalysis:ngram_2}.

\begin{figure}[h]
  \centering
  \makebox[\textwidth][c]{
    \includesvg[width=0.95\linewidth]{ngram-col_28.svg}
  }
  \caption{\textit{ds\_email} 3-gram frequencies}
  \label{fig:appendix:ngramanalysis:ngram_1}
\end{figure}

\begin{figure}[h]
  \centering
  \makebox[\textwidth][c]{
    \includesvg[width=0.95\linewidth]{ngram-col_30.svg}
  }
  \caption{\textit{ds\_tipo\_beneficiario} 3-gram frequencies}
  \label{fig:appendix:ngramanalysis:ngram_2}
\end{figure}

The figures show the frequency of the 3-grams in the strings in the two columns presented above (larger samples than in the example). The Y axis represents each string on the column and the X axis contains the positions of the characters in the strings. The colors represent the frequencies: white = high frequency, red = low frequency. Black is used as padding for shorter strings. Figure~\ref{fig:appendix:ngramanalysis:ngram_1} shows the results for the \textit{ds\_email} column. We notice how all the emails start with low frequencies (the local/username part) and end with high frequencies (the domain part). Moreover, we can also observe the different components of the domain: we have the top level domains with the highest frequencies (e.g. \verb|.com|, \verb|.br|, etc.), followed by the subdomains (e.g. \verb|gmail|, \verb|hotmail|, etc.). Even more, the word \verb|"mail"| is also representative in itself, since it is common for most email subdomains. Figure~\ref{fig:appendix:ngramanalysis:ngram_2} shows the results for the \textit{ds\_tipo\_beneficiario} column. It is clear how the majority of the strings start with the frequent substring \verb|"Outro: especificar"| and end in a low frequency substring.

The 3-gram frequency representation emphasizes the constituent subparts in which a column should be split in order to enable independent compression. The goal is to automatically find the split points, which is not a trivial task. The approach that we propose relies on the fact that we can view each string as a series of numbers (3-gram frequencies). The series contain edges and plateaus. An edge is a significant increase or decrease in frequency. After identifying the edges in each string, we need to find the common edges amongst the majority of the strings. A common edge is an edge that can be identified either by the same character (e.g. the \verb|'@'| delimiter) or by the same absolute position in all the strings. Once we found the common edges, we define the split points based on the character or position. During compression, we split each string according to these identifiers and mark as exception the values that cannot be split.

Due to time constraints, we decided not to proceed with the implementation and evaluation of this approach and leave it for future work.

\chapter{Column correlation graphs}
\label{appendix:corrgraphs}

The figures below show two examples of \nameref{subsec:pd:columncorrelation} graphs as defined in \ref{subsubsec:ps:correlation}. Figure~\ref{fig:appendix:corrgraphs:yale} shows a common correlation graph resulted from Configuration B (\ref{sec:eval:expsetup}) after applying the \nameref{subsec:pd:columncorrelation} pattern detector (\ref{subsec:pd:columncorrelation}) on the leaf columns of the tree produced by the \nameref{sub:learning:recursiveexhaustive} algorithm (\ref{sub:learning:recursiveexhaustive}) in \textit{Stage-1} (table: \textit{YaleLanguages\_1}).
Figure~\ref{fig:appendix:corrgraphs:generico} shows a highly complex correlation graph, resulted from Configuration A (\ref{sec:eval:expsetup}) (table: \textit{Generico\_1}). Both configurations resulted in graphs of similar complexities for the same tables. An important note is that these correlations are not directly between the logical columns, but between subcolumns resulted after decomposing them and separating the values into different subsets and exceptions as well. As we see in the figures, this process creates many correlation opportunities---and most of them could not be exploited on the original representation of the data.

Columns with high indegree can be determined by many other columns. This happens to columns that are (almost) constant and---in our case in particular---to exception columns with very high null ratios. The explanation is that, if a column has the same constant value (e.g. \verb|null|), any other column can determine it since all its values will map to the constant value. Columns with high outdegree can determine many other columns. This happens to columns with very low repetition factor (i.e. rather unique), since there is a perfect mapping between a unique column and any other column. However, this does not happen in our case, since we only use \nameref{subsec:pd:columncorrelation} on \nameref{subsec:pd:dict} compressible columns. Besides these special cases of nearly constant or unique columns, high degrees result from actual correlations between (sub)columns created by the learning algorithms.

% The correlation coefficients are computed as described in \ref{todo}. Recall that column correlation only applies on Dict represented columns and therefore the graph only contains these columns.

\begin{sidewaysfigure}[h]
  \centering
  \makebox[\textwidth][c]{
    \includegraphics[width=0.88\linewidth]{YaleLanguages_1-s_1_l_0-graph.pdf}
  }
  \caption{\textit{YaleLanguages\_1} correlation graph}
  \label{fig:appendix:corrgraphs:yale}
\end{sidewaysfigure}

\begin{sidewaysfigure}[h]
  \centering
  \makebox[\textwidth][c]{
    \includegraphics[width=1.08\linewidth]{Generico_1-corr-s_1_l_0-graph.pdf}
  }
  \caption{\textit{Generico\_1} correlation graph}
  \label{fig:appendix:corrgraphs:generico}
\end{sidewaysfigure}

% --------- notes --------- %

\iffalse
- interesting results:
  - correlation graphs \& heatmaps
  - pattern distribution
  - n-gram analysis
  - other plots that you already have
- concrete examples of whitebox representation of interesting columns (similar to the ones in presentation); with calculation of compression ratio, etc.
\fi

\iffalse
To better understand the impact of \textit{whitebox compression} and how it affects the compression ratio we selected \verb|todo| representative tables and analyzed their compression trees and size components. TODO-1: analyze 2-3 tables

analyze:
YaleLanguages - very good with both vw and wc, but small
Generico - large, decent with vw, significantly good with wb
Redfin1\_1 - not good overall, but good in used -> lack of compression opportunities
RealEstate2\_1 - not good with used either
\fi

% --------- end appendices --------- %

\end{appendices}

% ---------------------------------------------------------------------------
%: ----------------------- end of thesis sub-document ------------------------
% ---------------------------------------------------------------------------


\documentclass[pdf]{beamer}
\mode<presentation>{}

\usetheme{Frankfurt}
\usepackage{listings}
\usepackage{color}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{bm}

\usefonttheme[onlymath]{serif}

\newcommand\norm[1]{\left\lVert#1\right\rVert}

%=== Adding page numbers
\addtobeamertemplate{navigation symbols}{}{%
    \usebeamerfont{footline}%
    \usebeamercolor[fg]{footline}%
    \hspace{1em}%
    \insertframenumber/\inserttotalframenumber
}
\setbeamercolor{footline}{fg=blue}
\setbeamerfont{footline}{series=\bfseries}
%===

%% preamble
\title[Intro to Deep Learning]{A Very Brief Introduction to Neural Networks and Deep Learning}

\author{Ashley Lee, Isabel Restrepo, \& Paul Stey}
\date{\today}

\begin{document}

%% title frame
\begin{frame}
\titlepage
\end{frame}

\begin{frame}<beamer>{Table of Contents}
	\tableofcontents[currentsection,
				 currentsubsection,
				 sectionstyle=show,
				 subsectionstyle=show]
\end{frame}

\section{Background}

\subsection{Neural Network Basics}
	\begin{frame}{What is a neural network?}
		\begin{enumerate}
			\item A species of directed acyclic graphs (usually)
			\item ``Universal function approximator''
			\item ``An engineering solution to a statistics problem''
		\end{enumerate}
	\end{frame}

	\begin{frame}{What do neural networks do?}
	Like many other statistical or machine learning models (e.g., GLM, random forests, boosting), neural networks:

	\vspace{1em}

		\begin{enumerate}
			\item Attempt to approximate a data-generating mechanism
			\item Can be used for classification problems
			\item Can be used for regression problems
			\item Can also be used for dimension reduction like principal components analysis (PCA)

		\end{enumerate}
	\end{frame}

\subsection{History}
	\begin{frame}{History of Neural Networks}
	The history of neural networks is long and tumultuous.
	\vspace{2em}
		\begin{enumerate}
			\item McCulloch and Pitts (1943) ``\textit{A Logical Calculus of Ideas Immanent in Nervous Activity}''
			\item Rosenblatt (1958) ``\textit{The Perceptron: A Probabilistic Model For Information Storage And Organization In The Brain}''
		\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.08]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/pitts.jpg}
		\hspace{3em}
		\includegraphics[scale=0.25]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/mcculloch.jpeg}
	\end{center}
	\end{frame}

	\begin{frame}{More Recently}
	Neural networks are experiencing a major resurgence. There are at least two reasons.

		\begin{enumerate}
			\item Better algorithms for back-propagation
			\item GPUs are well suited to building neural networks
				\begin{itemize}
					\item Matrix multiplies can be made embarrassingly parallel
					\item GPUs have much better memory bandwidth
				\end{itemize}
			\item More labeled data
		\end{enumerate}
	\end{frame}

\subsection{Mechanics of Neural Networks}
	\begin{frame}{The Multilayer Perceptron}
	\begin{center}
		\includegraphics[scale=0.15]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/neural_network2.png}
	\end{center}
	\end{frame}

	\begin{frame}{Single Neuron}
	\begin{columns}
		\begin{column}{0.5\textwidth}
		A single neuron takes inputs, $x_j$, and applies the weights, $w_{\cdot j}$ to the input by computing the dot product of the vectors $x$ and $w$. The result is the input to the ``activation'' function.

		\end{column}

		\begin{column}{0.5\textwidth}
		\includegraphics[scale=0.15]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/neuron2.png}
		\end{column}
	\end{columns}
	\end{frame}

	\begin{frame}{Activation Functions}

	The notion of an activation function comes again from the theoretical relationship to neurons in the brain.
	\vspace{2em}

	Activation functions are analogous to ``link'' functions in generalized linear models (GLMs).

	\vspace{2em}
	In fact, one common activation function is the sigmoid function, which is just our old friend the logistic function which you are using when you fit logistic regression models.
	\end{frame}

	\begin{frame}{Purpose of Activation Functions}

	There are a few reasons we use activation functions.
	\vspace{2em}

	The most basic reason is that---like link functions in GLMs---we want to take some linear predictor and transform it so that it is bounded appropriate. For instance, the value of logistic function is in the range $(0, 1)$.

	\vspace{2em}

	And the second key reason is that this allows us to introduce non-linearities. Recall that a neural network (like many statistical or machine learning models) is trying to approximate a data-generating mechanism. So we are trying to approximate a function that might be very complex and include many non-linearities.
	\end{frame}

	\begin{frame}{Common Activation Functions}
	\begin{columns}
	\begin{column}{0.5\textwidth}
	Some common activation functions include the following:
	\vspace{1em}
	\begin{enumerate}
		\item Sigmoid (i.e., logistic)
		\item Hyperbolic tangent: $tanh$
		\item Rectified linear unit (ReLU)
		\item softplus
	\end{enumerate}
	\end{column}

	\begin{column}{0.5\textwidth}
	\begin{center}
		\includegraphics[scale=0.35]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/activation_functions.png}
	\end{center}
	\end{column}
	\end{columns}
	\end{frame}

	\begin{frame}{Sigmoid Functions}
	\begin{columns}
	\begin{column}{0.5\textwidth}
	Sigmoid function: $\phi(z) = \frac{1}{1 + e^{-z}}$
	\vspace{1em}
	\begin{enumerate}
		\item Range between 0 and 1
		\item Sometimes interpreted as probability
		\item Special case of the softmax: $\psi(z)_j = \frac{e^{z_j}}{\sum_{k = 1}^K e^{z_k}}$, which is used for vector-value outcome
	\end{enumerate}
	\end{column}

	\begin{column}{0.5\textwidth}
	\begin{center}
		\includegraphics[scale=0.35]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/sigmoid_activation.png}
	\end{center}
	\end{column}
	\end{columns}
	\end{frame}

	\begin{frame}{Rectified Linear Unit (ReLU)}
	\begin{columns}
	\begin{column}{0.5\textwidth}
	ReLU: $f(z) = max(0, z)$
	\vspace{1em}
	\begin{enumerate}
		\item Range between 0 and $\inf$
		\item Biological justification
	\end{enumerate}
	\end{column}

	\begin{column}{0.5\textwidth}
	\begin{center}
		\includegraphics[scale=0.30]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/relu_activation.png}
	\end{center}
	\end{column}
	\end{columns}
	\end{frame}

	\begin{frame}{Activation Functions}
	For the hidden layers of neural networks, ReLUs---or some variation thereof---tend to be the most common.

	\vspace{2em}

	Sigmoid (and probably $tanh$) are not frequently used in the hidden layers more because of the vanishing gradient problem (discussed later).
	\end{frame}

\section{CNNs}
	\subsection{Convolutional Neural Network (CNNs)}
	\begin{frame}{Convolutional Neural Networks}
	\begin{columns}
	\begin{column}{0.5\textwidth}
	\vspace{3em}

	Why do we need ConvNets?
	\end{column}

	\begin{column}{0.5\textwidth}
	\begin{center}
		\includegraphics[scale=0.3]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/large_neural_net.png}

	\end{center}

	\end{column}
	\end{columns}

	\begin{enumerate}
		\item Regular neural nets don't scale well to images
			\begin{itemize}
				\item For images of size $32 \times 32 \times 3$, a \textit{single} fully-connected neuron in the first layer would have $3072$ weights.
				\item Images of size $200 \times 200 \times 3$, a \textit{single} gives $120000$ weights.

			\end{itemize}
		\item Full connectivity is wasteful and the huge number of parameters would quickly lead to overfitting.

	\end{enumerate}
	\end{frame}

	\begin{frame}{CNNs}
	What are ConvNets?
	\vspace{1em}

	\begin{enumerate}
		\item ConvNets are very similar to neural networks discussed thus far. Dot product, followed by non-linearity, and loss function at the end.
		\item Explicit assumption that input are images.
		\item Layers have neurons arranged in 3 dimensions (width, height, depth) to form an \textbf{activation volume}
	\end{enumerate}

	\begin{center}
	\includegraphics[scale=0.15]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/cnn.png}

	\end{center}
	\end{frame}

	\begin{frame}{CNN Architecture}
	Types of layers used to build ConvNets

	\vspace{1em}

	\begin{enumerate}
		\item Convolutional Layer
			\begin{itemize}
				\item Input: 3-d volume
				\item Output: 3-d volume
				\item Convolve ``filters'' with small regions in the image
				\item Output depth, depends on the number of filters
			\end{itemize}
		\item Pooling Layer
			\begin{itemize}
				\item Downsampling along spatial dimensions (width, height)
			\end{itemize}
		\item Fully-Connected Layer (what we've seen so far)
			\begin{itemize}
				\item Compute class score. Dimensions are transformed to $1 \times 1 \times k$, where $k$ is number of classes
			\end{itemize}

	\end{enumerate}
	\end{frame}

	\begin{frame}{CNN Architecture}
	\begin{center}
		\includegraphics[scale=0.21]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/convnet.jpeg}
	\end{center}

	\vspace{1em}

	\begin{enumerate}
		\item CONV/FC layers perform transformations that are a function of not only the activations in the input volume, but also of the parameters (weights and biases of neurons).
		\item RELU/POOL layers will implement a fixed function (e.g., ReLU $= max(0, x)$)
	\end{enumerate}
	\end{frame}

	\begin{frame}{Classical Computer Vision}
	\begin{center}
		\includegraphics[scale=0.16]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/classic_compvision.png}
	\end{center}
	\end{frame}

	\begin{frame}{Convolutional Layer - Local Connectivity}
	Connecting each Neuron to the Input Volume
	\vspace{1em}
	\begin{enumerate}
		\item \textbf{Receptive Field (hyper-parameter):} Spatial extend of each neuron = filter size
		\item Connections are local in space (along width and height), but always full along the entire depth of the input volume.
			\vspace{1em}
			\begin{itemize}
				\item Example: Suppose an input volume had size $16 \times 16 \times 20$. Then using an example receptive field size $3 \times 3 \times 20 = 180$ connections to the input volume. Notice that, again, the connectivity is local in space (i.e., $3 \times 3$), but full along the input depth (20)
			\end{itemize}
	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.12]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/local_connectivity.png}
	\end{center}
	\end{frame}

	\begin{frame}{Convolutional Layer - Spatial Arrangement}

	\vspace{1em}
	\begin{columns}
	\column{\dimexpr\paperwidth-10pt}
	Number of Neurons in the Output Volume
	\begin{enumerate}
		\item \textbf{Depth (hyper-parameter):} Number of filters: Number of patterns to look for in input. Think back to steerable filters. The set of neurons looking at the same region may be referred to as \textbf{depth column} or \textbf{fiber}.
		\vspace{1em}
		\item \textbf{Stride (hyper-parameter):} Number of pixels used when sliding the filter. This controls the overlap between neurons. If stride is $1$, we move one pixel at a time (common, but depends on the width of filter).
		\vspace{1em}
		\item \textbf{Padding (hyper-parameter):} Number of zeros to add to the border of the volume. Convenient for convolution.

	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.078]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/local_connectivity.png}
	\end{center}
	\end{columns}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv1.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv2.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv3.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv4.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv5.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv6.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv7.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv8.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv9.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv10.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv11.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv12.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv13.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv14.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv15.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv16.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv17.png}
	\end{center}
	\end{frame}

	\begin{frame}
	\begin{center}
		\includegraphics[scale=0.32]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/conv18.png}
	\end{center}
	\end{frame}

\subsection{Real-World Constraints}
	\begin{frame}{Mutual Constraints of Hyper-Parameters}
	\begin{enumerate}
		\item If stride is one, $S = 1$, it's common to set padding, $P = \frac{F-1}{2}$, to ensure that input and output volumes have the same size.
		\vspace{1em}
		\item The number of neurons that fit is $\frac{W - F + 2P}{S} + 1$. Must set hyper-parameters so there number of neurons is an integer!
		\vspace{1em}
		\item \textbf{Real-World Example:} (Winner of ImageNet 2012) Images $227 \times 227 \times 3$, $F = 11$, $S = 4$, $P = 0$, $K = 96$.
			\begin{itemize}
				\vspace{1em}
				\item Number of neurons in \textit{first layer}
				\begin{enumerate}[]

					\item $\frac{227 - 11}{4} + 1 = 55$ (per width)
									\vspace{0.5em}
					\item $55 \times 55 \times 96 = 290400$ (width $\times$ height $\times$ depth)
									\vspace{0.5em}
					\item Each neuron has $11 \times 11 \times 3 = 363$ weights plus $1$ bias
									\vspace{0.5em}
					\item $290400 \times 364 = 105705600$ parameters \textit{in the first layer!}
				\end{enumerate}
			\end{itemize}

	\end{enumerate}
	\end{frame}

	\begin{frame}{Parameter Sharing}
	\textbf{If one feature (filter/kernel/depth slice) is useful at pixel $(x_1, y_1)$, then it is also useful at pixel $(x_2, y_2)$}

	\begin{enumerate}
		\item All neurons in the same depth slice have same weights and bias.
		\item Example before $96 \times 11 \times 11 \times 3 = 34848$ unique weights $ + 96 $ biases $ = 34944$ parameters
		\item Then forward pass can be seen as the convolution of that slice neuron's weights with the input volume.
		\item Weights $=$ filter/kernel that is convolved with the input
	\end{enumerate}

	\begin{columns}
	\begin{column}{0.7\textwidth}
	\begin{center}
		\includegraphics[scale=0.25]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/weights.jpeg}
	\end{center}
	\end{column}
	\begin{column}{0.3\textwidth}
		\tiny{Example filters learned by Krizhevsky et al. Each of the 96 filters shown here is of size $11 \times 11 \times 3$}
	\end{column}

	\end{columns}
	\end{frame}

\subsection{Pooling Layer}
	\begin{frame}{Pooling Layer}
	\begin{enumerate}
		\item Inserted in-between convolutional layers
		\item Goal: To reduce spatial size, amount of parameters, computation requirements, and overfitting
		\item Operates independently on every depth slice and resizes spatially using $max(\cdot)$ (or could be $mean(\cdot)$ or $L2$)
		\item Introduces no parameters, since it is a fixed operation
		\item Hyper-parameters: $F$ (spatial size) and $S$ (stride)
	\end{enumerate}
	\begin{columns}
	\begin{column}{0.25\textwidth}
	\begin{center}
		\includegraphics[scale=0.23]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/pool.jpeg}
	\end{center}
	\end{column}
	\begin{column}{0.75\textwidth}
	\begin{center}
		\includegraphics[scale=0.22]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/maxpool.jpeg}
	\end{center}
	\end{column}

	\end{columns}

	\end{frame}

\subsection{Normalization Layer}
	\begin{frame}{Normalization Layer}

	\textbf{Future - Hope of Getting Rid of Pooling}
	\vspace{1em}
		\begin{enumerate}
			\item Striving for Simplicity: Newer works propose discarding pooling. Suggest using larger strides
			\item Variational auto-encoders
			\item Generatie Adversarial Networks
		\end{enumerate}
	\vspace{2em}
	\textbf{Normalization Layer}
		\begin{enumerate}
			\item Have show no real advantages
		\end{enumerate}
	\end{frame}

	\begin{frame}{Practice}
	\textbf{Hyper-Parameters and Layer Sizing}
	\vspace{1em}
	\begin{enumerate}
		\item Input layer should be divisible by 2 many times
		\vspace{0.5em}
		\item Conv layer use small filters $(3 \times 3)$, $(5 \times 5)$, (notice odd) and stride $S = 1$ (reduce sizing headaches - downsampling done by pooling)
		\vspace{0.5em}
		\item Use padding that doesn't alter dimension
		\vspace{0.5em}
		\item Pool layers $(2 \times 2)$, $F = 2$, and $S = 2$. Discards $75\%$ of activations
		\vspace{0.5em}
		\item Compromise based on memory constraints (GPU). May have to sacrifice at the 1st layer.
	\end{enumerate}
	\end{frame}

	\begin{frame}{Popular Networks}
	\begin{enumerate}
		\item LeNet: LeCun 90's - read zip code (first successful ConvNet)
		\vspace{0.5em}
		\item AlexNet: Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Popularize ConvNets for computer vision. ImageNet. Deeper, bigger than LeNet
		\vspace{0.5em}
		\item ZF Net: Matthew Zeiler and Rob Fergus. ILSVRC 2013 Winner. Improved by expanding the size of the middle convolutional layers making the stride and filter size on the first layer smaller
		\vspace{0.5em}
		\item GoogLeNet: Szegedy et al. from Google. ILSVRC 2014 winner. Reduced the number of parameters in the network from 60M to 4M (inception model)
		\vspace{0.5em}
		\item ResNet: Residual Network. Kaiming He et al. ILSVRC 2015 winner. It features special skip connections and heavy use of batch normalization. State of the art as of 2016
	\end{enumerate}
	\end{frame}

\section{RNNs}

\subsection{RNNs and LSTMs}

	\begin{frame}{Recurrent Neural Networks (RNNs)}

		RNNs are a type of neural network that share parameters across time. They are used frequently for natural language processing (NLP) and time series data.

	\end{frame}

	\begin{frame}{RNNs}
	\begin{enumerate}
		\item Unlike standard neural networks or CNNs, RNNs introduce a temporal or sequential component to the system.
		\item While CNNs introduce idea of spatial dependence, RNNs rely on the notion that \textit{the current input relies on previous inputs}
	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.35]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/rnn_data.png}
	\end{center}
	\end{frame}

	\begin{frame}{RNN Architecture and Notation}
	\begin{enumerate}
		\item Current input: $x_t$
		\item Current hidden state: $h_t$
		\item Non-linear activation function $\sigma$ or $tanh$
		\item Predicted output: $\widehat{y_t}$
	\end{enumerate}

	\begin{center}
		\includegraphics[scale=0.2]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/rnn_unrolled2.png}
	\end{center}
	\end{frame}

	\begin{frame}{Long-Term Dependencies}
	\begin{enumerate}

		\item Suppose we want a model that predicts the next word in this sentence:
		\begin{itemize}
			\item ``\textit{The clouds are in the} \texttt{<WORD>}''
		\end{itemize}
		\vspace{2em}

		\item Now suppose we to model the next word in this sentence:
		\begin{itemize}
			\item ``\textit{I grew up in France and lived there most of my life. As a result I am a native speaker of} \texttt{<WORD>}''
		\end{itemize}
	\end{enumerate}

	\begin{center}
		\includegraphics[scale=0.2]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/long_depend.png}
	\end{center}
	\end{frame}

	\begin{frame}{Long Short Term Memory Networks (LSTM)}
		LSTMs are a species of RNN that is designed for learning long-term dependencies. While standard RNNs are able to model temporal dependencies, LSTMs have a much more elaborate structure.

	\begin{columns}
	\begin{column}{0.5\textwidth}
	\begin{center}
		\vspace{-4em}
		\includegraphics[scale=0.19]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/standard_rnn.png}
	\end{center}
	\end{column}

	\begin{column}{0.5\textwidth}
	\begin{center}
		\vspace{7em}
		\includegraphics[scale=0.14]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/lstm.png}
	\end{center}
	\end{column}
	\end{columns}
	\end{frame}

	\begin{frame}{Cell State}
	The key idea of LSTMs is that of cell state to propagate information.
	\vspace{2em}
	\begin{enumerate}
		\item Cell state is subject to linear operations
		\item Information in cell state can be ``forgotten'' using a system of gates
	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.3]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/lstm_cell_state.png}
	\end{center}
	\end{frame}

	\begin{frame}{LSTM Forget Gate Layer}
	Forget gate layer is a sigmoid layer that decides the values to be updated with
	\[
	f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)
	\]
	\begin{enumerate}

		\item This layer looks at $h_{t-1}$ and $x_t$ and updates the cell state
		\item $f_t$ has range $(0, 1)$
		\item $0$ indicates ``complete get rid of this'', and $1$ indicates ``completely keep this''.
	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.14]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/lstm_forget_gate.png}
	\end{center}
	\end{frame}

	\begin{frame}{Input Gate Layer}
	Input gate layer is a combination of a sigmoid layer and $tanh$ layer that decide the new information to keep with
	\[
	i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)
	\]
	\[
	\widetilde{C}_t = \text{tanh}(W_C \cdot [h_{t-1}, x_t] + b_C)
	\]
	\begin{enumerate}

		\item Sigmoid layer is our ``input gate layer''
		\item $tanh$ layer creates candidate values

	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.15]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/lstm_tanh_layer.png}
	\end{center}
	\end{frame}

	\begin{frame}{Update the Cell State}
	Now we update the cell state using
	\[
	C_t = f_t \cdot C_{t-1} + i_t \cdot \widetilde{C}_t
	\]
	\begin{enumerate}

		\item Multiply old state by $f_t$, forgetting what we had decided
		\item And add $i_t \cdot \widetilde{C}_t$, our new candidate values
	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.15]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/lstm_3rd_layer.png}
	\end{center}
	\end{frame}

	\begin{frame}{Output Layer}
	Now we decide what to output with
	\[
	o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)
	\]
	\[
	h_t = o_t \cdot \text{tanh}(C_t)
	\]
	\begin{enumerate}

		\item Take updated cell state to help compute output
		\item Apply sigmoid layer and $tanh$ layer
	\end{enumerate}
	\begin{center}
		\includegraphics[scale=0.15]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/lstm_4th_layer.png}
	\end{center}
	\end{frame}

\section{Autoencoders}

\subsection{Autoencoder Basics}
		\begin{frame}{Autoencoder Basics}
		\begin{enumerate}

			\item Feed-forward neural network
			\item Used for unsupervised learning
			\item Trained to reproduce input layer in the output layer
			\item Training uses gradient descent
		\end{enumerate}
		\end{frame}

		\begin{frame}{Autoencoder Basics cont.}
		\begin{enumerate}
			\item Similar to PCA*, but more flexible
			\item Autoencoders can accommodate non-linear transformations
			\begin{enumerate}[1]
				\item Consequence of flexible activation functions
				\item Turns out to be hugely useful
			\end{enumerate}
		\end{enumerate}

		\vspace{8em}
		{\tiny * Actually, with squared error loss and no sigmoid transformation, it is equivalent to PCA (Baldi \& Hornik, 1989)}
		\end{frame}

		\begin{frame}{Simple Autoencoder}
		\begin{figure}
			\includegraphics[scale = 0.20]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/autoencoder.png}
		\end{figure}
		\end{frame}

		\begin{frame}{Simple Autoencoder cont.}

		From input layer to hidden layer is``encoder'':
		\vspace{-1em}
		\begin{center}
		$$ f(x) = \sigma(Wx + b) $$
		\end{center}
		Hidden layer to the output is the ``decoder'':
		\vspace{-1em}
		\begin{center}
		$$ g(x) = \sigma(W'x + b') $$
		\end{center}
		where $\sigma$ is the activation function (often sigmoid) and $W'$ is often $W^\top$ (known as having ``tied'' weights).
		\end{frame}

		\begin{frame}{Simple Autoencoder cont.}
		\begin{enumerate}
			\item Input layer $\rightarrow$ hidden layer (fewer units) $\rightarrow$ output layer
			\item Hidden layers values can be thought of as lossy compression of input layer
			\item Hidden layer with fewer units than input is often called ``bottleneck'' or under-complete layer

		\end{enumerate}
		\end{frame}

		\begin{frame}{Question}
		In a sense, autoencoders want to learn an approximation to the identity function.
			\begin{center}
				Why would this be useful?
			\end{center}
		\end{frame}

		\begin{frame}{Answer}
		This turns out to have several practical uses.
		\vspace{2em}
		\begin{enumerate}
			\item Autoencoders learn a kind of latent representation of the input
			\item Can be used for initializing weights and biases prior to fitting neural net
			\item Very well suited to dimensionality reduction in NLP tasks
		\end{enumerate}
		\end{frame}

\subsection{Stacked and Denoising Autoencoders}
		\begin{frame}{Stacked Autoencoder}
		\begin{enumerate}
			\item Sometimes called deep autoencoder
			\item Feed-forward neural network
			\item Has additional layers
			\item Still ``unsupervised'' in the sense of no labels or real-valued outcome variable
			\item Output layer is still $\hat{x}$
			\item Trained in stages
		\end{enumerate}
		\end{frame}

		\begin{frame}{Stacked Autoencoder}
		\begin{figure}
			\includegraphics[scale = 0.20]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/stacked_autoencoder.png}
		\end{figure}
		\end{frame}

		\begin{frame} {Stacked Autoencoder cont.}
		\begin{enumerate}
			\item Beneficial for data with hierarchical organization
			\item Can be difficult to train using back propagation
		\end{enumerate}
		\end{frame}

			\begin{frame}{Denoising Autoencoder}
			\begin{enumerate}
				\item Stochastic version of autoencoder
				\item Random noise injected into the input layer
				\begin{enumerate}[1]
					\item \textit{Masking noise}: some fraction of elements* in $x$ set to $0$
					\item \textit{Gaussian additive noise}: $\tilde{x}|x \sim \mathcal{N}(x, \sigma^2I) $
					\item \textit{Salt-and-pepper noise}: fraction of elements* in $x$ set to either min or max possible value (often $0$ or $1$) according to coin toss
				\end{enumerate}
			\end{enumerate}
			\vspace{5em}
			\hspace{2em} {\tiny * chosen at random for each example}
			\end{frame}

			\begin{frame}{Denoising Autoencoder}
			\begin{enumerate}
				\item Output layer is evaluated against original (uncorrupted) data
				\item Goal is slightly different from simple autoencoder
				\begin{enumerate}[1]
					\item Not attempting to learn identity function
					\item We want robustness of representation
					\item Learned representation is insensitive to perturbations in the input
				\end{enumerate}
			\end{enumerate}
			\end{frame}

			\begin{frame}{Denoising Autoencoder}
			\begin{figure}
				\includegraphics[scale = 0.25]{/Users/pstey/projects_code/deep_learning_group/slides/intro_deep_learning/images/denoising_autoencoder2.png}
			\end{figure}
			\end{frame}

			\begin{frame}{Denoising Autoencoder cont.}
			\begin{enumerate}
				\item Training proceeds fairly similarly to standard autoencoder
				\item Crucial difference is the loss function is calculated using \textit{uncorrupted} input layer
			\end{enumerate}
			\end{frame}

		\begin{frame} {Extending Denoising Autoencoder}
		\begin{enumerate}
			\item Some noise types only corrupt subset of the input's components
			\begin{enumerate}[1]
				\item Masking noise
				\item Salt-and-pepper noise
			\end{enumerate}
			\item For these, we can extend denoising by adding emphasis on corrupted components.
			\item Use hyperparameters $\alpha$ and $\beta$ to control emphasis; for squared error loss, this yields

			$$ L(x, \hat{x}) = \alpha \left( \sum_{i \in \mathcal{I}(\tilde{x})} \left(x_i - \hat{x}_i \right) \right)   + \beta \left( \sum_{i \notin \mathcal{I}(\tilde{x})} \left(x_i - \hat{x}_i \right) \right)$$
			where $ \mathcal{I}(\tilde{x}) $ denotes indexes components of $x$ that were corrupted.
		\end{enumerate}
		\end{frame}

\section{Summary}

\subsection{When to use Neural Networks}
	\begin{frame}{When to use neural networks}
		Conditions under which you might consider using neural networks:
		\begin{enumerate}
			\item Have a huge amount of labeled training data
			\item Image classification (with huge amount of labeled images)
			\item Certain NLP tasks
			\item Some signal processing problems
		\end{enumerate}
	\end{frame}

	\begin{frame}{When \textit{not} to use neural networks}
		Probably should \textbf{not} use neural networks when:
		\vspace{1em}
		\begin{enumerate}
			\item You have specific hypotheses you want to test
				\begin{itemize}
					\item E.g., ``\textit{Drug $X$ improves condition $Y$}''.
				\end{itemize}
			\item Interested in estimating the effect of some variable(s) on some outcome variable
			\item You have highly structured data and/or few features
		\end{enumerate}

		\vspace{2em}

		In the case of (1.) and (2.), a traditional statistical model is better. In the case of (3.), using some ensemble-of-trees method will give as-good or better results with minimal tuning.
	\end{frame}

\begin{frame}{Sources}

	\begin{enumerate}[1]
	\item http://cs231n.github.io/convolutional-networks/
	\item https://colah.github.io/posts/2015-08-Understanding-LSTMs/
	\item Dave Berenbaum ``Activation Functions'' (2016) https://github.com/brown-data-science/deep\_learning\_group/blob/master/slides/activation\_functions.pdf
	\item Yinong Wang ``Recurrent Neural Networks'' (2016) https://github.com/brown-data-science/deep\_learning\_group/blob/master/slides/recurrent\_neural\_networks.pdf

	\item Vincent P., Larochelle, H., Bengio Y., Manzagol P.A. (2008) \textit{Extracting and Composing Robust Features with
Denoising Autoencoders}.

	\item Vincent P., Larochelle, H., LaJoie I., Bengio Y., Manzagol P.A. (2010) \textit{Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion}

	\item Makhzani A., Frey B. (2015) \textit{Winner-Take-All Autoencoders}

	\item Ng A. \textit{Sparse Autoencoders} CS294A Lecture notes
	\end{enumerate}

	\end{frame}
\end{document}


\documentclass[12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage[colorlinks,linkcolor=red]{hyperref}
\usepackage{graphicx}

\title{
    {Paper Note}\\
    {\large "GhostNet: More Features from Cheap Operations"} \\
    {\small Paper Link: \href{https://arxiv.org/pdf/1911.11907.pdf}{CVPR 2020}}
}

\author{Lisen Dai}
\date{Nov 8, 2020}

\begin{document}
    \maketitle
    \section{
        Background
    }
        \subsection{
            Problem
        }
            Traditional CNNs usually need a large number of parameters and floating point operations (FLOPs) to achieve a satisfactory accuracy. \\
        \subsection{
            Previous Work
        }
            1. Network pruning. \\
            2. Low-bit quantization. \\
            3. Knowledge distillation. \\
            4. Efficient structure design. \\
            \quad - MobileNet utilized the depthwise and pointwise convolutions to construct a unit for approximating the original convolutional layer with larger filters and achieved comparable performance. \\
            \quad - ShuffleNet further explore a channel shuffle operation to enhence the performance of lightweight models. \\

    \section{
        Contributions
    }
        \subsection{
            Ghost Module
        }
            \begin{center}
                \includegraphics[scale=0.6]{src/img/Ghost module.jpg} \\
            \end{center}
            It is about to change the following equation for CNN layers: \\
            $$
                Y = X * f + b
            $$
            to,
            $$
                Y^{\prime} = X * f^{\prime}
            $$
            where $f^{\prime} \in \mathcal{R}^{c \times k \times k \times m}$ is the utilized filters, and for $Y^{\prime}$ here goes,
            $$
                y_{ij} = \Phi_{i,j}(y_i^{\prime}), \forall i = 1,...,m, j = 1,...,s,
            $$
            where $ y_i^{\prime} $ is the i-th intrinsic feature map in $ Y^{\prime} $, $ \Phi_{i,j} $ in the
            above function is the j-th (except the last one) linear operation for generating the j-th ghost feature map $ y_{ij} $. \\

        \subsection{
            Applications
        }
            \subsubsection{
                Ghost Bottlenecks
            }
                \begin{center}
                    \includegraphics[scale=0.6]{src/img/Ghost bottleneck.jpg} \\
                \end{center}
                In practice, the primary convolution in Ghost module here is pointwise convolution for its efficiency. \\

            \subsubsection{
                GhostNet
            }
                Use the structure of \href{https://arxiv.org/pdf/1905.02244.pdf}{MobileNetV3} but re- place the bottleneck block in MobileNetV3 with our Ghost bottleneck. \\
                Also, there is a kind of GhostNet with width multiplier $ \alpha $ called GhostNet-$\alpha \times$. $\alpha$ here is a factor on the number of channels uniformly at each layer, To customize
                the network for smaller and faster models or higher accuracy on specific tasks. \\
                Width multiplier can control the model size and the computational cost quadratically by roughly $ \alpha^2 $. Usually smaller $\alpha$ leads to lower latency and lower performance, and vice versa. \\

    \section{
        Experiments
    }
        \subsection{
            setup
        }
            dataset: CIFAR-10, ImageNet ILSVRC 2012 dataset, MS COCO object detection benchmark. \\

        \subsection{
            Visualization of Feature Maps
        }
            \begin{center}
                \includegraphics[scale=0.4]{src/img/FMaps.jpg} \\
            \end{center}

        \subsection{
            Comparison
        }
            \subsubsection{
                CIFAR-10
            }
                \begin{center}
                    \includegraphics[scale=0.4]{src/img/Com_cifar10.jpg} \\
                \end{center}
            \subsubsection{
                ImageNet
            }
                \begin{center}
                    \includegraphics[scale=0.4]{src/img/Com_imagenet.jpg} \\
                \end{center}
            \subsubsection{
                Visual Benchmarks
            }
                \begin{center}
                    \includegraphics[scale=0.4]{src/img/acc_and_lat.jpg} \\
                    \includegraphics[scale=0.4]{src/img/Com_imagenet.jpg} \\
                \end{center}
            \subsubsection{
                MS COCO
            }
                \begin{center}
                    \includegraphics[scale=0.6]{src/img/MSCOCO.jpg} \\
                \end{center}

\end{document}


\chapter{Experiments}

For the designed system we haven chosen multiple experiments to
verify its accuracy. Experiments in this part focus on the localization process
and overall experiments. The tracker experiments are located in the chapter
\ref{ch:tracker}.

\section{Calibration and localization}
\label{s:experiment-static}

In order to measure a quality of calibration and localization we exclude a
tracking component. Therefore, we propose an experiment with static points in a
camera views. Then we estimate a position in 3D for these points.

It is complicated to measure distance from the origin of the coordinate system,
i.e. point inside the first camera to any given point. Therefore, we measure
distances between the static points in the real world and compute the distance
between their estimated position in 3D coordinates. Then we compare the results.

To skip the tracking part, we created a new tracker.  The new tracker always
returns the same position.  Doing this, we excluded tracker from our process.

We have chosen the grid as in the Figure \ref{fig:grid} as our pattern for experiments.
The vertical lines are circa 400~mm long and the distance between them is circa
200~mm. We measured distances between the crossings. We numbered the crossing,
the left column is from top to the bottom from 1 to 7 and the right column from
8 to 14 (see the Figure \ref{fig:ladder_numbered}).

\begin{figure}
\centering
\begin{tikzpicture}[]
	\draw (0, 0) -- (12, 0);
	\draw (0, 4) -- (12, 4);
	\draw (0, 0) -- (0, 4);
	\draw (2, 0) -- (2, 4);
	\draw (4, 0) -- (4, 4);
	\draw (6, 0) -- (6, 4);
	\draw (8, 0) -- (8, 4);
	\draw (10, 0) -- (10, 4);
	\draw (12, 0) -- (12, 4);
	\node [right] at (0.1, -0.25) {200mm};
	\node [right] at (2.1, -0.25) {200mm};
	\node [right] at (4.1, -0.25) {200mm};
	\node [right] at (6.1, -0.25) {200mm};
	\node [right] at (8.1, -0.25) {200mm};
	\node [right] at (10.1, -0.25) {200mm};
	\node [right] at (12.1, 2) {400mm};
\end{tikzpicture}
\caption{Pattern used for experiments}
\label{fig:grid}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.8\linewidth]{img/experiments/right-ladder-numbered.png}
\caption{Numbering the points}
\label{fig:ladder_numbered}
\end{figure}

The setup of the cameras was nearly parallel, which means, the center rays of
their views were parallel, looking in same direction. The distance between them
was circa 16~cm. Selected points are displayed in the Figure
\ref{fig:ladder_ground}.

We repeated the experiment ten times, with exactly same setup in order to get
more reliable results. The only difference was the frames chosen for the
calibration (frames for the calibration were randomly chosen from the same
video). The number of the frames was same in each run. Therefore, we obtained
slightly different calibration data on the same setup.

With each set of calibration data we estimated position of the static points.
Then we computed distances between them. From such results from ten runs, we
computed average errors.

The results are listed in the Table \ref{table:distances}. First two columns
define the pair of point between which are the results in corresponding row.
Third column denote the real length. The first column of the rests display the
average absolute error. Next columns display its relative value. The last
column displays the sample standard deviation for absolute error.

To observe the results better, we have used also box plot displayed in the
Figure \ref{fig:horizontal-boxplot}. This box plot represents same results from the
same experiment. The orange line for each pair of points represent median of
the results. The 50\% of the results lies within the box. We denote height of
the this box as interquartile range (IQR). The whiskers represents all other
measurements, which lies at most one and half of the IQR away from the box. All
measurement which do not belong in previously mentioned groups are considered
as outliers and  will be displayed as circles.

\begin{figure}
\includegraphics[width=\linewidth]{experiments/horizontal1.eps}
\caption{Box plot of relative errors of estimated horizontal distances between the columns (parallel setup)}
\label{fig:horizontal-boxplot}
\end{figure}

Based on the results displayed in the Table \ref{table:distances} and the box
plot \ref{fig:horizontal-boxplot} we can see that estimated results are least accurate
for the points furthest from the cameras. Movement by few pixels so far away
from the camera means great real movement.  On the other hand, the
movement by same amount pixels closer to the camera means far smaller
real distance. Because of that, the error is expected to increase for
further points.

On the contrary, we can see that results too close to the camera are actually a
bit less accurate than the results close to the center of the view. The points
close to the camera are at the edge of the view. These less accurate results
may be caused by wrongly estimated distortion coefficients, as distortion
coefficients have major effect at the edges of the image.

Other box plots for the same setup with displayed results for the left and
right column, and for the perpendicular plane are displayed in the Appendix
\ref{ss:16results}.

\begin{table}
\centering
\begin{tabular}{|r|r|r|r|r|r|}
\hline
\multicolumn{2}{|c|}{Points} & \multicolumn{1}{c|}{Real} & \multicolumn{3}{c|}{Error} \\
\cline{1-2} \cline{4-6}
From & To & length [mm] & Average [mm] & Average [\%] & Std. dev. [mm]\\
\hline
\hline
\input{distances.txt}
\hline
\end{tabular}
\caption{Results of the experiment focused on the distances}
\label{table:distances}
\end{table}

We were interested if the results are good not only on the one plane -- in our
case the ground -- but also in the others. Now we have tested the precision in
two axes, which were perpendicular to each other (vertical and horizontal
lines). We have noticed, that the results for the vertical lines are less
precise. Therefore, we decided to test the last axis and its precision. We place
a wall with the marks, which is perpendicular to the ground and measured the
distance between the marks. The setup can be seen in the Figure
\ref{fig:table}. The marks are numbered as displayed in the image
\ref{fig:tablenumbered}. The results are included in the Table
\ref{table:distances}.

We did the same experiment with another setup, when the cameras were 63~cm
apart from each other. We were interested, if the results are better in
parallel setup, or not. Based on the results, the parallel setup performed
better when the lines were parallel to the camera X-axis (i.e. horizontal). On
the other hand, the setup with cameras far from each other and not having
parallel view performed better in measuring non-horizontal lines.

Even though we explored this fact, we were not able to find out the cause. In
general, the mathematical background for parallel setup is more
straight-forward and thus provide more stable results. It also provides almost
full correspondence of the views, therefore it is preferable to use for depth
maps. On the other hand, the non-parallel setup should provide better precision
for closer objects. Similar results (parallel setup performing better in some
cases and non-parallel setup in others) were observed also for example
by \citet*{cardenas19953d}.

The results for this second experiment are listed in the appendix in the
Section \ref{ss:63results}. Same numbering of the marks was used.

\begin{figure}
\centering
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/left-ladder.png}
\end{subfigure}
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/right-ladder.png}
\end{subfigure}
\caption{Selecting the points from the videos}
\label{fig:ladder_ground}
\end{figure}

\begin{figure}
\centering
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/table-left.jpg}
	\caption{Left view of the camera with marks}
	\label{fig:tablenumbered}
\end{subfigure}
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/table-right.jpg}
	\caption{Right view of the camera}
\end{subfigure}
\caption{The views of the camera on the perpendicular plane to the ground}
\label{fig:table}
\end{figure}

\section{Complex experiments}

As the last step, we decided to test the whole system in a real environment. This
time we included all parts of the application -- calibration, tracking and also
localization. In the previous experiments, we tested the liability of the
trackers and the precision of the results. Now we test the usability of the
system as the whole.

\subsection{Experiment with the one small object}

The most important experiment -- the experiment which decides if the system is
usable -- is experiment under real conditions. We created a square to follow by
autonomous robot. This square have the length of the side equal 14.5~cm (see
Figure \ref{fig:robot-square}). The robot did four laps around the square. Its
speed was approximately $0.2 \text{ ms}^{-1}$. For tracking we used \hsv{}
tracker. The results are displayed in the Figure \ref{fig:square-results}.

We can see in the left picture, that the drawn line is similar to our shape.
Not having sharp edges, since our robot turns in the bends over the inner
wheel. On the other hand, the results from the another view are not so amazing.
The coordinates have a lot of noise over Z-axis. This noise is under 5~cm, on
average 1.4~cm. The noise is only present, if the robot is following horizontal
lines of the square. This corresponds to our results mentioned in the Section
\ref{s:experiment-static} -- the horizontal lines are less precise, because
the camera views are not parralel.

As the last step we computed maximum of the distances between any two points,
which were located. In this case, the expected value the length of the
diagonal, which is equal to 20.5061~cm. Our results are between 18.54~cm to
19.38~cm, depending on the run. That means, that the error is only usually
between 1 and 2~cm. This corresponds to previously measured inaccuracy in
non-parralel alignment.

\begin{figure}
\includegraphics[width=\linewidth]{img/experiments/square-robot.png}
\caption{Experiment with the robot}
\label{fig:robot-square}
\end{figure}

\begin{figure}
\centering
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/square-nice.png}
\end{subfigure}
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/square-ugly.png}
\end{subfigure}
\caption{Results of the localization -- four laps by the robot}
\label{fig:square-results}
\end{figure}

\subsection{Experiment with two objects}

This experiment will test the ability of the system to track multiple objects.
This effectively exclude \simback{} tracker as it is not able to track multiple
objects. Furthermore, we track objects with multicolored surface, making \hsv
tracker also unusable. The rest of the trackers are mostly slower, more
problematic.

We can see the setup for the experiment in the Figure \ref{fig:two-init}. We
can see two boxes, both in shades of blue. During testing many trackers failed
to track the object and lost it. We chose the \corr{} tracker as it provided
best results. It is medium fast tracker. The tracker sometimes lost the
images, but most of the time it was able to track. Because of the object lost,
sometimes an interaction of the user is needed, to reinitialize a tracker.

Sample results of the trajectories for two objects are displayed in the Figure
\ref{fig:two-trajectories}. During the experiment, the boxes were moving
between the yellow marks and swapped places. We can see these four marks easily
as the points, where the trajectories significantly changed directions.

\begin{figure}[p]
\includegraphics[width=\linewidth]{img/experiments/two-objects.png}
\caption{Initialization of the two objects}
\label{fig:two-init}
\end{figure}

\begin{figure}
\centering
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/trajectories1.png}
\end{subfigure}
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=\linewidth]{img/experiments/trajectories2.png}
\end{subfigure}
\caption{Sample trajectories from tracking two objects}
\label{fig:two-trajectories}
\end{figure}


\section{Task 2}

For this task I had to finish the implementation of a function for calculating
the ``Longest Satisfying Segment''. The code written for the solution can be
senn in Figure \ref{fig:t2code}. The part I finished is path of the LLS
operator. It takes information about two adjoining segments and creates
information about the segment resulting from joining the segments together.

\begin{figure}[ht]
    \begin{lstlisting}
    connect = p [lastx,firsty]
    newlss  = lssx `max` lssy `max` (if connect then lcsx+lisy else 0)
    newlis  = lisx `max` (if (connect && okx) then (tlx+lisy) else 0)
    newlcs  = lcsy `max` (if (connect && oky) then (tly+lcsx) else 0)
    newok   = if connect then okx && oky else False
    \end{lstlisting}
    \caption{The code written for the second task of the assignment.}
    \label{fig:t2code}
\end{figure}

\begin{itemize}
    \item[Line 1] Check wether the predicate $p$ holds over the edge between the
    segments, that is: If we have a list of the last element from list $x$ and
    the first element of list $y$, does $p$ still hold. The is used to determine
    how the two lists can be connected.

    \item[Line 2] Sets the length of the new longest satisfying segment in the
    joined list based on which value is larger: The LSS from list $x$, the LSS
    from list $y$ and if the \texttt{connect} value from Line 1 is ``true'' then
    the length of the concluding segment of $x$ joined with the length of the
    initial segment of $y$.

    \item[Line 3] Sets the length of the longest satisfying initial segment for
    the joined list by selecting the maximum of either: Length of the initial
    segment of list $x$ or if the \texttt{connect} and \texttt{okx} is true, it
    will take the total length of $x$ plus the length of the initial segment of
    $y$.

    \item[Line 4] Sets the length of the longest satisfying concluding segment
    for the joined list. Uses same method as above, and the length values of
    longest satisfying concluding segment for $y$ or if \texttt{connect} and
    \texttt{oky} is set, the total length of $y$ plus the length of the
    concluding segment of $x$.

    \item[Line 5] Determines wether or not the joined segment is ok by doing an
    ``and'' on the ok status of $x$ and $y$.
\end{itemize}