\chapter{Case Study}
\label{case_study}
\thispagestyle{empty}

In this chapter we focus on our research goal \textbf{RG1.2} as defined in section \ref{research_goals}. In the previous chapter we had worked with synthetic networks and after having conducted several experiments with them with various parameters we wish to now analyze the bias in recommendations for some empirical networks. We start the chapter by talking about the empirical data we wish to use in our thesis and how we process it and set it up for our experiment. We also list down names and few of the network properties for each of the datasets which we use. In the next section we look at the results of our experiment on each of these datasets individually. We primarily analyze the disparate visibility bias for each of the chosen datasets.

\section{Setup}

We work on the \textbf{Facebook100} \cite{traud2012social} dataset published by Traud et. al. in 2011. This huge dataset contains snapshots of Facebook \textit{`friendship networks'} for 100 universities across the United States of America. We choose 4 datasets among the 100 for our thesis and use our recommendation methods on them to receive the recommended nodes list on which we carry out the disparate visibility analysis as we had done previously for \textit{static networks} in section \ref{static_networks}.

\subsection{Dataset overview}

Table \ref{table_facebook_datadet} gives an overview of the networks we choose from the \textbf{Facebook100} dataset to carry out our experiments. The \textbf{NetworkID} is the same as used to identify individual networks in the original dataset. We choose \textit{gender} as the node attribute to distinguish nodes belonging to the \textit{minority} or \textit{majority} group. The \textbf{Minority} column in the table shows the gender representing the minority group in each network. The fraction of nodes which belong to the minority group is given as $f$. We estimate the homophily value for each group using the method outlined by Karimi et. al. \cite{karimi2018homophily}. The minority and majority homophily values are given as $h_{minority}$ and $h_{majority}$ respectively. As we had mentioned earlier in \ref{back_networks}, real-world networks exhibit asymmetric homophily, which we can observe here for the chosen empirical networks. All values for $f$, $h_{minority}$ and $h_{majority}$ have been rounded off to 2 decimal places for our experiments.

\begin{table}[h]
	\centering
	\begin{tabular}{ |c|c|c|c|c|c|c| }
		\hline
		\textbf{Network ID} & \textbf{Total nodes} & \textbf{Total edges} & \textbf{Minority} & \textbf{$f$} & \textbf{$h_{minority}$} & \textbf{$h_{majority}$} \\
		\hline
		Caltech36 & 703 & 15464 & female & 0.32 & 0.75 & 0.6 \\
		Reed98 & 865 & 15948 & male & 0.42 & 0.73 & 0.7 \\
		%Haverford76 & 1350 & 53904 & male & 0.46 & 0.73 & 0.82 \\
		Simmons81 & 1422 & 30486 & male & 0.01 & 0.47 & 0.87 \\
		Swarthmore42 & 1521 & 53726 & male & 0.49 & 0.72 & 0.78 \\
		\hline
	\end{tabular}
	\caption{Details of chosen empirical networks from \textbf{Facebook100} dataset}
	\label{table_facebook_datadet}
\end{table}

We parse the original dataset, which is provided as a sparse matrix to form our own network adjacency matrices. For each network multiple nodes are found to not possess the \textit{gender} information. We choose to discard such nodes and the edges they form from our final network.

\subsection{Experimental Setup}

For each of the networks $G(V,E)$ chosen by us (as listed in table \ref{table_facebook_datadet}) we use the \textit{Topological} and \textit{Reinforcement} recommender methods (as detailed in chapter \ref{recommender_methods}) to get a recommendation list $L_{v}$ for each node $v \in V$.

Once we have received this recommendation list, we measure the Disparate visibility \cite{fabbri2020effect} for the recommendations (similar to what we did in section \ref{static_networks}).

\section{Results}

\subsection{Caltech36}

For the network \textbf{Caltech36} \textit{females} form the minority group with 32\% of the nodes belonging to this group. The minority group has a homophily value of 0.75 which is slightly higher than the majority group homophily value of 0.6.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dd_caltech.png}
	\caption{Degree distribution for \textbf{Caltech36}}
	\label{dd_caltech}
\end{SCfigure}

Looking at the degree distribution plot (figure \ref{dd_caltech}) we can see that both majority and minority nodes hold quite high degrees. Since both the groups are homophilic in nature with minority being more homophilic, this kind of distribution can be expected for the minority fraction of 0.3.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dv_caltech.png}
	\caption{Disparate Visibility for \textbf{Caltech36}. Black dotted lines represent the range for the disparate visibility measure given the minority fraction and the solid black line shows the equal visibility mark at 0.}
	\label{dv_caltech}
\end{SCfigure}

Upon using our recommender methods, we find that the disparate visibility (figure \ref{dv_caltech}) is positive, which means that the minority nodes are more visible in recommendations than majorities. This result holds for all recommender methods used. The range of disparate visibility measure for this network lies between [-1.48, 3.08].

In our earlier experiments with measuring disparate visibility for static networks in section \ref{static_dv} we had assumed that for a homophilic regime the networks would have higher visibility for majorities. However, here we see that at the homophilic regime minorities are more visible. We however did not have information for the disparate visibility measure at homophily values of $h=0.6$ or $h=0.7$. Thus we might have missed information in our experiments which we can observe in this case.

For the \textbf{PA-Homophily} method we can deduce that minorities are more visible owing to only slightly high homophily value for majorities and a greater homophily among minorities. This would mean that for both type of nodes there is a higher chance of having minorities as node recommendations. This is also seen in the \textbf{reinforcement methods}. In the case of \textbf{Top-Rank} we see much more visibility for minorities at $r=0.0$ and $r=0.5$, but this visibility decreases from others in the case of $r=1.0$. Here we see that the ranking factor actually has a much more sever effect than what we had observed previously in our experiments with static networks. What we can deduce is that since both minority and majority nodes have higher degrees and they both have only slightly varying homophily, the ranking factor in this case plays a major role in boosting the probability for nodes in \textit{click model}. Lesser visibility for minorities would however mean that the Top-Rank method has a higher mix of majority and minority nodes at upper precedence partitions. This is what causes the minority visibility to drop.

\subsection{Reed98}

For the network \textbf{Reed98}, \textit{males} form the minority group with 42\% of the nodes belonging to this group. The minority group has a homophily value of 0.73 which is very close to the majority group homophily value of 0.7, thus having almost symmetric homophily in this network.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dd_reed.png}
	\caption{Degree distribution for \textbf{Reed98}}
	\label{dd_reed}
\end{SCfigure}

Looking at the degree distribution plot (figure \ref{dd_reed}) we can see that both the majority and minority nodes hold high degrees. Since the network is moderately homophilic, with the minority group size quite high this can be expected with both groups supporting themselves.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dv_reed.png}
	\caption{Disparate Visibility for \textbf{Reed98}. Black dotted lines represent the range for the disparate visibility measure given the minority fraction and the solid black line shows the equal visibility mark at 0.}
	\label{dv_reed}
\end{SCfigure}

From the disparate visibility plot (figure \ref{dv_reed}), upon using our recommender methods on the network, we find that visibility for both groups borders the equal visibility mark.

We had seen almost equal visibility for a minority size of $f=0.4$ in our synthetic network experiments too. The results corroborate with our findings for this network.

The range for disparate visibility measure for this network lies between [-1.72, 2.4]. Most values are slightly positive which shows that the minority group is at a very slight advantage, but this is very negligible ($\leq 5\%$). The \textbf{Top-Rank} method at $r=1.0$ however shows opposite behavior than the rest, putting majorities as more visible. The behavior which we had seen earlier in the case of \textbf{Caltech36} is thus seen here too for Top-Rank(r=1.0).

\subsection{Simmons81}

For the network \textbf{Simmons81}, \textit{males} form the minority group with only 1\% of the nodes belonging to this group. The minority group has a homophily value of 0.47 while the majority group has a homophily value of 0.87. This is a group with a very less minority who connects in an almost group-agnostic way while the majority has strong homophilic tendencies. So while the majority nodes have strong in-group support they also get connections from the minorities.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dd_simmons.png}
	\caption{Degree distribution for \textbf{Simmons81}}
	\label{dd_simmons}
\end{SCfigure}

Looking at the degree distribution plot (figure \ref{dd_simmons}) we see that the majority nodes hold much higher degrees compared to the minority as we could expect looking at the homophily values. While a fair amount of majority nodes have degrees above 100, the maximum degree a minority node holds is far less than 100. This shows that majority nodes are much more well-connected than minorities.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dv_simmons.png}
	\caption{Disparate Visibility for \textbf{Simmons81}. Black dotted lines represent the range for the disparate visibility measure given the minority fraction and the solid black line shows the equal visibility mark at 0.}
	\label{dv_simmons}
\end{SCfigure}

From the disparate visibility plot (figure \ref{dv_simmons}), upon using our recommender methods on the nodes in the network, we find that visibility is higher for majority nodes. The range of disparate visibility measure for this network lies in the range of [-1.01, 83.65].

Since the majorities have high homophily and the minorities have almost neutral homophily, the majorities would be winning in the recommendations game in the \textbf{PA-Homophily} and subsequently the \textbf{reinforcement methods} cases. For both the majority and minority nodes there would be more majority nodes which would be suggested. Also since the minorities are so few in number most of them would already be connected with other minorities thus further increasing chances of majorities to be recommended. If the minority group size was higher we would probably see a higher minority visibility for our network.

\subsection{Swarthmore42}

For the network \textbf{Swarthmore42}, \textit{males} barely form the minority group with 49\% of the nodes belonging to this group. The minority group has a homophily value of 0.72 while the majority group has a homophily value of 0.78. This is a network with equal node-group sizes exhibiting almost symmetric homophilic behavior.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dd_swarthmore.png}
	\caption{Degree distribution for \textbf{Swarthmore42}}
	\label{dd_swarthmore}
\end{SCfigure}

Looking at the degree distribution plot (figure \ref{dd_swarthmore}) we see that both majority and minority nodes have an almost equal distribution. The plot however does not strictly demonstrate the properties of a scale-free network as in there are lower degree nodes with lower probability also present in the network. The number of edges in this network is also quite high which would mean that there the nodes are well connected with a higher average degree.

\begin{SCfigure}[1][h!]
	\centering
	\includegraphics[trim=0 10 0 10, clip, width=0.75\textwidth]{images/dv_swarthmore.png}
	\caption{Disparate Visibility for \textbf{Swarthmore42}. Black dotted lines represent the range for the disparate visibility measure given the minority fraction and the solid black line shows the equal visibility mark at 0.}
	\label{dv_swarthmore}
\end{SCfigure}

From the disparate visibility plot (figure \ref{dv_swarthmore}), upon using our recommender methods on the network, we find that visibility for most cases is only negligibly higher for majority nodes. This suggests that the recommendations are quite balanced for both groups as we could have expected for a equal group size. The range of disparate visibility measure for this network lies between [-1.97, 2.03].

For the case of \textbf{Adamic-Adar} we see that the minority seems to have higher visibility than what is projected by the other methods. As we know that the Adamic-Adar method is completely structure dependent this could mean the network structure favors minorities in certain way. However this difference is very negligible to make a concrete comment on why this happens.

\section{Summary}

Using the various recommendation methods $R$ as we had defined in chapter \ref{recommender_methods} we carry out our experiment on empirical networks to get recommendations for each node in the networks. We use selected network from the \textbf{Facebook100} dataset to measure the disparate visibility found in the recommendations.

From our experiments we only see the \textbf{Caltech36} network give a higher visibility score in favor of minorities. The other networks either have an almost equal group-size or have a very small minority size which puts them close to the equal visibility measure.

In \textbf{Caltech36} we see that minorities are more visible which is different from what we were expecting from our experiments on synthetic networks. This clearly points to a limitation of our work since we do not know the disparate visibility behavior in the in-between homophily cases. Our consideration of homophily values for experiments is limited and the behavior in more such cases is something worth exploring in the future.


\chapter{Experiment Results}

\section{Introduction}
This chapter presents the results from performing the experiments. The characteristics of the dataset are summarized on the exploratory data analysis section. The last section compares PRFR and REINFORCE in terms of their effectiveness in training an AS model for solving subgraph isomorphism problems.

\section{Exploratory Data Analysis}
\subsection{Data Summary}

The following statistical measures were taken for the results of each feature and algorithm:

\begin{itemize}
	\item \textbf{min} - minimum
	\item \textbf{qu\_-1st} - 25\% quantile
	\item \textbf{med} - median
	\item \textbf{mean} -  arithmetic mean
	\item \textbf{qu\_3rd} - 75\%-quantile
	\item \textbf{max} - maximum
	\item \textbf{sd} - standard deviation
	\item \textbf{coeff\_var} - coefficient of variation (standard deviation / arithmetic mean)
\end{itemize}

\begin{table}[H]
	\centering
	\resizebox{\columnwidth}{!}{%
	\begin{tabular}{|l|r|r|r|r|r|r|r|r|}
		\hline
		\multicolumn{1}{|c|}{\textbf{Features}} & \multicolumn{1}{c|}{\textbf{min}} & \multicolumn{1}{c|}{\textbf{qu\_1st}} & \multicolumn{1}{c|}{\textbf{med}} & \multicolumn{1}{c|}{\textbf{mean}} & \multicolumn{1}{c|}{\textbf{qu\_3rd}} & \multicolumn{1}{c|}{\textbf{max}} & \multicolumn{1}{c|}{\textbf{sd}} & \multicolumn{1}{c|}{\textbf{coeff\_var}} \\ \hline
		cheap.pattern.time                      & 0.0000                            & 0.0000                                & 0.0000                            & 0.0035                             & 0.0000                                & 1.0000                            & 0.0590                           & 16.8908                                  \\ \hline
		cheap.pattern.vertices                  & 4.0000                            & 48.0000                               & 80.0000                           & 109.3787                           & 128.0000                              & 900.0000                          & 114.9942                         & 1.0513                                   \\ \hline
		cheap.pattern.edges                     & 4.0000                            & 112.0000                              & 240.0000                          & 598.8704                           & 696.0000                              & 12410.0000                        & 1073.5139                        & 1.7926                                   \\ \hline
		cheap.pattern.loops                     & 0.0000                            & 0.0000                                & 0.0000                            & 0.2451                             & 0.0000                                & 9.0000                            & 1.3121                           & 5.3543                                   \\ \hline
		cheap.pattern.meandeg                   & 1.7867                            & 3.6364                                & 5.4286                            & 10.4274                            & 10.0000                               & 99.0000                           & 14.4378                          & 1.3846                                   \\ \hline
		cheap.pattern.maxdeg                    & 2.0000                            & 8.0000                                & 11.0000                           & 20.6091                            & 26.0000                               & 269.0000                          & 27.3313                          & 1.3262                                   \\ \hline
		cheap.pattern.degisfixed                & 0.0000                            & 0.0000                                & 0.0000                            & 0.1132                             & 0.0000                                & 1.0000                            & 0.3168                           & 2.7993                                   \\ \hline
		cheap.pattern.density                   & 0.0045                            & 0.0441                                & 0.0978                            & 0.1683                             & 0.1830                                & 1.0000                            & 0.2194                           & 1.3038                                   \\ \hline
		cheap.target.time                       & 0.0000                            & 0.0000                                & 0.0000                            & 3.9988                             & 2.0000                                & 55.0000                           & 9.5821                           & 2.3963                                   \\ \hline
		cheap.target.vertices                   & 10.0000                           & 216.0000                              & 561.0000                          & 1147.3385                          & 1430.0000                             & 6671.0000                         & 1440.1846                        & 1.2552                                   \\ \hline
		cheap.target.edges                      & 15.0000                           & 930.0000                              & 2074.0000                         & 7176.6114                          & 5994.0000                             & 209000.0000                       & 21594.1676                       & 3.0090                                   \\ \hline
		cheap.target.loops                      & 0.0000                            & 0.0000                                & 0.0000                            & 1.6306                             & 0.0000                                & 144.0000                          & 13.3146                          & 8.1656                                   \\ \hline
		cheap.target.meandeg                    & 0.4357                            & 4.1983                                & 6.0000                            & 19.4312                            & 11.9172                               & 270.6950                          & 39.8146                          & 2.0490                                   \\ \hline
		cheap.target.maxdeg                     & 2.0000                            & 10.0000                               & 19.0000                           & 102.2134                           & 52.0000                               & 4973.0000                         & 384.0174                         & 3.7570                                   \\ \hline
		cheap.target.degisfixed                 & 0.0000                            & 0.0000                                & 0.0000                            & 0.1492                             & 0.0000                                & 1.0000                            & 0.3563                           & 2.3885                                   \\ \hline
		cheap.target.density                    & 0.0005                            & 0.0030                                & 0.0119                            & 0.0729                             & 0.0588                                & 1.0000                            & 0.1585                           & 2.1736                                   \\ \hline
		distance.pattern.time                   & 0.0000                            & 0.0000                                & 0.0000                            & 2.0349                             & 1.0000                                & 93.0000                           & 8.0467                           & 3.9543                                   \\ \hline
		distance.pattern.isconnected            & 0.0000                            & 1.0000                                & 1.0000                            & 0.7976                             & 1.0000                                & 1.0000                            & 0.4019                           & 0.5039                                   \\ \hline
		distance.pattern.meandistance           & 1.0000                            & 2.0010                                & 2.7778                            & 3.9349                             & 4.1227                                & 80.6687                           & 5.6418                           & 1.4338                                   \\ \hline
		distance.pattern.maxdistance            & 1.0000                            & 3.0000                                & 5.0000                            & 8.9328                             & 9.0000                                & 240.0000                          & 16.9947                          & 1.9025                                   \\ \hline
		distance.pattern.proportiondistancege2  & 0.0000                            & 0.6000                                & 0.8466                            & 0.7406                             & 0.9319                                & 0.9929                            & 0.2661                           & 0.3593                                   \\ \hline
		distance.pattern.proportiondistancege3  & 0.0000                            & 0.0974                                & 0.5000                            & 0.4605                             & 0.8237                                & 0.9834                            & 0.3507                           & 0.7615                                   \\ \hline
		distance.pattern.proportiondistancege4  & 0.0000                            & 0.0000                                & 0.1957                            & 0.3091                             & 0.6094                                & 0.9752                            & 0.3225                           & 1.0432                                   \\ \hline
		distance.target.time                    & 0.0000                            & 5.0000                                & 32.0000                           & 553.0529                           & 208.0000                              & 9962.0000                         & 1474.8373                        & 2.6667                                   \\ \hline
		distance.target.isconnected             & 0.0000                            & 0.0000                                & 1.0000                            & 0.7095                             & 1.0000                                & 1.0000                            & 0.4540                           & 0.6399                                   \\ \hline
		distance.target.meandistance            & 1.0000                            & 2.4484                                & 3.7010                            & 7.4919                             & 7.1508                                & 100.6250                          & 11.7237                          & 1.5648                                   \\ \hline
		distance.target.maxdistance             & 1.0000                            & 5.0000                                & 8.0000                            & 16.3768                            & 16.0000                               & 200.0000                          & 24.8081                          & 1.5148                                   \\ \hline
		distance.target.proportiondistancege2   & 0.0000                            & 0.6082                                & 0.9502                            & 0.7944                             & 0.9900                                & 0.9993                            & 0.2896                           & 0.3645                                   \\ \hline
		distance.target.proportiondistancege3   & 0.0000                            & 0.2472                                & 0.7260                            & 0.5990                             & 0.9502                                & 0.9985                            & 0.3708                           & 0.6190                                   \\ \hline
		distance.target.proportiondistancege4   & 0.0000                            & 0.0288                                & 0.3875                            & 0.4514                             & 0.8813                                & 0.9973                            & 0.3838                           & 0.8502                                   \\ \hline
		lad.values.removed                      & 0.0000                            & 0.0000                                & 0.0000                            & 28733.8725                         & 10673.0000                            & 844304.0000                       & 81827.2598                       & 2.8478                                   \\ \hline
		lad.values.removed.percent              & 0.0000                            & 2.3503                                & 49.2371                           & 51.7472                            & 100.0000                              & 100.0000                          & 41.8261                          & 0.8083                                   \\ \hline
		lad.values.removed.min                  & 0.0000                            & 0.0000                                & 0.0000                            & 6.7000                             & 0.3190                                & 99.9700                           & 19.4023                          & 2.8959                                   \\ \hline
		lad.values.removed.max                  & 0.0000                            & 0.0000                                & 0.0000                            & 33.1602                            & 88.2143                               & 99.9816                           & 42.8695                          & 1.2928                                   \\ \hline
		lad.time                                & 0.0000                            & 0.0000                                & 0.0000                            & 7.1210                             & 3.0000                                & 3964.0000                         & 80.7295                          & 11.3367                                  \\ \hline
	\end{tabular}%
}
\caption{Summary of features}
\label{tbl:featuresummary}
\end{table}

\begin{table}[H]
	\centering
	\resizebox{\columnwidth}{!}{%
		\begin{tabular}{|l|r|r|r|r|r|r|r|r|}
			\hline
			\multicolumn{1}{|c|}{\textbf{Algorithm}} & \multicolumn{1}{c|}{\textbf{min}} & \multicolumn{1}{c|}{\textbf{qu\_1st}} & \multicolumn{1}{c|}{\textbf{med}} & \multicolumn{1}{c|}{\textbf{mean}} & \multicolumn{1}{c|}{\textbf{qu\_3rd}} & \multicolumn{1}{c|}{\textbf{max}} & \multicolumn{1}{c|}{\textbf{sd}} & \multicolumn{1}{c|}{\textbf{coeff\_var}} \\ \hline
			lad                                      & 0                                 & 3                                     & 33                                & 5935035.15                         & 1214                                  & 1.00E+08                          & 22874350.97                      & 3.8541                                   \\ \hline
			supplementallad                          & 0                                 & 3                                     & 22                                & 4841120.76                         & 402                                   & 1.00E+08                          & 20810140.52                      & 4.2986                                   \\ \hline
			vf2                                      & 0                                 & 6                                     & 115                               & 27565110.75                        & 1.00E+08                              & 1.00E+08                          & 44243664.13                      & 1.6051                                   \\ \hline
			glasgow1                                 & 0                                 & 4                                     & 25                                & 4158094.27                         & 199                                   & 1.00E+08                          & 19437830.22                      & 4.6747                                   \\ \hline
			glasgow2                                 & 0                                 & 12                                    & 61                                & 3196878.81                         & 414                                   & 1.00E+08                          & 17126177.54                      & 5.3572                                   \\ \hline
			glasgow3                                 & 0                                 & 33                                    & 168                               & 3212027.69                         & 1072                                  & 1.00E+08                          & 17153623.91                      & 5.3404                                   \\ \hline
			glasgow4                                 & 0                                 & 113                                   & 478                               & 4333647.25                         & 7178                                  & 1.00E+08                          & 19233485.20                      & 4.4382                                   \\ \hline
		\end{tabular}%
	}
	\caption[Summary of algorithms]{Summary of algorithms. All values represent runtime in milliseconds.}
	\label{tbl:algosummary}
\end{table}

\subsection{Distributions}
Histograms for features and algorithm runtimes are displayed in the following plots. The red dashed lines denote the median of the distribution.

\begin{figure}[H]
	\centering
	\scalebox{.65}{\includegraphics{./img/featuredist.png}}
	\caption{Feature distributions}
	\label{fig:featuredist}
\end{figure}

\begin{figure}[H]
	\centering
	\scalebox{.55}{\includegraphics{./img/algodist.png}}
	\caption{Algorithm distributions}
	\label{fig:algodist}
\end{figure}

\begin{figure}[H]
	\centering
	\scalebox{.55}{\includegraphics{./img/logalgodist.png}}
	\caption{Log-scaled algorithm distributions}
	\label{fig:logalgodist}
\end{figure}

\section{AS Model Performance Evaluation}

The mean MCP of the AS model trained using REINFORCE is 4x worse than one trained using PRFR. REINFORCE is not able to surpass the performance standard set by PORTSUB, which was hoped to be surpassed in this study.

\begin{table}[H]
	\begin{tabular}{|l|l|l|l|l|}
		\hline
		\textbf{AS Model} & \textbf{Mean MCP} & \textbf{\# Solved problems} & \textbf{Mean Performance} & \textbf{Median Performance} \\ \hline
		VBS               & 0.0               & 2,219                       & 5,822,809.0               & 79.0                        \\ \hline
		PRFR              & 621,662.5         & 2,208                       & 6,446,129.0               & 1,748.5                     \\ \hline
		SBS               & 1,963,498.6       & 2,173                       & 7,786,308.0               & 539.0                       \\ \hline
		REINFORCE         & 2,318,832.0       & 2,156                       & 8,168,306.0               & 929.0                       \\ \hline
	\end{tabular}
	\caption{AS model performance evaluation results}
	\label{tbl:asresults}
\end{table}

Despite REINFORCE failing to obtain better MCP than PRFR, it showed improvement on median performance, reporting almost 2x better than PRFR.  Its significance becomes clearer by analyzing the cumulative density function (CDF) plot which visualizes the distribution of runtimes of the selected algorithms on all problems, as illustrated Figure \ref{fig:cdf}.  REINFORCE is able to select more optimal algorithms than PRFR on the easier problems (solvable within $10-10^4$ ms) on the dataset. For difficult problems which are solvable within at least 105-108 ms, REINFORCE selected worse-performing algorithms than PRFR. Although the easier problems constitute the majority (75\%) of the dataset, its influence on the mean MCP is much lesser than the harder problems. REINFORCE performed better than PRFR in solving the easy problems, but underperformed when it came to solving the more difficult problems.

\begin{figure}[H]
	\centering
	\scalebox{.9}{\includegraphics{./img/cdf.png}}
	\caption{Cumulative density plot of AS model performance}
	\label{fig:cdf}
\end{figure}


\pagebreak

\noindent\makebox[\textwidth]{\textbf{NAMES:}\enspace\dotfill}

\section*{Exercise 1: Computation of aerobic and anaerobic growth rates}
\setcounter{section}{1}

This section will show how to perform basic FBA calculations.
We will simulate the growth of \textit{E. coli} with glucose as the sole carbon source under aerobic (high O$_2$) and anaerobic (low O$_2$) conditions.

In flux balance analysis we need to define an objective function for the optimisation problem.
Objective functions are defined by setting the \pythoninline{objective_coefficient} attribute of a reaction object to a value $> 0$.
Normally, this has already been done in the model provided.
To find all the reactions with an \pythoninline{objective_coefficient > 0}, just type the following code.
\begin{code-python}
for reaction in model.reactions:
	if reaction.objective_coefficient > 0:
		print reaction
		objective = reaction
\end{code-python}
You should identify the flux towards biomass (also called growth rate in FBA) which is used by default in most models.
To inspect its composition, enter:
\begin{code-python}
print objective.reaction
\end{code-python}
You can print the full names of the reactants and products in the reaction by typing:
\begin{code-python}
for reactant in objective.get_reactants():
	print reactant.name
for product in objective.get_products():
	print product.name
\end{code-python}

\begin{questions}
\setcounter{question}{0}
\question Comment on the composition of the objective function: which general class of metabolites can you identify? Where does the stoichiometry come from?

\begin{solutionorbox}[5cm]
\input{./Solutions/solution11}
\end{solutionorbox}

\end{questions}

\noindent We will now compute the distribution of fluxes maximizing this function, and thus maximizing the growth rate.
We will set the maximum glucose uptake rate to 18.5 mmol gDW$^{-1}$ hr$^{-1}$ (millimoles per gram dry cell weight per hour, the default flux units used in COBRA).
Note that by convention, uptake reactions have a negative flux because they are written as export reactions (e.g., \pythoninline{glc_D_e <==>}).
\begin{code-python}
# Setting the lower_bound of the uptake of glucose
model.reactions.EX_glc_e.lower_bound = -18.5
\end{code-python}
By doing this, we constrain the glucose uptake rate to a maximum value of 18.5 mmol gDW$^{-1}$ hr$^{-1}$, a biologically realistic uptake rate.

\subsection*{Growth simulation in different oxygen conditions}

We will first allow unlimited oxygen availability by changing the boundaries of the oxygen uptake reaction.
To do so, we set the lower bound of the oxygen uptake reaction to a large number (making it practically unbounded).
\begin{code-python}
model.reactions.EX_o2_e.lower_bound = -1000
\end{code-python}
We will now try to optimize the objective function in these conditions.
\begin{code-python}
model.optimize()
\end{code-python}
The solution is stored in \pythoninline{model.solution}.
A \pythoninline{Solution} object has several attributes, including
\begin{itemize}
\item \pythoninline{f}: the objective value;
\item \pythoninline{status}: the exit status of the LP solver (\pythoninline{'optimal'} if a solution has been found, \pythoninline{'infeasible'} if no solution exists);
\item \pythoninline{x_dict}: a dictionary of pairs of reaction identifier and flux value;
\item \pythoninline{x}: a list for \pythoninline{x_dict}.
\end{itemize}
For example, the growth rate (the objective value of the solution) can be printed using the following command:
\begin{code-python}
print model.solution.f
\end{code-python}
A good way to analyse the solution is to draw the map of the fluxes using the module escher.
\begin{code-python}
import escher
solution = model.solution.x_dict
modelname = "e_coli:iJO1366:central_metabolism"
modelmap = escher.Builder(modelname, reaction_data=solution)
modelmap.display_in_browser()
# Press ctrl+C to continue in the console
\end{code-python}
You can change the size-scale or the color-scale of the arrows in \textit{Map>Settings}.
For instance, a good way to visualize which fluxes are non-zero is to set the ladder to [0 0 0.01].
If you want to zoom in on a component of the map (reaction or metabolite), try using \textit{ctrl+F}.

\begin{questions}
\setcounter{question}{1}
\question What is the growth rate predicted by FBA (\pythoninline{model.solution.f}) in aerobic conditions?
Which pathways carry significant (non-zero) flux under these conditions?
\begin{solutionorbox}[3cm]
\input{./Solutions/solution12}
\end{solutionorbox}
\end{questions}

\noindent Next, the same simulation is performed under anaerobic conditions.
Keep the same model and disable oxygen uptake.
\begin{code-python}
model.reactions.EX_o2_e.lower_bound = 0
\end{code-python}
Now perform the optimization and draw the map as before.

\begin{questions}
\setcounter{question}{2}
\question What is the growth rate predicted under anaerobic conditions?
Which pathways carry significant flux under these conditions?
In particular, list all the metabolites that are significantly excreted by the bacteria (transport reactions by diffusion are indicated by the suffix \pythoninline{-tex}).
Based on what you know of central carbon metabolism, comment on the discrepancies between the solutions in aerobic and anaerobic conditions.
\begin{solutionorbox}[6cm]
\input{./Solutions/solution13}
\end{solutionorbox}
\end{questions}

\section*{Exercise 2: Growth on alternate substrates}
\setcounter{section}{2}

Just as FBA was used to calculate growth rates of \textit{E. coli} on glucose in the previous exercise, it can also be used to simulate growth on other substrates.
The \textit{E. coli} model used contains exchange reactions for several organic compounds, each of which can be used as the sole carbon source under aerobic conditions.
For example, to simulate growth on succinate instead of glucose, first set the lower bound of the glucose exchange reaction \pythoninline{EX_glc_e} to \pythoninline{0} and the lower bound of the oxygen exchange reaction to \pythoninline{-1000}.
\begin{code-python}
model.reactions.EX_glc_e.lower_bound = 0
model.reactions.EX_o2_e.lower_bound = -1000
\end{code-python}
Then set the lower bound of the succinate exchange reaction \pythoninline{EX_succ_e} to \pythoninline{-20} mmol gDW$^{-1}$ hr$^{-1}$ (an arbitrary uptake rate).
\begin{code-python}
model.reactions.EX_succ_e.lower_bound = -20
\end{code-python}
You can now solve this new optimisation problem.
\begin{code-python}
model.optimize()
\end{code-python}

\begin{questions}
\setcounter{question}{0}
\question What is the growth rate of \textit{E. coli} on succinate in aerobic condition?
If you try the same optimisation in anaerobic conditions, what is the new growth rate?
\begin{solutionorbox}[2cm]
\input{./Solutions/solution21}
\end{solutionorbox}
\end{questions}

\noindent Based on what you learned during the previous exercise, predict the growth rates of \textit{E. coli} on all 13 organic substrates under \textbf{both aerobic and anaerobic conditions}. For each, use a substrate uptake rate of 20 mmol gDW$^{-1}$ hr$^{-1}$.
Such a task can be easily performed using a \pythoninline{for} loop (tips: look at the \pythoninline{get_by_id()} attribute).

\begin{questions}
\setcounter{question}{1}
\question Fill the following table (substrate names and growth rates).

\begin{tabular}{|c|c|c|c|}
\hline
\multicolumn{2}{ |c| }{\textbf{Substrate}} & \multicolumn{2}{ |c| }{\textbf{Growth rate (h$^{-1}$)}} \tabularnewline \hline
\textbf{Name} & \textbf{Id} & \textbf{Aerobic} & \textbf{Anaerobic} \tabularnewline \hline
\hspace{3cm} & ac\_e & & \tabularnewline \hline
 & acald\_e & & \tabularnewline \hline
 & akg\_e & & \tabularnewline \hline
 & etoh\_e & & \tabularnewline \hline
 & fru\_e & & \tabularnewline \hline
 & fum\_e & & \tabularnewline \hline
 & glc\_e & & \tabularnewline \hline
 & gln\_\_L\_e & & \tabularnewline \hline
 & glu\_\_L\_e & & \tabularnewline \hline
 & lac\_\_D\_e & & \tabularnewline \hline
 & mal\_\_L\_e & & \tabularnewline \hline
 & pyr\_e & & \tabularnewline \hline
 & succ\_e & & \tabularnewline \hline
\end{tabular}
\begin{solution}
\input{./Solutions/solution22}
\end{solution}

\question What is the highest growth rate? Looking at the maps, can you explain why some substrates cannot be used by \textit{E. coli} in anaerobic conditions?
\begin{solutionorbox}[3cm]
\input{./Solutions/solution23}
\end{solutionorbox}
\end{questions}

\section*{Exercise 3: Simulation of the maximum growth rate following genetic perturbations}
\setcounter{section}{3}

You can perform \textit{in-silico} knock-outs in a genome-scale model by
setting the reaction fluxes associated with the genes to 0.
For example, the gene responsible for the expression of PCK (phosphoglycerate kinase, an enzyme in glycolysis), can be deleted as follows:
\begin{code-python}
# Don't forget to reinitialize the model before doing this
model.reactions.EX_glc_e.lower_bound = -10
model.reactions.EX_o2_e.lower_bound = -1000
model.reactions.PCK.lower_bound = 0
model.reactions.PCK.upper_bound = 0
model.optimize()
print model.solution.f
\end{code-python}

\begin{questions}
\setcounter{question}{0}
\question Does the PCK knock-out strain grow?
By looking at the flux map, can you explain what is happening?
\begin{solutionorbox}[2cm]
\input{./Solutions/solution31}
\end{solutionorbox}
\end{questions}

You can perform all possible single-gene deletions in the model by using the function\linebreak \pythoninline{cobra.flux_analysis.single_deletion()}.
For each gene in the model, it sets the reaction flux corresponding to the enzyme to 0 and performs the optimization.
The function returns a dictionary containing the gene identifier and the computed growth rate, and a dictionary containing the gene identifier and the exit status of the optimization process.
\begin{code-python}
grates, status = cobra.flux_analysis.single_deletion(model)
\end{code-python}
Before using the \pythoninline{single_deletion()} function, remember to set the constraints bounding the uptake rate of the substrates.
If you want to get a list of all deletions with growth rate > 0.95~h$^{-1}$, you may execute the following code:
\begin{code-python}
listgenes = []
for gene in grates:
	if grates[gene] > 0.95:
		listgenes.append(gene)
print listgenes
\end{code-python}
Using a similar script, try to answer the following questions.

\begin{questions}
\setcounter{question}{1}
\question In an aerobic environment with glucose as the sole carbon source (uptake rate at most 10 mmol gDW$^{-1}$ hr$^{-1}$), what is the percentage of single-gene deletions that are lethal? (\textit{i.e.}, with a growth rate < 0.01.)
Select one of these genes and explain why you think it is essential.
\begin{solutionorbox}[2cm]
\input{./Solutions/solution32}
\end{solutionorbox}

\question What is the percentage of single-gene deletions with no measurable effect on growth? (\textit{i.e.}, with a growth rate $\ge$ 99\% of the wild-type growth rate.)
\begin{solutionorbox}[2cm]
\input{./Solutions/solution33}
\end{solutionorbox}

\question The same question as above for anaerobic growth. Comment the differences with the result obtained for aerobic growth.
\begin{solutionorbox}[5cm]
\input{./Solutions/solution34}
\end{solutionorbox}

\end{questions}

\section*{Exercise 4: Alternate optimal solutions: flux variability analysis}
\setcounter{section}{4}

The flux distribution calculated by FBA is often not unique. In many cases, it is possible for a biological system to achieve the same objective value by using alternate pathways, so phenotypically different alternate optimal solutions are possible.
A method that uses FBA to identify alternate optimal solutions is Flux Variability Analysis (FVA).
This is a method that identifies the maximum and minimum possible fluxes through a particular reaction with the objective value constrained to be close to or equal to its optimal value. Performing FVA on a single reaction using the basic COBRA module functions is simple.
First, calculate the maximal growth rate like before.
Get the optimal solution (\pythoninline{model.solution.f}), and then set both the lower and upper bounds of the objective reaction to exactly this value.
Next, set the reaction of interest as the objective using \pythoninline{model.change_objective()}, and use FBA to minimize and maximize this new objective in two separate steps.
This will give the minimum and maximum possible fluxes through this reaction while contributing to the optimal objective value.
All the steps described above are performed by the script below, in the case of growth on succinate.
\begin{code-python}
model.reactions.EX_glc_e.lower_bound = 0
model.reactions.EX_succ_e.lower_bound = -20
solution = model.optimize()
obj = model.reactions.Ec_biomass_iJO1366_core_53p95M
obj.lower_bound = solution.f
obj.upper_bound = solution.f

model.change_objective('ME1');
ME1_max = model.optimize(objective_sense='maximize')
ME1_min = model.optimize(objective_sense='minimize')

modelname = "e_coli:iJO1366:central_metabolism"
modelmap_max = escher.Builder(modelname,
	reaction_data=ME1_max.x_dict)
modelmap_min = escher.Builder(modelname,
	reaction_data=ME1_min.x_dict)
modelmap_max.display_in_browser()
#ctrl+C
modelmap_min.display_in_browser()
\end{code-python}

\begin{questions}
\setcounter{question}{0}
\question Draw the two alternative flux distributions on the map when the \pythoninline{ME1} reaction is set to either its minimum or maximum possible flux. Comment on the differences between the two flux distributions.
\begin{solutionorbox}[3cm]
\input{./Solutions/solution41}
\end{solutionorbox}
\end{questions}
The COBRApy module includes a built-in function for performing global FVA called \linebreak \pythoninline{cobra.flux_analysis.flux_variability_analysis()}.
This function performs the steps described above on every reaction in a model.

\begin{questions}
\setcounter{question}{1}
\question For aerobic growth on succinate, with the maximum uptake rate set to -20 mmol gDW$^{-1}$ hr$^{-1}$, what is the percentage of reactions that are completely determined, that is, reactions for which FVA returns strictly equal maximum and minimum fluxes? Comment this result.

\begin{solutionorbox}[3cm]
\input{./Solutions/solution42}
\end{solutionorbox}
\end{questions}

\section*{Exercice 5: Industrial contest}
\setcounter{section}{5}

Like \textit{Saccharomyces cerevisiae}, \textit{E. coli} is capable of producing ethanol from glucose in certain conditions.
In this question we aim at maximizing this production rate.
Before doing expensive experiments, you want to test \textit{in silico} what could be the best growth conditions and genome modifications to optimize ethanol production.
We choose a glucose uptake rate equal to 10 mmol gDW$^{-1}$ hr$^{-1}$ in anaerobic conditions.

\begin{questions}
\setcounter{question}{0}
\question By changing the objective function to ethanol excretion, what is the maximum attainable carbon yield? (\textit{i.e.}, the ratio of the number of carbon atoms included in the ethanol produced versus glucose consumed.)
What is the growth rate in this condition? (stored in \pythoninline{model.solution.x_dict}.)
\begin{solutionorbox}[2cm]
\input{./Solutions/solution51}
\end{solutionorbox}
\end{questions}

\noindent Bacteria do not naturally optimize ethanol production, but are believed to maximize their growth rate. Therefore, in order to find a more realistic solution, we need to find ways to improve ethanol production while the overall objective of the cell remains growth-rate maximization.

\begin{questions}
\setcounter{question}{1}
\question Which conditions can you find that most increase the ethanol excretion rate?
Which growth rate is attained in these conditions?
What is the carbon yield?
You can choose every (combination of) substrates (with a maximum uptake rate of 10 mmol gDW$^{-1}$ hr$^{-1}$ for each substrate).
All gene deletions are allowed, but no more than 5 in a single strain.
\begin{solutionorbox}[10cm]
\input{./Solutions/solution52}
\end{solutionorbox}
\end{questions}

\noindent We have asked you to solve the above question by trial-and-error, but one should be aware that several extensions of standard FBA have been developed to address the above question in a principled and more rigorous way. For example, MOMA (Minimization of Metabolic Adjustment) has been proposed for growth-rate maximization in mutant strains (Burgard \textit{et al.} (2003), \textit{Biotechnol. Bioeng.}, 84(6):647-57), while OptKnock and its derivatives use bilevel optimization methods to find gene deletions that force the production of desired compounds as a byproduct of optimal growth (Segr\`{e} \textit{et al.} (2002), \textit{Proc. Natl. Acad. Sci. USA}, 99(23):15112-7). MOMA and OptKnock have been integrated as functions in some of the COBRA implementations. (See for instance \pythoninline{cobra.flux_analysis.moma}.)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Author: Sébastien Gardoll copyright all rights reserved %
% email: sebastien@gardoll.fr                             %
% LaTeX Project Public License 1.3c                       %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Title of the chapter}
\label{chap:title_of_the_chapter}
Some preambule.

\section{Text emphasis}

\textbf{bold text}
\textit{italicied text}
\underline{underlined text}
\emph{emphasising text}

\section{Itemize example}

\begin{itemize}
\item {Item1: \lipsum[1]}
\item {Item2}
\item {Item3}
\end{itemize}
\begin{itemize}
\item {carriaged return item:}
\end{itemize}
\lipsum[1]

\section{Section}

\lipsum[1]
\lipsum[2]

\subsection{Subsection}

\lipsum[1]

\subsubsection{Subsubsection}

\lipsum[1-2]

\section{Image inclusions}

\ref{fig:pic1.png} is an example of a full width page inclusion
of a picture (aspect ratio is kept):

\includeimg{1}{pic1.png}{The caption of the picture.}

\ref{fig:pic2.png} is an example of a scaled down picture:

\includeimg{0.5}{pic2.png}{The caption of the picture.}

\ref{fig:pic3.png} is an example of a rotated picture:

\includeimgsideways{0.5}{pic3.png}{The rotated caption of the picture.}

\section{Citations}

According to \cite{inbook-full}...
This is a footnote citation\footfullcite{incollection-minimal}.

\section{Some fancy maths}

\[\textit{MSE}=E(Y-\hat{Y})^2=\underbrace{\left[F(X)-\hat{F}(X)\right]^2}_{\text{reductible}} + \underbrace{\textit{Var}(\varepsilon)}_{\text{irréductible}}\]

\[d=\left\{
        \begin{array}{l}
        1\ si\ A>\alpha \\
        0\ si\ A\leqslant\alpha
        \end{array}
        \right.
      \]

\[d(x_1,\ x_2)=\sqrt{\ \sum_{j=1}^n\frac{1}{f_n}(f_{1j}-f_{2j})^2}\]

\begin{theo}[Morgan's Law]
\[\overline{P \wedge Q} \Leftrightarrow \bar{P} \vee \bar{Q}\]
\[\overline{P \vee Q} \Leftrightarrow \bar{P} \wedge \bar{Q}\]
\end{theo}

\section{Tables}

From the CTABLE man (link \href{https://ctan.org/pkg/ctable}{here}):

\ctable[
cap     = The Skewing Angles,
caption = The Skewing Angles ($\beta$) for
$\fam0 Mu(H)+X_2$ and $\fam0 Mu(H)+HX$~\tmark,
label   = nowidth,
pos     = h
]{rlcc}{
\tnote{for the abstraction reaction,
$\fam0 Mu+HX \rightarrow MuH+X$.}
\tnote[b]{1 degree${} = \pi/180$ radians.}
\tnote[c]{this is a particularly long note, showing that
footnotes are set in raggedright mode as we don’t like
hyphenation in table footnotes.}
}{                                                          \FL
&            & $\fam0 H(Mu)+F_2$     & $\fam0 H(Mu)+Cl_2$ \ML
&$\beta$(H)  & $80.9^\circ$\tmark[b] & $83.2^\circ$       \NN
&$\beta$(Mu) & $86.7^\circ$          & $87.7^\circ$       \LL
}