\chapter{System architecture of HyCube - a distributed hash table based on a variable metric}
\label{sec:hycubeArchitecture}

This chapter presents the routing architecture of \emph{HyCube} - a distributed hash table system based on a hierarchical hypercube geometry. Sections \ref{sec:hycubeHierarchicalHypercube} and \ref{sec:hycubeRoutingTables} introduce the hierarchical hypercube concept and describe the structure of routing tables maintained by nodes. Section \ref{sec:routing} presents the basic routing algorithm, and the subsequent sections present various optimizations of the routing algorithm. The optimizations of the basic routing algorithm include the use of a variable routing metric (defining distances between nodes) adopting Steinhaus transform (Section \ref{sec:metric}) - the routing metric is modified by nodes on routes in a way resulting in a very high degree of flexibility in next hop selection (and this a very high level of resilience to node failures), preserving the routing efficiency of the basic algorithm. The concepts presented in this document have been published in \cite{hycubePpam2009} and \cite{hycubeICSReportFeb2013}.

\section{Hierarchical hypercube geometry}
\label{sec:hycubeHierarchicalHypercube}

The routing geometry of \emph{HyCube} is a combination of the tree geometry and the hypercube geometry. It is similar to \emph{Plaxton mesh} \cite{plaxton1}, \cite{plaxton2}, but nodes are also logically located in vertices of a $d$-dimensional hierarchical hypercube. A hierarchical hypercube is a hypercube whose vertices are also (lower level) hypercubes. Vertices of the hypercubes at the lowest level are positions which may be assigned to nodes.

\begin{figure}
\centering
\includegraphics[scale=.6]{img/hh.pdf}
\caption{A hierarchical hypercube (3 dimensions and 2 levels of hierarchy)}
\label{fig:hierarchicalHypercube}
\end{figure}

Figure \ref{fig:hierarchicalHypercube} presents the structure of an exemplary hierarchical hypercube with 3 dimensions and 2 hierarchy levels. Node IDs are determined by their positions - the identifier of a node is a string of $d$-bit groups determining positions of the node in hypercubes at individual levels (starting with the hypercube at the highest level). The position in a hypercube at a particular level is a number built of bits corresponding to the positions of the node in the hypercube in individual dimensions. The length of the identifier equals $d \cdot l$, where $d$ is the number of dimensions, and $l$ is the number of levels.

The hierarchical hypercube and the tree geometries are isomorphic. However, visualizing the structure as a hierarchical hypercube gives an idea of the spatial arrangement of nodes in a $d$-dimensional space - the numbers formed from bits corresponding to individual dimensions relate to the coordinates of the node in these dimensions in the system of coordinates with the center in point $0$. Thus, considering the ID space as a segment of $\mathbb{Z}^d$ space ($\mathbb{Z}$ denotes the set of integer numbers), the distance between nodes may be defined by any metric applicable to $\mathbb{Z}^d$, or $\mathbb{R}^d$ ($\mathbb{R}$ - the set of real numbers) as $\mathbb{Z}$ is a subset of $\mathbb{R}$. However, the geometry of \emph{HyCube} should be seen as a $d$-dimensional torus with the perimeter equal to $2^l$ in each dimension (the set of coordinates in each dimension is treated as on a ring). That means that after the point $2^l-1$, point $0$ is located, and all arithmetic is done modulo $2^l$. This fact is important in determining distances between nodes - in every dimension the distance is determined like on a ring - it is the shorter of the distances in either direction.

In \emph{HyCube}, the default number of dimensions is 4 and the number of levels is 32, resulting in a 128-bit address space (which allows avoiding conflicts of identifiers in majority of applications).

\section{Routing tables}
\label{sec:hycubeRoutingTables}

\subsubsection{Primary routing table.}

The primary routing table has the same structure as in \emph{Plaxton mesh}. It has $l$ levels (the number of hierarchy levels), and, at each level, there are $2^d$ slots ($d$ - the number of dimensions). In the primary routing table of a node $X$, the slot $j$ at level $i$ ($i \geq 0$) contains a reference to a node that is located in the same hypercube at level $i+1$ and in the hypercube corresponding to the number $j$ at level $i$ (lower level). At each level $i>0$, one slot corresponds to the hypercube in which the node $X$ is located - this slot is left empty, as the routing table contains a whole level corresponding to this hypercube.

An exemplary primary routing table for a 2-dimensional hierarchical hypercube with 6 hierarchy levels for node $X$ = 112013 is presented in Table \ref{tab:rt1Example}. For clarity, groups of bits are represented by quaternary digits (base-4 numeral system). The digits in bold represent the sub-hypercube addresses corresponding to routing table slots at individual levels. The underlined digits represent the hypercubes corresponding to the routing table slots matching the hypercubes of node $X$ at individual levels.

\begin{table}
\scriptsize
\begin{center}
%\begin{tabular}{p{2cm}|p{1.3cm}|p{1.3cm}|p{1.3cm}|p{1.3cm}|}
\begin{tabular}{c|c|c|c|c|}
	\hhline{*{1}{~}*{4}{|-}|~|}
																				& \texttt{\textbf{0}}									& \texttt{\textbf{1}}											& \texttt{\textbf{2}}											& \texttt{\textbf{3}}										\\
    \hline
    \multicolumn{1}{|c|}{\texttt{\textbf{Level 5}}}								& \texttt{\underline{\textbf{0}}11033}					& \texttt{-}													& \texttt{\underline{\textbf{2}}31011}							& \texttt{\underline{\textbf{3}}00232}						\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 4}}}								& \texttt{\underline{1\textbf{0}}2223}					& \texttt{-}													& \texttt{\underline{1\textbf{2}}1301}							& \texttt{\underline{1\textbf{3}}0001}						\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 3}}}								& \texttt{\underline{11\textbf{0}}113}					& \texttt{\underline{11\textbf{1}}201}							& \texttt{-}													& \texttt{\underline{11\textbf{3}}302}						\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 2}}}								& \texttt{-}											& \texttt{\underline{112\textbf{1}}01}							& \texttt{\underline{112\textbf{2}}03}							& \texttt{\underline{112\textbf{3}}12}						\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 1}}}								& \texttt{\underline{1120\textbf{0}}3}					& \texttt{-}													& \texttt{\underline{1120\textbf{2}}1}							& 															\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 0}}}								& 														& 																& 																& \texttt{-}												\\
    \hline
\end{tabular}
\end{center}
\caption{Exemplary primary routing table for node 112013 (for a 2-dimensional hierarchical hypercube with 6 hierarchy levels).}
\label{tab:rt1Example}
\end{table}

\subsubsection{Secondary routing table.}

The secondary routing table of a node $X$ contains nodes from adjacent hypercubes to the hypercube of node $X$ in each dimension, in both directions, at each level. An adjacent hypercube (at any level) is the one whose coordinate in one dimension is greater or smaller by 1 than the coordinate of the same level hypercube of $X$ (modulo $2^l$), and coordinates in all other dimensions are equal to those of $X$. The secondary routing table does not contain nodes in slots at the highest level, as hypercubes corresponding to them are covered by the primary routing table. Also, one of the adjacent hypercubes at each level in each dimension is covered by a primary routing table slot.

The secondary routing table increases the level of flexibility in the next hop selection. If the distance between nodes is defined by a metric in $\mathbb{R}^d$ space, it is very likely that the secondary routing table contains nodes that are closer to any arbitrarily chosen node. Furthermore, it provides additional shortcut references when a message is routed to a node that is close in the $\mathbb{R}^d$ space, but is not close in terms of the \emph{Plaxton mesh} distance. With the use of the primary routing table, routing a message between nodes sharing a short ID prefix would require many steps of traversing the tree structure.

Table \ref{tab:rt2Example} presents an exemplary secondary routing table for node $X$ = 113012 (binary 01'01'11'00'01'10) - for a 2-dimensional hierarchical hypercube with 6 hierarchy levels. For clarity, in this example, the IDs are represented as binary numbers. Addresses of adjacent hypercubes corresponding to the routing table slots are marked in bold, and bits of addresses of adjacent hypercubes corresponding to the particular dimension are underlined - the numbers built of these bits are larger by 1 or smaller by 1 than the numbers formed of the corresponding bits of $X$. All other hypercube address bits (the remaining dimensions) are equal to the corresponding ones of $X$.

\begin{table}
\scriptsize
\begin{center}
%\begin{tabular}{p{1.7cm} p{2.7cm} p{2.7cm} p{2.7cm} p{2.7cm}}
\begin{tabular}{c|c|c|c|c|}
    \hhline{*{1}{~}*{4}{|-}|~|}
														& \multicolumn{2}{c|}{\texttt{\textbf{Dimension 0}}}																																		& \multicolumn{2}{c|}{\texttt{\textbf{Dimension 1}}} 																																							\\
    \hhline{*{1}{~}*{4}{|-}|~|}
														& $\leftarrow$																			& $\rightarrow$																						& $\leftarrow$																						& $\rightarrow$																						\\
	\hline
    \multicolumn{1}{|c|}{\texttt{\textbf{Level 5}}}		& \texttt{-}																			& \texttt{-}																						& \texttt{-}																						& \texttt{-}																						\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 4}}}		& \texttt{\textbf{0\underline{1}'0\underline{0}}'01'01'00'11}							& \texttt{\textbf{0\underline{0}'0\underline{0}}'10'01'10'10}										& \texttt{\textbf{\underline{1}1'\underline{1}1}'10'01'00'11}										& \texttt{\textbf{\underline{0}1'\underline{1}1}'00'10'10'01}										\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 3}}}		& \texttt{\textbf{0\underline{1}'0\underline{1}'1\underline{0}}'10'01'00}				& \texttt{\textbf{0\underline{0}'0\underline{0}'1\underline{0}}'11'01'01}							& \texttt{\textbf{\underline{0}1'\underline{0}1'\underline{0}1}'11'01'01}							& \texttt{\textbf{\underline{0}1'\underline{1}1'\underline{0}1}'01'00'01}							\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 2}}}		& \texttt{\textbf{0\underline{1}'0\underline{1}'1\underline{0}'0\underline{1}}'11'10}	& \texttt{\textbf{0\underline{1}'0\underline{1}'1\underline{1}'0\underline{1}}'01'01}				& \texttt{\textbf{\underline{0}1'\underline{0}1'\underline{0}1'\underline{1}0}'01'10}				& \texttt{\textbf{\underline{0}1'\underline{0}1'\underline{1}1'\underline{1}0}'10'00}				\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 1}}}		& 																						& \texttt{\textbf{0\underline{1}'0\underline{1}'1\underline{1}'0\underline{1}'0\underline{0}}'10}	& \texttt{\textbf{\underline{0}1'\underline{0}1'\underline{0}1'\underline{1}0'\underline{1}1}'00}	& \texttt{\textbf{\underline{0}1'\underline{0}1'\underline{1}1'\underline{0}0'\underline{1}1}'10}	\\
	\multicolumn{1}{|c|}{\texttt{\textbf{Level 0}}}		& 																						& 																									& 																									& 																									\\
     \hline
\end{tabular}
\end{center}
\caption{Exemplary secondary routing table for node 113012 (01'01'11'00'01'10) - for a 2-dimensional hierarchical hypercube with 6 hierarchy levels.}
\label{tab:rt2Example}
\end{table}

\subsubsection{Neighborhood set (closest neighbors set).}

In addition to the routing tables described above, nodes maintain sets of closest to them (according to the chosen metric) nodes existing in the system - called \emph{neighborhood sets}. These sets may allow finding a next hop (routing/lookup), decreasing the distance left to the destination, even if there are no appropriate nodes in both routing tables. The closest neighbors sets are expected to increase the probability of delivering messages in the presence of node failures. The neighborhood sets have also very good properties for supporting joining and leaving procedures, as well as maintenance and recovery algorithms, and they help to keep the DHT consistent. Moreover, their existence is crucial for searching closest nodes for a given key, as well as for replicating resources, which will be discussed in detail in Chapters \ref{sec:lookupSearch} and \ref{sec:resourcesManagement}. In \emph{HyCube}, the default size of the neighborhood set is 16.

\section{Basic routing algorithm}
\label{sec:routing}

Let us consider routing a message from node $X$ to node $Y$. Every node $R$ along the route first checks if there is a reference to the destination node in its neighborhood set, in which case, the message is sent directly to that node. Otherwise, the routing tables are searched for an appropriate next hop - the node that shares at least one $d$-bit group longer prefix of ID with $Y$ (than with $R$) or shares the same number of $d$-bit groups of the ID but is closer to $Y$ than $R$ in terms of the chosen routing metric. The routing metric of \emph{HyCube} is discussed in Section \ref{sec:metric}. For the time being, let us assume that routing converges according to the Euclidean metric. The detailed algorithm is presented below:

\begin{enumerate}

\item Initially, the routing algorithm finds the slot in the primary routing table that corresponds to nodes sharing at least one group of $d$ bits longer prefix of ID with $Y$ than with the current node ($R$). In a hierarchical hypercube, this slot corresponds to a hypercube in which the destination node is located, at a lower level than the lowest-level hypercube containing both, the current and the destination node. If for the current node and $Y$, the common prefix length equals $i$, and the next $d$-bit group of the ID of $Y$ equals $j$, $j$-th routing table slot at level $l - 1 - i$ is used. If this slot is not empty, the message is routed to the node found in the slot - increasing the common prefix length with the destination node $Y$ by at least one $d$-bit group.

\item If no node is found in the appropriate primary routing table slot, nodes sharing the same prefix length with $Y$ as with $R$ (in terms of the number of bit groups), but closer to $Y$ in terms of the chosen metric, are also considered - both routing tables and the neighborhood set are checked. From the set of nodes found, the node sharing the longest prefix of the ID with $Y$ (number of $d$-bit groups) is chosen, and, if there are more than one such nodes, the node closest to $Y$ (according to the routing metric) is chosen for the next hop.

\end{enumerate}

The primary routing table supports routing based on extending the ID prefix (in terms of $d$-bit groups) - tree-based routing (\emph{Plaxton mesh}), and the secondary routing table supports finding a closer node in any dimension - such nodes are likely to be closer to the destination node also in terms of the Euclidean metric. Both routing tables and the neighborhood set are used for determining the best possible next hop, to which the message is routed.

It can be shown that the expected route length equals $\lceil\log_{2^d} N\rceil$ hops and, on average, $\lceil\log_{2^d} N\rceil \cdot (2^d-1)$ slots are populated in the primary routing table and $(\lceil\log_{2^d} N\rceil - 1) \cdot d$ in the secondary routing table ($N$ is the number of nodes in the network)\footnote{Based on the assumption that nodes are uniformly distributed in the hierarchical hypercube}.

\subsubsection{Message TTL}

To limit the maximum number of hops for messages, every message contains the TTL information (determining the maximum number of hops) and the number of hops that the message already passed. The TTL value should be decremented, and the number of hops is incremented by every node along the route (including the initiating node) before passing it to the next hop. When the TTL falls below 0, after decrementing, the message should not be routed any further and be dropped.

\subsubsection{Acknowledging message delivery and detecting duplicates}

Optionally, after receiving messages (DATA messages - sent at the application level), depending on configuration, nodes may send acknowledgments (DATA\_ACK messages) confirming receiving of the messages (either directly to the sending node, or routed via the system to the message sender). When the sending node receives the DATA\_ACK message, it knows that the original message was successfully received and processed, and, when no acknowledgment is received (timeout), the node might resend the message (automatic resending may also be configured - up to the defined maximum number of retries). In cases when the acknowledgment mechanism is implemented at the application level (beyond the scope of \emph{HyCube}), the native \emph{HyCube} mechanism may be switched off.

\emph{HyCube} also implements message duplicate detection. In case of receiving a message duplicate, the duplicate is dropped. Message duplicates may be received as a result of incorrect routing or network problems, or an ACK message not being delivered. Duplicates are detected based on the header of the message (the details are specified in Chapter \ref{sec:protocol}).

\section{Number of dimensions versus number of levels}

The address (node identifiers) space should be large enough to prevent potential conflicts of identifiers (existence of two nodes with the same ID). The size of the address space in \emph{HyCube} equals:

\begin{equation}
N_{max} = 2^{d \cdot l}
\end{equation}

\noindent
which means that increasing the number of dimensions or the number of hierarchy levels would increase the number of possible identifiers that nodes may be assigned. However, the number or dimensions and levels of the hierarchical hypercube influences the system characteristics.

Adding additional levels of hierarchy causes the address space to increase, but it also proportionally increases the number of routing table slots that nodes maintain. Moreover, the pessimistic route length would also increase, as in the pessimistic scenario, routing steps correspond to individual routing table levels. Nevertheless, the expected route length remains at the same level, because, on average, similar number of routing table slots would be populated (with high probability the lower-level slots are empty).

Increasing the number of dimensions, on the other hand, has a very strong impact on routing characteristics. Although the routing tables grow sharply with the increase of the number of dimensions, the routing algorithm is more specific in selecting next hops - in each routing step, the common prefix with the destination node ID is increased by a larger number of bits, maintaining the same pessimistic path length and decreasing the average path length. As the base of the logarithm (Equation \ref{equ:expPathLenRT1}) grows exponentially, the expected path length is inversely proportional to the number of dimensions:

\begin{equation}
\label{equ:expPathLenRT1}
\log_{2^d} N = \frac{1}{d} \log_2 N
\end{equation}

\noindent
However, the maintenance cost is significant, as the primary routing table size grows exponentially with the number of dimensions:

\begin{equation}
\log_{2^d} N \cdot (2^d-1) = \frac{\log_{2} N \cdot (2^d-1)}{d}
\end{equation}

At one extreme, when the number of dimensions is equal to the number of identifier bits (1 level of hierarchy), every node would maintain references to all other nodes in the system, and the routing table slots would correspond to all possible values of node identifiers. Increasing the number of dimensions may influence any algorithm used in the distributed hash table and may require adjusting the parameter values for the specified number of dimensions. That is why it is crucial to determine a reasonable value for the number of dimensions that would ensure good routing properties and reasonable sizes of the routing tables, while controlling the identifier length should be done by increasing/decreasing the number of hierarchy levels. Whenever the number of dimensions is changed, it should be verified again that the DHT algorithms parameters still have their optimal values.

\section{Routing table nodes selection}
\label{sec:HyCubeNodeSelection}

The routing table node selection technique used in \emph{HyCube} is a variant of the LNS approach, which, at the same time, provides means to remove failed nodes from routing tables.

\subsubsection{LNS (liveness node selection) in HyCube}
\label{sec:HyCubeNodeSelectionLNS}

\emph{HyCube} uses a variant of LNS which bases the neighbor choice on node liveness information discovered locally, working together with a background process checking node's responsiveness. However, the \emph{HyCube} LNS implementation does not calculate the liveness information based on the node's join and leave times. Every node periodically sends keepalive (PING) messages to all the nodes in its routing tables and updates the stored nodes' liveness values. The value is increased when the node responds to the PING message (sending a PONG message), and is decreased if the keepalive response is not received (timeout). The initial liveness value (new nodes) is defined by the system parameter $L_{init}$, and the values are updated as follows:

\begin{equation}
L = L_{prev} \cdot p + (1-p) \cdot L_{max}
\end{equation}

\noindent
if the keepalive confirmation (PONG message) was received, or:

\begin{equation}
L = L_{prev} \cdot p
\end{equation}

\noindent
when the keepalive fails (no PONG message is returned). $L_{prev}$ is the previous value of liveness, $L$ is the new, updated liveness value, $L_{max}$ and $p$ are system parameters. When $L$ falls below the threshold value $L_{deactivate}$ (system parameter), the node is marked as deactivated and is not used in the routing procedure until $L$ falls below the threshold value $L_{remove}$ (parameter), in which case the node is permanently removed from routing tables, or until $L$ reaches above $L_{deactivate}$ again. Whenever the value of $L$ is below the value of the parameter $L_{replace}$, when possible, the current node in the routing table slot is replaced by a new node, whose value $L$ is then given the initial value $L_{init}$. The value of $L$ for any node removed from the routing table should however be stored by nodes for some time to prevent replacing new nodes that temporarily fall below $L_{replace}$ with nodes that were permanently unresponsive. The values $0 < p < 1$ (update coefficient), $L_{deactivate}$, $L_{replace}$ and $L_{remove}$ determine the sensitivity of the algorithm, and $L_{max}$ defines the maximum value of $L$. This technique provides a good way to keep long-living responsive nodes in routing tables and eliminate failed or overloaded nodes that are not able to handle requests. Reasonable values of $p$, $L_{deactivate}$, $L_{replace}$ and $L_{remove}$ should be chosen to prevent the algorithm from removing nodes from routing tables in case of temporary delays in sending responses by nodes, but still, to be able to efficiently remove failed nodes. If $L_{max} = 2$, the values of $L_{init} = 1.5$, $p = 0.5$, $L_{deactivate} = 1$, $L_{replace} = 0.5$ and $L_{remove} = 0.05$ are good defaults as they allow avoiding failed paths due to temporary nodes' unresponsiveness, at the same time allowing unresponsive nodes to be replaced by new nodes. Using these default values would cause deactivating nodes after a single failed keep-alive, however, not replacing/removing them immediately. The parameter values (as well as the keep alive interval) may be fine-tuned depending on the system characteristics. The flexibility in next hop selection guaranteed by \emph{HyCube} ensures that no significant increase in failed paths rate, nor the average path length, is observed even if large portions of nodes temporarily do not respond to keep-alive messages and are deactivated.

\subsubsection{\emph{HyCube} mixed mode node selection}

The LNS technique described above allows replacing a routing table node only when its value of $L$ drops below $L_{replace}$ (or when $L$ reaches $L_{remove}$, in which case the node is removed). In some cases, however, it would be desirable to take another criterion (or criteria) into account, like for example proximity or any application specific measure. To achieve that, the condition $L < L_{replace}$ may be relaxed, and the neighbor selection could be based on a function measuring both factors - the liveness and the second factor (algorithm-specific). The liveness fulfillment factor may be defined as follows:

\begin{equation}
	Fact_{L} = \left|\frac{L - L_{replace}}{L_{replace}}\right|^{e_L} \cdot \mathop{\mathrm{sgn}}(L - L_{replace})
\end{equation}

\noindent
where the exponent $e_L$ determines the exponential growth of $Fact_{L}$ depending on the difference $L - L_{replace}$. Depending on the algorithm, this factor may be used to calculate overall criteria fulfillment at a node level, improvement of the criteria fulfillment for a new node, relative to current node(s), or may be used in calculating the quality function value for a set of nodes (for example for the neighborhood set). An exemplary neighbor selection criterion at a node level might be a weighted sum of two factors:

\begin{equation}
	Q = \alpha \cdot Fact_{L} + \beta \cdot Fact_{X}
\end{equation}

\noindent
where $Fact_{X}$ may be any neighbor selection algorithm specific function.

\section{Prefix mismatch heuristic}
\label{sec:pmh}

In the final part of a route, when the message is already relatively close to the destination node, the routing algorithm may omit some nodes that are close to the destination, but do not share the same long or longer prefix of ID with the destination node than with the current node. This phenomenon becomes more significant when a multidimensional metric is used. That is why, like in \cite{pastry}, \emph{HyCube} uses a heuristic switching to routing based only on the distance left, when a message is already in vicinity of the destination. Nodes should therefore be able to determine how close the message is to the destination in relation to the density of nodes in the space. In \emph{HyCube}, before choosing the next hop, each node checks if the distance to the destination is shorter than the average distance to the nodes in the neighborhood set multiplied by the factor $\lambda$:

\begin{equation}
d_{dest} < \mathop{\mathrm{avg}}(d_{neigh}) \cdot \lambda
\end{equation}

\noindent
If this condition is satisfied, all further nodes on the route are chosen based only on their distance to the destination node - they might not share the same long or longer prefix of the identifier. All nodes from both routing tables and the neighborhood set are checked and the closest node is chosen for the next hop.

The prefix mismatch heuristic may be also enabled, when no next hop is found with routing based on extending the common prefix length with the destination node. This behavior may be configured by changing a system parameter value. In many cases, the number of neighborhood set nodes matching the destination is much larger if the selection is based only on the distance. Obeying the prefix condition is a much stronger constraint on the next hops, and may cause more failed paths, especially in the presence of many node failures. Thus, although the path length might increase, by default, \emph{HyCube} switches to routing based only on the distance left whenever no next hop is found. To maintain routing convergence, the prefix mismatch heuristic is followed by all subsequent nodes along the path.

The greater is the value of $\lambda$, the longer parts of routes will be determined based only on the distance left. The value should be large enough to ensure high probability of message delivery. However, too large values of $\lambda$ could cause an increase in path lengths. The performed simulations indicated that the value $\lambda = 1.5$ ensures good static resilience, maintaining the path lengths at a low level.

\section{Routing metric}
\label{sec:metric}

In DHT systems, distances between pairs of nodes are defined by a certain metric, and the routing converges according to this metric. The choice of the metric has a great impact on the average route length and the probability of message delivery. \emph{HyCube} uses a variable multi-dimensional metric adopting the Steinhaus transform, described in this section. It was confirmed during simulations that such an approach allows reaching a very high level of resilience to node failures and short pessimistic and average routing/lookup paths.

\subsection{Steinhaus transform}
\label{sec:steinhaus}

In \cite{nearNeighSearchMetrSpDim} and \cite{geomCutsMetrics}, the authors present the Steinhaus transform. The terminology comes from the fact that this distance was used in biological problems for the study of biotopes \cite{certDistSetsCorrDistOfFunc}. The theorem presented says that if $D$ is a metric on a set $X$, $D'$ is also a metric on $X$ for any $a \in X$, where:

\begin{equation}
\label{equ:steinhaus}
D'(x,y) = \frac{2D(x,y)}{D(x,a) + D(y,a) + D(x,y)}
\end{equation}

The default base metric used in \emph{HyCube} ($D$ in the equation above) is the Euclidean metric. Applying a metric with the Steinhaus transform to every route, setting the value of $a$ to the ID of the source node, causes the next hops to be chosen in such a way that they are closer to the destination node and more distant from the source node. Such an approach increases the expected number of neighborhood set nodes to which messages may be routed in each routing step - although some closer nodes (according to metric $D$) might be considered more distant when the Steinhaus transform is applied, it allows sending messages using more roundabout routes, while still being convergent to the destination node.

One important remark should be made regarding the Steinhaus transform. In the case when $x = y = a$, Equation \ref{equ:steinhaus} does not have a value (division by zero). Therefore, the value of the distance should be considered 0 if $x = y$, regardless of the value of $a$.

\subsection{Variable metric adopting Steinhaus transform}
\label{sec:varSteinhaus}

The use of the Steinhaus transform yields very good routing parameters and very high resilience to node failures for networks containing relatively few nodes. However, for networks containing much more nodes (denser), in final parts of routes, the addend $D(x,a)$ of the denominator in Equation \ref{equ:steinhaus} (where $x$ is the current node), has less influence on the value of the distance as its changes in individual steps are very small compared to the value of the entire denominator. Thus, the more nodes in the network, the lesser is the influence of the Steinhaus transform on routing. However, a certain modification can be introduced - a variable metric adopting the Steinhaus transform, where point $a$ would be changed by intermediate nodes along routes. The value of $a$ would initially be set to the source node $ID$, and subsequent nodes, before choosing the next hops, would check whether they are closer (in terms of the Euclidean metric) to the destination than the current point $a$. In such a case, point $a$ would be updated - would be given the value of the current node. Such a way of changing point $a$ ensures that the routing is convergent to the destination (there will be no cycles on routes) and yields a very high level of flexibility in the next hop selection along the whole route, regardless of the network size. It is easy to notice that whenever the Steinhaus point has the value of the current node ID, messages may be passed to any other node, decreasing the Steinhaus distance left to the destination. With great probability the node closest to the destination in terms of the Euclidean metric is chosen. It may however not be a node that is closer (Euclidean) to the destination. Nevertheless, in such a case, the subsequent next hop selections would be more restrictive and converge to the destination faster. The expected route length is still proportional to $\sqrt[d]{N}$, and, owing to the increase in the flexibility in the next hop selection, the resilience to node failures reached is very high.

It should be noted that whenever the Steinhaus point is equal to any of the node IDs between which the distance is measured, the distance value equals 1, regardless of the second argument value:

\begin{equation}
D'(x,y)\bigg|_{a=x} = \frac{2D(x,y)}{D(x,x) + D(y,x) + D(x,y)} = 1
\end{equation}

\begin{equation}
D'(x,y)\bigg|_{a=y} = \frac{2D(x,y)}{D(x,y) + D(y,y) + D(x,y)} = 1
\end{equation}

\noindent
When calculating distances from multiple nodes to the destination node, if the Steinhaus point is given the value of the destination node ID, it is impossible to differentiate the nodes, as all the distances are then equal to 1. The routing algorithm is not exposed to such a situation - the Steinhaus point value would be equal to the destination point only when the destination is already reached. However, any other algorithm using Steinhaus distances, modifying the Steinhaus point, should take that fact into account (e.g. search algorithm, described in Sec. \ref{sec:lookupSearch}).

To limit the average route length increase caused by employing the variable Steinhaus metric, one more modification was introduced - the Steinhaus transform should be used only when the prefix mismatch heuristic is already applied. The use of the Steinhaus transform is the most important when the message is already in the proximity of the destination node, and the probability of finding the next hop in the neighborhood set drops sharply with the use of the Euclidean metric. If the Steinhaus transform is used only when the prefix mismatch heuristic is applied, it is enabled when the message gets close to the destination, or when no next hop is found (which would also enable the prefix mismatch heuristic, allowing the Steinhaus transform to be used). Such a modification decreases the average path length, preventing messages from being routed to more distant areas (according to the Euclidean metric) in initial steps, when the average hop distances are much larger. Despite a significant decrease of the average path length, the modification did not cause any static resilience decrease.

\section{Euclidean distance versus Steinhaus distance - re-routing using regular metric}
\label{sec:euclideanAfterSteinhaus}

The final steps of routing with the use of a metric with the Steinhaus transform applied may cause a message to be sent to a node that is more distant from the destination than it would be if the Steinhaus transform was not applied. Therefore, when, at some point, a node on a route cannot find the next hop in its routing tables and neighborhood set, it is possible that the route is ended in a point (node) that is not the closest one to the destination in terms of the Euclidean metric. For some applications, if the destination node itself cannot be reached, it is crucial to reach the closest possible node. Thus, \emph{HyCube} introduces one more modification - when a message cannot be routed by a node, the node tries to route it again, based only on the Euclidean distance left to the destination. All consecutive next hops after that should be chosen in the same way. Such an approach will cause that, in the case the message is dropped, a relatively close node to the destination is reached. From the experiments (for a network containing 10'000 nodes, with 50\% failed nodes, routing using only neighborhood sets), it appears that applying this phase in routing allows messages to be sent to a closer node in about 80\% cases. Furthermore, this additional phase of routing also increases the resilience of the system to node failures.

There is, however, one drawback of such an approach - the failed path lengths (undelivered messages) may possibly increase due to re-routing with another metric after the next hop is not found. Nevertheless, messages usually get relatively close to the destination node, and, in most cases, only a small number of additional hops (if any) are performed.

\section {Uniform distribution of neighborhood set nodes}
\label{sec:uniformNSDistribution}

The neighborhood set should provide possibility to route messages regardless of the direction in which the destination node is located. Therefore, it is important that nodes in neighborhood sets be uniformly distributed in terms of directions. There might be a scenario where some nodes would have more closest neighbors in one direction and no or very few neighbors in other directions. In such a case, the nodes would not be able to route messages in all directions (using neighborhood sets). The issue becomes more important in the presence of node failures, when the number of matching next hops should be as large as possible. In a multi-dimensional space, it is not trivial to ensure uniform distribution of the closest neighbors set, ensuring that certain subset of these nodes would match any potential direction (message destination). Uniform distribution of nodes in terms of directions may be defined in a variety of ways, and many different algorithms may be employed to maximize this uniformity. Both, proximity and even distribution, should be considered, because ensuring uniform distribution of nodes in respect of directions may cause some more distant nodes to be included in neighborhood sets and pass over some closer nodes. The key is to maintain the good properties of neighborhood sets (as being the closest existing nodes), and to make these good properties valid regardless of the direction.

\emph{HyCube} adopts a simple technique for ensuring uniform distribution of neighborhood set nodes in respect of directions. The technique splits the space into fragments (orthants\footnote{An \emph{orthant} is the generalization (in $d$-dimensional Euclidean space) of a quadrant (2-dimensional space)} of the system of coordinates with the center at the address of the node whose neighborhood set is considered) and attempts to ensure that in each orthant, the number of nodes is the same. Within individual orthants, nodes are chosen based on their distances. Such a solution is very simple, efficient and does not require much computational overhead.

\section{Hypercube-aware next hop selection}

When the routing algorithm cannot find a node sharing a longer ID prefix with the destination node than with the current node, a node sharing the same long prefix, but closer to the destination according to the chosen metric is selected as the next hop. When the message is routed within a hypercube corresponding to the common prefix length according to the distance only, it is completely independent of the hypercube hierarchy at lower levels. It is however possible to introduce one more criterion in next hop selection. If the message was routed to a node sharing the largest possible number of common bits in the first different digit (group of $d$ bits), this would be the node in the closest possible hypercube at a lower level. If there were more than one such nodes, the next hop would be then chosen by the remaining distance (among the nodes with the same number of common bits in the first different digit). Such an approach increases the probability of finding the next hop sharing longer prefix (in terms of entire bit groups) by the next node(s) on the path - with the support of secondary routing tables.

\section{Routing table slots overlapping}
\label{sec:rtOverlapping}

Hypercubes corresponding to slots of primary and secondary routing tables may contain nodes that are covered by a slot at a lower level in the secondary routing table. If multiple different routing table slots contain the same node, when the node fails, all these slots become unusable. That is why, ideally, the routing tables should contain references to different nodes for all such overlapping hypercubes.

To overcome the overlapping problem, for the secondary routing table, it is enough to consider a node as a candidate for a matching routing table slot only if the corresponding adjacent hypercube is the lowest-level hypercube containing that node (for individual dimensions and directions). That would ensure that no secondary routing table slot would contain nodes that are located in overlapping lower-level adjacent hypercubes.

As far as the primary routing table slots are concerned, it may also be easily verified whether a certain node is covered by a lower-level secondary routing table slot (let us denote by $Y$ its identifier, and by $X$ the ID of the node whose routing table is being considered). $X$ and $Y$ are in adjacent hypercubes at levels $l - 1 - j$ to $l - 1 - i$ if and only if all $k<i$ first digits ($d$-bit groups) of $X$ and $Y$ are equal, and $i$-th to $j$-th digits differ on one (the same for all these digits) bit - this bit corresponds to the dimension in which the two hypercubes are adjacent. $l$ is the number of levels, and $d$ is the number of dimensions of the hierarchical hypercube. If $l - 1 - j$ is smaller than the corresponding primary routing table level of $Y$, $l_{RT1}$, that means that $Y$ is located in an adjacent hypercube at a lower level than $l_{RT1}$ and is thus covered by a lower-level secondary routing table slot. There might also be a case that $Y$ is not in an adjacent hypercube in any dimension at any level - if the first different digit ($d$-bit group) of $X$ and $Y$ differs on more than one bit.

This modification, however, should not be applied to the neighborhood set, because the neighborhood set is crucial for maintaining high resilience and should always contain the closest nodes (as discussed in the previous sections). Thus, the neighborhood set might contain some nodes that are also included in routing tables. Furthermore, when considering a candidate for routing tables, no check whether the node is in the neighborhood set should be performed. The neighborhood set is continuously updated, and such a check might very soon be ``out-of-date'', while still having left the routing table slot empty. Therefore, due to its properties, the neighborhood set should be built independently from the routing tables.

% ex: set tabstop=4 shiftwidth=4 softtabstop=4 noexpandtab fileformat=unix filetype=tex encoding=utf-8 fileencodings= fenc= spelllang=pl,en spell:


\subsection{Performance metrics}

    Most of our algorithms are supervised and multiclass. The most suitable performance metric is the test accuracy, basically the ratio between hits and tries. We also present some test intraclass metrics, such as class precision (the fraction of BP $k$ predicted documents that are correct, for some BP $k$) and class recall (the fraction of correct BP $k$ documents that were predicted, for some BP $k$). In general, we also present a test confusion matrix, which shows how many test documents were predicted in each class.


\subsection{Measurement functions in \tuner}
\label{sec:oracles}
This section describes our implementation of the measurement oracles used by \tuner: \textproc{CurvatureRange}, \textproc{Variance}, and \textproc{Distance}.
We design the measurement functions with the assumption of a negative log-probability objective; this is in line with typical losses in machine learning, e.g. cross-entropy for neural nets and maximum likelihood estimation in general.
Under this assumption, the Fisher information matrix---i.e.\ the expected outer product of noisy gradients---approximates the Hessian of the objective~\citep{johnfisherinfo2016,pascanu2013revisiting}. This allows for measurements purely being approximated from minibatch gradients with overhead linear to model dimensionality.
These implementations are not guaranteed to give accurate measurements.
Nonetheless, their use in our experiments in Section~\ref{sec:experiments} shows that they are sufficient for \tuner to outperform the state of the art on a variety of objectives. We also refer to Appendix~\ref{sec:practical_impl} for details on zero-debias~\citep{kingma2014adam}, slow start~\citep{schaul2013no} and smoothing for curvature range estimation.

%\begin{minipage}{0.37\textwidth}
%\algrenewcommand\alglinenumber[1]{\scriptsize #1:}
%	\begin{algorithm}[H]
%	\scriptsize
%	\caption{\small Curvature range}
%	\begin{algorithmic}
%		\State \textbf{state: } $h_{\max}$, $h_{\min}$, $h_i, \forall i \in\{1,2,3,...\}$
%		\Function{CurvatureRange}{gradient $g_t$, $\beta$}
%			\State $h_t \gets \| g_t \|^2$
%			\State $h_{\max,t}\!\!\gets\!\!\!\!\!\!\!\max\limits_{t - w \leq i \leq t}\!h_i$, $h_{\min,t}\!\!\gets\!\!\!\!\!\!\!\min\limits_{t - w \leq i \leq t}\!h_i$
%			\State $h_{\max} \gets \beta \cdot h_{\max} + (1 - \beta) \cdot h_{\max,t}$ %\hfill Smoothed largest curvature.
%			\State $h_{\min} \gets \beta \cdot h_{\min} + (1 - \beta) \cdot h_{\min,t}$ %\hfill Smoothed smallest curvature.
%			\Return $h_{\max}$, $h_{\min}$
%		\EndFunction
%	\end{algorithmic}
%	\label{alg:curv_func}
%	\end{algorithm}
%\end{minipage}
%\begin{minipage}{0.315\textwidth}
%\algrenewcommand\alglinenumber[1]{\scriptsize #1:}
%	\begin{algorithm}[H]
%	\scriptsize
%	\setstretch{1.5}
%	\caption{\small Gradient variance}
%	\begin{algorithmic}
%	\State \textbf{state: } $\overline{g^2}\gets0$, $\overline{g}\gets0$
%	\Function{Variance}{gradient $g_t$, $\beta$}
%		\State $\overline{g^2}\gets\beta \cdot \overline{g^2} + (1 - \beta) \cdot g_t \odot g_t$
%		\State $\overline{g}\gets\beta \cdot \overline{g} + (1 - \beta) \cdot g_t$
%		\Return $\| \overline{g^2} - \overline{g}^2 \|_1$ %\hfill Sum of elements in the vect
%	\EndFunction
%	\end{algorithmic}
%	\label{alg:var_func}
%	\end{algorithm}
%\end{minipage}
%\begin{minipage}{0.3\textwidth}
%\algrenewcommand\alglinenumber[1]{\scriptsize #1:}
%	\begin{algorithm}[H]
%	\scriptsize
%	\setstretch{1.25}
%	\caption{\small Distance to opt.}
%	\begin{algorithmic}
%	\State \textbf{state: } $\overline{\|g\|}\gets0$, $\overline{h}\gets0$
%		\Function{Distance}{gradient $g_t$, $\beta$}
%		\State $\overline{\|g\|}\gets \beta \cdot \overline{\|g\|} + (1 - \beta) \cdot \|g_t\|$
%		\State $\overline{h} \gets \beta \cdot \overline{h} + (1 - \beta) \cdot \| g_t \|^2$
%		\State $D \gets \beta \cdot D + (1 - \beta) \cdot \overline{\|g\|} /\overline{h}$
%		\Return $D$
%	\EndFunction
%	\end{algorithmic}
%	\label{alg:dist_func}
%	\end{algorithm}
%\end{minipage}

\paragraph{Curvature range}
Let $g_t$ be a noisy gradient, we estimate the curvatures range in Algorithm~\ref{alg:curv_func}. We notice that the outer product $g_tg_t^T$ has an eigenvalue $h_t=\| g_t \|^2$ with eigenvector $g_t$. Thus under our negative log-likelihood assumption, we use $h_t$ to approximate the curvature of Hessian along gradient direction $g_t$. Note here we use empirical Fisher $g_tg_t^T$ instead of Fisher information matrix. Empirical Fisher is typically used in practical natural gradient methods~\citep{martens2014new, roux2008topmoumoute, duchi2011adaptive}. For practically efficient measurement, we use the empirical Fisher as a coarse proxy of Fisher information matrix which approximates the Hessian of the objective.
Specifically in Algorithm~\ref{alg:curv_func}, we maintain $h_{\min}$ and $h_{\max}$ as running averages of extreme curvature $h_{\min, t}$ and $h_{\max, t}$, from a sliding window of width 20\footnote{We use window width 20 across all the models and experiments in our paper. We refer to Section~\ref{sec:experiments} for details on selecting the window width}.
As gradient directions evolve, we estimate curvatures along different directions. Thus $h_{\min}$ and $h_{\max}$ capture the curvature variations.

%\vspace{-0.5em}
\paragraph{Gradient variance}
To estimate the gradient variance in Algorithm~\ref{alg:var_func},
we use running averages $\overline{g}$ and $\overline{g^2}$ to keep track of $g_t$ and $g_t \odot g_t$, the first and second order moment of the gradient.
As $\Var(g_t) = \E{g_t^2} - \E{g_t} \odot \E{g_t}$, we estimate the gradient variance $C$ in \eqref{equ:noisy_min} using $C=\bm{1}^T\!\!\cdot(\overline{g^2} - \overline{g}^2)$. %To get stable estimates, we use $C$, the running average of $C_t$ as the quantity representing gradient variance.

%\vspace{-0.25em}
\paragraph{Distance to optimum}
In Algorithm~\ref{alg:dist_func}, we estimate the distance to the optimum of the local quadratic approximation.
Inspired by the fact that $\| \nabla f(\mat{x}) \| \leq \| \mat{H} \| \| \mat{x} - \mat{x}^{\star}\|$ for a quadratic $f(x)$ with Hessian $\mat{H}$ and minimizer $\mat{x}^{*}$,
we first maintain $\overline{h}$ and $\overline{\|g\|}$ as running averages of curvature $h_t$ and gradient norm $\| g_t \|$. Then the distance is approximated using $\overline{\|g\|} / \overline{h}$. %according to inequality $\| \nabla f(\mat{x}) \| \leq \| \mat{H} \| \| \mat{x} - \mat{x}^{\star}\|$.

%\begin{minipage}{0.37\textwidth}
%\algrenewcommand\alglinenumber[1]{\scriptsize #1:}
%	\begin{algorithm}[H]
%	\scriptsize
%	\caption{\small Curvature range}
%	\begin{algorithmic}
%		\State \textbf{state: } $h_{\max}$, $h_{\min}$, $h_i, \forall i \in\{1,2,3,...\}$
%		\Function{CurvatureRange}{gradient $g_t$, $\beta$}
%			\State $h_t \gets \| g_t \|^2$
%			\State $h_{\max,t}\!\!\gets\!\!\!\!\!\!\!\max\limits_{t - w \leq i \leq t}\!h_i$, $h_{\min,t}\!\!\gets\!\!\!\!\!\!\!\min\limits_{t - w \leq i \leq t}\!h_i$
%			\State $h_{\max} \gets \beta \cdot h_{\max} + (1 - \beta) \cdot h_{\max,t}$ %\hfill Smoothed largest curvature.
%			\State $h_{\min} \gets \beta \cdot h_{\min} + (1 - \beta) \cdot h_{\min,t}$ %\hfill Smoothed smallest curvature.
%			\Return $h_{\max}$, $h_{\min}$
%		\EndFunction
%	\end{algorithmic}
%	\label{alg:curv_func}
%	\end{algorithm}
%\end{minipage}
%\begin{minipage}{0.315\textwidth}
%\algrenewcommand\alglinenumber[1]{\scriptsize #1:}
%	\begin{algorithm}[H]
%	\scriptsize
%	\setstretch{1.5}
%	\caption{\small Gradient variance}
%	\begin{algorithmic}
%	\State \textbf{state: } $\overline{g^2}\gets0$, $\overline{g}\gets0$
%	\Function{Variance}{gradient $g_t$, $\beta$}
%		\State $\overline{g^2}\gets\beta \cdot \overline{g^2} + (1 - \beta) \cdot g_t \odot g_t$
%		\State $\overline{g}\gets\beta \cdot \overline{g} + (1 - \beta) \cdot g_t$
%		\Return $\| \overline{g^2} - \overline{g}^2 \|_1$ %\hfill Sum of elements in the vect
%	\EndFunction
%	\end{algorithmic}
%	\label{alg:var_func}
%	\end{algorithm}
%\end{minipage}
%\begin{minipage}{0.3\textwidth}
%\algrenewcommand\alglinenumber[1]{\scriptsize #1:}
%	\begin{algorithm}[H]
%	\scriptsize
%	\setstretch{1.25}
%	\caption{\small Distance to opt.}
%	\begin{algorithmic}
%	\State \textbf{state: } $\overline{\|g\|}\gets0$, $\overline{h}\gets0$
%		\Function{Distance}{gradient $g_t$, $\beta$}
%		\State $\overline{\|g\|}\gets \beta \cdot \overline{\|g\|} + (1 - \beta) \cdot \|g_t\|$
%		\State $\overline{h} \gets \beta \cdot \overline{h} + (1 - \beta) \cdot \| g_t \|^2$
%		\State $D \gets \beta \cdot D + (1 - \beta) \cdot \overline{\|g\|} /\overline{h}$
%		\Return $D$
%	\EndFunction
%	\end{algorithmic}
%	\label{alg:dist_func}
%	\end{algorithm}
%\end{minipage}


%% LyX 2.3.5.2 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[conference]{IEEEtran}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{fontspec}
\usepackage{float}
\usepackage{graphicx}

\makeatletter

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
%% Because html converters don't know tabularnewline
\providecommand{\tabularnewline}{\\}
%% A simple dot to overcome graphicx limitations
\newcommand{\lyxdot}{.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsfonts}\usepackage{algorithmic}
\usepackage{textcomp}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}

\makeatother

\begin{document}
\title{ELL409 Assignment 1}
\author{\IEEEauthorblockN{Harman Singh} \and \IEEEauthorblockN{Aayush Srivastava} }
\maketitle
\begin{abstract}
Solutions for Assignment 3 of ELL409.
\end{abstract}

\section{Q1 SVM on health data}

We first ran PCA on the health data which has 3 features and ran SVM
to classify into the 2 classes.Then to make the decision boundaries,
we changed the problem as suggested on piazza and took the first 2
principal components of the data to. We fit SVM's with multiple kernels
and plot the decision boundaries. We also implement the SMO algorithm
and compare its performance, training time with libSVM.

\subsection{Running SVM with different Kernels}

\subsection{Decision Boundaries}

Decision boundaries for all kernels (Linear, RBF, polynomia with different
degrees) is shown in Figure \ref{fig:Question1_fig1}. These figures
correspond to the optimal values of 'C' and gamma (in case of RBF)
kernel. The yellow dots signify that they are support vectors.

\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.9\columnwidth]{images/Q1\lyxdot 3supportvecs_decision}
\par\end{centering}
\caption{Decision boundary for various kernels \label{fig:Question1_fig1}}
\end{figure}

\subsection{Decision Boundaries with removed support vectors}

We removed the old support vectors and then ran the SVM again to fit
it to the remaining dataset. Our decision boundaries are as shown
in Figure \ref{fig:Question1_fig2}. The yellow marked points are
the new support vectors. As we can see these are much smaller in number.

\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.9\columnwidth]{images/Q1\lyxdot 3without_supportvecs_decision}
\par\end{centering}
\caption{Decision boundary for various kernels when old support vectors are
removed \label{fig:Question1_fig2}}
\end{figure}

\subsection{Implementing SMO algorithm (bonus part)}

We referred to Platt's original paper, P.S.Sastri's tutorial and CS231
notes and implemented the SMO algorithm in python itself. The performance
of our code was comparable to the sklearn's LIBSVM implementation
in terms of accuracy but time taken by our code was larger.
\begin{itemize}
\item The time taken by our code was more than that of the LIBSVM as shown
in Figure \ref{fig:Question1_fig3} a) and it continues to increase
as the data scales.
\item The accuracy of LIBSVM and our implementation of SMO isplotted and
we can see that for a large enough dataset they are nearly equal to
each other (Figure \ref{fig:Question1_fig3} b))
\end{itemize}
We feel that ours is not the most optimal implementation of the algorithm
and many more techniques can be used for eg in choosing the 2 vectors
at each step and in other parts of the code. We also feel that LIBSVM
may have been implemented in C/C++ which makes it much faster than
some parts of our code in Python (like loops etc) which cannot be
vectorized using numpy arrays and hence have to suffer the slow-ness
of python.

\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.45\columnwidth]{images/Q1\lyxdot 4timecomparison}\includegraphics[width=0.45\columnwidth]{images/Q1\lyxdot 4accuracycomparison}
\par\end{centering}
\caption{a)Time taken SMO implementation (smotime) vs Sklearn's LIBSVM (svmtime)
and b) accuracy of SMO implemenation (smoacc) and Sklearn's LIBSVM
(svmacc) \label{fig:Question1_fig3}}
\end{figure}

\section{Neural Networks}

\subsection{Ordinary Least Squares}

\section{Learning++}

Part A to E correspond to the 5 subquestions of Q3

\subsection{Class Imbalance in MNIST}

MNIST class 0, 1, 2 were considered in the ratio 70:25:5. We first
used a comparitively larger self made (non standard) neural network
which gave close to 100\%(>99.8 on both train and test datasets) accuracy.
This neural network was large for the MNIST dataset and was able to
generalize well despite the class imbalance.

We then used a smaller feedforward neural network with 2 hidden layers
to first visualize the problems that arose due to class imbalance
and then try to solve them. We first tried out the following methods
and modifications to our losses.
\begin{itemize}
\item Cross entropy loss: We did not modify the loss in this case and the
results we obtained are shown in the folloing table having precision
recall and F1 score for all three classes. (Table \ref{tab3.1})
\end{itemize}
\begin{table}[H]
\begin{centering}
\begin{tabular}{c|c|c|c}
\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.955 & 0.998 & 0.976\tabularnewline
\hline
1 & 0.955 & 0.995 & 0.975\tabularnewline
\hline
2 & \textbf{0.995} & \textbf{0.908} & \textbf{0.949}\tabularnewline
\hline
Macro F1 &  &  & \textbf{0.967}\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\centering{}\caption{Simple Cross entropy loss \label{tab3.1}}
\end{table}

Missclassified samples are as follows, these are randomly picked misclassified
samples and as we can see most of them are from the class 2 which
has the least samples (Figure \ref{fig3.1})

\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.9\columnwidth]{images/Fig3\lyxdot 1Missclassifiedsamples}
\par\end{centering}
\caption{Missclassified samples, random. T=True, P =Predicted\label{fig3.1}}
\end{figure}

\begin{itemize}
\item Weighted Cross entropy loss: We gave more wight to the class with
fewer sample. The weights are inversely proportional to the ratio
of classes. We get an increase in the performance metrics, particularly
the F1 score of the class 1 and 2 which have less samples. results
are as follows.(Table \ref{tab3.2})
\end{itemize}
\begin{table}[H]
\begin{centering}
\begin{tabular}{c|c|c|c}
\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.976 & 0.996 & 0.985\tabularnewline
\hline
1 & 0.977 & 0.997 & 0.986\tabularnewline
\hline
2 & \textbf{0.992} & \textbf{0.951} & \textbf{0.972}\tabularnewline
\hline
Macro F1 &  &  & \textbf{0.981}\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\centering{}\caption{Weighted cross entropy loss stats \label{tab3.2}}
\end{table}

\begin{itemize}
\item Focal loss: We experiemnted with the focal loss and obtained an increase
in performance metrics, when compared with the vanilla cross entropy
loss.(Table \ref{tab3.3})
\end{itemize}
\begin{table}[H]
\begin{centering}
\begin{tabular}{c|c|c|c}
\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.977 & 0.995 & 0.987\tabularnewline
\hline
1 & 0.981 & 0.997 & 0.988\tabularnewline
\hline
2 & \textbf{0.993} & \textbf{0.965} & \textbf{0.980}\tabularnewline
\hline
Macro F1 &  &  & \textbf{0.985}\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\centering{}\caption{Focal loss stats \label{tab3.3}}
\end{table}

\begin{itemize}
\item MSE loss: Simple MSE loss was tried out, performance is much worse
than cross entropy loss.(Table \ref{tab3.4})
\end{itemize}
\begin{table}[H]
\begin{centering}
\begin{tabular}{c|c|c|c}
\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.863 & 0.997 & 0.925\tabularnewline
\hline
1 & 0.853 & 0985 & 0.914\tabularnewline
\hline
2 & \textbf{0.996} & \textbf{0.685} & \textbf{0.811}\tabularnewline
\hline
Macro F1 &  &  & \textbf{0.8894}\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\centering{}\caption{Simple MSE loss stats \label{tab3.4}}
\end{table}

\begin{itemize}
\item Weighted MSE: Similar to Weghted Crossentropy but with this time using
MSE.(Table \ref{tab3.5})
\end{itemize}
\begin{table}[H]
\begin{centering}
\begin{tabular}{c|c|c|c}
\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.881 & 0.998 & 0.936\tabularnewline
\hline
1 & 0.862 & 0.988 & 0.926\tabularnewline
\hline
2 & \textbf{0.994} & \textbf{0.718} & \textbf{0.833}\tabularnewline
\hline
Macro F1 &  &  & \textbf{0.896}\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\centering{}\caption{Weighted MSE loss stats \label{tab3.5}}
\end{table}

\begin{itemize}
\item Focal loss type weighted MSE: We took the ideas of focal loss and
used it in MSE to increase the performance of the classification.(Table
\ref{tab3.6})
\end{itemize}
\begin{table}[H]
\begin{centering}
\begin{tabular}{c|c|c|c}
\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.872 & 0.998 & 0.930\tabularnewline
\hline
1 & 0.859 & 0.986 & 0.918\tabularnewline
\hline
2 & \textbf{9.993} & \textbf{0.703} & \textbf{0.823}\tabularnewline
\hline
Macro F1 &  &  & \textbf{0.890}\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\centering{}\caption{Focal loss MSE stats\label{tab3.6}}
\end{table}

\subsection{KMeans on SVHN + classification}

\subsection{Class Imbalance in MNIST}

\subsection{Class Imbalance in MNIST}

\subsection{2D PCA, tSNE on SVHN dataset}
\begin{itemize}
\item The result of performing PCA on SVHN is as shown in Figure \ref{figQ3.5.1}.
\end{itemize}
\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.9\columnwidth]{images/SVHN_PCA.jpeg}
\par\end{centering}
\caption{SVHN 2D PCA \label{figQ3.5.1}}
\end{figure}

\begin{itemize}
\item The result of performing tSNE on SVHN is as shown in Figure \ref{figQ3.5.2}.
tSNE was performed on a subset (10k images) of data which took considerable
hours on HPC.
\end{itemize}
\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.9\columnwidth]{images/SVHN_tSNE.jpeg}
\par\end{centering}
\caption{SVHN 2D tSNE \label{figQ3.5.2}}
\end{figure}

\begin{itemize}
\item To have a comparison, we went ahead and performed PCA, tSNE on MNIST
data as well (Figure \ref{fig:Q3.5.3}) and based on all this we present
our observations and colnclusions below.
\end{itemize}
\begin{figure}[H]
\begin{centering}
\includegraphics[width=0.45\columnwidth]{images/mnist_PCA.jpeg}\includegraphics[width=0.45\columnwidth]{images/mnist_tSNE.jpeg}
\par\end{centering}
\caption{a)2D PCA and b)2D tSNE MNIST \label{fig:Q3.5.3}}
\end{figure}

\subsubsection{Observations and Conclusions}
\begin{itemize}
\item The PCA result on SVHN is very homogeneous, and the classes are not
separated at all. We can see that all the class labels are placed
at the same point in the figure, ie teh centroid of all the classes
is the same and hence wPCA hasn't been able to separate the classes
very well in 2 dimesnions.
\item tSNE on the other hand isn't alot better but we can ee more clustering
than PCA and the centroids of the classes are also displaced from
each other if we look closely in the tSNE plot. tSNE being a non linear
technique, it is expected that it would perform better than PCA, however
in a lot more amount of time due to (O(N\textasciicircum 2) complexity)
\item Comparing the results with MNIST we get class separationg using PCA
on MNIST. Classes 7, 9, 4 and classes 6 ,5, 8, 2 are close to each
other which is expected due to the struture of the digits. tSNE is
able to completely separate all classes from each other as shown in
the tSNE plots. An interesting thing to note is that the digit 4 is
separated into 2 groups and one is closer to 9, this is expected because
some digits, 4 look very cose to 9.
\item We can conclude that due to the noisse, background, overlapping digits
and classes, and being in more natural setting, SVHN is difficult
to separate using PCA,tSNE but MNIST is highly separable due to the
nature of the dataset, having contrast between digits and background,
uniform background, no overlapping of labels, classes etc
\end{itemize}

\subsection{smdlsk}

n param classifier, KNN, is shown here and the other one's in the
Appendix.

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.996 & 0.996 & 0.996\tabularnewline
\hline
1 & 0.995 & 0.985 & 0.99\tabularnewline
\hline
2 & 0.994 & 0.999 & 0.996\tabularnewline
\hline
3 & 0.97 & 0.978 & 0.974\tabularnewline
\hline
4 & 0.967 & 0.932 & 0.949\tabularnewline
\hline
5 & 0.953 & 0.984 & 0.969\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.996 & 0.0 & 0.004 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.0 & 0.985 & 0.0 & 0.0 & 0.001 & 0.015\tabularnewline
\hline
\textbf{2} & 0.001 & 0.0 & 0.999 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{3} & 0.001 & 0.0 & 0.002 & 0.978 & 0.019 & 0.0\tabularnewline
\hline
\textbf{4} & 0.002 & 0.0 & 0.0 & 0.03 & 0.932 & 0.035\tabularnewline
\hline
\textbf{5} & 0.0 & 0.004 & 0.0 & 0.0 & 0.012 & 0.984\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{KNN on Multi-Class Dataset}
\end{table}

\appendix

\section{A}

\subsection{Binary Classification - Question 1 : Health Dataset Visualization}

\begin{figure}[H]
\begin{centering}
\includegraphics[width=7cm,height=7cm,keepaspectratio,bb = 0 0 200 100, draft, type=eps]{C:/Users/shubhammittal/Desktop/assignment/Q1_Binary_Classifier/health dataset.png}\caption{Health Data\label{fig:Health-Data}}
\par\end{centering}
\end{figure}

\subsection{Binary Classification - Question 1: Logistic Regression - Stochastic
Gradient Descent}

\begin{table}[H]

\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{MAE} & \textbf{MSE} & \textbf{CrossEntropy}\tabularnewline
\hline
Val Acc & 0.8286 & 0.8469 & 0.8469\tabularnewline
\hline
Val F1 & 0.8278 & 0.8179 & 0.8165\tabularnewline
\hline
Train Acc & 0.851 & 0.851 & 0.851\tabularnewline
\hline
Test Acc & 0.8714 & 0.8714 & 0.8714\tabularnewline
\hline
Precision & 0.8571 & 0.8571 & 0.8571\tabularnewline
\hline
Recall & 0.8478 & 0.8478 & 0.8478\tabularnewline
\hline
F1 & 0.8525 & 0.8525 & 0.8525\tabularnewline
\end{tabular}
\par\end{centering}
\caption{Logistic Regression Classifiers with different loss functions (SGD)
\label{tab:Logistic-Regression-Classifiers-1}}

\end{table}

\subsection{Regression problem - Question 2: Elastic Net Regularization}

Here, it is seen that $\alpha$, coefficient of L2 penalty penalizes
the training loss more than $\beta$, coefficient of L1 penalty.

\begin{figure}[H]
\begin{centering}
\includegraphics[width=6cm,height=6cm,keepaspectratio,bb = 0 0 200 100, draft, type=eps]{C:/Users/shubhammittal/Desktop/assignment/Q2_Regression/ElasticNet.png}\caption{Elastic Net Regularization\label{fig:Elastic-Net-Regularization}}
\par\end{centering}
\end{figure}

\subsection{Regression problem - Question 2: Linear regression using MAE Loss
function}

\begin{figure}[H]
\begin{centering}
\includegraphics[width=5cm,height=5cm,keepaspectratio,bb = 0 0 200 100, draft, type=eps]{C:/Users/shubhammittal/Desktop/assignment/Q2_Regression/MAE_regressor.png}\includegraphics[width=5cm,height=5cm,keepaspectratio,bb = 0 0 200 100, draft, type=eps]{C:/Users/shubhammittal/Desktop/assignment/Q2_Regression/MSE_regressor.png}\caption{Linear regression comparison between MAE and MSE\label{fig:Linear-regression-comparison}}
\par\end{centering}
\end{figure}

\subsection{Question 3: MLE - Bayes Gaussian}

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
\textbf{0} & 0.997 & 0.986 & 0.992\tabularnewline
\hline
\textbf{1} & 0.947 & 0.985 & 0.966\tabularnewline
\hline
\textbf{2} & 0.986 & 0.998 & 0.992\tabularnewline
\hline
\textbf{3} & 0.962 & 0.984 & 0.973\tabularnewline
\hline
\textbf{4} & 0.974 & 0.922 & 0.947\tabularnewline
\hline
\textbf{5} & 0.95 & 0.942 & 0.946\tabularnewline
\hline
\end{tabular}\\
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.986 & 0.0 & 0.014 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.0 & 0.985 & 0.0 & 0.0 & 0.0 & 0.015\tabularnewline
\hline
\textbf{2} & 0.0 & 0.0 & 0.998 & 0.0 & 0.002 & 0.0\tabularnewline
\hline
\textbf{3} & 0.001 & 0.0 & 0.0 & 0.984 & 0.014 & 0.0\tabularnewline
\hline
\textbf{4} & 0.002 & 0.0 & 0.0 & 0.039 & 0.922 & 0.037\tabularnewline
\hline
\textbf{5} & 0.0 & 0.05 & 0.0 & 0.0 & 0.008 & 0.942\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{}

\end{table}

\subsection{Question 3: MLE - Naive Bayes Gaussian}

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
\textbf{0} & 0.997 & 0.986 & 0.992\tabularnewline
\hline
\textbf{1} & 0.937 & 0.996 & 0.966\tabularnewline
\hline
\textbf{2} & 0.986 & 0.998 & 0.992\tabularnewline
\hline
\textbf{3} & 0.941 & 0.981 & 0.96\tabularnewline
\hline
\textbf{4} & 0.958 & 0.906 & 0.931\tabularnewline
\hline
\textbf{5} & 0.964 & 0.918 & 0.941\tabularnewline
\hline
\end{tabular}\\
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.986 & 0.0 & 0.014 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.0 & 0.996 & 0.0 & 0.0 & 0.0 & 0.004\tabularnewline
\hline
\textbf{2} & 0.0 & 0.0 & 0.998 & 0.002 & 0.0 & 0.0\tabularnewline
\hline
\textbf{3} & 0.001 & 0.0 & 0.0 & 0.981 & 0.018 & 0.0\tabularnewline
\hline
\textbf{4} & 0.002 & 0.0 & 0.0 & 0.06 & 0.906 & 0.031\tabularnewline
\hline
\textbf{5} & 0.0 & 0.06 & 0.0 & 0.0 & 0.022 & 0.918\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{}
\end{table}

\subsection{Question 3: MLE - Bayes GMM}

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
\textbf{0} & 0.987 & 0.983 & 0.992\tabularnewline
\hline
\textbf{1} & 0.946 & 0.979 & 0.923\tabularnewline
\hline
\textbf{2} & 0.956 & 0.998 & 0.912\tabularnewline
\hline
\textbf{3} & 0.912 & 0.980 & 0.983\tabularnewline
\hline
\textbf{4} & 0.984 & 0.921 & 0.951\tabularnewline
\hline
\textbf{5} & 0.951 & 0.932 & 0.956\tabularnewline
\hline
\end{tabular}\\
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.936 & 0.0 & 0.064 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.01 & 0.945 & 0.0 & 0.0 & 0.0 & 0.045\tabularnewline
\hline
\textbf{2} & 0.0 & 0.0 & 0.993 & 0.0 & 0.007 & 0.0\tabularnewline
\hline
\textbf{3} & 0.011 & 0.0 & 0.0 & 0.985 & 0.004 & 0.0\tabularnewline
\hline
\textbf{4} & 0.002 & 0.0 & 0.0 & 0.041 & 0.920 & 0.037\tabularnewline
\hline
\textbf{5} & 0.0 & 0.03 & 0.0 & 0.0 & 0.008 & 0.962\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{}
\end{table}

\subsection{Question 3: Logistic Regression over MAE Loss}

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
\textbf{0} & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.57 & 1.0 & 0.726\tabularnewline
\hline
\textbf{2} & 0.495 & 1.0 & 0.663\tabularnewline
\hline
\textbf{3} & 0.936 & 0.984 & 0.96\tabularnewline
\hline
\textbf{4} & 0.724 & 0.908 & 0.806\tabularnewline
\hline
\textbf{5} & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\end{tabular}\\
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{2} & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{3} & 0.0 & 0.0 & 0.006 & 0.984 & 0.009 & 0.0\tabularnewline
\hline
\textbf{4} & 0.0 & 0.013 & 0.012 & 0.067 & 0.908 & 0.0\tabularnewline
\hline
\textbf{5} & 0.0 & 0.664 & 0.0 & 0.0 & 0.336 & 0.0\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{}
\end{table}

\subsection{Question 3: Logistic Regression over MSE Loss}

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
\textbf{0} & 0.779 & 0.026 & 0.051\tabularnewline
\hline
\textbf{1} & 0.782 & 1.0 & 0.878\tabularnewline
\hline
\textbf{2} & 0.503 & 1.0 & 0.67\tabularnewline
\hline
\textbf{3} & 0.94 & 0.983 & 0.961\tabularnewline
\hline
\textbf{4} & 0.843 & 0.896 & 0.869\tabularnewline
\hline
\textbf{5} & 0.965 & 0.6 & 0.74\tabularnewline
\hline
\end{tabular}\\
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.026 & 0.0 & 0.974 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{2} & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{3} & 0.0 & 0.0 & 0.007 & 0.983 & 0.008 & 0.002\tabularnewline
\hline
\textbf{4} & 0.008 & 0.007 & 0.006 & 0.062 & 0.896 & 0.02\tabularnewline
\hline
\textbf{5} & 0.0 & 0.242 & 0.0 & 0.0 & 0.158 & 0.6\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{}
\end{table}

\subsection{Question 3 : Parzen Windows}

\begin{table}[H]
\begin{centering}
\begin{tabular}{l|l|l|l}
 & \textbf{Precision} & \textbf{Recall} & \textbf{F1}\tabularnewline
\hline
0 & 0.982 & 0.99 & 0.986\tabularnewline
\hline
1 & 0.999 & 0.974 & 0.987\tabularnewline
\hline
2 & 0.99 & 1.0 & 0.995\tabularnewline
\hline
3 & 0.967 & 0.981 & 0.974\tabularnewline
\hline
4 & 0.978 & 0.92 & 0.948\tabularnewline
\hline
5 & 0.946 & 0.991 & 0.968\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\begin{centering}
\begin{tabular}{l|l|l|l|l|l|l}
\hline
 & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5}\tabularnewline
\hline
\textbf{0} & 0.99 & 0.0 & 0.01 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{1} & 0.0 & 0.974 & 0.0 & 0.0 & 0.001 & 0.025\tabularnewline
\hline
\textbf{2} & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0\tabularnewline
\hline
\textbf{3} & 0.006 & 0.0 & 0.0 & 0.981 & 0.012 & 0.0\tabularnewline
\hline
\textbf{4} & 0.012 & 0.0 & 0.0 & 0.034 & 0.92 & 0.034\tabularnewline
\hline
\textbf{5} & 0.0 & 0.0 & 0.0 & 0.0 & 0.008 & 0.991\tabularnewline
\hline
\end{tabular}
\par\end{centering}
\caption{Parzen Windows on Multi-Class Dataset}
\end{table}

\end{document}


\section{Algorithmic Typing: \declang}

Soundness of \declang trivially reduces to soundness of implication checking.
Here we give the detailed proof of the Approximation Theorem:
\begin{theorem}{[Approximation]}\label{thm:approximation}
  If \decissubref{\Env}{p_1}{p_2} then \isimplied{\Env}{p_1}{p_2}.
\end{theorem}
\begin{proof}
To prove the above, let ${\VC \defeq \VCOND{\Env}{p_1}{p_2}}$. First, note that
if $\VC$ is u-valid then it is valid as the addition of axioms preserves
validity. Next, we prove that if the \VC is valid then \isimplied{\Env}{p_1}{p_2}.
%
We fix a $\theta$ for which
$ \theta \in \interp{\Gamma}$ and
$\evals{\thetasub{\theta}{q_1}}{\etrue}$
%
It suffices to prove that
$\evals{\thetasub{\theta}{q_2}}{\etrue}$.

For all $(x_i, \tlref{x_i}{B}{\trivial}{p_i}) \in \Gamma$
there exists $(x_i, e_i) \in \theta$ and
\begin{align*}
	e_i \in \interp{\tref{x_i}{B}{\trivial}{p_i}}
&\Leftrightarrow
	e_i \in \interp{\tref{v}{B}{}{p_i}}
	\land
	\exists v. \evals{e_i}{v_i}  \\
&\Leftrightarrow
	\exists v. \evals{e_i}{v} \Rightarrow
	\evals{\thetasub{\theta}{p_i\sub{x_i}{v_i}}}{\etrue}
	\land
	\exists v. \evals{e_i}{v_i}  \\
&\Leftrightarrow
	\evals{\thetasub{\theta}{p_i}}{\etrue}\\
\end{align*}

Thus we have that
$\evals{\theta\ (\bigwedge p_i \land q_1)}{\etrue}$.
%
By Lemma~\ref{lemma:teval}
$\tevals{\theta}{(\bigwedge p_i \land q_1)}{\theta'}{\etrue}$.
%
Let $\rho = \mkbot{\theta'}$;
thus, $\trackevals{\theta}{\rho}$.
By Lemma~\ref{lemma:proofs:lifting}
$\evals{\thetasub{\rho}{\bigwedge p_i \land q_1}}{\etrue}$.
%
Moreover, by the construction of $\rho$,
\hastype{\emptyset}{\thetasub{\rho}{\bigwedge p_i \land q_1}}{\tbool}.
Thus, by Equivalence Theorem~\ref{thm:equiv}
$\forall \sigma \in \embed{\rho}.
\lmodels{\sigma}{\bigwedge p_i \land q_1}$.
%
By which and validity of $VC$
$\forall \sigma \in \embed{\rho}.
\lmodels{\sigma}{q_2}$.
%
Using the other direction of Equivalence Theorem~\ref{thm:equiv}
$\evals{\rho\ q_2}{\etrue}$.
%
Finally, using the other direction of Lemma~\ref{lemma:proofs:lifting}
$\eval{\thetasub{\theta}{q_2}}{\etrue}$.
\end{proof}

\renewcommand\botsto{\ensuremath{\theta^\ebot}}
To conclude the proof we prove Equivalence Theorem.
Let \botsto be a substitution from variables to \textit{lifted values}.
We define the embedding of the substitution \embed{\botsto}
that maps \ebot to arbitrary elements of the logical domain:

\begin{definition}
$$
\instance{\rho}=
	\{(x_1, \botv_1), \dots, (x_n, \botv_n) \mid \botv_i \in \instance{\rho(x_i)}  \}
$$
\begin{align*}
\instance{\ebot}& = \dom &
\instance{D\ \overline{v}} &=
	\{ D\ \overline{v} \mid v_i \in \instance{v} \} \\
\instance{n} &= \{n\} &
\instance{v} &= \{c_v\}, \text{otherwise}
\end{align*}
\end{definition}

Then we prove that
given a lifted substitution a predicate goes to \etrue
if and only if for any embedding the
predicate holds.

\begin{theorem}{[Equivalence]}\label{thm:equiv}
If \hastype{\emptyset}{\botsto(p)}{\tbool}, then
\begin{itemize}
\item $\evals{\botsto(p)}{\etrue}\ \mbox{iff}\
	\forall \sigma \in \embed{\botsto}. \lmodels{\sigma}{p}$.
\item $\evals{\botsto(p)}{\efalse}\
	\mbox{iff}\
	\forall \sigma \in \embed{\botsto}. \lmodels{\sigma\not}{p}$.
\end{itemize}
\end{theorem}
\begin{proof}
\newcommand\sigmamodel{\ensuremath{\sigma}}
\newcommand\interpI{\ensuremath{\mathcal{I}}}
\newcommand\interpIEq[3]{\ensuremath{\interpI_{\sigmamodel}(#2) = #3}}
\newcommand\interpIGEq[3]{\ensuremath{\interpI_{\sigmamodel}(#2) \sqsupseteq #3}}
\newcommand\interpINGEq[3]{\ensuremath{\interpI_{\sigmamodel}(#2) \not\sqsupseteq #3}}
\newcommand\interpINEq[3]{\ensuremath{\interpI_{\sigmamodel}(#2) \not = #3}}
\newcommand\interpIENGEq[4]{\ensuremath{\interpI_{\sigmamodel}(#2) = {#3}\not \sqsupseteq #4}}

To begin with we define a comparison between lifted values and
elements of the logical domain:
\begin{definition}
\begin{align*}
\ebot &\sqsubset d & d &\sqsubseteq d & v &\sqsubseteq c_v
\end{align*}
\end{definition}

and a function $\interpI_\sigmamodel (t)$
that given a model $\sigmamodel$ and an (open) logical term $t$ returns an element in the logic:

\begin{definition}{[Interpretation]}
$$\interpI_{\sigmamodel} :: t \rightarrow d $$
%
\begin{align*}
\interpIEq{\sigmamodel}{n&}{n} &
\interpIEq{\sigmamodel}{f\ \overline{t}&}{f_D\ (\overline{\interpI_{\sigmamodel}{t}})}\\
& & \interpIEq{\sigmamodel}{D\ \overline{t}&}{D\ \overline{\interpI_{\sigmamodel}{t}}}\\
\interpIEq{\sigmamodel}{x&}{\sigmamodel(x)}& \interpIEq{\sigmamodel}{t_1 \oplus t_2 &}{\interpI_{\sigmamodel}{t_2}\oplus_D \interpI_{\sigmamodel}{t_2}}\\
\end{align*}
\end{definition}

We relate the evaluation of logical terms with their interpretation
into the logic:
%
\begin{lemma}
If \hastype{\Gamma}{\thetasub{\botsto}{t}}{\tau}, then
$	\evals{\thetasub{\botsto}{t}}{\botv}
	\Leftrightarrow
	\forall\sigma\in \instance{\rho}.\interpIGEq{\sigma}{t}{\botv}$
\end{lemma}
\begin{proof}
By induction on the structure of $t$.
\begin{itemize}
\item $t \equiv n$: \evals{\rho\ n}{n} and
	  $\forall\sigmamodel \in \instance{\rho}\interpIEq{\sigmamodel}{n}{n}$
\item $t \equiv x$:
		\evals{\thetasub{\rho}{x}}{\rho(x)} and
		$\forall\sigmamodel \in \instance{\rho}.
		\interpIEq{\sigmamodel}{x}{\sigmamodel(x)} \sqsupseteq \rho(x)$
\item $t \equiv f\ \overline{t}$:

	$$
		\evals{\rho\ (f\ \overline{t})}{\botv} \Leftrightarrow
		\evals{f\ \overline{\rho\ t}}{\botv} \Leftrightarrow
	$$
	$$
		\exists \botv_i. \evals{\thetasub{\rho}{t_i}}{\botv_i} \text{ and } \evals{f ({\overline{\botv}})}{\botv} \Leftrightarrow
	$$
	$$
		\exists\botv_i\forall\sigmamodel \in \instance{\rho}.\interpIGEq{\instance{\rho}}{t_i}{\botv_i} \text{ and }
		\forall d_i\sqsupseteq\botv_i. f_D(\overline{d}) \sqsupseteq \botv \xLeftrightarrow{(*)}
	$$
	$$
		\forall\sigmamodel \in \instance{\rho}.\exists d_i\interpIEq{\sigmamodel}{t_i}{d_i} \text{ and }
		f_D(\overline{d}) \sqsupseteq \botv \Leftrightarrow
	$$
	$$
		\forall\sigmamodel \in \instance{\rho}.
		f_D(\overline{\interpI_{\sigmamodel}(t_i)}) \sqsupseteq \botv \Leftrightarrow
	$$
	$$
		\forall\sigmamodel \in \instance{\rho}.\interpIGEq{\sigmamodel}{f\ \overline{t}}{\botv}
	$$

$(*)$ We can show that for each $f_D$ and $\botv$
	$$
		\exists\botv_i\forall d_i. d_i\sqsupseteq\botv_i \Leftrightarrow f_D(\overline{d}) \sqsupseteq \botv
	$$
ie, $\botv_i$ contains the least information required by $f_D$
to produce a result less than \botv.
Now, say $$ \exists\sigmamodel \in \instance{\rho}\forall d_i.\interpIENGEq{\sigmamodel}{t_i}{d_i}{\botv_i} $$
Then, by definition of $\botv_i$,
$f_D(\overline{d}) \not\sqsupseteq \botv $, which is a contradiction.
\item $t \equiv D\ \overline{t}$:
	$$
		\evals{\rho\ (D\ \overline{t})}{D\ \overline{\botv}} \Leftrightarrow
		\evals{\rho\ t_i}{\botv_i} \Leftrightarrow
	$$
	$$
		\forall\sigmamodel \in \instance{\rho}.\interpIGEq{\sigmamodel}{t_i}{\botv_i} \Leftrightarrow
		\forall\sigmamodel \in \instance{\rho}.\interpIGEq{\sigmamodel}{D\ \overline{t}}{D\ \overline{\botv}}
	$$
\item $t \equiv t_1 \oplus t_2 $
	$$
		\evals{\rho\ (t_1 \oplus t_2)}{d} \Leftrightarrow
		\evals{(\rho\ t_1) \oplus (\rho\ t_2)}{d} \Leftrightarrow
	$$
	$$
		\exists d_1.\evals{\rho\ t_1}{d_1} \text{ and }  \evals{\oplus_{d_1} (\rho\ t_2)}{d} \Leftrightarrow
	$$
	$$
		\exists d_1, d_2.\evals{\rho\ t_1}{d_1} \text{ and }  \evals{\rho\ t_2}{d_2} \text{ and } d_1 \oplus_D d_2 = d \Leftrightarrow
	$$
	$$
		\exists d_1, d_2.\forall\sigmamodel \in \instance{\rho}.\interpIEq{\sigmamodel}{t_1}{d_1} \text{ and }
		\forall\sigmamodel \in \instance{\rho}.\interpIEq{\sigmamodel}{t_2}{d_2} \text{ and } d_1 \oplus_D d_2 = d \xLeftrightarrow{(*)}
	$$
	$$
		\forall\sigmamodel \in \instance{\rho}.
		\exists d_1, d_2.\interpIEq{\sigmamodel}{t_1}{d_1} \text{ and }
		\interpIEq{\sigmamodel}{t_2}{d_2} \text{ and } d_1 \oplus_D d_2 = d \Leftrightarrow
	$$
	$$
		\forall\sigmamodel \in \instance{\rho}.\interpIEq{\sigmamodel}{t_1 \oplus t_2}{d}
	$$
$(*)$ For $i = 1 , 2$, fix two instantiations
$\sigmamodel_1, \sigmamodel_2 \in \instance{\rho}$.
Assume that $d_{i_{\sigmamodel_1}} \not = d_{i_{\sigmamodel_2}}$.
Then $\lnot \forall \sigmamodel \in \instance{\rho} \interpIEq{\sigmamodel}{t_i}{d} \Rightarrow \evals{\rho\ t_i \not}{d} \Rightarrow \lnot \hastype{\Gamma}{t_i}{b^\finite} \Rightarrow \lnot \hastype{\Gamma}{p}{\tbool}$.
\end{itemize}
\end{proof}

We use the above Lemma to prove the Theorem by induction on the structure of $p$.
\begin{itemize}
\item $ p \equiv \etrue$:
\begin{itemize}
\item \eval{\rho\ \etrue}{\etrue} and $\forall\forall\sigmamodel \in  \instance{\rho}. \sigmamodel \models \etrue$
\item \eval{\rho\ \etrue\not}{\efalse} and $\exists \forall\sigmamodel \in  \instance{\rho}. \sigmamodel \models \etrue$
\end{itemize}

\item $ p \equiv \efalse$:
\begin{itemize}
\item $\eval{\rho\ \efalse \not}{\etrue}$ and $\exists \sigmamodel \in \instance{\rho}. \sigmamodel \not \models \efalse$
\item $\eval{\rho\ \efalse}{\efalse}$ and $\forall \sigmamodel \in \instance{\rho}.\sigmamodel \not \models \efalse$
\end{itemize}
\item $ p \equiv \lnot q$:
\begin{itemize}
\item
$
	\evals{\rho\ (\lnot q)}{\etrue} \Leftrightarrow
	\evals{\lnot (\rho\ q)}{\etrue} \Leftrightarrow
	\evals{\rho\ q}{\efalse} 		\Leftrightarrow
	\forall \sigmamodel \in \instance{\rho}. \sigmamodel \not \models q	 \Leftrightarrow
	\forall \sigmamodel \in \instance{\rho}. \sigmamodel \models \lnot q \Leftrightarrow
	\forall \sigmamodel \in \instance{\rho}. \sigmamodel \models p
$
\item
$
	\evals{\rho\ (\lnot q)}{\efalse} \Leftrightarrow
	\evals{\lnot (\rho\ q)}{\efalse} \Leftrightarrow
	\evals{\rho\ q}{\etrue} 		\Leftrightarrow
	\forall \sigmamodel\in\instance{\rho}. \sigmamodel \models q	 \Leftrightarrow
	\forall \sigmamodel\in\instance{\rho}. \sigmamodel \not \models \lnot q \Leftrightarrow
	\forall \sigmamodel\in\instance{\rho}. \sigmamodel \not \models p
$
\end{itemize}

\item $ p \equiv p_1 \land p_2$:
\begin{itemize}
\item
$
	\evals{\rho\ (p_1 \land p_2)}{\etrue} \Leftrightarrow
$
$
	\evals{(\rho\ p_1) \land (\rho\ p_2) }{\etrue} \Leftrightarrow
$
$	\evals{\rho\ p_1}{\etrue} \text{ and } \evals{\rho\ p_2}{\etrue} \Leftrightarrow
$\\
$
	{\forall\sigmamodel\in\instance{\rho}. \sigmamodel\models p_1} \text{ and }
	{\forall\sigmamodel\in\instance{\rho}. \sigmamodel\models p_2} \Leftrightarrow
$\\
$
	{\forall\sigmamodel\in\instance{\rho}. \instance{\rho}\models p_1 \land p_2}  \Leftrightarrow
	\forall\sigmamodel\in\instance{\rho}. \instance{\rho}  \models p
$
\item
$
	\evals{\rho\ (p_1 \land p_2)}{\efalse} \Leftrightarrow
	\evals{(\rho\ p_1) \land (\rho\ p_2) }{\efalse} \Leftrightarrow
	\left\{
	\begin{array}{c}
		\evals{\rho\ p_1}{\efalse} \\
		 \text{OR}\\
		\evals{\rho\ p_2}{\efalse} \\
	\end{array}
	\right.
	\Leftrightarrow
	\left.
	\begin{array}{c}
		{\forall\sigmamodel\in\instance{\rho}. \sigmamodel\not\models p_1} \\
		 \text{OR}\\
		{\forall\sigmamodel\in\instance{\rho}. \sigmamodel\not\models p_2} \\
	\end{array}
	\right\}
$\\
$
	  \Leftrightarrow
	{\forall\sigmamodel\in\instance{\rho}. \sigmamodel\not\models p_1 \land p_2}  \Leftrightarrow
	\forall\sigmamodel\in\instance{\rho}. \sigmamodel \not\models p
$
\end{itemize}

\item $p \equiv t_1 = t_2$:
\begin{itemize}
\item
$
\begin{array}{lclclcl}
	&&\evals{\rho\ (t_1 = t_2)}{\etrue}
	\\&\Leftrightarrow&
	\evals{(\rho\ t_1) = (\rho\ t_2)}{\etrue} &&\\
	&\Leftrightarrow&
	\exists d_1, d_2.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{=_{d_1} (\rho\ t_2)}{\etrue} &\\
	&\Leftrightarrow&
	\exists d_1, d_2.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{\rho\ t_2}{d_2} \\
	&&&\text{and}& d_1 =_D d_2\\
	&\Leftrightarrow&
	\exists d_1, d_2\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 =_D d_2\\
	&\xLeftrightarrow{(*)}&
	\forall\sigmamodel\in\instance{\rho}\exists d_1, d_2.\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 =_D d_2\\
	&\Leftrightarrow&
	\forall\sigmamodel\in \instance{\rho}. \sigmamodel \models t_1 = t_2  &&\\
\end{array}
$

\item
$
\begin{array}{lclclcl}
	&&\evals{\rho\ (t_1 = t_2)}{\efalse} \\&\Leftrightarrow&
	\evals{(\rho\ t_1) = (\rho\ t_2)}{\efalse} &&\\
	&\Leftrightarrow&
	\exists d_1, d_2.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{=_{d_1} (\rho\ t_2)}{\efalse} &\\
	&\Leftrightarrow&
	\exists d_1, d_2.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{\rho\ t_2}{d_2} \\&&&\text{and}& d_1 \not=_D d_2\\
	&\Leftrightarrow&
	\exists d_1, d_2\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 \not =_D d_2\\
	&\xLeftrightarrow{(*)}&
	\forall\sigmamodel\in\instance{\rho}\exists d_1, d_2.\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 \not =_D d_2\\
	&\Leftrightarrow&
	\forall\sigmamodel\in \instance{\rho}. \sigmamodel \not \models t_1 = t_2  &&\\
\end{array}
$
\end{itemize}
$(*)$ For $i = 1 , 2$, fix two instantiations
$\sigmamodel_1, \sigmamodel_2\in\instance{\rho}$.
Assume that $d_{i_{\sigmamodel_1}} \not = d_{i_{\sigmamodel_2}}$.
Then $\lnot \forall \sigmamodel\in\instance{\rho} \interpIEq{\sigmamodel}{t_i}{d} \Rightarrow \evals{\rho\ t_i \not}{d} \Rightarrow \lnot \hastype{\Gamma}{t_i}{b^\finite} \Rightarrow \lnot \hastype{\Gamma}{p}{\tbool}$

\item $p \equiv t_1 < t_2$:
\begin{itemize}
\item
$
\begin{array}{lclclcl}
	&&\evals{\rho\ (t_1 < t_2)}{\etrue} \\&\Leftrightarrow&
	\evals{(\rho\ t_1) < (\rho\ t_2)}{\etrue} &&\\
	&\Leftrightarrow&
	\exists d_1.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{<_{d_1} (\rho\ t_2)}{\etrue} &\\
	&\Leftrightarrow&
	\exists d_1, d_2.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{\rho\ t_2}{d_2} \\&&&\text{and}& d_1 <_D d_2\\
	&\Leftrightarrow&
	\exists d_1, d_2.\forall\instance{\rho}.\interpIEq{\instance{\rho}}{t_1}{d_1} &\text{and}&
	\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_2}{d_2}
	\\&&&\text{and}& d_1 <_D d_2\\
	&\Leftrightarrow&
	\exists d_1, d_2.\forall\sigmamodel\in\instance{\rho}.
	\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 <_D d_2\\
	&\xLeftrightarrow{(*)}&
	\forall\sigmamodel\in\instance{\rho}\exists d_1, d_2.
	\interpIEq{\sigmamodel}{t_1}{d_1}&\text{and}&\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 <_D d_2\\
	&\Leftrightarrow&
	\forall \sigmamodel\in\instance{\rho}. \sigmamodel \models t_1 < t_2  &&\\
\end{array}
$
\item
$
\begin{array}{lclclcl}
	&&\evals{\rho\ (t_1 < t_2)}{\efalse} \\&\Leftrightarrow&
	\evals{(\rho\ t_1) < (\rho\ t_2)}{\efalse} &&\\
	&\Leftrightarrow&
	\exists d_1.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{<_{d_1} (\rho\ t_2)}{\efalse} &\\
	&\Leftrightarrow&
	\exists d_1, d_2.\evals{\rho\ t_1}{d_1} &\text{and}& \evals{\rho\ t_2}{d_2} \\&&&\text{and}& d_1 \not <_D d_2\\
	&\Leftrightarrow&
	\exists d_1, d_2\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 \not <_D d_2\\
	&\Leftrightarrow&
	\exists d_1, d_2\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 \not <_D d_2\\
	&\xLeftrightarrow{(*)}&
	\forall\sigmamodel\in\instance{\rho}\exists d_1, d_2.
	\interpIEq{\sigmamodel}{t_1}{d_1} &\text{and}&
	\interpIEq{\sigmamodel}{t_2}{d_2} \\&&&\text{and}& d_1 \not <_D d_2\\
	&\Leftrightarrow&
	\forall\sigmamodel\in \instance{\rho}. \sigmamodel\not\models t_1 < t_2  &&\\
\end{array}
$
\end{itemize}
$(*)$ For $i = 1 , 2$, fix two instantiations
$\sigmamodel_1, \sigmamodel_2 \in \instance{\rho}$.
Assume that $d_{i_{\sigmamodel_1}} \not = d_{i_{\sigmamodel_2}}$.
Then $\lnot \forall\sigmamodel\in \instance{\rho} \interpIEq{\sigmamodel}{t_i}{d} \Rightarrow \evals{\rho\ t_i \not}{d} \Rightarrow \lnot \hastype{\Gamma}{t_i}{b^\finite} \Rightarrow \lnot \hastype{\Gamma}{p}{\tbool}$

\item $p \equiv t$:
\begin{itemize}
\item
$
\begin{array}{lclclcl}
	\evals{\rho\ t}{\etrue} &\Leftrightarrow&
	\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t}{\etrue}
	\\&\Leftrightarrow&
	\forall\sigmamodel\in \instance{\rho}. \sigmamodel \models t\\
\end{array}
$
\item
$
\begin{array}{lclclcl}
	\evals{\rho\ t}{\efalse} &\Leftrightarrow&
	\forall\sigmamodel\in\instance{\rho}.\interpIEq{\sigmamodel}{t}{\efalse}
	\\&\Leftrightarrow&
	\forall\sigmamodel\in\instance{\rho}. \sigmamodel \not \models t\\
\end{array}
$
\end{itemize}
\end{itemize}
\end{proof}


\documentclass[simplex.tex]{subfiles}
% NO NEED TO INPUT PREAMBLES HERE
% packages are inherited; you can compile this on its own
\begin{document}
\subsection{Multiscale Network Testing for Two-Graph}

We invented multiscale network test via diffusion maps and \texttt{MGC}, and extends its utility into testing two graphs of the same node set with different edge sets. Assume two graphs $\mathbf{G}_{1}$ and $\mathbf{G}_{2}$ are generated via a latent variable $\mathbf{u}_{i} = ( u_{1i}~ u_{2i}~ \cdots ~u_{5i} ) \in \mathbb{R}^{5}$ as follows:
\begin{equation}
\begin{split}
u_{ki} & \overset{i.i.d.}{\sim} Unif(0, 1), \quad i = 1,2, \ldots, n;~k = 1,2,\ldots, 5 \\
w_{i} & := (1- u_{i1} )^{2}, \quad i = 1,2, \ldots, n \\
A^{(1)}_{ij} \big| \mathbf{u}_{i}, \mathbf{u}_{j} & \sim Bernoulli \big( <\mathbf{u}_{i}/5, \mathbf{u}_{j}/5  > \big), \quad \forall i < j;~i,j=1,2,\ldots,n;~\mathbf{u}_{i}, \mathbf{u}_{j} \in \mathbb{R}^{5} \\
A^{(2)}_{ij} \big| w_{i}, w_{j} & \sim Bernoulli \big( <w_{i}, w_{j}  > \big), \quad \forall i < j;~i,j=1,2,\ldots,n.
\end{split}
\label{eq:twoGraphs}
\end{equation}
That is, Each graph is generated by a random dot product graph (RDPG), and the underlying dependency is reflected via the quadratic function of one-dimensional latent variable; this implies both multi-dimensional and nonlinear relationship where \texttt{MGC} is preferred to other benchmarks in testing network dependency in nodal attributes.

\begin{figure}[h!]
\begin{cframed}
\centering
	\includegraphics[width=0.7\textwidth]{../../figs/Graphs}
	\caption{The power curve with respect to increasing number of nodes for the two-graph dependency testing simulation (Equation~\ref{eq:twoGraphs}). The proposed approach achieves higher power than other methods.}
    \label{fig:graphtest}
		\end{cframed}
\end{figure}

Figure~\ref{fig:graphtest} shows the testing power of \texttt{MGC}, \texttt{mCorr}, and \texttt{HHG} against the number of nodes $n$, all based on the diffusion maps, and it demonstrates that the proposed approach is able to achieve higher testing power under relatively small number of nodes. Note that if noise is included in the set-up, or the nonlinear relationship is more complex than quadratic, the proposed approach still enjoys the same advantage, i.e., the testing power converges to $1$ faster than all other methods, though the actual number of nodes to achieve perfect power will likely increase under noisy and complex dependency.

The draft is submitted this month and available on arXiv.

\clearpage
\end{document}