\section{Inversion properties}
An inversion is a template class of the underlying data vector Vec and the matrix type Mat.
It is initialised with one of the following constructors:
\begin{lstlisting}
//! very simple and empty inversion
Inversion( bool verbose = false, bool dosave = false )
//! Usual inversion constructor with data and FOP
Inversion( Vec data, ModellingBase & forward, bool verbose, dosave)
//! Complete constructor including transformations
Inversion( Vec data, ModellingBase & forward,
    Trans & transData, Trans & transModel, bool verbose, bool dosave )
\end{lstlisting}
The properties are not visible itself, instead there are setter and getter functions. Setters:
\begin{lstlisting}
setRelativeError( double/Vec error);  //! set relative data error
setAbsoluteError( double/Vec error);  //! set absolute data error
setForwardOperator( ModellingBase & forward ); //! set forward operator
setTrans( transData, transModel );   //! set transformations
setTransData( transData );           //! set data transformation
setTransModel( transModel );         //! set model transformation
setLineSearch( bool isLinesearch );  //! switch line search on/off
setRobustData( bool isRobust );      //! IRLS (robust) data weighting
setBlockyModel( bool isBlocky );     //! IRLS (blocky) model constraints
setLambda( double lambda );          //! regularisation strength
setOptimizeLambda( bool optLambda ); //! L-curve optimization
setMaxIter( int maxIter );           //! define maximum iteration number
setModel( Vec model );               //! set model vector
setModelRef( Vec referenceModel );   //! set reference model vector
setCWeight( Vec cWeight );           //! set constraint control vector
setMWeight( mWeight );               //! set model control vector
\end{lstlisting}
Getter%Vec error();
\begin{lstlisting}
ModellingBase & forwardOperator();     //! forward operator
uint boundaryCount(), modelCount(), dataCount();//! # boundaries/cells/data
bool lineSearch(), blockyModel(), robustData(), optimizeLambda();//!options
double getLambda();                   //! regularisation strength
int maxIter();                        //! maximum iteration number
Vec model(), response(), roughness(); //! model/response/roughness vector
Vec cWeight(), mWeight();             //! constraint/model control vector
getPhiD(), getPhiM(), getPhi(), getChi2(); //! objective function parts
\end{lstlisting}
Run inversion and other actions
\begin{lstlisting}
Vec model = run();               //! runs the whole inversion
Vec model = oneStep();           //! runs one inversion step
Vec model = runChi1();           //! runs changing lambda such that chi^2=1
robustWeighting();               //! applies robust data weighting
constrainBlocky();               //! apply blocky model contraints
echoStatus();                    //! echo chi2/phi/phiD/phiM/iteration
Vec modelCellResolution( iRes ); //! compute a column of resolution matrix
\end{lstlisting}

\clearpage
\section{Mesh types and mesh generation}\label{app:meshes}
There are different mesh types and ways how to generate them. There is only one base mesh class holding the nodes/vertices/coordinates, cells (defined by the bounding vertices) and boundaries revealing the neighboured cells.
Every node, cell and boundary has a marker that defines the behaviour in modelling (e.g. an electrode node or boundary conditions) or inversion (e.g. the region number or a known sharp boundary).

\subsection*{0d mesh}
Zero dimension means that there are several parameters without any neighbouring relation.
Consequently 0th order constraints are used.
There is no special mesh generator, instead a 1d mesh is created and the constraint type is set to zero.
Alternatively, the forward operator is initialised without mesh and the parameter number is set by setParameterCount.
\subsection*{1d mesh}
A real 1d mesh subdivides the subsurface in vertically or horizontally aligned elements and can be created by the following functions:
\begin{lstlisting}
/*! Generate 1d mesh with nCells cells (size 1) and nCells+1 nodes */
Mesh createMesh1D( uint nCells, uint nClones = 1 );
/*! Generate simple one dimensional mesh with nodes in RVector pos */
Mesh createMesh1D( const RVector & x );
\end{lstlisting}
nClones can be used to create models for different parameters such as resistivity and phase.
Result is one mesh with two sub-meshes that are individual regions.
See section~\ref{sec:dc1dsmooth} for an example.

\subsection*{1d block model}
A 1d block model consists of (nLayers-1) thicknesses and nLayers values for a number of properties.
The thicknesses and properties are individual 1d meshes.
\begin{lstlisting}
/*! Generate 1D block model of thicknesses and properties */
Mesh createMesh1DBlock( uint nLayers, uint nProperties = 1 );
\end{lstlisting}
See section~\ref{sec:dc1dblock} for a resistivity block inversion or \ref{sec:blockjoint} for joint block inversion using two properties.

\subsection*{2d regular mesh}
A regular (FD like) 2d model consists of regularly spaced rectangles.
They can be created by the number of elements in x- or y-direction or vectors of the enclosing nodes:
\begin{lstlisting}
/*! Generate simple 2d mesh with xDim*yDim cells of size 1 */
Mesh createMesh2D( uint xDim, uint yDim );
/*! Generate simple 2d mesh from node vectors x and y */
Mesh createMesh2D( const RVector & x, const RVector & y );
\end{lstlisting}

\subsection*{2d general mesh}
Generally a 2d mesh - a regular one is just a special case - can consist of triangles or quadrangles (deformed rectangles) or a mix of it.
They are created by mesh generators such as triangle \citep{triangle}.
Input for the mesh is a piece-wise linear complex (PLC) comprising nodes edges and region markers.
Meshing is done externally and loaded using \lstinline|Mesh.load(filename);|.

\subsection*{3d regular mesh}
A regular (FD like) 3d model consists of regularly spaced hexahedra.
They can be created by the number of elements in x/y/z-direction or vectors of the enclosing nodes:
\begin{lstlisting}
/*! Generate regular 3d mesh with xDim*yDim*zDim cells of size 1 */
Mesh createMesh3D( uint xDim, uint yDim, uint zDim );
/*! Generate regular 3d mesh from node vectors x, y and z */
Mesh createMesh3D( const RVector & x, & y, & z );
\end{lstlisting}

\subsection*{3d general mesh}
At the moment, a 3d mesh can consist of tetrahedrons or hexahedrons, but prisms or pyramids could be easily implemented.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\clearpage
\section{Region properties and region map file}\label{app:regions}
Some properties can be set for each region individually using \lstinline|f.region(i).|
\begin{lstlisting}
setMarker( int marker );
setBackground( bool background );
setSingle( bool single );
setStartVector( const RVector & start );
setStartValue( double start );
setModelControl( double mc );
setModelControl( const RVector & mc );
setBoundaryControl( double or RVector bc );
%setZPower( double zpower );
setZWeight( double zweight );
setTransModel( Trans & tM );
setConstraintType( uint type );
setLowerBound( double lb );
setUpperBound( double ub );
\end{lstlisting}

\begin{lstlisting}
uint parameterCount(), boundaryCount(); //! parameters/boundaries
\end{lstlisting}

The region manager controls the individual regions.
It is initialised from the forward operator and interprets the mesh with its markers
\begin{lstlisting}
setMesh( const Mesh & mesh );
Region * createRegion( int marker, const Mesh & mesh ); //! create region
Region * region( int marker ); //! get an individual region by marker
uint parameterCount(), constraintCount(), boundaryCount(); //! counters
RVector createStartVector();  //! create starting model vector
RVector createModelControl(), createBoundaryControl(); //! create vectors
RVector createFlatWeight( double zPower, double minZWeight ); //! zpower
loadMap( const std::string & fname ); //! set region properties from file
\end{lstlisting}

The latter region map file simplifies the setting of region properties and is comfortable for testing different values.
It is a column file with the description of the columns in the first row after a \# sign:
\begin{verbatim}
#No start Ctype MC  zWeight Trans lBound uBound
0   100   1     1   0.2  	  log   50     1000
1   30    0     0.2 1       log   10     200
\end{verbatim}
The example represents a two layer case, e.g. an unsaturated (0) and a saturated (1) zone with different starting resistivities.
Smoothness constraints with enhanced layering is applied in the first and minimum length in the latter.
Both use a log/logLU transformation with specific upper and lower bounds.

Instead of the number, an asterisk (*) can be used to set properties for all regions.
In one region file, several blocks as above can be stated, e.g.
\begin{verbatim}
#No start Trans
*   100   log
#No lBound uBound
1   20     300
2   50     1000
\end{verbatim}
defines an equal model transformation and starting value, but different lower and upper bounds.

There are two special types of regions: background and single regions.
The background region is not part of the inversion, the values are prolongated (filled up) for the forward run.
On the contrary, the cells of a single region are held constant and treated as one parameter in inversion.
They are specified as above using the keywords \verb|background| and \verb|single|, e.g.
\begin{verbatim}
#No single
*	1
#No background
0	1
\end{verbatim}
defines all regions as single parameter regions except number zero, which is background.

By default, regions are decoupled from each other, i.e. there are no smoothness contraints at the boundary.
However, it might be useful to have those, e.g. by constraining a region of known parameters by borehole data to the neighboring cells or just to stabilize inversion.
In this case, inter-region constraints can be defined in the region file.
The text
\begin{verbatim}
#Inter-region
* 	*	  0.1
1   2   1
\end{verbatim}
sets weak connection between all regions, except regions 1 and 2 are normally connected.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\clearpage
\section{Transformation functions}\label{app:trans}
For data and model parameter arbitrary transformations can be applied.
A transformation is a C++ class derived from a base class (the identity transformation), which mainly consists of four functions, each returning a vector for a given vector:
\begin{description}
\item[trans] the forward transformation function: $y(x)$
\item[invTrans] the inverse of the function: $x(y)$
\item[deriv] the derivative of the function: $y'(x)$
\item[error] the transformation of associated errors $\delta y(\delta x)$
\end{description}
The latter is defined in the base class.
The inversion as mathematical operation is done in the $y$ domain, whereas the physics is described in the $x$ domain.

Besides the presented transformations, you can define your own transformations by deriving from the base class and overwriting the first three functions.
If the inverse transformation is not known analytically, there is a class \textbf{TransNewton}, in which the inverse function is obtained by Newton iteration.

However, there are a lot of already existing transformation classes that can be used or combined:

\subsection*{Basic transforms}
\begin{description}
\item[TransLinear](a,b): $y(x)=a*x+b$
\item[TransPower]($n$): $y(x)=x^n$
\item[TransExp]($x_0$,$y_0$): $y(x)=y_0\cdot e^{-x/x_0}$
\item[TransInv]: $y(x)=1/x$ (specialisation of TransPower)
\item[TransLog]: $y(x)=\log(x)$
\end{description}

\subsection*{Range transforms}
The logarithm restricts $x$ to be positive, i.e. sets a lower bound 0.
Instead of 0, a lower bound $x_l$ can be set.
By combining two logarithmic functions a lower and an upper bound can be combined.
Similar can be obtained by a cotangens function.

\begin{description}
\item[TransLog]($x_l$): $y(x)=\log(x-x_l)$
\item[TransLogLU]($x_l$,$x_u$): $y(x)=\log(x-x_l)-\log(x_u-x)$
\item[TransCotLU]($x_l$,$x_u$): $y(x)=-\cot((x-x_l)/(x_u-x)\cdot\pi)$
\end{description}

\subsection*{Combination}
Different transformations can be combined by either\\
\textbf{TransNest}($y^1$,$y^2$): $y(x)=y^1(y^2(x))$\\
\textbf{TransAdd}($y^1$,$y^2$): $y(x)=y^1(x)+y^2(x)$\\
Since for the latter the inverse cannot be combined by the two inverses, it is derived from \lstinline|transNewton|, a base class whose inverse is achieved by a Newton iteration.
So any not-so-easily-invertible function can be derived from \lstinline|transNewton| and does not require to define \lstinline|invTrans|.

There is a cumulative transformation \lstinline|CumulativeTrans|, which applies a vector of transformations for different part of the model.
This meta-transformation is applied in the region technique but can also be defined for one region.
More often it is used to combine different data or model types, see sections \ref{sec:mt1d}, \ref{sec:ttoffset} and \ref{sec:jointdcem} for examples.

\subsection*{Special transformations}
Some geophysically relevant transformations have been already defined:
\begin{description}
\item[TransLogMult]($y_0$) is a \textbf{TransNest} of \textbf{TransLog} and \textbf{TransMult}: $y(x)=y_0\log(x_0)$, e.g. for using the logarithm of the apparent resistivity ($\rho^a=G R$)
\item[TransCRIM]($\phi$,$\epsilon_m$,$\epsilon_w$): the complex refractive index model (CRIM) - derived from \textbf{TransLinear}
\item[TransArchie]($\rho_w$) - Archie's equation, derived from \textbf{TransPower}
\end{description}

Note that there are predefined types based on real (double) vectors beginning with an R, e.g. \lstinline|RTransLogLU| is actually \lstinline|TransLogLU< RVector >|.

\section{Vectors and Matrix types}\label{app:matrix}
\sperre

\lstinline|std::vector|

\lstinline|template< class ValueType > class Vector|

\lstinline|RVector|,\lstinline|FVector|, \lstinline|BVector|,\lstinline|IVector|

\lstinline|load/save|

complex values using Complex = \lstinline|std::complex<double>| as a vector \lstinline|CVector|

\lstinline|template < class ValueType > class Matrix|

\lstinline|RMatrix|, \lstinline|FMatrix|

\lstinline|load/saveMatrixCol|

\lstinline|template< class ValueType, class IndexType > class SparseMapMatrix|

\lstinline|RSparseMapMatrix| for double and unsigned int

modelling column-compressed sparse matrix \lstinline|SparseMatrix|

\subsection{Block-Matrices}
Large matrices for Jacobian (J) and constraint matrix (C)

holding individual matrices by stacking matrices saves space and makes allocation easier

Horizontal stacking (H types): individual models (or model parts)

Vertical stacking (V types): individual data (J) or constraints (C)

2 matrices (2) of arbitrary type, N matrices (N) of identical type, repetition (R) of identical matrices

\lstinline|H2Matrix< Matrix1, Matrix2 >|
example: 2 data sets for 1 model

\lstinline|V2Matrix< Matrix1, Matrix2 >| example: 1 model, 2 data sets

\lstinline|D2Matrix< Matrix1, Matrix2 >| example: 2 models, 2 data sets

\lstinline|HNMatrix< Matrix >| example: LCI

\lstinline|VNMatrix< Matrix >| (N vertical identical matrices)

\lstinline|DNMatrix< Matrix >| (diagonal matrices, Block-Jacobi)
example: LCI (identical models with individual Jacobians)

\lstinline|DRMatrix< Matrix >| (
example:


\section{Our Construction}\label{sec:construction}

\subsection{Overview}
First we describe our join algorithm without any privacy and then we will discuss how this translates to the secret shared setting. As depicted in \figureref{fig:mapping}, the algorithm can be broken into the following phases:
\begin{enumerate}
	\item \label{step:overview1}  $Y$ is inserted into a cuckoo hash table $\ytable$ based on the join-key(s). That is, let us assume the columns $Y_1$ and $X_1$ are the join keys. Then
	row $Y[i]$ is inserted at $\ytable[j]$ for some $j\in \{h_0(Y_1[i]), h_1(Y_1[i])\}$.
	\item \label{step:overview2} Each row $X[i]$ will need to be compared with the rows $\ytable[j]$ for $j\in \{h_0(X_1[i]), h_1(X_1[i])\}$. This will be achieved by mapping $\ytable[h_0(X_1[i])]$ to a new row $\widehat{Y}^0[i]$ and $\ytable[h_1(X_1[i])$ to $\widehat{Y}^1[i]$.
	\item \label{step:overview3} It is now the case that if row $X[i]$ has a machining key in $Y$, the this row will be located at $\widehat{Y}^0[i]$ or $\widehat{Y}^1[i]$. As such, these rows can directly be compared to determine if there is a match on the join keys and the where clause evaluates to true. Let $b_i=1$ if there is such a match and $0$ otherwise.
	\item \label{step:overview4} Various types of joins can then be constructed from locally comparing row $i$ from these tables, i.e. $X[i],\widehat{Y}^0[i], \widehat{Y}^1[i]$. For example, an inner join is constructed from all the rows where $b_i=1$ by selecting the values from $X[i]$ and either $\widehat{Y}^0[i]$ or $\widehat{Y}^1[i]$ depending on which one matches. If there is no match, then that output row is set to \Null.
\end{enumerate}

The main challenge in bringing the described algorithm to the secret shared setting is constructing the cuckoo hash table $\ytable$ and selecting rows from $\ytable$ without leaking sensitive information. We achieve this with the use an MPC friendly \emph{randomized encoding} and a new three-party protocol called an \emph{oblivious switching network}.

%Our core protocol is a technique for obliviously mapping together rows with equal join-keys. For the $i$th secret shared row $\share{X}[i]$ and for some $j_0,j_1$, our protocol obliviously maps rows $\share{Y[j_0]},\share{Y[j_1]}$ to the $i$th row of two new tables $\share{\widehat{Y}^0},\share{\widehat{Y}^1}$ such that if $Y$ contains a row with matching keys then either $\share{\widehat{Y}^0}[i]$ or $\share{\widehat{Y}^1}[i]$ will be this row. If no such key exists in $Y$ then an arbitrary rows from $Y$ are mapped to these locations. Once the mapping is performed the final output table can be constructed by an MPC protocol\cite{aby3} that directly compares the keys for the row of $\share{X}[i]$ with $\share{\widehat{Y}^0}[i]$ and $\share{\widehat{Y}^1}[i]$. If the keys match then the output row is constructed and otherwise a dummy \Null row is constructed.

Let us continue to assume that the columns $X_1$ and $Y_1$ are the join-keys. Our protocol begins by generating a \emph{randomized encoding} for each of the secret shared join-key $\share{x_i}\in \share{X_1}$ and $ \share{y_i}\in \share{Y_1}$. \figureref{fig:randomized-encode-ideal} contains the ideal functionality for this encoding which takes secret shares from the parties, apply a PRF $F_k$ to the reconstructed value using a internally sampled key $k$, and returns the resulting value to one of the three parties. For $\share{ x_i}:= \share{X_1}[i]$, \Party{0} will learn $F_k(x_i)$ while \Party{1} will learn $F_k(y_i)$ for $\share{ y_i}:=\share{Y_1}[i]$. Since the join-keys $x_i$ (resp. $y_i$) are unique and $k$ is not known, this reveals no information to \Party{0} (resp. \Party{1}).

Party \Party{1} proceeds by constructing a \emph{secret shared} cuckoo hash table $\shareTwo{\ytable}$ from the rows of $\share{Y}$ where the hash function values for row $i$ are defined as $h_j(y_i) = H( j || F_k(y_i))$. Note that \Party{1} knows only the randomized encodings $F_k(y_i)$ of each row $Y[i]$, and not the contents of the row itself. The goal in this step is to construct a secret shared cuckoo table $\shareTwo{\ytable}$ such that row $Y[i]$ is located at $\ytable[h_j(y_i)]$ for some $j$. We construct  $\shareTwo{\ytable}$ using a three-party \emph{oblivious permutation protocol} where \Party{1} inputs a permutation $\pi$, all parties input secret shares of $Y$, and the result is secret shares of ``$Y$ permuted according to $\pi$'' which forms $\ytable$ (details follow later). This completes \stepref{step:overview1} and is the first transformation shown in \figureref{fig:mapping}.

It is now the case that $\shareTwo{\ytable}$ is a valid cuckoo hash table of $\share Y$ which is secret shared between \Party{0} and \Party{1}. Party \Party{0}, who knows the randomized encodings $F_k(x_i)$ for all $\share{ x_i}:= \share{X_1}[i]$, now must compare the rows of $\shareTwo{\ytable}$ indexed by $h_j(x_i)= H( j || F_k(x_i))$ with the row $\share X[i]$. In particular, assuming we use two cuckoo hash functions, then \Party{0} constructs two \emph{oblivious switching networks} that maps the shares $\shareTwo{\ytable[{h_0(x_i)}]}$ and $\shareTwo{\ytable[{h_1(x_i)}]}$ to be ``aligned" with $\share X[i]$. Exactly how such a network operates is discussed later but the result is two new tables $\shareTwo{\widehat{Y}^0},\shareTwo{\widehat{Y}^1}$ such that $\ytable[{h_j(x_i)}]=\widehat{Y}^j[i]$. This completes \stepref{step:overview2} and is the second transformation shown in \figureref{fig:mapping}.

Once the shares of $\widehat{Y}^0[i]={\ytable[{h_0(x_i)}]}, Y_1'[i]={\ytable[{h_1(x_i)}]}$ are obtained using the switching network, the parties employ an MPC protocol to directly compare these rows with $\share{X}[i]$. That is, they compute a bit $\share{b}$ which equals one if the join-keys are equal and the \texttt{where} clause $P(\shareTwo{\widehat{Y}^j}[i],\share{X}[i])$ outputs one for some $j$. For each row, the output row for an inner join is constructed as $S(\shareTwo{\widehat{Y}^j}[i],\share{X}[i])$ using MPC where $S$ is the user defined selection circuit. In addition, the MPC circuit outputs the secret shared flag $\share{b}$ indicating whether this row is set to \texttt{NULL}.

Left joins work in a similar way except that all rows of $X$ are output and marked not \texttt{NULL}. Finally, unions can be computed by including all of $Y$ in the output and all of the rows of $X$ where the comparison bit $\share{b}$ is zero. Regardless of the type of join, the protocols do not reveal any information about the tables. In particular, not even the cardinality of the join is revealed due to the use of \texttt{NULL} rows.

\begin{figure}\centering
	\frame{	\begin{tikzpicture}[scale=0.48, every node/.style={scale=0.48}]
		\draw (0, 0) node[inner sep=0] {{\includegraphics[trim={0cm 7cm 14cm 1cm},clip,width=\textwidth]{diga.pdf}}};
		\draw (-7.9cm,4cm) node {\large$\share{Y}$};
		\draw (7.2cm,4cm) node {\large$X$};
		\draw (-3.7cm,4cm) node {\large$\ytable=\textsf{Cuckoo}(Y)$};
		%	\draw (3.8cm,4.6cm) node {\large Selections w/ $h_0(x),h_1(x)$};
		\draw (3.3cm,4cm) node { \large$\widehat{Y}^0$};
		\draw (5.2cm,4cm) node { \large$\widehat{Y}^1$};

		\draw (-5.8cm,-4.5cm) node { 1) Cuckoo hash  $Y$  };
		\draw (-5.8cm,-5.0cm) node { using oblv. permutation.};
		\draw (-5.8cm,-5.5cm) node { $\exists j$ s.t. $T[h_j(Y[i])]=Y[i]$. };

		\draw (-0.3cm,-4.5cm) node { 2) Select Cuckoo locations $T[h_j(X[i])]$  };
		\draw (-0.3cm,-5.0cm) node { using oblv. switching network.};
		\draw (-0.3cm,-5.5cm) node { $\widehat{Y}^j[i]=T[h_j(X[i])]$.};

		\draw (5.4cm,-4.5cm) node { 3) Compare $\widehat{Y}^0[i],\widehat{Y}^1[i]$ w/ $X[i]$};
		\draw (5.4cm,-5.0cm) node { using MPC circuit and  };
		\draw (5.4cm,-5.5cm) node {construct output row. };
		\end{tikzpicture}}
	\caption{Overview of the join protocol using oblivious switching network.\label{fig:mapping}}
\end{figure}

\subsection{Randomized Encodings}

The randomized encoding functionality \f{encode} of \figureref{fig:randomized-encode-ideal} enables the parties to coordinate their secret shares without revealing the underlying values. In particular, the parties will construct a cuckoo hash table using these encodings. The functionality takes as input several tuples $(\share{B_i},\share{X_i},\Party{i})$ where $B_i\in\{0,1\}^d$ is an array of $d$ bits, $ X_i\in(\{0,1\}^\sigma)^{d}$ is a array of $d$ strings and  $\Party{i}$ that denotes that party $\Party{i}$ should be output the encodings for this tuple. The functionality assigns a random $\ell$ bit encoding for each input $x\in \{0,1\}^\sigma$. For $j\in[d]$, if the bit $B_i[j]=0$ then the functionality outputs the encoding for $X_i[j]$ and otherwise a random $\ell$ bit string. Looking forward, $B_i[j]=0$ will mean that the key $X_i[j]$ is actually set to \Null and a random encoding should be returned.

\begin{figure}[ht]
	\framebox{\begin{minipage}{0.95\linewidth}
			Parameters: Input string size of $\sigma$ bits and output encoding size of $\ell$ bits.\\

			{\bf [Encode]} Upon receiving command $(\textsc{Encode},$ $\{(\share{B_i},\share{X_i}, \Party{i})\})$ from all parties where $X_i\in (\{0,1\}^\sigma)^{d_i}, B_i\in\{0,1\}^{d_i}$ for some $d_i\in \mathbb{Z}^*$.
			\begin{enumerate}
				\item Sample a uniformly random $F:\{0,1\}^\sigma \rightarrow \{0,1\}^\ell$. Define $F':\{0,1\}\times \{0,1\}^\sigma \rightarrow \{0,1\}^\ell$ as  $F'(b,x) = \overline{b}F(x) + br$ where $r\gets\{0,1\}^\ell$ is sampled each call.

				\item For each $(\share{ B_i},\share{ X_i},\Party{i})$, send $\{F'(b,x)\mid (b,x)\in \textsc{zip}(B_i,X_i)\}$ to $\Party{i}$.
			\end{enumerate}
	\end{minipage}}
	\caption{The Randomized Encoding ideal functionality \f{encode}}
	\label{fig:randomized-encode-ideal}
\end{figure}

\paragraph{LowMC Encodings}
We realize this functionality using the LowMC block cipher\cite{lowmc}. When implemented with the honest majority MPC protocols\cite{highthroughput}, this approach results in extremely high throughput, computing up to one million encodings per second. Once the parties have their secret shared inputs, they sample a secret shared LowMC key uniformly and encrypt each input under that key using the MPC protocol. These encryptions are revealed as the encodings to the appropriate party.

The LowMC cipher is parameterized by a block size $\ell$, keys size $\kappa$, s-boxes per layer $m$ and the desired data complexity $d$. To set these parameters, observe that the adversary only sees a bounded number of block cipher outputs (encodings) per key. As such, the data complexity can be bounded by this value. For our implementation we upper bound the number of outputs by $d= 2^{30}$. The remaining parameters are set to be $\ell\in\{80, 100\}$ and $m=14$ which results in $r=13$ rounds and computational security of $\kappa=128$ bits\cite{lowmc}. The circuit for $\ell=80$ contains 546 \textsc{and} gates (bits of communication).

%We show simulation of this approach with statistical correctness of $\lambda$ bits. First, by the assumption that LowMC is indistinguishable from an ideal cipher,
%
%\paragraph{Long Encodings}

One issue with the LowMC approach alone is that the input size is fixed to be at most $\ell\in\{80,100\}$ bits. However, we will see that the larger join protocol requires an arbitrary input size $\sigma$. This is accommodated by applying a universal hash function to the input shares. Specifically, the parties jointly pick a random matrix $E\gets\{0,1\}^{\sigma\times \ell}$. The parties can then locally multiply each secret shared input before it is sent into the LowMC block cipher.

The security of this transformation follows from $xE\neq x'E$ with overwhelming probability if $x\neq x'$. In particular, $f(x)=xE$ is a universal hash function given that $E$ is independent of $x$. As such the probability that $f(x)=f(x')$ for any $x\neq x'$  is $2^{-\ell}$. Applying the birthday bound we obtain that probability of any collisions among the tuples is $2^{-\ell+p}$ where $p=\log_2 D^2/2=2\log_2(D)-1$ and $D=\sum_i d_i$.

Conditioned on the inputs to the block cipher being unique, the outputs of the block cipher is also distinct and indistinguishable from random $\ell$ bit strings. As such, in the simulation the real outputs can be replaced with that of the ideal functionality so long as $2^{-\ell+p}$ is statistically negligible, i.e. $\ell-p\geq\lambda$.

%$n*n=n^2$
%$(D-1) + (D-2)+...+1 = \sum_{i=1}^D (D-i)= D^2/2 = (2n)^2/2=2n^2$

%
%To preserve security we require the probability of the following game outputting 1 be negligible in $\lambda$.
%\begin{quote}
%Have the adversary select two sets $X,Y\subset \{0,1\}^{m}$ of size $n_1,n_2\in poly(\lambda)$  respectively and then a uniformly random matrix $E\gets \{0,1\}^{m\times \sigma}$ is sampled. If there exists distinct $x\in X$ and $y\in Y$ such that $xE = yE$, output 1, otherwise 0.
%\end{quote}

%First observe that this use of $E$ is a universal hash function.

%Therefore, fixing any distinct $x\in X, y\in Y$, it holds that $\Pr[xE=yE]=2^{-\sigma}$. Applying a union bound over all pairs of items, the overall probability of a collision is $2^{-\sigma}n_1n_2$. By setting $\sigma \geq \lambda + \log_2(n_1) + \log_2(n_2)$, we ensure that the collision probability is bounded by the statistical  security $2^{-\lambda}$.

%In practice, we set $\lambda=40$ and choose $\sigma =80$ if $n_1=n_2=n\leq 2^{20}$ and $\sigma=100$ otherwise (supporting $n\leq 2^{30}$).

\begin{figure}[ht]
	\framebox{\begin{minipage}{0.95\linewidth}
			Parameters: Input, output size of $\sigma$, $\ell$ bits (respectively). Computational security parameter $\kappa$.\\

			{\bf [Encode]} Upon receiving command $(\textsc{Encode},$ $\{(\share{B_i}, \share{X_i}, \Party{i})\})$ from all parties where each $X_i\in (\{0,1\}^\sigma)^{d_i}$. Let $d=\max_i(d_i)$.
			\begin{enumerate}
				\item If $\sigma >\ell,$ the parties jointly sample a matrix $E\in\{0,1\}^{\sigma\times \ell}$. Otherwise $E$ is the ${\sigma\times \ell}$ identity matrix.

				 \item The parties have \f{mpc} evaluate the following circuit:
				\begin{enumerate}
					\item Uniformly sample a key $k$ for a LowMC cipher with block size $\ell$, computational security $\kappa$ and data complexity at least $d$ blocks.
					\item For each $(\share{B_i}, \share{X_i}, \Party{i})$ input pair, reveal $\{ F'(b,x) \mid (b,x)\in \textsc{zip}(B_i, X_i)\}$ to \Party{i} where $F'(b,x)=\textsf{LowMC}_k(xE)\oplus br$ and $r\gets\{0,1\}^\ell$ is sampled for each call.
				\end{enumerate}

		\end{enumerate}

	\end{minipage}}
	\caption{The randomized encoding LowMC protocol.}
	\label{fig:randomized-encode-lowMC}
\end{figure}

\subsection{Oblivious Switching Network}

The ideal functionality of a switching network was introduced by Mohassel and Sadeghian\cite{MS13}. It obliviously transform a vector $A=\{A_1,...,A_n\}$ such that the output is $A'=\{A_{\pi(1)}, ..., A_{\pi(m)}\}$ for an arbitrary function $\pi : [m]\rightarrow[n]$. The accompanying protocol of \cite{MS13} was designed in the two party setting where the first party inputs $A$ while the second party inputs a description of $\pi$. \iffullversion
This switching network require  $O(n\log n)$ cryptographic operations. Building on this general paradigm, we
\else
We
\fi
introduce a \emph{new} oblivious switching network protocol tailored for the honest majority setting with significantly efficiency improves. Our protocol has $O(n)$ overhead and is constant round. \cite{MS13} requires $O(n\log n)$ communication/computation and log rounds. %Moreover, our protocol can be instantiated with information theoretic security.

\newcommand{\programmer}{\ensuremath{P_{\textsf{p}}}\xspace}
\newcommand{\sender}{\ensuremath{P_{\textsf{s}}}\xspace}
\newcommand{\receiver}{\ensuremath{P_{\textsf{r}}}\xspace}

The ideal functionality of our protocol is given in \figureref{fig:perm-ideal} with three parties, a programmer \programmer, a sender \sender and a receiver \receiver. \programmer has a description of  $\pi$ while \sender has a vector $A$ containing $n$ elements each consisting of $\sigma$ bits. \programmer and  \receiver are each output a share of $\shareTwo{A'}$ s.t. $A'=\{A_{\pi(1)}, ..., A_{\pi(m)}\}$.  Later we will discuss the case where $A$ is secret shared.

\begin{figure}[ht]
	\framebox{\begin{minipage}{0.95\linewidth}
			Parameters: $3$ parties denoted as the \programmer, \sender and \receiver. Elements are strings in $\{0,1\}^\sigma$. An input vector size of $n$ and output size of $m$.

			{\bf [Switch]} Upon the command $(\textsc{switch}, \pi)$ from the \programmer and $(\textsc{switch}, A)$ from the \sender:
			\begin{enumerate}
				\item Interpret $\pi: [m]\rightarrow [n]$ and $A\in (\{0,1\}^{\sigma})^n$.
				\item Compute $B$ s.t. $\forall i\in [m], A_{\pi(i)} = B_i$.
				\item Generate $\shareTwo{B}$ and send $\shareTwo{B}_0$ to \programmer and $\shareTwo{B}_1$ to \receiver.
			\end{enumerate}
	\end{minipage}}
	\caption{The Oblivious Switching Network ideal functionality \f{switch}}
	\label{fig:perm-ideal}
\end{figure}

\paragraph{Permutation Network}\label{sec:perm}

We begin with a restricted class of switching networks where the programming function $\pi$ is injective. {That is, each input element $A_i$ will be mapped to a maximum of one location in the output.}  \programmer samples two random functions $\pi_0,\pi_1$ such that $\pi_1 \circ \pi_0 = \pi$, $\pi_0:[n]\rightarrow [n]$ is bijective and $\pi_1:[m]\rightarrow [n]$ is injective.  \programmer sends   $\pi_1$ to  \receiver and $\pi_0, S\gets \{0,1\}^{\sigma\times n}$ to  \sender who sends $B := \{A_{\pi(1)} \oplus S_0, ...,A_{\pi(n)} \oplus S_n \}$ to  \receiver. The final shares of $A'=\pi(A)$ are defined as  \programmer holding $\shareTwo{A'}_0:=\{S_{\pi_1(1)}, ..., S_{\pi_1(m)}\}$ and the \receiver holding $\shareTwo{A'}_1:=\{B_{\pi_1(1)}, ..., B_{\pi_1(m)}\}$.

The simulation of this protocol is perfect. The view of \sender contains a uniform permutation $\pi_0$ and vector $S$.  Similarly, the view of  \receiver contains $\pi_1$ which is uniformly distributed (when $\pi_0$ is unobserved) and the uniform vector $B$. A simulator can sample these directly on behalf of the honest parties.

In our computational secure setting,  $\pi_0,S$ can be generated locally by \Party{0} and \Party{1} using a common source of randomness, e.g. a seeded PRG. This reduces the rounds to 1.

\paragraph{Duplication Network}\label{sec:dup}

Next we consider a second type of network where $\pi : [n]\rightarrow[n]$,  $\pi(1)=1$ and $\pi(i)\in \{i, \pi(i-1)\}$ for $i=2,...,n$. That is, each output position is either a copy of the same input position or is a duplicate of the previous output position, e.g. $A'=\{A_1,A_1,A_3,A_4,A_4,A_4\}$ where $A_1,A_4$ were duplicated into the next position(s). This transformation can be characterized by a vector $b\in\{0,1\}^{n}$ where $b_i=1$ denotes that the output position $i$ should be a copy of output position $i-1$.

\iffullversion
This observation gives rise to a natural protocol: for $i=1,...,m$, if $b_i=1$ then use MPC to copy $A_{i-1}$ into $A_{i}$. The primary challenge is to achieve this while using a constant number of communication rounds which prevents the use of a generic (secret sharing) MPC protocol such as \cite{aby3, highthroughput}.
\fi

As a warm-up, suppose that the \sender inputs $A_{i-1},A_{i}$ and that  \programmer  \& \sender should receive a share of $\shareTwo{B_{i}}$ such that $B_{i}:=A_{i-b_i}$. \sender samples three uniform strings $\shareTwo{B_{i}}_1, w_0,w_1\gets \{0,1\}^\sigma$ and a uniform bit $\phi\gets \{0,1\}$. \sender constructs two messages $m_0=A_{i}^1\oplus \shareTwo{B_{i}}_1\oplus w_\phi$ and $m_1= A_{i-1}\oplus \shareTwo{B_{i}}_1 \oplus w_{\phi\oplus 1}$.  \sender sends $w_0,w_1$ to the \receiver and sends $m_0,m_1,\phi$ to  \programmer who sends $\rho=\phi\oplus b_i$ to  \receiver. The final shares are constructed by having  \receiver send $w_\rho$ to  \programmer who computes $\shareTwo{B_{i}}_0:=m_{b_i}\oplus w_{\rho}$.

The simulation of this protocol is also perfect. \programmer learns $m_0=A_{i}\oplus \shareTwo{B_{i}}_1\oplus w_\phi, m_1=A_{i-1}\oplus \shareTwo{B_{i}}_1\oplus w_{1\oplus \phi}, w_{b_i\oplus \phi}$. As such, they can compute $\shareTwo{B_i}_0=A_{i-b_i}\oplus \shareTwo{B_{i}}_1$ but not $A_{i-\overline{b_i}}\oplus \shareTwo{B_{i}}_1$. \receiver only learns $w_0,w_1$ and $\rho=b_i\oplus \phi$ which can be trivially simulated. In our computationally secure setting observe that $w_0,w_1,\phi$  are uniformly random and sending them can be optimized away with a pre-shared PRG seed.

The protocol just described considers the setting where the messages $A_{i-1},A_{i}$ are the private input of  \sender. However, we require that at each iteration the messages being selected is either $B_{i-1}$ or $A_{i}$ where $\shareTwo{B_{i-1}}$ was computed in the previous iteration. In this case  \sender inputs their share of $B_{i-1}$ instead of $A_{i-1}$ while the \programmer computes $\shareTwo{B_{i}}_0:=m_{b_i}\oplus w_{\rho}\oplus b_i\shareTwo{B_{i-1}}_0$.

Note that this protocol outputs shares to \sender as opposed to  \receiver. This can be corrected by having \sender secret share $\shareTwo{B}_1$ between \receiver \& \programmer but we prefer to leave the protocol as is for composability reasons.

\paragraph{Shared Inputs}

The protocols and functionality described above assume the vector being transformed is the private input of the \sender. However, our larger  protocols will require the transformations to be applied to secret shared vectors. In particular, the parties hold  $\share{A}$.
As described in [shared Switch] of \figureref{fig:switching-net}, the parties first locally convert $\share{A}$ into $\shareTwo{A}$ between \sender and \programmer. They run \f{switch} where  \sender inputs their share $B=\shareTwo{A}_0$. \programmer and \receiver receive $\shareTwo{\pi(B)}$ from \f{switch} where \programmer holds $\shareTwo{\pi(B)}_0$. \programmer  locally define $\shareTwo{C}_0:=\shareTwo{\pi(B)}_0\oplus \pi(\shareTwo{A}_1)$ and \receiver defines $\shareTwo{C}_1:=\shareTwo{\pi(B)}_1$. It is easy to verify that $C=\pi(A)$. Simulation of this protocol essentially equivalent to simulating the call to \f{switch} since there is no added communication.

\paragraph{Universal Switching Network}\label{sec:switch}

A universal switching network supporting an \emph{arbitrary}  $\pi : [m]\rightarrow [n]$ can be constructed in three phases\cite{MS13}: %In particular, the input vector $A$ will have three transformations applied
 $A\overset{\pi_1}{\rightarrow}B\overset{\pi_2}{\rightarrow}C\overset{\pi_3}{\rightarrow}D=\pi(A)$.
\begin{enumerate}
	\item $B:=\pi_1(A)$:  The input vector $A$ is permuted by the injective function $\pi_1:[m]\rightarrow[n]$ such that if $\pi$ maps an input position $i$ to $k$ outputs positions (i.e. $k=|preimage(\pi,i)|=|\{ j : \pi(j)=i \}|$), then there exists a $j$ such that $\pi_1(j)=i$  and $\{\pi_1(j)+ 1,...,\pi_1(j )+k \} \cap image(\pi) = \emptyset$. That is, wherever position $i$ is mapped by $\pi_1$, it should be followed by $k-1$ positions that do not appear in the final output.

	\item $C:=\pi_2(B)$: The intermediate vector $B$ is transformed by a \emph{duplication network}  $\pi_2:[m]\rightarrow[m]$ which is defined as follows. If position $A_i$ is mapped to $k$ positions in $\pi(A)$, then $\{ C_{j},...,C_{j+k}\} = \{A_i\}=\{B_j\}$ where $\pi_1(j)=i$. That is, copies $B_{j}$ into the next $k-1$ positions.

	\item $D:=\pi_3(C)$: The final transformation $\pi_3:[m]\rightarrow[m]$  permutes $C$ to have the same ordering as $\pi(A)$. That is, the elements $\{ C_{j},...,C_{j+k}\}$ which all have the value  $A_i$ are arbitrary mapped to the $k$ positions $\{ j : \pi(j)=i \}$.
\end{enumerate}
Observe that steps $\pi_1,\pi_3$ can both be implemented using the oblivious permutation protocol while $\pi_2$ can be implemented with a duplication network.
\figureref{fig:switching-net} provides a formal description of the full switching network protocol.

The simulation of the full protocol ([switch] of \figureref{fig:switching-net}) essentially follows from the simulation of the permutation and duplication protocols. That is, simulation of step b) and d) of [switch] follows the simulation of \sectionref{sec:perm} and step c) follows from the simulation of \sectionref{sec:dup}.

\begin{figure}[ht!]
	\framebox{\begin{minipage}{0.95\linewidth}\small
			Parameters: $3$ parties denoted as \programmer, \sender and \receiver. Elements are strings in $\{0,1\}^\sigma$. An input, output vector size of $n, m$.\\

			{\bf [Permute]} Upon the command $(\textsc{Permute}, \pi)$ from  \programmer and $(\textsc{Permute}, A)$ from  \sender.  $\pi: [m]\rightarrow [n]$ is parsed as a \emph{injective} function and  $A\in \{0,1\}^{n\times \sigma}$ as a vector of $n$ elements. Then:
				\begin{enumerate}[leftmargin=.8cm]
					\item If $n<m$,  \sender redefines $A$ to be $A := A || \{0\}^{(m-n)\times \sigma}$ and all parties redefine $n:=m$.
					\item  \programmer samples a uniformly random bijective function $\pi_0 : [n]\rightarrow[n]$ and computes the injective function $\pi_1 :[n] \rightarrow[m]$ such that $\pi_1\circ \pi_0 = \pi$.  $\pi_0 $ and a random vector $S\gets \{0,1\}^{n\times \sigma}$ are sent to  \sender.
					\item  \sender computes and sends $B := \{ A_{\pi_0(1)} \oplus S_1, ..., A_{\pi_0(n)} \oplus S_n\}$ to  \receiver.
					\item  \programmer sends $\pi_1$ and a random vector $T\gets\{0,1\}^{m\times\sigma}$ to  \receiver who outputs $C^0:=\{B_{\pi_1(1)} \oplus T_1,...,B_{\pi_1(m)}\oplus T_m\}$.  \programmer outputs $C^1:=\{ S_{\pi_1(1)}\oplus T_1,...,S_{\pi_1(m)}\oplus T_m\}$.
				\end{enumerate}

				{\bf [Switch]} Upon the command $(\textsc{Switch}, \pi)$ from  \programmer and $(\textsc{Switch}, A)$ from  \sender. $\pi: [m]\rightarrow [n]$ is parsed as a function and  $A\in \{0,1\}^{n\times \sigma}$ as a vector of $n$ elements. Then:
				\begin{enumerate}[leftmargin=.8cm]

					\item  \programmer samples an injective function $\pi_1:[m]\rightarrow [n]$ such that for $i\in image(\pi)$ and $k=|preimage(\pi, i)|$, there exists a $j$ where $\pi_1(j)=i$ and $\{\pi_1(j+1), ...,\pi_1(j+k) \}\cap image(\pi)=\emptyset$.

					 \programmer  sends $(\textsc{Permute}, \pi_1)$ to \proto{switch} and  \sender sends $(\textsc{Permute}, A)$.  \programmer receives $B^{0}\in \{0,1\}^{m\times \sigma}$ in response and  \receiver receives $B^{1}\in \{0,1\}^{m\times \sigma}$.

					\item  \programmer  computes the vector $b\in\{0,1\}^{m}$ such that for $i\in image(\pi)$ and $k=|preimage(\pi, i)|$, $b_j = 0$ and $b_{j+1}=...=b_{j+k}=1$ where $\pi_1(j)=i$.

					 \receiver samples three $m$ element vectors $C^{1}, W^0,W^1\gets \{0,1\}^{m\times \sigma}$ and $\phi\gets\{0,1\}^m$. They set $C^{1}_1:=B^{1}_1$ and computes
					\begin{align*}
						M^0_i&:= B^1_{i}\ \ \, \oplus C^{1}_i \oplus W^{\phi_i}_i\\
						M^1_i&:= C^1_{i-1} \oplus C^{1}_i \oplus W^{\phi_i\oplus 1}_i
					\end{align*}
					for $i\in \{2,...,m\}$.  \receiver sends $M,\phi$ to  \programmer and $C^{1},W$ to  \sender.  \programmer sends $\rho:=\phi\oplus b$ to  \sender who responds with $\{ W^{\rho_i}_i : i\in [m] \}$.  \programmer defines $C^{0}_1:=B^{0}_1$ and computes
					$$
						C^{0}_i:= M^{b_i}_i \oplus W^{\rho_i}_i\oplus b_iC^{0}_{i-1}
					$$
					for $i\in \{2,...,m\}$.
					\item \programmer computes the permutation $\pi_3$ such that for  $i\in image(\pi)$ and $k=|preimage(\pi, i)|$, $\{\pi_3(\ell) : \ell\in preimage(\pi, i)\}=\{j, ..., j +k\}$ where $i=\pi_1(j)$.	 \programmer sends $(\textsc{Permute}, \pi_3)$ to \proto{switch} and  \sender sends $(\textsc{Permute}, C^{1})$.  \programmer receives $S\in \{0,1\}^{m\times \sigma }$ in response.  \receiver receives and outputs $D^{1}\in \{0,1\}^{m\times \sigma }$.

					 \programmer outputs $D^{0}_i:=S_i\oplus C^{0}_{\pi_3(i)}$ for $i\in [m]$.
				\end{enumerate}

				{\bf [Shared Switch]} Upon the command $(\textsc{SharedSwitch},$ $\pi, \shareTwo{A}_0)$ from \programmer and $(\textsc{SharedSwitch}, \shareTwo{A}_1)$ from \sender. $\pi: [m]\rightarrow [n]$ is parsed as a function and  $A\in \{0,1\}^{n\times \sigma}$ as a vector of $n$ elements.
				\begin{enumerate}
					\item \programmer sends $(\textsc{Switch}, \pi)$ to \proto{switch} and \receiver sends $(\textsc{Switch}, \shareTwo{A}_1)$.
					\item In response, \programmer and \receiver respectively receive $\shareTwo{B}_0$ and $\shareTwo{B}_1$. \programmer outputs $\shareTwo{B}_0\oplus \pi(\shareTwo{A}_0)$ and \receiver outputs $\shareTwo{B}_1$.
				\end{enumerate}
	\end{minipage}}
	\caption{The Oblivious Switching Network protocol \proto{switch}. }
	\label{fig:switching-net}
\end{figure}

\subsection{Join Protocols}\label{sec:join}

Our join protocol can be divided into four phases:

\begin{enumerate}
	\item Compute randomized encodings of the join-columns/keys.
	\item Party \Party{1} constructs a cuckoo table $T$ for table $Y$ and arranges the secret shares using a permutation protocol.
	\item For each row $x$ in $X$, \Party{0} uses an oblivious switching network to map  the corresponding location $i_1,i_2$ of the cuckoo hash table to a secret shared tuple $(x, T[{i_1}], T[{i_2}])$.
	\item The join-key(s) of $x$ is compared to that of $T[{i_1}], T[{i_2}]$. If one of them match then the corresponding output row is populated; otherwise the output row is set to \texttt{NULL}.
\end{enumerate}
Steps 1 through 3 are performed by the Map routine of \figureref{fig:full_proto} while step 4 is performed in \figureref{fig:full_proto2}. \figureref{fig:full_ideal} contains the ideal functionality of the join protocol.

\paragraph{Randomized Encodings}
We begin by generating randomized encodings of the columns being used for the join-keys. For example,
\iffullversion
$$
	\texttt{select }* \texttt{ from } X \texttt{ inner join } Y \texttt{ on } X_1 = Y_1 \texttt{ and } X_2 = Y_3
$$

\else
selecting all columns of $X$ and $Y$ where $X_1 = Y_1 \texttt{ and } X_2 = Y_3$.
\fi
In this case there are two join-keys, $X_1,X_2$ from $X$ and $Y_1,Y_3$ from $Y$. The protocol has \Party{0} learn the randomized encoding for each row of $X$ and \Party{1} learn them for $Y$. Importantly, is that after a previous join operation, some (or all) of the rows being joined can be \texttt{NULL}. We require that the randomized encodings of these rows not reveal that they are \texttt{NULL}. For table $X$, a special column $\XNull$ encodes if for each row is logically \texttt{NULL}. The \f{encode} functionality will then return a random encoding for all \texttt{NULL} rows. Specifically, the parties will send $(\textsc{Encode}, \{(\share{\XNull}, \share{X_{j_1}||...||X_{j_l}}, \Party{0}), (\share{\YNull}, \share{Y_{k_1}||...||$ $Y_{k_l}}, \Party{1})\})$ to \f{encode} where $j_1,...,j_l$ and $k_1,...,k_l$ index the join-keys of $X$ and $Y$. Let $\mathbb{E}_x,\mathbb{E}_y\in(\{0,1\}^{\ell})^n$ be the encodings that \Party{0} and \Party{1} respectively receive from \f{encode}.

For correctness, we require the encoding bit-length $\ell$ to be sufficiently large such that the probability of a collision between encodings is statistically negligible. Given that there are a total of $D=2n$ encodings, the probability of this is at most $2^{-\ell+2\log_2 D-1}$ which we require to be less than $2^{-\lambda}$, therefore $\ell\geq \lambda+2\log_2 D -1$. Our implementation uses $\lambda=40$ and $\ell\in\{80,100\}$ depending on $D$.

% Following the same logic as before, the probability that $x'$ collides with some $y'$ is $2^{-\sigma}n_1n_2$ and therefore the resulting encodings are uniformly distributed and unique with overwhelming probability.

%\todo{Pr. of collision.}

%Putting everything together, the parties sample $E\gets\{0,1\}^{m\times \sigma}$ if $m>\sigma$ and $E=I$ otherwise. For $i\in[n]$ and $Z\in\{X[i],Y[i]\}$, let  $\share{Z_{j_1}},...,\share{Z_{j_l}}$ be the join-keys of row $i$. A secret shared value $r\gets\{0,1\}^\sigma$ is sampled and the parties compute $\share{z'}:=\share{Z_{j_1}||...||Z_{j_l}}E \oplus \share{\overline{b}}\share{r}$. \share{z'} is sent to \f{encode}. Party \Party{0} receives the randomized encodings $\mathbb{E}_x$ for the join-keys of $X$ and \Party{1} receives the encodings $\mathbb{E}_y$ for $Y$.

\paragraph{Constructing the Cuckoo Table}

The next phase of the protocol is for \Party{1} to construct a secret shared cuckoo table for $Y$ where each row is inserted based on its encoding in $\mathbb{E}_y$. \Party{1} locally inserts the encodings $\mathbb{E}_y$ into a plain cuckoo hash table $t$ with $m$ slots using the algorithm specified in \sectionref{sec:prelim}. We assume two hash functions are used. \Party{1} samples an injective function $\pi : m\rightarrow m$ such that  $t[j]=\mathbb{E}_y[i]$, then $\pi(j)=i$.
\iffullversion
 That is, $\pi$ defines the mapping from each row's original position in the table $Y$ to the corresponding position in the cuckoo table $t$.
\fi

Parties \Party{0} and \Party{1} convert $\share{Y}$ to $\shareTwo{Y}$ such that \Party{0} holds $\shareTwo{Y}_0$.
\Party{1} sends $(\textsc{SharedSwitch}, \pi, \shareTwo{Y}_1)$ to \f{switch} and \Party{0} sends $(\textsc{SharedSwitch}, \shareTwo{Y}_0)$.
In response \f{switch} sends $\shareTwo{\ytable}_{1}$ to \Party{1}  and $\shareTwo{\ytable}_{0}$ to \Party{2}.
It is now the case that $\ytable$ is a valid secret shared cuckoo hash table of $Y$.
\iffullversion
 In particular, for a given row $Y[i]$ with encoding $e=\mathbb{E}_y[i]$, there exists a $j\in \{h_1(e),h_2(e)\}$ such that  $\ytable[j] = Y[i]$. Here, the $h_0,h_1$ functions are hash functions used to construct the cuckoo table $\ytable$. Another important observation is that $\pi$ is a permutation and therefore the more efficient permutation protocol can be used in place of the universal switching protocol.

We note that some of the columns of the tables may be secret shared in arithmetic group as opposed to binary shares. In this case the switching network will use the appropriate arithmetic operation as note in \sectionref{sec:switch}.
\fi

\paragraph{Selecting from the Cuckoo Table}

The next phase of the protocol is for each row of $X$, select the appropriate rows of $\ytable$ so the keys can be compared. \Party{0} knows that if the join-keys of the $X[i]$ row will match with a row from $Y$, then this row will be at $\ytable[j]$ for some $j\in \{h_1(e),h_2(e)\}$ where  $e=\mathbb{E}_x[i]$.

To obliviously compare these rows, \Party{0} will construct two switching networks with programming $\pi_1,\pi_2 : n\rightarrow m$ such that if $h_l(\mathbb{E}_x[i])=j$ then $\pi_l(i)=j$. Each of these will be used to construct the tables $\shareTwo{\widehat{Y}^1},\shareTwo{\widehat{Y}^2}$ which are the result of applying the switching networks $\pi_1,\pi_2$ to $\shareTwo{\ytable}$.
\iffullversion
In particular, for the $i$th row $X[i]$ it is now the case that if $X[i]$ has a matching row in $Y$ then it will be contained at  $\widehat{Y}^1[i]$ or $\widehat{Y}^2[i]$.
\fi

\paragraph{Inner Join}

Given the three secret shared tables $\share{X},\shareTwo{\widehat{Y}^1},\shareTwo{\widehat{Y}^2}$ as described above, the parties do a linear pass over the $n$ rows to construct the join between $X$ and $Y$. Recall that the inner join consists of all the selected columns from the rows $X[i],Y[j]$ where  the join-keys of the rows $X[i]$ and $Y[j]$ are equal.

If row $X[i]$ has a matching row in $Y$ then this row will occupy either ${\widehat{Y}^1}[i]$ or ${\widehat{Y}^2}[i]$. To determine which, the parties input the secret shares of these rows to an MPC protocol where the join-keys are compared. For each $i$ and rows ${\widehat{Y}^1}[i],{\widehat{Y}^2}[i]$ the bits $c_1[i],c_2[i]$ are generated where $c_l[i]=1$ iff the join-keys of ${\widehat{Y}^l}[i]$ are equal to that of $X[i]$. The MPC circuit then computes  $Y'[i]:=c_1[i]{\widehat{Y}^1}[i]\oplus c_2[i]{\widehat{Y}^2}[i]$ and $\YNull'[i]:=\YNull'[i]\vee \neg(c_1[i]\oplus c_2[i])$. That is, $Y'[i]$ is a \Null row if it was already \Null or none of the comparisons were equal. % $c[i]$ encodes whether the $i$th row of $Y'$ is not \texttt{NULL}. If so, then $Y'[i]$ is the row of $Y$ which has matching join-keys with $X[i]$.

Next, the \texttt{where} clause further filters the output table as a function of $Y'[i]$ and $X[i]$.
\iffullversion
For example, the query may specify that only rows where $Y_2'[i] + X_3[i] > 22$  are to be selected. Regardless of the exact where clause, the
\else
The
\fi
MPC protocol sets the \texttt{NULL}-bit of the final output table $Z$ as $\ZNull[i] :=\XNull[i] \vee \YNull'[i] \vee \neg  P(Y'[i], X[i])$ where $P$ is the predicate function specified by the \texttt{where} clause.
Finally, the computation specified by the \texttt{select} query is performed, e.g. copying the columns of $X,Y$ or computing a function of them.

%Specifically, the columns of the output table can either be directly copied from the input tables $X,Y$ or can be a function of the given row.
%\iffullversion
%For example, the query could be of the form
%$$
%\texttt{select } X_1, Y_2 + X_3 \texttt{ from } X \texttt{ inner join } Y \texttt{ on } X_1 = Y_1
%$$
%In this case the first column of the output will be $X_1$ while the second column will consist of the second column of $Y$ plus the third column of $X$.
%\fi
%In general we view the \texttt{select} clause as a function which takes the rows $X[i]$ and $Y'[i]$ and computes a new row with the specified columns.

\iffullversion
\paragraph{Optimizations}
Several optimizations can be applied to this protocol. First, observe that only columns of $Y$ which explicitly appear in the query need to be input to the switching networks. This reduces the amount of data to be sent and improves performance. Secondly, when comparing the join-columns, instead of computing the equality circuit between all of these columns it suffices to compare the randomized encodings. In the event that the join-column(s) contain many bits, comparing the encodings can reduce the size of the equality circuit. In addition, observe that including columns from $X$ in the output table is essentially free due to these secret share columns simply being copied from $X$. Leveraging this the queries can be optimized by ensuring that the majority of the output columns are taken from $X$. Moreover, if a join-column from $Y$ is in the \texttt{select} clause, this output column can be replaced with the matching column in $X$.

Also observe that the computation perform heavily lends itself to SIMD instructions. That is, the same computation is repeatedly applied to each row of the output table. Modern MPC protocol such as the ABY$^3$ framework \cite{aby3,highthroughput} are optimized for this setting and can process billions of binary circuit gates per second\cite{highthroughput}. In addition the ABY$^3$ framework can switch between using binary and arithmetic circuits based on which is most efficient for the given computation.
\fi

\paragraph{Left/Right Join}

A left join query is similar to an inner join except that all of the rows from the left table $X$ are included. All rows that are in the inner join are computed as before. For rows only in $X$, the bit $\YNull'[i]$ will equal one and is used to initialize the missing columns from $Y$ to a default, typically \texttt{NULL}. A right join can be implemented symmetrically.

\paragraph{Union and Set Minus}

Our framework is also capable of computing the union of two tables with respect to the join-keys. Specifically, we define the union operator as taking all of the rows from the left table and all of the rows from the right table that would not be present in the inner join. First we compute $Y\backslash X$ by only including $X[i]$ if $Y'[i]$ is \Null, i.e. $X[i]$ has no matching row in $Y$. The union of $X$ and $Y$ is then constructed as $(Y\backslash X) || X$ where the $||$ operator denotes the row-wise concatenation of $X$ to the end of $Y\backslash X$.

\paragraph{Full Join}

We construct a full join as $(X$ left join $Y)$ union $Y$. The left join merge the rows in the intersection and the union includes the missing rows of $Y$. The overhead of this protocol is effectually twice that of the other protocols.

We note that under some restrictions on the tables being joined, a more efficient protocol for full joins can be achieved. We defer an explanation of this technique to \sectionref{sec:threatlog}.

\paragraph{Security} The simulation of these protocols directly follow from the composibility of the subroutines \f{encode}, \f{switch} and \f{mpc}. First, the output of \f{encode} simply outputs random strings and it is therefore straightforward to simulate.  \f{switch} and \f{mpc} both output secret shared values. Finally, correctness is straight forward to analysis and holds so long as there is no encoding collisions and cuckoo hashing succeeds. Parameters are chosen appropriately so these failure events happen with probability at most $2^{-\lambda}$.

\begin{figure}[ht]
	\framebox{\begin{minipage}{0.95\linewidth}
			Parameters: Table size $n$. For all command, $X,Y$ are tables and $\{X_j \mid j\in J\}$ and $\{Y_k \mid k\in K\}$ are the join-keys of $X$ and $Y$ respectively. $S$ is the \texttt{select} function and $P$ is the \texttt{where} predicate.\\

			{\bf [Map]} Upon receiving command $(\textsc{Map},\share{X},J, \share{Y}, K)$ from all parties. %Let $n_x$ and $n_y$ be the number of rows $X$ and $Y$ has respectively.
			\begin{enumerate}
				\item The parties send $(\textsc{Encode}, \{(\share{\XNull}, \share{X_{J_1}||...||X_{J_l}}, $ $\Party{0}), (\share{\YNull}, \share{Y_{K_1}||...||Y_{K_l}}, \Party{1})\})$ to \f{encode} where $l=|J|=|K|$. \Party{0} receives $\mathbb{E}_x$ and \Party{1} receives $\mathbb{E}_y$ from \f{encode}.

				\item \Party{1} constructs a cuckoo hash table $t$ for the set $\mathbb{E}_y$. Define $\pi_0$ such that $\pi_0(j)=i$ where $\mathbb{E}_y[i]=t[j]$.

				\item \Party{0} and \Party{1} convert \share{Y} to \shareTwo{Y}. \Party{1} sends $(\textsc{SharedSwitch}, \pi_0, \shareTwo{Y}_1)$ to \f{switch} and \Party{0} sends $(\textsc{SharedSwitch}, \shareTwo{Y}_0)$.
				\Party{1} receives $\shareTwo{\ytable}_{1}$ and \Party{2} receives $\shareTwo{\ytable}_{0}$ from \f{switch}.

				\item Let $h_1,...,h_w$ be the cuckoo hash functions. \Party{1} defines $\pi_1,...,\pi_w$ such that $\pi_l(i)=j$ where $h_l(\mathbb{E}_x[i])=j$.

				\item For $l\in[w]$, \Party{1} sends $(\textsc{SharedSwitch}, \pi_l, \shareTwo{T}_1)$ to \f{switch} and \Party{2} sends $(\textsc{SharedSwitch}, \shareTwo{T}_0)$. \Party{0} receives $\shareTwo{\widehat{Y}^l}_0$ and \Party{1} receives $\shareTwo{\widehat{Y}^l}_1$ from \f{switch}.

				\item The parties convert  $\{\shareTwo{\widehat{Y}^l}\}_{l\in[w]}$ to $\{\share{\widehat{Y}^l}\}_{l\in[w]}$ and output them.
			\end{enumerate}
		\end{minipage}}
		\caption{Join protocols $\proto{join}$.}
	\label{fig:full_proto}
\end{figure}

\begin{figure}[ht]
	\framebox{\begin{minipage}{0.95\linewidth}
%			{\bf [Compare]} Upon receiving command $(\textsc{Compare},\share{X},$ $J, \{\share{\widehat{Y}^l}\}_{l\in[w]}, K)$ from all parties. The parties have \f{mpc} evaluate the following circuit:
%			\begin{enumerate}
%				\item For $l\in[w]$ and $i\in[n]$, $\share{c_l}[i]:= \wedge_j( \share{X_{J_j}}[i]=\share{\widehat{Y}^l_{K_j}}[i])$.
%				$\share{c}[i]:= \vee_l \share{c_l}[i]$ and $\share{Y'}[i]:= \oplus_l { \share{c_l}[i] \cdot \share{\widehat{Y}^l}[i]}$.
%
%				\item Output $\share{c}$ and \share{Y'}.
%			\end{enumerate}
%

			{\bf [Join]}  Upon receiving command $(\textsc{Join}, type, \share{X},$ $J, \share{Y}, K, S, P)$ from all parties.
			\begin{enumerate}
				\item The parties send $(\textsc{Map},\share{X},J, \share{Y}, K)$ to \proto{join} and receive $\{\share{\widehat{Y}^l}\}_{l\in[w]}$.

				\item The parties have \f{mpc} evaluate the following circuit:
				For $l\in[w]$ and $i\in[n]$, $\share{c_l}[i]:= \wedge_j( \share{X_{J_j}}[i] \overset{?}{=} \share{\widehat{Y}^l_{K_j}}[i])$.
				For  $i\in[n],$ $\share{Y'}[i]:= \oplus_l { \share{c_l}[i] \cdot \share{\widehat{Y}^l}[i]}$ and $\share{\YNull'}[i]:= \share{\YNull'}[i]\wedge \neg(\oplus_l \share{c_l}[i])$.

				\item If $type=\textsc{InnerJoin}$, define the output table $Z$ by having \f{mpc} evaluate:
				 For $i\in [n],$ $\share{\ZNull}[i]:=\share{\XNull}[i] \vee \share{\YNull'}[i] \vee \neg P(\share{X}[i], \share{Y'}[i])$ and $\share{Z}[i]:=S(\share{X}[i], \share{Y'}[i])$.

				\item If $type=\textsc{LeftJoin}$, define the output table $Z$ by having \f{mpc} evaluate:
				For $i\in [n],$ $\share{\ZNull}[i]:=\share{\XNull}[i] \vee \neg P(\share{X}[i], \share{Y'}[i])$ and $\share{Z}[i]:=S(\share{X}[i], \share{Y'}[i])$.

				\item If $type=\textsc{Union}$, define the output table $Z$ by having \f{mpc} evaluate:
				For $i\in [n], \share{\ZNull}[i]:= \share{\XNull}[i] \vee \neg P(\share{X}[i], \Null)$ and $\share{Z}[i]:=S(\share{X}[i], \Null)$.

				For $i\in [n+1,2n], \share{\ZNull}[i]:= \share{\YNull}[i] \vee \neg \share{\XNull}[i] \vee \neg P(\Null, \share{Y'}[i])$ and $\share{Z}[i]:=S(\Null, \share{Y'}[i])$.

				\item If  $type=\textsc{FullJoin}$, all parties sending $(\textsc{Join}, \textsc{LeftJoin}, \share{X}, J, \share{Y}, K, S', P)$ to $\proto{join}$ and receiving $\share{X'}$ in response. They then send $(\textsc{Join}, \textsc{Union}, \share{X'}, J, \share{Y}, K, S'', P')$  to $\proto{join}$ and output the response, where $S',S''$ and $P'$ are appropriately updated version of $S,P$.
			\end{enumerate}
	\end{minipage}}
	\caption{Join protocols $\proto{join}$ continued.}
	\label{fig:full_proto2}
\end{figure}

\begin{figure}[ht]
	\framebox{\begin{minipage}{0.95\linewidth}
			{\bf [Join]}  Upon receiving command $(\textsc{Join}, type, \share{X},$ $J, \share{Y}, K, S, P)$ from all parties. Let $n_X$ and $n_Y$ denote the number of rows in $X,Y$ respectively.
			\begin{enumerate}
				\item Define $\textsc{Keys}(X,J,i)=(X_j[i])_{j\in J}$.

				\item If the collections $\{ \textsc{Keys}(X,J,i) \mid i\in n \wedge \XNull[i]=0 \}$ or $\{\textsc{Keys}(Y,K,i) \mid i\in n\wedge \YNull[i]=0 \}$ contains duplicates, output $\bot$.

				\item If $type=\textsc{InnerJoin}$, let the rows of $Z$ be $\{ S(X[i],Y[j]) \mid \exists i,j\text{ s.t. }
				\neg\XNull[i]\wedge \neg\YNull[j]\wedge \textsc{Keys}(X,J,i)\overset{?}{=}\textsc{Keys}(Y,K,j) \wedge P(X[i],Y[i]) \}$ along with zero or more \Null rows s.t. $Z$ has $n_X$ rows.

				\item If $type=\textsc{LeftJoin}$, let the rows of $Z$ be $\{ S(X[i],Y[j]) \mid \exists i,j\text{ s.t. }
				\neg\XNull[i]\wedge \neg\YNull[j]\wedge \textsc{Keys}(X,J,i)\overset{?}{=}\textsc{Keys}(Y,K,j) \wedge P(X[i],Y[i])\}
				\cup
				\{ S(X[i],\Null) \mid \exists i,\forall j\text{ s.t. } \neg\XNull[i]\wedge \textsc{Keys}(X,J,i) \neq\textsc{Keys}(Y,K,j) \wedge P(X[i],\Null)\}$ along with zero or more \Null rows s.t. $Z$ has $n_X$ rows.

				\item If $type=\textsc{Union}$, let the rows of $Z$ be $\{ S(X[i],\Null) \mid \exists i \text{ s.t. }
				\neg\XNull[i]\wedge P(X[i],\Null)\}
				\cup
				\{ S(\Null, Y[i]) \mid \exists i,\forall j\text{ s.t. } \neg\YNull[i]\wedge \textsc{Keys}(X,J,j) \neq\textsc{Keys}(Y,K,i) \wedge P(\Null, Y[i])\}$
				along with zero or more \Null rows s.t. $Z$ has $n_X+n_Y$ rows.

				\item If  $type=\textsc{FullJoin}$, let the rows of $Z$ be $\{ S(X[i],Y[j]) \mid \exists i,j\text{ s.t. }
				\neg\XNull[i]\wedge \neg\YNull[j]\wedge \textsc{Keys}(X,J,i)\overset{?}{=}\textsc{Keys}(Y,K,j) \wedge P(X[i],Y[i]) \}
				\cup
				\{ S(\Null, Y[i]) \mid \exists i,\forall j\text{ s.t. }
				\neg\YNull[i]\wedge \textsc{Keys}(X,J,j)\neq\textsc{Keys}(Y,K,i) \wedge P(\Null, Y[i])\}
				\cup
				\{ S(X[i], \Null) \mid \exists i,\forall j\text{ s.t. }
				\neg\XNull[i]\wedge \textsc{Keys}(X,J,i)\neq\textsc{Keys}(Y,K,j) \wedge P(X[i], \Null)\}$

				\item Output $\share{Z}$ to the parties.
			\end{enumerate}
	\end{minipage}}
	\caption{Join functionality $\f{join}$.}
	\label{fig:full_ideal}
\end{figure}

\subsection{Non-unique Join on Column}

When values in the join-column are not unique within a single table, the security guarantees begin to erode. Recall that the randomized encodings for $X,Y$ are revealed to \Party{0}, \Party{1} respectively. Repeated values in the join-columns will lead to duplicate randomized encodings and therefore reveal their location. Learning the distribution of these duplicates reveals that the underlying table has the same distribution. In the event that only one of the tables contains duplicates, the core protocol can naturally be extended to compute the various join operations subject to \Party{0} learning the duplicate distribution. This is achieved by requiring the left table $X$ contain the duplicates rows. After learning the randomized encodings for this table \Party{0} can program the switching networks appropriately to query the duplicate locations in the cuckoo hash table.

When both tables contain duplicates we fall back to a less secure protocol architecture. This is required due to the cuckoo table not supporting duplicates. First, \Party{1} samples two random permutations $\pi_0,\pi_1$ and computes $X'=\pi_1(X),Y'=\pi_2(Y)$ using the oblivious permutation protocol. \Party{0} then learns all of the randomized encodings for the permuted tables $X'$ and $Y'$. Given this, \Party{0} can compute the size of the output table and inform the other two parties of it. Alternatively, an upper bound on the output table size can be communicated. Let $n'$ denote this value. \Party{0} can then construct two switching networks which map the rows of $X'$ and $Y'$ to the appropriate rows of the output table. The main disadvantage of this approach is that \Party{0} learns the size of the output, the distribution of duplicate rows and how these duplicate rows are multiplied together. However, unlike \cite{LTW13} which takes a conceptually similar approach, our protocol does not leak any information to \Party{1} and \Party{2}, besides the value $n'$.

%When both tables contain duplicates we fall back to the less secure protocol architecture of Laur et al.\cite{LTW13}. In particular, this style of protocol  performs an oblivious shuffling of the table rows and then reveals all of the randomized encodings to all of the parties. Given this information the parties can construct the desired join. We suggest that the performance of these two primitive can be improved over \cite{LTW13} by 1) implementing the random shuffle using two random permutation networks from \sectionref{sec:switch} where party 0 and 1 both privately sample one of the permutations. 2) Replace the use of AES with our improved randomized encodings (LowMC and random binary matrix). Given these optimization the overhead of these protocols should be comparable to our standard join techniques. The major shortcoming of this approach is that the duplicate distribution and the size of join is revealed to all of the parties. As discussed in the related work section, this can limit several important application such as threat log comparison.

\subsection{Revealing Results}

Revealing a secret shared table $\share{X}$ requires two operations. First observe that the data in the \texttt{NULL} rows is not cleared out by the join protocols. This is done as an optimization. As such naively reconstructing these rows would lead to significant leakage. Instead $X[i]$  is updated as $X[i]=(\neg\XNull[i])\cdot X[i]$. %This ensures that all \texttt{NULL} rows have a deterministic value and therefore can be simulated.
The second operation is to perform an oblivious shuffle of the rows. This operation randomly reorders all the rows without revealing the ordering to any of the parties. This step is necessary since the original ordering of the result table is input-dependent. For example, say $X$ is a list of patents info, $Y$ is patent billing status, and $Z$ is a list of patent diseases. Say we reveal $\texttt{select } X.name, Y.balance \texttt{ from } X,Y \texttt{ on } X.id=Y.id$ and $\texttt{select } X.gender, Z.desease \texttt{ from } X,Z \texttt{ on } \allowbreak X.id=Z.id$. Without reordering you could connect $X.name, X.gender, Y.balance$ and $Z.desease$ by the row index and infer secret information. However, by randomly shuffling this connection is destroyed and the reveal can be simulated.

%Concretely, this ordering can be used to correlate between two different output tables. For example, the row ordering of say $X\cap Y$ and $X\cap Z$ will be the same up to some rows being \texttt{NULL} while other may not. %In the case of sets this ordering does not reveal additional information since it can be inferred given the ideal output, e.g. set intersection, then the shuffling can be omitted. However, this is not true in general when the items being joined are key-value pairs.

\section{Computing a Function of a Table}\label{sec:card}

In addition to join queries, our framework can perform computation on a single secret shared table. For example, selecting $X_1+X_2$  where $X_3>42$. For each row $i$ we generate the corresponding output row $Z[i]$ by computing the new \texttt{NULL}-bit as $\ZNull[i] := \XNull[i] \vee P(X[i])$ where $P(\cdot)$ is the \texttt{where} predicate. The new column(s), e.g. $Z_1=X_1+X_2$, can then be constructed in a straightforward MPC protocol, e.g. \cite{aby3,highthroughput}. The key property is that all of the operations are with respect to a single row of $X$, allowing them to be evaluated in parallel.

Our framework also considers a second class of functions on a table that allow computation between rows. For example, computing the sum of a column. We refer to this broad class of operations as an aggregation function. Depending on the exact computation, various levels of efficiencies can be achieved. Our primary approach is to employ the ABY$^3$ framework \cite{aby3} to express the desired computation in an efficient way and then to evaluate the resulting circuit. Next we highlight a sampling of some important aggregation operations:
\begin{itemize}
	\item Sum: For a column $\share{X_j},$ compute $\share{s}=\sum_i \share{X_j}[i]$ where $X_j[i]\in \mathbb{Z}_{2^\ell}$ and $i$ indexes only non-\Null rows. The parties compute $\shareA{s}:=\sum_{i\in [n]}\textsf{B2A}((\neg\share{\XNull}[i]) \cdot \share{X_j}[i])$ where $\textsf{B2A}$ is the boolean to arithmetic share conversion of \cite{aby3}. In total this requires $2n\ell$ binary gates and $\ell+1$ rounds\cite{aby3}. The parties can then convert $\shareA{s}$ back to \share{s} if desired. %Alternatively, $n\ell\log \ell$ binary gates and $\log \ell$ rounds can be used\cite{aby3}.

	\item Count/Cardinality: Here, we consider two cases. 1) In the general case there is an arbitrary table over which the count is being computed. This is performed by computing $\shareA{s}:=\sum_{i\in [n]}\textsf{B2A}(\neg\share{\XNull}[i])$
	\iffullversion
	The more efficient bit injection protocol\cite{aby3} can be used to convert each bit to an arithmetic sharing in a constant number of rounds. In particular, the malicious secure bit injection protocol provided in \cite{aby3} is suggested due to it reducing the overall communication.
	\fi

	2) Consider case where some of the parties should learn the cardinality of a join without a \texttt{where} clause. %The core idea is that given the randomized encodings for these tables $\mathbb{E}_x,\mathbb{E}_y$, the cardinality/count is exactly $|\mathbb{E}_x \cap \mathbb{E}_y|$.
	First, w.l.o.g. let us assume that \Party{2} should learn the cardinality. The randomized encodings $\mathbb{E}_x,\mathbb{E}_y$ are respectively revealed \Party{0} and \Party{1}  as done in the standard join protocol. These encodings are then sent to \Party{2} in a random order. \Party{2} outputs $|\mathbb{E}_x \cap \mathbb{E}_y|$ as the count/cardinality. In the event that \Party{0} or \Party{1} should also learn the cardinality, \Party{2} sends $|\mathbb{E}_x \cap \mathbb{E}_y|$ to them.

	\item  Min/Max: We propose a recursive algorithm where the min/max of the first and second half of the rows is recursively computed. The final result is then the min/max of these two values.  Concerning \texttt{NULL} rows, the corresponding value can be initialized to a maximum or minimum sentential value which guarantee that the other value will be propagated. The overall complexity of this approach is $O(n\ell)$ binary gates and $O(\ell log n)$ rounds when using a basic comparison circuit\cite{aby3}.
\end{itemize}

More generally, any polynomial time function can generically be expressed using the ABY$^3$ framework\cite{aby3}. However, the resulting efficiency may not be adequate for practical deployment.

\iffalse
\section{Beyond Three Parties}
In some settings where many parties are providing tables to be computed on, the three party requirement with at most one corruption may not be adequate. In particular, a larger corruption threshold such as 3-out-of-5 or 4-out-of-7 may be desirable. Our protocol can naturally be extended to these setting with some caveats. First, the binary circuit based MPC protocol\cite{highthroughput}, which our standard three party protocol relies on, can be naturally extended to these larger (honest majority) corruption thresholds. One limitation is that the number of shares is exponential in the number of parties.  Alternatively, several other protocol without this limitation such as\cite{DBLP:conf/crypto/ChidaGHIKLN18} have been proposed. Another implementation challenge is how to instantiate share conversion that some of the aggregation steps rely on. In the three party case \cite{aby3} provides specification. We note that such share conversions can generically be implemented using an MPC protocol but leave efficient instantiations to future work.

This change immediately implies that no minority set of corrupted parties can decrypt/reconstruct the secret shared tables. However, our three party protocol also requires \Party{0} and \Party{1} learning the randomized encodings in the clear. If the adversary learns both sets of randomized encodings they can infer information about the cardinality of the join. As such, our protocol offers two levels of security. When the adversary corrupts \Party{0} and \Party{1} the cardinality of $\mathbb{E}_x \cap \mathbb{E}_y$ is revealed. Otherwise, if an honest majority is present then the view of the corrupted parties can be simulated in the semi-honest setting.

\fi


\section{Abstract}
\label{sec:abstract_en}

Financial companies from all around the world have started to focus their investments in quantitative and algorithmic funds. Those methods run in server applications that execute automatic trades. It is important to distinguish high frequency trading from machine learning trading. The latter is used and analyzed in detail in the present work.

This project explains the development of a trading strategy on Bitcoins based on machine learning techniques. A pipeline proposal is shown which is based on Lopez de Prado's book (\cite{lopez_de_prado}). Some modifications are introduced in the book's pipeline to adjust a momentum primary model on Bitcoins, and to incorporate and study features that would let estimate the size of the primary model bets (secondary model to be trained on top of the first model). The range of features to analyze goes from financial metrics derived from Bitcoin prices and volumes, to Bitcoin and blockchain related features and finally social indexes which incorporate interest and animosity towards Bitcoin itself.

The pipeline proposed in \cite{lopez_de_prado} and implemented in this thesis rigorously handles the dataset, the involved models and finally the posterior backtesting strategies. Details about statistical foundation of the involved methods, algorithm complexity and implementation and domain explanations (such as those related to cryptocurrencies) can be found. The pipeline allows to gather enough information to compare and decide whether a propose strategy is good enough to be implemented. We will use this to compare models that introduce microstructure indexes such as SADF (Supremum Augmented Dickey Fuller) in comparison and conjunction with social indexes.


\documentclass[11pt]{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{pifont}
\usepackage{setspace}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{color}
\usepackage{sectsty}
% \usepackage{raisebox}
% \usepackage{geometry}
% \usepackage{fancyhdr}
% \pagestyle{fancy}
% \lfoot{\emph{CV: Christopher Harman after discussion with Dr. S. Flockton}}
\definecolor{darkblue}{rgb}{0.,0.,0.7}
\sectionfont{\color{darkblue}}
\subsectionfont{\color{darkblue}}

\oddsidemargin  -0.25in
\evensidemargin 0.25in
\textwidth      7.0in
\headheight     0.0in
\topmargin      -0.8in
%\textheight=9.5in
\textheight=10.0in
\onehalfspacing

\begin{document}
\thispagestyle{empty}

\title{Clear Cut: Algorithm Design}
% \author{Christopher Harman}
\maketitle
\tableofcontents

\newpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%% INPUT DATA DESCRIPTION %%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Input data}
The input data is an image of arbitrary size, $M \times N$, where $M$ is the horizontal dimension and $N$ is the vertical dimension of the image. Most images have three channels (RGB = ``Red-Green-Blue"), this means an array with three $M \times N$ arrays. Images with exif data are checked and modified so that the image is imported in the intended orientation, e.g. in case the image is a photo taken from a phone camera.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%% IMAGE SIZE REDUCTION %%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Image size reduction}
To improve the efficiency of the edge detection algorithm, the image size is reduced to that the average image dimension is less than 500 pixels in length. The final image size to be thrown into the algorithm is $M_{\text{eff}} \times N_{\text{eff}}$, where
\begin{equation}
\dfrac{M_{\text{eff}} N_{\text{eff}}}{2}<500~\text{pxl} \text{.}
\label{im_reduction_condition}
\end{equation}
The image is currently reduced slowly by max pooling. This relies on calculating the smallest kernel size and repeating the max pooling process until the condition in Equation~\ref{im_reduction_condition} is satisfied. This can be implemented more effectively by calculating the smallest kernel that would satisfy the condition in Equation~\ref{im_reduction_condition} after just one implementation of max pooling.

As it currently exists, the process consists of determining the smallest number that divides the height (width) of the image to return an integer. If there is no such factor, i.e. the height (width) is a prime number, the image is cropped by removing the single pixel row (column) at the $M_{\text{eff}}^{th}$ ($N_{\text{eff}}^{th}$) index. By definition this would result in that image dimension being divisible by 2, which is then the number of pixels of the smallest kernel in that dimension. Throughout the image reduction process, the values ~{($M_{\text{eff}}$,~$N_{\text{eff}}$,~$M_{\text{kernel}}$,~$N_{\text{kernel}}$)} are stored in a Python dictionary to keep a record of the image reduction history.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%% EDGE DETECTION PROCEDURE %%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Edge detection procedure}
\subsection{\label{sec:grad_img}The gradient image}
\begin{center}
\texttt{For Python code, refer to the {\bf traceObjectsInImage()} function in edgeUtility.py}
\end{center}
In order to determine the edges of an image, we must choose a criteria to distinguishing a single pixel located at $(i_0,j_0)$ as being part of ``an edge''. We would refer to such pixels as ``edge pixels'' or ``non-edge pixels'' in future. This is achieved by mathematically comparing its value to those of its surrounding pixels. The simplest method to adopt is to consider only the pixels neighbouring pixels $(i,j)$, i.e. any combination of pixels that satisfy
\begin{equation}
i_0-1 \leq i \leq i_0+1\text{, and }j_0-1 \leq j \leq j_0+1\text{.}
\end{equation}
A more advanced technique could extend the range of neighbouring pixels. For a pixel not located on the perimeter of the image, it would have 8 neighbouring pixels.

The mathematical quantity we are interested in is the difference in value between a selected pixel and it's neighbouring pixels. For the simple case, an edge pixel would be identified as having a large difference in value between any adjacent neighbouring pixel and itself. This simplest case works well for an image with sharp edges, but not so well for an image with blurry edges. With eight neighbouring pixels, we would have to calculate eight gradients per pixel. It turns out that we can more efficient in calculating the gradients and only count four instead. This is because when the neighbouring pixel has the gradient calculated, it would be the same magnitude but a different sign. The sign is irrelevant for determining the ``sharpness'' between pixels. This means that the eight gradient per pixel method would double-count the number of gradients; therefore we need only calculate four gradients per pixel instead.

Thus for a generic pixel at $(i,j)$, the difference in value between pixels $(i-1,j-1)$, $(i+1,j)$, $(i,j+1)$, and $(i+1,j+1)$ is calculated. These four directions are a choice and are not expected to make a difference in the final result. Note that not all gradients can be calculated if the pixel under consideration sits along the perimeter of the image. That each pixel in the ``image space'' has four unique gradients calculated, means that we will obtain a ``gradient space'' of size $2M_{\text{eff}} \times 2N_{\text{eff}}$.

\subsection{Dealing with image channels}
\subsubsection{Obtaining gradient images}
\begin{center}
\texttt{For Python code, refer to the {\bf traceObjectsInImage()} function in edgeUtility.py}
\end{center}
Each of the RGB channels have the edge detection algorithm specified in Section~\ref{sec:grad_img} applied to them. This is achieved by writing out the results to a single array of size $6M_{\text{eff}} \times 2N_{\text{eff}}$, where each channel's gradient image is offset as follows:
\begin{itemize}
  \item the Red channel gradient array exists in the domain $(0,~2M_{\text{eff}}-1)$,
  \item the Green channel gradient array exists in the domain $(2M_{\text{eff}},~4M_{\text{eff}}-1)$, and
  \item the Blue channel gradient array exists in the domain $(4M_{\text{eff}},~6M_{\text{eff}}-1)$.
\end{itemize}
Recall that we could only reduce the number of gradients per pixel if we ignore the gradients sign. Therefore, we must take care to only add up the gradient images element-wise AFTER taking the magnitude of each element first. The initial step in edge detection is to determine whether an (rgb) pixel in the reduced image is, or is not, part of an edge; this turns each pixel into a ``yes'' or ``no'' (boolean) answer. We therefore set a numerical upper and lower limit on what is an edge pixel, and what is not. We implement this through a variable parameter called \texttt{imCut}, whereby an edge pixel is one in which the gradient lives between:
\begin{equation}
255\times \texttt{imCut}< \text{Gradient}_{\text{edge pixel}} < 255\times (1-\texttt{imCut})\text{.}
\end{equation}
We have found $\texttt{imCut} = 0.07 \pm 0.02$ to work effectively. We have found both the upper limit and lower limit are necessary to extract clear edge pixels for any image. The lower limit is understandable because it determines adjacent pixels with too similar a colour as not being an edge pixel, however, the upper limit is less trivial. We believe the upper limit is necessary to remove random fluctuations in light/darkness within an image. These may be the result of tiny defective regions of photographical film that cannot be seen by the human eye when the image view at a larger scale that the size of the defect.

\subsubsection{Returning detected edges in reduced image size}
\begin{center}
\texttt{For Python code, refer to the {\bf mergeChannelsTracedImage()} function in edgeUtility.py}
\end{center}
We now have three gradient space arrays. However, we only want one gradient space array to say whether a single (rgb) pixel in the reduced image is an edge or not. This means we that we need to merge each of the gradient arrays into a single gradient array, and then reduce this gradient array to be the same size as the reduced image. The idea behind the merge is that \textbf{the more channels that detect an edge, the more likely it is to be an edge}. This is in agreement with human perception that an orange boat on a blue sea is far more distinct object than a blue boat on a blue sea. We therefore simply add up each gradient image element-wise to form our final gradient image.

This final gradient image is still roughly double the width and double the height of the reduced image. We reduce the gradient image to the same size as the reduced image using max pooling of $(2 \times 2)$-sized kernels. Therefore the final ``edge value'' of each (rgb) pixel of the reduced image is determined by the largest gradient of that pixel and its neighbouring pixel. This in turn determines whether it is an edge pixel or not.

\subsection{Cleaning the edge data}
Now that we have two arrays of the same size, we can mask the edge image on top of the reduced image to view our current results. In doing so for a number of unique images, we observe  two main features:
\begin{enumerate}
  \item There exist spurious regions of edge pixels with a size of the order of a few pixels. \textbf{Solution: remove tiny regions of edge data.}
  \item There are edge pixels around the perimeter of objects in the reduced image, but with the occasional break, whereby no edge pixels exist. \textbf{Solution: extrapolate long regions of edge data.}
\end{enumerate}

\subsubsection{Removing small scale edge data}
\begin{center}
\texttt{For Python code, refer to the {\bf edgeKiller()} function in edgeUtility.py}
\end{center}
To accomplish this task, we first need to decide what a small scale is. This is necessary for object recognition because many complicated objects are made of smaller, perhaps even more complicated objects. Fundamentally, if an object is identified as being surrounded by a continuous border of edge pixels, then the smallest object would be three pixels in size. This because the border must be at least one pixel thick and the central pixel must not be recognised as an edge (two pixels either side of a non-edge pixel in the $x$ and $y$ direction), i.e. a square drawn around the perimeter of a $3 \times 3$ pixel grid. Practically, it is the choice of the user to specify a tolerance to the size of the object. Take the example of a cat. Users may want to extract certain features of the cat, depending on some objective scaling into the cat (see Table~\ref{cat_table}).

\begin{table}[h]
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|}
\hline
{\bf Feature/object} & {\bf User 1} & {\bf User 2} & {\bf User 3} & {\bf User 4}\\
\hline
{\bf Cat} & y & y & y & y\\
\hline
{\bf Nose} & n & y & y & y\\
\hline
{\bf Eye} & n & n & y & y\\
\hline
{\bf Iris} & n & n & n & y\\
\hline
\end{tabular}
\end{center}
\label{cat_table}
\end{table}

To this end, a user may specific their pixel tolerance using the parameter \texttt{objectTolerance}. Any edge pixels that are found within a border of size $(2\times \texttt{objectTolerance}+1) \times (2\times \texttt{objectTolerance}+1)$, where the border does not contain any edge pixels, are killed off.

% include visualisations here

\subsubsection{Expand large-length edge data}
\begin{center}
\texttt{For Python code, refer to the {\bf edgeFiller()} function in edgeUtility.py}
\end{center}
To address the issue of small gaps within edges, we extrapolate any edges subject to number of consecutive edge pixels in a given direction. The minimum number of consecutive edge pixels required for the extrapolation to happen is embodied in the parameter \texttt{edge\_bias}. The extrapolation basically runs through each edge pixels and determines the number of consecutive edge pixels in each of the eight directions $(-1,-1)$, $(-1,0)$,$(-1,+1)$, $(0,-1)$, $(0,+1)$, $(+1,-1)$, $(+1,0)$,and $(+1,+1)$. If the number of edge pixels in a direction is greater than $\texttt{edge\_bias}$, change the first non-edge pixel in that direction to an edge pixel.

% include visualisations here

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%% OBJECT EXTRACTION PROCEDURE %%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Object extraction procedure}
\subsection{Theoretical set up}
Randomly pick an initial edge pixel. Determine the directions in which the neighbouring pixel is also an edge pixel. For the directions which have an edge pixel, (recursively) determine if the pixel in that same direction is also an edge pixel until it is not an edge pixel. Note down the number of edge pixels in this direction and add it to the number of edge pixels (until a non-edge pixel is found) in the exact opposite direction. This will result in four lengths - which we will call \textit{thicknesses} - in each of the directions:
\begin{itemize}
  \item horizontal: $(-1,0)\rightarrow (+1,0)$,
  \item vertical: $(0,-1)\rightarrow (0,+1)$,
  \item positive gradient diagonal: $(-1,-1)\rightarrow (+1,+1)$,
  \item negative gradient diagonal: $(-1,+1)\rightarrow (+1,-1)$.
\end{itemize}

The shortest thickness of consecutive edge pixels at the initial edge pixel will be referred to as the \textbf{race start line}. The length of the race start line is defined to be in the $\hat{y}$-direction of the Cartesian coordinate system $\hat{C}: (\hat{x},\hat{y})$. The initial direction of the \textbf{race path} must have a non-zero $\hat{x}$-component, i.e. the initial path vector is $\vec{O}=a\hat{x}+b\hat{y}$, where $a \neq 0$.

\subsection{Definition of an object}
\begin{center}
\texttt{We define an object within an image as the set of pixels that residue\\ within an enclosed path of edge pixels. The enclosed path must return\\ through the race start line with an $\hat{\texttt{x}}$-component that has the same sign\\ as that in
	 which the path was initialised.}
\end{center}
There are two very important concepts to understand within this definition:
\begin{itemize}
  \item \textit{... an enclosed path ...}: this means we should be able to draw a continuous path of edge pixels that start and end at the same edge pixel coordinate.
  \item \textit{... $\hat{x}$-component that has the same sign ...}: this means that the path of edge pixels must have an initial path vector with an $\hat{x}$-component with the same sign as the $\hat{x}$-component of the (final) path vector that crosses past the start race line. Since the path is determined by when the path has crossed the start race line, it means the path could not have gone back on itself whereby it remains along the ``same edge''. % Need to include visualisations here of starting at the bottom of a hangmans knot, and starting at the top of the knot.
\end{itemize}

\subsection{Stepping into the edge path}
The edge path cannot be stepped by an entirely random process as the code would take too long to run and possible return nonsense (see Brownian motion). However, neither can the steps be too systematic as this may guarantee that some forks in the edges are never taken, thus losing potential objects. At each step, there must be some notion of moving in a direction roughly orthogonal to the shortest pixel thickness that is also away from pixel at the previous step. Therefore, the step is chosen to be close to the vector orthogonal to the start line for that current pixel, but with a small random fluctuation in the ($\hat{y}$-)direction of the start line.

\begin{figure}[h!]
\centering
\includegraphics[width=15.0cm]{visuals/edge_id_aids/Path_algorithm.png}
\caption{\small{Visualisation of the random path method. This image shows two paths starting at the same initial edge pixel (white square) with the same steps up $i=3$. At this point the path the random vector off the $i=3$ start line may either go into the upper or lower fork of the edge. The green cone has been drawn to detail that the random path vector has a specified length (the radius of the cone) but the random direction is constrained to within the green cone.}}
\label{singleField_PtPlot_fig}
\end{figure}

\end{document}


\documentclass[a4paper]{article}

%
% Packages
%

%% styling for this document
\usepackage{tex/docstyle}

%% Test
\usepackage{blindtext}

%
% Metadata
%

\title{A Template for a Notebook}
\author{
  Naoki Pross
}
\date{\today}

%
% Document
%

\begin{document}
\maketitle
\tableofcontents

\section{Introduction}
\blindtext
\[
  U = R \cdot I \iff
  \int_L \mathbf{E \cdot \hat{l}} \mathrm{d}l =
  R \int_{S} \sigma \mathbf{E \cdot \hat{n}} \mathrm{d}s
\]

\newpage

\section{Atomic structures}
\makefigure[A sample figure]{6cm}
\makegrid{18}

\end{document}