%Chapter 3
\chapter{Per-Pixel Calibration and \gls{3D} Reconstruction on \gls{GPU}} % Main chapter title
\label{chapterDataBasedCalibration}
%
%
%*.  aims to convert \gls{RGBD} camera streams in to XYZRGB on \gls{GPU}.
%*. Kai showed us how to convert the pinhole camera matrix into beam equation (XYZ from P) so that it could be processed on \gls{GPU}
%#. pinhole camera model
%#. \gls{GPU} process, each pixel call an independent frame shader and do processing based on its screen position.
%
%*. We want to go beyond Kai:
%#. Add radial distortion correction
%#. want more generic \gls{D} -> Z transformation
%#. align RGB value from a separate RGB sensor.

%Of all calibration systems, we think the best a moving XY plane, because:
%*. can take infinite number of calibration points
%*. fill up the field of view
%*. Dense \gls{D} -> Z transformation

%procedures:
%1. Move the slider to the nearest position.
%a. use laser to measure distance from the slider to the pattern plane, and record the distance as $\gls{worldZ}$
%b. Grab \gls{NIR}, Depth, RGB video frame
%c. Detect closest centers in \gls{NIR} + RGB frame
%d. assign XY to extracted CR of dots' centers, find high order mapping.
%e. generate dense XY for all pixels based on the high order mapping, in \gls{NIR} and RGB respectively
%
%2. move the rail to the next position, and repeat step 1.
%
%3. Unify Staggered Origin (make sure the point XY=0 is at same dot)
%4. determine per-pixel mapping coefficients of \gls{D} -> Z, Z -> XY, and generate row by column by 6 look-up table
%5. RGB values alignment based on pinhole camera model.
%
%
\gls{RGBD} cameras are famous for their \gls{3D} reconstruction application. In this chapter, we will show how to reconstruct \gls{3D} image naturally on \gls{GPU} in camera space based on raw data without calibration. Considering the deformation of raw data resulted from lens distortions and \emph{depth distortion}, we propose a per-pixel calibration method based on a rail calibration system. Instead of using a pinhole-camera model, a two-dimensional high-order polynomial mapping model is determined and employed for lens distortions removal. Corresponding data collection procedures and look-up table based undistorted real-time world space \gls{3D} reconstruction on \gls{GPU} are discussed in detail. Color values alignment is also discussed at last.
%
\section{\(\gls{RGBD}\) to \(X^CY^CZ^CRGB\)}
\label{sectionCameraSpaceReconstruction}
As described in Chap.\ref{chapterIntroduction}, there are applications like \gls{SLAM} and KinectFusion that require \(\gls{D}\) to be converted into \(X^CY^CZ^C\) coordinates on a per-pixel basis. From Chap.\ref{chapterTraditionalCalibration}, we know that the depth sensor measures \(\gls{cameraZ}\), and a pinhole camera model (more specifically the intrinsic matrix) in homogeneous coordinates offers the relationship between \(\gls{cameraZ}\) and \(\gls{cameraX}/\gls{cameraY}\) respectively. In this section, we will introduce how to generate the camera space \gls{3D} coordinates without calibration, and draw a camera space \gls{3D} reconstruction on \gls{GPU} using a \gls{KinectV2} camera.
\\\indent
%
\begin{figure}[!b]
\centering
\includegraphics[width=0.7\textwidth]{VHfieldOfView}
\caption{Field of View in Pinhole-Camera Model}
\label{VHfieldOfView}
\end{figure}%
%
The \gls{KinectV2} depth sensor measures \(\gls{cameraZ}\) in millimeter and supports its positive data in unsigned-short data-type. Those data will be automatically converted into \gls{dataTypeFloat} type with its range from 0.0f to +1.0f when uploaded onto \gls{GPU}. Considering that in practice \(\gls{D}\)s from \gls{KinectV2} are always positive whereas \(\gls{cameraZ}\)s should be always negative, we will add a negative sign in the un-scaling step to recover the \(\gls{cameraZ}\) in metric on \gls{GPU}:
\begin{equation}
\gls{cameraZ} [m, n] = - \beta \gls{D} [m, n] \, ,
\label{unscalingZc}
\end{equation}%
\noindent
where \(\beta\) constantly equals to 65535.0 (range of unsigned short in \gls{dataTypeFloat}) for all pixels. Besides the depth stream, \gls{KinectV2} supports both of the horizontal and vertical field of view (\gls{FoV}) that can help generate per-pixel \(\gls{cameraX}\) and \(\gls{cameraY}\), which share the same credits with the intrinsic parameters in a pinhole camera model. Figure~\ref{VHfieldOfView} shows an intuitive view of the horizontal and vertical FoVs (\(\gls{symHorFoV}, \, \gls{symVerFoV}\)) in a pinhole camera model, based on which we can derive the \(\gls{cameraX}\) and \(\gls{cameraY}\) values given a random \(\gls{cameraZ}\) value on the per-pixels basis. Assuming the depth sensor, with its size \gls{imRowSize} by \gls{imColumnSize}, is observing right perpendicularly to a wall where all pixels share the same \(|\gls{cameraZ}_0|\) from the sensor to the wall. Talking about the horizontal \gls{FoV} \(\gls{symHorFoV}\) only, it is easy to get the range of the camera space \gls{FoV} along
\(\gls{cameraX}\)-axis from \(-|\gls{cameraX}_{\text{MAX}}|\) to \(|\gls{cameraX}_{\text{MAX}}|\), as shown in Fig.~\ref{horizontalFoVtoColumns}. %
\begin{figure}[!t]
\centering
\includegraphics[width=0.6\textwidth]{horizontalFoVtoColumns}
\caption{map \(\gls{imageColumn}\) to \(\gls{cameraX}\) via horizontal \gls{FoV}}
\label{horizontalFoVtoColumns}
\end{figure}%
%
The horizontal range value \(|\gls{cameraX}_{\text{MAX}}|\) depends on \(|\gls{cameraZ}_0|\) and \(\gls{symHorFoV}\), given by:
%
\begin{equation}
|\gls{cameraX}_{\text{MAX}}| = |\gls{cameraZ}_0| \cdot tan(\gls{symHorFoV} / 2)
\label{cameraXMaxfromZ}
\end{equation}%
\noindent
where the pixel observing on \(\gls{cameraX} = |\gls{cameraX}_{\text{MAX}}|\) has its column address of \gls{imColumnSize}. Similarly, given a random pixel of column address \(n_0\), its horizontal view \(\gls{cameraX}_0\) could be expressed based on its own horizontal view angle \(\alpha_0\):
%
\begin{equation}
|\gls{cameraX}_{0}| = |\gls{cameraZ}_0| \cdot tan(\alpha_0)  .
\label{alphaZeroHorFOV}
\end{equation}%
\noindent
To combine eqn.~(\ref{cameraXMaxfromZ}) and eqn.~(\ref{alphaZeroHorFOV}), we get
%
\begin{equation}
\frac{\gls{cameraX}_0}{|\gls{cameraX}_{\text{MAX}}|} = \frac{tan(\alpha_0)}{tan(\gls{symHorFoV} / 2)} = \frac{n_0}{\gls{imColumnSize}/2} - 1\, ,
\label{alphaZeroHorFOV}
\end{equation}%
\noindent
which shows how to get the per-pixel \(\gls{cameraX}_0\) from \(|\gls{cameraX}_{\text{MAX}}|\) based on its column address, while \(|\gls{cameraX}_{\text{MAX}}|\) depends on the depth sensor's horizontal filed of view. It is intuitively better to change eqn.~(\ref{alphaZeroHorFOV}) a little by substituting \(|\gls{cameraX}_{\text{MAX}}|\) with eqn.~(\ref{cameraXMaxfromZ}) such that we can get eqn.~(\ref{proportionalEqnFromZtoX}), a proportional per-pixel mapping based on the column addresses from \(\gls{cameraZ}\) to \(\gls{cameraX}\).
%
\begin{equation}
\gls{cameraX} [m, n] = tan(\gls{symHorFoV} / 2) \cdot (\frac{n}{\gls{imColumnSize}/2} - 1) \cdot |\gls{cameraZ} [m, n]|\, ,
\label{proportionalEqnFromZtoX}
\end{equation}%
\noindent
where [\(m,n\)] is the discrete space \(\gls{imageRow}\) and \(\gls{imageColumn}\) coordinate of each pixel in the depth sensor. Similarly, we can also get the proportional per-pixel mapping from \(\gls{cameraZ}\) to \(\gls{cameraY}\), based on the vertical \gls{FoV} and row addresses:
%
\begin{equation}
\gls{cameraY} [m, n] = tan(\gls{symVerFoV} / 2) \cdot (\frac{m}{\gls{imRowSize}/2} - 1) \cdot |\gls{cameraZ} [m, n]| \  .
\label{proportionalEqnFromZtoY}
\end{equation}%
\noindent
Note that, \(\gls{symHorFoV}\) and \(\gls{symVerFoV}\) are constant during image processing, such that the mapping functions from per-pixel \(\gls{cameraZ}\) to per-pixel \(\gls{cameraX}/\gls{cameraY}\) totally depend on a pixel's address [\(m, n\)]. Therefore eqn.~(\ref{proportionalEqnFromZtoX}) and eqn.~(\ref{proportionalEqnFromZtoY}) could be expressed as
%
\begin{equation}
\begin{aligned}
\gls{cameraX} [m, n] &= a [m, n] \cdot |\gls{cameraZ} [m, n]|
\\%
\gls{cameraY} [m, n] &= b [m, n] \cdot |\gls{cameraZ} [m, n]|
\end{aligned}
\label{proportionalBeamEqn}
\end{equation}%
\noindent
where
%
\begin{equation}
\begin{aligned}
a[m, n] &= tan(\gls{symHorFoV} / 2) \cdot (\frac{n}{\gls{imColumnSize} / 2} - 1)
\\%
b[m, n] &= tan(\gls{symVerFoV} / 2) \cdot (\frac{m}{\gls{imRowSize} / 2} - 1) \, \, .
\end{aligned}
\label{parametersABofProportional}
\end{equation}%
\\\indent
%
\begin{figure}[!t]
\centering
\includegraphics[width=0.8\textwidth]{cameraSpaceDiagram}
\caption{Diagram for Camera Space \gls{3D} Reconstruction without calibration}
\label{cameraSpaceDiagram}
\end{figure}%
%
Now that we have the per-pixel mapping from \(\gls{cameraZ}\) to \(X^CY^C\), it is time to draw the camera space \gls{3D} image on \gls{GPU}. Figure~\ref{cameraSpaceDiagram} shows the streams flow diagram. We will retrieve depth from the \gls{KinectV2} camera, save them into corresponding buffers on CPU and upload the the streams onto \gls{GPU} as textures. Then the per-pixel's camera space \gls{3D} coordinates \(X^CY^CZ^C\) will be generated during its fragment-shader processing from depth texture based on eqn.~(\ref{proportionalBeamEqn}). The fragment shader is programmed as below.
\\\\%
{\ttfamily
\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{\ \ \ uniform }}\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
 }\textcolor[rgb]{0.5019608,0.5019608,0.0}{sampler2D }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}qt\_depthTexture;\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
\ \ }}

{\ttfamily
\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{uniform }}\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
 }\textcolor[rgb]{0.5019608,0.5019608,0.0}{sampler2D }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}qt\_spherTexture;\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{ \ }}

\bigskip

{\ttfamily
\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{layout}}(\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{location } }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}=\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}0,\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{index } }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}=\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}0)\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{out } }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
 }\textcolor[rgb]{0.5019608,0.5019608,0.0}{vec4 }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}qt\_fragColor;}

{\ttfamily
\textcolor[rgb]{0.5019608,0.5019608,0.0}{void }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}main()}

{\ttfamily
\{}

{\ttfamily
\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{\ \ \ \  }\textcolor[rgb]{0.5019608,0.5019608,0.0}{ivec2 }\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}CRD\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}=\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
 }\textcolor[rgb]{0.5019608,0.5019608,0.0}{ivec2}(
 \textcolor[rgb]{0.5019608,0.0,0.5019608}{gl\_FragCoord}.x, \textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
 }\textcolor[rgb]{0.5019608,0.0,0.5019608}{gl\_FragCoord}.y);}

\bigskip

{\ttfamily
\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{\ \ \ \ }\textcolor[rgb]{0.5019608,0.5019608,0.0}{float}\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}z\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
} = \textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{texelFetch}}(qt\_depthTexture,\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}CRD,\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textcolor[rgb]{0.0,0.0,0.5019608}{0}).r
*\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textcolor[rgb]{0.0,0.0,0.5019608}{65535.0};}

{\ttfamily
\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{\ \ \ \ }\textcolor[rgb]{0.5019608,0.5019608,0.0}{vec4}\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
\ }a\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
} = \textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textbf{\textcolor[rgb]{0.5019608,0.5019608,0.0}{texelFetch}}(qt\_spherTexture,\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}CRD,\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}\textcolor[rgb]{0.0,0.0,0.5019608}{0});}

\bigskip

{\ttfamily
\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{\ \ \ \ }qt\_fragColor.x\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
} = \textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}{}-z\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}*\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{ }a.r;}

{\ttfamily
\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{\ \ \ \ }qt\_fragColor.y\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
} = \textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}{}-z\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
}*\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{ }a.g;}

{\ttfamily
\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{\ \ \ \ }qt\_fragColor.z\textcolor[rgb]{0.7529412,0.7529412,0.7529412}{
} = \textcolor[rgb]{0.7529412,0.7529412,0.7529412}{ }{}-z;}

{\ttfamily
\}}
\\\\
\indent%
The uniform \emph{qt\_spherTexture} is a \(\gls{cameraZ}\) to \(\gls{cameraX}/\gls{cameraY}\) proportional mapping texture, which contains the per-pixel parameters \(a/b\) based on eqn.~(\ref{parametersABofProportional}). Note that we add three negative signs in front of \(X^CY^CZ^C\) respectively to account for the pinhole-imaging. %
%
\begin{figure}[t]
\centering
\subfloat[Front View][Front View]{
\includegraphics[height=0.42\textwidth]{distortedRGB}
\label{distortedRGB}}
\subfloat[Left View][Left View]{
\includegraphics[height=0.42\textwidth , width = 0.4\textwidth]{distortedSideViewRGB}
\label{distortedSideViewRGB}}
%\qquad
\caption{Colored Camera Space \gls{3D} Reconstruction}
\label{reconstructionInCameraSpace}
\end{figure}%
%
Figure~\ref{reconstructionInCameraSpace} shows the view of camera space \gls{3D} reconstruction when observing uniform grid dots pattern on the flat wall. We can tell from the image that this reconstruction is totally based on raw data, without any calibration. Not only the \(X^CY^C\) plane is apparently deformed (lens distortions) in the front view, but the \(\gls{cameraZ}\) is also not as flat as it should be, which we will call \emph{depth distortion}. The \emph{depth distortion} comes from the imperfect depth resolutions among pixels, \textit{i.e.}, \(\gls{cameraZ}  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  - \gls{D}  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] \neq E_{\text{Constant}}\), which lead to bumps and hollows in the camera space reconstruction when observing a flat wall. Therefore, a per-pixel calibration is necessary for an undistorted \gls{3D} image.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Rail Calibration System}
\label{sectionRailCalibrationSystem}
Talking about camera calibration, the pinhole-camera matrix \(\gls{pinHoleCameraM}\) will come up in most people's mind. As discussed in Chap.~\ref{chapterTraditionalCalibration}, the pinhole-camera matrix \(\gls{pinHoleCameraM}\) consists of an intrinsic matrix \(\gls{intrinsicMatrixK}\) and an extrinsic matrix \([\gls{exRotationR}, \gls{exTranslationT}]\). The camera space \gls{3D} reconstruction method we discussed in section~\ref{sectionCameraSpaceReconstruction} utilizes horizontal and vertical field of views, which work in the same way with the intrinsic matrix \(\gls{intrinsicMatrixK}\)'s principle. However, a camera's calibration needs external help from world space objects, which means neither of the FoVs nor intrinsic matrix \(\gls{intrinsicMatrixK}\) alone is able to do calibration,  and extrinsic parameters that can link to world space are necessary in calibration.
\\\indent
Kai \cite{Kai10} did a good job on structured light \gls{3D} scanner parallel calibration on \gls{GPU}, and derived the per-pixel beam equation~(\ref{kaiBeamEquationCh3}) directly from a pinhole camera matrix \(\gls{pinHoleCameraM}\), which offers the possibility of natural \gls{3D} reconstruction on \gls{GPU} similar to eqn.~(\ref{proportionalBeamEqn}).
%
\begin{equation}
\begin{aligned}
\gls{worldX} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = a [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] \gls{worldZ} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] +b [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]
\\%
\gls{worldY} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = c [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] \gls{worldZ} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] +d [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]
\end{aligned} \ ,
\label{kaiBeamEquationCh3}
\end{equation}%
\noindent
where \([\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] \) is the discrete space \(\gls{imageRow}\) and \(\gls{imageColumn}\) coordinate of each pixel in a \gls{imRowSize} by \gls{imColumnSize} sensor. This per-pixel beam equation shows per-pixel linear mappings from \(\gls{worldZ}\) to \(\gls{worldX}/\gls{worldY}\), and the per-pixel \(\gls{worldZ}\) can be mapped from features of structured light. It is not specially proportional like eqn.~(\ref{proportionalBeamEqn}), because it contains space translation infos from camera space to world space. Although it is in world space now and the intrinsic parameters are able to be determined, however, lens distortions are still not able to be handled. To make it ideal, we need to not only remove the lens distortions and \emph{depth distortion}, but also realize the \gls{3D} reconstruction on \gls{GPU} in a natural method similar to eqn.~(\ref{kaiBeamEquationCh3}).
\\\indent
%
In this section, we will find a best-fit calibration system for a \gls{KinectV2} camera's natural calibration and reconstruction, which is able the handle both of lens distortion and \emph{depth distortion}. %
%The \gls{KinectV2} camera has one depth sensor and one RGB sensor. The RGB sensor offers RGB streams, whose color infos will be aligned to pixels according to their corresponding world space coordinates after the \gls{3D} reconstruction. The depth sensor offers \gls{NIR} and Depth streams, both of which share the same size ($row$ and $column$) and lens distortions of the depth sensor, and can both be utilized for lens distortion correction. We will use the Depth stream to recover world space \gls{3D} coordinates $X^WY^WZ^W$ for reconstruction, during which the \gls{NIR} stream will be used for lens distortion correction.
%\\\indent
To easily show \gls{3D} reconstruction in a parallel way on the \gls{GPU}, we would like our calibration system to be able to offer a per-pixel mapping from \(\gls{D}\) to \(\gls{worldZ}\), which then could be used to map to \(\gls{worldX}/\gls{worldY}\) using eqn.~(\ref{kaiBeamEquationCh3}). In this way, the \emph{depth distortion} could also be corrected during the per-pixel \(\gls{D}\) to \(\gls{worldZ}\) mapping. Of all different kinds of calibration systems, a camera on rail system with a planar pattern on wall is finally decided, which offers a moving plane with respect to the camera when the camera moves along the rail. %
\\\indent
As Fig.~\ref{trackingModuleOnKinectV2CalibrationSystemCh3} shows, a canvas on which printed an uniform grid dots pattern is hung on the wall, and the rail is required to be perpendicular to the wall.
%
\begin{figure}[t]
\centering
\includegraphics[width=0.55\textwidth]{trackingModuleOnKinectV2CalibrationSystem}
\caption{\gls{KinectV2} Calibration System}
\label{trackingModuleOnKinectV2CalibrationSystemCh3}
\end{figure}%
%
The \gls{RGBD} camera waiting to calibrate is mounted on the slider. Note that, in this calibration system, the only unit that needs to be perpendicular to the wall is the rail, whereas the \gls{RGBD} camera has no need to require its observation orientation. Because the per-pixel calibration requires only accurate world space coordinates which will be decided by the rail and the wall, whereas the camera's space is not considered at all. We will assign the pattern plane as the \(X^WY^W\) plane in world space, and the rail to be along with (not exactly on) \(\gls{worldZ}\)-axis. The world coordinate is static with the camera on the slider. %
%
%
\\\indent
\begin{figure}[!t]
\centering
%calibrated \(X^{W}\)/\(Y^{W}\) and \(Z^{W}\)
\includegraphics[width=0.7\textwidth, height= 0.5\textwidth]{OneFrameXYZ_Calibrated}
\caption{\gls{NIR} \(X^{W}Y^{W}Z^{W}\) \gls{3D} Reconstruction}
\label{OneFrameXYZ_Calibrated}
\end{figure}%
%
Figure~\ref{OneFrameXYZ_Calibrated} shows one frame of \gls{NIR} \(X^WY^WZ^W\) \gls{3D} reconstruction, which can help explain how the world space origin is assigned. Inside the figure, both of the origin and \(Z\)-axis are high-lighted in blue, and the origin of world space is on the left end of the blue line. On the pattern plane with dot-clusters marked in circles, we can see one dot-cluster is high-lighted inside a thick circle, and its center point is where \(\gls{worldX}/\gls{worldY} = 0\). The dot-cluster which will be sitting on the \(\gls{worldZ}\)-axis is the one whose center point is closest to the center pixel of the sensor. All pixels in this frame share the exact same \(\gls{worldZ}\), which is also why we require the rail to be perpendicular to the pattern. And the value of \(\gls{worldZ}\) is measured by a laser distance measurer mounted on the camera's carriage. The final origin of the world space will be decided by both of the camera's observing orientation and the laser distance measurer's position. This kind of world coordinate assignment is totally for simplifying image processing during calibration.  Practically, we do not even care where exactly the origin is, as long as the rail is perpendicular to the pattern and the distance measurer is static with the camera.
\\\indent
With this rail calibration system, an infinite number of frames could be utilized for training a calibration model. With more and more dots walking into the camera's field of view under a certain rhythm as the slider moves further from the pattern plane, the dots (which will be extracted as calibration points) are able to cover all pixels of a sensor. What's more, a moving-plane system (multiple frames calibration instead of one frame calibration) makes it possible to do dense \(\gls{D}\) to \(\gls{worldZ}\) mapping, which will handle \emph{depth distortion}.

\section{Data Collection}
\label{perpixelDataCollection}
%
With the calibration system built up and world coordinate assigned, we are now ready to calibrate. \(Z^{W}\) values for all pixels of every frame will be supported from external laser distance measurer. To simplify potential calculation during image processing, we assign the world coordinate \emph{Unit One} based on the uniform grid dots patter, to be same with the side of pattern's unit-square. Concretely, the distance between every two adjacent dots' centers in real-world is 228mm. Therefore, \(Z^{W}\) = -\(Z\)(mm) / 228(mm), where \(Z\) is the vertical distance to the pattern plane in reality measured by the laser distance measurer. Note that, \(Z^{W}\) values are always negative, based on the assignment of Cartesian world coordinate. The outline of calibration procedures is listed below.
%
\begin{enumerate}
   	\item Mount both of the camera and laser distance measurer onto the slider.
   	\item Move the slider to the nearest position to the pattern plane.
   	\item Record one frame of RGB data with one frame of \gls{NIR} data at this position.
   	\label{repeatMovingSlider}
   	\begin{enumerate}
     		\item measure \(|\gls{worldZ}|\) using the laser distance measurer.
     		\item grab RGB, \gls{NIR} and Depth streams from \gls{KinectV2} camera.
     		\item extract center points (\(R/C\)) of dot-clusters from RGB and \gls{NIR} streams respectively.
     		\item assign \(\gls{worldX}/\gls{worldY}\) values to the extracted points, RGB and \gls{NIR} respectively.
     		\item train and determine the best-fit high order polynomial model that map from \(RC\) to \(\gls{worldX}/\gls{worldY}\), RGB and \gls{NIR} respectively.
     		\item generate dense \(\gls{worldX}/\gls{worldY}\) for all pixels using the model, for \gls{NIR} and RGB streams respectively.
     		\item save the RGB and \gls{NIR} images as \(X^WY^WZ^WRGBD\) for RGB stream, and \(X^WY^WZ^WID\) for \gls{NIR} stream, where the channel \(I\) in \gls{NIR} frame denotes \emph{Intensity}.
   	\end{enumerate}
	\item Move the slider to the next position, and repeat step~\ref{repeatMovingSlider}.
%	\item Check all \(\gls{worldX}/\gls{worldY}\) values of all frames, unify the staggered origin by adding or subtracting a corresponding integer (make sure the point \(X^WY^W = 0\) is at the same dot-cluster for all frames).
%	\item Determine the per-pixel mapping from \(\gls{D}\) to \(\gls{worldZ}\) and from \(\gls{worldZ}\) to \(\gls{worldX}/\gls{worldY}\) for all pixels using the \gls{NIR} frames of data, and generate a \(\gls{imageRow}\)-by-\(\gls{imageColumn}\)-by-6 look-up table.
%	\item Determine a pinhole-camera matrix that can map \(\gls{worldX}/\gls{worldY}/\gls{worldZ}\) to \(R/C\) using RGB frames of data (for the RGB values' alignment to pixels after \gls{3D} reconstruction).
\end{enumerate}
%
Concretely, we will record RGB and \gls{NIR} frames every 25mm. During every time, \(|\gls{worldZ}|\) will be measured by the laser measurer and manually input into the shader at the very beginning of streams recording. After RGB, \gls{NIR} and Depth streams have been retrieved from \gls{KinectV2}, we will utilize digital image processing (\gls{DIP}) techniques to extract the center points' image addresses (\(R/C\)) of dot-clusters, and then determine a high-order polynomial model to build a mapping from \(\gls{imageRow}/\gls{imageColumn}\) to \(\gls{worldX}/\gls{worldY}\) for dense world coordinates generation.

\subsection{\gls{DIP} Techniques on (\(\gls{imageRow},\, \gls{imageColumn}\)) Extraction}
\label{sectionDIPTechniques}
%  add noises analysis
\indent
A robust \gls{DIP} process on the extraction of a customer's geometric center pixel coordinate determines the accuracy of calibration. In this project, the extraction steps consist of gray-scaling, histogram equalization, adaptive thresholding and a little trick on black pixel counting. OpenGL is selected as the \gls{GPU} image processing language. The default data type of streams saved on \gls{GPU} during processing is single precision floating point (\gls{dataTypeFloat}), with its range being from 0.0 (total balck) to 1.0 (total white). %
%\\\indent
Gray-scaling is done to unify processing steps of both RGB and \gls{NIR} streams. For \gls{NIR} stream, its data contains only color gray, and data will be saved on \gls{GPU} as \gls{dataTypeFloat} automatically. Whereas for RGB stream, a conversion from RGB to gray value is needed. Typically, there are three converting methods: lightness, average, and luminosity. The luminosity method, which uses a weighted average value to account for human perception, is finally chosen as a human-friendly way for gray-scaling, give by
%
\begin{equation}
I_0 [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] =  0.21 R  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] \,  + \, 0.72 G [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  \, + \, 0.07 B  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]   \, \, ,
\label{luminosityGrayScaling}
\end{equation}%
\noindent
where \(R/G/B\) denotes the data of \(Red/Green/Blue\) channels from RGB stream, and \(I_0\) denotes the intensity value after gray-scaling.
\\\indent%
In practice, \gls{NIR} stream image is always very dark, as shown in Fig.~\ref{Raw_Single_NIR} (with their intensity values every close to zero).  In order to enhance the contrast of \gls{NIR} image for a better binarizing, histogram equalization technique is utilized to maximize the range of valid pixel intensity distributions. Same process is also compatible on the RGB stream. %
%
Commonly, Probability Mass Function (\gls{PMF}) and Cumulative Distributive Function (\gls{CDF}) will be calculated to determine the minimum valid intensity value (\(I_{\text{MIN}}\)) and maximum valid value (\(I_{\text{MAX}}\)) for rescaling, whereas tricks could be used by taking advantage of the \gls{GPU} drawing properties.%
\\\indent%
\gls{PMF} means the frequency of every valid intensity value for all of the pixels in an image. The calculation of \gls{PMF} on \gls{GPU} will be very similar to points' drawing process, both of which are on a per-pixel basis. Dividing all of the pixels in terms of their intensity values into \(\gls{graySize}\) levels, every pixel belongs to one level of them, which is called \emph{Gray Level}. With a proper selection of \(\gls{graySize}\) (\textit{e.g.}, \(\gls{graySize}\)=1000) to make sure a good accuracy, the intensity value of a pixel could be expressed based on its gray level \(\gls{grayLevel}\)
%
\begin{equation}
I_0 [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = \frac{\gls{grayLevel} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] }{\gls{graySize}} * (1.0 - 0.0) + 0.0 = \frac{\gls{grayLevel} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] }{\gls{graySize}} \, ,
\end{equation}%
\noindent
where \(\gls{grayLevel}\) and \(\gls{graySize}\) are integers, and \(1 \leqslant \gls{grayLevel} \leqslant \gls{graySize}\). We will count \gls{PMF} by drawing all pixels with \enquote{1.0} being their intensity (color) value, zero being \(y\)-axis all the time and their original intensity being their position on \(x\)-axis. Thus, \gls{PMF} value at each gray level could be drawn into a framebuffer object.
\\\indent%
With \gls{PMF} determined, \gls{CDF} at gray leverl \(\gls{grayLevel}\) could be calculated by
%
\begin{equation}
\gls{CDF}(\gls{grayLevel}) = \frac{ \sum_{l=1}^{\gls{grayLevel}} \gls{PMF} [l] }{\gls{imRowSize} \times \gls{imColumnSize}} \, ,
%\label{lensDistortion}
\end{equation}%
%
where \(l\) in the \gls{PMF} summation calculation denotes integer gray level with its range from 1 to \(\gls{grayLevel}\), and \(\gls{imRowSize} \times \gls{imColumnSize}\) gives how may pixels totally in a sensor. %
%
Then, the limit of valid intensities \(I_{\text{MIN}}\) and \(I_{\text{MAX}}\) could be determined by
%
\begin{equation}
\begin{aligned}
I_{\text{MIN}} &=  \frac{\gls{grayLevel}_{min}}{\gls{graySize}}%
\\%
I_{\text{MAX}} &=  \frac{\gls{grayLevel}_{max}}{\gls{graySize}}
\end{aligned}
\label{intensityFloorCeilingDetermination}
\end{equation}%
%
where \(\gls{grayLevel}_{min}\) and \(\gls{grayLevel}_{max}\) are determined by choosing appropriate CDFs, \textit{e.g.}, \(\gls{CDF}(\gls{grayLevel}_{min}) = 0.01\) and \(\gls{CDF}(\gls{grayLevel}_{max}) = 0.99\). Finally, a new intensity value of every single pixel in an image could be rescaled by
%
\begin{equation}
I_1 [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  = \frac{I_0  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  - I_{\text{MIN}}}{I_{\text{MAX}} - I_{\text{MIN}}} \, ,
\end{equation}%
%
\noindent
and the image will get a better contrast, as compared in Fig.~\ref{Histogram_Equalization}.%
%
 \begin{figure}[t]
\hspace*{-0.5cm}
\centering
\subfloat[Raw \gls{NIR}][Raw]{
\includegraphics[width=0.5\textwidth]{Raw_Single_NIR}
\label{Raw_Single_NIR}}
\subfloat[Histogram Equalized \gls{NIR}][Histogram Equalized]{
\includegraphics[width = 0.5\textwidth]{NIR_After_HistogramEqualization}
\label{NIR_After_HistogramEqualization}}
%\qquad
\caption{\gls{NIR} Streams before / after Histogram Equalization}
\label{Histogram_Equalization}
\end{figure}%
%
\\\indent%
%% Adaptive Thresholding
Affected by radial dominated lens distortions, the intensity value tend to decrease as the position of a pixel moves from the center of an image to the borders, in the case of observing a singular color view. Therefore, an adaptive thresholding process is needed, whereas using fixed thresholding will generate too much noise around borders. To segment the black dots from white background, we could simply subtract an image's background from an textured image, where the background comes from a blurring process of that image. %
%
There are three common types of blurring filters: mean filter, weighted average filter, and gaussian filter. Mean filter is selected for this background-aimed blurring process, because it has the smallest calculation and also a better effect of averaging than the others. After the blurred image containing background texture is obtained, the binarizing (subtraction) process for every single pixel could be written as
%
\begin{equation}
%
I_2 [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = %
%
\begin{cases}
\,\, 1 , \quad \quad I_1  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  - I_b  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]   -  C_0 \,\, > \,\,0 %
\\%
\,\, 0 , \quad \quad \quad \quad else%
\end{cases}
 \, ,%
\end{equation}%
%
where the per-pixel \(I_1\) value comes from the texture after histogram equalization, the per-pixel \(I_b\) comes from the background texture after a mean blurring filter, and \(C_0\) is a small uniform constant offset that could be adjusted depending on various thresholding situations. In this project, \(C_0\) is around 0.1. %
%
\begin{figure}[t]
\hspace*{-0.5cm}
\centering
\subfloat[Histogram Equalized \gls{NIR}][Histogram Equalized]{
\includegraphics[width = 0.5\textwidth]{NIR_After_HistogramEqualization}
\label{NIR_After_HistogramEqualization_2}}
\subfloat[After Adaptive Thresholding][Binarized]{
\includegraphics[width=0.5\textwidth]{Binarized_Single_NIR}
\label{Binarized_Single_NIR}}%\qquad
\caption{\gls{NIR} Streams before / after Adaptive Thresholding}
\label{Adaptive_Thresholding}
\end{figure}%
%%
To sharpen the edge of the binarized image for a better \enquote{circle} shape detection, a median filter could be added as the last step of adaptive thresholding. As shown in Fig.~\ref{Adaptive_Thresholding}, background is removed in the binarized image after adaptive thresholding.
\\\indent
%
%
% Sniffer for Round Dot Center
After the adaptive thresholding, image data saved on \gls{GPU} is now composed of circle-shaped \enquote{0}s within a background of \enquote{1}s. In order to locate the center of those \enquote{0}s circle, which is the center of captured round dot, it is necessary to know the edge of those circles. A trick is used to turn all of the edge data into markers that could lead a pathfinder to retrieve circle information. %
%
The idea that helps to mark edge data is to reassign pixels' values (intensity values) based on their surroundings. Using letter \(O\) to represent one single pixel in the center of a $3\times3$ pixels environment, and letters from \(A\)\texttildelow \(H\) to represent surroundings, a mask of 9 cells for pixel value reassignment could be expressed as below.

\begin{center}
  \begin{tabular}{ | c | c | c | }
    \hline
    \(E\) & \(A\) & \(F\) \\ \hline
    \(B\) & \(O\) & \(C\) \\ \hline
    \(G\) & \(\gls{D}\) & \(H\) \\
    \hline
  \end{tabular}
\end{center}
%
To turn the surroundings \(A\)\texttildelow \(H\) into marks, different weights will be assigned to them. Those markers with different weights have to be non-zero data, and should be counted as the edge-part of circles. Therefore, the first step is to inverse the binary image, generating an image that consists of circle-shaped \enquote{1}s distributed in a background of \enquote{0}s. After reversing, the next step is to assign weight to the surroundings. %
\\\indent
OpenGL offers convenient automatic data type conversion, which means the intensity values from \enquote{0.0} to \enquote{1.0} of \gls{dataTypeFloat} data type save on \gls{GPU} could be retrieved to CPU as unsigned-byte data type from \enquote{0} to \enquote{255}. Considering a bitwise employment of markers, a binary calculation related weight assignment is used in the shader process for pixels. The reassigned per-pixel intensity could be determined by:
%
\begin{equation}
\hspace*{-0.1cm}%
%
I_3  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = I_2  [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] * \frac{(128I_{\text{A}} + 64I_{\text{B}} + 32I_{\text{C}} + 16I_{\text{\gls{D}}} + 8I_{\text{E}} +  4I_{\text{F}} +  2I_{\text{G}}+I_{\text{H}})}{ 255 } ,
%
\label{snifferDistributionWeight}
\end{equation}%
\noindent%
where the per-pixel \(I_3\) is calculated based on its surroundings. After this reassignment, the image is not binary any more. Every non-zero intensity value contains marked information of its surroundings, and data at the edge of circles are now turned into fractions. In other words, the image data saved on \gls{GPU} at the moment is composed of \enquote{0}s as background and \enquote{non-zero}s circles, which contains fractions at the edge and \enquote{1}s in the center. %
\\\indent%
Now, it is time to discover dots through an inspection over the whole path-marked image, row by row and pixel by pixel. Considering that, a process of one single pixel in this step may affect the processes of the other pixels (which cannot be a parallel processing), it is necessary to do it on CPU. The \gls{dataTypeFloat} image data will be retrieved from \gls{GPU} to a buffer on CPU as unsigned-byte data, waiting for inspection. And correspondingly the new CPU image will have its \enquote{non-zero}s circles composed of fractions at the edge and \enquote{1}s in the center. Whenever a non-zero value is traced, a dot-circle is discovered and a singular-dot analysis could start. The first non-zero pixel will be called as an anchor, which means the beginning of a singular-dot analysis. %
%
During the singular-dot analysis beginning from the anchor, very connected valid (non-zero) pixel will be a stop, and a \enquote{stop-address} queue buffer is used to save addresses of both visited pixels and the following pixels waiting to be visited. On very visit of a pixel, there is a checking procedure to find out valid (non-zero) or not. Once valid, the following two steps are waiting to go. The first step is to sniff, looking for possible non-zero pixels around as the following stops. And the second step is to colonize this pixel, concretely, changing the non-zero intensity value to zero. Every non-zero pixel might be checked 1\texttildelow 4 times, but will be used to sniff for only once.%
\\\indent%
As for the sniffing step, base on the distribution table of \(A \texttildelow H\) that has been discussed above and their corresponding weight given by equation \ref{snifferDistributionWeight}, the markers \(A\)/\(B\)/\(C\)/\(\gls{D}\) will be checked if valid or not, using an \(AND\) bitwise operation between \(I_O\) and 0x80 / 0x40 / 0x20 / 0x10 respectively. The the bitwise operation result equals to one, then that marker is valid. %
%are valid (non-zero) as long as the intensity value of pixel \(O\) satisfies the following conditions shown as below.%
%
%\begin{equation*}
%%
%\begin{aligned}
%if \,\, ( I_{\text{O}} \,\, \& \,\, \text{0x80}  == 1 ),\quad &then, \quad \text{marker } A\,\, \text{is valid \,\,( go Up )}%
%\\%
%if \,\, ( I_{\text{O}} \,\, \& \,\, \text{0x40}  == 1 ),\quad &then, \quad \text{marker } B\,\, \text{is valid \,\,( go  Left)}%
%\\%
%if \,\, ( I_{\text{O}} \,\, \& \,\, \text{0x20}  == 1 ),\quad &then, \quad \text{marker } C\,\,\text{is valid \,\,( go Right )}%
%\\%
%if \,\, ( I_{\text{O}} \,\, \& \,\, \text{0x10}  == 1 ),\quad &then,\quad \text{marker } \gls{D}\,\,\text{is valid \,\,( go Down )}%
%\\%
%\end{aligned}
%%
%\end{equation*}%
%%
\noindent
Once a valid marker is found, its address [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] will be saved into the \enquote{stops-address} queue. One pixel's address might be saved for up to 4 times, but \enquote{colonizing} procedure will only happen once at the first time, so that the sniffing will stop once all of the connected valid pixels in a singular dot-cluster are colonized as zeros. %
\\\indent%
In the second step \enquote{colonizing},  \(I_{\text{O}}\) is changed to zero, variable \(area\) of this dot-cluster pluses one, and bounding data \(RowMax\) / \(RoxMin\) / \(ColumnMax\) / \(ColumnMin\) are also updated. %
%
Finally, the Round Dot Centers could be determined as the center of bounding boxes with their borders \(RowMax\) / \(RoxMin\) / \(ColumnMax\) / \(ColumnMin\). After potential noises being removed based on their corresponding \(area\) and \(shape\) (ratio of width and height), the data left are taken as valid dot-clusters. As shown in Fig.~\ref{Dots_Extracted_Single_NIR}, the centers of valid dot-clusters are marked within their corresponding homocentric circles.
%
 \begin{figure}[t]
\hspace*{-0.5cm}
\centering
\subfloat[After Adaptive Thresholding][before Extraction]{
\includegraphics[width=0.5\textwidth]{Binarized_Single_NIR}
\label{Binarized_Single_NIR}}
%\qquad
\subfloat[Dot Centers Extraction][Extracted and Marked]{
\includegraphics[width = 0.5\textwidth]{Dots_Extracted_Single_NIR}
\label{Dots_Extracted_Single_NIR}}
%
\caption{Valid Dot-Clusters Extracted in \gls{NIR}}
\label{DotCentersExtraction}
\end{figure}
%%%
%%
%%
\subsection{\((\gls{worldX}, \,\gls{worldY})\) Fitting based on Uniform Grid}
\label{uniformGridFittingXY}
%%
With a list of image space points (\(R, C\))s extracted, the following is to assign those points with their corresponding world coordinates \((\gls{worldX}, \,\gls{worldY})\)s, so that we can get coordinate-pairs to train a mapping model for dense \(X^WY^W\) generation. The world coordinates are based on the uniform grid. Taking the side of unit-square (distance between two adjacent dots) as \enquote{Unit One} in the world coordinates and one dot as the origin of plan \(X^WY^W\), all fitted \(\gls{worldX}\)/\(\gls{worldY}\) values will be integers. %
\\\indent%
Ideally, a $3\times3$ perspective transformation matrix could help set a linear mapping between two different 2D coordinates, and 3 dot centers with known coordinates pair of \((R, C)\) and \((\gls{worldX}, \,\gls{worldY})\) are enough to determine the transformation matrix. Once four points with a squared-shape \(R\)/\(C\) distribution is found, a $3\times3$ perspective transformation matrix \(\textbf{\textit{A}}_0\) could be determined by solving
\begin{equation}
%
\left[ \begin{array}{c} %
zX^W \\ zY^W \\ z \end{array} \right] %
= %
\textbf{\textit{A}}_0 \left[ \begin{array}{c} %
C \\ R \\ 1 \end{array} \right] %
= %
\begin{bmatrix}
a_{11} & a_{12} & a_{13} \\
a_{21} & a_{22} & a_{23} \\
a_{31} & a_{32} & a_{33} \\
\end{bmatrix}%
 \left[ \begin{array}{c} %
C \\ R \\ 1 \end{array} \right] %
 \ ,%
\label{perspectiveDistortionCorrectionEquation}
\end{equation}%
%
\noindent
where \(C\) and \(R\) are vectors consist of four squared-shape distributed points' addresses; \(\gls{worldX}\) and \(\gls{worldY}\) are vectors consist of four points \((0, 0)\), \((0, 1)\), \((1, 1)\), and \((1, 0)\); \(z\) denotes the third axis in the homogenous system connecting two coordinates. %
\\\indent%
Deformed by lens distortions, the cluster centers in image plane are not uniformly distributed, and this $3\times3$ transformation matrix can only generate corresponding decimal \(\gls{worldX}\)/\(\gls{worldY}\) values that are close integers. But in practice, the correct integer values \(\gls{worldX}\)/\(\gls{worldY}\) could still be located through rounding. Considering that a list of cluster centers' image coordinate \((R, C)\)s will give many groups of four squared-shape distributed points, and each of them gives a different image coordinate distance will be mapped to the \enquote{Unit One} in world coordinates, we will traverse all of the possible groups of four squared-shape distributed points and pick out the group whose mapping matrix \(\textbf{\textit{A}}_0\) generates the most points that are close to integers. %
%
In this way, we find the transformation matrix \(\textbf{\textit{A}}_0\) can give the best \enquote{Unit One} distance in world coordinates; however, its generated point \(\gls{worldX}/\gls{worldY} = 0\) is usually not at the dot-cluster that is closest to the center of cameras' \gls{FoV}. A translation matrix \(T\) could be used to refine the transformation matrix \(\textbf{\textit{A}}_0\) and help to translate the origin point to be at the dot cluster we want:%
%
\begin{equation}
%
\textbf{\textit{A}}_1%
= %
\textbf{\textit{T}} \cdot \textbf{\textit{A}}_0 %
= %
\begin{bmatrix}
1 & 0 & -x_h \\%
0 & 1 & -y_h \\%
0 & 0 &   1 \\%
\end{bmatrix}%
\cdot \textbf{\textit{A}}_0%
\, \, \, \ \ ,%
\end{equation}%
%
where \((x_h, \, y_h)\) is the integer world space address rounded after the transformation from image space center point:%
%
\begin{equation}
%
\left[ \begin{array}{c} %
z x_h \\ z y_h \\ z \end{array} \right] %
= %
\textbf{\textit{A}}_0 \cdot \left[ \begin{array}{c} %
c_h \\ r_h \\ 1 \end{array} \right] %
 \ .%
\end{equation}%
\noindent
where \((c_h, \, r_h)\) denotes the center point of \gls{FoV} in image space. Eventually, the refined transformation matrix $\textbf{\textit{A}}_1$ can help assign world coordinates to the extracted image space coordinates, with the point \(\gls{worldX}/\gls{worldY}=0\) at the dot-cluster nearest to the image center, as shown in Fig.~\ref{XY_GridFitting_Matlab}.
%
 \begin{figure}[t]
\hspace*{-0.3cm}
\centering
\subfloat[Image Plane Coordinates][ImagePlane Coordinate]{
\includegraphics[width=0.54\textwidth, height = 0.425\textwidth]{Grid_Centered_Single_NIR}
\label{Grid_Centered_Single_NIR}}
%\qquad
\subfloat[World Coordinates][World Coordinate]{
\includegraphics[width = 0.47\textwidth, height = 0.425\textwidth]{XY_GridFitting_Matlab}
\label{XY_GridFitting_Matlab}}
%
\caption{Coordinates-Pairs: (\(R, C\))s and (\(\gls{worldX}\), \(\gls{worldY}\))s}
\label{Grid_Fitting}
\end{figure}%

%%
\subsection{Dense \(\gls{worldX}/\gls{worldY}\) Generation}
\indent
%With a list of points with image space addresses (\(R, C\))s extracted,
The generation of undistorted dense \(X^WY^W\) needs to consider lens distortion removal. In the traditional calibration method, world space \(\gls{worldX}/\gls{worldY}/\gls{worldZ}\) are mapped to undistorted (\(R', C'\)) by linear pinhole camera matrix \(\gls{pinHoleCameraM}\). And then the undistortion step from undistorted (\(R', C'\)) to distorted (\(R, C\)) is done by eqn.~(\ref{lensDistortion}), which uses a high order (higher than 2nd order) polynomial equation. Assuming that there is a high order mapping relationship directly from the distorted image space (\(R, C\)) to world space (\(\gls{worldX}, \, \gls{worldY}\)), we will do different orders of two-dimensional polynomial prototypes in Matlab using its application of \enquote{Curve Fitting Toolbox}, and then decide a best-fit mapping model with a high accuracy and a relative small number of parameters.
%
\\\indent
A two-dimensional polynomial model means surface mapping between two different spaces. Equation.~(\ref{perspectiveDistortionCorrectionEquation}) (perspective correction) as \(1^{st}\) order polynomial mapping is not able to handle lens distortions.
%
The second order polynomial mapping has $2\times6=12$ parameters, written as %
%
\begin{equation}
\begin{aligned}
\gls{worldX} &=  a_{11}C^2 + a_{12}CR + a_{13}R^2 + a_{14}C + a_{15}R + a_{16}
\\%
\gls{worldY} &=  a_{21}C^2 + a_{22}CR + a_{23}R^2 + a_{24}C + a_{25}R + a_{26} \, ,
\end{aligned}
\label{secondOrderPolynomial}
\end{equation}%
%
\noindent
and similarly, the fourth order polynomial mapping has $2\times15=30$ parameters, given by
%
\begin{equation}
\hspace*{-0.3cm}%
\begin{aligned}
\gls{worldX} &=  a_{11}C^4 + a_{12}C^3R + a_{13}C^2R^2 + a_{14}CR^3 + a_{15}R^4 + a_{16}C^3 + a_{17}C^2R \ \ \ ...\\%
&\,\,\,\,\,\,+ a_{18}CR^2 + a_{19}R^3 + a_{110}C^2 + a_{111}CR + a_{112}R^2 + a_{113}C + a_{114}R + a_{115}
\\\\%
\gls{worldY} &=  a_{21}C^4 + a_{22}C^3R + a_{23}C^2R^2 + a_{24}CR^3 + a_{25}R^4 + a_{26}C^3 + a_{27}C^2R \ \ \ ...\\%
&\,\,\,\,\,\,+ a_{28}CR^2 + a_{29}R^3 + a_{210}C^2 + a_{211}CR + a_{212}R^2 + a_{213}C + a_{214}R + a_{215}      \,\,.
\end{aligned}
\label{fourthOrderPolynomial}
\end{equation}%
%
\indent
To prototype equation \ref{secondOrderPolynomial} and \ref{fourthOrderPolynomial} in Matlab, \enquote{Curve Fitting Toolbox} is used to obtain the $2\times6$ and $2\times15$ parameters, using 107 points' coordinates pairs of image space \(R/C\) and world coordinates \(\gls{worldX}/\gls{worldY}\). Based on the \enquote{Goodness of fit} of transformation parameters from Matlab, the Root-Mean-Square Error (RMSE) of (\(\gls{worldX}\), \(\gls{worldY}\)) is (0.06796, 0.05638) for the \(2^{nd}\) order polynomial, and (0.02854, 0.02343) for the \(4^{th}\) order polynomial. As the order of polynomial goes higher, the RMSE gets smaller, and the cost of parameters' calculation gets much higher as well. After the prototyping in Matlab, the different orders of polynomial models will be applied into real-time streams transformation to get live transformed world space reconstruction. Eventually, the \(4^{th}\) order polynomial transformation model is selected as the distortion removal mapping model, which can give an intuitive undistorted world space image while costing relative less calculations. This mapping model will be trained by coordinate-pairs of the extracted points, and then be applied to image space addresses to generate dense \(\gls{worldX}/\gls{worldY}\) for all pixels.
%% mapping model parameters determination
%
%
\section{Data Process and \gls{LUT} Generation}
\label{sectionDataProcessLUTgeneration}
Till now, we have collected frames of \(X^WY^WZ^WRGBD\) data from RGB streams and \(X^WY^WZ^WID\) from \gls{NIR} stream. The \(X^WY^WZ^WRGBD\) data from RGB streams can be used for color values' alignment after the \gls{3D} reconstruction based on depth stream. Since the \gls{NIR} stream has same pixel distribution with depth stream, we will process the \(X^WY^WZ^WID\) data and generate per-pixel mapping parameters, which can be saved as \gls{LUT} for undistorted \gls{3D} reconstruction. From section \ref{sectionRailCalibrationSystem}, we know that our whole idea of a calibrated natural \gls{3D} reconstruction on \gls{GPU} is to make use of per-pixel eqn.~(\ref{kaiBeamEquationCh3}), which requires undistorted \(X^WY^WZ^W\) data and a per-pixel mapping from \(\gls{D}\) to \(\gls{worldZ}\). With enough \(X^WY^WZ^WID\) data already been collected, we will determine a per-pixel \(\gls{D}\) to \(\gls{worldZ}\) mapping, and then generate \gls{LUT} for calibrated \gls{3D} reconstruction.
\\\indent
\begin{figure}[b]
\centering
\includegraphics[width=0.55\textwidth]{ZD_CurveFitting}
\caption{Polynomial Fitting between \gls{D} and \(\gls{worldZ}\)}
\label{ZD_CurveFitting}
\end{figure}%
%
Both of \(\gls{D}\) and \(\gls{worldZ}\) are continuous data, so that their function could written as a polynomial expression, based on Taylor series. We will determine a best-fit mapping model from \(\gls{D}\) to \(\gls{worldZ}\) using Matlab. Figure~\ref{ZD_CurveFitting} shows the polynomial fitting result in Matlab \enquote{Curve Fitting Tool} toolbox, with 32 points of \(DZ^\text{W}\) values (at pixel \(\gls{imageColumn}\)=256 and \(\gls{imageRow}\)=212) from 32 frames. It is apparent that \(\gls{worldZ}\) is linear with \(\gls{D}\). Therefore, for every single pixel, \(\gls{worldZ}\) could be mapped from \(\gls{D}\) through \par
%
\begin{equation}
\gls{worldZ} [m, \, n] = e [m, n] \gls{D} [m, \, n]+f [m, n] ,
\label{fromD_To_Z}
\end{equation}%
%
\noindent
where \([m, \, n]\) denotes the image address \(\gls{imageRow}\) and \(\gls{imageColumn}\) of a pixel, \(e/f\) are the corresponding linear coefficients that help map from \(\gls{D}\) to \(\gls{worldZ}\). With eqn.~(\ref{fromD_To_Z}) supporting the per-pixel \(\gls{worldZ}\) values and eqn.~(\ref{kaiBeamEquationCh3}) help generating the per-pixel \(X^WY^W\) values, we found all of the six per-pixel parameters \((a/b/c/d/e/f)\) we need. We are now ready to process the collected data off-line and generate a \(\gls{imRowSize}\) by \(\gls{imColumnSize}\) by 6 \gls{LUT}.
\\\indent
%
 \begin{figure}[t]
\hspace*{-0.5cm}
\centering
\subfloat[Staggered]{
\includegraphics[width=0.5\textwidth]{staggeredPiles}
\label{staggeredPiles}}
%\qquad
\subfloat[Unified]{
\includegraphics[width = 0.5\textwidth]{unifiedPiles}
\label{unifiedPiles}}
%
\caption{World Space Unification of Collected Data}
\label{worldSpaceDataUnification}
\end{figure}
%
The collected data cannot be used directly for \gls{LUT} generation, because \(\gls{worldX}/\gls{worldY}\) may not be well aligned and pre-processes of data are needed. Our calibration system did not require the camera's observing orientation to be along with \(\gls{worldZ}\)-axis, which results to the fact that, the collected frames might be staggered when the centered dot-cluster in camera's \gls{FoV} moves. Figure~\ref{staggeredPiles} shows the raw collected data before world space unification, which has a staggered piles of frames. In order to make the data available for generating valid per-pixel \gls{3D} reconstruction parameters, we need to unify their world space by adding or subtracting a corresponding integer to all pixels in every staggered frame, making sure the point \(X^WY^W = 0\) is at the same dot-cluster for all frames.	After the data unification of world space, as shown in Fig.~\ref{unifiedPiles}, we are all ready to determine the per-pixel mapping parameters \(a/b/c/d/e/f\) and generate the \gls{LUT}.
%\subsection{Mathematical tools}
%\subsubsection{Singular Value Decomposition (SVD)}
%\subsubsection{Least Square with Pseudo-Inverse}
%
\section{Alignment of RGB Pixel to Depth Pixel}
Now an undistorted \gls{3D} reconstruction could be displayed with the help of the calibrated \gls{LUT}. However, we have not figured out yet what the color of each pixel is. To generate a colored \gls{3D} reconstruction with a combination of a random depth sensor and a random RGB sensor, we need to align the RGB pixels to depth pixels. The intermediate between the depth sensor image space and RGB sensor image space is the world space. As long as we figure out the mapping from world space to RGB sensor image space, the color of pixel with known \(X^WY^WZ^W\) could be looked up from the RGB image space. The pinhole camera matrix \(\gls{pinHoleCameraM}\) is used to map from world space to RGB image space. Using the frame data from Kinect RGB streams and Kinect \gls{NIR} streams, a Matlab prototype of RGB pixels alignment is shown in Fig.~\ref{RGBAligned_Matlab}, where the RGB textured is mapped onto its corresponding \gls{NIR} image, who has same pixels with depth sensor. The total black area on the top edge and bottom edge is where the depth sensor's view goes beyond the RGB sensor's view. Figure~\ref{RGBvaluesAligmentMatlabQt} shows the screen-shot of live video after calibration with the RGB pixels aligned by a pinhole camera model \(\gls{pinHoleCameraM}\).

%
 \begin{figure}[h]
\hspace*{-0.5cm}
\centering
\subfloat[][]{
\includegraphics[width = 0.5\textwidth]{RGBAligned_Matlab}
\label{RGBAligned_Matlab}}
\subfloat[][]{
\includegraphics[width=0.5\textwidth]{RGBAligned_Qt}
\label{RGBAligned_Qt}}
\caption{Alignment of RGB Texture onto \gls{NIR} Image}
\label{RGBvaluesAligmentMatlabQt}
\end{figure}%
%


\documentclass[a4paper]{article}

\def\npart{III}

\def\ntitle{Combinatorics}
\def\nlecturer{I.\ B.\ Leader}

\def\nterm{Michaelmas}
\def\nyear{2018}

\input{header}

\let\SO\undefined
\usepackage{tkz-graph}

\newcommand{\shadow}{\partial}
\renewcommand{\P}{\mathbb P}

\begin{document}

\input{titlepage}

\tableofcontents

\section{Set systems}

\begin{definition}[set system]\index{set system}
  Let \(X\) be a set. A \emph{set system} on \(X\) (or \emph{family of subsets}) of \(X\) is a family \(\mathcal A \subseteq \powerset (X)\).
\end{definition}

\begin{eg}
  We write \(X^{(r)} = \{A \subseteq X: |A| = r\}\)
\end{eg}

In this course we almost deal exclusively with finite sets so unless otherwise stated, in this course we assume all sets are finite and \(X = [n] = \{1, 2, \dots, n\}\).

For example, \(|X^{(r)}| = \binom{n}{r}\). More concretely, for example,
\[
  [4]^{[2]} = \{12, 13, 14, 23, 24, 34\}
\]
where \(12\) denotes \(\{1, 2\}\) to avoid heavy notation. Therefore \(|[4]^{[2]}| = 6\).

What is the mental picture for power set? Often we make \(\powerset (X)\) into a graph, called \(Q_n\), by joining \(A\) to \(B\) if \(|A \Delta B| = 1\) where \(\Delta\) is the symmetric difference, i.e.\ if \(A = B \cup \{i\}\) for some \(i \notin B\) (or vice verse).

\begin{eg}
  \(Q_3\)
\end{eg}

\begin{eg}
  General picture for \(Q_n\) where \(n\) is even:

  and when \(n\) is odd:
\end{eg}

If we identify a set \(A \subseteq X\) with a \(\{0, 1\}\) sequence of length \(n\) (e.g.\ \(134 \leftrightarrow 1011000 \cdots 0\)), via \(A \leftrightarrow \mathbf{1}_A \text{ or } \chi_A\). In this way, we can represent \(Q_n\) as a \(n\)-dimensional cube:

For this reason, \(Q_n\) is often called the \emph{hypercube} or \emph{discrete cube} or \emph{\(n\)-cube}. In this way, the study of a set system becomes the study of a graph.

\subsection{Chains \& Antichains}

\begin{definition}[chain]\index{chain}
  A family \(\mathcal A \subseteq \powerset (X)\) is a \emph{chain} if for all \(A, B \in \mathcal A, A \subseteq B \text{ or } B \subseteq A\).
\end{definition}

\begin{eg}
  \(\{12, 125, 123589\}\).
\end{eg}

On the other hand, we have

\begin{definition}[antichain]\index{antichain}
  A family \(\mathcal A \subseteq \powerset (X)\) is an \emph{antichain} if for all \(A, B \in \mathcal A\) with \(A \neq B\), \(A \nsubseteq B\).
\end{definition}

\begin{eg}
  \(\{1, 467, 2456\}\).
\end{eg}

A natural question is: how large can a chain be? Obviously we can achieve \(|\mathcal A| = n + 1\). We also cannot exceed \(n + 1\) since a chain must meet each ``level'' \(X^{(r)}\) for \(0 \leq r \leq n\) in at most one place.

A less trivial question is how large an antichain can be. We could achieve \(|\mathcal A| = n\) by enumerating all the singletons. It is maximal since adjoining any nonempty set to the family will result in an inclusion. However, it is not the largest antichain. Indeed, we could take \(\mathcal A = X^{(r)}\) for any \(r\). Thus we can achieve \(|\mathcal A| = \binom{n}{\floor{n/2}}\). Can we beat it?

Pause for a moment and consider the longest chain problem. Why can a chain meet each level at only one place? One way to see this is that each level is an antichain: as we can decompose \(Q_n\) into \(n + 1\) antichains, we cannot have a chain longer than that. Inspired by this, we try to decompose \(Q_n\) into chains.

\begin{theorem}[Sperner's lemma]\index{Sperner's lemma}
  Let \(\mathcal A \subseteq \powerset(X)\) where \(|X| = n\) be an antichain. Then
  \[
    |\mathcal A| \leq \binom{n}{\floor{n/2}}.
  \]
\end{theorem}

\begin{proof}
  Sufficient to partition \(\powerset(X)\) into \(\binom{n}{\floor{n/2}}\) chains. For this sufficient to show:
  \begin{enumerate}
  \item for all \(r < \frac{n}{2}\), there exist matchings from \(X^{(r)}\) to \(X^{(r + 1)}\) where a matching is just a set of non-adjacent edges,
  \item for all \(r > \frac{n}{2}\), there exist matchings from \(X^{(r)}\) to \(X^{(r - 1)}\).
  \end{enumerate}
  Then put these matchings together to form chains. Each passes through \(X^{(n/2)}\) so there are \(\binom{n}{\floor{n/2}}\) of them.

  By taking complements, sufficient to prove 1. Consider the subgraph of \(Q_n\) spanned by \(X^{(r)} \cup X^{(r + 1)}\) which is bipartite. For any \(\mathcal B \subseteq X^{(r)}\), let \(\Gamma(\mathcal B)\) be the neighbourhood (in \(X^{(r + 1)}\)) of \(\mathcal B\). Then we have

  \[
    \# (\mathcal B - \Gamma(\mathcal B) \text{ edges}) = |\mathcal B| (n - r)
  \]
  as each point in \(X^{(r)}\) has degree \(n - r\). Meanwhile
  \[
    \# (\mathcal B - \Gamma(\mathcal B) \text{ edges}) \leq |\Gamma(\mathcal B)| (r + 1)
  \]
  as each point in \(X^{(r + 1)}\) has degree \(r + 1\). Thus
  \[
    |\Gamma(\mathcal B)| \geq |\mathcal B| \frac{n - r}{r + 1} \geq |\mathcal B|
  \]
  as \(r < \frac{n}{2}\). Hence by Hall's theorem there exist matchings.
\end{proof}

\begin{remark}\leavevmode
  \begin{enumerate}
  \item \(\binom{n}{\floor{n/2}}\) is achievable, e.g.\ \(\mathcal A = X^{(\floor{n/2})}\).
  \item Note that the theorem says nothing about extremal cases --- which antichains have this size?
  \end{enumerate}
\end{remark}

The aim is to show that for \(\mathcal A\) an antichain,
\[
  \sum_{r = 0}^n \frac{\mathcal A \cap X^{(r)}}{\binom{n}{r}} \leq 1.
\]
Note that this trivially implies Sperner's lemma. We will use the same setup but bound the numbers more carefully. To justify the following definition, we will write \(X^{(r)}\) above \(X^{(r - 1)}\) in \(Q_n\).

\begin{definition}
  Let \(\mathcal A \subseteq X^{(r)}\) for some \(1 \leq r \leq n\). The \emph{shadow} or \emph{lower shadow} of \(\mathcal A\) is
  \[
    \shadow A = \shadow^- A = \{A - \{i\}: A \in \mathcal A, i \in A\}.
  \]
  so \(\shadow A \subseteq X^{(r - 1)}\).
\end{definition}

\begin{eg}
  Let \(\mathcal A = \{123, 124, 134, 135\} \subseteq X^{(3)}\). Then
  \[
    \shadow A = \{12, 13, 23, 14, 24, 34, 15, 35\} \subseteq X^{(2)}.
  \]
\end{eg}

\begin{lemma}[Local LYM]\index{local LYM}
  Let \(\mathcal A \subseteq X^{(r)}\) where \(1 \leq r \leq n\). Then
  \[
    \frac{|\shadow \mathcal A|}{\binom{n}{r - 1}} \geq \frac{|\mathcal A|}{\binom{n}{r}}.
  \]
\end{lemma}

Informally, the fraction of the later occupied increases when we take the shadow.

\begin{proof}
  \[
    \#(\mathcal A - \shadow \mathcal A \text{ edges in } Q_n) = r|\mathcal A|
  \]
  by counting from above and
  \[
    \#(\mathcal A - \shadow \mathcal A \text{ edges in } Q_n) \leq (n - r + 1) |\shadow \mathcal A|
  \]
  counting from below so
  \[
    \frac{|\shadow \mathcal A|}{|\mathcal A|} \geq \frac{r}{n - r + 1}
  \]
  but
  \[
    \frac{\binom{r}{r - 1}}{\binom{n}{r}} = \frac{r}{n - r + 1}.
  \]
\end{proof}

When does equality hold in local LYM? We'll need
\[
  (A - \{i\}) \cup \{j\} \in \mathcal A
\]
for all \(a \in \mathcal A, i \in A, j \notin A\). Hence \(\mathcal A = X^{(r)}\) or \(\emptyset\).

\begin{theorem}[LYM inequality]\index{LYM inequality}
  Let \(\mathcal A \subseteq \powerset(X)\) be an antichain. Then
  \[
    \sum_{r = 0}^n \frac{\mathcal A \cap X^{(r)}}{\binom{n}{r}} \leq 1.
  \]
\end{theorem}

\begin{proof}
  The whole idea of the proof can be summarised by ``bubble down with local LYM''. Let \(\mathcal A_r = \mathcal A \cap X^{(r)}\). Obviously
  \[
    \frac{|\mathcal A_n|}{\binom{n}{n}} \leq 1.
  \]
  Also \(\shadow \mathcal A_n\) and \(\mathcal A_{n - 1}\) are distinct as \(\mathcal A\) is an antichain. Thus
  \[
    \frac{|\shadow \mathcal A_n|}{\binom{n}{n - 1}} + \frac{|\mathcal A_{n - 1}|}{\binom{n}{n - 1}}
    = \frac{|\shadow \mathcal A_n \cup \mathcal A_{n - 1}|}{\binom{n}{n - 1}} \leq 1
  \]
  so
  \[
    \frac{|\mathcal A_n|}{\binom{n}{n}} + \frac{|\mathcal A_{n - 1}|}{\binom{n}{n - 1}} \leq 1
  \]
  by local LYM.

  Also \(\shadow (\shadow \mathcal A_n \cup \mathcal A_{n - 1})\) is disjoint from \(\mathcal A_{n - 2}\), again since \(\mathcal A\) is an antichain. Thus
  \[
    \frac{|\shadow (\shadow \mathcal A_n \cup \mathcal A_{n -1})|}{\binom{n}{n - 2}} +\frac{|\mathcal A_{n - 2}|}{\binom{n}{n - 2}} \leq 1,
  \]
  so
  \[
    \frac{|\shadow \mathcal A_n \cup \mathcal A_{n - 1}|}{\binom{n}{n - 1}} + \frac{|\mathcal A_{n - 1}|}{\binom{n}{n - 2}} \leq 1,
  \]
  so
  \[
    \frac{|\mathcal A_n|}{\binom{n}{n}} + \frac{|\mathcal A_{n - 1}|}{\binom{n}{n - 1}} + \frac{|\mathcal A_{n - 2}|}{\binom{n}{n - 2}} \leq 1
  \]
  Keep going we have the result desired.
\end{proof}

Again we can ask when we have equality in LYM inequality. This happens if and only if we have equality in each use of local LYM, so the ``first'' (greatest) \(r\) with \(\mathcal A_r \neq \emptyset\) must have \(\mathcal A_r = X^{(r)}\) so \(\mathcal A = X^{(r)}\). Thus we know equality in Sperner's lemma (\(\mathcal A = X^{(n/2)}\) for \(n\) even, similar for \(n\) odd)

\begin{proof}[Proof 2]
  Choose uniformly at random a maximal chain \(\mathcal C\) (i.e.\ \(C_0 \subseteq C_1 \subseteq \dots \subseteq C_n\) with \(|C_i| = i\)). For a given \(r\)-set \(A\), \(\P(A \in \mathcal C) = \binom{n}{r}^{-1}\) as all \(r\)-sets are equally likely. Therefore
  \[
    \P(\mathcal A_r \text{ meets } \mathcal C) = \frac{|\mathcal A_r|}{\binom{n}{r}}
  \]
  since the events are disjoint. Furthermore for different \(r\) the events that \(\mathcal A\) meets \(X^r\) are also disjoint so
  \[
    \P(\mathcal A \text{ meets } \mathcal C) = \sum_{r = 0}^n \frac{|\mathcal A_r|}{\binom{n}{r}}
  \]
  and of course it is less than \(1\).
\end{proof}

%how do we think of the first line (from which the rest follows)? inspired by the first proof of LYM %maybe no?

\begin{remark}
  Equivalently, the number of maximal chains is \(n!\) and the number of them containing a given \(r\)-set is \(r! (n - r)!\), so
  \[
    \sum_{r = 0}^n |\mathcal A_r| r! (n - r)! \leq n!
  \]
  so this is probability in disguise
\end{remark}

\subsection{Shadows}

For \(\mathcal A \subseteq X^{(r)}\), we know \(|\shadow A| \geq |A| \frac{r}{n - r + 1}\) --- but equality is rare (only for \(\mathcal A = \emptyset\) or \(\mathcal A = X^{(r)}\)). The natural question is: given \(|\mathcal A|\), how should we choose \(\mathcal A \subseteq X^{(r)}\) to minimise \(|\shadow A|\)? Informally, this asks how ``tightly'' can we pack some \(r\)-sets.

If \(|A| = \binom{k}{r}\), it is believable that we would take \(A = [k]^{(r)}\), which gives \([k]^{(r - 1)}\). What if \(\binom{k}{r} < |\mathcal A| <\binom{k + 1}{r}\), it is believable that we'd take \([k]^{(r)}\) and some other \(r\)-sets from \([k + 1]^{(r)}\). For example if \(\mathcal A \subseteq X^{(3)}\) with \(|\mathcal A| = \binom{7}{3} + \binom{4}{2}\), we would take
\[
  \mathcal A = [7]^{(3)} \cup \{A \cup \{8\}: A \in [4]^{(2)}\},
\]
i.e.\ take those in \([7]\) of size \(3\) so that they are as tightly packed as possible and then choose some other stuff.

If we increment the size of \(\mathcal A\) by \(1\), it is believable that we should take the above \(\mathcal A\) and adjoin another element from \([4]\). Thus it seems that there is a total order on subsets of \(X\) of a given size, and we just take the first \(|\mathcal A|\).

\subsubsection{Two total orderings on \(X^{(r)}\)}

\begin{definition}[lexicographic order]\index{lexicographic order}
  Given \(A, B \in X^{(r)}\), say \(A = a_1, \dots, a_r, B = b_1, \dots, b_r\), where we use the notation to mean \(a_1 < \dots < a_r\), say \(A < B\) in the \emph{lexicographic} or \emph{lex} order if for some \(i\) have \(a_i < b_i\) and \(a_j = b_j\) for all \(j < i\).
\end{definition}

Equivalently, \(a_i < b_i\), where \(i = \min\{j: a_j \neq b_i\}\). The slogan is ``use small number''.

\begin{eg}
  Lex on \([4]^{(2)}\):
  \[
    12, 13, 14, 23, 24, 34.
  \]

  Lex on \([6]^{(3)}\):
  \begin{align*}
    &123, 124, 125, 126, 134, 135, 136, 145, 146, 156, \\
    &234, 235, 236, 245, 246, 256, 345, 346, 356, 456.
  \end{align*}
\end{eg}

However, in the shadow minimisation problem we want to avoid large numbers as much as possible, i.e.\ keep the largest number as small as possible.

\begin{definition}[colexicographic order]\index{colexicographic order}
  Say \(A < B\) in the \emph{colexicographic} or \emph{colex} order if for some \(i\) have \(a_i < b_i\) and \(a_j = b_j\) for all \(j > i\).
\end{definition}

Equivalently, \(a_i < b_i\) where \(i = \max\{j: a_j \neq b_j\}\). The slogan is ``avoid large number''. Equivalently, \(A < B\) if \(\sum_{i \in A} 2^i < \sum_{i \in B} 2^i\).

\begin{eg}
  Colex on \([4]^{(2)}\):
  \[
    12, 13, 23, 14, 24, 34.
  \]

  Colex on \([6]^{(2)}\):
  \begin{align*}
    &123, 124, 134, 234, 125, 135, 235, 145, 245, 345,\\
    &126, 136, 236, 146, 246, 346, 156, 256, 356, 456.
  \end{align*}
\end{eg}

\begin{note}
  In colex, \([k]^{(r)}\) is an initial segment of \([k + 1]^{(r)}\), meaning that it is the first \(t\) elements for some \(t\). Therefore we could view colex as an enumeration of \(\N^{(r)}\). Try this with lex and see what happens!
\end{note}

Following our heuristics just now, the aim is to show initial segments of colex minimise \(\shadow\), i.e.\ if \(\mathcal A \subseteq X^{(r)}\) and \(\mathcal C \subseteq X^{(r)}\) is the first \(|\mathcal A|\) \(r\)-sets in colex then \(|\shadow \mathcal A| \geq |\shadow \mathcal C|\), This is Kruskal-Katona theorem, the first theorem in combinatorics.
In particular, \(|\mathcal A| = \binom{k}{r}\) implies \(|\shadow \mathcal A| \geq \binom{k}{r - 1}\). However, unless \(\mathcal A\) is written in very nice form, it is very difficult to estimate \(\shadow \mathcal A\).

\subsubsection{Compressions}

The idea is to ``replace'' \(\mathcal A \subseteq X^{(r)}\) with some \(\mathcal A' \subseteq X^{(r)}\) such that
\begin{enumerate}
\item \(|\mathcal A'| = |\mathcal A|\),
\item \(|\shadow \mathcal A'| \leq |\shadow \mathcal A|\),
\item \(\mathcal A'\) ``looks more like \(\mathcal C\)'' than \(\mathcal A\) did.
\end{enumerate}

Ideally, we would compress
\[
  \mathcal A \to \mathcal A' \to \mathcal A'' \to \dots \to \mathcal B
\]
where either \(\mathcal B = \mathcal C\), or \(\mathcal B\) is so similar to \(\mathcal C\) that we can see directly that \(|\shadow \mathcal B| \geq |\shadow \mathcal C|\).

\paragraph{Colex prefers \(1\) to \(2\)}

\begin{definition}[\(ij\)-compression]\index{\(ij\)-compression}
  For \(1 \leq i < j \leq n\), the \emph{\(ij\)-compression} \(C_{ij}\) is defined by for \(A \subseteq X\),
  \[
    C_{ij} (A) =
    \begin{cases}
      A - j + i & \text{if } j \in A, i \notin A \\
      A & \text{otherwise}
    \end{cases}
  \]
  and for \(\mathcal A \subseteq \powerset(X)\),
  \[
    C_{ij} (\mathcal A) = \{C_{ij}(A): A \in \mathcal A\} \cup \{A \in \mathcal A: C_{ij}(A) \in \mathcal A\}.
  \]
\end{definition}

\begin{eg}
  If \(\mathcal A = \{123, 134, 234, 235, 247\}\), then
  \[
    C_{12}(\mathcal A) = \{123, 134, 234, 135, 147\}.
  \]
\end{eg}

\(|C_{ij}(\mathcal A)| = |\mathcal A|\) and after the compression it looks ``more'' like colex than lex. Say \(\mathcal A\) is \emph{\(ij\)-compressed} if \(C_{ij}(\mathcal A) = \mathcal A\). It is also intuitively obvious that the operation is indeed a compression in the sense that it decreases the shadow.

\begin{proposition}
  \label{prop:ij-compression}
  Let \(\mathcal A \subseteq X^{(r)}\), \(1 \leq i < j \leq n\) then
  \[
    |\shadow C_{ij}(\mathcal A)| \leq |\shadow \mathcal A|.
  \]
\end{proposition}

\begin{proof}
  Write \(\mathcal A'\) for \(C_{ij} (\mathcal A)\). We'll show that if \(B \in \shadow \mathcal A' - \shadow \mathcal A\) then \(i \in B, j \neq B\) and \(B \cup j - i \in \shadow \mathcal A - \shadow \mathcal A'\) (so \(B\) has a preimage under \(C_{ij}\)). Then done.

  We have \(B \cup x \in \mathcal A'\) for some \(x \notin B\) and \(B \cup x \notin \mathcal A\). Hence \(i \in B \cup x, j \notin B \cup x\) and \((B \cup x) \cup j - i \in \mathcal A\) (so \(B \cup x\) is the set that has been compressed). Note that \(x \neq i\) else \(B \cup j \in \mathcal A\), contradicting \(B \notin \shadow \mathcal A\).

  Certainly \(B \cup j - i \in \shadow \mathcal A\). Claim that \(B \cup j - i \notin \shadow \mathcal A'\), thereby completing the proof: suppose \((B \cup j - i) \cup y \in \mathcal A'\). We cannot have \(y = i\), else \(B \cup j \in \mathcal A'\), whence \(B \cup j \in \mathcal A\), contradiction. Thus \(j \in (B \cup j - i) \cup y\) and \(i \notin (B \cup j - i) \cup y\), so \((B \cup j - i) \cup y \in \mathcal A\) and \(B \cup y \in \mathcal A\) by definition of \(C_{ij}\). Contradiction.
\end{proof}

\begin{remark}
  We have actually showed that
  \[
    \shadow C_{ij}(\mathcal A) \subseteq C_{ij} (\shadow \mathcal A),
  \]
  ``shadow of compression lives inside compression of shadow''.
\end{remark}

\begin{definition}[left-compressed]\index{left-compressed}
  Say \(\mathcal A \subseteq X^{(r)}\) is \emph{left-compressed} if \(C_{ij}(\mathcal A) = \mathcal A\) for all \(i < j\).
\end{definition}

\begin{proposition}
  \label{prop:left compression decreases shadow}
  Let \(\mathcal A \subseteq X^{(r)}\). Then there exists left-compressed \(\mathcal B \subseteq X^{(r)}\) with \(|\mathcal B| = |\mathcal A|\) and \(|\shadow \mathcal B| \leq |\shadow \mathcal A|\).
\end{proposition}

\begin{proof}
  Morally we only have to show that any sequence of compression terminate. Among all \(\mathcal B \subseteq X^{(r)}\) with \(|\mathcal B| = |\mathcal A|\) and \(|\shadow \mathcal B| \leq |\shadow \mathcal A|\), choose one with
  \[
    \sum_{A \in \mathcal B} \sum_{x \in A} x
  \]
  minimal. Then \(\mathcal B\) is left compressed, as if \(C_{ij}(\mathcal B) \neq \mathcal B\) then we contradict minimality.
\end{proof}

\begin{note}\leavevmode
  \begin{enumerate}
  \item Alternatively, we may apply one \(C_{ij}\) then another and so on --- it must terminate.
  \item In fact we can apply each \(C_{ij}\) at most once if we choose a sensible order.
  \end{enumerate}
\end{note}

Certainly initial segments of colex are left-compressed. The converse is blatantly false, e.g.\ \(A = \{123, 124, 125, 126, 127\}\). We need to do more.

\paragraph{colex prefers \(23\) to \(14\)}

We can compress not only singletons but also sets of larger sizes:

\begin{definition}[\(UV\)-compression]\index{\(UV\)-compression}
  For \(U, V \subseteq X\) with \(|U| = |V|\) and \(U \cap V = \emptyset\), the \emph{\(UV\)-compression} \(C_{UV}\) is defined by for \(A \subseteq X\),
  \[
    C_{UV} (A) =
    \begin{cases}
      A \cup U - V & \text{if } V \subseteq A, U \cap A = \emptyset \\
      A & \text{otherwise}
    \end{cases}
  \]
  and for \(\mathcal A \subseteq X^{(r)}\),
  \[
    C_{UV} (\mathcal A) = \{C_{UV}(A): A \in \mathcal A\} \cup \{A \in \mathcal A: C_{UV}(A) \in \mathcal A\}.
  \]
\end{definition}

\begin{eg}
  If \(\mathcal A = \{123, 134, 235, 145, 146, 157\}\), then
  \[
    C_{23, 14}(\mathcal A) = \{123, 134, 235, 145, 236, 157\}.
  \]
\end{eg}

Similar observation: \(|C_{UV}(\mathcal A)| = |\mathcal A|\). Say \(\mathcal A\) is \emph{\(UV\)-compressed if \(C_{UV}(\mathcal A) =\mathcal A\)}. Sadly, \(C_{UV}\) need \emph{not} decrease shadow --- e.g.\ \(\mathcal A = \{146, 467\}\) then \(C_{23, 14}(\mathcal A) = \{236, 467\}\) so \(|\shadow \mathcal A| = 5, |\shadow C_{23, 14}(\mathcal A)| = 6\). Intuitively, removing one elements and adding another sends a set to a ``close neighbour'' which shares largely the same shadow, but once we start doing \(UV\)-compression, some things are ``moved a long way''.

However, note that \(\mathcal A\) in the above example is not left-compressed. It turns out once you have done the ``smaller'' compressions, doing a larger compression always decrease the shadow. Formally,

\begin{proposition}
  \label{prop:UV-compression}
  Let \(\mathcal A \subseteq X^{(r)}\) and \(U,V \subseteq X\) with \(|U| = |V|\) and \(U \cap V = \emptyset\). Suppose that for all \(x \in U\) exists \(y \in V\) such that \(\mathcal A\) is \((U - x, V - y)\)-compressed then
  \[
    |\shadow C_{UV}(\mathcal A)| \leq |\shadow \mathcal A|.
  \]
\end{proposition}

\begin{proof}
  Write \(\mathcal A'\) for \(C_{UV}(\mathcal A)\). Given \(B \in \shadow \mathcal A' - \shadow \mathcal A\), we'll show that \(U \subseteq B, V \cap B = \emptyset\) and \(B \cup V - U \in \shadow \mathcal A - \shadow \mathcal A'\).

  We have \(B \cup x \in \mathcal A'\) for some \(x \notin B\), with \(B \cup x \notin \mathcal A\). So \(U \subseteq B \cup x, V \cap (B \cup x) = \emptyset\) and \((B \cup x) \cup V - U \in \mathcal A\). Thus certainly \(V \cap B = \emptyset\).

  If \(x \in U\), have \(\mathcal A\) is \((U - x, V - y)\)-compressed for some \(y \in V\) so from \((B \cup x) \cup V - U \in \mathcal A\) we obtain \(B \cup y \in \mathcal A\), contradicting \(B \notin \shadow \mathcal A\). Hence \(x \notin U\) and so \(U \subseteq B\). Also \(B \cup V - U \in \shadow \mathcal A\) (as \((B \cup x) \cup V - U \in \mathcal A\)).

  Suppose \(B \cup V - U \in \shadow \mathcal A'\) then \((B \cup V - U) \cup w \in \mathcal A'\) for some \(w\).
  \begin{enumerate}
  \item if \(w \notin U\), then \(V \subseteq (B \cup V - U) \cup w\) and \(U \cap ((B \cup V - U) \cup w) = \emptyset\) so from \((B \cup V - U) \cup w \in \mathcal A'\) we conclude that both \((B \cup V - U) \cup w \in \mathcal A\) and \(B \cup w \in \mathcal A\), contradicting \(B \notin \shadow \mathcal A\).
  \item if \(w \in U\), we have \(\mathcal A\) is \((U - w, V - z)\)-compressed for some \(z \in V\). From \((B \cup V - U) \cup w \in \mathcal A\) (as it is in \(\mathcal A'\) and contains \(V\), so could not have moved), we deduce \(B \cup z \in \mathcal A\), contradicting \(B \notin \shadow \mathcal A\).
  \end{enumerate}
\end{proof}

\begin{remark}
  We have actually showed that
  \[
    \shadow C_{UV}(\mathcal A) \subseteq C_{UV}(\shadow \mathcal A).
  \]
\end{remark}

\begin{theorem}[Kruskal-Katona]\index{Kruskal-Katona theorem}
  \label{thm:Kruskal-Katona}
  Let \(\mathcal A \subseteq X^{(r)}\) where \(1 \leq r \leq n\) and let \(\mathcal C\) be the initial segment of colex on \(X^{(r)}\) with \(|\mathcal C| = |\mathcal A|\), then
  \[
    |\shadow \mathcal A| \geq |\shadow \mathcal C|.
  \]
  In particular if \(|\mathcal A| = \binom{k}{r}\) then
  \[
    |\shadow \mathcal A| \geq \binom{k}{r - 1}.
  \]
\end{theorem}

\begin{proof}
  Let
  \[
    \Gamma = \{(U, V): U, V \subseteq X, |U| = |V| > 0, U \cap V = \emptyset, \max U < \max V\}
  \]
  which are the ordered pairs \((U, V)\) with \(U < V\) in colex, which are exactly the sensible pairs to do \(UV\) compression on in order to decrease shadow. Define a sequence of set systems \(\mathcal A_0, \mathcal A_1, \dots\) in \(X^{(r)}\) as follow:
  \begin{enumerate}
  \item \(\mathcal A_0 = \mathcal A\).
  \item If \(\mathcal A_k\) is \((U, V)\)-compressed for all \((U, V) \in \Gamma\), then stop the sequence with \(\mathcal A_k\).
  \item If not, choose \((U, V) \in \Gamma\) such that \(\mathcal A_k\) is not \((U, V)\)-compressed with \(U\) \emph{minimal}. Set \(\mathcal A_{k + 1} = C_{UV}(\mathcal A_K)\). Note that for all \(x \in U\), we have \((U - x, V - y) \in \Gamma \cup \{(\emptyset, \emptyset)\}\) for \(y = \min V\) so by \Cref{prop:UV-compression} have \(|\shadow \mathcal A_{k + 1}| \leq |\shadow \mathcal A_k|\).
  \end{enumerate}

  The sequence must terminate, for example because as \(\sum_{A \in \mathcal A_k}\sum_{i \in A} 2^i\) is decreasing in \(k\).

  The final system \(\mathcal B = \mathcal A_k\) satisfies \(|\mathcal B| = |\mathcal A|\) and \(|\shadow B| \leq |\shadow \mathcal A|\). Moreover, \(\mathcal B\) is \((U, V)\)-compressed for all \((U, V) \in \Gamma\). Claim that \(\mathcal B = \mathcal C\):
  \begin{proof}
    Suppose \(\mathcal B\) is not an initial segment of colex. Then there exists \(A < B\) in colex with \(A \notin \mathcal B\) and \(B \in \mathcal B\). But then \(U = A - B\) and \(V = B - A\) have \((U, V) \in \Gamma\) and \(C_{UV}(B) = A\). Absurd.
  \end{proof}
\end{proof}

\begin{remark}\leavevmode
  \begin{enumerate}
  \item Equivalently, we may state the theorem in numerical form by translating initial segments of colex into the size of its constituent ``lex'' parts: if \(\mathcal A \subseteq X^{(r)}\) with
    \[
      |\mathcal A| = \binom{k_r}{r} + \binom{k_{r - 1}}{r - 1} + \dots + \binom{k_s}{s}
    \]
    where \(k_r > k_{r - 1} > \dots > k_s\) and \(s > 0\), then
    \[
      |\shadow \mathcal A| \geq \binom{k_r}{k - 1} + \binom{k_{r - 1}}{r - 2} + \dots + \binom{k_s}{s - 1}.
    \]
  \item In proof of Kruskal-Katona we used only \Cref{prop:UV-compression}, but neither \Cref{prop:ij-compression} nor \Cref{prop:left compression decreases shadow}. However, deriving simpler results for \(ij\)-compression provides motivation and intuition for the further full-fledged proof.
  \item Can we ask for uniqueness? We can check that if \(|\shadow \mathcal A| = |\shadow \mathcal C|\) and \(|\mathcal A| = \binom{k}{r}\) then \(\mathcal A = Y^{(r)}\) for some \(k\)-set \(Y\). Thus this is unique up to isomorphism. But in general it is not true that \(|\shadow \mathcal A| = |\shadow \mathcal C|\) implies \(\mathcal A\) is isomorphic to \(\mathcal C\) (\(\mathcal A \subseteq \powerset(X), \mathcal B \subseteq \powerset(Y)\) are isomorphic if there exists a bijection between \(X\) and \(Y\) sending \(\mathcal A\) to \(\mathcal B\)).
  \end{enumerate}
\end{remark}

What about upper shadow?

\begin{definition}[upper shadow]\index{upper shadow}
  For \(\mathcal A \subseteq X^{(r)}\) where \(0 \leq r \leq n - 1\), the \emph{upper shadow} of \(\mathcal A\) is
  \[
    \shadow^+ \mathcal A = \{A \cup x: A \in \mathcal A, x \notin A\}.
  \]
\end{definition}

Note that \(A < B\) in colex if and only if \(A^c < B^c\) in lex with ground-set order reversed. You can mess around with complement and other set operations, but think about it until it becomes clear!

\begin{corollary}
  Let \(\mathcal A \subseteq X^{(r)}\) where \(0 \leq r \leq n - 1\) and let \(\mathcal C\) be the initial segment of lex with \(|\shadow C| = |\mathcal A|\). Then
  \[
    |\shadow^+ \mathcal A| \geq |\shadow^+ \mathcal C|.
  \]
\end{corollary}

\begin{proof}
  Take complements.
\end{proof}

Also the shadow of an initial segment of colex is again an initial segment of colex. Indeed, if
\[
  \mathcal C = \{A \subseteq X^{(r)}: A \leq a_1,a_2,\dots,a_r\}
\]
then
\[
  \shadow \mathcal C = \{B \subseteq X^{(r - 1)}: B \leq a_2,\dots,a_r\}.
\]

Kruskal-Katona also proves a generalised version of itself:

\begin{corollary}
  Let \(\mathcal A \subseteq X^{(r)}\) and let \(\mathcal C \subseteq X^{(r)}\) be the initial segment of colex with \(|\mathcal C| = |\mathcal A|\). Then
  \[
    |\shadow^t \mathcal A| \geq |\shadow^t \mathcal C|
  \]
  for all \(1 \leq t \leq r\). In particular if \(|\mathcal A| = \binom{k}{r}\) then
  \[
    |\shadow^t \mathcal A| \geq \binom{k}{r - t}.
  \]
\end{corollary}

\begin{proof}
  If \(|\shadow^t \mathcal A| \geq |\shadow^t \mathcal C|\) then \(|\shadow^{t + 1}\mathcal A| \geq |\shadow^{t + 1} \mathcal C|\) by \nameref{thm:Kruskal-Katona}.
\end{proof}

\subsection{Intersecting families}

\begin{definition}[intersecting family]\index{intersecting}
  A family \(\mathcal A \subseteq \powerset(X)\) is \emph{intersecting} if \(A \cap B \neq \emptyset\) for all \(A, B \in \mathcal A\).
\end{definition}

How large can \(|\mathcal A|\) can be? We can achieve \(|\mathcal A| = 2^{n - 1}\) by fixing an element, e.g.\ \(\mathcal A = \{A \subseteq X: 1 \in \mathcal A\}\). We can't beat it either.

\begin{proposition}
  Let \(\mathcal A \subseteq \powerset(X)\) be intersecting. Then \(|\mathcal A| \leq 2^{n - 1}\).
\end{proposition}

\begin{proof}
  For each \(A \subseteq X\), we have at most one of \(A\) or \(A^c\) in \(\mathcal A\).
\end{proof}

\begin{note}
  There are many examples with \(|\mathcal A| = 2^{n - 1}\). In the discrete cube, we can take the ``top'' layer along any element. In the blob graph, we can take \(\{A \subseteq X: |A| > \frac{n}{2}\}\).
\end{note}

Now consider the same problem restricted to one level. How larger can \(|\mathcal A|\) be if \(\mathcal A \subseteq X^{(r)}\) is intersecting? If \(r > \frac{n}{2}\), it is trivial as we can take \(\mathcal A = X^{(r)}\). If \(r = \frac{n}{2}\), the maximum is \(\frac{1}{2} \binom{n}{r}\) --- just choose one of \(A, A^c\) for each \(A \in X^{(r)}\). Thus from now on assume \(r < \frac{n}{2}\).

Take
\[
  \mathcal A = \{A \in X^{(r)}: 1 \in A\},
\]
we have
\[
  |\mathcal A| = \binom{n - 1}{r - 1} = \frac{r}{n} \binom{n}{r},
\]
where the second equality comes from the probability an element of \(X^{(r)}\) containing \(1\). We may also take another family, say
\[
  \mathcal B = \{A \in X^{(r)}: |A \cap \{1, 2, 3\}| \geq 2\}.
\]
For example, in case of \([8]^{(3)}\),
\[
  |\mathcal A| = \binom{3}{2} = 2, |\mathcal B| = 1 + \frac{3}{2} \binom{5}{1} = 16 < 21
\]
so \(\mathcal A\) beats \(\mathcal B\).

\begin{theorem}[Erdős-Ko-Rado]\index{Erdős-Ko-Rado theorem}
  Let \(r < \frac{n}{2}\) and let \(\mathcal A \subseteq X^{(r)}\) be intersecting. Then
  \[
    |\mathcal A| \leq \frac{n - 1}{r - 1}.
  \]
\end{theorem}

\begin{proof}[Proof 1]
  ``Bubble down with Kruskal-Katona''

  For \(A, B \in \mathcal A\) have \(A \cap B \neq \emptyset\), i.e.\ \(A \nsubseteq B^c\). Write
  \[
    \overline{\mathcal A} = \{A^c: A \in \mathcal A\} \subseteq X^{(n - r)}.
  \]
  This says that \(\shadow^{n - 2r}\overline{\mathcal A}\) is disjoint from \(A\). Suppose \(|\mathcal A| > \binom{n - 1}{r - 1}\). Then \(|\overline{\mathcal A}| = \binom{n - 1}{r - 1} = \binom{n - 1}{n - r}\) so by iterated Kruskal-Katona, have
  \[
    |\shadow^{n - 2r} \overline{\mathcal A}| \geq \binom{n - 1}{r}.
  \]
  But \(\binom{n - 1}{r - 1} + \binom{n - 1}{r} = \binom{n}{r}\), i.e.
  \[
    |\shadow^{n - 2r} \overline{\mathcal A}| + |\mathcal A| > |X^{(r)}|,
  \]
  absurd.
\end{proof}

\begin{remark}
  Note that in the proof we ``happen'' to have \(|\overline{\mathcal A}| = \binom{n - 1}{n - r}\). In fact, the numbers have to work as we get equality for \(\mathcal A = \{A \in X^{(r)}: 1 \in A\}\).
\end{remark}

\begin{proof}[Proof 2]
  Consider a cyclic ordering \(c\) of \([n]\), i.e.\ a bijection \(c: [n] \to \Z/(n\Z)\). How many \(A \in \mathcal A\) are intervals (sets of \(r\) consecutive elements) in our ordering? The answer is at most \(r\). Indeed, suppose \(c_1, \dots, c_r \in A\). Then for each \(1 \leq i \leq r - 1\), at most one of the two intervals \(\dots, c_{i - 1} c_i\) and \(c_{i + 1}, c_{i + 2} \dots\) can belong to \(\mathcal A\) so that \(\mathcal A\) is intersecting. Also a given \(r\)-set \(A\) is an interval in exactly \(n \cdot r! \cdot (n - r)!\) of the \(n!\) cyclic orderings. Therefore
  \[
    |\mathcal A| n \cdot r! (n - r)! \leq n! r
  \]
  i.e.
  \[
    |\mathcal A| \leq \frac{(n - 1)!}{(r - 1)! (n - r)!} = \binom{n - 1}{r - 1}.
  \]
\end{proof}

\begin{remark}\leavevmode
  \begin{enumerate}
  \item Equivalently, we are double-counting the edges in the bipartite graph with vertex classes \(\mathcal A\) and all cyclic orderings, in which \(A \in \mathcal A\) is joined to \(c\) if \(A\) is an interval in \(c\).
  \item This method, by constructing an clever auxiliary object and counting a parameter of it in two ways, is called \emph{averaging} or \emph{Katona's method}.
  \end{enumerate}
\end{remark}

We may ask when we have equality in Erdős-Ko-Rado. We can check that equality holds if and only if \(\mathcal A = \{A \in X^{(r)}: i \in A\}\) for some \(i\). This follows (proof 1) from equality case of Kruskal-Katona (when size is nice), or (proof 2) by considering changing the cyclic ordering bit by bit.

\section{Isoperimetric inequalities}

The general theme in this chapter is the question ``how tightly can we pack a subset of given size in a space''. Many results in combinatorics follow from this chapter, and Kruskal-Katona is almost one of the cases.

To motivate the name ``isoperimetric'', observe the basic fact in geometry that among subsets of \(\R^2\) of given area, a disc has smallest perimeter. Similarly among subsets of \(\R^3\) of given volume, a ball has smallest surface area. Among subsets of \(S^2\) of given area, the cap has the smallest perimeter. What are the corresponding notions in discrete settings, say subgraphs of a graph? Obviously the ``volume'' should be the number of vertices. What about the ``surface area'' and more generally, its boundary?

\begin{definition}[boundary]
  For a set \(A\) of vertices in a graph \(G\), the \emph{boundary} of \(A\) is
  \[
    b(A) = \{x \in V(G): x \notin A, xy \in E \text{ for some } y \in A\}.
  \]
\end{definition}

\begin{eg}
  In the following graph if \(A = \{1, 2, 3\}\) then \(b(A) = \{4, 5\}\).
  \begin{center}
    \begin{tikzpicture}
      \Vertex[x=0,y=0]{1}
      \Vertex[x=1,y=0]{2}
      \Vertex[x=0,y=-1]{3}
      \Vertex[x=1,y=-1]{4}
      \Vertex[x=2,y=0]{5}
      \Vertex[x=3,y=0]{6}
      \Vertex[x=2,y=-1]{7}
      \Vertex[x=3,y=-1]{8}
      \Edges(1, 2, 4, 3, 1)
      \Edges(2, 5, 6, 8, 7, 5)
    \end{tikzpicture}
  \end{center}
\end{eg}

An \emph{isopertimetric inequality}\index{isoperimetric inequality} of \(G\) is an inequality of the form
\[
  b(A) \geq f(|A|)
\]
for \(A \subseteq V(G)\). This should be a ``good'' (at least nontrivial) inequality, which warrants that such an inequality should describe a fixed graph, and not applicable to graphs in general.

To minimise the size \(b(A)\) is equivalently to (and usually easier to) the size of

\begin{definition}[neighbourhood]
  A \emph{neighbourhood} of \(A\)
  \[
    N(A) = A \cup b(A) = \{x \in G: d(x, A) \leq 1\}
  \]
  where \(d\) the usual graph distance.
\end{definition}

A natural guess for boundary minimising \(A\) is often
\[
  B(x, r) = \{y \in G: d(x, y) \leq r\}.
\]
Let's test it. Take \(|A| = 4\) in \(Q_3\). Then in the discrete \(A\) is either a hyperplane or a ball. We have \(|b(A)| = 3\) for ball and \(|b(A)| = 4\) for hyperplane. This works so we conjecture that ``balls are the best'', i.e.\ if \(|A| = |X^{(\leq r)}|\) then \(|N(A)| \geq |X^{(\leq r + 1)}|\). When \(|A|\) is strictly between \(\sum_{i = 0}^r \binom{n}{i}\) and \(\sum_{i = 0}^{r + 1} \binom{r}{i}\), conjecture that \(b(A)\) is minimised if \(A = X^{(\leq r)} \cup B\) for some \(B \subseteq X^{(r + 1)}\). This is a \emph{Hamming ball}\index{Hamming ball}. Note that if we assume this result then
\[
  N(A) = X^{(\leq r + 1)} \cup \shadow^+ B,
\]
so by Kruskal-Katona, we'd take \(B\) to be an initial segment of lex on \(X^{(r + 1)}\). This strongly suggests that there is a ordering on \(\powerset(X)\), where subsets are ordered by sizes and lex is used to decide order among subsets of equal size, such that the boundary minimising sets are the initial segments of this order.

\begin{definition}[simplicial order]\index{simplicial order}
  The \emph{simplicial order} on \(Q_n\) is defined by \(x < y\) if either \(|x| \leq |y|\) or \(|x| = |y|\) and \(x < y\) in lex.
\end{definition}

\begin{notation}
  In this chapter we use roman instead of curly upper case letters such as \(A, B\) to denote a family of subsets of \(X\), and lower case letters such as \(x, y\) to denote element of \(X\). This is compatible with the vertex convention in graph theory.
\end{notation}

Our aim is to show that initial segments of simplicial are the best. As we have Kruskal-Katona at disposal, we would like to do similar ``compressions'' to move a set ``closer'' to Hamming ball. In the proof of Kruskal-Katona the compressions \(C_{UV}\) can be seen as operations of dimension \(|U|\) in the discrete cube. Here we consider ``codimension \(1\)'' compression.

\begin{definition}[section]\index{section}
  Given \(A \subseteq Q_n\) and \(1 \leq i \leq n\), the \emph{\(i\)-sections} are the set systems \(A^{(i)}_+, A^{(i)}_- \subseteq \powerset(X - i)\), thought as \(Q_{n - 1}\), given by
  \begin{align*}
    A^{(i)}_+ &= \{x \in A: i \notin x\} \\
    A^{(i)}_- &= \{x - \{i\}: x \in A, i \in x\}
  \end{align*}
  If it is clear which \(i\) is being considered we may omit the superscripts for simplicity.
\end{definition}

\begin{definition}[\(i\)-compression]\index{\(i\)-compression}
  Define the \emph{\(i\)-compression} \(C_i(A)\) of \(A\) by compression on its \(i\)-sections, i.e.\ \(C_i(A)^{(i)}_+\) is initial segment of \(Q_{n - 1}\) of size \(|A^{(i)}_+|\) and \(C_i(A)^{(i)}_-\) is initial segment of \(Q_{n - 1}\) of size \(|A^{(i)}_-|\).
\end{definition}

\begin{eg}
  (graph)
\end{eg}

Certainly \(|C_i(A)| = |A|\). Moreover, \(C_i(A)\) ``looks more like'' the initial segment of simplicial than \(A\) did.

Say \(A\) is \emph{\(i\)-compresses} if \(C_i(A) = A\).

\begin{theorem}[Harper]\index{Harper's theorem}
  Let \(A \subseteq Q_n\) and \(C\) be the initial segment of simplicial order with \(|C| = |A|\). Then
  \[
    |N(A)| \geq |N(C)|.
  \]
\end{theorem}

Same as in Kruskal-Katona, what we are really interested in is the ``nice'' size, i.e.\ if \(|A| \geq \sum_{i = 0}^r \binom{n}{i}\) then
\[
  |N(A)| \geq \sum_{i = 0}^{r + 1} \binom{n}{i}.
\]

\begin{remark}\leavevmode
  If \(A\) is a Hamming ball then done by Kruskal-Katona. Conversely, Harper implies Kruskal-Katona. Indeed, given \(B \subseteq X^{(r)}\), apply Harper to \(A = B \cup X^{(\leq r - 1)}\).
\end{remark}

% Note to self: proofread until told to stop

\begin{proof}
  Induction on \(n\): \(n = 1\) is trivial. Given \(A \subseteq Q_n\) where \(n > 1\), fix \(1 \leq i \leq n\). Claim that
  \[
    |N(C_i(A))| \leq |N(A)|.
  \]
  \begin{proof}
    % TODO: proofread this proof
    Write \(B = C_i(A)\). Given an element in the neighbourhood of \(A\), it is either in the downstairs part \(A_-\) or the upstairs part \(A_+\) so
    \[
      |N(A)| = |A_+ \cup N(A_-)| + |A_- \cup N(A_+)|
    \]
    Similarly
    \[
      |N(B)| = |B_+ \cup N(B_-)| + |B_- \cup N(B_+)|.
    \]
    Now \(|B_+| = |A_+|\) and \(|N(B_-)| \leq |N(A_-)|\) by induction. But \(N(B_-)\)  is an initial segment of simplicial (on \(Q_{n - 1}\)), as is \(B_+\), so \(N(B_-)\) and \(B_+\) are nested. Hence
    \[
      |B_+ \cup N(B_-)| \leq |A_+ \cup N(A_-)|
    \]
    and similarly
    \[
      |B_- \cup N(B_+)| \leq |A_- \cup N(A_+)|
    \]
    thus
    \[
      |N(B)| \leq |N(A)|.
    \]
  \end{proof}

  Among all \(B \subseteq Q_n\) with \(|B| = |A|\) and \(|N(B)| \leq |N(A)|\), choose one with \(\sum_{x \in B} f(x)\) minimal, where \(f(x)\) is position of \(x\) in simplicial ordering of \(Q_n\). Then \(B\) is \(i\)-compressed for all \(i\).

  Must such \(B\) be an initial segment of simplicial? Unfortunately, no. For example \(A = \{\emptyset, 1, 2, 12\} \subseteq Q_3\). However, we have

  \begin{lemma}
    Let \(B \subseteq Q_n\) be \(i\)-compressed for all \(i\) but not an initial segment of simplicial order. Then if \(n\) is odd, say \(n = 2k + 1\), we have
    \[
      B = X^{(\leq k)} - \underbrace{\text{ last \(k\)-set }}_{(k + 2)(k + 3) \dots (2k)(2k + 1)} \cup \underbrace{\text{ first \((k + 1)\)-set}}_{12\dots k(k + 1)}
    \]
    whereas if \(n\) is even, say \(n = 2k\), we have
    \[
      B = X^{(\leq k - 1)} \cup \{x \in X^{(k)}: 1 \in x\} - \underbrace{\text{ last \(k\)-set with \(1\) }}_{1(k + 2)(k + 3) \dots (2k)} \cup \underbrace{\text{ first \(k\)-set without \(1\)}}_{234\dots k (k + 1)}
    \]
  \end{lemma}
  Then done as in each case have \(|N(B)| \geq |N(C)|\).

  \begin{proof}
    Have \(x \notin B, y \in B\) for some \(x < y\) in simplicial. Cannot have \(i \in X, i \in y\) as \(B\) is \(i\)-compressed, and cannot have \(i \notin x, i \notin y\), again as \(B\) is \(i\)compressed. So for each \(i\), \(i\) belongs to exactly one of \(x\) or \(y\). Thus \(y = x^c\).

    Hence for each \(y \in B\), at most one of \(x < y\) has \(x \notin B\) (namely \(y^c\)) and for each \(x \notin B\), at most one of \(y > x\) has \(y \in B\) (namely \(x^c\)). Hence
    \[
      B = \{z: z \leq y\} - \{x\}
    \]
    where \(x\) is the immediate predecessor of \(y\) and \(x = y^c\).

    But then \(x\) is the last \(k\)-set (if \(n = 2k + 1\)) or last \(k\)-set containing \(1\) (if \(n = 2k\)), by definition of simplicial ordering.
  \end{proof}
\end{proof}

\begin{remark}\leavevmode
  \begin{enumerate}
  \item It is also possible to prove Harper by \(UV\)-compression.
  \item We can also use these ``codimension \(1\)'' compressions to prove Kruskal-Katona directly.
  \end{enumerate}
\end{remark}

In analysis, we often use isoperimetric inequality in blow-up form, e.g.\ give a disk, if we expand it a little bit by \(\varepsilon > 0\) at some point then the resulting shape has perimeter at least as large as that of the disk.

\begin{definition}[\(t\)-neighbourhood]
  For \(A \subseteq Q_n\), the \emph{\(t\)-neighbourhood} of \(A\) is
  \[
    A_{(t)} = N^t(A) = \{x \in Q_n: d(x, A) \leq t\}
  \]
\end{definition}

\begin{corollary}
  Let \(A \subseteq Q_n\) with \(|A| = |X^{(\leq r)}|\). Then for \(1 \leq t \leq n - r\), have
  \[
    |A_{(t)}| \geq |X^{(\leq r + t)}|.
  \]
\end{corollary}

\begin{proof}
  Harper and induction.
\end{proof}

To get a better for for what the corollary is saying, we'll need some estimates and things like \(\sum_{i = 0}^r \binom{n}{i}\).
% c.f. central limit theorem

\begin{proposition}
  Let \(0 < \varepsilon < \frac{1}{4}\), then
  \[
    \sum_{i = 0}^{\floor{(\frac{1}{2} - \varepsilon) n}} \binom{n}{i} < \frac{1}{\varepsilon} \exp (- \frac{\varepsilon^2 n}{2}) 2^n.
  \]
\end{proposition}

i.e.\ for fixed \(\varepsilon\), the sum is an exponentially small fraction of \(2^n\)

In standard deviation language, going \(\sim \varepsilon\) standard deviates from mean

\begin{proof}
  For \(i \leq (\frac{1}{2} - \varepsilon) n\),
  \[
    \frac{\binom{n}{i - 1}}{\binom{n}{i}}
    = \frac{i}{n - i + 1}
    \leq \frac{\frac{1}{2} - \varepsilon}{\frac{1}{2} + \varepsilon}
    = 1 - \frac{2\varepsilon}{\frac{1}{2} + \varepsilon}
    \leq 1 - 2\varepsilon.
  \]
  Hence by comparison with geometric progression
  \[
    \sum_{i = 0}^{\floor{(\frac{1}{2} - \varepsilon) n}} \binom{n}{i}
    \leq \frac{1}{2\varepsilon} \binom{n}{\floor{(\frac{1}{2} - \varepsilon) n}}.
  \]
  Similarly
  \[
    \binom{n}{\floor{(\frac{1}{2} - \varepsilon)n}}
    \leq (1 - \varepsilon)^{\varepsilon n / 2 - 1} \binom{n}{\floor{(\frac{1}{2} - \frac{\varepsilon}{2}) n}} %\text{ by replacing \varepsilon with \varepsilon/2}
    \leq 2(1 - \varepsilon)^{\varepsilon n/2} 2^n
    \leq 2 \exp{-\varepsilon \varepsilon n /2} \cdot 2^n
  \]
  Therefore
  \[
    \sum_{i = 0}^{\floor{(\frac{1}{2} - \varepsilon)n}} \leq \frac{1}{2\varepsilon} 2 \exp{- \varepsilon^2n /2} 2^n.
  \]
\end{proof}

% end of proofread

Missed a lecture on 06/11/18

\printindex
\end{document}

% Three parts:
% Chapter 1: set systems
% Chapter 2: isoperimetric inequalities
% Chapter 3: projections
% Books: Combinatorics, Bollobas, CUP 1986, excellent for Chapter 1 and Chapter 2 (and gentle!) and for future development of the course;
% Combinatorics of finite sets, Anderson, OUP 1987, simple and clear, good for Chapter 1