\documentclass[../../../../main]{subfiles}
\graphicspath{{images/cv-basics/}}

\begin{document}

\paragraph{Computer vision basics}

\textbf{Digital images} are the first step in any computer vision process. They represent an electronic snapshots of something that we see, for instance, captured scene by photo camera or some scanned document and etc. It got used to sample all digital images as a set of pixels (picture elements or grid of dots, in other words). Any pixel is represented as a binary value (set of zeros and ones) so these values illustrate tonality of pixels (white, black, etc.). The binary digits or bits are interpreted and read by a computer to produce an analog version of the image in order to display or print it \cite{digital_images}.

Images have a property called "Bit Depth", its value is constituted by number of bits used to represent a pixel. Image tonality is dependant on bits depth so the larger its value the larger variety of possible tones for the pixel. Digital images could be monochrome (black and white), gray-scale, or color. For example, pixels in monochrome images are depicted as one bit (zero or one, black and white); gray-scale images, on the contrary, can be made of multiple bits, usually from two to eight but can be more. Color digital images are another side of coin, they are typically represented as a set of bits in range from 8 to 24 or more. Taking the 24-bit images into the view, their values are usually divided into 3 group, first 8 bits for red color, next 8 for green, and last 8 for blue.
In case of \ac{RGB} color format, here is a list that shows how possible variety of tones is dependant on number of bits per pixel:

\begin{itemize}
    \item 1 bit, ($2^1$) = 2 tones
    \item 2 bits, ($2^2$) = 4 tones
    \item 3 bits, ($2^3$) = 8 tones
    \item 4 bits, ($2^4$) = 16 tones
    \item 8 bits, ($2^8$) = 256 tones
    \item 16 bits, ($2^{16}$) = 65,536 tones
    \item 24 bits, ($2^{24}$) = 16.7 million tones
\end{itemize}

\textbf{Computer vision pipeline} can be divided into 3 main phases: low-level processing where image is converted into another image (blurring, sharpening, thresholding), mid-level processing where image results into set of features (\ac{FAST}, \ac{SURF} feature detection algorithms), high-level processing where features are analyzed and some data is received (\ac{CCA}) \cite{cv_basics_linda_shapiro}.

\textbf{Thresholding}. In order to remove noise and make gray-scale image more precise for computer vision it should be thresholded. There are also some algorithms that can threshold color images but they produce gray-scale images as a result. There are multiple thresholding algorithms and they produces very different result due to complexity of computations.
There are two the most simplest thresholding algorithms: global and adaptive thresholding. The first one is very primitive, it is provided with a threshold value and compares each pixel intensity whether its larger or lower this value. The intensity of a pixel is calculated as $((R+G+B)/3))$. Based on the comparison result it replaces pixel intensity with either 0 or maximal intensity and this is displayed in figures \ref{fig:Global Threshold}, \ref{eq:global_threshold}. There are multiple variants of this operation, it can be binary, inverted binary, truncated, truncated to zero, threshold to zero and inverted \cite{digital_image_processing_gonzalez}.

\begin{figure} [ht!]
    \begin{center}
        \includegraphics[width=200pt]{Threshold_Tutorial_Theory_Base_Figure}
        \includegraphics[width=200pt]{Threshold_Tutorial_Theory_Binary}
        \caption{Global Threshold example \cite{opencv_docs_basic_tresholding_operations}.}
        \label{fig:Global Threshold}
    \end{center}
\end{figure}

\begin{figure} [ht!]
  \begin{center}
     newIntensity =
        \begin{cases}
            maxValue, & \mbox{if } currentIntensity\mbox{ > } tresh\\
            0, & \mbox{if } currentIntensity\mbox{ < } tresh
        \end{cases}
      \label{eq:global_threshold}
      \caption{Equation of Global Threshold.}
  \end{center}
\end{figure}

\newpage

Truncated threshold \cite{opencv_docs_basic_tresholding_operations}, on the contrary to binary, replaces only values that are larger or lower the threshold value, depending on whether operation is inverted or not, not both at the same time. Figures \ref{fig:Global Truncated Threshold} and \ref{eq:global_truncated_thresholding} illustrates the computations.

\begin{figure} [ht!]
    \begin{center}
        \includegraphics[width=200pt]{Threshold_Tutorial_Theory_Truncate}
        \caption{Global Truncated Threshold example \cite{opencv_docs_basic_tresholding_operations}.}
        \label{fig:Global Truncated Threshold}
    \end{center}
\end{figure}

\begin{figure} [ht!]
  \begin{center}
     newIntensity =
        \begin{cases}
            thresh, & \mbox{if } currentIntensity\mbox{ > } tresh\\
            currentIntensity, & \mbox{if } currentIntensity\mbox{ < } tresh
        \end{cases}
    \label{eq:global_truncated_thresholding}
    \caption{Equation of Global Truncated Threshold.}
  \end{center}
\end{figure}

Threshold to zero is quite similar to the simple threshold but instead of putting maximal value it keeps the current pixel value. This is illustrated in figures \ref{fig:Global Threshold to Zero} and \ref{eq:global_thresholding_to_zero}.

\begin{figure} [!ht]
    \begin{center}
        \includegraphics[width=200pt]{Threshold_Tutorial_Theory_Zero}
        \caption{Global Threshold to Zero example \cite{opencv_docs_basic_tresholding_operations}.}
        \label{fig:Global Threshold to Zero}
    \end{center}
\end{figure}

\begin{figure} [!ht]
  \begin{center}
     newIntensity =
        \begin{cases}
            currentIntensity, & \mbox{if } currentIntensity\mbox{ > } tresh\\
            0, & \mbox{if } currentIntensity\mbox{ < } tresh
        \end{cases}
     \label{eq:global_thresholding_to_zero}
     \caption{Equation of Global Threshold to Zero.}
  \end{center}
\end{figure}

\newpage

Adaptive thresholding \cite{learning_opencv}, on the other hand, does not use fixed threshold value, it computes this value from surrounding pixels, for instance, it takes an eleven by eleven square where the needed pixel is in the center and computes threshold value from this block of pixels. The demonstration of this operation can be seen in figure \ref{fig:adaptiveThresholdingLogic} This algorithm is much slower but it produces much better results and it is more appropriate for real world tasks, since it works better for images with varying illumination.

\begin{figure} [!ht]
    \begin{center}
        \includegraphics[width=200pt]{adaptive_thresholding_logic}
        \caption{Adaptive thresholding logic demonstration.}
        \label{fig:adaptiveThresholdingLogic}
    \end{center}
\end{figure}

\textbf{Otsu's Binarization} uses different approach to calculate threshold value, it works with bi-modal images (histogram has 2 peaks) so the threshold value is taken approximately in the middle of these two peaks. The figure \ref{fig:otsuThresholding} illustrates the results of thresholding for images with 2 peaks histograms. The algorithm is accurate only for bi-modal images, it's better not to use it for others, results will be unsatisfying \cite{opencv_docs_thresholding}.

\begin{figure} [!ht]
\begin{center}
\includegraphics[width=200pt]{OpenCV_Otsu_Thresholding2}
\caption{Otsu's Binarization demonstration.}
\label{fig:otsuThresholding}
\end{center}
\end{figure}

\newpage

\textbf{Image Filtering}. In computer vision filtering is used to clear images from noise. There are multiple common types of it: "Salt and pepper" noise that contains multiple randomly located black and white pixels, "Impulse noise" that contains randomly located white pixels, and Gaussian noise where noise is Gaussian-distributed \cite{opencv_docs_image_smoothing}.
One of the most important entities in the image filtering is convolution matrix, also known as kernel or mask, which is a small matrix used for image filtering. It is related to a form of mathematical convolution. In addition, it should be mentioned that convolution is not traditional matrix multiplication, even though it is denoted by * (asterisk). It is the process of adding each element of the image to its local neighbors, weighted by the kernel. The figure \ref{eq:mathematical_convolution} demonstrates 2 matrices convolution process.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        \begin{pmatrix}
           \begin{bmatrix}
               a & b & c \\
               d & e & f \\
               g & h & i
           \end{bmatrix}
           $*$
           \begin{bmatrix}
               1 & 2 & 3 \\
               4 & 5 & 6 \\
               7 & 8 & 9
           \end{bmatrix}
        \end{pmatrix}
        \left[ \begin{array}{cc} 2, 2 \end{array} \right] = (i*1) + (h*2) + (g*3) + (f*4) + \\ (e*5)+ (d*6) + (c*7) + (b*8) + (a*9)
    \end{equation}
  \label{eq:mathematical_convolution}
  \caption{Example of convolution}
\end{figure}

There are many filters for various purposes developed, some of them are \ac{LPF} that are used for removing noise or blurring and \ac{HPF} used to find edges, etc.

In order to blur an image it should be convolved with a low-pass filter mask. In fact, this operation removes high frequency content, such as noise or edges, for instance, it may result in edges being blurred. It is important to mention that edges blurring effect is the result of only some specific techniques.

The Gaussian blur is often used in computer graphics software and as initial processing step in computer vision algorithms for enhancing images. Applying this filter to an image is convolving the pixels with a Gaussian function. As a result, the image with reduced high frequency components effect is produced. The mentioned action is known as a two dimensional Weierstrass transformation.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        G(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{x^2}{2\sigma^2}}
    \end{equation}
    \begin{equation}
        G(x,y)=\frac{1}{2\pi\sigma^2}e^{-\frac{x^2+y^2}{2\sigma^2}}
    \end{equation}
    \label{fig:gaussian_functions}
    \caption{Gaussian functions for one and two dimensions}
\end{figure}

The figure \ref{fig:gaussian_functions} illustrates Gaussian functions for one and two dimensions. In the Gaussian functions $x$ and $y$ represent the distances from the origin in the horizontal and vertical axis while $\sigma$ represents the standard deviation of the Gaussian distribution. In order to build a convolution matrix two dimensional function is used to get concentric circles with a Gaussian distribution. The values of this distribution from the center points are used in convolution matrix.

Median Filtering is quite simple, it computes the median of all the pixels from the current pixel neighborhood, so called window, and the central pixel is replaced with this median value. The convolution matrix size for this technique must be a positive odd integer. This type filtering is effective for removing such kind of noise as so called "salt and pepper".

Average filtering or Mean filter is very similar to Median but the value for replacement is computed as the average value for the pixel neighborhood.

Bilateral filtering is significantly different from the others, this filtering technique removes noise without blurring edges, it preserves them. It makes much deeper analysis then Gaussian filter, checks pixel intensity or if some of them lie on the same edge and etc. This filter has a Gaussian filter under the hood for space domain and also uses a function of pixel intensity differences, that is also a multiplicative Gaussian filter component. This analysis helps to keep edges unblurred while noise is removed but operation is taking some time therefore its much slower than other competitors.

\begin{figure} [!ht]
  \centering
    \begin{equation}
    \label{eq:bilateral_filter}
        I^{filtered}(x)=\frac{1}{W_p}\sum_{x_i\in\Omega} I(x_i)f_r(||I(x_i)-I(x)||)g_s(||x_i-x||)
    \end{equation}
    \begin{equation}
       W_p=\sum_{x_i\in\Omega} f_r(||I(x_i)-I(x)||)g_s(||x_i-x||)
    \end{equation}
  \caption{The bilateral filter definition and its normalization term}
\end{figure}

$I^{filtered}$ from the equation \ref{eq:bilateral_filter} is the resulting image; $I$ is the initial image, $x$ is the coordinate of the current pixel; $\Omega$ is the neighborhood, so called window, with $x$ in its center; $f_r$ and $g_s$ are two convolution matrices for smoothing and they could be Gaussian functions.

\textbf{Morphological transformations} are widely used and simple operations based on the image shape, they are techniques for analysis and processing of geometrical structures. Initially they were designed for monochrome digital images but nowadays they can also be applied to gray-scale image and different spatial structures as graphs, solids, or meshes. The most common morphological operations are erosion, dilation, opening and closing \cite{morphology_website}. The mentioned morphological operations will be explained in the following text.

A structuring element in morphological transformations is a shape mask. It can be of any shape or size that can be represented digitally.

\textbf{Erosion operation} moves the boundaries of foreground object away. The main idea of erosion transformation is similar to soil erosion. This operation can be described as the window moving through the image, action is similar to 2D convolution and checks whether all items in the window for the current pixels are the same value as pixel value itself. If the pixel is 1 and all kernel items are ones then its value is kept, otherwise eroded to zero. Definitely, this is related to binary images erosion that is widely used in computer vision. This morphological operation is useful for removing small white noise objects or for detaching two connected objects and etc. Figure \ref{fig:erosion} demonstrates mathematical definition of erosion for monochrome and gray-scale images. Equation \ref{eq:binary_erosion} illustrates definition for binary images where $A$ is an image and $B$ is a structuring element. On the other hand, equation \ref{eq:gray_scale_erosion} displays the same for gray-scale images where the image is denoted as $f()$, structuring element is denoted as $b()$ and $infimum$ is the \ac{GLB}.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        A \ominus B = \underset{b\in B}{\cap} A_b
        \label{eq:binary_erosion}
    \end{equation}
    \begin{equation}
       (f \ominus b)(x) = \underset{y\in E}{\text{infimum}}[f(y)-b(y-x)]
       \label{eq:gray_scale_erosion}
    \end{equation}
  \label{fig:erosion}
  \caption{Definitions of erosion for binary and gray-scale images}
\end{figure}

\textbf{Dilation}, on the contrary, keeps pixel value "1" in case if at least one of the elements in the neighborhood is "1". This operation enlarges the white space in the image and usually used straight after erosion. The idea under this action is that erosion has removed noise and increasing white area won't bring it back while it may connect broken parts of objects and restore shrinked objects. Figure \ref{fig:dilation} illustrates the mathematical definition of dilation where $supermum$  is the \ac{LUB}.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        A \oplus B = \underset{b\in B}{\cup} A_b
    \end{equation}
    \begin{equation}
       (f \oplus b)(x) = \underset{y\in E}{\text{supermum}}[f(y)+b(y-x)]
    \end{equation}
    \label{fig:dilation}
  \caption{Definitions of dilation for binary and gray-scale images}
\end{figure}

\textbf{Opening} is the name for two operations, erosion followed by dilation. Its mathematical definition is showed in figure \ref{fig:opening}. As was mentioned above, it is useful for removing noise.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        A \circ B = (A \ominus B) \oplus B
    \end{equation}
    \label{fig:opening}
  \caption{Definition of opening}
\end{figure}

\textbf{Closing} is the name of reversed Opening where dilation is followed by erosion. Its mathematical definition is displayed in figure \ref{fig:closing}. Most commonly used in computer vision for closing little holes in the foreground objects and etc.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        A \bullet B = (A \oplus B) \ominus B
    \end{equation}
  \label{fig:closing}
  \caption{Definition of closing}
\end{figure}

\textbf{Morphological gradient} in computer vision is simply a difference between dilation and erosion of an input image. This operation is useful for edge detection and segmentation purposes because it creates an image where each pixel value represents the contrast intensity among neighboring pixels. Figure \ref{fig:morphological_gradient} illustrates mathematical definition of morphological gradient where $f$ is a gray-scale image and $b$ is structuring element.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        G(f) = f \oplus b - f \ominus b
    \end{equation}
    \label{fig:morphological_gradient}
  \caption{Definition of morphological gradient.}
\end{figure}

\textbf{Feature detection and description}. In image processing there is no way to define image features as something exact and there is no universal algorithm to get them from an image. Usually the definition of features comes from the problem that is trying to be solved but in general they can be characterized as \ac{POI} for a given problem.
Feature detection is an operation of finding and analyzing image for some interesting points; it often results in the form of points, lines, curves, or regions lists. Detection and analyzing operations could be really time consuming or require large computational power thus feature detection algorithms could become a part of more high level algorithms so that they are called only on some localized regions to improve speed and time complexity \cite{opencv_docs_understanding_features}.

Some popular types of features have become very popular and are widely used, they are edges \cite{cv_basics_3_linda_shapiro}, corners (points of intersection), and blobs (regions of interesting points).
Edges, in general, are defined as points on the edge (boundary) in between image blobs. Corners are referred as point-like features,however the name is just used by tradition. In fact those features are not corners at every scenario. They were developed from the edge detection when people tried to analyze rapid changes in direction, which is a corner. Blob features are good when images are too smooth so that corners can't be detected. These features contain additional information about image structures in terms of regions, in contrast to how point like features perform \cite{feature_matching_presentation}.

Feature extraction, on the other hand, is the operation applied to the image with the information retrieved from the detection step, it proceeds the extraction of image region around the feature. This action gives a feature descriptor as a result despite the fact it may consume a large amount of computational power.

Nowadays, many different types of features have been defined, thus many different feature detection and extraction algorithms have been developed. The following text covers feature detection algorithms such as Canny edge detection, Harris corner detection, \ac{SIFT}, \ac{SURF} and etc.

\textbf{Canny edge detection} is one of the most popular edge detection algorithms. It is named by its author, John Canny. This algorithm has multiple stages: noise reduction, computing intensity gradient, non-maximum suppression, hysteresis thresholding \cite{opencv_docs_canny}. It uses 5 by 5 Gaussian filter to reduce noise because edge detection operation is very vulnerable to noise. Image is filtered with Sobel matrix \cite{opencv_docs_sobel_derivatives} to get horizontal and vertical direction first derivatives which is done to find edge gradient and direction. Figure \ref{fig:canny_gradient} illustrates the edge gradient definition where $G_x$ and $G_y$ are first derivatives in horizontal and vertical directions. After this, image is filtered to reduce pixels that might not be components of the edge. Filtering is done by checking local maximum of the pixel in the direction of gradient through the neighboring pixels. In short, if the result of this check gives a local maximum, pixel is kept for the next stage, otherwise put to zero. Binary image is given in the end of this stage. Last step is analysis of detected edges, they are filtered (thresholded) on the basis of two thresholding values, minimum and maximum. If the pixel intensity gradient is larger then maximum, it is marked as edge, if lower then minimum then discarded. The most interesting case is when the value lies in between, in this case edges are checked for connectivity with that are already marked as real edge, if the connection exists they are also marked as edges, otherwise discarded.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        gradient(G) = \sqrt{G_x^2+G_y^2}
    \end{equation}
     \begin{equation}
        angle(\theta) = \tan^{-1}\bigg(\frac{G_y}{G_x}\bigg)
    \end{equation}
    \label{fig:canny_gradient}
  \caption{Definition of gradient and direction.}
\end{figure}

\textbf{Harris corner detection} algorithm is developed by Chris Harris and Mike Stephens. They were one of the earliest researchers who were trying to find corners, regions in the image where intensity value varies in all directions in the large range \cite{opencv_docs_harris_corners_1}. The idea was to get the difference in intensity in all directions.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        E(u,v) = \sum_{x,y} w(x,y) [I(x+u,y+v) - I(x,y)]^2
    \end{equation}
    \label{fig:harris_function}
  \caption{Definition of intensity difference.}
\end{figure}

For corner detection $E(u,v)$ function is maximized by applying Taylor Expansion. This function is demonstrated in figure \ref{fig:harris_function}  where $w(x,y)$ is a window function, $I(x+u,y+v)$ is a shifted intensity and $I(x,y)$ is the current intensity.  This was done to create a score function that will determine whether the corner is in the window or not. Eigenvalues $\lambda_1$ and $\lambda_2$ are retrieved from the matrix created in the process of maximizing $E(u,v)$ and these define what region is corner. The region is determined to be an edge when its score is less then zero, that happens when $\lambda_1>>\lambda_2$ or $\lambda_1<<\lambda_2$, which means that the difference between them should be significant. On the contrary, if the score is large, which happens when lambdas are large and approximately equal, region is determined as a corner \cite{opencv_docs_harris_corners_2}. Scoring function is illustrated in the figure \ref{fig:harris_scoring}.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        R = \lambda_1\lambda_2 - k(\lambda_1+\lambda_2)^2
    \end{equation}
    \label{fig:harris_scoring}
  \caption{Definition of the Harris scoring function.}
\end{figure}

On the other hand, Jianbo Shi and Carlo Thomasi in 1994 wrote a paper called "Good features to track" \cite{good_feature_to_track_shi_tomasi}, where they suggested to take another scoring function which gave much better results in comparison with Harris Corner Detection. The idea was to compare results of scoring function to a threshold value. In their version, the determined region is a corner only if both $\lambda_1$ and $\lambda_2$ are larger then the score, which is minimum from both lambda values. This is illustrated in figure \ref{fig:shi_tomasi_scoring}.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        R = min(\lambda_1,\lambda_2)
    \end{equation}
  \label{fig:shi_tomasi_scoring}
  \caption{Definition of the Shi-Tomasi scoring function.}
\end{figure}

\textbf{\ac{SIFT}} is an algorithm authored by David Lowe in 2004 and described in the article "Distinctive Image Features from Scale-Invariant Keypoint" \cite{distinctive_features_si_lowe}. The author has created it due to the scale problems for corner detection in already created algorithms, while corner remains a corner after rotation, scaling breaks the logic. Algorithm contains 5 stages: scale-space extrema detection, key point localization, orientation assignment, key point descriptor and key point matching. First of all, it uses different windows to detect key points with different scale and it uses scale-space filtering for this. \ac{LoG} is used as a blob detector with a scale parameter $\sigma$. Then it searches for the local maxima through scale-space and results into a list of tuples $(x,y,\sigma)$ that can be a potential key points. Since the  \ac{LoG} is a bit costly operation \ac{SIFT} uses \ac{DoG} as it is less computationally expensive and, in fact, is \ac{LoG}'s approximation. It is done by getting difference of blurring image with two different values ($\sigma$). After \ac{DoG} is complete, scale-space is searched for a local extrema and if it is found, that is a potential key point. Paper gives some optimal values retrieved by empirical research, $\sigma$ value is $1.6$, and another $\sigma$ is $1.6\sqrt{2}$. Potential key points are found but they are not accurate enough.To get more precise location of extrema algorithm uses Taylor series expansion of scale-space and compares intensity of current extrema with threshold value thus if the intensity is lower then threshold value the key point is rejected. It also uses similar to Harris Corner detection edge filtering since \ac{DoG} is not good at that and discard those key points whose ration is greater then a threshold. Next, in order to be invariant to image rotation, each key point is assigned with orientation. To achieve this orientation histogram covering 360 degrees is created. The next step is creation of key point descriptor from the 16 by 16 neighborhood of the current key point. This region is divided into 16 sub-locks where for each an orientation histogram is created. It also performs the number of operations to be resistant to illumination changes, rotation changes and etc. Total of 128 bin values are represented as a vector to be the definition of a key point descriptor. The last step is key points matching, that is done by identifying the nearest neighbour. Due to the number reasons like noise or others the second nearest match may be very close to the first one and in this case ratio between them is taken and depending on its value they might be rejected or not \cite{opencv_docs_sift}.

In 2006 3 people, Herbert Bay, Tinne Tuytelaars, Luc Van Gool created a replacement for \ac{SIFT} called \textbf{\ac{SURF}} \cite{surf_bay}. This algorithm brings speed improvement at each step of \ac{SIFT}. For the fist step, it goes much deeper and approximates \ac{LoG} with box filter, which is a convolution filter. This type of operation has some advantage: easily calculated with a help of integral images, can be performed in parallel for different scales. During orientation assignment it uses wavelet responses in both directions. It also estimates the dominant orientation as a sum of all responses in the orientation window with angle of 60\degree. Since many tasks do not require rotation invarience, \ac{SURF} has a concept called Upright-SURF or U-SURF where there is no orientation findings that speeds up the process by 15\%. On the feature description step it uses wavelet responses again in both directions. The neighborhood of the key point is divided into 4 by 4 sub-regions and for each of them it forms a vector from wavelet responses.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        v = (\sum{d_x},\sum{d_y},\sum{|d_x|},\sum{|d_y|})
    \end{equation}
    \label{fig:surf_wavelet_responses}
  \caption{Definition of the vector formed from wavelet responses.}
\end{figure}

The definition of the vector formed from wavelet responses is displayd in the figure \ref{fig:surf_wavelet_responses}. This results into the feature descriptor with 64 dimensions but \ac{SURF} also has an extended 128 dimension version where $\sum{d_x}$ and $\sum{|d_x|}$ are computed separately for $d_y<0$ and $d_y\geq0$, the same is done for $\sum{d_y}$ and $\sum{|d_y|}$ depending on $d_x$ sign.

\ac{SURF} also has a great improvement which is a use of a sign of Laplacian. It allows to distinguish bright blobs on dark background and the opposite situation. During matching, it uses minimal information, compares features by having the same type of contrast, which allows to do matching faster without reducing performance \cite{opencv_docs_surf}. Figure \ref{fig:Surf_Matching} depicts \ac{SURF} matching.

\begin{figure} [ht]
    \begin{center}
        \includegraphics[width=200pt]{surf_matching}
        \caption{Surf matching example}
        \label{fig:Surf_Matching}
    \end{center}
\end{figure}

All mentioned features detection algorithms are not fast enough to be used in real time tasks, one of examples is \ac{SLAM}. In 2006 Tom Drummond and Edward Rosten have written an article "Machine learning for high-speed corner detection" where they proposed new algorithm called \textbf{\ac{FAST}} \cite{machine_learning_high_speed_corner_rosten}.

The idea is pretty simple, it takes a circle of 16 pixels around the current pixel and if at least $n$ of them are contiguous and one of two requirements must be met: intensities of $n$ pixels must be larger then the sum of the current pixel (center of created circle) intensity and threshold value or these intensities must be less then difference of the current pixel and threshold value. Only in case these regiments are met it implies that the pixel is a corner. Figure \ref{fig:fast_state} illustrates \ac{FAST} intensity checkups. To improve the speed it also has a concept of a high-speed test to reject more non-corners then it already dose. It takes only 4 pixels, first it takes 1-st and 9-th and if they meet the condition next couple of 5-th and 13-th are checked. Only to those who passed high-speed test full check is applied. The method has its pros and cons so to fix some of cons machine learning non-maximal suppression is used.

Machine learning part of algorithm works as follows, it takes a set of images from the same domain and extract features as described before then forms vectors of 16 surrounding pixels for each feature point in all images from the set. Then it divides the vector into 3 sub vectors under \ref{fig:fast_states} conditions. After that decision tree classifier is applied to these sub-vectors using the knowledge of whether feature point is a corner or not, let this value be $K_p$. It picks some value $x$ which yields the information measured by the entropy of $K_p$. This is applied to each sub-vector recursively until the entropy becomes zero. This decision tree is used afterwards for fast feature detection. Applying machine learning helps to get better performance in cases when number of contiguous pixels is less then 12 and to improve not optimal pixel's picking mechanism. There is also one additional option that improves feature detection and it is rejecting unnecessary feature points in locations where more then one feature were found way too close to each other. This is done by applying non-maximal suppression where depending on value of score function $V$, which is the sum of absolute differences between current feature point and its 16 surrounding pixel values, features are discarded or not. \ac{FAST} is at least twice faster then other algorithms however it is strongly influenced by the presence of noise in the image. It can be adjusted by the threshold value.

\begin{figure} [!ht]
  \centering
    \begin{equation}
        S_{p->x} =
        \begin{cases}
            darker, & \mbox{if } I_{p->x} \leq I_p-thresh \\
            similar, & \mbox{if } I_p-thresh < I_{p->x} < I_p+thresh\\
            brighter, & \mbox{if } I_{p->x} \geq I_p+thresh
        \end{cases}
        \label{fig:fast_states}
    \end{equation}
    \label{fig:fast_state}
  \caption{FAST feature state conditions.}
\end{figure}

\textbf{\ac{BRIEF}} was published by Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua in 2010 \cite{brief_2010}. This algorithm brings a significant improvement to the way how features are matched. In fact, it is not a fully functional feature detection algorithm, it is only matching already found features thus have to be used in tandem with some feature detection algorithm. \ac{SIFT} and \ac{SURF} use vectors as feature descriptors but they take enormous amount of space, 256 bytes per feature which is not sufficient for embedded systems, for instance. These vectors can be compressed to binary string that can be used to match features with Hamiltonian distance which is a simple XOR on bit count operations. Unfortunately compression does not solve the problem of memory consumption because it still has to compute descriptors. Here comes \ac{BRIEF}, it can find this binary strings without actually finding feature descriptors. In order to do this, it uses unique algorithm to get a set of location pairs from the smoothed image patch. Iterating over this set it compares elements of pairs and compares their intensity, depending on the result of comparison it creates a bit-string of zeroes and ones. Afterwards, Hamiltonian distance can be used to match the results.

In 2011, Vincent Rabaud, Ethan Rublee, Kurt Konolige and Gary Bradski from "OpenCV Labs" created an alternative to \ac{SIFT} and \ac{SURF}. They described it in the paper called "ORB: An efficient alternative to \ac{SIFT} or \ac{SURF}" \cite{orb_rublee}. The main attractiveness of \textbf{\ac{ORB}} is that it is free from patents while is good as \ac{SIFT} and \ac{SURF}.
In fact, it uses \ac{FAST} for key points detection and by applying Harris corner it finds $N$ best points. It also uses \ac{BRIEF} as the feature descriptor. To fix the issues with rotations for both \ac{FAST} and \ac{BRIEF} it suggests some additional computations. As an improvement to \ac{FAST}, to get the rotation the algorithm computes the direction vector from the corner point to the intensity weighted centroid with this corner placed in the center. For \ac{BRIEF} it controls the process according to key points orientation. For a set of binary tests at some location it creates a matrix, with a size twice larger then the set, with coordinates of pixels. Afterwards using the orientation of key point it computes rotation matrix. In addition, it uses multi-probe local sensitivity hashing instead of original version of it for matching. The algorithm is good for inefficient devices with low computational power \cite{opencv_docs_orb}.

\textbf{Matchers} are algorithms for checking feature descriptors for similarity. Query descriptor can be compared with all other descriptors via Brute Force approach or use other algorithms with better heuristic like \ac{kNN} or \ac{FLANN}. The main problem is that each matcher can't work with all feature descriptors. For instance simple \ac{FLANN} can't work with bit strings as descriptors while \ac{FLANN} with locality sensitive hashing index can do this \cite{opencv_docs_matchers}.

The most popular combinations of feature detector, extractor and matcher are \ac{SURF} with \ac{SURF} and \ac{FLANN}, \ac{SIFT} with \ac{SIFT} and \ac{FLANN}, \ac{ORB} with \ac{ORB} and Brute force, \ac{ORB} with \ac{BRIEF} and Brute force. \ac{FAST} can be used as a feature detector in all these combinations.

Brute-Force matcher simply takes the descriptor of some feature from the first image and compares it with all other features descriptors from the second image using distance calculation so the closest feature is its match.

\ac{FLANN} is used for large data-sets because it provides much better performance then Brute force. It consists of a set of algorithms for nearest neighbour search so that those algorithms are optimized to be faster.

\textbf{Homography}, in computer vision, is a relation between two point's sets from the same planar surface which is true for the pinhole camera model. It can be computed with knowledge of camera parameters so that rotation and translation information can be extracted. This information is useful in many tasks; they are augmented reality, navigation and etc \cite{opencv_docs_feature_matching_homography}.

In computer vision, machine learning techniques are widely used. One of them is classification, analyzing new observed features for belonging to some defined categories. The process of creating those categories is also known as clustering.

In pattern recognition, the \textbf{\ac{kNN}} is a non-parametric method used for classification and regression. In both cases, the input consists of the $k$ closest training examples in the feature space. The output depends on whether \ac{kNN} is used for classification or regression. \ac{kNN} algorithm is one of the simplest algorithms in machine learning. It is used for classification and regression and belongs to a type of lazy learning algorithms, which means that the function is just approximated but all the computations are delayed until classification begins. The idea beneath is quite simple, to classify the object it takes into consideration the majority of votes from the object $k$ neighbours, while $k$ is user-defined. This means that the object will be labeled with the most repeated class through out the neighborhood.
During the training phase algorithm only stores the feature vectors and labels of classes retrieved from training data.
Two commonly used distance metrics in the algorithm are Euclidean and Hamming depending on what should be classified, for example, Hamming is used for text classification.

\textbf{\ac{SVM}} algorithm was developed by Vladimir Vapnik and Alexey Chervonenkis in 1963 but in 1992, Bernhard Boser, Isabelle Guyon and Vladimir Vapnik suggested a way to make \ac{SVM} nonlinear classifier. \ac{SVM} as it is known today, with soft margin, was published in 1995 by one of its initial authors, Vapnik, and Corinna Cortes \cite{support_vector_networks_vapnik}. \ac{SVM} algorithm is a set of controlled learning models used to analyze data for regression and classification analysis. With a help of training data this algorithm builds a model that can classify new samples to one of the two categories. Thus it is linear binary classifier however with a help "kernel trick" it performs implicit mapping to a high-dimensional feature spaces and can become non-linear classifier. In addition, it can work with unlabeled sample data with a help of support vector clustering algorithm. In general, \ac{SVM}s are very useful for working with text, especially handwritten, because they make pre-labeling stage unnecessary. They also show high search accuracy after multiple runs of relevance feedback.

\textbf{Connected components analysis} or labeling is a category of algorithms that are used to identify and analyze connected set of pixels. In general it is done by taking binary image and producing another image where each pixel is marked for belonging to some group, object on the image represented as a blob of connected pixels \cite{binary_image_analysis_washington}. Figure \ref{fig:binaryImageCca} illustrates connected components.

\begin{figure} [!ht]
  \centering
    \includegraphics[width=120pt]{binary_image_cca}
    \label{fig:binaryImageCca}
  \caption{A binary image with 5 connected components denoted with white areas.}
\end{figure}

There are many various algorithms for the connected component analysis. Some of them can keep whole image in memory, which is not suitable for low-performance computers, and process one component at a time while moving across the image. On the other hand, there are some algorithms that can process very large images while working only with some blob at a time, keeping some piece of image in the memory but not the whole image.

\begin{figure} [!ht]
  \centering
    \begin{tabular}{ |c|c|c| }
        \hline
        &1&\\
        \hline
        2& *& 3\\
        \hline
        & 4&\\
        \hline
    \end{tabular}
     \begin{tabular}{ |c|c|c| }
        \hline
        1& 2& 3\\
        \hline
        4& *& 5\\
        \hline
        6& 7& 8\\
        \hline
    \end{tabular}
    \label{fig:recursiveCcPatterns}
  \caption{Scan-line order.}
\end{figure}

Recursive labeling algorithm works as follows, it take binary image and negates it to make all pixels with value $1$ to be $-1$, this is done to distinguish unprocessed pixels from the labeled, whose label is $1$. After that it takes pixel with $-1$ value and assigns it a new label then it searches for a neighbour and recursively repeats the process. Pixel neighbours are returned according to the pattern specified on the figure \ref{fig:recursiveCcPatterns}, which is scan-line order.

Row by row labeling algorithm or Two-pass algorithm is based on connected components algorithm for graphs \cite{rosenfeldpfaltz}. It performs two passes, in the first one it records equivalences and marks pixels with temporary labels. Before the second pass it analyses binary relations from the stored equivalences to generate equivalence classes of the relation. Afterwards, in the second pass it reassigns pixels with its equivalence class instead of temporary value. Figure \ref{fig:labeled_binary_image} demonstrates labeled binary image where each cell of the right grid contains the number of its component.

\begin{figure} [!ht]
  \centering
     \begin{tabular}{ |c|c|c|c|c|c|c|c| }
        \hline
        1  & 1  & 0  & 1  & 1  & 1  & 0  & 1\\
        \hline
        1  & 1  & 0  & 1  & 0  & 1  & 0  & 1\\
        \hline
        1  & 1  & 1  & 1  & 0  & 0  & 0  & 1\\
        \hline
        0  & 0  & 0  & 0  & 0  & 0  & 0  & 1\\
        \hline
        1  & 1  & 1  & 1  & 0  & 1  & 0  & 1\\
        \hline
        0  & 0  & 0  & 1  & 0  & 1  & 0  & 1\\
        \hline
        1  & 1  & 0  & 1  & 0  & 0  & 0  & 1\\
        \hline
        1  & 1  & 0  & 1  & 0  & 1  & 1  & 1\\
        \hline
    \end{tabular}
    \begin{tabular}{ |c|c|c|c|c|c|c|c| }
        \hline
        1& 1& 0& 1& 1& 1& 0& 2\\
        \hline
        1& 1& 0& 1& 0& 1& 0& 2\\
        \hline
        1& 1& 1& 1& 0& 0& 0& 2\\
        \hline
        0& 0& 0& 0& 0& 0& 0& 2\\
        \hline
        3& 3& 3& 3& 0& 4& 0& 2\\
        \hline
        0& 0& 0& 3& 0& 4& 0& 2\\
        \hline
        5& 5& 0& 3& 0& 0& 0& 2\\
        \hline
        5& 5& 0& 3& 0& 2& 2& 2\\
        \hline
    \end{tabular}
    \label{fig:labeled_binary_image}
  \caption{Grid representations of binary image and it's labeled version.}
\end{figure}

\end{document}


\subsection{k-Nearest Neighbour Search}


\documentclass[12pt]{article}
\usepackage{fullpage,enumitem,amsmath,amssymb,graphicx,grffile,float,listings}

\begin{document}
    %Title Section
    \begin{flushleft}
    \LARGE CS229 Fall 2017\\
    \LARGE Problem Set \#1 Solutions: Supervised Learning\\
    \textbf{\normalsize Author: LFhase \quad rimemosa@163.com}
    \end{flushleft}
    \noindent
    \rule{\linewidth}{0.4pt}
    %Title Section

    %Problem and Solution

    \section*{Logistic regression  }

    \begin{enumerate}[label=(\alph*)]

    \item
    {
        Given that
        $$J(\theta)= \frac{1}{m} \sum_{i=1}^{m}log(1+e^{-y^{(i)}\theta^{T}x^{(i)}})$$\\
        we can get
        $$\frac{\partial J(\theta)}{\partial \theta_{i}}
            =  \frac{1}{m} \sum_{k=1}^{m} \frac{-y^{(k)}x^{(k)}_i}{1+e^{y^{(k)}\theta^{T}x^{(k)}}}
        $$\\
        then
        $$\frac{\partial J(\theta)}{\partial \theta_{i} \partial \theta_{j}}
            =  \frac{1}{m} \sum_{k=1}^{m}
            \frac{x^{(k)}_ix^{(k)}_je^{y^{(k)}\theta^{T}x^{(k)}}}
            {(1+e^{y^{(k)}\theta^{T}x^{(k)}})^2}
        $$\\
        which is $ H_{ij} $\\
        so
        \begin{equation*}
        \begin{split}
            Z^THZ &= \sum_{i=1}^n \sum_{j=1}^n z_iH_{ij}z_j \\
            &= \frac{1}{m} \sum_{k=1}^{m}
            \frac{\sum_{i=1}^n \sum_{j=1}^n z_ix^{(k)}_ix^{(k)}_jz_je^{y^{(k)}\theta^{T}x^{(k)}}}{(1+e^{y^{(k)}\theta^{T}x^{(k)}})^2}
            \\
        \end{split}
        \end{equation*}
        known that
        $$ \sum_{i=1}^n \sum_{j=1}^n z_ix^{(k)}_ix^{(k)}_jz_j = (X^TZ)^2 \geq 0 $$
        $$ \frac{e^{y^{(k)}\theta^{T}x^{(k)}}}{(1+e^{y^{(k)}\theta^{T}x^{(k)}})^2} > 0 $$
        we can easily get $$ Z^THZ \geq 0 $$
    }
    \newpage
    \item
    {
        Firstly, we plot the original data

        \begin{figure}[H]
            \centering
            \includegraphics[width=0.80\textwidth]{figure/LogisticRegression/original_data.png}
        \end{figure}

        To implement Newton’s Method, we calculate the value of
        $$\frac{1}{m} \sum_{k=1}^{m} \frac{-y^{(k)}x^{(k)}_i}{1+e^{y^{(k)}\theta^{T}x^{(k)}}}
        $$
        and $\textbf{Hessian}$\\
        then using the \textbf{update rule}
        $$ \theta := \theta - H^{-1}\bigtriangledown_\theta l(\theta) $$
    }

    \newpage
    \item
    {
        Through 6 iterations, we finally get the result
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.80\textwidth]{figure/LogisticRegression/Iteration6.png}
        \end{figure}
        and the $ \theta $ is  [-2.6205116,   0.76037154,  1.17194674].
    }

    \end{enumerate}

    \newpage
    \section*{ Poisson regression and the exponential family}

    \begin{enumerate}[label=(\alph*)]

    \item
    Firstly, we get
    \begin{equation*}
        \begin{split}
            p(y;\lambda) &= \frac{e^{-\lambda}\lambda^{y}}{y!} \\
            &= \frac{1}{y!}exp(log\lambda^y - loge^\lambda) \\
            &= \frac{1}{y!}exp(log\lambda y - \lambda) \\
    \end{split}
    \end{equation*}
    It's easy to know that
    $$ log\lambda = \eta, \lambda = e^\eta $$
    $$ T(y) = y $$
    $$ a(\eta) = -e^\eta $$
    $$ b(y) = \frac{1}{y!} $$
    and Poisson distribution is in the exponential family.

    \item
    Known that if $ Y \sim P(\lambda) $, then $ E[Y] = \lambda $ ,
    \begin{equation*}
        \begin{split}
            h_\theta(x) &= E[y|x;\theta] \\
                        &= \lambda \\
                        &= e^\eta = e^{\theta^Tx} \\
        \end{split}
    \end{equation*}
    so the response function for the family is
    $$ g(\eta) = e^\eta $$.

    \item
    Firstly, we can get likelihood function $L(\theta)$
    \begin{equation*}
        \begin{split}
            L(\theta) &= \Pi_{k=1}^m P(y^{(k)}|x^{(k)};\lambda) \\
                      &= \Pi_{k=1}^m \frac{e^{-\lambda}\lambda^{y^{(k)}}}{y^{(k)}!} \\
        \end{split}
    \end{equation*}
    Then, the log-likelihood function $l(\theta)$
    \begin{equation*}
        \begin{split}
            l(\theta)
            &= \sum_{k=1}^m (-logy^{(k)}!) + (-\lambda) + y^{(k)}log\lambda \\
            &= \sum_{k=1}^m (-logy^{(k)}!) + (-e^{\theta^Tx^{(k)}}) + y^{(k)}\theta^Tx^{(k)} \\
        \end{split}
    \end{equation*}

    Then, we can get the derivative of the log-likelihood with respect to $\theta_i$
    $$ \frac{\partial l(\theta)}{\partial \theta_i}
       = \sum_{k=1}^m (y^{(k)}-e^{\theta^Tx^{(k)}})x^{(k)}_i
    $$
    So, the stochastic gradient ascent learning rule is
    $$ \theta_i = \theta_i + \alpha(y^{(k)}-e^{\theta^Tx^{(k)}})x^{(k)}_i
    $$

    \item

    Firstly, we can know that
    $$ p(y|x;\theta) = b(y)exp(\eta^Ty-a(\eta)) $$
    then, for a simple training data $(x,y)$ in stochastic gradient ascent,
    $$ \frac{\partial p(y|x;\theta)}{\partial \theta_i}
        = x_i(y-\frac{\partial a(\eta)}{\partial \eta})
    $$
    known that,
    $$ \int_{y}p(y|x;\eta) = 1 $$
    we can imply derivation in both sides, and get
    $$ \int_{y}p(y|x;\eta)(y-\frac{\partial a(\eta)}{\partial \eta}) = 0 $$
    finally, we get
    $$ \frac{\partial a(\eta)}{\partial \eta} = \int_{y}p(y|x;\eta)y = h_\theta(x) $$
    so the gradient we get is
    $$ \frac{\partial p(y|x;\theta)}{\partial \theta_i}
        = x_i(y-h_\theta(x))
    $$
    the update rule is
    $$ \theta_i = \theta_i - \alpha(h_\theta(x)-y)x_i $$
    \end{enumerate}

    \newpage
    \section*{Gaussian discriminant analysis   }
    \begin{enumerate}[label=(\alph*)]

    \item
    Firstly, we have
    \begin{equation*}
        \begin{split}
            p(y=1|x;\phi, \Sigma, \mu_1, \mu_-1)
             &= \frac{p(x|y=1)p(y=1)}{p(x|y=1)p(y=1)+p(x|y=-1)p(y=-1)}  \\
             &= \frac{1}{1+\frac{p(x|y=-1)p(y=-1)}{p(x|y=1)p(y=1)}} \\
        \end{split}
    \end{equation*}
    apply such calculattion, we can get
    \begin{equation*}
        \begin{split}
            p(y=1|x;\phi, \Sigma, \mu_1, \mu_-1)
             &= \frac{1}{1+exp(-y(ln\frac{\phi}{1-\phi}+\frac{1}{2}((x-\mu_{-1})^T\Sigma^{-1}(x-\mu_{-1})-(x-\mu_{1})^T\Sigma^{-1}(x-\mu_{1}))} \\
             &= \frac{1}{1+exp(-y(ln\frac{\phi}{1-\phi}+(\mu_1-\mu_{-1})\Sigma^{-1}x+\frac{1}{2}(\mu_{-1}\Sigma^{-1}\mu_{-1}-\mu_{1}\Sigma^{-1}\mu_{1})))} \\
        \end{split}
    \end{equation*}
    It's the same when $y=-1$. They can be written in the form of a logistic function where
    $$ \theta = (\mu_1-\mu_{-1})\Sigma^{-1} $$
    $$ \theta_0 = ln\frac{\phi}{1-\phi} + 1/2(\mu_{-1}\Sigma^{-1}\mu_{-1}-\mu_{1}\Sigma^{-1}\mu_{1}) $$

    \item
    Let $m_1$ be the number of samples with label $y=1$, and $m_{-1}$ be the number of samples with label $y=-1$.
    The log-likelihood function is
    $$ l(\theta,\mu_{-1}, \mu_1, \Sigma)
       = mlog(\frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}})+m_1log\phi+m_{-1}log(1-\phi)
        +\sum_{i=1}^{m}(-\frac{1}{2})(x^{(i)}-\mu_{y^{(i)}})^T\Sigma^{-1}(x^{(i)}-\mu_{y^{(i)}}) $$
    so we can derive the partial derivate of each parameter,
    $$ \frac{\partial l}{\partial \phi} = \frac{m_1-m\phi}{\phi(1-\phi)}$$
    $$ \frac{\partial l}{\partial \mu_1} = \frac{\sum_{i=1}^{m_1}(x^{(i)}-\mu_1)}{\Sigma}$$
    $$ \frac{\partial l}{\partial \mu_{-1}} = \frac{\sum_{j=1}^{m_1}(x^{(j)}-\mu_{-1})}{\Sigma}$$
    $$ \frac{\partial l}{\partial \Sigma} = \frac{-\frac{1}{2}(m\Sigma-\sum_{i=1}^{m}(x^{(i)}-\mu_{y^(i)})^T(x^{(i)}-\mu_{y^(i)}))}{\Sigma^{2}}$$
    let these partial derivates to be zero, we can get the maximum likelihood estimates the same as problem statement.

    \item
    As shown in (b)
    \end{enumerate}

    \newpage
    \section*{ Linear invariance of optimization algorithms  }
    \begin{enumerate}[label=(\alph*)]

    \item

    According to Newton’s Method, we know that
    $$ x^{(i+1)} = x^{(i)} - \frac{f(x^{(i)})}{f'(x^{i})} $$
    So
    \begin{equation*}
        \begin{split}
        z^{(i+1)} &= z^{(i)} - \frac{g(z^{(i)})}{g'(z^{(i)})} \\
                  &= z^{(i)} - \frac{f(Az^{(i)})}{f'(Az^{(i)})} \\
                  &= A^{(-1)}x^{(i)} - A^{(-1)}(x^{(i)}-x^{(i+1)}) \\
                  &= A^{(-1)}x^{(i+1)}
        \end{split}
    \end{equation*}
    so that Newton’s Method is invariant to linear reparameterizations.

    \item
    In gradient descent algorithm
    \begin{equation*}
        \begin{split}
        z^{(i+1)} &= z^{(i)} - \alpha g'(z^{(i)}) \\
                  &= A^{(-1)} x^{(i)} - A( x^{(i)} - x^{(i+1)} ) \\
                  &\neq A^{(-1)}x^{(i+1)}
        \end{split}
    \end{equation*}

    \end{enumerate}

    \newpage
    \section*{ Regression for denoising quasar spectra}
    \begin{enumerate}[label=(\alph*)]

    \item
    \begin{enumerate}[label=(\roman*)]
        \item
        We know that the form can be written in
        $$ \sum_{i,j} W_{ij}(\Theta^Tx^{(i)}-y^{i})(\Theta^Tx^{(j)}-y^{j}) $$
        to let them fit, just let $W_{ii} = \frac{1}{2}w^{(i)}$
        \item
        Firstly, we get the derivative
        $$\bigtriangledown_{\theta}J(\theta) = 2(W(X\theta-\vec y)^TX)$$
        let the derivative be zero, we get
        $$ \theta =  (X^TWX)^{-1}X^TW\vec{y} $$
        since $W$ is normal, $\theta$ is what we want.
        \item
        Write the log-likelihood function
        $$ l = \sum_{i=1}^m log\frac{1}{\sqrt{2\pi}\sigma^{(i)}}
             - \sum_{i=1}^m \frac{(y^{(i)}-\theta^Tx^{(i)})^2}{2(\sigma^{(i)})^2} $$
        The left part is a constant, and the right part need to be minimized.\\
        Let $w_{i}=\frac{1}{2(\sigma^{(i)})^2}$, we get a form same as the weight linear regression.\\
    \end{enumerate}

    \item
    \begin{enumerate}[label=(\roman*)]
        \item
        Let $\theta = (X^TX)^{-1}X^T \vec{y}$, we can get the result
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.80\textwidth]{figure/Regression4Quasar/b_i.png}
        \end{figure}
        \item
        Let $\theta = (X^TWX)^{-1}X^TW \vec{y}$, we can get the result
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.60\textwidth]{figure/Regression4Quasar/b_ii.png}
        \end{figure}
        \item
        let $\tau$ be different value, we can get different fitting result.
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.80\textwidth]{figure/Regression4Quasar/b_iii.png}
        \end{figure}
        From the figure above, we know that, when $\tau = 1$, we get a perfect fitting result, but it's likely to be overfitting.\\
        As $\tau$ grows, the fitting gets worse and worse. So if we want to get a good result, we'd better trade off and select $\tau = 10$ (just an example) to get a balanced performance.

    \end{enumerate}
    \item
    \begin{enumerate}[label=(\roman*)]
        \item Just apply locally weighted linear regression to all 200 samples
        \item Through the algorithm described in problem statement, we get the estimation of left part in training set.
        \\ The average error is $7.511903$.
        \\ Here are some examples.$[1, 50, 100, 150]$
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.80\textwidth]{figure/Regression4Quasar/c_ii.png}
        \end{figure}
        \item  Through the algorithm described in problem statement, we get the estimation of left part in test set.
        \\ Here are some examples.$[1, 6, 10, 15]$
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.80\textwidth]{figure/Regression4Quasar/c_iii.png}
        \end{figure}
    \end{enumerate}

    \end{enumerate}
\end{document}


% declare document class and geometry
%\documentclass[12pt]{article} % use larger type; default would be 10pt
%\usepackage[margin=1in]{geometry} % handle page geometry

\input{../plaheader.tex}

\title{Astro 270 -- Astrophysical Dynamics -- Lec04}
\author{UCLA, Fall 2014}
%\date{\formatdate{02}{10}{2014}} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed
\date{}

\begin{document}
\setlength{\unitlength}{1mm}
\maketitle

\section{Mestel Disk}
Defined in terms of the surface mass density
\begin{equation}
\Sigma(R) = \Sigma_0 \left(\frac{R_0}{R}\right)
\end{equation}
\begin{equation}
M(r) = \int_0^R 2\pi R' dR' \Sigma_0 \left(\frac{R_0}{R}\right) = 2\pi \Sigma_0 R_0R
\end{equation}
Use Poisson's equation
\begin{equation}
\nabla^2 \Phi = 0
\end{equation}
everywhere but at z = 0
\begin{equation}
\frac{1}{R} \pd{}{R} \left(R \pd{\Phi}{R}\right) + \pdd{\Phi}{z} = 0
\end{equation}
The potential can be written as um two um product um of two functions um we're assuming this is of course incorrectly um that it is separable in two fucntions so when put this um formulation of phi into this um equation we end up with uh the equation that 1/r um I have to write my Rs carefully because I''m using two this is uh script R and this is um capital R and so um we have the following

Can't write my capital Rs if I''m writing it as a script R. Um so since this is equal to zero we'll um bring this to teh other size of the equation and um end up with the function

\begin{equation}
\Phi (R,z) = R(r) Z(z)
\end{equation}
\begin{equation}
\frac{1}{Rr} \d{}{R} (R \d{r}{R}) = -\frac{1}{Z} \dd{Z}{z}
\end{equation}
Each side must equal a constant $-k^2$
\begin{equation}
\dd{Z}{z} - k^2Z = 0
\end{equation}
\begin{equation}
Z = \alpha e^{\pm z}
\end{equation}
We choose the minus sign so it converges
\begin{equation}
\frac{1}{R} \d{}{R} (R \d{r}{R}) + k^2R=0
\end{equation}

By doing a substitution $u = kR$ we find that our function r goes as the zeroeth order bessel functions
\begin{equation}
\Phi(R,z) = e^{-kz} J_0(kR)
\end{equation}
We have a discontinuity in the derivative of the phi at z=0. Let's look at Gauss' law again

\begin{equation}
\sigma_k(R) = -\frac{k J_0(kR)}{2\pi G}
\end{equation}
We still need coefficients such that the surface mass density will yeild ourioriginal density

\begin{equation}
\Sigma(R) = \int_0^{\infty} S(k) \sigma_k(R) dk = -\frac{1}{2\pi G}\int_0^{\infty} kdk S(k)J_0(kR)
\end{equation}
WELL WHAT WHOOPIDY FUCKIN DO ITS A FT. Actually more than a Henkel function. We can invert it to find the coefficients
\begin{equation}
S(k) = -2\pi G \int_0^{\infty} J_0(kR) \Sigma(R) RdR
\end{equation}
Plug in our Mestel disk
\begin{equation}
S(k) = -2\pi G \Sigma_0 R_0 \int_0^{\infty} J_0(kR)  dR = -2\pi G \Sigma_0 R_0 /k
\end{equation}
\begin{equation}
\Phi = \int_0^{\infty} S(k) \Phi_k (R, z)dk
\end{equation}
\subsection{Circular Orbit in Z=0 Plane}
\begin{equation}
v_c^2(R) = R\d{\Phi}{R}
\end{equation}
\begin{equation}
\d{J_0(x)}{x} = -J_1(R)
\end{equation}
\begin{equation}
v^2_c(R) = -R\int S(k) J_1(kR)kdk
\end{equation}
\begin{equation}
v^2_c(R) = -2\pi G \Sigma_0 R_0R\int S(k) J_1(kR)kdk = -\frac{2\pi G \Sigma_0 R_0R}{R}
\end{equation}
\begin{equation}
v_c^2 = 2\pi G \Sigma_0R_0
\end{equation}

\section{Orbits in Axisymmetric}
We have our integrals of motion $E, L_z$. We can define an effective potential again
\begin{equation}
\Phi_{eff} = \Phi + \frac{\L_z^2}{2R^2}
\end{equation}
\begin{equation}
E = \frac{1}{2} (\dot{R}^2 + R\dot{\phi}^2 + \dot{z}^2) + \Phi = \frac{1}{2} (\dot{R}^2 + \dot{z}^2 + \Phi_{eff}
\end{equation}
This means taht the orbit is restricted to the portions of the R-Z plane where $E\ge \Phi_{eff}$

$R_z$ plane is the meridianal plane. We can limit our phase space to $R, z, \dot{R}, \dot{z}$ plane
\begin{equation}
\ddot{R} = -\pd{\Phi_{eff}}{R}
\end{equation}
\begin{equation}
\ddot{z} = -\pd{\Phi_{eff}}{z}
\end{equation}
When
\begin{equation}
E = \Phi_{eff}
\end{equation}
Then the velocities are zero.

THis leads to epicicylic orbits. See homework problem 1e on classical. Do the Taylor expansion thing.

\end{document}


\documentclass{article}
\usepackage{amsmath,mathtools,amssymb}
\usepackage{graphicx}
\usepackage{gensymb}
\usepackage{verbatim}
\usepackage{cancel}
\usepackage{mathrsfs}
\usepackage{bbm}
\usepackage{braket}
\usepackage{xcolor}
\definecolor{orange}{RGB}{255,127,0}
\usepackage{verbatim}
\usepackage[margin=1.0in]{geometry}
\newcommand{\ol}{\overline}
\newcommand{\ve}{\varepsilon}
\newcommand{\ap}{\ensuremath{a_p} }
\newcommand{\dg}{\ensuremath{^\dagger} }
\newcommand{\cd}{\ensuremath{\cdots} }
\newcommand{\apd}{\ensuremath{a_p^\dagger} }
\def\*#1{\mathbf{#1}}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}

\title{Lecture 3.2: The Hamiltonian in Second Quantization}
\date{February 10$^{\text{th}}$, 2020}
\begin{document}
\maketitle
\noindent
\section{Introduction }
So far, we have see that any determinant can be written as a string of creation operators acting on a vacuum state.
Our next task is to formulate the Hamiltonian in terms of creation and annihilation operators so we have a unified language.
We will first generalize the effect of an annihilation operator acting on a determinant, which will then be used to derive the one-body and two-body operators
in the second quantization formalism.

\section{Re-expressing the effect of operators on the determinant }
For an arbitrary set of indices $\{p_1 \cdots p_N\}$, the actions of \ap and \apd are given by:
 \begin{align}
   a_p\Phi_{(p_1\cd p_N)}
 =
   (-)^{k-1}\Phi_{(p_1\cd\cancel{p_k}\cd p_N)}
   \ \text{if $p=p_k\in(p_1\cd p_N)$}
 &&
   a_p\Phi_{(p_1\cd p_N)}
 =
   0
   \ \text{if $p\notin(p_1\cd p_N)$}
 \\
   a_p\dg\Phi_{(p_1\cd p_N)}
 =
   (-)^{k-1}\Phi_{(p_1\cd p_{k-1}pp_k\cd p_N)}
   \ \text{if $p\notin(p_1\cd p_N)$}
 &&
   a_p\dg\Phi_{(p_1\cd p_N)}
 =
   0
   \ \text{if $p\in(p_1\cd p_N)$}
 \end{align}
We can prove (Appendix I) that Equation 1, instead of two parts, can be generally expressed as:
\begin{equation}
a_q \Phi_{(p_1\cd p_N)} =  \sum_i^N (-1)^{i-1} \delta_{qp_i} \Phi_{(p_1\cd\cancel{p_i}\cd p_N)}
\end{equation}
or writing the determinant as a product of creation operators,
\begin{equation}
a_q a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}=  \sum_i^N (-1)^{i-1} \delta_{qp_i} a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}
\end{equation}
We can adopt a shorthand notation where we express
\[a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} = \prod a\dg \ket{0}\]
and
\[a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} =  \prod{}^{p_i} a\dg \ket{0} \]
where $\prod{}^{p_i} a\dg $ stands for the string of creation operators without $a_{p_i}\dg$.
In this notation, we rewrite Equation 3 a final time:
\begin{equation}
a_q \prod a\dg \ket{0} =  \sum_i^N (-1)^{i-1} \delta_{qp_i} \prod{}^{p_i} a\dg \ket{0}
\end{equation}
You can show that two annhilation operators acting on a determinant can be similarly rewritten to be
\begin{equation}
a_r a_s \prod a\dg \ket{0} = \sum\limits_{i < j} ^N  (-1)^{i + j }   \delta_{rp_i} \delta_{sp_j}  \prod{}^{p_ip_j} a\dg \ket{0} +
\sum\limits_{i > j} ^N  (-1)^{i + j -1} \delta_{rp_i} \delta_{sp_j} \prod{}^{p_ip_j}a\dg \ket{0}
\end{equation}
by applying Equation 5 twice.

\section{Deriving the one-electron operator in second quantization}

We can come to a definition of the one-electron operator $\hat{O}_1 = \sum\limits_i^N \hat{h} (\*{r}_i) $  in the second quantization formalism by acting it a determinant $ \Phi_{p_1 \cdots p_N} = \prod a\dg \ket{0} $:  \\ \\
\[\hat{O}_1 \prod a\dg \ket{0} = \hat{O}_1\Phi_{p_1 \cdots p_N} = \hat{O}_1 \sqrt{N!} \hat{A} (\psi_{p_1} \cdots \psi_{p_N}) \]
Next, we recognize that  $\hat{A}$ and $\hat{O}_1$ commute, so we can switch their order (for general comments on commuting operators, see Appendix II):
\[ = \sqrt{N!}  \hat{A} \hat{O}_1  (\psi_1 \cdots \psi_N) \]
We can now re-express $\hat{O}_1$:
\[=  \sqrt{N!}  \hat{A}  \sum_i^N \hat{h} (\*{r}_i) (\psi_1 \cdots \psi_N) \]
Since $\hat{h} (\*{r}_i) $ only applies to one spin-orbital $\psi_{p_i}(\*{r}_i)$, we can move it out of the product of spin-orbitals:
\[=  \sqrt{N!}  \hat{A} {\color{blue}  \sum_i^N \hat{h} (\*{r}_i) \psi_{p_i} (\*{r}_i)} (\psi_{p_1} \cdots \cancel{\psi_{p_i}} \cdots \psi_{p_N}) \]
Using the resolution of the identity, we can rewrite the term in {\color{blue}blue} as:
\[=  \sqrt{N!}   \hat{A}  {\color{blue} \sum_i^N  \sum_p \braket{p | \hat{h} | p_i } \psi_p } (\psi_{p_1} \cdots \cancel{\psi_{p_i}} \cdots \psi_{p_N}) \]
 We can move $\psi_p$ into the product of spin-orbitals, and move the antisymmetrizer again to recover a Slater determinant:
\begin{align*}
&= \sqrt{N!}  \hat{A}   \sum_i^N  \sum_p \braket{p | \hat{h} | p_i }  (\psi_{p_1} \cdots \psi_{p_{i-1}} \psi_p \psi_{p_{i+1}} \cdots \psi_{p_N}) \\
&=\sum_i^N  \sum_p \braket{p | \hat{h} | p_i } \sqrt{N!}  \hat{A} (\psi_{p_1} \cdots \psi_{p_{i-1}} \psi_p \psi_{p_{i+1}} \cdots \psi_{p_N}) \\
&=\sum_i^N  \sum_p \braket{p | \hat{h} | p_i } \Phi_{{p_1} \cdots {p_{i-1}} p {p_{i+1}} \cdots {p_N}} \\
\end{align*}
Notice that the Slater determinant is not the original one we are acting on. Instead of $\psi_{p_i}$, it contains $\psi_p$.
Our next step is to express this Slater determinant as a product of creation operators acting on the vacuum, and move $a_p\dg$ in front using anticommutation rules:
\begin{align*}
&= \sum_i^N  \sum_p \braket{p | \hat{h} | p_i } a_{p_1}^\dagger \cdots a_{p_{i-1}}^\dagger {\color{blue}a_p^\dagger} a_{p_{i+1}}^\dagger \cdots a_{p_N}^\dagger \ket{0}\\
&= \sum_i^N  \sum_p \braket{p | \hat{h} | p_i } {\color{blue}a_p^\dagger} (-1)^{i-1} a_{p_1}^\dagger \cdots a_{p_{i-1}}^\dagger a_{p_{i+1}}^\dagger \cdots a_{p_N}^\dagger \ket{0}\\
\end{align*}
From here, we apply the resolution of the identity once again, and notice that $\braket{q | p_i }$ is just $\delta_{q p_i}$:
\begin{align*}
&= \sum_i^N  \sum_{p{\color{blue}q}} \bra{p}\hat{h}{\color{blue} \ket{ q} \bra{q }}p_i \rangle a_p^\dagger (-1)^{i-1} a_{p_1}^\dagger \cdots a_{p_{i-1}}^\dagger a_{p_{i+1}}^\dagger \cdots a_{p_N}^\dagger \ket{0}\\
&= \sum_i^N  \sum_{pq} \braket{p | \hat{h} | q} {\color{red}\delta_{q p_i}} a_p^\dagger (-1)^{i-1} a_{p_1}^\dagger \cdots a_{p_{i-1}}^\dagger a_{p_{i+1}}^\dagger \cdots a_{p_N}^\dagger \ket{0}\\
\end{align*}
Now we separate the terms by the appropriate summation, and notice that the terms following the summation of indices i is exactly the definition we obtained in Equation 4.
\begin{align*}
&=  \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger {\color{blue} \sum_i^N \delta_{q p_i} (-1)^{i-1} a_{p_1}^\dagger \cdots a_{p_{i-1}}^\dagger a_{p_{i+1}}^\dagger \cdots a_{p_N}^\dagger \ket{0}}\\
&=  \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger {\color{blue} a_q a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}}\\
&=  \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger a_q  {\color{red} a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}}\\
&=  \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger a_q  {\color{red} \prod a\dg \ket{0}}\\
\end{align*}
where the term in {\color{red} red} is just our original Slater determinant.  \\ \\
Our final equation is:
\begin{equation}
\hat{O}_1 {\color{red} \prod a\dg \ket{0}}  = \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger a_q  {\color{red} \prod a\dg \ket{0}}
\end{equation}
Since Equation 6 is true for any arbitrary determinant, conclude that
\begin{equation}
\hat{O}_1 = \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger a_q
\end{equation}
is an appropriate definition for the one-electron operator in the second quantization formalism.

\section{Deriving the two-electron operator in second quantization}
\subsection{The two-body operator}
We can come to a definition of the two-electron operator $\hat{O}_2 = \sum\limits_{i < j} ^N \hat{g} (\*{r}_i, \*{r}_j) $
in the second quantization formalism in a similar process as the one-electron operator.
\\ \\
We start by acting it a determinant $ \Phi_{p_1 \cdots p_N} = \prod a\dg \ket{0} $:  \\ \\
\[\hat{O}_2 \Phi_{p_1 \cdots p_N} = \sum\limits_{i < j} ^N \hat{g} (\*{r}_i, \*{r}_j)  \sqrt{N!} \hat{A} (\psi_{p_1} \cdots \psi_{p_N}) =  \frac{1}{2} \sum\limits_{i j} ^N \hat{g} (\*{r}_i, \*{r}_j)  \sqrt{N!} \hat{A} (\psi_{p_1} \cdots \psi_{p_N}) \]
where we can rewrite the summation $i < j$ as $ij$ with a factor of $\frac{1}{2}$ in front. \\ \\
Just as in the case of the one-electron operator, $\hat{A}$ and $\hat{O}_2$ commute, so we can switch their order:
\[= \frac{1}{2} \sqrt{N!}  \hat{A} \sum\limits_{i j} ^N \hat{g} (\*{r}_i, \*{r}_j)  (\psi_1 \cdots \psi_N) \]
Since $\hat{g} (\*{r}_i, \*{r}_j)  $ only applies to spin-orbitals $\psi_{p_i}(\*{r}_i)$ and $\psi_{p_j}(\*{r}_j)$, we can move them out of the product of spin-orbitals:
\[= \frac{1}{2} \sqrt{N!}  \hat{A} {\color{blue}  \sum\limits_{i j} ^N \hat{g} (\*{r}_i, \*{r}_j) \psi_{p_i} (\*{r}_i) \psi_{p_j} (\*{r}_i)} (\psi_{p_1} \cdots \cancel{\psi_{p_i}} \cd  \cancel{\psi_{p_j}} \cdots \psi_{p_N}) \]
Using the resolution of the identity, we can rewrite the term in {\color{blue}blue} as:
\[= \frac{1}{2}  \sqrt{N!}   \hat{A}  {\color{blue}  \sum\limits_{i j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } \psi_p \psi_q }  (\psi_{p_1} \cdots \cancel{\psi_{p_i}} \cd  \cancel{\psi_{p_j}} \cdots \psi_{p_N}) \]
\\
 We can move $\psi_p$  and $\psi_q$ into the product of spin-orbitals and move the antisymmetrizer again to recover a Slater determinant, but there are now two cases to consider.
 There are two Slater determinants we can form, one for cases $i < j$ and one for cases $i > j$.
 Thus, we split our equation into two terms:
\begin{align*}
= &\frac{1}{2} \sum\limits_{i < j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i }  \sqrt{N!}   \hat{A}  (\psi_{p_1} \cd \psi_{p_{i - 1}} {\color{blue} \psi_{p}} \psi_{p_{i +1}} \cd  \psi_{p_{j - 1}} {\color{red} \psi_{q}}\psi_{p_{j +1}} \cd \psi_{p_N})  \\
+ &\frac{1}{2} \sum\limits_{i > j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } \sqrt{N!}   \hat{A}  (\psi_{p_1} \cd \psi_{p_{j - 1}}  {\color{red} \psi_{q}}\psi_{p_{j +1}} \cd  \psi_{p_{i - 1}} {\color{blue} \psi_{p}}\psi_{p_{i +1}} \cd \psi_{p_N})  \\
= &\frac{1}{2} \sum\limits_{i < j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } \Phi_{{p_1} \cd p_{i - 1} {\color{blue} {p}} {p_{i +1}} \cd  {p_{j - 1}} {\color{red} {q}}{p_{j +1}} \cd {p_N}} \\
+ &\frac{1}{2} \sum\limits_{i > j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } \Phi_{{p_1} \cd {p_{j - 1}}  {\color{red} {q}}{p_{j +1}} \cd  {p_{i - 1}} {\color{blue} {p}}{p_{i +1}} \cd {p_N}}  \\
\end{align*}
Our next step is to express the Slater determinants in our expression as a product of creation operators acting on the vacuum, and move $a_p\dg$ in front using anticommutation rules:
\begin{align*}
= &\frac{1}{2} \sum\limits_{i < j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i }  (a\dg_{p_1} \cd a\dg_{p_{i - 1}} {\color{blue} a\dg_{p}} a\dg_{p_{i +1}} \cd  a\dg_{p_{j - 1}} {\color{red} a\dg_{q}}a\dg_{p_{j +1}} \cd a\dg_{p_N}) \ket{0}  \\
+ &\frac{1}{2} \sum\limits_{i > j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i }  (a\dg_{p_1} \cd a\dg_{p_{j - 1}}  {\color{red} a\dg_{q}}a\dg_{p_{j +1}} \cd  a\dg_{p_{i - 1}} {\color{blue} a\dg_{p}}a\dg_{p_{i +1}} \cd a\dg_{p_N})  \ket{0}  \\ \\
= &\frac{1}{2} \sum\limits_{i < j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } (-1)^{i - 1 + j - 1 - 1}  {\color{blue}  a\dg_{p}}  {\color{red} a\dg_{q}} (a\dg_{p_1} \cd a\dg_{p_{i - 1}}  a\dg_{p_{i +1}} \cd  a\dg_{p_{j - 1}} a\dg_{p_{j +1}} \cd a\dg_{p_N}) \ket{0}  \\
+ &\frac{1}{2} \sum\limits_{i > j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } (-1)^{j - 1 + i -1 -1}  {\color{red} a\dg_{q}}   {\color{blue} a\dg_{p}} (a\dg_{p_1} \cd a\dg_{p_{j - 1}} a\dg_{p_{j +1}} \cd  a\dg_{p_{i - 1}}a\dg_{p_{i +1}} \cd a\dg_{p_N})  \ket{0}  \\
\end{align*}
We can switch the order of ${\color{red} a\dg_{q}}$ and  ${\color{blue} a\dg_{p}}$ with an additional phase factor, and simplify the phase factor:
\begin{align*}
= &\frac{1}{2} \sum\limits_{i < j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } (-1)^{i + j - 1}  {\color{blue}  a\dg_{p}}  {\color{red} a\dg_{q}} (a\dg_{p_1} \cd a\dg_{p_{i - 1}}  a\dg_{p_{i +1}} \cd  a\dg_{p_{j - 1}} a\dg_{p_{j +1}} \cd a\dg_{p_N}) \ket{0}  \\
+ &\frac{1}{2} \sum\limits_{i > j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } (-1)^{i + j}   {\color{blue} a\dg_{p}}  {\color{red} a\dg_{q}}  (a\dg_{p_1} \cd a\dg_{p_{j - 1}} a\dg_{p_{j +1}} \cd  a\dg_{p_{i - 1}}a\dg_{p_{i +1}} \cd a\dg_{p_N})  \ket{0}  \\
= &\frac{1}{2} \sum\limits_{i < j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } (-1)^{i + j - 1}  {\color{blue}  a\dg_{p}}  {\color{red} a\dg_{q}} \prod{}^{p_ip_j}a\dg \ket{0}
 + \frac{1}{2} \sum\limits_{i > j} ^N   \sum_{pq}  \braket{pq | \hat{g} | p_i q_i } (-1)^{i + j}   {\color{blue} a\dg_{p}}  {\color{red} a\dg_{q}} \prod{}^{p_ip_j}a\dg \ket{0}  \\
\end{align*}

From here, we apply the resolution of the identity to introduce two more summations over $r$ and $s$:
\begin{align*}
= &\frac{1}{2} \sum\limits_{i < j} ^N  {\color{orange} \sum_{pq}  \sum_{rs}  \braket{pq | \hat{g} | rs } \braket{r| p_i} \braket{s | p_j} }(-1)^{i + j - 1}  a\dg_{p} a\dg_{q} \prod{}^{p_ip_j}a\dg \ket{0} \\
 + &\frac{1}{2} \sum\limits_{i > j} ^N  {\color{orange}  \sum_{pq}  \sum_{rs} \braket{pq | \hat{g} | rs} \braket{r| p_i} \braket{s | p_j} } (-1)^{i + j}   a\dg_{p}a\dg_{q} \prod{}^{p_ip_j}a\dg \ket{0}  \\
\end{align*}
and recognize that $ \braket{r| p_i}  = \delta_{rp_i}$ and $ \braket{s| p_j}  = \delta_{sp_j}$:
\begin{align*}
= &\frac{1}{2} \sum\limits_{i < j} ^N  {\color{orange} \sum_{pqrs} \braket{pq | \hat{g} | rs }  \delta_{rp_i} \delta_{sp_j}}(-1)^{i + j - 1}  a\dg_{p} a\dg_{q} \prod{}^{p_ip_j}a\dg \ket{0} \\
 + &\frac{1}{2} \sum\limits_{i > j} ^N  {\color{orange}  \sum_{pqrs}  \braket{pq | \hat{g} | rs} \delta_{rp_i} \delta_{sp_j}} (-1)^{i + j}   a\dg_{p}a\dg_{q} \prod{}^{p_ip_j}a\dg \ket{0}  \\
\end{align*}
We can now rearrange the equation using like terms:
\begin{align*}
= &\frac{1}{2}  \sum_{pqrs}  \braket{pq | \hat{g} | rs}  a\dg_{p}a\dg_{q} {\color{blue} \left[  \sum\limits_{i < j} ^N  (-1)^{i + j - 1}   \delta_{rp_i} \delta_{sp_j}  \prod{}^{p_ip_j} a\dg \ket{0} +
\sum\limits_{i > j} ^N  (-1)^{i + j} \delta_{rp_i} \delta_{sp_j} \prod{}^{p_ip_j}a\dg \ket{0}  \right]} \\
\end{align*}
The term in {\color{blue} blue} almost equals Equation 6
\[a_r a_s \prod a\dg \ket{0} = \sum\limits_{i < j} ^N  (-1)^{i + j }   \delta_{rp_i} \delta_{sp_j}  \prod{}^{p_ip_j} a\dg \ket{0} +
\sum\limits_{i > j} ^N  (-1)^{i + j -1} \delta_{rp_i} \delta_{sp_j} \prod{}^{p_ip_j}a\dg \ket{0}
\]
except for the switched summations.
We see that we can flip the terms after the summations if we flip the order of our operators $a_r$ and $a_s$:
\[a_s a_r \prod a\dg \ket{0}  = - a_r a_s \prod a\dg \ket{0} = \sum\limits_{i < j} ^N  (-1)^{i + j -1 }   \delta_{rp_i} \delta_{sp_j}  \prod{}^{p_ip_j} a\dg \ket{0} +
\sum\limits_{i > j} ^N  (-1)^{i + j } \delta_{rp_i} \delta_{sp_j} \prod{}^{p_ip_j}a\dg \ket{0}
\]
We therefore substitute $a_s a_r \prod a\dg \ket{0} $ in to get our final expression:
\begin{align}
\hat{O}_2  {\color{red} \prod a\dg \ket{0} }= &\frac{1}{2}  \sum_{pqrs}  \braket{pq | \hat{g} | rs}  a\dg_{p}a\dg_{q} a_s a_r {\color{red} \prod a\dg \ket{0} } \label{eqx}
\end{align}
where the term in {\color{red} red} is just our original Slater determinant. \textit{Notice that the order of the indices in the operator string is different than the order of the indices in the matrix element. } \\ \\
Since Equation~\ref{eqx}  is true for any arbitrary determinant, we conclude that
\begin{equation}
\hat{O}_2 =\frac{1}{2}  \sum_{pqrs}  \braket{pq | \hat{g} | rs}  a\dg_{p}a\dg_{q} a_s a_r
\end{equation}
is an appropriate definition for the two-electron operator in the second quantization formalism.
\\
\subsection{The antisymmetrized form of the two body operator}
In general, the N-body operator in the second quantization formalism can be expressed as :
\[ \hat{O}_k = \frac{1}{k!}   \sum_{\substack{p_1 \cd p_k \\q_1 \cd q_k }}  \braket{ p_1 \cd p_k | \hat{o}_k | q_1 \cd q_k} a_{p_1}\dg \cd \ a_{p_k}\dg a_{q_k} \cd a_{q_1} \]
This way of expressing the N-body operator, however, only gives us non-symmetric matrix elements.
What I mean by this is that the matrix element is only the same under simultaneous permutations of the $p$s and the $q$s.
If you only permute the $p$s \textbf{or} the $q$s, you would not get the same matrix element.
We want to rewrite the expression in a way to incorporate more symmetry.
We want to get the same matrix element if we permute the $p$s and $q$s at the same time, but also if we permute them separately.
To do this, we have to write instead:
\[ \hat{O}_k = \left( \frac{1}{k!} \right)^2  \sum_{\substack{p_1 \cd p_k \\q_1 \cd q_k }}  \braket{ p_1 \cd p_k | \hat{o}_k | q_1 \cd q_k}_\mathscr{A} a_{p_1}\dg \cd \ a_{p_k}\dg a_{q_k} \cd a_{q_1} \]
where $\braket{ p_1 \cd p_k | \hat{o}_k | q_1 \cd q_k}_\mathscr{A}$ stands for a linear combination of nonsymmetric matrix elements corresponding to all permutations of either p or q, but \textit{not both}.
We can write our two-body operator in this anti-symmetrized form. We will choose to permute the $q$s:
\[\hat{O}_2 =\left( \frac{1}{2!} \right)^2  \sum_{pqrs}  \braket{ pq | \hat{g} | rs}_\mathscr{A} a\dg_{p}a\dg_{q} a_s a_r = \frac{1}{4}\sum_{pqrs} ( \braket{ pq | \hat{g} | rs} -  \braket{ pq | \hat{g} | sr} )  a\dg_{p}a\dg_{q} a_s a_r  \]
\section{The final form of the Hamiltonian}
We have now derived the one-body and the two-body operator in the second quantization formalism:
\begin{equation}
\hat{H} = \sum_{pq} \braket{p | \hat{h} | q}  a_p^\dagger a_q  +  \frac{1}{2}  \sum_{pqrs}  \braket{pq | \hat{g} | rs}  a\dg_{p}a\dg_{q} a_s a_r
\end{equation}
From here on, we will often use the following notations to write the matrix elements to make lives a little easier for ourselves:
\[ h_{pq} = \braket{p | \hat{h} | q}\]
\[ \braket{pq | rs}  = \braket{pq | \hat{g} | rs} \]
\[\braket{pq || rs}  = \braket{pq | \hat{g} | rs} - \braket{pq | \hat{g} | sr} \]
 Using these shorthand notations, we can write our Hamiltonian as:
\[\hat{H} = \sum_{pq} h_{pq} a_p^\dagger a_q  +  \frac{1}{2}  \sum_{pqrs} \braket{pq | rs} a\dg_{p}a\dg_{q} a_s a_r  \]
or in the antisymmetrized form as:
\[\hat{H} = \sum_{pq} h_{pq} a_p^\dagger a_q  +  \frac{1}{4}  \sum_{pqrs} \braket{pq || rs} a\dg_{p}a\dg_{q} a_s a_r  \]
Note the effect of the Hamiltonian operator on the determinant.
In the case of the one body operator, it is removing a spin-orbital $q$ and replacing it with a spin-orbital $p$, multiplied by a matrix element.
We can think of this as a single excitation.
In the case of the two-body operator, it is removing two spin-orbitals $s$ and $r$ and replacing it with spin-orbitals $p$ and $q$, multiplied by a matrix element.
We can think of this as a double excitation. \\
\\
Notice something else: we have now gotten rid of the dependence on particle number in our Hamiltonian!
It is a completely general operator that applies to \textit{any} system size.
It is only dependent on the basis that we choose.
\\ \\
Previously, we have said our energy can be obtained by:
\[E = \braket{\Psi  | \hat{H}  | \Psi} \]
and that our wavefunction $\Psi$ can be expanded in a basis of determinants:
\[E = \sum_{\substack{p_1 < \cd p_N \\q_1 < \cd q_N }} \braket{\Phi_{q_1 \cd q_N}  | c_q^* \hat{H}  c_p | \Phi_{p_1 \cd p_N} } = \sum_{\substack{p_1 < \cd p_N \\q_1 < \cd q_N }} c_q^* c_p \braket{\Phi_{q_1 \cd q_N}  |  \hat{H}  | \Phi_{p_1 \cd p_N} }   \]
We now have a way to express both the determinant and the Hamiltonian in terms of strings of creation and annihilation operators.
The matrix element $\braket{\Phi_{q_1 \cd q_N}  |  \hat{H}  | \Phi_{p_1 \cd p_N} }$  can just be expressed in second quantization as a string of creation and annihilation operators acting on the vacuum state.
 Thus we see that the major mathematical manipulations we have to do to get to the energy reduces down to finding the expectation value of strings of creation and annihilation operators acting in the vacuum!
The next task we have is finding an efficient way of doing so.

\section*{Appendix I. Annihilation operators acting on the determinant}

We adopt a shorthand notation where
\[\Phi_{(p_1\cd p_N)}  = a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} = \prod a\dg \ket{0}\]
and
\[ \Phi_{(p_1\cd\cancel{p_i}\cd p_N)} = a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} =  \prod{}^{p_i} a\dg \ket{0} \]
\\
The task is to prove that
\[a_q \prod a\dg \ket{0} =  \sum_i^N (-1)^{i-1} \delta_{qp_i} \prod{}^{p_i} a\dg \ket{0} \]
First, we note that if $ q \neq p_i $, we can move $a_q$ all the way to the right to act on the vacuum state, which will give us zero.
We thus only have to consider the case where  $q = p_i$:

\begin{align*}
a_q a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} &= (-1)^{i-1} a_{p_1}\dg \cd a_{p_{i -1}}\dg {\color{blue} a_q a_{p_i}\dg} a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}  \\
&=  (-1)^{i-1} a_{p_1}\dg \cd a_{p_{i -1}}\dg {\color{blue} (1 - a_{p_i}\dg a_q) } a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}  \\
&=  (-1)^{i-1} a_{p_1}\dg \cd a_{p_{i -1}} {\color{blue} (1) }  a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}  - (-1)^{i-1}a_{p_1}\dg \cd a_{p_{i -1}}\dg {\color{blue} (a_{p_i}\dg a_q) } a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}
\end{align*}
The second term goes to zero because we can now move $a_q$ to the right until it annihilates the vacuum.
We are thus just left with:
\[a_q a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}  = (-1)^{i-1}  \delta_{pq_i} a_{p_1}\dg \cd a_{p_{i -1}} a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} \]
where we inserted $ \delta_{pq_i}$ to account for the case where $q \neq p_i$.
If we want this result to apply generally such that $p_i$ can represent any indices in the string of creation operators, we can just sum over $i$:
\[a_q a_{p_1}\dg \cd a_{p_{i -1}}\dg a_{p_i}\dg a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0}  = \sum_i^{N} (-1)^{i-1}  \delta_{pq_i} a_{p_1}\dg \cd a_{p_{i -1}} a_{p_{i +1}}\dg \cd a_{p_N}\dg \ket{0} \]
We see that all the terms in the summation will be zero except for the case $q = p_i$.

\section*{Appendix II. A note on commutation of operators}

In general, if two operators commute,
\[ [\hat{A}, \hat{B}] = \hat{A}\hat{B} -\hat{B}\hat{A} = 0 \]
This means that
\[\hat{A}\hat{B} = \hat{B}\hat{A} \]
The order that you write the operators does not matter.
Qualitatively, we can say that two operators act independently of each other.

\end{document}


\chapter{Introduction}
\label{cha:introduction}

RZ is a program that translates mathematical theories to programming
specifications. The translation is done according to the realizability
interpretation of logic, discovered by Kleene~\cite{KleeneSC:intint}.

The original motivation for RZ came from the desire to make explicit
the relationship between constructive mathematics and programming
practice. As is well known, there is a tight relationship between
constructive mathematics and programming, formally expressed as
Curry-Howard propositions-as-types correspondence, and similarly (but
not equivalently) as the realizability interpretation of logic. It
makes sense to exploit this relationship not only in theory but also
in practice. Unfortunately, there seem to be few people who are
trained in constructive logic and are at the same time experienced
programmers. Thus we developed a tool that connects mathematician's
and programmer's worlds by translating mathematical theories to
programming specifications.

Experts in the area of type theory and program specifications often
wonder why there is a need to translate mathematical theories to
specifications. In their view a specification \emph{is} a theory
(whose model is an implementation)? While this is true in a certain
sense, it is not very helpful. Specifications are typically expressed
in a specialized language, such as a variant of Martin-L\"of type
theory, that only specialists are able to use as their ``working
language''. And even they will quickly get lost when it comes to
writing down complex mathematical theories. For a workable enviroment,
we need a mathematical language that is sufficiently close to what is
used in practice. But this makes the language so rich that a tool
which automatically translates it to the world of programing languages
turns out to be very helpful.

RZ can be used also as an aid to teach constructive mathematics.
Because it translates mathematical statements expressed in
constructive logic to specifications expressed in classical
mathematics, it should prove useful as a way of explaining the
computational content of constructive statements. In fact, in the
computable mathematics community a fair amount of time is spent on
finding correct representations of various mathematical
structures---either in terms of numbered sets, as is done in Recursive
Mathematics, or admissible Baire space representations, as is done in
Type Two Effectivity. This task is automated by RZ, which finds
representations in terms of abstract data types that can actually be
compiled on a computer.

\subsection*{What RZ is not}
\label{sec:what-rz-not}

RZ translates theories to specifications. It does \emph{not} translate
proofs to programs, and in fact there is no way to even write down a
proof in RZ. Neither does RZ extract programs from specifications.
While program extraction is an important idea and a worthwhile
endeavor, there are several reasons why we have avoided it in RZ.
First, if one wants to extract programs from proofs or specifications
that are expressed in a rich mathematical language, such as the one
supported by RZ, then it is already non-trivial to compute the types
of extracted programs and assertions they should satisfy. Hence it
makes sense to write such a tool as a first step. Second, formal
proofs are much harder to write and to come by than formalized
theories. We cannot realistically expect ordinary programers to write
formal proofs---at best we can hope they will worry about obtaining a
specification of what is needed \emph{before} writing the program.
Third, until program extraction has made major advances, programers
will write more efficient programs than the current state-of-the-art
program extractors do. Last, it is not our aim to create a tool that
will impress logicians, but rather one that will help in the real
world of programming. Because of this, it is better not to attempt to
write programs that need to solve arbitrarily hard mathematical
problems in order to perform their tasks, which is what the ideal
program extractor is expected to do.

\subsection*{How to read this document}
\label{sec:how-read-this}

RZ relates several aspects of mathematics, logic, and the theory of
programming languages. People who work in these areas have different
points of view and different backgrounds. We have targeted the
document at a reader who poseses basic knowledge of mathematics, and
is interested in programming. In particular, we assume familiarity
with an ML-like programming language, although we explain the
important programming concepts as we go along. Formal logic and
realizability theory are not prerequisites. Naturally, knowledge of
these topics will make it much easier to understand the subject.

Those who are familiar with the theory of realizability will see that
RZ is just an application of familiar techniques and ideas. It is a
demonstration of how realizability can be used in practice.

Most mathematicians are classical by virtue of their education.
Nevertheless, many are curious about constructive logic. RZ may serve
them as (one possible) explanation of constructive logic, motivated by
examples from computer science.

For computer scientists RZ is a connection to constructive
mathematics, as it can be used to translate constructive mathematics
so the language of program specifications.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "userman"
%%% End:


    \foldertitle{SVAR}{Structural Vector Autoregressions: SVAR Objects and Functions}{SVAR/Contents}

	SVAR methods:

\paragraph{Constructor}\label{constructor}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \href{SVAR/SVAR}{\texttt{SVAR}} - Convert reduced-form VAR to
  structural VAR.
\end{itemize}

SVAR objects can call any of the \href{VAR/Contents}{VAR} functions. In
addition, the following functions are available for SVAR objects.

\paragraph{Getting information about SVAR
objects}\label{getting-information-about-svar-objects}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \href{SVAR/get}{\texttt{get}} - Query SVAR object properties.
\end{itemize}

\paragraph{Simulation}\label{simulation}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \href{SVAR/srf}{\texttt{srf}} - Shock (impulse) response function.
\end{itemize}

\paragraph{Stochastic properties}\label{stochastic-properties}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \href{SVAR/fevd}{\texttt{fevd}} - Forecast error variance
  decomposition for SVAR variables.
\end{itemize}

\paragraph{Manipulating SVAR objects}\label{manipulating-svar-objects}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \href{SVAR/sort}{\texttt{sort}} - Sort SVAR parameterisations by
  squared distance of shock reponses to median.
\end{itemize}

See help on \href{VAR/Contents}{VAR} objects for other functions
available.

\paragraph{Getting on-line help on SVAR
functions}\label{getting-on-line-help-on-svar-functions}

\begin{verbatim}
help SVAR
help SVAR/function_name
\end{verbatim}

\paragraph{Getting on-line help on SVAR functions that are inherited
from
VARs}\label{getting-on-line-help-on-svar-functions-that-are-inherited-from-vars}

\begin{verbatim}
help VAR
help VAR/function_name
\end{verbatim}


\section{Security Considerations}
\subsection{Threat Model}
This format is designed to protect files at rest and in transport from accidental disclosure.
%
Using authenticated encryption in individual segments mirrors solutions like Transport Layer Security (TLS) as described in~\cite{RFC5246} and prevents undetected modifications of segments.

\subsection{Selection of Key and Nonce}
The security of the format depends on attackers not being able to guess the encryption key (and to a lesser extent the nonces).
%
The encryption key and nonce SHOULD be generated using a cryptographically-secure pseudo-random number generator and MUST NOT be repeated with the same encryption key.
%
This makes the chance of guessing a key vanishingly small.
%
Additional security can be provided by using `Associated Data' when encrypting a file.
%
This data must be used to decrypt the data, although it is not part of the encrypted file \cite{RFC8439}

\subsection{Message Forgery}
The encrypter's public key component is required to calculate the encryption key used for the header in the Curve25519 function.
If this public key is not specified in the header directly, it must be obtained from the encrypter, and can then be used to validate the origin of the file.
The encrypted header part is authenticated using the Poly1305 algorithm over the header data.
%
Using ChaCha20-ietf-Poly1305, the Poly1305 algorithm produces an authentication tag for each encrypted segment of the encryoted data.

\subsection{No File Updates Permitted}
Implementations MUST NOT update encrypted files.
%
Once written, a section of the file must never be altered.


% !TEX encoding = UTF-8 Unicode
\section{Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods}
%%%%%%%%%%%%%%%%
	\begin{frame}[allowframebreaks]{Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods}
		\begin{itemize}
			\item Jacobi,
			\item Gauss-Seidel,
			\item Succesive-Over-Relaxation (SOR),
			\item Symmetric SOR;
			\item Cojugate Gradient (CG),
			\item Minimal Residual (MINRES),
			\item Symmetric LQ (SYMMLQ),
			\item Conjugate Gradients on the Normal Equations (CGNE, CGNR),
			\item Generalized Minimal Residual (GMRES),
			\item Biconjugate Gradient (BICG),
			\item Quasi-Minimal Residual (QMR),
			\item Conjugate Gradient Squared (CGS),
			\item Biconjugate Gradient Stabilized (BiCGSTAB),
			\item Chebyshev Iteration.
		\end{itemize}
		Wprowadzenie do metod iteracyjnych $Ax = b$; \\
		Odsyłacze do literatury (klasyczne pozycje); \\
		Programy w:
		\begin{itemize}
			\item Fortranie i w C (pojedyncza i podwójna precyzja),
			\item Matlab scripts,
			\item C++.
		\end{itemize}
	\end{frame}