Machine learning (ML) is the study of algorithms and mathematical models with a feature of progressively improve a performance on a specific task. Machine learning is a subfield of Artificial Intelligence (AI). The main difference between Artificial Intelligence and Machine learning is that machine learning performance results primary depends on the data set and it makes data driven decisions while Artificial Intelligence involves agents at the top. Another subfield of machine learning is Deep Learning that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Hierarchy representation of Artificial Intelligence, Machine Learning and Deep Learning could be seen in the \ref{fig:ml_hierarchy} figure.

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{Pictures/ml_hierarchy.png}
\caption{\label{fig:ml_hierarchy}{}Hierarchy representation of AI, ML and Deep Learning}
\captionsetup{font={footnotesize,bf,it}}
\caption*{Source: https://www.codesofinterest.com/2016/11/difference-artificial-intelligence-machine-learning-deep-learning.html}
\end{figure}

Machine learning models is divided into two types:

\begin{enumerate}
    \item \textbf{Supervised learning} - is the machine learning task of learning a function that maps an input to an output based on example input-output pairs.
    \item \textbf{Unsupervised learning} - is the machine learning tasks that learns from test data that has not been labeled, classified or categorized.
\end{enumerate}

\subsection{Supervised Learning}

Supervised Machine learning models are all about finding appropriate representations for their input data and it requires 3 things:
\begin{enumerate}
    \item \textbf{Input data}. A good data set is the key of creating good Machine Learning model. The input data depends on the problem which the developer want to solve - if the task is speech recognition, then the input data could be sound files converted to the form that computer could proceed, for example: binary. If the task is image tagging, then the data could be a pictures where each pixel is converted to the RGB (red-green-blue) format or HSV (hue-saturation-value) format. The input data depends on the problem and the final goal of the problem.
    \item \textbf{Output data}. Output data in other words could be called as results of the input data. Supervised machine learning models should know the results of each data input entry point in order to find out a pattern which helps to predict a results.
    \item \textbf{Validation}. Validation is a way to measure whether the algorithm is doing a good job to determine the distance between the algorithm's current output and its expected output. The validation for supervised machine learning models is split into 2 parts: training data and testing data. Training data is input data and output data which is used for machine learning model training and it from where model learns the patterns of the data which produce certain output. The testing data is used after the machine learning training and from this data the model is evaluated of how successfully it predicted the output results from the input data.
\end{enumerate}

Supervised learning is grouped into two types:

\begin{enumerate}
    \item \textbf{Classification} A classification type is when the output variable is representing a category
    \item \textbf{Regression} A regression type is when the output variable is representing a real value
\end{enumerate}

\subsubsection{Classification}

Classification is the process of predicting the class of given data points. Classes are usually named as a term of \textbf{Labels}. Classification main goal is from given input data points predict an output which would mark a label. Classification predictive modeling: approximating a mapping function (f) from input variables (X) to discrete output variable (y).

Classification Machine Learning algorithms:

\begin{enumerate}
    \item \textbf{Linear Classifiers} is the statistical classification group of identifying classes of the object's characteristics. A linear classifier methods makes a classification decisions based on the values of a linear combinations of the characteristics. In this paper two linear classifiers would be described: \textit{Logistic Regression} and \textit{Naive Bayes Classifier}
    \begin{enumerate}
        \item \textbf{Logistic Regression}
        \label{sssec:logistic_regression}
        Logistic Regression is a statistical method for analysing a data set in which there are one or more independent variables that determine an measured(outcome) with a dichotomous variable. It predicts the probability of an outcome that have two values.

        The main goal of logistic regression is to find the best fitting model to describe relationship between the dichotomous characteristics of interest and a set of independent variables.

        Logistic regression generates a logistic curve (yellow color) (the Linear and logistic model representation figure \ref{fig:logistic_regression}) which y-axis is limited to values between 0 and 1; [0, 1].

        \begin{figure}[H]
            \centering
            \includegraphics[width=0.8\textwidth]{Pictures/logistic_regression.png}
            \caption{\label{fig:logistic_regression}{} Linear model and Logistic model representation \cite{16}}
        \end{figure}

        \textbf{Logistic model} mathematical representation: $p = \frac{1}{1 + e^-(b_{0} + b_{1}x)}$

        \textbf{Linear model} mathematical representation: $y = b_{0} + b_{1}x$

        \textbf{Logistic regression} could be expressed in the formula:

        \begin{equation}
             \frac{p}{1 - p} = \exp{b_{0} + b_{1}x}
        \end{equation}
        \begin{description}
            \item[$p$] Logistic Model
            \item[$b_{0}$] Logistic regression constant
            \item[$b_{1}$] The slope that defines the steepness of the curve
        \end{description}

        \item \textbf{Naive Bayes Classifier}.

        Naive Bayes classifier assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature. Bayes theorem could be expressed in the mathematical equation:

        \begin{equation}
                P(A|B) = \frac{P(B|A)P(A)}{P(B)}
        \end{equation}
        \begin{description}
            \item[$A$ and $B$] are events
            \item[$P(B)$] $\ne 0$
            \item[$P(A | B)$] is a conditional probability: the likelihood of event $A$ occurring given that $B$ is true.
            \item[$P(B | A$] is also a conditional probability: the likelihood of event $B$ occurring given that $A$ is true.
            \item[$P(A)$ and $P(B)$] are the probabilities of observing $A$ and $B$ independently of each other; this is known as the marginal probability
        \end{description}

        Naive Bayes \cite{BIB4} is effective in many practical applications including text classification, performance management and medical diagnosis. The effectiveness of Naive Bayes classifier comes from it is presence of feature dependencies: optimality in terms of zero-one loss (classification error) is not necessarily related to the quality of the fit to a probability distribution.

    \end{enumerate}
    \item \textbf{Support Vector Machines(SVM)}

    Support vector machine(SVM) is a discriminative classifier defined by a separating hyperplane. Hyperplane is a line dividing a plane i two parts where in two parts. SVM algorithm main objective is to find a optimal hyperplane in an N-dimensional (N - is the number of features) space that distinctly classifies the data points.

    When using Support Vector Machine it should be considered these parameters:

    \begin{itemize}
        \item Input. Set of training pair samples. Call the input sample features $x_{1}, x_{2}...x_{n}$ and the output result y
        \item Output. Set of weights $w$. One for each feature, whose linear combination predicts the value of y
    \end{itemize}

    SVM usage in the real world applications:

    \begin{itemize}
        \item Text and Hypertext categorization. SVM method could significantly reduce the need for labeled training instances in both the standard inductive and transductive settings.
        \item Classification of images. SVM achieve higher search accuracy than traditional query refinement schemes
        \item Biological and other sciences. SVM performed good results in classification of proteins schemes or classifying permutation tests.
    \end{itemize}

    \item \textbf{Decision Trees}

    Decision tree are flowchart-like structures that classifies input data points or predict output values given inputs.
    A decision tree is a decision-making device which assigns a probability to each of the possible choices based on the context of the decision: $P(f|h)$, where $f$ is an element of the set of choices and $h$ is the context of the decision. Probability $P(f|h)$ is determined by asking the sequence of questions $q_{1}, q_{2},...,q_{n}$ about the context, where the $ith$ question asked is uniquely determined by the answers to the $i - 1$ previous questions \cite{BIB5}.

    Decision tree builds classification or regression models in the form of a tree data structure. The data sets is break into smaller subsets. Decisions are represented as nodes and leaf nodes in the tree.

    A decision tree consist of three types of nodes:

    \begin{enumerate}
        \item Decision nodes - represented by squares
        \item Chance nodes - represented by circles
        \item End nodes - represented by triangles
    \end{enumerate}

    Advantages of Decision trees classification:

    \begin{itemize}
        \item Easy to understand and interpret because of the tree structure.
        \item The results could be achieved even with the little data.
        \item Could determine best, worst and expected values for different scenarios
    \end{itemize}

    Disadvantages of Decision trees classification:

    \begin{itemize}
        \item It could be unstable. Small changes could imply large changes in the structure of the decision tree
        \item They are often relatively inaccurate comparing with others classification algorithms
        \item Information gain in decision trees is biased in favor of those attributes with more levels
    \end{itemize}
\end{enumerate}

\subsubsection{Regression}

Another type of Machine learning is called Regression. Regression algorithms tries predict a value for an input based on previous information. The main difference of classification type and regression type is that regression main goal is to estimate a value while classification type main goal is to estimate a class of an observation.

Regression models have the following parameters and variables:

\begin{itemize}
    \item \textbf{The unknown parameters}, denotes as $\beta$, which may represent a scalar or a vector
    \item \textbf{The dependent variable, Y}. This is a main factor that has to be understood and predicted.
    \item \textbf{The independent variables, X}. This is a factor which have an impact on dependent variable
\end{itemize}
A regression model relates Y to a function of X and $\beta$: $Y \approx f(X, \beta)$

Regression Machine Learning algorithms:
\begin{enumerate}
    \item \textbf{Linear Regression}

    Linear regression \cite{BIB6} is a technique that analyze the relationships between variables and how they contribute and related to producing a particular outcome. Linear regression establishes a relationship between \textbf{dependent variable (Y)} and \textbf{independent variables (X)} using straight line also known as regression line.

    Linear regression is represented in the formula:

    \begin{equation}
     Y = a + b\times X + e
    \end{equation}
    \begin{description}
        \item[Y] Dependent variable
        \item[X] Independent variables
        \item[a] Intercept
        \item[b] Slope
        \item[e] Error term
    \end{description}
    This equation represents of how Linear Regression method predicts value of target variable on given predictor variables.

    Example of the linear regression scatter plot could be seen in the figure \ref{fig:linear_regression}. The black line consists of the predictions, the points are the data and the vertical lines between the points and  the black line represent errors of prediction.
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.55\textwidth]{Pictures/linear_regression.png}
        \caption{\label{fig:linear_regression}{}Scatter plot of the linear regression method \cite{BIB17}}
    \end{figure}

    Logistic regression \tectbg{properties}:

    \begin{itemize}
        \item Linear regression method requires linear relationship between independent variable (Y) and dependent variables X
        \item Linear regression results could be drastically dependent on Outliers. Outliers are the Data points that diverge in a big way from the overall pattern.
        \item Multicollinearity could increase the variance of the coefficient estimates and make the estimates sensitive to minor changes in the model. Multicollinearity is a statistical process in which predictor variables are correlated.
    \end{itemize}

    \textbf{Advantages} of using logistic regression model:

    \begin{itemize}
        \item Space complexity of the logistic regression model is low because it needs only to save the weights at the end of training
        \item Good interpretability, simple to understand
        \item Feature importance is generated at the time model building. Dimensionality reduction could be achieved by handling feature selection and by using hyperparameters
    \end{itemize}

    \textbf{Disadvantages} of using logistic regression model:

    \begin{itemize}
        \item Multicollinearity should be avoided
        \item Prone to outliers
        \item Linear regression assumes that the data is independent
    \end{itemize}

    \item \textbf{Polynomial Regression}

    Polynomial Regression method is the relationship between the \textbf{independent variables X } and the \textbf{dependent variable Y} which is modelled as an $nth$ degree polynomial. This method is good for handling non-linearly separable data. This method fits a nonlinear relationship between the value of x and the corresponding conditional mean of Y, denoted as $E(Y|X)$.

    $nth$ degree polynomial regression method could be described in the mathematical formula:
    \begin{equation}
     Y = \beta_{0} + \beta_{1}x + \beta_{2}x^2 + \beta_{3}x^3 + ... + \beta_{n}x^n + e
    \end{equation}
    \begin{description}
        \item[Y] Dependent variable
        \item[x] Independent variables
        \item[$\beta$] Unknown parameters
        \item[e] Error term
    \end{description}

    Polynomial regression properties:
    \begin{itemize}
        \item Otherwise than linear regression, polynomial regression method could model non-linearly seperable data
        \item Full control over the modelling of feature variables
        \item Prone to over fitting if exponents are not selected according to model design.
    \end{itemize}

    Polynomial regression have few problems that should be taken to consideration before using this algorithm:
    \begin{enumerate}
        \item When the X values are large, the model could be leaded to the math overflow which could make results inaccurate.
        \item The parameters of the model are intertwined so that lead=s of having covariance and dependency cases between the X values.
    \end{enumerate}

    \item \textbf{Ridge Regression}

    Ridge regression is a remedial measure taken to alleviate multicollinearity amongst regression predictor variables in a model. By adding a degree of bias to the regression estimates, ridge regression reduces the standart errors rate.

    Ridge regression model formula:
    \begin{equation}
     Y = X\beta + e
    \end{equation}
    \begin{description}
        \item[Y] Dependent variable
        \item[X] Independent variables
        \item[$\beta$] Regression coefficients to be estimated
        \item[e] Error term
    \end{description}

    The presence of multicollinearity could be detected in a few ways:
    \begin{itemize}
        \item A regression coefficient is not significant even though variables should be correlated with dependent variable Y
        \item The regression coefficients changes dramatically if X independent variables are added or deleted
        \item X independent variables have high pairwise correlations
    \end{itemize}

    Features of ridge regression model:
    \begin{itemize}
        \item The assumption is the same as least squared regression
        \item The value of coefficients does not reach zero
    \end{itemize}

    \item \textbf{Lasso Regression}

    Lasso regression \cite{BIB7} method performs both variable selection and regularization in order to enhance the prediction. Lasso regression and ridge regression are similar methods but lasso regression is using an absolute value bias while ridge regression method is using squared value bias.

    The goal of lasso method is to obtain the subset of predictors that minimizes prediction error for a quantitative response variable. This is done by shrinking variables towards zero.

    The lasso estimate is defined by formula:

    $\beta^{\mathit{lasso}} = argmin \sum_{i=1}^N(y_{i} - \beta{0} - \sum_{j=1}^P x_{ij}\beta_{j})^2$ subject to $\sum_{j=1}^P |\beta_{j}| <= t$

    Lasso regression translates each coefficient by a constant factor $\lambda$, truncating at zero. This process is called "soft thresholding," and is used in the context of wavelet-based smoothing.

    Lasso regression is often an effective technique for shrinkage and feature selection.

\end{enumerate}

\subsection{Unsupervised Learning}

Unsupervised machine learning clusters only need to have an input data which would be used for detecting a patterns of the data.

\subsubsection{Clustering}

Clustering is a task of grouping a set of objects by their data property patterns. Clustering methods are used for identifying similar groups to entities of that group than those of the other groups. Clustering methods are diverse by their types:
\begin{enumerate}
    \item \textbf{Centroid-based}

    Clustering model which is related to the notion of similarity is derived by the closeness of a data point to the centroid of the clusters. Using algorithms with this type it is important to know the number of clusters before grouping data.

    \item \textbf{Distributed-based}

    Clustering model which is based on the notion of how probable is it that all data points in the cluster belong to the same distribution. Distributed-based models tends more suffer from overfitting.

    \item \textbf{Connectivity-based}

    Clustering model which is based on the notion that the data points closer in data space exhibit more similarity to each other than the data points lying farther away. This model could be diverse into two approaches:
    \begin{enumerate}
        \item Classifying all data points into separate clusters and then aggregating them as the distance decrease.
        \item All data points are classified as a single cluster and then partitioned as the distance increases.
    \end{enumerate}
    Models are very easy to interpret but lacks scalability for handling big data sets

    \item \textbf{Density-based}

    Clustering model which is search the data space for areas of varied density of data points in the data space. It isolates various different density regions and assign the data points within these regions in the same cluster.

\end{enumerate}

\begin{enumerate}

    \item \textbf{K-Means Clustering}

    K-mean is a centroid model based clustering algorithm which the main goal is to partition the inputs into sets $S1,...,S_{k}$ in a way that minimizes the total sum of squared distances from each point to the mean of its assigned cluster. $k$ representing the number of clusters.

    K-means \cite{BIB8} clustering abstract algorithm:

    \begin{enumerate}
        \item Decide on a value for K, the number of clusters
        \item Initialize the K cluster centers
        \item Decide the class memberships of the N objects by assigning them to the nearest cluster center
        \item Re-estimate the K cluster centers, by assuming the memberships found above are correct
        \item Repeat c and d until none of the N objects changed membership in the last iteration
    \end{enumerate}

    \textbf{Advantages} of K-mean clustering :
    \begin{itemize}
        \item K-mean clustering is fast. Computing the distances between points and group centers requires few computations. The linear complexity of the algorithm is \textbf{O(n)}
        \item An instance could move to another cluster when the centroids are recomputed
        \item Converges to local minimum of within-cluster squared error
    \end{itemize}

    \textbf{Disadvantages} of K-mean clustering:
    \begin{itemize}
        \item The exact number of clustering groups/classes should be known before using k-means clustering algorithm
        \item K-means starts with a random choice of cluster centers and it could perform different results for each algorithm run
        \item The order of the data has an impact to the final results
        \item Sensitive to outliers
        \item Detects spherical clusters only
    \end{itemize}

    \item \textbf{Mean-Shift Clustering}

    Mean-shift \cite{BIB9} clustering algorithm is a non parametric clustering technique which does not require prior knowledge of the number of clusters and does not constrain the shape of the clusters.

    \begin{equation}
        K(x) =
        \left \{
          \begin{tabular}{ccc}
          1 if ||x|| <= \lambda \\
          0 if ||x|| > \lambda
          \end{tabular}
        \right \}
    \end{equation}
    \begin{description}
         Let data be a finite set $S$ embedded in the n-dimensional Euclidean space, $X$. Let $K$ be a flat kernel that is the characteristic function of the $\lambda$-ball in $X$
    \end{description}
    The sample mean at x \in X is:

    \begin{equation}
        m(x) = \frac{\Sigma_{s \in S} K(s-x)s}{\Sigma_{s \in S} K(s-x)}
    \end{equation}

    The difference $m(x) - x$ is called \textit{mean shift}. In each iteration of the algorithm, $s \Leftarrow m(s)$ is performed for all $s \in S$ simultaneously. The mean shift vector always points toward the direction of the maximum increase in the density.

    Mean-shift clustering abstract algorithm:
    \begin{enumerate}
        \item Circular sliding window centered at a randomly selected point $C$ and having radius $r$ as the kernel.
        \item At every iteration the sliding window is shifted towards regions of higher density by shifting the center points to the mean of the points withing the window while there is direction at which a shift can accommodate more points inside the kernel
        \item  This process of steps (a) and (b) is done with many sliding windows until all points lie within a window.
    \end{enumerate}

    \textbf{Advantages} of Mean-Shift clustering:
    \begin{itemize}
        \item Model free - does not assume any prior shape on data clusters
        \item It requires single parameter of window size $h$
        \item Robust to outliers
    \end{itemize}

    \textbf{Disadvantages} of Mean-Shift clustering:
    \begin{itemize}
        \item Results depends on window size
        \item Window size selection is not trivial
        \item Computationally expensive
        \item Does not scale well with dimension of feature space
    \end{itemize}

    \item \textbf{Density-Based Spatial Clustering of Applications with Noise (DBSCAN)}

    Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is a density based clustering algorithm. Given a set of points in some space, it groups together points that are closely packed together, marking as outliers points that lie alone in low-density regions. Overall average complexity of DBSCAN clusetering algorithm is \textbf{O(log n)}.

    DBSCAN abstract algorithm:
    \begin{enumerate}
        \item Find the points in the $\epsilon$ neighborhood of every point, and identify the core points with more than $minPts$ neighbors
        \item Find the connected components of core points on the neighbor graph, ignoring all non-core points
        \item Assign each non-core point to a nearby cluster if the cluster is an $\epsilon$ neighbor, otherwise assign it to noise
    \end{enumerate}

    \textbf{Parameters}  of DBSCAN clustering algorithm:
    \begin{itemize}
        \item $MinPoints$: The minimum number of points to form a dense region
        \item $\epsilon$: The minimum distance between two points. The points are considered neighbors if the distance between two points is lower or equal to $\epsilon$ value
    \end{itemize}

    \textbf{Advantages} of DBSCAN clustering algorithm:
    \begin{itemize}
        \item DBSCAN does not require one to specify the number of clusters in the data
        \item DBSCAN can find arbitrarily shaped clusters
        \item DBSCAN has a notion of noise, and is robust to outliers
        \item DBSCAN requires just two parameters and is mostly insensitive to the ordering of the points in the database
        \item DBSCAN is designed for use with databases that can accelerate region queries, e.g. using an R* tree
    \end{itemize}

    \textbf{Disadvantages} of DBSCAN clustering algorithm:
    \begin{itemize}
        \item DBSCAN is not entirely deterministic: border points that are reachable from more than one cluster can be part of either cluster, depending on the order the data are processed
        \item The results of DBSCAN model depends on the distance measure
        \item DBSCAN cannot cluster data sets well with large differences in densities
        \item Choosing a meaningfull distance treshhold $\epilepson$ could be difficult if the data is not proper for this algorithm
    \end{itemize}

    \item \textbf{Expectation–Maximization (EM) Clustering}

    The Expectation–Maximization clustering algorithm is an iterative method to find maximum likelihood or maximum a posteriori (MAP) which computes probabilities of cluster memberships based on one or more probability distributions. The goal of the EM clustering algorithm is to maximize the overall probability or likelihood of the data.

    The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying these two steps:
    \begin{enumerate}
        \item Expectation step (E step): Define $\Theta(\theta|\theta^{(t)})$ as the expected value of the log likelihood function of $theta$, with respect to the current conditional distribution of $Z$  given $X$  and the current estimates of the parameters $\theta^{(t)$:
        \begin{equation}
            \Theta(\theta|\theta^{(t)}) = E_{Z|X_{1}\theta^{(t)}}[\log L(\Theta;X;Z)]
        \end{equation}
        \item Maximization step (M step): Find the parameters that maximize this quantity:
        \begin{equation}
            \theta^{(t+1)} = arg max_\theta \Theta(\theta|\theta^{(t)}
        \end{equation}
    \end{enumerate}

    \begin{description}
        \item [X] Observed data
        \item [Z] a set of unobserved latent data or missing values $Z$
        \item [$\theta$] a vector of unknown parameters
        \item [$L(\theta,X, Z)$] likelihood function
    \end{description}

    Expectation–Maximization abstract algorithm:
    \begin{enumerate}
        \item Initialize the parameters $\theta$ to some random values
        \item Compute the probability of each possible value of $Z$, given $\theta$
        \item Use the just-computed values of $Z$  to compute a better estimate for the parameters $\theta$
        \item Iterate steps (b) and (c) until convergence
    \end{enumerate}

    \textbf{Advantages} of Expectation–Maximization clustering algorithm:
    \begin{itemize}
        \item Likelihood is guaranteed to increase for each iteration
        \item Is a derivative-free optimizer
        \item Is fast if analytical expressions for the M-step are available
        \item Parameter constraints are often dealt with implicitly
    \end{itemize}

    \textbf{Disadvantages} of Expectation–Maximization clustering algorithm:
    \begin{itemize}
        \item Requires both forward and backward probabilities
        \item Significant implementational effort required compared to numerical optimization
        \item I Convergence may be slow if analytical expressions for the M-step are not available since numerical optimization must be applied
        \item Hessian must be calculated manually
    \end{itemize}

    \item \textbf{Hierarchical Clustering}

   Hierarchical clustering \cite{BIB10} is a method of cluster analysis which seeks to build a hierarchy of clusters. Hierarchical clustering algorithm run once and create a dendrogram which is a tree structure containing a k-block set partition for each value of k between 1 and n, where n is the number of data points to cluster allowing the user to choose a particular clustering.

   Hierarchical clustering consist of two types:
   \begin{enumerate}
       \item \textbf{Agglomerative}:  This is a "bottom-up" approach: each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy. Bottom-up algorithms treat each data point as a single cluster at the outset and then successively merge  pairs of clusters until all clusters have been merged into a single cluster that contains all data points. The time complexity of this method is $O(n^3)$.
       \item \textbf{Divisive}: This is a "top-down" approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy. Divisive clustering complexity is $O(2^n)$.
   \end{enumerate}

    Agglomerative Hierarchical clustering abstract algorithm \cite{BIB11}:
    \begin{enumerate}
        \item Compute the similarity between all the pairs of clusters
        \item Combine the foremost similar two clusters
        \item Update the similarity matrix to replicate the pairwise similarity between the new cluster and the original clusters
        \item Repeat steps b and c until only a single cluster remains
    \end{enumerate}

    \textbf{Advantages} of Hierarchical clustering algorithms:
    \begin{itemize}
        \item Hierarchical clustering does not require us to specify the number of clusters
        \item Algorithm is not sensitive to the choice of distance metric
    \end{itemize}

    \textbf{Disadvantages} of Hierarchical clustering algorithms:
    \begin{itemize}
        \item Sensitivity to noise and outliers
        \item Breaking large clusters
        \item Difficulty handling different sized clusters and convex shapes
    \end{itemize}

\end{enumerate}

\subsubsection{Dimensionality reduction}

Dimensionality reduction \cite{BIB13} is the transformation of high-dimensional data into a meaningful representation of reduced imensionality.Ideally, the reduced representation should have a dimensionality that corresponds to the intrinsic dimensionality of the data. The intrinsic dimensionality of data is the minimum number of parameters needed to account for the observed properties of the data.

The problem of dimeansionality reduction could be defined as follows: Data set is represented in a $n$ x $D$ matrix $X$ consisting of $n$ datavectors $x_{i} (i \in {1, 2, ..., n})$ ) with dimensionality $D$. The data set has intrinsic dimensionality $d (where d < D)$, in mathematical terms, intrinsic dimensionality means that the points in data set $X$ are
lying on or near a manifold with dimensionality $d$ that is embedded in the D-dimensional space.
Dimensionality reduction techniques transform dataset $X$ with dimensionality $D$ into a new data set $Y$ with dimensionality $d$, while retaining the geometry of the data as much as possible. High-dimensional data point is denoted by $x_{i}$ , where $x_{i}$ is the $i$th row of the D-dimensional data matrix $X$. The low-dimensional counterpart of $x_{i}$ is denoted by $y_{i}$ , where $y_{i}$ is the $i$th row of the d-dimensional data matrix $Y$. The data set $X$ is assumed as a zero-mean.

There are two components of dimensionality reduction:
\begin{enumerate}
    \item \textbf{Feature selection}. The subset of the original set of variables are found to get a smaller subset which could be used to model the problem. This component is divided into three separate parts:
    \begin{enumerate}
        \item \textit{Filter}. Filter out features with small potential to predict outputs. Filter operation could be expressed in mathematical representation:
        \begin{enumerate}
            \item Let $\Phi$ be a current set of features
            \item Removing feature $\phi_}k}(x)$ is possible only when:
            \begin{equation}
                \Tilde{P}(y|\Phi|\phi_{k}) \approx \Tilde{P}(y|\Phi)
            \end{equation}
            For all values of $\phi_{k}, y$
        \end{enumerate}

        \item \textit{Wrapper}. Select features that directly optimize the accuracy of the classifier
        \item \textit{Embedded}. Features are selected to add or be removed while building the model based on the prediction errors
    \end{enumerate}
    \item \textbf{Feature extraction}. Reduces the data in a high dimensional space to a lower dimension space
\end{enumerate}

\textbf{Advantages} of Dimensionality Reduction:
\begin{itemize}
    \item Improves performance in data compression
    \item Reduces computation time
    \item Removes redundant features
    \item Hence reduced storage space
\end{itemize}

\textbf{Disadvantages} of Dimensionality Reduction:
\begin{itemize}
    \item It may lead to some amount of data loss.
\end{itemize}

\begin{enumerate}
    \item \textbf{Principal Component Analysis (PCA)}

    Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. PCA is a linear transformation of $d$ dimensional input $x$ to $M$ dimensional feature vector $z$ such that under which the retained variance is maximal. It is also considered as the linear projection for which the sum of squares reconstruction cost is minimized.

    \textbf{Goals} of Principal Component Analysis (PCA):
    \begin{itemize}
        \item Simplification
        \item Data reduction
        \item Outlier Detection
        \item Variable selection
        \item Classification
        \item Prediction
        \item Unmixing
    \end{itemize}

    Many of the goals of PCA are concerned with finding relationships between objects. PCA estimates the correlation structure of the variables. The importance of a variable in a PC model is indicated by the size of its residual variance.

    PCA \cite{BIB14} in matrix form is the least squares model:
    \begin{equation}
        X = 1\Tilde{x} + TP' + E
    \end{equation}
    \begin{description}
        \item [$\Tilde{x}$] Mean vector which is included in the model formulation
        \item [P'] Projection matrix. It is also called loading vector
        \item [T] Object coordinates in the plane. It is also called scoring vectors
        \item [X] The projection
        \item [E] The deviations between projections and the original coordinates are termed the residuals.
    \end{description}

    \item \textbf{Linear Discriminant Analysis (LDA)}

    Linear Discriminant Analysis (LDA) \cite{BIB15} is a method to find a linear combination of features that characterizes or separates two or more classes of objects or events.

    Given a data matrix $A \in \R^{N×n}$, classical LDA aims to find a transformation $G \in \R^{N×t}$ that maps each column $a_{i}$ of $A$, for $1 ≤ i ≤ n$, in the N-dimensional space to a vector $b_{i}$ in the $\iota$-dimensional space. That is $G$ : $a_{i} \in \R \Leftarrow b_{i} = G^T a_{i} \in \R^{\iota} (\iota < N)$. Equivalently, classical LDA aims to find a vector space $G$ spanned by {g$_{i}}^{\iotai}_{i=1}$}, where $G = [g_{1},..., g_{\iota}]$, such that each $a_{i}$ is projected onto $G$ by $(g^{T}_{1} · a_{i},..., g^{T} · a_{i})T \in \R^{\iota}$.

    Assume that the original data in $A$ is partitioned into $k$ classes as $A = {\Pi_{1}, ... , \Pi_{k}}$, where $\Pi_{i}$ contains $n_{i}$ data points from the $i$th class, and $\sum\limits{i=1}^{k} n_{i} = n$. Classical LDA aims to find the optimal transformation $G$ such that the class structure of the original high-dimensional space is preserved in the low-dimensional space. In general, if each class is tightly grouped, but well separated from the other classes, the quality of the cluster is considered to be high. In discriminant analysis, two scatter matrices, called within-class ($S_{w}$) and between-class ($S_{b}$) matrices, are defined to quantify the quality of the cluster, as follows [4]: $S_{w} = \sum\limits{i=1}^{k} \sum\limits{x \in \Pi_{i}} (x − m_{i})(x − m_{i})T$, and $S_{b} = \sum\limits{i=1}^{k} n_{i}(m_{i} − m)(m_{i} − m)T$, where $m_{i} = \frac{1}{n_{i}} \sum\limits{x \in \Pi_{i}} x$ is the mean of the $i$th class, and $m = \frac{1}{n}\sum\limits{i=1}^{k} \sum\limits{x \in \Pi|{i}} x$ is the global mean.

    Notation:
    \begin{description}
        \item [$n$] number of instances in the data set
        \item [$k$] number of classes in the data set
        \item [$A_{i}$] $i$th instance in matrix representation
        \item [$a_{i}$] $i$th instance in vectors representation
        \item [$r$] r number of rows in $A_{i}$
        \item [$c$] number of columns in $A_{i}$
        \item [$N$] dimension of $a_{i} (N = r ∗ c)$
        \item [$\Pi$] $j$th class in the data set
        \item [$L$] transformation matrix (left) by Two-Dimensional Linear Discriminant Analysis
        \item [$R$] transformation matrix (right) by Two-Dimensional Linear Discriminant Analysis
        \item [$I$] number of iterations in Two-Dimensional Linear Discriminant Analysis
        \item [$B_{i}$] reduced representation of $A_{i}$ by Two-Dimensional Linear Discriminant Analysis
        \item [$\iota_{1}$] number of rows in $B_{i}$
        \item [$\iota_{2}$] number of columns in $B_{i}$
    \end{description}

    The goal of Linear Discriminant Analysis (LDA) is to project a data set onto a lower-dimensional space with good class-separability in order avoid overfitting and also reduce computational costs.

    \textbf{Pseudo algorithm} for using LDA approach:
    \begin{enumerate}
        \item Compute the $d$-dimensional mean vectors for the different classes from the data set
        \item Compute the scatter matrices
        \item Compute the eigenvectors ($e_{1},e_{2},...,e{d}$) and corresponding eigenvalues ($\lambda_{1},\lambda_{2},...,\lambda_{d}$) for the scatter matrices
        \item Sort the eigenvectors by decreasing eigenvalues and choose $k$ eigenvectors with the largest eigenvalues to form a $d$×$k$ dimensional matrix $W$
        \item Use $d$×$k$ eigenvector matrix to transform the samples onto the new subspace. This can be summarized by the matrix multiplication: $Y$=$X$×$W$, where
        \begin{description}
            \item [X] is a $n$×$d$-dimensional matrix representing the $n$ samples
            \item [Y] are the transformed $n$×$k$-dimensional samples in the new subspace
        \end{description}
    \end{enumerate}

\end{enumerate}

\subsubsection{Association analysis}

Association analysis is a method which is useful for discovering relationships hidden in large data sets. The uncovered relationships can be represented in the form of sets of items present in many transactions, which are known as \textbf{association rules} that represents relationships between two item sets. Association rule mining finds all rules in the database that satisfy some minimum support and minimum confidence constraints.

One of the real life examples in where association rules could be used is two items expression:

\textbf{Olives} \rightarrow \textbf{Wine}

The rule suggest a relationship between olives usage with wine because olives is well know a good pair with wine.

An \textbf{association rule} \cite{BIB12} is an implication expression of the form $X \rightarrow Y$ , where $X$ and $Y$ are disjoint item sets, i.e., $X ∩ Y = ∅$. The strength of an association rule can be measured in terms of its support and confidence. Support determines how often a rule is applicable to a given data set, while confidence determines how frequently items in $Y$ appear in transactions that contain $X$. These metrics has an formal definitions:

\begin{equation}
    Support, s(x \rightarrow Y) = \frac{\delta(X \cup Y)}{N}
\end{equation}
\begin{equation}
    Confidence, c(x \rightarrow Y) = \frac{\delta(X \cup Y)}{\delta(X)}
\end{equation}

\subsection{Machine Learning models evaluation}
Machine Learning models performance could be evaluated of how well they are classifying websites categories. Evaluation scores is a first level indicator which lets to know of Machine Learning capabilities of predicting categories according to the primary training features data sets.

Machine Learning models could predict website categories when training features set are fitted into model. After that model is fed with training features set and it outputs prediction set of predicted categories of websites. These predicted categories are compared with training labels set. Models predictions set and training labels set allows to generate evaluation scores and analysis of how well model is trained to predict categories of websites.

Calculating models predictions scores, there are 4 special terms which appears in the formulas:
\begin{itemize}
    \item \textbf{True Positives (TP)} - is an outcome where the model correctly predicts the positive class. Example of true positive condition: \textit{Person with disease was diagnosed a disease}.
    \item \textbf{True Negatives (TN)} - is a true negative is an outcome where the model correctly predicts the negative class. Example of true negative condition: \textit{Person with no disease was not diagnosed a disease}.
    \item \textbf{False Positives (FP)} - is a result that indicates a given condition exists, when it does not. A false positive error is a type I error where the test is checking a single condition, and wrongly gives an affirmative decision.  Example of false positive condition:: \textit{Healthy person was diagnosed with a specific disease}.
    \item \textbf{False Negatives (FN)} - is a test result that indicates that a condition does not hold, while in fact it does. A false negative error is a type II error occurring in a test where a single condition is checked for and the result of the test is erroneously that the condition is absent.  Example of false negative condition:: \texit{Person with disease was diagnosed with no disease}.
\end{itemize}

There are several methods to evaluate machine learning models :
\begin{enumerate}

    \item \textbf{Accuracy score}

    Classification accuracy is the number of correct predictions made as a ratio of all predictions made.

    The accuracy score is calculated by formula:
    \begin{equation}
        Accuracy = \frac{TP + TN}{TP + TN + FP + FN}
    \end{equation}

    \item \textbf{Recall score}

    Recall score is the number of true positives divided by the number of true positives plus the number of false negatives.

    The recall score is calculated by formula:
    \begin{equation}
        Recall = \frac{TP}{TP + FN}
    \end{equation}

    \item \textbf{Precision score}

    Precision evaluation method determines of how precise/accurate machine learning model of how many positives predictions have been predicted of total predictions. Precision score is calculated by formula:
    \begin{equation}
        Precision = \frac{TP}{\mathit{TP} + \mathit{FP}}
    \end{equation}

    Precision is a good measure to determine, when the costs of False Positive is high.

    \item \textbf{F1 score}

    F1 score is the harmonic mean of precision and recall taking both metrics into account. F1 score is calculated by formula:
    \begin{equation}
        F1 = 2 * \frac{Precision * Recall}{Precision + Recall}
    \end{equation}
    \item \textbf{Confusion Matrix}

    A confusion matrix is a technique for summarizing the performance of a classification algorithm. Confusion matrix is a method to better understand the performance of classification models. It is a summary of of prediction results on a classification models: The number of correct and incorrect predictions are summarized with count values and broken down by each class. This is the key to the confusion matrix.

\end{enumerate}


\subsection{Gradient Properties}
\noindent
Let $f$ and $g$ be functions of multiple variables, let $\vec{r}$ be a VVF, and let $c \in \mathbb{R}$.
\begin{enumerate}
	\item $\nabla(f \pm g) = \nabla f \pm \nabla g$
	\item $\nabla(cf) = c\nabla f$
	\item $\nabla(fg) = f\nabla g + g\nabla f$
	\item $\nabla(f\circ\vec{r}(t)) = \nabla f \cdot \vec{r}(t)$
\end{enumerate}


\documentclass{article}
\usepackage{leonine,amsmath,amssymb,amsthm,graphicx}
\setkeys{Gin}{width=\linewidth,totalheight=\textheight,keepaspectratio}
\graphicspath{{graphics/}}
% Prints a trailing space in a smart way.
\usepackage{xspace}
% Inserts a blank page
\newcommand{\blankpage}{\newpage\hbox{}\thispagestyle{empty}\newpage}
% \usepackage{units}
% Typesets the font size, leading, and measure in the form of 10/12x26 pc.
\newcommand{\measure}[3]{#1/#2$\times$\unit[#3]{pc}}

\theoremstyle{definition}
\newtheorem{pred}[thm]{Prediction}

\title{Cerebellum: Mathematical Preliminaries} \author{Eric Purdy}

\begin{document}

\maketitle

\section{Probability Space}

A finite probability space\footnote{It is slightly more complicated to
  define an infinite probability space, so we will assume that our
  probability spaces are finite.} is a finite set $\Omega$ together
with a function $P$ from $\Omega$ to the real numbers $\RR$, such that
\begin{itemize}
\item For all $\omega \in \Omega$, $0 \le P[\omega] \le 1$
\item $\sum_{\omega\in \Omega} P[\omega] = 1$
\end{itemize}

The elements $\omega$ of $\Omega$ are called {\em elementary events}.
An {\em event} is a subset $S$ of $\Omega$, and its probability is the
sum of the probabilities of the elementary events that make it up:
$$P[A] = \sum_{\omega \in A} P[\omega].$$ We say that the event $A$
happens if we pick a sample $\omega$ from $\Omega$ and $\omega \in A$,
and that it does not happen if $\omega \notin A$.

For us, $\Omega$ will generally be the space of all possible
activities of neurons in the brain at a single instant in time. One of
the most common events that we will be interested in is the event that
a particular neuron $s$ fires; there are many possible states of the
brain that include $s$ firing (for example, every single neuron in the
brain firing, or $s$ alone firing and no other neuron in the brain
firing), and these are the elementary events that make up the event
``$s$ fires''.

\section{Random Variables}

A random variable is a function $X$ from a finite probability space
$\Omega$ to some set. For our purposes, the set will always be $\RR$,
the real numbers.

For a random variable $X$, and a real number $a \in \RR$, the event
that $X=a$ is just the subset of those $\omega$ in $\Omega$ such that
$X(\omega) = a$. The probability of this event is written as $P[X=a]$.
If we write just $P[X]$, this should be understood as a function that
maps values of $X$ to probabilities, i.e., $P[X](a) = P[X = a]$.

If $X$ and $Y$ are random variables, then so are $aX$, $X+Y$, $X-Y$,
$X\cdot Y$, and $\frac{X}{Y}$. These are the functions
\begin{align*}
(aX)(\omega) &= a \cdot X(\omega)\\
(X+Y)(\omega) &= X(\omega) + Y(\omega)\\
(X-Y)(\omega) &= X(\omega) - Y(\omega)\\
(X\cdot Y)(\omega) &= X(\omega) \cdot Y(\omega)\\
\left(\frac{X}{Y}\right)(\omega) &= \frac{X(\omega)}{Y(\omega)},\\
\end{align*}
respectively. In this way, we can build up more complicated random
variables from simpler ones.

The {\em expected value} or {\em average value} of a random variable
$X$ is defined to be
$$E[X] = \sum_{\omega \in \Omega} P[\omega] X(\omega).$$

\section{Conditional Probabilities}

We will be working with conditional probabilities. The {\em
  conditional probability of A given B}, written $P(A|B)$, is the
probability that event $A$ happens given that we already know that $B$
has happened. It is defined as the probability that both $A$ and $B$
happen (written $A\cap B$) divided by the probability that $B$
happens.
$$P[A|B] = \frac{P[A \cap B]}{P[B]}$$

Two events $A$ and $B$ are said to be independent if $P[A|B] = P[A]$.

If $X$ and $Y$ are random variables, then the events we are interested
in are the event that they take on a particular value. A conditional
probability $P[X|Y]$, where $X$ and $Y$ are random variables, should
be understood as a function that maps values of $X$ and $Y$ to
probabilities, i.e., $P[X|Y](\alpha, \beta) = P[X=\alpha | Y=\beta]$.

It is always the case that, for every $\beta$, $\sum_\alpha
P[X=\alpha|Y=\beta] = 1$, where the sum is taken over all possible
values of $A$.

Two random variables $X$ and $Y$ are said to be independent if the
events $X=a$ and $Y=b$ are independent for every $a$ and $b$.

\section{Covariance}

The {\em covariance} between two random variables is defined as
$$Cov[X,Y] = E[XY] - E[X]E[Y].$$
The covariance is higher if $X$ and $Y$ tend to be large at the same
time and small at the same time. If it is zero, $X$ and $Y$ are said
to be ``uncorrelated''. Two independent random variables are always
uncorrelated, but two uncorrelated random variables need not be
independent.

\section{Partial Derivatives}

\begin{figure}
\includegraphics[width=\linewidth]{contour3d_demo3.png}
\caption{A function of two variables. The partial derivative is the
  derivative of the curves shown on the left and right walls.}
\label{fig-partial}
\end{figure}

The partial derivative of a function $f(x_1, \dots, x_n)$, denoted by
$\frac{\partial f(x_1, \dots, x_n)}{\partial x_i}$, is the derivative of
the function
$$g(x) = f(a_1, \dots, a_{i-1}, x, a_{i+1}, \dots, a_n),$$
where the $a_j$ are constants. Some functions $g$ are shown for
different values of the $a_j$ in Figure \ref{fig-partial}, for a
function of two variables $f(x_1, x_2)$.

\section{Stochastic Gradient Ascent/Descent}

Let $f$ be a function of $n$ real variables $x_1, \dots, x_n$. We want
to find the values of $x_1, \dots, x_n$ for which $f(x_1, \dots, x_n)$
is largest. For completely general $f$, this is impossible to do
except by a brute force search over the entire space, which is
impossible, since there are infinitely many settings for each
$x_i$. For functions $f$ which are continuous and which have
continuous derivatives, it is still nontrivial, and often the best we
can do is to find a setting of the $x_i$ which is locally highest,
i.e., such that there is no $x_1', \dots, x_n'$ close to $x_1, \dots,
x_n$ such that $f(x_1', \dots, x_n') > f(x_1, \dots, x_n)$.

We can find local maxima of continously differentiable $f$ by
repeatedly taking small steps in the direction that makes the function
grow fastest:
$$\Delta x_i = \eta \cdot \frac{\partial f(x_1, \dots, x_n)}{\partial
  x_i}.$$
The notation $\Delta x_i$ means that we replace $x_i$ by $x_i + \Delta
x_i$.  Here $\frac{\partial f}{\partial x_i}$ is a partial derivative;
it specifies how fast $f$ will grow if we take a small step in the
direction of increasing $x_i$. This procedure is called {\em gradient
  ascent}; if we are trying to minimize a function, we simply go in
the opposite direction:
$$\Delta x_i = -\eta \cdot \frac{\partial f(x_1, \dots, x_n)}{\partial
  x_i}.$$ This is called {\em gradient descent}.

The parameter $\eta$, called the ``learning rate'', controls how fast
we move in the direction of the gradient. Lower rates result in more
accurate gradient computations (since we are updating our direction
more frequently), but also result in gradient ascent taking
longer. Learning rates that are too high can result in oscillation
around the desired solution, as the algorithm keeps overshooting the
best point. Learning rates are generally set empirically, by seeing
what works best in a particular context.

It is often easier to find a random variable whose average value is
the gradient, and use that instead of the true gradient in our
updates. This is called {\em stochastic} gradient ascent or descent,
depending on whether we are trying to maximize $f$ or minimize $f$.
More formally, if we have random variables $Y_i$ ($i=1, \dots, n$)
such that:
$$\frac{\partial f(x_1, \dots, x_n)}{\partial x_i} = E[Y_i] = \sum_a P[Y_i=a]
a,$$ and we have a source of random samples $y_i(1), \dots, y_i(T)$
of $Y_i$, then we can maximize $f$ via the update
$$\Delta x_i = \eta \cdot y_i(t).$$

In general, the partial derivative of $f$ will change as the $x_i$
change, so we will need a new random variable $Y_i$ at each time
step. Fortunately, this is often possible. In general the approach
taken is to use a single sample from each random variable, then update
the $x_i$ according to the update rule above, then pick a new random
variable whose expected value is the new gradient.

It is important to note that there may be many different random
variables whose expectation is equal to the gradient. For instance, if
we have a random variable such that $E[X]$ is equal to a gradient of
interest, then the same is true of the random variable
$$X + a\frac{I[X \ge 0]}{P[X \ge 0]} - a \frac{I[X<0]}{P[X < 0]},$$
where $I[X \ge 0]$ is one if $X \ge 0$ and zero otherwise, and
$I[X<0]$ is one if $X<0$ and zero otherwise. We therefore do not
require that the neurological evidence exactly match the stochastic
gradient descent equations we give.

One random variable that seems especially biologically plausible is
$$T_{X, a} = E[X | X\ge a] I[X \ge a] + E[X | X < a] I[X < a],$$
where
$$E[X | X \ge a] = \sum_{\omega: X(\omega) \ge a}X(\omega)
\frac{P[\omega]}{P[X \ge a]},$$
and similarly for $E[X | X < a]$. The random variable $T_{X,a}$ has
the same expectation as $X$, but has the very simple form of a step
function: it has one constant value when $X<a$ and another constant
value when $X\ge a$.
\end{document}


\section{Graph-based Clustering}

% ===
%\textbf{Metric} relational data:
%relations correspond to Euclidean distances.

\textbf{Non-metric relations:}
might assume negative values or violate the triangular inequality.

% ===
\emph{Setting:}\enspace
objects $\bm o_i, \bm o_j \in \mathcal O$;\:
relations with weights $\mathcal D \coloneqq \{ D_{ij} \}$ on the edges $(i, j)$.
\\
\begin{itemize}
    \item Cluster $\alpha$:\enskip
        $\mathcal G_\alpha \equiv \brace{\bm o\in \mathcal O : c(\bm o) = \alpha}$
    \item Inter-cluster edges:\enskip
        $\mathcal E_{\alpha\beta} = \brace{(i,j) \in \mathcal E : \bm o_i \in \mathcal G_\alpha \land \bm o_j \in \mathcal G_\beta}$
    \item $\mathrm{cut}(A,B) = \sum_{i\in A, j\in B} W_{ij}$\enskip
        {\small $\to$ weight matrix $W$}
    \item $\mathrm{assoc}(A,\mathcal V) = \sum_{i\in A, j\in \mathcal V} W_{ij}$
        \enskip{\small $\to$ total connection strength from nodes in $A$ to all nodes in the graph}
\end{itemize}

\emph{Correlation clustering:}\\
%Maximise agreement within cluster and disagr. between clusters.
Minimise the sum of \textit{pairwise} intracluster distances.

\begin{tabular}{@{} l @{} l @{}}
    $R\ap{cc} (c; \mathcal D)$ &
        $= - \sum\limits_{\nu\leq k} \: \sum\limits_{(i,j) \in \mathcal E_{\nu\nu}} S_{ij} + \sum\limits_{\nu\leq k} \: \sum\limits_{\substack{\mu\leq k \\ \color{red}\mu\neq\nu}} \: \sum\limits_{(i,j) \in \mathcal E_{\nu\mu}} S_{ij}$
    \\ &
        $= - \textcolor{red}{2}\sum\limits_{\nu\leq k} \: \sum\limits_{(i,j)\in \mathcal E_{\nu\nu}} S_{ij} + \cancel{ \sum\limits_{(i,j)} S_{ij} }$
    \\ &
        \quad $\hookrightarrow$ intra-cluster
        \quad $\hookrightarrow$ const
    \\
    \multirow{2}{*}{\small \shortstack{up to \\ thresh. $u$}} &
        $\overset{\ast}{=}- \frac12 \sum\limits_{\nu\leq k} \: \sum\limits_{(i,j) \in \mathcal E_{\nu\nu}} (\abs{S_{ij} {-} u} + S_{ij} {-} u)$
    \\ &
        $\phantom{=} + \frac12 \sum\limits_{\nu\leq k} \: \sum\limits_{\substack{\mu\leq k \\ \color{red}\mu\neq\nu}} \: \sum\limits_{(i,j) \in \mathcal E_{\nu\mu}} (\abs{S_{ij} {+} u} - S_{ij} {-} u)$
    \\ \multicolumn{2}{@{\quad}l}{
        \small $\ast :$ altern. def. where $\frac12 (\abs{X} \pm X) = \max \{ 0, \pm X \}$
    }
\end{tabular}

\columnbreak
\emph{Graph partitioning:}
\quad $D_{ij} \in \mathbb R$\\
\begin{tabular}{@{} l @{} l @{}}
    $R\ap{gp} (c; \mathcal D)$ &
        $= \mathit{const} - \sum_{\nu\leq k} \mathrm{cut} \paren{ \mathcal G_\nu (\mathcal D), \mathcal V \setminus \mathcal G_\nu (\mathcal D) }$
    \\ &
        $= \mathit{const} + \sum_{\nu\leq k} \mathrm{cut} \paren{ \mathcal G_\nu (\mathcal S), \mathcal V \setminus \mathcal G_\nu (\mathcal S) }$
\end{tabular}

\emph{Bias in \textit{\rmfamily R(c;D)}:}\enspace
Cost should scale prop. to \#objects,\\
i.e. $R(c;D) = \mathcal O(n)$.
\qquad $\ast: \text{ use } D_{ij} = D(1 - \delta_{ij})$

\textbf{Tipp:}\enspace
$\frac{\mathrm{cut}(\mathcal G_\alpha, \mathcal V \backslash \mathcal G_\alpha)}{\mathrm{assoc}(\mathcal G_\alpha, \mathcal V)}
\overset\ast= \frac{n\cdot p_\alpha \cdot n(1-p_\alpha) \cdot D}{n\cdot p_\alpha \cdot n \cdot D} = 1 - p_\alpha$

% ===
\subsection{Pairwise Clustering}

\emph{Cost:}\enspace
$R\ap{pc} (c; \mathcal D)
= \sum\limits_\alpha \sum\limits_{(i,j) \in \mathcal E_{\alpha\alpha}} \frac{D_{ij}}{\abs{\mathcal G_\alpha}}
\color{gray}
= \sum\limits_\alpha \sum\limits_{(i,j) \in \mathcal E_{\alpha\alpha}} \abs{\mathcal G_\alpha} \frac{D_{ij}}{\abs{\mathcal E_{\alpha\alpha}}}$

\emph{Equivariance to \textit{\rmfamily k}-means:}
{\small\color{gray}\quad(if $\color{gray} D_{ij} = \norm{\bm x_i - \bm x_j}^2$)}\\
$\sum\limits_{i\leq n} \norm{\bm x_i - \bm y_{c(i)}}^2 = \sum\limits_{i\leq n} \sum\limits_{j\leq n} \sum\limits_{\alpha\leq k} \frac{\mathbb I_{\brace{c(i) = \alpha}} \mathbb I_{\brace{c(j) = \alpha}}}{\abs{\mathcal G_\alpha}} D_{ij}$

\emph{Invariance properties:}\\
\begin{itemize}
    \item Symmetrisation:
        \enskip $R\ap{pc} (c; \mathcal D\ap{s}) \equiv R\ap{pc} (c; \mathcal D)$
    \item Off-diagonal shift:
        \enskip $R\ap{pc} (c; \tilde{\mathcal D}) = R\ap{pc} (c; \mathcal D) - \lambda\ped{min}\cdot n$
\end{itemize}

\emph{Theorem:}\enspace
If $S\ap{c}$ is p.s.d., then $D$ derives from squared Eucl. space.
$\implies$ Make $S$ \textbf{p.s.d.}: \enskip $\tilde S \coloneqq S - \lambda\ped{min} \mathbb I$

\emph{Constant Shift Embedding:}\\
\begin{enumerate}
    \item \textbf{Symmetrise} $D \to D\ap{s}$:
        \enskip $\highlight*{D_{ij}\ap{s}} \coloneqq \frac12 (D_{ij} + D_{ji})$
    \item \textbf{Centralise} $D$, then $S$:
        \enskip $X\ap{c} \coloneqq Q X\ap{s} Q^\top$\\
        $Q = \mathbb I - \frac1n \bm e_n \bm e_n^\top$
        \qquad \highlight*{$S\ap{c} = -\frac12 D\ap{c}$}\\
        $X\ap{c}_{ij} = X_{ij} - \frac1n \sum_k X_{ik} - \frac1n \sum_k X_{kj} + \frac{1}{n^2} \sum_{k,\ell} X_{k\ell}$\\
        $\implies$ sum over column/rows = 0
    \item \textbf{(Off-)Diagonal shift}:
        \enskip Find $\lambda\ped{min}$ of $S\ap{c}$\\
        $\highlight*{\tilde S} \coloneqq S\ap{c} - \lambda\ped{min} \mathbb I$
        \qquad $\tilde D \coloneqq D - \lambda\ped{min} (\bm 1 - \mathbb I)$\\
        $\highlight*{\tilde D_{ij}} = \tilde S_{ii} + \tilde S_{jj} - 2 \tilde S_{ij} \color{gray} = \norm{\bm x_i - \bm x_j}^2$
\end{enumerate}

\emph{Reconstruction:}\\
\begin{enumerate}
    \item EVD:
        \enskip \highlight*{$\tilde S = \bm V \Lambda \bm V^\top$}
        \enskip via \enskip $(\tilde S - \lambda \mathbb I) \bm v \overset!= 0$
        \enskip $\color{gray} (\abs{\bm v} = 1)$\\
        where \: $\Lambda {=} \mathrm{diag}(\lambda_1 \ldots \lambda_n)$
        \: and \: $\bm V {=} [\bm v_1 \ldots \bm v_n]$
    \item Find $p$ s.t. $\lambda_1 \geq \ldots \lambda_p > \lambda_{p+1} = \ldots = \lambda_n = 0$
    \item $\implies \highlight*{\bm X_p} = \bm V_p (\Lambda_p)^{1/2}$
        \enskip (each row is a vector)
    \item $\implies \bm X_t = \bm V_t (\Lambda_t)^{1/2}$
        \enskip (approx. \& denoising)
\end{enumerate}

\columnbreak
\emph{Cluster membership of new data:}\\
\textit{Note:}\enspace $S\ap{new}$ is def. by\enspace $D_{ij}\ap{new} = S_{ii}\ap{new} + \tilde S_{jj} - 2 S_{ij}\ap{new}$
\begin{enumerate}
    \item $\begin{aligned}[t]
        (S\ap{new})\ap{c} = -\tfrac12 \big[ &
            D\ap{new} (\mathbb I_n - \tfrac1n \bm e_n \bm e_n^\top)
        \\[-3pt] &
            - \tfrac1n \bm e_m \bm e_n^\top + \tilde D (\mathbb I_n - \tfrac1n \bm e_n \bm e_n^\top) \big]
    \end{aligned}$
    \item Project:
        \enspace $X_p\ap{new} = (S\ap{new})\ap{c} \bm V_p (\Lambda_p)^{-1/2}$
    \item Assign:
        \enskip $\hat c_i = \arg\min_c \norm*{(x_p\ap{new})_i - y_{c(i)}}$
\end{enumerate}

% ===
\iffalse
    \subsection{Alternative Costs}
    \todo{}
\fi

% ===


\section*{Web Appendix C}

As mentioned in Section 2, any \textit{p}-hacking model can be written on the form of a selection model. Observe that
\begin{eqnarray*}
\int_{[0,1]}\phi_\alpha^{\star}(x_{i}\mid\theta_{i},\sigma_{i}^2)d\omega(\alpha) & = & \int_{[0,1]}\phi(x_{i}\mid\theta_{i}, \sigma_i^2)P(u_i\in\left[0,\alpha\right]\mid\theta_{i},\sigma^2_{i})^{-1}d\omega(\alpha)\\
 & = & \phi(x_{i}\mid\theta_{i}, \sigma^2_i)\int_{[0,u_i)}P(u_i\in\left[0,\alpha\right]\mid\theta_{i},\sigma_{i}^2)^{-1}d\omega(\alpha).
\end{eqnarray*}
where $\phi_\alpha^{\star}$ is a Gaussian density truncated so that the \textit{p}-value associated to $x_i$, $u_i$, lies in the interval $\left[0,\alpha\right)$. This is a publication bias model if $$h(u_i)=\int_{[0,u_i]}P(u_i\in\left[0,\alpha\right]\mid\theta_{i},\sigma^2_{i})^{-1}d\omega(\alpha)$$ is bounded for each $u_i$ and $h(u_i)$ is independent of $\theta_{i},\sigma^2_{i}$. While $h(u_i)$ can be bounded, it is typically dependent of $\theta_{i},\sigma^2_{i}$, with the fixed effect model under complete selection for significance being a notable exception.

On the other hand, any selection model with
$$
I = \int \phi(x;\theta_{i},\sigma^2_{i})w(u_i)du_i<\infty
$$
can be written as a mixture model. For then there is a finite measure $d\omega(\alpha)$ satisfying
\[
w(u_i)=\int_{[0,u_i)}\frac{1}{P(u_i\in\left[0,\alpha\right)\mid\theta_i,\sigma^2_{i})}d\omega(\alpha)
\]
Just take $d\omega(\alpha)=d w(\alpha)P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\sigma^2_{i})$, where $d\rho(\alpha)$ is defined by $\int_{0}^{u_i}d w(\alpha)= w(u_i)$. The size of the measure is
\begin{eqnarray*}
\int_{0}^{1}d\omega(\alpha;\theta_{i},\sigma^2_{i}) & = & \int_{0}^{1}P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\sigma^2_{i})d w(\alpha)\\
 & = & \int_{0}^{1}\phi(u_i;\theta_{i},\sigma^2_{i})\int_{0}^{u_i}d w(\alpha)du_i\\
 & = & I
\end{eqnarray*}
Hence $I d\omega'(\alpha;\theta_{i},\sigma^2_{i})$ is a probability measure. This probability measure makes
\[
I^{-1}\phi(x_{i};\theta_{i},\sigma^2_{i})w(u_i)=\int_{[0,1]}\phi_\alpha(x_{i};\theta_{i},\sigma^2_{i})d\omega'(\alpha)
\]
as can be seen by the following computation,
\begin{eqnarray*}
I^{-1}\phi(x_{i};\theta_{i},\sigma^2_{i})w(u_i) & = & I^{-1}\int_{[0,u_i)}\frac{\phi(x_{i};\theta_{i},\sigma^2_{i})}{P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\sigma^2_{i})}d\omega(\alpha)\\
 & = & I^{-1}\int_{[0,1]}\frac{\phi(x_{i};\theta,\sigma^2_{i})1_{\left[0,\alpha\right)}(u_i)}{P(u_i\in\left[0,\alpha\right)\mid\theta_{i},\sigma^2_{i})}d\omega(\alpha)\\
 & = & I^{-1}\int_{[0,1]}\phi_\alpha(x_{i};\theta_{i},\sigma^2_{i})d\omega(\alpha)\\
 & = & \int_{[0,1]}\phi_\alpha(x;\theta_{i},\sigma^2_{i})d\omega'(\alpha)
\end{eqnarray*}

Proposition 1 shows the form of the one-sided normal step function selection probability publication bias model when it is written as a mixture model of the form (5). But most such mixture models are not true \textit{p}-hacking models, as the mixing probabilities $\pi_{i}^{\star}$ depend on $\theta$. There is no way for the \textit{p}-hacker to know
$\theta$, so we cannot regard the publication bias model as a \textit{p}-hacking model.


\documentclass{article}
\usepackage[hmargin=1in,vmargin=1.5in]{geometry}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{bm}
\renewcommand{\a}{\bm a}
\renewcommand{\c}{\bm c}
\newcommand{\p}{\bm p}
\newcommand{\q}{\bm q}
\newcommand{\1}{\bm 1}
\title{Homework 1}
\author{Xinyi Gu, Songchen Tan}
\date{\today}
\begin{document}
\maketitle
\section{}
\subsection*{(a)}

To deal with the absolute value in the objective function, We let vector $\a=\{a_0,\cdots,a_{T-1}\}=\p-\q$, where $p_t=\max(a_t,0)$ and $q_t=-\min(a_t,0)$; it is easy to verify that $a_t=p_t-q_t$ and $|a_t|=p_t+q_t$. Furthermore we denote $m=\max_{t\in\{0,\cdots,T-1\}}|a_t|$. Let $\c=(c_0,\cdots,c_{T-1})$ and $\1=(1,\cdots,1)$. The linear programming formulation is

minimize $m$, subject to

$$
\begin{cases}
    m >= (p_t + q_t), \quad 0\le t\le T-1\\
    \displaystyle
    \sum_{t=0}^{T-1}(p_t - q_t)=0\\
    \displaystyle
    \sum_{t=0}^{T-1}(T-t-1)(p_t - q_t)=d\\
    -\delta \le (p_{t-1} - q_{t-1})-(p_t - q_t) \le\delta, \quad 1\le t\le T-1\\
    \displaystyle
    \sum_{t=0}^{T-1}c_t(p_t+q_t)\le f
\end{cases}
$$

\subsection*{(b)}

The optimization gives $m\approx 0.0213$, and the $a, x, v$ are shown below.

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{a.png}
        \caption{$a$}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{v.png}
        \caption{$v$}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{x.png}
        \caption{$x$}
    \end{subfigure}
       \caption{$a, x, v$ during the motion}
\end{figure}

\section{}

\subsection*{(a)}

We have

\begin{align*}
    \min \quad & c_1 x_1 + c_2 | x_2 -10| \\
    \text{subject to}\quad & c_3 | x_1 + 2 | + c_4 | x_2 | \le 5
\end{align*}

Moving the objective into the constraints and making constraints with absolute valunes linear, we get

\begin{align*}
    \min \quad & t\\
    \text{subject to}\quad  z_1  & \geq  x_1 +2, \\
    z_1  & \geq -(x_1 + 2) \\
    z_2  & \geq  x_2 ,\\
    z_2  & \geq  -x_2\\
    c_3 z_1 + c_4 z_2 & \le 5, \\
    z_3  & \geq x_2 - 10 ,\\
    z_3  & \geq -(x_2 - 10)\\
    c_1 x_1 + c_2 z_3 & \le  t
\end{align*}

To formulate a linear programming problem, $c_1$ free, $c_2, c_3, c_4 \geq 0 $.

\subsection*{(b)}

\begin{align*}
    f(x) & = \max{\{-x+1, \quad 0, \quad 2x - 4\}}\\
    c^{'}x + f(d^{'}x) & = c^{'}x + \max{\{-(d^{'}x) + 1, \quad 0, \quad 2(d^{'}x) - 4\}}
\end{align*}

Write max$\{\cdot\}$ as z and 3 constraints:

\[ z \geq - d^{'}x + 1, \quad z \geq 0, \quad z \geq 2d^{'}x - 4 \]

We get

\begin{align*}
    \min \quad & c^{'}x + z\\
    \text{subject to} \quad z & \geq - d^{'}x + 1,\\
    z & \geq 0,\\
    z & \geq 2d^{'}x - 4, \\
    Ax & \geq b
\end{align*}

\section{}

When $m\ge n$ this is trivial, so we consider $m < n$ only. For any $y\in\operatorname{ran}_+(A)$, consider the polyhedron

$$
C=\left\{x \in \mathbb{R}^{n} \mid y=A x, x \geq 0\right\}
$$

Assuming the rank of $A$ is $r\le m<n$, we can always select $r$ linearly independent rows such that, by row operations, they can eliminate other rows (including the corresponding elements in $y$). Therefore it suffices to prove the case $r=m$.

According to the corollary in class, every nonempty polyhedron in standard form has a BFS. $C$ is certainly nonempty since $y\in\textrm{ran}_+(A)$, so there exists a group of indices $B(1), \cdots, B(m)$ such that the matrix $B=(A_{B(1)}, \cdots, A_{B(m)})$ satisfies $B^{-1}y\ge 0$. This group of indices are the indices of coefficients required by the problem.

\section{}

\subsection*{(a)}

True.

$n = m+1$ means that $P$ lives in an 1-D subspace, therefore every point in $P$ can be written as $x_0 + \lambda d$. This subspace is either a segment or a ray, so it can't have more than 2 extreme points.

\subsection*{(b)}
False.

Counter example:
minimize $x_1$, subject to $x_1=1$ and $x_1,x_2\ge0$, optimal solution set is $\{(x_1,x_2)|x_1=1,x_2\geq0\}$ which is unbounded.

\subsection*{(c)}
False.

Counter example: minimize $x_1+x_2$, subject to $x_1 +x_2= 1$, $x_1, x_2 \geq 0$. At a optimal solution $(0.5, 0.5)$, 2 (more than 1) variables are positive.

\subsection*{(d)}
True.

Convex combination of any 2 (or more) solutions can be optimal.

\subsection*{(e)}
False.

Same example as (b).

\subsection*{(f)}
False.

Counter example: minimize $\max\{x_1-x_2,x_2-x_1\}$ subject to $x_1+x_2=1,x_1,x_2\ge0$. The optimal solution is $(0.5, 0.5)$.

\section{}

\subsection*{(a)}

We noticed that $\bar c_2<0$, therefore $\beta=0$ (or else it can be further optimized). We choose $(\alpha,\beta,\gamma,\delta,\eta)=(0, 0, 0, 0, 0)$.

\subsection*{(b)}

We need $d_j>0,\forall j\in N$, so we choose $j=1$ and $\alpha,\gamma,\delta <0$. We choose $(\alpha,\beta,\gamma,\delta,\eta)=(-1, 0, -1, -1, 0)$.

\subsection*{(c)}

We choose $j=2$ to be optimized, so we require $\beta>0$. We choose $(\alpha,\beta,\gamma,\delta,\eta)=(0, 3, 0, 1, 0)$.

\section{}

\subsection*{(a)}

Standard form:

\begin{align*}
    \min \quad  -2x_1 - x_2 & \\
    \text{subject to} \quad x_1 - x_2 + s_1 & = 2 \\
    x_1 + x_2 + s_2 & = 6 \\
    x_1, x_2, s_1, s_2 & \geq 0
\end{align*}

\[\begin{bmatrix}
    1 & -1 & 1 & 0 \\
    1 & 1 & 0 & 1
\end{bmatrix} \begin{bmatrix}
    x_1 & x_2 & s_1 & s_2
\end{bmatrix}^{T} = \begin{bmatrix}
    2 \\
    6
\end{bmatrix}
\]
Want ($x_1$, $x_2$) = (0, 0), pick $s_1$, $s_2$ as basis,
B $ = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $,
BFS: $x^T = B ^{-1} b = $ (0, 0, 2, 6).

\subsection*{(b)}

Full tableau implementation:

start with BFS: x = (0, 0, 2, 6)'
B = [\boldsymbol{$A_3$}, \boldsymbol{$A_4$}]

\begin{tabular}{l|l|l|l|l|l|}
\cline{2-6}
&  & $x_1$ & $x_2$ & $s_1$ & $s_2$ \\ \cline{2-6}
& 0 & -2 & -1 & 0 & 0 \\ \cline{2-6}
 $s_1$ = & 2 & 1 & -1 & 1 & 0 \\ \cline{2-6}
 $s_2$ = & 6 & 1 & 1 & 0 & 1\\ \cline{2-6}
\end{tabular}
\\

$x_1$ and $x_2$ have negative reduced costs, pick $x_1$ to enter basis

\begin{tabular}{l|l|l|l|l|l|}
\cline{2-6}
&  & $x_1$ & $x_2$ & $s_1$ & $s_2$ \\ \cline{2-6}
& 1 & 0 & 1 & 0 & 2 \\ \cline{2-6}
 $x_1$ = & 6 & 1 & 1 & 0 & 1 \\ \cline{2-6}
 $s_1$ = & -4 & 0 & -2 & 1 & -1\\ \cline{2-6}
\end{tabular}
\\

$s_1 < $  0 here, infeasible solution, pick $s_2$

\begin{tabular}{l|l|l|l|l|l|}
\cline{2-6}
&  & $x_1$ & $x_2$ & $s_1$ & $s_2$ \\ \cline{2-6}
& -3 & 0 & -3 & 2 & 0 \\ \cline{2-6}
 $x_1$ = & 2 & 1 & -1 & 1 & 0 \\ \cline{2-6}
 $s_2$ = & 4 & 0 & 2 & -1 & 1\\ \cline{2-6}
\end{tabular}
\\
Then we have reduced costs of $x_2$ to be negative, pick $x_2$ to enter basis,

\begin{tabular}{l|l|l|l|l|l|}
\cline{2-6}
&  & $x_1$ & $x_2$ & $s_1$ & $s_2$ \\ \cline{2-6}
& 10 & 0 & 0 & 0.5 & 1.5 \\ \cline{2-6}
 $x_1$ = & 4 & 1 & 0 & 0.5 & 0.5 \\ \cline{2-6}
 $x_2$ = & 2 & 0 & -1 & -0.5 & 0.5\\ \cline{2-6}
\end{tabular}
\\
All the reduced costs in this tableau are nonnegative, representing an optimal solution at $x_1$ = 4 and $x_2$ = 2, $s_1$ = 0, $s_1$ = 0. In this solution, min $-2x_1 -x_2$ = $c^{'}_B x_B$ = -10

\end{document}

\\


\chapter{Categories of filters}

In~\cite{filt-cat} two categories, whose objects are related with filters on sets, are defined and researched.

Accordingly~\cite{filt-cat} infinite product is defined just in the first (denoted $\mathscr{F}$ there) of these two categories.
So we will for now consider the first category. (Usefulness of the second category for our research is questionable.)

Let $f:A\rightarrow B$ be a function, $\mathcal{A}$ be a filter on~$A$.

\begin{prop}
$\setcond{Y\in\subsets B}{\rsupfun{f^{-1}}Y\in\mathcal{A}}$ is a filter.
\end{prop}

\begin{proof}
That it is an upper set is obvious.

Let $Y_0,Y_1\in\setcond{Y\in\subsets B}{\rsupfun{f^{-1}}Y\in\mathcal{A}}$. Then
$\rsupfun{f^{-1}}Y_0\in\mathcal{A}$ and $\rsupfun{f^{-1}}Y_1\in\mathcal{A}$.
We have
\[ \rsupfun{f^{-1}}(Y_0\cap Y_1) = \rsupfun{f^{-1}}Y_0 \cap \rsupfun{f^{-1}}Y_1 \in \mathcal{A} \]
since $f$ is monovalued.
Thus $Y_0 \cap Y_1\in\setcond{Y\in\subsets B}{\rsupfun{f^{-1}}Y\in\mathcal{A}}$.
\end{proof}

\begin{thm}
\fxwarning{Should be moved above in the book.}
$\setcond{Y\in\subsets B}{\rsupfun{f^{-1}}Y\in\mathcal{A}}$ is equal to the filter generated
by the filter base $\rsupfun{\rsupfun{f}}\mathcal{A}$, for every filter~$\mathcal{A}$.
\end{thm}

\begin{proof}
Denote $\mathcal{B} = \setcond{Y\in\subsets B}{\rsupfun{f^{-1}}Y\in\mathcal{A}}$,
$\mathcal{C} = \rsupfun{\rsupfun{f}}\mathcal{A}$.

Let $Y\in\mathcal{C}$. Then $Y=\rsupfun{f}A$ where $A\in\mathcal{A}$.
Then $\rsupfun{f^{-1}}\rsupfun{f}A\supseteq A$ and so $\rsupfun{f^{-1}}\rsupfun{f}A\in\mathcal{A}$.
This proves $\rsupfun{f}A\in\mathcal{B}$, that is $Y\in\mathcal{B}$.

Let now $Y\in\mathcal{B}$. Then $\rsupfun{f}\rsupfun{f^{-1}}Y\subseteq Y$. Since $\rsupfun{f^{-1}}Y\in\mathcal{A}$,
we have that $Y$ is a supset of some set of the form $\rsupfun{f}A$, so $Y\in\mathcal{C}$.
\end{proof}

\begin{cor}
$\up\supfun{f}\mathcal{A} = \setcond{Y\in\subsets B}{\rsupfun{f^{-1}}Y\in\up\mathcal{A}}$.
\end{cor}

\begin{defn}
The \emph{category of filtered sets} $\mathbf{Filt}$ is the category defined as follows:
\begin{enumerate}
\item Objects are pairs $(A,\mathcal{A})$ where $A$ is a (small) set and $\mathcal{A}$ is a filter on~$A$.
\item Morphisms from $(A,\mathcal{A})$ to $(B,\mathcal{B})$ are functions $f:A\rightarrow B$ such that
$\supfun{f}\mathcal{A} \sqsubseteq \mathcal{B}$.
\item Identities are identity functions.
\end{enumerate}
\end{defn}

To verify that it is a category is straightforward.

It is the same category as $\mathscr{F}$ in \cite{filt-cat}, as follows from an above proposition.

We will prove that starred reloidal product is a categorical product in this category.
First we will prove the special case that binary reloidal product is a categorical product in this category.

\begin{thm}
  $\times^{\mathsf{RLD}}$ (together with projections $\Pr_0$ and
  $\Pr_1$) is a categorical product in $\mathbf{Filt}$.
\end{thm}

\begin{proof}
  Let our objects be $\mathcal{A}$, $\mathcal{B}$.

  Denote $p$ the left projection from $\Base (\mathcal{A}) \times
  \Base (\mathcal{B})$ to $\Base (\mathcal{A})$.

  We need to check that $p$ is a $\mathbf{Filt}$-morphism that is $p
  (\mathcal{A} \times^{\mathsf{RLD}} \mathcal{B}) \sqsubseteq
  \mathcal{A}$ what is obvious.

  Similarly for the right projection $q$.

  It remains to check the universal property: Let $\mathcal{C}$ be a filter
  and $f : \mathcal{C} \rightarrow \mathcal{A}$, $g : \mathcal{C} \rightarrow
  \mathcal{B}$. We need to prove that there are a unique $u : \mathcal{C}
  \rightarrow \mathcal{A} \times^{\mathsf{RLD}} \mathcal{B}$ such that
  $f = p \circ u$ and $g = q \circ u$. Denote $h (z) = (f (z) , g (z))$.

  $h$ is the unique function $\Base (\mathcal{C}) \rightarrow
  \Base (\mathcal{A}) \times \Base (\mathcal{B})$ such that $f = p
  \circ h$ and $g = q \circ h$, so it remains to check that $h$ is a morphism
  of $\mathbf{Filt}$ that is $\langle h \rangle \mathcal{C}
  \sqsubseteq \mathcal{A} \times^{\mathsf{RLD}} \mathcal{B}$, what
  obviously follows from $\supfun{f} \mathcal{C} \sqsubseteq
  \mathcal{A}$ and $\supfun{g} \mathcal{C} \sqsubseteq \mathcal{B}$.
\end{proof}

\begin{thm}
  $\prod^{\mathsf{RLD} \ast}$ together with projections $\Pr_k$ is a
  categorical product in $\mathbf{Filt}$.
\end{thm}

\begin{proof}
  Consider an indexed family $\mathcal{A}$ of objects.

  Denote $p_k$ the $k$-th projection from $\prod_{i \in \dom
  \mathcal{A}} \Base (\mathcal{A}_i)$.

  We need to check that $p_k$ s a $\mathbf{Filt}$-morphism that is
  $p_k \left( \prod^{\mathsf{RLD} \ast} \mathcal{A} \right) \sqsubseteq
  \mathcal{A}_k$ what is obvious.

  It remains to check the universal property: Let $\mathcal{C}$ be a filter
  and $f_k : \mathcal{C} \rightarrow \mathcal{A}_k$. We need to prove that
  there are a unique $u : \mathcal{C} \rightarrow \prod^{\mathsf{RLD}
  \ast} \mathcal{A}$ such that $f_k = p_k \circ u$. Denote $h (z) = \lambda i
  \in \dom \mathcal{A} : f_i z$.

  $h$ is the unique function $\Base (\mathcal{C}) \rightarrow \prod_{i
  \in \dom \mathcal{A}} \Base (\mathcal{A}_i)$ such that $f_k =
  p_k \circ h$, so it remains to check that $h$ is a morphism of
  $\mathbf{Filt}$ that is $\langle h \rangle \mathcal{C} \sqsubseteq
  \prod^{\mathsf{RLD} \ast} \mathcal{A}$. It follows from
  \[ \Pr^{\mathsf{RLD}}_i \langle h \rangle \mathcal{C} = \bigsqcap
     \rsupfun{Pr_i} \langle h \rangle^{\ast} \up
     \mathcal{C} = \bigsqcap \langle \Pr_i \circ h \rangle^{\ast} \up
     \mathcal{C} = \bigsqcap \langle f_i \rangle^{\ast} \up \mathcal{C}
     = \langle f_i \rangle \mathcal{C} \sqsubseteq \mathcal{A}_i . \]
\end{proof}


\section{Image segmentation}
Identify groups of pixels that belong together - goal: Delineate objects and background regions \& group similar-looking pixels for efficientcy of further processing

Examples of Groupin in vision
\begin{itemize}
	\item similar appearance
	\item symmetry
	\item common fate (direction etc)
	\item proximity
\end{itemize}

\subsection{Segmentation as clustering}
\subsubsection{Clustering}
Chicken and Egg problem  - either know the centers and allocate the points to it OR know the point membership and create center from them

Feature space - depending on that choice we can group pixels in different ways

\begin{itemize}
	\item pixel intensity
	\item pixel color
	\item pixel texture
	\item pixel intensity + position (proximity)
\end{itemize}

\subsubsection{k-Means}
Idea: randomly initialize the k cluster centers, and iterate between the two steps from clustering

Properties: will always converge to some solution (which can be a local maximum)

Pro:
\begin{itemize}
	\item simple, fast to compute
	\item converges (to local maximum)
\end{itemize}

Cons:
\begin{itemize}
	\item number of cluster has to be defined
	\item sensitive to initial centers
	\item sensitive to outliers
	\item detects spherical clusters only
	\item assuming means can be computed
\end{itemize}
\subsubsection{Mixture of Gaussians, EM}
Idea: instead of treating the data as a bunch of points, assume that they are all generated by sampling a continous function - this function is called a \textit{generative model}, defined by a vector of parameters $\theta$

\includegraphics[width=\columnwidth]{pictures/EM}

E-Step: Compute probability that point $x$ is in blob b, given current guess of $\theta$

$$ P(b|x,\mu_b, V_b) = \frac{\alpha_b P(x|\mu_b, V_b)}{\sum_{i=1}^{K}\alpha_i P(x|\mu_i, V_i)} $$

M-Step: Compute overall probability that blob b is selected (N: data points)\\

Weight:
$$\alpha_b^{new} = \frac{1}{N} \sum_{i=1}^{N} P(b|x_i, \mu_b, V_b) $$
Mean of blob b:
$$ \mu_b^{new} = \frac{\sum_{i=1}^{N} x_i P(b|x_i, \mu_b, V_b)}{\sum_{i=1}^{N} P(b|x_i, \mu_b, V_b)}$$
Covariance of blob b:
$$ V_b^{new} = \frac{\sum_{i=1}^{N} (x_i-\mu_b^{new})(x_i-\mu_b^{new})^T P(b|x_i, \mu_b, V_b)}{\sum_{i=1}^{N} x_i P(b|x_i, \mu_b, V_b)} $$

EM Application is useful for all sorts of problems
\begin{itemize}
	\item Any clustering problem
	\item Many model estimation problems
	\item Missing data problems
	\item Finding outliers
	\item Segmentation problems
	\begin{itemize}
		\item based on color
		\item based on motion
		\item Foreground/background separation
\end{itemize}
\end{itemize}
Pro:
\begin{itemize}
	\item Probabilistic interpretation
	\item Soft assignments between data points and clusters
	\item Generative model, can predict novel data points
	\item Relatively compact storage ($O(Kd^2$)
\end{itemize}
Cons:
\begin{itemize}
	\item Initialization – often a good idea to start from output of k-means
	\item Local minima
	\item Need to know number of components K – Solutions: model selection (AIC, BIC), Dirichlet process mixture
	\item  Need to choose blob generative model (math form of a cluster?)
	\item Numerical problems are often a nuisance
\end{itemize}

\subsubsection{Mean-Shift Algorithm}
Choose features (color, gradients, texture, etc)
\begin{enumerate}
	\item Initialize random seed center and window size $W$
	\item Calculate center of gravity (the “mean”) of $W$: $ \sum_{x \in W} x H(x) $
	\item Shift the search window to the mean
	\item Repeat steps 2+3 until convergence for all centers
	\item merge windows that end up near the same peak/center
\end{enumerate}

\includegraphics[width=\columnwidth]{pictures/meanshift}

Pro:
\begin{itemize}
	\item General, application-independent tool
	\item Model-free, does not assume any prior shape (spherical, elliptical, etc.) on data clusters
	\item Just a single parameter (window size h) – h has a physical meaning (unlike k-means) == scale of clustering
	\item Finds variable number of modes given the same h
	\item Robust to outliers
\end{itemize}
Cons:
\begin{itemize}
	\item Output depends on window size h
	\item Window size (bandwidth) selection is not trivial
	\item Computationally rather expensive
	\item Does not scale well with dimension of feature space
\end{itemize}

\subsection{Hough transform}
see chapter before

\subsection{Interactive Segmentation with GraphCuts}
Markov Random Fields

\includegraphics[width=\columnwidth]{pictures/markovfields}

Field Joint Probability

\includegraphics[width=0.7\columnwidth]{pictures/fieldjointprobability}

\includegraphics[width=\columnwidth]{pictures/energyfunction}
Unary potentials $\Phi$
\begin{itemize}
	\item Encode local information about the given pixel/patch
	\item How likely is a pixel/patch to be in a certain state (e.g. foreground/background)
\end{itemize}
Pairwise potentials $\Psi$
\begin{itemize}
	\item Ecode neighborhood information
	\item How different is a pixel/patch's label from that of its neighbor (e.g. here independent of image data, but later based on intensity/color/texture difference)
\end{itemize}

Goal: minimum cost cut

\includegraphics[width=\columnwidth]{pictures/s-t-min-cut}

Maxflow Algorithm
\begin{enumerate}
	\item Find path from source to sink with positive ! capacity
	\item push maximum possible flow through this path (subtract it from all segments)
	\item repeat until no path can be found
\end{enumerate}

Pro:
\begin{itemize}
	\item Powerful technique, based on probabilistic model (MRF).
	\item Applicable for a wide range of problems.
	\item Very efficient algorithms available for vision problems.
	\item Becoming a de-facto standard for many segmentation tasks.
\end{itemize}
Cons:
\begin{itemize}
	\item Graph cuts can solve a limited (but useful) class of models
	\item Submodular energy functions
	\item Can capture only part of the expressiveness of MRFs
	\item Only approximate algorithms available for multi-label case (except for special cases with particular topology and/or pairwise potentials)
\end{itemize}

\subsection{Learning-based approaches}
Extract the same features used during training \& Use the mapping learned before to label

\subsubsection{K-nearest neighbor}
For every point find the k nearest neighbors (labeled in the training set). The point is then defined through the lables of these neighbors (majority voting of these k neighbors)

Pro:
\begin{itemize}
	\item Very simple to implement
	\item Very simple to understand
	\item efficient implementations possible for approx. NNs
	\item distance definition is flexible
\end{itemize}
Cons:
\begin{itemize}
	\item highly depends on the definitions and k
	\item need to keep the entire data in memory for distance computations
	\item for high dimensional problems might need many training samples for accuracy
	\item other methods have better generalization ability
\end{itemize}

\subsubsection{Random forests}
Binary decision trees - Random Forests are slow to train, but fast to apply during testing.

\includegraphics[width=\columnwidth]{pictures/tree}
TRaining: During training, samples are used for which the true class is known.
Having arrived at a node, one randomly generates a set of possible tests to carry out in order to split it up.
Among the possibilities one selects the test that maximally reduces the uncertainty. (biggest difference)\\
\\
Testing: Feed the current sample into every tree. See in which leaf node it ends -> (most probable) class. Combine the results, e.g. which class/label was found most often

Pro:
\begin{itemize}
	\item Easy to implement
	\item very efficient during testing
	\item can easily use diverse features
	\item can handle high dimensional spaces
\end{itemize}
Cons:
\begin{itemize}
	\item Lots of parametric choices
	\item needs large number of data
	\item training can take time
\end{itemize}


% File:		makepass.tex
% Author:	See Below
% Date:	        See Below
%
% The authors have placed this file in the public domain;
% they make no warranty and accept no liability for this file.

\documentclass[12pt]{article}
\usepackage{times}
\begin{document}
\newcommand{\problem}[1]{\underline{\Large \bf #1}}
\renewcommand{\section}[1]{\bigskip\underline{\bf #1}\\}
\newcommand{\header}[1]{\underline{\bf #1}}
\newcommand{\file}[1]{{\bf #1}}
\newcommand{\blankpage}{\newpage\vspace*{3.5in}%
    \centerline{\Large This Page is Intentionally Left Blank}}
\setlength{\parindent}{0.0in}
\setlength{\parskip}{1ex}

\problem{Make Passwords}

Ideally a password would be a string of 10 (or more)
randomly chosen decimal digits.  But this tends to be hard
to remember or type.

You are being asked to convert strings of decimal digits
into a more typeable and perhaps more memorable form using a 1-1 encoding.
In this encoding, groups of more than 3 digits are encoded
as 3-letter word randomly chosen from a dictionary.

Some examples are:\hspace{0.2in}
\begin{tabular}[t]{r@{~~~~~{\em is encoded as}~~~~~}l}
161954595198532 & 88bem06fis \\
174570539285673 & 5bid012loq \\
417744241130258 & dav034zeq9 \\
566373677809620 & sah9qub784 \\
270281313141987 & 4dud456niz \\
\end{tabular}

The encoding algorithm uses a dictionary of all words
of the form: \\
\hspace*{1in}$<consonant><vowel><consonant>$ \\
sorted in
lexical order, with {\tt y} treated as a consonant.
Dictionary word $0$ is {\tt "bab"} and word
$21*5*21-1=2204$ is {\tt "zuz"}.

Then the encoding algorithm with input number $N$ is as follows:
\begin{enumerate}
\item Divide $N$ by 6 and use the remainder to select the
encoding format thus:
\begin{tabular}{r@{~~~~~{\em selects}~~~~~}l}
0 & \tt "WWWDWWWDDD" \\
1 & \tt "WWWDDWWWDD" \\
2 & \tt "WWWDDDWWWD" \\
\end{tabular}
\hspace{0.3in}
\begin{tabular}{r@{~~~~~{\em selects}~~~~~}l}
3 & \tt "DWWWDDDWWW" \\
4 & \tt "DDWWWDDWWW" \\
5 & \tt "DDDWWWDWWW" \\
\end{tabular}
\item Process the format left to right.  If the next
character is {\tt D}, divide $N$ by 10 and output the
remainder.  If the next character is {\tt W}, divide $N$
by $21*5*21 = 2205$, use the remainder to look up
a word in the dictionary, output the word, and skip to
after the 3 {\tt W}'s in the format.
\end{enumerate}

For example, $0+6*(0+2205*(1+10*(2204+2205*(2+10*(3+10*4)))))= 126315290430$
is encoded as {\tt bab1zuz234}.  Note you must use `{\tt long}'
integers for C, C++, and JAVA, and that numbers input are treated
module $6*(21*5*21)^2*10^4=291721500000$.

\section{Input}
One more lines each containing a non-negative integer
with at most 12 digits.  Input ends with an end of line.

\section{Output}
For each input line, output one line containing the
encoded input integer.

\bigskip

\begin{center}
\begin{tabular}{ll}
\begin{minipage}[t]{2.5in}
\header{Sample Input}
\\[1ex]
\file{00-000-makepass.sin:}
\begin{verbatim}
0
6
13230
132300
291721500
2917215000
29172150000
\end{verbatim}
\file{00-001-makepass.sin:}
\begin{verbatim}
0
1
2
3
4
5
\end{verbatim}
\file{00-002-makepass.sin:}
\begin{verbatim}
126315290430
854595198532
870539285673
144241130258
273677809620
281313141987
\end{verbatim}
\end{minipage}
&
\begin{minipage}[t]{2.5in}
\header{Sample Output}
\\[1ex]
\file{00-000-makepass.sout:}
\begin{verbatim}
bab0bab000
bac0bab000
bab1bab000
bab0bac000
bab0bab100
bab0bab010
bab0bab001
\end{verbatim}
\file{00-001-makepass.sout:}
\begin{verbatim}
bab0bab000
bab00bab00
bab000bab0
0bab000bab
00bab00bab
000bab0bab
\end{verbatim}
\file{00-002-makepass.sout:}
\begin{verbatim}
bab1zuz234
88pat25yiq
5yuz930zok
vac975yuf4
lod3fan839
4qol723zeh
\end{verbatim}
\end{minipage}
\end{tabular}
\end{center}

\bigskip

\begin{tabular}{ll}
Author:	      & Robert L.~Walton $<$walton@acm.org$>$ \\
Date:         & Thu Oct  8 06:35:04 EDT 2020
\end{tabular}

The authors have placed this problem in the public domain;
they make no warranty and accept no liability for this problem.

\end{document}


\chapter{Factoring Polynomials}

We factor a polynomial into two or more polynomials of lower
degree. For example, let's say that you wanted to factor
$5x^3 - 45x$. You would note that you can factor out $5x$ from every term. Thus,
\begin{equation*}
5x^3 - 45x = (5x)(x^2 - 9)
\end{equation*}
And then, you might notice that the second factor looks like the difference of squares, so
\begin{equation*}
5x^3 - 45x = (5x)(x + 3)(x - 3)
\end{equation*}
That is as far as we can factorize this polynomial.\index{factoring polynomials}

Why do we care? The factors make it easy to find the roots of the
polynomial. This polynomial evaluates to zero if and only if at least
one of the factors is zero. Here we see that
\begin{itemize}
\item The factor $(5x)$ is zero when $x$ is zero.
\item The factor $(x + 3)$ is zero when $x$ is -3.
\item The factor$(x - 3)$ is zero when $x$ is 3.
\end{itemize}
So looking at the factorization, you can see
that $5x^3 - 45x$ is zero when $x$ is 0, -3, or 3.

This is a graph of that polynomial with its roots circled:

\includegraphics{factor4roots}

\section{How to factor polynomials}

The first step when you are trying to factor a polynomial is to find
the greatest common divisor for all the terms, and pull that out. In
this case, the greatest common divisor will also be a monomial: its
degree is the least of the degrees of the terms, its coefficient will
be the greatest common divisor of the coefficients of the terms.

For example, what can you pull out of this polynomial?
\begin{equation*}
12x^100 + 30x^31 + 42x^17
\end{equation*}
The greatest common divisor of the coefficients (12, 30, and 42) is 6.  The least of the degrees of terms (100, 31, and 17) is 17.  So you can pull out $6x^17$:
\begin{equation*}
12x^100 + 30x^31 + 42x^17 = (6x^17)(2x^83 + 5x^14 + 7)
\end{equation*}

\begin{Exercise}[title={Factoring out the GCD monomial}, label=gcdmonomial]

\end{Exercise}
\begin{Answer}[ref=gcdmonomial]

\end{Answer}

So, now you have the product of a monomial and a polynomial. If you
are lucky, the polynomial part looks familiar, like the difference of
squares or a row from Pascal's triangle.

Often you are trying factor a quadratic like $x^2 + 5x + 6$ in a pair
of binomials. In this case, the result would be $(x + 3)(x + 2)$. Let's check that:
\begin{equation*}
  (x + 3)(x + 2) = (x)(x) + (3)(x) + (2)(x) + (3)(2) = x^2 + 5x + 6
\end{equation*}
Notice that 3 and 2 multiply to 6 and add to 5. If I were trying to
factor $x^2 + 5x + 6$, I would ask myself''What are two numbers that
when multiplied equal 6 and when added equal 5?'' And I would might
guess wrong a couple of times. For example, I might say to myself
``Well, 6 times 1 is 6. Maybe those work. But 6 and 1 add 7. So those
don't work.''

Solving these sorts of problems are like solving a Sudoku puzzle: you
try things and realize they are wrong, so you backtrack and try
something else.

The numbers are sometimes negative. For example, $x^2 + 3x - 10$ factors into $(x + 5)(x - 2)$.

\begin{Exercise}[title={Factoring quadratics}, label=factorquadratics]

\end{Exercise}
\begin{Answer}[ref=factorquadratics]

\end{Answer}


\chapter{Functional form for the prior}\label{appendix:prior_function}
\paragraph{} \cite{2006A&A...457..841I} devise a functional form for the prior $P(z, T | m_{0})$, by once more using Bayes' theorem to obtain:

\begin{equation}
P(z, T | m_{0}) = P(T | m_{0}) P(z | T, m_{0}). \label{eqn:prior}
\end{equation}

$P(z|T, m_{0})$ is parameterised as:

\begin{equation}
P(z|T,m_{0})  \propto z^{\alpha_{T}}\exp{-\left[ \frac{z}{z_{oT}+k_{mT}(m_{0} -20)} \right]},\label{eqn:redshift_distr}
\end{equation}

\noindent where $\alpha_{T}$, $z_{0T}$, $k_{mT}$ are free parameters and $T$ encapsulates the dependence on spectral type. The other part of Equation \ref{eqn:prior}, $P(T|m_{0})$, gives the magnitude distribution of spectral types and is parameterised as

\begin{equation}
P(T|m_{0}) = f_{T} e^{-k_{T}(m_{0}-20)},\label{eqn:spectral_distr}
\end{equation}

where $k_{T}$ is a free parameter  and $f_{T}$ is the fraction of each type at $m_{0} = 20 \textrm{ mag AB}$. Following the formalism in \cite{2000ApJ...536..571B}, \cite{2006A&A...457..841I} calibrated the values for all four free parameters using a deep and representative $I$-selected spectroscopic sample from VVDS \citep{2005A&A...439..845L} using data from a (different) $i'$ band for $m_{0}$. To quantify the dependence on spectral type, the sample was divided into four different spectral type groups (according to the four optimised CWW  templates [\cite{1980ApJS...43..393C}, \cite{2006A&A...457..841I}]). For each group,  $f_T$ and $k_T$ were determined by fitting Equation \ref{eqn:spectral_distr} to the observed magnitude distribution. Likewise, the redshift distrubution of each spectral type group was fitted to {\color{red} Equation \ref{eqn:redshift_distr}} to obtain $\alpha_{T}$, $z_{0,T}$ and $k_{mT}$.


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{tfrridb}
\section*{\hspace*{-1.6cm} tfrridb}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Reduced Interference Distribution with Bessel kernel.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[tfr,t,f] = tfrridb(x)
[tfr,t,f] = tfrridb(x,t)
[tfr,t,f] = tfrridb(x,t,N)
[tfr,t,f] = tfrridb(x,t,N,g)
[tfr,t,f] = tfrridb(x,t,N,g,h)
[tfr,t,f] = tfrridb(x,t,N,g,h,trace)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        Reduced Interference Distribution with a kernel based on the Bessel
        function of the first kind.  {\ty tfrridb} computes either the
        distribution of a discrete-time signal {\ty x}, or the cross
        representation between two signals. This distribution writes
\begin{eqnarray*}
RIDB_x(t,\nu)&=&\int_{-\infty}^{+\infty} h(\tau) R_x(t,\tau)\,
e^{-j2\pi\nu\tau}\ d\tau\\
{\rm with}\quad
R_x(t,\tau)&=&
\int_{t-|\tau|}^{t+|\tau|}\
\dfrac{2\ g(v)}{\pi|\tau|}\ \sqrt{1-\left(\frac{v-t}{\tau}\right)^2}
x(v+\frac{\tau}{2})\ x^*(v-\frac{\tau}{2})\ dv.
\end{eqnarray*}

\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{8cm} c}
Name & Description & Default value\\
\hline
        {\ty x}     & signal if auto-RIDB, or {\ty [x1,x2]} if cross-RIDB ({\ty
			Nx=length(x)})\\
        {\ty t}     & time instant(s)          & {\ty (1:Nx)}\\
        {\ty N}     & number of frequency bins & {\ty Nx}\\
        {\ty g}     & time smoothing window, {\ty G(0)} being forced to
		{\ty 1}, where {\ty G(f)} is the Fourier transform of {\ty g(t)}
                                         & {\ty window(odd(N/10))}\\
        {\ty h}     & frequency smoothing window, {\ty h(0)} being forced to {\ty 1}
                                         & {\ty window(odd(N/4))}\\
        {\ty trace} & if nonzero, the progression of the algorithm is shown
                                         & {\ty 0}\\
     \hline {\ty tfr}   & time-frequency representation\\
        {\ty f}     & vector of normalized frequencies\\

\hline
\end{tabular*}
\vspace*{.2cm}

When called without output arguments, {\ty tfrridb} runs {\ty tfrqview}.
\end{minipage}

\newpage

{\bf \large \sf Example}
\begin{verbatim}
         sig=[fmlin(128,0.05,0.3)+fmlin(128,0.15,0.4)];
         g=window(31,'rect'); h=window(63,'rect');
         tfrridb(sig,1:128,128,g,h,1);
\end{verbatim}
\vspace*{.5cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
all the {\ty tfr*} functions.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Reference}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
[1] Z. Guo, L.G. Durand, H.C. Lee ``The Time-Frequency Distributions of
Nonstationary Signals Based on a Bessel Kernel'' IEEE Trans. on Signal
Proc., vol 42, pp. 1700-1707, july 1994.
\end{minipage}