%!TEX root = ../dokumentation.tex
\chapter{Theory}

\section{Classification of Data}
%Data Types in ML (Cross-Sectional-, Time Series- and Panel Data)
Most data found in academic and industrial projects can be broadly classified into three categories:
\begin{itemize}
	\item Cross-sectional data
	\item Time series data
	\item Panel data
\end{itemize}

Cross-sectional data is data taken from multiple individuals at one point in time. The cross section of a population is obtained by taking observations from multiple individuals at the same time our without taking time into consideration at all. One example for this could be the test scores of all students for one particular exam. Another example is shown here:

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.475\textwidth]{images/example_cross-sectional_data.PNG}}%\scalebox{1}{\includegraphics[width=05\textwidth]{images/empty-transparent.png}}\scalebox{1}{\includegraphics[width=0.475\textwidth]{images/example_cross-sectional_data.PNG}}
	\caption{Example for cross-sectional data \textsuperscript{\cite{-1}}}
\end{figure}

%https://www.cnbc.com/2018/02/16/black-panther-already-setting-records-with-thursday-box-office.html
%example_cross-sectional_data.PNG

Time series data is data taken from one individual at multiple points in time. A time series is made up of quantitative observations on one or more measurable characteristics of an individual entity and taken in an interval of time. The data is typically characterized by several internal structural elements such as trend, seasonality, stationarity, autocorrelation and noise. A common example for this would be sensor data, but also stock data when measured over time or the average income of politicians over the course of their career.

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.475\textwidth]{images/example_timeseries_data.PNG}}
	\caption{Example for time series data \textsuperscript{\cite{-1}}}
\end{figure}

The last category is a combination of the first two. Panel Data can be defined as data taken from multiple individuals at multiple points in time and also known as longitudinal data.

An example for panel data would be Military Expenses of all European countries from 2001 to 2010. And one of the methods used to analyses this type of dataset is called “fixed effects model” which itself is also type of regression model.

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.475\textwidth]{images/example_panel_data.PNG}}
	\caption{Example for panel data \textsuperscript{\cite{-1}}}
\end{figure}

Methods for analyzing any of this data include plotting of variables to visualize statistical properties and calculation of central tendency, variance, skewness, and kurtosis.

%\pagebreak
\section{Time Series Analysis}

Analyzing time series is about understanding and modeling the different characteristics that time series data can exhibit. The most commonly studied features of time series data are:
\begin{itemize}
	\item Noise
	\item General trend
	\item Cyclic movements
	\item Seasonality
	\item Pulses and Steps
	%\item Outliers
	\item Stationarity
\end{itemize}

The first one that is mostly referred to as \textit{noise} reflects unexpected variations in the time series. It is often used to describe a type of error in a time series model that is due to a lack of information about explanatory variables that can model these variations or due to presence of random noise.

And there can be different models used in which there is no trend nor seasonal component. One of the those models describes the \acl{iid} noise or \acs{iid} noise. In this model observations are simply \acl{iid} variables with zero mean and it is also called white noise. This means that this model doesn't not have dependencies between observations.

However there is also another model that does show dependencies. And it is know as \textit{Random Walk Model}. The random walk \(S_t \) obtained by cumulatively summing \acs{iid} random variables \([Z_t]\) with \(S_0=0\) and can therefore be defined as:

\begin{equation}\label{eq:random_walk_model}
S_t = Z_1 + Z_2 + Z_3 + ... + Z_t
\end{equation}

If furthermore the random variables \([Z_t] \) are Bernoulli distributed with \(p=0.5\) then this is called a simple symmetric random walk. An example for this could be a pedestrian who starts a position zero at time zero and at each integer time tosses a fair coin, stepping one unit to the right each time a head appears and one unit to the left for each tail. Both these model are referenced to as \textit{zero-mean models}.
An example of the random walk model as well as an \acs{iid} model can be seen in Figure 2.4.

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.66\textwidth]{images/example_noise.png}}
	\caption{Example for iid model (top) and random walk model (bottom)}
\end{figure}

In several time series there is also a clear trend to be observed in the data. This means it exhibits an upward or downward movement in the long run. In those cases a zero mean model is not sufficient to describe the series.
Trend models aim to capture this long run trend in a time series. They can be fitted as linear regressions of the time index. And for this a trend model \(X_t\) can be expresses as the sum of the trend component \(m_t\) being a slowly changing function and a zero mean model \(z_t\):

\begin{equation}\label{eq:simple_trend_model}
X_t = m_t + z_t
\end{equation}

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.66\textwidth]{images/example_trendednoise.png}}
	\caption{Example of trend models using  \acs{iid} and random walk as zero mean models}
\end{figure}

Besides general trend and noise many time series are also influenced by factors that occurs periodically. These repetitive fluctuations are called \textit{cyclic movements}. These can further be divided into nonseasonal cycles and seasonal cycles.

Nonseasonal cycles are repetitive, possibly unpredictable patterns in time series values and defined by a periodicity that varies of time.

In contrast to that, seasonal cycles have a known \textbf{constant} periodicity. And in that case the time series is to be said to exhibit seasonality.  One example causing seasonality in a series is dependency of the observed system on  weather. It itself is seasonal and when the weather then interferes with the observed system this interference can be seen in periodic fluctuations that occurs in a fixed period.

%2002 Brockwell, Introduction to Time Series and Forecasting
In order to represent such seasonal effects, allowing for noise but assuming no trend, we can use a model that can be expressed as combination of an harmonic regression and the zero mean model

\begin{equation}\label{eq:simple_seasonal_model}
X_t = s_t + z_t
\end{equation}
where \(s_t\) is a periodic function of t with period d (\(s_{t-d}= s_t\)) and is defined as:

\begin{equation}\label{eq:harmonic_regression}
s_t = a_0 + \displaystyle\sum_{j=1}^{k} (a_j \cos(\lambda t) + b_j \sin(\lambda t ))
\end{equation}

here \(a_0, a_1, . . . , a_k\) and \(b_1, . . . , b_k\) are unknown parameters and \(\lambda_1, . . . , \lambda_k\) are fixed frequencies, each being some integer multiple of \(2\pi/d\).

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.66\textwidth]{images/example_season_and_noise.png}}
	\caption{Example of seasonal models using \acs{iid} and random walk as zero mean models}
\end{figure}

By using an additive model \(X_t= m_t + s_t + z_t\) combining a trend component \(m_t\), a seasonal component \(s_t\) and the random walk model as the noise or zero-mean component \(z_t\) a time series can be decomposed and visually represented like in figure 2.6. However real time series data is usually not that easily modeled, because they usually also show other more complex characteristics that are harder to model and therefore predict. These include \textit{pulses} and \textit{steps} as well as different types of \textit{outliers}.

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.6\textwidth]{images/example_timeseries_characteristics.png}}
	\caption{Example of time series decomposed into seasonal, trending and zero mean model}
\end{figure}
\textit{Pulses} and \textit{steps} are abrupt changes in level that a series might exhibit:
A \textit{pulse} is defined as a temporary shift and a \textit{step} is a permanent shift in the series.

%"":
When steps or pulses are observed, it is important to find a plausible explanation. Time series models are designed to account for gradual, not sudden, change. As a result, they tend to underestimate pulses and be ruined by steps, which leads to poor model fits and uncertain forecasts. If a disturbance can be explained, it can be modeled using an intervention or event.
%https://www.ibm.com/support/knowledgecenter/SS3RA7_17/components/dt/timeseries_pulses.html?view=embed

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.66\textwidth]{images/example_pulse.jpg}}
	\caption{Example of a pulse in a time series}
\end{figure}

%if not enough pages add outliers: https://www.ibm.com/support/knowledgecenter/SS3RA7_17/components/dt/ts_outliers_overview.html

%https://people.duke.edu/~rnau/411diff.htm
%https://pdfs.semanticscholar.org/0f08/bcca67b3db328edfa5d3f48331dc71d8789e.pdf
%2002 Brockwell, Introduction to Time Series and Forecasting
A particular importance to many time series forecasting models has the characteristic that is called stationarity. This is because a lot of time series models use the assumption that the series to be forecast is already stationarity or can be made approximately stationary through the means of mathematical transformations.
A time series is called stationary if its statistical properties stay constant over time. These restrictions only have to apply to those properties that depend only on the first- and second-order moments of \(X_t\):

Let \({X_t}\) be a time series with \(E(X_t^2 ) < \infty\). The mean function of \({X_t }\) is

\begin{equation}\label{eq:mean_function}
\mu_X(t) = E(X_t )
\end{equation}
The covariance function of \({X_t }\) is

\begin{equation}\label{eq:covariance_function}
\gamma_X(r, s)  = Cov(X_r,X_s)  E[(X_r - \mu_X(r))(X_s - \mu_X(s))]
\end{equation}
for all integers r and s.
\({X_t }\) is (weakly) stationary if
\begin{enumerate}
	\item [i] \(\mu_X(t)\) is independent of t,
	\item [and]
	\item [ii] \(\gamma_X(t + h, t) \) is independent of t for each h.
\end{enumerate}

%https://people.duke.edu/~rnau/411diff.htm
To obtain meaningful statistical properties such as mean, variance, and correlations a timer series has to be stationarized first. Only then such statistics can be used as descriptors for future behavior.
For example, if the series is consistently increasing over time, the sample mean and variance will grow with the size of the sample, and they will always underestimate the mean and variance in future periods. And therefore based on the mean and variance the correlation with other variables would be under- or overestimated too.

This is why there is a need for mathematical transformations that can make a non-stationary time series approximately stationary.

One of those mathematical transformations is called \textit{de-trending} and can be used when the series has a stable long-run trend and no seasonality. This is achieved by fitting a trend model to the time series and then subtracting it from the original series. The resulting time series can then be further analyzed and modeled and is called \textit{trend-stationary} if the process was able to stationarize the series. An equivalent process is working for time series which do show seasonality. Of course in those cases a seasonal model has to be fitted and then subtracted from the series. There is also the possibility of combining both of those approaches.

However, not all time series can be stationarized by these processes. For some series this is insufficient and they might have to be differenced, either from period to period or from season to season. The idea is that if, even after removing a trend and/or seasonal model, the statistical characteristics are still not constant over time, then the statistics of the changes in the series between periods or seasons might be constant. In that case it is said to be \textit{difference-stationary}. Using unit-root tests can help identifying what stationarizing method might be more successful.

For the process of nonseasonal \textit{first order differencing} the \textit{lag-1} operator \(\Delta\) is defined by:

\begin{equation}\label{eq:first_difference}
\Delta X_t = X_t - X_{t-1} = (1-B)X_t
\end{equation}

where B is the backward shift or \textit{backshift} operator

\begin{equation}\label{eq:bachshift_operator}
B X_t = X_{t-1}
\end{equation}

Powers of operators \(B\) and \(\Delta\) are defined by \(B^j X_t = X_{t-j}\) and \(\Delta^j X_t = \Delta (\Delta^{j-1} X_t), j >= 1\) with \(\Delta^0 X_t = X_t\). Polynomials in \(B\) and \(\Delta\) are manipulated in precisely the same way as polynomial functions of real variables. For example:

\begin{equation}\label{eq:example_delta_squared}
\Delta^2 X_t = \Delta (\Delta X_t) = (1-B)(1-B)X_t = (1-2B + B^2) X_t = X_t - 2 X_{t-1} +  X_{t-2}
\end{equation}

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.66\textwidth]{images/example_firstorderdiff.png}}
	\caption{Example of a time series with its first order non-seasonal difference}
	\label{fig:example_non_seasonal_diff}
\end{figure}

\textit{First order differencing} is an example for a period to period differencing and is used to removed the trend from a time series. However, as seen in \ref{fig:example_non_seasonal_diff} the seasonality is still exhibited in the same way and to remove this as well seasonal differencing as to be applied additionally.

For the process of seasonal \textit{first order differencing} the \textit{lag-1} operator \(\Delta_s\) with seasonality \(s\) is defined by:

\begin{equation}\label{eq:seasonal_first_difference}
\Delta_s X_t = X_t - X_{t-s} = (1-B^s)X_t
\end{equation}

And the \textit{D-th order difference} is defined by:

\begin{equation}\label{eq:seasonal_difference}
\Delta_s^D X_t = (X_t - X_{t-s})^D = (1-B^s)^DX_t
\end{equation}

\begin{figure}[ht]
	\centering
	\scalebox{1}{\includegraphics[width=0.66\textwidth]{images/example_seasonaldiff.png}}
	\caption{Example of a time series with its first order non-seasonal difference}
	\label{fig:example_seasonal_diff}
\end{figure}

%Characteristics of Time Series Data and how to model them
%Smoothing
%Smoothing is often referred to as filtering.
%There are two distinct groups of smoothing methods
%- Averaging Methods (Moving Average)
%- Exponential Smoothing Methods
%Autocovariance and (Partial) Autocorrelation

\section{ARIMA Model}\label{arimamodel}
The \acl{ARIMA} (\acs{ARIMA}) model is used to better understand and forecast time series data. It is based on the \acl{ARMA} (\acs{ARMA}) model but can also handle data that shows evidence of non-stationarity.
\acs{ARMA} is composed out of the \acl{AR} (\acs{AR}) model and the \acl{MA} (\acs{MA}) model.

\acl{AR} models are used to predict future values of a time series by using linear combinations of previous values of the series and \acl{MA} models try to forecast by combining the prediction error that has been made for previous predictions in the time series.

Given a univariate time series \(X_t\) that is stationary and \(e_t\) a random variable with an independent and identical distribution representing the error that can occur in any prediction, the \acs{AR}\((p)\) model is defined by:
\begin{equation}\label{eq:AR_p}
X_t = e_t + \displaystyle\sum_{i=1}^{p} (\phi_i B^i) X_t = e_t + \phi_1 X_{t-1}+ \phi_2 X_{t-2}+ ... + \phi_p X_{t-p}
\end{equation}

By defining the error of a prediction as the difference between the correct value from the time series and the value from the approximation \(\hat{X_t}\) as:

\begin{equation}\label{eq:error_term}
e_t = X_t - \hat{X_t}
\end{equation}

The approximation of \(X_t\) using an \acs{AR}\((4)\) can be written as:

\begin{equation}\label{eq:AR_four}
\hat{X_t} = \phi_1 X_{t-1}+ \phi_2 X_{t-2}+ \phi_3 X_{t-3}+ \phi_4 X_{t-4}
\end{equation}

The \acl{MA} model is based on the definition \eqref{eq:error_term} of the error term \(e_t\). In contrast to the \acs{AR} model it combines \(e_t\) terms instead of just combining\(X_t\). Hence the definition of \acs{MA}\((q)\) is:

\begin{equation}\label{eq:MA_q}
X_t = e_t + \displaystyle\sum_{k=1}^{q} (\theta_k B^k) e_t = e_t + \theta_1 e_{t-1}+ \theta_2 e_{t-2}+ ... + \theta_p e_{t-q}
\end{equation}

This makes the calculation of an \acs{MA}\((q)\) model a little bit more complicated, as demonstrated on the example of \acs{MA}\((3)\):

\begin{equation}\label{eq:example_MA_three_1}
X_t = e_t + \theta_1 e_{t-1}+ \theta_2 e_{t-2} + \theta_3 e_{t-3}
\end{equation}

Replacing \(e_t\) with its definition \eqref{eq:error_term} and transforming the equation to \(\hat{X_t}\) makes the complication clearer:

\begin{equation}\label{eq:example_MA_three_2}
\hat{X_t} =\theta_1 (X_{t-1} - {\hat{X}}_{t-1}) + \theta_2 (X_{t-2} - {\hat{X}}_{t-2}) + \theta_3 (X_{t-3} - {\hat{X}}_{t-3})
\end{equation}

In this form it is clear that \(\hat{X_t}\) is depended on \({\hat{X}}_{t-1}\), \({\hat{X}}_{t-2}\), \({\hat{X}}_{t-3}\) which are also unknown variables that have to be calculated first. Using the same formula for \({\hat{X}}_{t-1}\) and so on, this leads to more dependencies which lead to even more. What one eventually ends up with is a system of linear equation that, assuming \(X_t\) is a series of length 5 looks like this:

\begin{equation}\label{eq:example_MA_three_system_1}
\begin{array}{lcl}
{\hat{X}}_{5} & = & \theta_1 (X_{4} - {\hat{X}}_{4}) + \theta_2 (X_{3} - {\hat{X}}_{3}) + \theta_3 (X_{2} - {\hat{X}}_{2}) \\
{\hat{X}}_{4} & = & \theta_1 (X_{3} - {\hat{X}}_{3}) + \theta_2 (X_{2} - {\hat{X}}_{2}) + \theta_3 (X_{1} - {\hat{X}}_{1}) \\
{\hat{X}}_{3} & = & \theta_1 (X_{2} - {\hat{X}}_{2}) + \theta_2 (X_{1} - {\hat{X}}_{1})\\
{\hat{X}}_{2} & = & \theta_1 (X_{1} - {\hat{X}}_{1})\\
{\hat{X}}_{1} & = & 0
\end{array}
\end{equation}

To be able to calculate this for larger time series, some kind of solving mechanism is needed. To be able to use one of those the system of linear equations needs to be in the form of:

\begin{equation}\label{eq:syslinequation}
\mathbf{A}  \vec{\hat{x}} = \vec{b}
\end{equation}

with $\mathbf{A} \in \mathbb{R}^{n\times n}$ being a $n\times n$ Matrix,  $\vec{\hat{x}} \in \mathbb{R}^n$ and  $\vec{b}\in \mathbb{R}^n$ both n-dimensional vectors. The vector of $\vec{b}$ representing the known values for each equation, $\vec{\hat{x}}$ the unknown variables and $\mathbf{A}$ the coefficients of all the $\vec{\hat{x}}$ for each equation.

To see how $\mathbf{A}$ and  $\vec{b}$ have to be constructed the system of linear equations \eqref{eq:example_MA_three_system_1} can be transformed to:
\begin{equation}\label{eq:example_MA_three_system_2}
\begin{array}{rcrcrcrcrclll}
{\hat{X}}_{1}&&&&&&&&& = &0&&\\
\theta_1 {\hat{X}}_{1} &+& {\hat{X}}_{2} & & & & & & &= &\theta_1 X_{1}&&\\
\theta_2 {\hat{X}}_{1}&+&\theta_1 {\hat{X}}_{2} &+& {\hat{X}}_{3}&  &  &&& = &\theta_1 X_{2} &+ \theta_2 X_{1}&\\
\theta_3 {\hat{X}}_{1} &+& \theta_2 {\hat{X}}_{2} &+&  \theta_1 {\hat{X}}_{3}&+& {\hat{X}}_{4}& && = &\theta_1 X_{3} &+ \theta_2 X_{2} &+ \theta_3 X_{1}\\
&&\theta_3 {\hat{X}}_{2} &+& \theta_2 {\hat{X}}_{3}&+& \theta_1 {\hat{X}}_{4} &+&{\hat{X}}_{5}& = &  \theta_1 X_{4} &+\theta_2 X_{3} &+ \theta_3 X_{2}
\end{array}
\end{equation}

This leads directly to the form required in \eqref{eq:syslinequation}:

\begin{equation}\label{eq:example_MA_three_system_3}
\left(\begin{array}[c]{lllll}
1 & 0 & 0 & 0 & 0\\
\theta_1 & 1 & 0 & 0 & 0\\
\theta_2 & \theta_1& 1 & 0 & 0\\
\theta_3 & \theta_2 & \theta_1& 1 & 0\\
0 & \theta_3 & \theta_2 & \theta_1& 1
\end{array}\right) \;\vec{\hat{x}} =
\left(\begin{array}[c]{rrr}
0 &&\\
\theta_1 X_{1} &&\\
\theta_1 X_{2} &+ \theta_2 X_{1} &\\
\theta_1 X_{3} &+ \theta_2 X_{2} &+ \theta_3 X_{1} \\
\theta_1 X_{4} &+ \theta_2 X_{3} &+ \theta_3 X_{2}
\end{array}\right)
\end{equation}

Having the system of linear equations in this form, the \acs{MA}$(4)$ model with given $\theta_k$ for $k \in [1,...,4]$ can be calculated using a numerical solver for systems of linear equations like the Jacobi (chapter \ref{jacobi}) or Conjugate Gradient solver.

The \acs{ARMA}$(p,q)$ model can now be composed out of the \acl{AR}$(p)$ and \acl{MA}$(q)$ model defined previously by the equations \eqref{eq:AR_p} and \eqref{eq:MA_q} and can be represented as:

\begin{equation}\label{eq:ARMA_1}
\begin{array}{ccc}
(1-\displaystyle\sum_{i=1}^{p} \phi_i B^i) X_t & = & (1+\displaystyle\sum_{k=1}^{q} \theta_k B^k)e_t\\
\Leftrightarrow X_t - \phi_1 X_{t-1} - \phi_2 X_{t-2} - ... - \phi_p X_{t-p} & = & e_t + \theta_1 e_{t-1}+ \theta_2 e_{t-2}+ ... + \theta_p e_{t-q}
\end{array}
\end{equation}

The approximation $\hat{X_t}$ can therefore be calculated by:
\begin{equation}\label{eq:ARMA_2}
\rightarrow  \hat{X}_t = \phi_1 X_{t-1} + \phi_2 X_{t-2} + ... + \phi_p X_{t-p} + \theta_1 e_{t-1}+ \theta_2 e_{t-2}+ ... + \theta_p e_{t-q}
\end{equation}

Because this equation \eqref{eq:ARMA_2} is just a combination of the equations \eqref{eq:AR_p} and \eqref{eq:MA_q} it also comes with the same difficulties as previously described and has therefore to be transformed the same way, so it is in the form of equation \eqref{eq:syslinequation}.

This is achieved with the same procedures as described for the example of \acs{MA}$(3)$ eventually leading to an equation in the form of \eqref{eq:example_MA_three_system_3} with all the \acs{AR} terms of the equation contained within $\vec{b}$.
%Maybe add example for Matrix and more detailed explanation later

However, like already mentioned in the beginning of this chapter: \acs{ARIMA} is a generalization of \acs{ARMA} with the additional ability to also work with non-stationary time series. It accomplishes this by differencing the time series $d$ times. Hence, the \acs{ARIMA}$(p,d,q)$ model also includes the equation for first order differencing \eqref{eq:first_difference} - modified to equal d-th order differencing - and is therefore defined as:

\begin{equation}\label{eq:ARIMA_1}
\begin{array}{cccccc}
(1-\displaystyle\sum_{i=1}^{p} \phi_i B^i) & (1-B^d)& X_t & = & (1+\displaystyle\sum_{k=1}^{q} \theta_k B^k) & e_t
\end{array}
\end{equation}

But this model won't be able to fit any seasonal behavior, because it is missing the seasonal parts of all three components \acs{AR}, \acs{MA} and d-th order differencing.

The most commonly used \acs{ARIMA} model for forecasting seasonal time series  is the multiplicative \acl{SARIMA} (\acs{SARIMA}) model. This model assumes that there is a significant parameter as a result of multiplication between nonseasonal and seasonal parameters. But there also is an additive (\acs{SARIMA}) model that can be used to forecast.

The additive models for \acl{SAR} \acs{SAR}$(p,P)$ are defined as:
\begin{equation}\label{eq:additive_SAR_pP}
\begin{array}{rcl}
X_t & = & e_t + \displaystyle\sum_{i=1}^{p} \phi_i B^i X_t + \displaystyle\sum_{j=1}^{P} \Phi_j B^{i\cdot s} X_t\\
X_t & = & e_t + \phi_1 X_{t-1}+ ... + \phi_p X_{t-p} + \Phi_1 X_{t-1 s}+ ... + \Phi_P X_{t-P s}
\end{array}
\end{equation}

And analog to this the additive \acl{SMA} \acs{SMA}$(q,Q)$ model is defined by:

\begin{equation}\label{eq:additive_SMA_q}
\begin{array}{rcl}
X_t & = & e_t + \displaystyle\sum_{k=1}^{q} \theta_k B^k e_t + \displaystyle\sum_{l=1}^{Q} \Theta_l B^{l\cdot s} e_t\\
X_t & = & e_t + \theta_1 e_{t-1}+ ... + \theta_q e_{t-q} + \Theta_1 e_{t-1 s}+ ... + \Theta_Q e_{t-Q s}
\end{array}
\end{equation}

The differencing component is always multiplicative and therefore by also adding the equation \eqref{eq:seasonal_difference} for the seasonal D-th order difference the additive \acs{SARIMA}$(p,d,q)(P,D,Q)_s$ is defined as:

\begin{equation}\label{eq:additive_SARIMA}
\begin{array}{cccccc}
(1-(\displaystyle\sum_{i=1}^{p} \phi_i B^i + \displaystyle\sum_{j=1}^{P} \Phi_j B^{i\cdot s})) & (1-B^d)& X_t & = & (1+(\displaystyle\sum_{k=1}^{q} \theta_k B^k + \displaystyle\sum_{l=1}^{Q} \Theta_l B^{l\cdot s})) & e_t
\end{array}
\end{equation}

The multiplicative models are a little bit more complicated for both \acs{SAR} as well as \acs{SMA}. With the multiplicative \acl{SAR} \acs{SAR}$(p,P)$ model being the simpler one defined by:

\begin{equation}\label{eq:multiplicative_SAR_pP}
\begin{array}{rcl}
e_t & = & (1-\displaystyle\sum_{i=1}^{p} \phi_i B^i) (1-\displaystyle\sum_{j=1}^{P} \Phi_j B^{i\cdot s}) X_t\\
e_t & = & (1-\phi_1 B-\phi_2 B^2 - ... -\phi_p B^p) (1-\Phi_1 B^s-\Phi_2 B^{2s} - ... -\Phi_p B^{Ps}) X_t\\
e_t & = & (1-\phi_1 B-\phi_2 B^2 - ... -\phi_p B^p -\Phi_1 B^s-\Phi_2 B^{2s} - ... -\Phi_p B^{Ps} \\
& & + \phi_1 \Phi_1 B^{1+s} + \phi_2 \Phi_1 B^{2+s} + \phi_2 \Phi_2 B^{2+2s} + ... + \phi_p \Phi_P B^{p+Ps}) X_t\\
X_t & = & e_t + \displaystyle\sum_{i=1}^{p} \phi_i B^i X_t + \displaystyle\sum_{j=1}^{P} \Phi_j B^{i\cdot s} X_t - \displaystyle\sum_{i=1}^{p}\displaystyle\sum_{j=1}^{P} \phi_i \Phi_j B^{i + js} X_t
\end{array}
\end{equation}

And the multiplicative \acl{SMA} \acs{SMA}$(q,Q)$ model defined as:
\begin{equation}\label{eq:multiplicative_SMA_qQ}
\begin{array}{rcl}
X_t & = & (1+\displaystyle\sum_{k=1}^{q} \theta_i B^k) (1+\displaystyle\sum_{l=1}^{Q} \theta_j B^{k\cdot s}) e_t\\
X_t & = & (1+\theta_1 B-\theta_2 B^2 + ... +\theta_p B^q +\theta_1 B^s+\theta_2 B^{2s} + ... +\theta_p B^{Qs} \\
& & + \theta_1 \theta_1 B^{1+s} + \theta_2 \theta_1 B^{2+s} + \theta_2 \theta_2 B^{2+2s} + ... + \theta_p \theta_P B^{q+Qs}) e_t\\
X_t & = & e_t + \displaystyle\sum_{k=1}^{q} \theta_i B^k e_t + \displaystyle\sum_{l=1}^{Q} \theta_j B^{k\cdot s} e_t + \displaystyle\sum_{k=1}^{q}\displaystyle\sum_{l=1}^{Q} \theta_i \theta_j B^{k + js} e_t
\end{array}
\end{equation}

This directly leads to the definition of multiplicative \acs{SARIMA}$(p,d,q)(P,D,Q)_s$ as:

\begin{equation}\label{eq:multiplicative_SARIMA}
\begin{array}{rcl}
(1-\displaystyle\sum_{i=1}^{p} \phi_i B^i)(1-\displaystyle\sum_{j=1}^{P} \Phi_j B^{i\cdot s}) (1-B^d) (1-B^s)^D X_t & = & (1+\displaystyle\sum_{k=1}^{q} \theta_k B^k) (1+\displaystyle\sum_{l=1}^{Q} \theta_j B^{k\cdot s}) e_t
\end{array}
\end{equation}

The complication with this model originates from the third sum that is now also combining  all the $\phi$ and $\Phi$ in case of \acs{AR} and all the $\theta$ and $\Theta$ for \acs{MA}.

The terms of the third sum increase exponentially if both parameters are increased, which leads to an equivalent increase in compile-time. Additionally this further complicates the transformation of the system of linear equations for \acl{MA}.

For example, the equation for the approximation ${\hat{X}}_t$ of multiplicative \acs{SARIMA}$(2,0,1)(1,0,2)_{3}$ would look like:

\begin{equation}\label{eq:example_multiplicative_SARIMA_1}
\begin{array}{rcl}
\hat{X_t} & = & \phi_1 X_{t-1} + \phi_2 X_{t-2} + \Phi_1 X_{t-2} + \phi_1 \Phi_1 X_{t-3} + \phi_2 \Phi_1 X_{t-4}\\
&& + \theta_1 (X_{t-1} - {\hat{X}}_{t-1})+ \Theta_1 (X_{t-2} - {\hat{X}}_{t-2}) + \Theta_2 (X_{t-4} - {\hat{X}}_{t-4})\\
&& + \theta_1 \Theta_1 (X_{t-3} - {\hat{X}}_{t-3}) + \theta_1 \Theta_2 (X_{t-5} - {\hat{X}}_{t-5})
\end{array}
\end{equation}

Bringing this in the form required in \eqref{eq:syslinequation} assuming that the time series $X_t$ is only of length $8$:

\begin{frame}
	\footnotesize
	\medmuskip = 1mu % default: 4mu plus 2mu minus 4mu
	\begin{equation}\label{eq:example_multiplicative_SARIMA_2}
	\resizebox{\linewidth}{!}{%
		$\displaystyle
		\left(\begin{array}[c]{lllllll}
		1 & 0 & 0 & 0 & 0 & 0 \\
		\theta_1 & 1 & 0 & 0 & 0 & 0 \\
		\Theta_1 & \theta_1& 1 & 0 & 0 & 0\\
		\theta_1 \Theta_1 & \Theta_1 & \theta_1& 1 & 0 & 0\\
		\Theta_2 & \theta_1 \Theta_1 & \Theta_1 & \theta_1 & 1 & 0 \\
		\theta_1 \Theta_2 & \Theta_2 & \theta_1 \Theta_1 & \Theta_1 & \theta_1 & 1
		\end{array}\right)
		\left(\begin{array}[c]{c}
		\hat{X}_1\\
		\hat{X}_2\\
		\hat{X}_3\\
		\hat{X}_4\\
		\hat{X}_5\\
		\hat{X}_6
		\end{array}\right) =
		\left(\begin{array}[c]{lrrrr}
		0\\
		\phi_1\theta_1 X_1\\
		\phi_1\theta_1 X_2 & + \Phi_1\phi_2\theta_2 X_1\\
		\phi_1\theta_1 X_3 & + \Phi_1\phi_2\theta_2 X_2 & + \Phi_1\Theta_1 X_1\\
		\phi_1\theta_1 X_4 & + \Phi_1\phi_2\theta_2 X_3 & + \Phi_1\Theta_1 X_2 + &\Theta_2X_1\\
		\phi_1\theta_1 X_5 & + \Phi_1\phi_2\theta_2 X_4 & + \Phi_1\Theta_1 X_3 + &\Theta_2X_2 & + \theta_1\Theta_2X_1
		\end{array}\right)
		$}
	\end{equation}
\end{frame}

For the estimation of the \acs{SARIMA} coefficients $\phi$, $\Phi$, $\theta$ and $\Theta$ an objective function is needed to be optimized over. And there are different estimators that can be used for this purpose:
\begin{itemize}
	\item \acl{ML} (\acs{ML}) estimation
	\item Yule-Walker estimation
	\item Least Squares or \acl{CSS} (\acs{CSS}) method
\end{itemize}

With the Yule-Walker and the \acl{ML} estimator described in detail in Peter J. Brookwell's 2002 "Introduction to Time Series and Forecasting, Second Edition" and also being the more complicated two.

The \acl{CSS} method in contrast to that can, for $X_t$ with size $T$, be simple put as the sum of squared residuals:
\begin{equation}\label{eq:ARIMA_CSS}
\begin{array}{rcl}
ARIMA_{CSS}(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q}) = \frac{1}{2}\displaystyle\sum_{t=1}^{T} (e_t)^2 = \frac{1}{2}\displaystyle\sum_{t=1}^{T} (X_t - \hat{X}_t)^2
\end{array}
\end{equation}

\section{Optimization Algorithms}\label{optimalgorithms}
To forecast using \acs{ARIMA} the \acl{AR} and \acl{MA} coefficients have to be estimated first. This is done by solving an optimization problem:
\begin{equation}\label{eq:min}
\begin{array}{c}
\displaystyle\max_{\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q} \in R} \; g(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q})\\
or \\
\displaystyle\min_{\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q} \in R} \; g(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q})
\end{array}
\end{equation}

with $g(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q})$ being the \acl{ML}, Yule-Walker or \acl{CSS} estimator. There are different algorithms that have been developed to solve such a optimization problem. Some of the most common ones that are also provided by the general-purpose optimization function $optim()$  of R are called:
\begin{itemize}
	\item Nelder-Mead method
	\item \acl{BFGS} (\acs{BFGS}) method
	\item \acl{L-BFGS} (\acs{L-BFGS}) method
	\item Brent method
\end{itemize}

\subsection{BFGS}\label{bfgs}
The \acl{BFGS} (\acs{BFGS}) algorithm belongs to the class of \textit{Quasi-Newton} methods and can therefore be used to find roots or local maxima and minima of real-valued functions. \textit{Quasi-Newton} methods do this faster then the "full" \textit{Newton} method by approximating the Hessian or Jacobian instead of calculating it exactly.

If the function $f: R \rightarrow R$ its \textit{derivative}, $f'(x)$ and an initial guess $\hat{x}_0$ is given then the \textit{Newton} method's iterative approximation $\hat{x}_{n+1}$ of the root of $f(x)$ is given by:
\begin{equation}\label{eq:newtons_method_univariate}
	\begin{array}{c}
		\hat{x}_{n+1} = \hat{x}_{n} - f(\hat{x}_{n})/f'(\hat{x}_{n})
	\end{array}
\end{equation}

%Die Gleichung (6.36) lässt sich geometrisch interpretieren: Legen wir im Punkt hxn; g(xn)i eine Tangente an die Funktion g, so schneidet diese Tangente die x-Achse im Punkt

%Multivariate Newtons method for maxima using hessian: http://people.duke.edu/~kh269/teaching/b553/newtons_method.pdf

%Explanation multivariate newton raphson (Jacobi) method:http://fourier.eng.hmc.edu/e176/lectures/NM/node21.html

%Quasi-Netwon Method with hessian: https://www.rose-hulman.edu/~bryan/lottamath/quasinewton.pdf

%More quasi newton: https://www.cs.ccu.edu.tw/~wtchu/courses/2014s_OPT/Lectures/Chapter%2011%20Quasi-Newton%20Methods.pdf

This of course is only for univariate functions. The definition of the iterative approximation $\hat{x}_{n+1}$ of the root for multivariate function $f(x): R^k -> R$ is:

\begin{equation}\label{eq:newtons_method_multivariate_root}
	\begin{array}{c}
		\hat{x}_{n+1} =\hat{x}_{n} - \frac{f(\hat{x}_{n})}{J_f(\hat{x}_{n})} = \hat{x}_{n} - f(\hat{x}_{n}) \cdot J_f^{-1}(\hat{x}_{n})
	\end{array}
\end{equation}

With $J_f(\hat{x}_{n})$ being the Jacobian matrix, the matrix of all \textit{first-order} partial derivatives.

Using the \textit{Newton} method to optimize the function $f$ is equivalent to finding the root of the \textit{derivative}  $f'$ which means that the approximation $\vec{\hat{x}}_{n+1}$ for the extrema of $f$ is given by:

\begin{equation}\label{eq:newtons_method_multivariate_extrema}
	\begin{array}{c}
		\hat{x}_{n+1} =\hat{x}_{n} - \frac{\Delta f(\hat{x}_{n})}{H_f(\hat{x}_{n})} = \hat{x}_{n} - \Delta f(\hat{x}_{n}) \cdot H_f^{-1}(\hat{x}_{n})
	\end{array}
\end{equation}

With $H_f(\hat{x}_{n})$ being the Hessian, a square matrix of all \textit{second-order} partial derivatives.

%http://people.duke.edu/~kh269/teaching/b553/newtons_method.pdf :
The drawback of using the \textit{Newton} method is that it requires inverting the Hessian, which means a computation of $O(n^3)$ using standard techniques. Additionally the exact calculation of the Hessian takes $O(n^2)$ function evaluations (partial derivatives). Consequently using the Newton method is getting expensive for large n.

\textit{Quasi-Newton} methods try to overcome these limitations by approximating the Hessian instead of calculating it directly. This can be achieved by leveraging the \textit{secant} method.

The \textit{secant} approximation of the second derivative of the univariate function $f(x)$ is:

\begin{equation}\label{eq:secant_method_univariate}
	\begin{array}{lc}
		&f''(x_k) \approx \frac{f'(x_k) - f'(x_{k-1})}{x_k - x_{k-1}}\\
        \Leftrightarrow &f''(x_k) \cdot (x_k - x_{k-1})\approx f'(x_k) - f'(x_{k-1})
	\end{array}
\end{equation}

The generalization of \eqref{eq:secant_method_univariate} for a multivariate function is:

\begin{equation}\label{eq:secant_method_multivariate}
	\begin{array}{lc}
		\Delta^2 f(x_k) \cdot (x_k - x_{k-1})\approx \Delta f(x_k) - \Delta f(x_{k-1})
	\end{array}
\end{equation}

\textit{Quasi-Newton} methods try to find the Hessian $H_f(x_k)\approx \Delta^2f(x_k)$ to make \eqref{eq:secant_method_multivariate} an equality. For this there is an initial guess needed for $H_0 = I$ which is then incrementally improved by updating $H_{k+1}$.

Using this approach the Hessian $H_f(x_k)$ still needs to be inverted which is also a time consuming computation, that can be avoided by instead approximating the inverse Hessian $H_f^{-1}(x_k)$ directly.

In that case we define $B_k$ as the inverse Hessian $H_f^{-1}(x_k)$ and the search direction $p_k$ that has to be computed are defined as:
\begin{equation}\label{eq:searchdirection}
	\begin{array}{lc}
		p_k = -B_k \cdot \Delta f(x_k)
	\end{array}
\end{equation}

To calculate the updated approximation of $B_k$ an acceptable step size $\alpha$ in the direction $p_k$ has to be found by using a line search:
\begin{equation}\label{eq:linesearch_objfunction}
	\begin{array}{lc}
		\displaystyle \min_\alpha \;\;\; h(\alpha) = f(x_k + \alpha p_k)
	\end{array}
\end{equation}

An exact line search algorithm would determine a value for $\alpha$ that exactly minimizes $h(\alpha)$. However this is not always necessary or even desirable because of the additional computing costs it would require.
Instead using, for example the backtracking line search, the step size $\alpha$ can be approximately reasonably well, which is sufficient for most cases.

The backtracking line search starts the same way an exact one would by guessing $\alpha_0 > 0$ and then shrinking it in every iteration by multiplying it with a constant $r \in ]0,1[$:
\begin{equation}\label{eq:linesearch_ak}
	\begin{array}{lc}
		\alpha_{k+1} = r \cdot \alpha_k
	\end{array}
\end{equation}

This is repeated as long as the Armijo-Goldstein condition is fulfilled, which tests whether the new smaller step size  achieves a adequately corresponding decrease in the objective function $h(\alpha)$ defined in \eqref{eq:linesearch_objfunction}.
The Armijo-Goldstein condition is fulfilled if:
\begin{equation}\label{eq:linesearch_armijo-goldstein}
	\begin{array}{lc}
		f(x+\alpha p) <= f(x) + a c m
	\end{array}
\end{equation}

With $m= p^T \Delta f(x)$ and $c\in]0,1[$ being a pre defined control parameter.

Given $p_k$ and $\alpha_k$ the update formula of \acl{BFGS} algorithm for $B$ is defined by:
\begin{equation}\label{eq:lbfgs_update_1}
	\begin{array}{lc}
		B_{k+1} = B_k + \frac{(s_k^T y_k + y_k^T B_k y_k)(s_k s_k^T)}{(s_k^T y_k)^2} - \frac{B_k (s_k^T y_k + s_k y_k^T)}{s_k^T y_k}
	\end{array}
\end{equation}

With $y_k = \Delta f(x_{k+1}) - \Delta f(x_k)$ and $s_k = x_{k+1}-x_k = a_k \cdot p_k$.

The only thing left to be able to optimize using \acs{BFGS} is the gradient $\Delta f(x)$. This can either be done by approximating using finite differencing or by calculating the exact partial differentials of $f(x)$.

To get more accurate results calculating the partial differentials is preferred. In case of minimizing the objective function $ARIMA_{CSS}$ defined in \eqref{eq:ARIMA_CSS} and $\hat{X}_t$
\begin{equation}\label{eq:ARIMA_CSS_long}
	\begin{array}{rrlll}
    	\hat{X}_t &=& \displaystyle\sum_{i=1}^{p} \phi_i B^i X_t &+ \displaystyle\sum_{j=1}^{P} \Phi_j B^{i\cdot s} X_t &- \displaystyle\sum_{i=1}^{p}\displaystyle\sum_{j=1}^{P} \phi_i \Phi_j B^{i + js} X_t \\
        &+& \displaystyle\sum_{k=1}^{q} \theta_i B^k e_t &+ \displaystyle\sum_{l=1}^{Q} \theta_j B^{k\cdot s} e_t &+ \displaystyle\sum_{k=1}^{q}\displaystyle\sum_{l=1}^{Q} \theta_i \theta_j B^{k + js} e_t
	\end{array}
\end{equation}

the partial differentials are as follows:
\begin{equation}\label{eq:gradient_arima_phi}
	\begin{array}{lcl}
		\frac{\delta}{\delta \phi_n} ARIMA_{CSS}(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q}) &=& \frac{1}{2}\displaystyle\sum_{t=1}^{T} 2 \cdot (X_t - \hat{X}_t) \cdot (\frac{\delta}{\delta \phi_n} (X_t - \hat{X}_t))\\
        &=& \displaystyle\sum_{t=1}^{T} e_t \cdot  (-B^n X_t +  \displaystyle\sum_{j=1}^{P} \Phi_j B^{n + js} X_t)
	\end{array}
\end{equation}

\begin{equation}\label{eq:gradient_arima_theta}
	\begin{array}{lcl}
		\frac{\delta}{\delta \theta_n} ARIMA_{CSS}(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q}) &=& \displaystyle\sum_{t=1}^{T} e_t \cdot  (-B^n e_t - \displaystyle\sum_{l=1}^{Q} \Theta_j B^{n + ls} e_t)
	\end{array}
\end{equation}

\begin{equation}\label{eq:gradient_arima_Phi}
	\begin{array}{lcl}
		\frac{\delta}{\delta \Phi_n} ARIMA_{CSS}(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q}) &=& \displaystyle\sum_{t=1}^{T} e_t \cdot  (-B^{n \cdot s} X_t + \displaystyle\sum_{i=1}^{p} \phi_i B^{i + ns} X_t)
	\end{array}
\end{equation}

\begin{equation}\label{eq:gradient_arima_Theta}
	\begin{array}{lcl}
		\frac{\delta}{\delta \Theta_n} ARIMA_{CSS}(\phi_{1..p}, \Phi_{1..P}, \theta_{1..q}, \Theta_{1..Q}) &=& \displaystyle\sum_{t=1}^{T} e_t \cdot  (-B^{n \cdot s} e_t - \displaystyle\sum_{k=1}^{q} \theta_j B^{k +ns} e_t)
	\end{array}
\end{equation}

%\subsection{Nelder-Mead}\label{neldermead}

\section{Solvers of Linear Systems}\label{linsys_solvers}

The hardest part of calculating an approximation $\hat{X}_t$ for the \acs{ARIMA} model is finding a solution for the system of linear equation depicted in the equations \eqref{eq:example_MA_three_system_1} and \eqref{eq:example_multiplicative_SARIMA_2}.
And again there are different approaches that can be taken. The first differentiation is between exact and numerical methods. One example for an exact one would be the \textit{ Gaussian Elimination} solver. However, the complexity of this algorithm is $O(n^3)$ which means that is not suitable for large systems of equations. And especially for \acs{ARIMA} this is almost always the case, because the system of linear equations will always be equal to the size of the time series used to train the model. So instead an iterative approach is used. The \textit{Jacobi} method is one of these and it is commonly used, because of its simplicity and robustness. Additionally each iteration is quite fast.

\subsection{Jacobi} \label{jacobi}
Given a system of linear equations given in the form of the equation \eqref{eq:syslinequation} that can also be express as:

\begin{equation}\label{eq:syslineqaution_long}
	\begin{array}{lcl}
		\displaystyle\sum_{j=1}^{n} a_{ij} \cdot x_j = b_i
	\end{array}
\end{equation}

Using the \textit{fixed-point iteration} method this equation \eqref{eq:syslineqaution_long} can be transformed to be:

\begin{equation}\label{eq:syslineqaution_fixedpoint}
	\begin{array}{lrcl}
		&\displaystyle\sum_{j=1}^{n} a_{ij} \cdot x_j &=& b_i\\
        \Leftrightarrow & a_{ii} \cdot x_i \displaystyle\sum_{\substack{j=1 \\ j\neq i}} a_{ij} \cdot x_j &=& b_i\\
        \Leftrightarrow & a_{ii} \cdot x_i  &=& b_i \displaystyle\sum_{\substack{j=1 \\ j\neq i}} a_{ij} \cdot x_j\\
        \Leftrightarrow & x_i  &=& a_{ii} - (b_i \displaystyle\sum_{\substack{j=1 \\ j\neq i}} a_{ij} \cdot x_j)
	\end{array}
\end{equation}

Leading directly to the update function for the \textit{Jacobi} method:
\begin{equation}\label{eq:jacobi_update}
	\begin{array}{lrcl}
		x_i^{(n+1)}  = a_{ii} - (b_i \displaystyle\sum_{\substack{j=1 \\ j\neq i}} a_{ij} \cdot x_j^{(n)})
	\end{array}
\end{equation}

%\subsection{Conjugate Gradient}\label{cg}

\section{SystemML}
SystemML is Machine Learning Platform built for large scale analytics. It enables flexible and scalable machine learning while also accelerating exploratory algorithm development.

It accomplishes this by providing the \acl{DML} (\acs{DML}). It can either be written in the default R-Like ("DML") or a Python-Like ("PyDML") syntax.

The goal of \acs{DML} is to automatically scale any algorithm by translating all the instruction within the script into a set of \textit{Spark} \acs{API} calls so it can be run on a cluster in multiple nodes if necessary. Beforehand SystemML also uses code optimization methods to remove dead code and common sub expressions.

Each script is optimized based on data and cluster characteristics, which means that the script is not only optimized once, but a second time as soon as all free variables - or rather parameters the script can be run with - are known and therefore for example all the sizes of the matrices needed can be calculated. Using this information combined with the details about cluster are used to optimize the script further by for example calculating the number of nodes needed to run the script.

DML and PyDML scripts can be run in different modes. Either in Spark, Hadoop or Standalone mode. Additionally it can also be access via Scala or Python to be used in a Spark Shell, Jupyter or Zepplin Notebook. This enables easy and fast algorithm development in a well established development environment for ML.


\documentclass[runningheads, a4paper, oribibl]{llncs}

\setcounter{tocdepth}{3}
\usepackage{graphicx}
\usepackage{float}
\graphicspath{{../images/}}
\usepackage{epstopdf}
\usepackage{standalone}
\usepackage{xcolor}
\usepackage{tikz}
\usetikzlibrary{fit}
\usetikzlibrary{shapes,snakes,calc}

\usepackage{listings, color}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  language=Matlab,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=false,
  tabsize=2,
  numbers=left,
  numbersep=5pt,
  title=\lstname
}

\usepackage[section]{placeins}

\usepackage{amsmath,amssymb, cancel}
%
\usepackage{url}
\urldef{\mailsa}\path|201501005@daiict.ac.in|
\urldef{\mailsb}\path|201501422@daiict.ac.in|
\newcommand{\keywords}[1]{\par\addvspace\baselineskip
\noindent\keywordname\enspace\ignorespaces#1}

\renewcommand\thesubsection{\thesection(\alph{subsection})}

\begin{document}

\mainmatter

\title{High Performance Computing Report}

\titlerunning{High Performance Computing Report}

\author{Amarnath Karthi\\Chahak Mehta}%
%
\authorrunning{Amarnath Karthi \& Chahak Mehta}
\institute{Dhirubhai Ambani Institute of Information and Communication Technology\\
  \mailsa\\
  \mailsb\\
}

\maketitle
\section{Implementation Details (Basic matrix multiplication)}
\subsection{Brief and clear description about the Serial implementation}
The serial implementation involves basic matrix multiplication using 3 nested loops.
\begin{equation}
    C_{ij} = \sum A_{ik}B_{kj}
\end{equation}
\subsection{Brief and clear description about the implementation of the approach (Parallelization Strategy, Mapping of computation to threads)}
The parallelization is achieved by load distribution of the middle loop over several processors, using the OpenMP library. We will be using the \textbf{omp parallel for directive}.
\section{Complexity and Analysis}
\subsection{Complexity of serial code}
Cubic time complexity : $O(n^3)$.
This is because of the 3 nested loops, each having an $O(n)$ runtime complexity by itself.
\subsection{Theoretical Speedup (using asymptotic analysis, etc.)}
For an n core system, the theoretical speedup is approximately n.
\newpage
\section{Curve Based Analysis}
\subsection{Time Curve related analysis (as no. of processor increases)}
\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{p1.png}
    \caption{Execution time for simple matrix multiplication}
    \label{fig:p1}
\end{figure}
For small problem sizes, the overhead of adding 1 extra core to the solution outweighs the advantages given by it. Thus for small input sizes, the execution time is higher for a higher number of cores. As the problem size increases, the overhead becomes insignificant in front of the compute time, which decreases drastically when a larger number of cores are used. This is quite natural because the work is shared equally amongst all processors.

\subsection{Speedup Curve related analysis (as problem size and no. of processors increase)}
The speedup is bound by "n", the number of cores. Therefore, irrespective of the problem size, the speedup is always lesser than n for n cores. For small problem sizes, the speedup is less than one if we use more cores. This again indicates that the load is too small to be parallelized efficiently. For very large problem sizes we see almost a constant speedup. This indicates that saturation has been achieved.  For 4 cores and a problem size of 512,  we get a speedup of approximately 3.2, whereas for 2 cores we get a speedup of 1.78.
\newpage
\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{p2.png}
    \caption{Speedup vs problem size}
    \label{fig:p2}
\end{figure}
\begin{figure}[b]
    \centering
    \includegraphics[width=\textwidth]{p3.png}
    \caption{Efficiency vs problem size}
    \label{fig:p3}
\end{figure}
\newpage

\section{Implementation Details(block matrix multiplication)}
\subsection{Brief and clear description about the Serial implementation}
This is a divide and conquer approach on traditional matrix multiplication. We recursively divide each matrix into smaller and smaller blocks, and perform multiplication and summation operations on them.
\subsection{Brief and clear description about the implementation of the approach (Parallelization Strategy, Mapping of computation to threads)}
Initially we distribute a different set of blocks to each thread. In the combine step of a level, the results of 2 threads are added into one matrix, and one thread hands over its entire data to another and dies. This step goes on until there is one thread surviving. This thread will hold the final result of the matrix multiplication.
\section{Complexity and Analysis}
\subsection{Complexity of serial code}
Cubic complexity $O(n^3)$
\subsection{Theoretical Speedup (using asymptotic analysis, etc.)}
n, where n is the total number of processors.
\section{Curve Based Analysis}
\subsection{Time Curve related analysis (as no. of processor increases)}
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{p4.png}
    \caption{Execution time for simple matrix multiplication}
    \label{fig:p4}
\end{figure}
For small problem sizes, the overhead of adding 1 extra core to the solution outweighs the advantages given by it. Thus for small input sizes, the execution time is higher for a higher number of cores. As the problem size increases, the overhead becomes insignificant in front of the compute time, which decreases drastically when a larger number of cores are used. This is quite natural because the work is shared equally amongst all processors.
\subsection{Speedup Curve related analysis (as problem size and no. of processors increase)}
The speedup is bound by "n", the number of cores. Therefore, irrespective of the problem size, the speedup is always lesser than n for n cores. For small problem sizes, the speedup is less than one if we use more cores. This again indicates that the load is too small to be parallelized efficiently. For very large problem sizes we see almost a constant speedup. This indicates that saturation has been achieved.  For 4 cores and a problem size of 512,  we get a speedup of approximately 2.09, whereas for 2 cores we get a speedup of 1.28.
\newpage
\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{p5.png}
    \caption{Speedup vs problem size}
    \label{fig:p5}
\end{figure}
\begin{figure}[b]
    \centering
    \includegraphics[width=\textwidth]{p6.png}
    \caption{Efficiency vs problem size}
    \label{fig:p6}
\end{figure}
\newpage
\end{document}


\documentclass[11pt, twoside, withdegree]{bhthesis}

\usepackage[ngerman, english]{babel}

\newcommand{\OEIS}[1]{\text{\href{https://oeis.org/#1}{{\small \tt (#1)}}}}
\DeclarePairedDelimiter{\abs}{\lvert}{\rvert}
\DeclarePairedDelimiter{\iverson}{\llbracket}{\rrbracket}

\title{Eine supertolle Masterarbeit}
\author{Maria Musterfrau}

\reporttype{Masterarbeit}
\studname{Angewandte Informatik}
\degree{Diplom-Ingenieurin}

\involvedpeople{
  \person{0.45\linewidth}{Betreuerin}{
    \hbox{Univ.-Prof.\ Dr.\ Alexandra Musterfrau}\\
    Institut für Angewandte Informatik\\
    Alpen-Adria-Universität Klagenfurt
  }\hfill
  \person[\flushright]{0.45\linewidth}{Gutachter}{
    Dr.\ Erich Mustermann\\
    Institut für Mathematik\\
    Technische Universität Graz\\
  }\\[2em]
}

\university{
  \hfill\includegraphics{aau-logo.pdf}\\[1em]
}
\universityname{Alpen-Adria-Universität Klagenfurt}
\fakultaetname{Fakultät für Technische Wissenschaften}

\begin{document}
\selectlanguage{ngerman}
\maketitle

\selectlanguage{english}
\chapter*{Abstract}
This is a short summary of the contents of this thesis.

\tableofcontents

\chapter{Introduction}\label{chap:intro}

\section{Tasty!}

\begin{definition}[Open Set]
  A set $\Omega\subseteq\mathbb{C}$ is said to be \emph{open}, if for every
  $z_{0}\in\Omega$ there is a positive number $\varepsilon > 0$ such
  that for all $z\in\mathbb{C}$ that satisfy $\abs{z - z_{0}} <
  \varepsilon$ we find $z\in\Omega$.
\end{definition}

\begin{theorem}[Cauchy's Integral Theorem]\label{thm:cauchy}
  Let $\Omega\subseteq\mathbb{C}$ be an open and simply connected
  set. Let $\gamma:[0,1]\to \Omega$ be a closed path in $\Omega$, and
  let $f\colon\gamma^{*} \to \mathbb{C}$ be a holomorphic function.
  Then the relation
  \begin{equation}\label{eq:cauchy-integral}
    \oint_{\gamma} f(z)~dz = 0
  \end{equation}
  holds.
\end{theorem}

The following example provides verification for this important result.

\begin{example}
  Consider $f\colon \mathbb{C} \to \mathbb{C}$ with $z\mapsto
  z^{2}$. Then, by Theorem~\ref{thm:cauchy} we find
  \[ \oint_{\abs{z} = 42} f(z)~dz = 0.  \]
  In this case, it is not too difficult to verify that the theorem
  holds by straightforward computation of the line integral; we use
  the curve $\gamma\colon [0,1]\to \mathbb{C}$ with $\gamma(t) =
  42\cdot\exp(2\pi i t)$. Straightforward computation yields
  \begin{align*}
    \oint_{\abs{z} = 42} f(z)~dz
    &= \int_{0}^{1} f(\gamma(t)) \gamma'(t)~dt\\
    &= 42^{3}\cdot 2\pi i \cdot \biggl[\frac{\exp(6 \pi i t)}{6 \pi
      i}\biggr]_{0}^{1} = 42^{3}\cdot 2\pi i\cdot \frac{1 - 1}{6\pi
      i} = 0,
  \end{align*}
  which verifies the theorem.
\end{example}

\begin{theorem*}
  Theorems do not have to be numbered, but they can stretch over
  multiple lines and maybe even over to the next page. A very nice
  formula is
  \[ e^{\pi i} + 1 = 0, \]
  and it should help to illustrate a page break.
\end{theorem*}

\section{Improvements?}

Feel free to adapt / polish the styling suggested by this template
in any way you like. This template is hosted at
\url{https://github.com/behackl/thesis-template} -- I am happy to
discuss ideas and suggestions for general improvement of this template.

\end{document}


\documentclass[main.tex]{subfiles}
\begin{document}

\marginpar{Monday\\ 2020-4-20, \\ compiled \\ \today}

% The Fourier transform is stochastic since the noise is stochastic: however, the PSD encompasses the statistical properties of the signal in a way that is stationary and well-defined.

A physical system is a \textbf{functional} \(F\) which transforms one or many input time series \(i_j (t)\) into one or many output time series \(o_j (t) = F(i_j(t))\).

In principle, any output time series at any time could be a function of any input time series at any time.
However, real systems are causal: there cannot be causality going backward in time, so \(o(t_0 ) = \eval{F (i(t))}_{t \leq t_0 }\).

% Also, often we can approximate systems as linear ones. as long as we work near a single point.
% Also, we can sometimes approximate them as stationary.
We can also make two assumptions: that the functional \(F\) is \textbf{linear} --- this is justified as long as we are always working near a fixed point, so that the higher order terms are negligible; and that the functional \(F\) is \textbf{stationary}: this means that it is invariant under translations \(t \to t + a\).

Under all of these assumptions, we can express the effect of the system through an \textbf{impulse response function} \(h\), defined so that:
%
\begin{align}
o(t) = F \qty[\int \dd{\widetilde{t}} i (\widetilde{t}) \delta(t - \widetilde{t})]
= \int \dd{\widetilde{t}} i(\widetilde{t}) F( \delta (t - \widetilde{t}))
= \int \dd{\widetilde{t}} i(\widetilde{t}) h(t - \widetilde{t})
\,.
\end{align}

We can say that \(F[i(\widetilde{t}) \delta] = i(\widetilde{t}) F[\delta ]\) because \(t'\) is fixed inside the integral, so \(i(\widetilde{t})\) is just a constant.

We used stationarity to write \(h(\widetilde{t}, t)\) as \(h( \widetilde{t} - t)\); also, causality tell us that the IRF \(h(\tau )\) must satisfy \(h(\tau ) = 0\) for \(\tau < 0\).

The expression for the output as a function of the input is a convolution, so in Fourier space it is a product;
%
\begin{align}
o(\omega ) = i(\omega ) h(\omega )
\,.
\end{align}

The power spectral density then transforms as \(S_o (\omega ) = \abs{h(\omega )}^2 S_i (\omega )\), and if we have systems in series we can just multiply the impulse responses together, like
%
\begin{align}
o (\omega ) = i(\omega )\prod_{j = 1}^{N} h_j (\omega )
\,.
\end{align}

\subsection{Sampling}

Often we sample signals digitally.
Analogic systems can be faster, but electronics are getting very fast as well, and they are easier to use.

The signal is quantized in two ways: we quantize both in time by sampling at an interval \(t_s\) and in amplitude, by encoding it with a finite number of bits.
This introduces noise, which is well-known and easy to calculate.
We still need to do Fourier analysis, but we will use a discrete Fourier transform.

\subsubsection{Aliasing}

If we have a signal at a frequency \(f\), and we want to reconstruct it, we need to sample at a frequency \(\geq 2f\).

If we sample at \SI{100}{Hz}, we can only accurately describe signals up to \SI{50}{Hz}.
This is the \textbf{Nyquist-Shannon sampling theorem}.

This is true if we want to fit the data with the slowest sinusoid possible; if we know in which frequency range we should look we can try to fit higher-frequency sinusoids but this is risky business.
If we work below the Nyquist frequency we can be sure of each frequency we see.

\section{Resonant bar detectors}

\subsection{Two paths to GW detection}

Most modern and planned GW detectors operate by constructing \textbf{free-falling masses}: ground-based interferometers bounce signals off of suspended mirrors, space based ones have masses in actual geodesic motion.
These detectors are \emph{broad-band}, which is useful, but each frequency has to be detected at its native amplitude with no amplification.
They can be made on \emph{large scales}, on the order of \SI{e3}{m} on Earth, \SI{e9}{m} in space. This is very useful scientifically, since then the GW-induced displacement is larger; however it requires a lot of infrastructure and investment.
These must then be built and operated by large collaborations, with hundreds of people at least.

Another option, which was quite popular a few years ago, is to use an \textbf{elastic body} which resonates at a specific frequency.
This might \emph{enhance} the effect of a GW through resonance and \emph{extend} the duration of burst signals.

However, this kind of detector is only sensitive \emph{around its resonant frequency}.
Also, since it extends the signal it is hard to precisely reconstruct the \emph{temporal profile} of the signal.

These need to be isolated solid objects: they will fit in a lab (at scales of \SI{e1}{m} at most), but the GW-induced displacements will be small.

% On the other side, we have \textbf{interferometers} which measure the distance between free-falling masses.

\subsection{Harmonic oscillators and GW}

\subsubsection{Harmonic oscillators}

Suppose we have a perfect harmonic oscillator with a time-dependent rest position \(x_0(t) \) and a time-dependent external force \(F _{\text{ext}} (t)\): its evolution will be determined by the differential equation
%
\begin{align}
m \ddot{x} = -k (x(t) - x_0 (t) ) + F _{\text{ext}} (t)
\,,
\end{align}
%
which in Fourier space can be written as
%
\begin{align}
- m \omega^2 x(\omega ) &= - k (x(\omega ) - x_0 (\omega )) + F _{\text{ext}} (\omega )  \\
x(\omega ) &= \frac{k x_0 (\omega ) + F _{\text{ext}}(\omega )}{k - m \omega^2} = \frac{k x_0 (\omega ) + F _{\text{ext}}(\omega )}{k \qty(1 - \omega^2 / \omega_0^2)}  \\
&= \underbrace{\frac{\omega_0^2}{1 - \omega^2 /\omega_0^2}}_{H_{x_0 }(\omega )}
x_0 (\omega ) +
\underbrace{\frac{F _{\text{ext}} (\omega )}{k (1 - \omega^2 / \omega_0^2)} }_{H_{F _{\text{ext}}} (\omega )}
F _{\text{ext}}(\omega )
\,,
\end{align}
%
where we defined \(\omega_0 = \sqrt{k / m}\) and the two transfer functions \(H_{x_0 }\) and \(H_{F _{\text{ext}}}\).

This diverges for \(\omega = \omega_0 \); but let us consider the effect of \textbf{velocity damping}: we add a term \(- \beta \dot{x}\) to the RHS of the differential equation,
%
\begin{align}
m \ddot{x} = - k \qty(x(t) - x_0 (t)) - \beta \dot{x}(t) + F _{\text{ext}}(t)
\,,
\end{align}
%
which in Fourier space becomes:
%
\begin{align}
x(\omega ) = \frac{k x_0 (\omega ) + F _{\text{ext}}(\omega )}{k \qty(1 - \qty( \frac{\omega}{\omega_0 })^2 - \frac{i \omega \beta }{k})}
\,,
\end{align}
%
since every derivative becomes \(- i \omega \).

Another kind of damping we can have is called \textbf{structural internal damping}, which means modifying the differential equation as:
%
\begin{align}
m \ddot{x} = - k (1 + i \delta ) \qty(x(t) - x_0 (t)) + F _{\text{ext}}(t)
\,;
\end{align}
%
concretely speaking this means that there is some \emph{delay} between the action of the force and the response of the system. In Fourier space, this means
%
\begin{align}
x(\omega ) = \frac{k x_0 (\omega ) + F _{\text{ext}}(\omega )}{k \qty(1 - \qty( \frac{\omega}{\omega_0 })^2 + i \delta )}
\,,
\end{align}
%
so, since both terms add a purely imaginary constant to the denominator, we encapsulate them into a term \(i / Q\), for an arbitrary \(Q\).

This \(Q\) quantifies damping (inversely: large \(Q\) means small damping). For non-infinite values of \(Q\), the transfer function does not diverge.
% The transfer function is more peaked for less damping.

\subsubsection{GW interactions}

How do we see the effect of GW on an elastic body?
Consider two masses, which start out free-falling, and connect them by a spring: they now will not move along geodesics.
If we move to the \textbf{proper detector frame}, the effect of a GW can be described as a Newtonian force on the test masses, which together with the reaction of the spring determines the motion of the system:
%
\begin{align}
F _{\text{GW}} - k (L - \Delta x) = m \Delta \ddot{x}
\,,
\end{align}
%
where the force is given by (equation \eqref{eq:geodesic-deviation-detector-frame} multiplied by \(m\)):
%
\begin{align}
F_{\text{GW}} = \frac{m}{2} \ddot{h}^{TT}_{xx} \Delta x \approx \frac{m}{2} L \ddot{h}^{TT}_{xx}
\,.
\end{align}

Since we can only see \(h_{xx}\), we are only sensitive to the \(h_{+}\) polarization: this is not surprising, since our detector is one-dimensional.

Note that this expression is only valid as long as we are in the short arm approximation: \(L \ll \lambda _{\text{GW}}\), which means \(f _{\text{GW}} \ll c/ L \approx \SI{3e8}{Hz}\), if \(L \approx \SI{1}{m}\).

Intuitively, what the equation is describing is the force of the GW competing with the intrinsic one of the oscillator to move the mass.

The oscillator was a convenient approximation to give an idea of the system, but really for our detector we will use a continuous \textbf{resonant bar}; we can describe its movement by introducing the variable \(u(x, t)\), which denotes the displacement from equilibrium at a certain point (still in only \emph{one dimension}). The dynamics of the bar can be shown to obey the law
%
\begin{align}
\dd{m} \qty(\pdv[2]{u}{t} - v_s^2 \pdv[2]{u}{x}) = \dd{F_x} = \dd{m} \frac{1}{2} x \ddot{h}_{xx}^{TT}
\,,
\end{align}
%
where \(v_s\) is the speed of sound in the medium.
\todo[inline]{Lagrangian or Eulerian?}

We assume that the ends of the bar are kept stationary:
%
\begin{align}
\eval{\pdv{u}{x}}_{x = \pm L/2} = 0
\,.
\end{align}

The general solution will be given by a sum of sines and cosines, but the cosines will move the center of the bar.

\todo[inline]{They will, but we imposed the ends being stationary, not the center (and why should the center be stationary)! Why should we not use that condition instead? It works just as well, since \emph{cosines} satisfy the condition of being zero at \(\pm L/2\), and we get cosines from the first derivative of sines.}

Keeping only the physical sines we will then have the harmonic decomposition
%
\begin{align}
u(t, x) = \sum _{n=0}^{ \infty } \xi_{n} \sin( \frac{\pi x}{L} (2 n + 1))
\,,
\end{align}
%
which we can plug into the differential equation: computing the derivatives explicitly we find
%
\begin{align}
\sum _{n=0}^{ \infty } \bigg(\ddot{\xi}_{n} + \underbrace{\qty( \frac{v_s \pi (2n+1)}{L})^2}_{\mathclap{\omega_{n}^2}}\bigg) \sin( \frac{\pi x}{L} (2 n+1)) = \frac{1}{2} x \ddot{h}^{TT}_{xx}
\,,
\end{align}
%
which, in \(L^2\) space, is in the form \(\sum _{n} c_n \hat{e}_{n} = \vec{v}\), where \(\hat{e}_n\) are orthogonal basis vectors while \(\vec{v}\) is a vector (recall that \(\ddot{h}_{xx}^{TT}\) is approximately  constant with respect to \(x\), but it is multiplied by \(x\)).
In order to solve it, we take its scalar \(L^2\) product with an arbitrary basis vector, which amounts to multiplying by another sinusoid and integrating.

The sinusoids \(\hat{e}_{n} = \sin((2n+1) \pi x / L)\) are not orthonormal, they instead satisfy \(\hat{e}_n \cdot \hat{e}_{m} = (L/2) \delta_{nm}\) as can be checked by direct computation. On the other side of the equation we find
%
\begin{align}
\hat{e}_{m} \cdot \vec{v} &= \frac{1}{2} \ddot{h}^{TT}_{xx} \int_{-L/2}^{L/2}  \dd{x} x \sin( \frac{\pi x}{L} (2 m + 1))  \\
&= \frac{1}{2} \ddot{h}^{TT}_{xx} \frac{L^2}{\pi^2 (2m+1)^2} \underbrace{\eval{\sin( \frac{\pi x}{L} (2 m + 1))}_{-L/2}^{L/2}}_{= 2 (-)^{m}} \marginnote{The indefinite integral also has a term like \(x \cos(x)\), which is odd and vanishes.}
\,,
\end{align}
%
so the final equation reads:
%
\begin{align}
\ddot{\xi}_{n} + \omega^2_{n} \xi_{n} = \frac{(-)^{n}}{(2n+1)^2} \frac{2L}{\pi^2} \ddot{h}^{TT}_{xx}
\,.
\end{align}

We have eliminated (``integrated out'') the spatial part: we can analyze the time evolution by itself.

\end{document}


\documentclass[main.tex]{subfiles}
\begin{document}

\subsection{Sychrotron absorption}

\marginpar{Tuesday\\ 2020-9-1, \\ compiled \\ \today}

We have seen what the total absorption coefficient (accounting for stimulated emission) is for a two-level system.

We want to use this expression in order to evaluate the total absorption due to the synchrotron process: so, we need to generalize it to a system with a continuum of energy levels, the free particle states with arbitrary velocity.

We will approach this by discretizing the space of possible energies of the particle.
There is a slight complication, in that for a given energy \(h \nu \) there are now \emph{many} pairs of levels having that energy between them. We will need to sum over them, still denoting all the higher-energy states in the pairs by ``2'' and the lower-energy states in the pairs by ``1''.
In this discretized description, then, we shall have an expression like
%
\begin{align}
\alpha _\nu  = \frac{h \nu }{4 \pi } \sum _{E_1 } \sum _{E_2 }
\qty[n(E_1 ) B_{12} - n(E_2 ) B_{21} ] \phi_{12}
\,,
\end{align}
%
where the sum is performed across all the possible energy levels \(E_1 \) and \(E_2 \) such that \(E_2 - E_1 = h \nu \), while \(n(E_1 )\) and \(n(E_2 )\) are the respective number densities of the two energy states.
On the other hand, \(\phi_{12} \) is the transition width.

The result we derived earlier was found by assuming that emission and absorption are isotropic; this is not true anymore since the magnetic field offers a preferential direction.
However, we can work around this problem by assuming that the magnetic field is ``tangled'', so that the direction of its value at a randomly chosen point in space is essentially uniformly distributed along the sphere.

We can also quickly evaluate the emission coefficient \(j_\nu \) (which is the power emitted per unit volume, solid angle and frequency): if the power per unit frequency emitted by a single electron is \(P(E, \nu )\) then the coefficient is
%
\begin{align}
j_\nu = \frac{ \dd{w}}{ \dd{t} \dd{\nu }} \frac{n(E)}{4 \pi }
= P(E, \nu ) \frac{n(E)}{4 \pi }
\,,
\end{align}
%
where \(n(E)\) is the density of electrons at a specific energy \(E\).
Note that this is not a differential quantity since we discretized the energy levels, so that they are countable.
We are also assuming isotropicity, by the same reasoning as before.

The coefficient \(j_\nu \) can also be expressed in terms of the Einstein coefficients, as
%
\begin{align}
j_\nu = \frac{h \nu }{4 \pi } \phi_{21}  (\nu ) n_2 A_{21}
\,,
\end{align}
%
which like before can be generalized to the situation in which we have many possible energy levels giving rise to the same transition as
%
\begin{align}
j_\nu = \sum _{E_1 } \frac{h \nu }{4 \pi }  \phi_{21} n_2 A_{21}
= \frac{h \nu }{4 \pi } n_2  \sum _{E_1 } \phi_{21} A_{21}
= \frac{n_2 }{4 \pi } P(E_2, \nu  )
\,,
\end{align}
%
meaning that now we know
%
\begin{align}
P(E_2 , \nu ) = h \nu \sum _{E_1} \phi_{21} A_{21}
\,.
\end{align}

From the \emph{detailed balance} relations we can also relate the Einstein coefficients as
%
\begin{align}
A_{21} = \frac{2 h \nu^3}{c^2} B_{21}
\,,
\end{align}
%
which we can substitute in, to find
%
\begin{align} \label{eq:spectral-power-from-einstein-B}
P(E_2 , \nu ) = h \nu \qty( \frac{2 h \nu^3}{c^2}) \sum _{E_1} \phi_{21} B_{21}
\,.
\end{align}

The reasoning we went to these great lengths to express the spectral power as a function of the Einstein coefficients is that we hope to invert the relation we found, giving us the coefficients as a function of the spectral power.

We have an expression for \(\alpha_\nu \) in terms of the coefficients \(B_{21} \) and \(B_{12} \); however we know from the detailed balance relations that, as long as the statistical weights of the energy levels are equal (which holds for us since we are considering free states), \(B_{21} = B_{12} \), so we can write the expression as
%
\begin{align}
\alpha _\nu &=  \frac{h \nu }{4 \pi } \sum _{E_1 } \sum _{E_2 } B_{21} \qty(n(E_1 ) - n(E_2 ) ) \phi_{21} \\
&= \frac{h \nu }{4 \pi }
\sum _{E_2 } \qty(n(E_2 - h \nu ) - n(E_2 ))
\sum _{E_1 } B_{21} \phi_{21} \marginnote{Used \(E_2 - E_1 = h \nu \).}  \\
&= \frac{h \nu }{4 \pi }
\sum _{E_2 } \qty(n(E_2 - h \nu ) - n(E_2 ))
\frac{P(E_2, \nu )}{h \nu \frac{2 h \nu^3}{c^2}}  \marginnote{Used equation \eqref{eq:spectral-power-from-einstein-B}.}\\
&= \frac{c^2}{8 \pi h \nu^3 }
\sum _{E_2 } \qty(n(E_2 - h \nu ) - n(E_2 ))
P(E_2 , \nu )
\,.
\end{align}

This expression makes some intuitive sense: in order to see how much radiation is absorbed at a frequency \(\nu \) we consider all the pairs of levels with that gap, and for each multiply  the difference of electrons in the low vs high state by the power of the synchrotron process at that energy and frequency.

This works well for a system which we can discretize: however, if we want to use the language of the continuum of states we will need to turn the sum into an integral.
In order to do this, we need to introduce the distribution function of electrons in momentum space, \(f(\vec{p})\), which we assume to be isotropic. Using it, we write
%
\begin{align}
\alpha _\nu =
\frac{c^2}{8 \pi h \nu^3}
\int \dd[3]{p_2 } \qty[ f(\overline{p}_2 )- f(p_2 )]P (E_2 , \nu )
\,,
\end{align}
%
where \(p_2 \) is the modulus of the momentum associated with \(E_2 \), and similarly \(\overline{p}_2\) is associated with \(E_1 = E_2 - h \nu \).

Let us apply this for the well-known case in which the electron distribution is thermal, so that
%
\begin{align}
f(\vec{p}) = k \exp(- \frac{E}{k_B T})
\,,
\end{align}
%
which means that
%
\begin{align}
f(\overline{p}_2) - f(p_2) &= k \qty(\exp(- \frac{E - h \nu }{k_B T}) - \exp(- \frac{E}{k_B T}))  \\
&= k \exp(- \frac{E}{k_B T}) \qty(\exp(\frac{h \nu }{k_B T}) - 1)
= f(p_2) \qty(\exp(\frac{h \nu }{k_B T}) - 1)
\,.
\end{align}

We can plug this into the expression we have for the absorption  coefficient  to find
%
\begin{align}
\alpha _\nu &=
\frac{c^2}{8 \pi h \nu^3} \int \dd[3]{p} f(p) \qty(\exp(\frac{h \nu }{k_B T}) - 1) P(E, \nu )  \\
&= \frac{c^2}{8 \pi h \nu^3} \qty(\exp(\frac{h \nu }{k_B T}) - 1) \underbrace{\int \dd[3]{p} f(p) P(E, \nu ) }_{4 \pi j_\nu }  \\
&= \frac{j_\nu}{B_\nu }
\,,
\end{align}
%
since the rest of the expression we have is precisely the inverse of the Planck function.
We have \textbf{recovered Kirkhoff's law}, which is expected since the electrons are assumed to be in thermal equilibrium.

Now we want to proceed in the general case in which the distribution of the electron energies is not Maxwellian; a specific case of interest is a power-law spectrum, which as we have seen can be the effect of synchrotron emission and absorption.

% \todo[inline]{Are we sure about this? was the powerlaw tail not the effect of Comptonization?}

We wish to switch our integral from one over the particle momentum to one over the particle energy; in order to simplify the change of variable we will make the assumption that the particles are ultrarelativistic, therefore \(E \approx p c\). The change of variable is then
%
\begin{align}
4 \pi p^2 \dd{p}=  4 \pi \frac{E^2}{c^2} \frac{\dd{E}}{c} = \frac{4 \pi E^2}{c^3} \dd{E}
\,.
\end{align}
%
while the number of particles in this differential element is given by
%
\begin{align}
\dd{N} = N(E) \dd{E} = 4 \pi p^2 \dd{p} f(p) = \frac{4 \pi E^2}{c^3} \dd{E} f(p)
\,,
\end{align}
%
so we can identify
%
\begin{align}
N(E) = \frac{4 \pi E^2}{c^3} f(p)
\qquad \text{or} \qquad
f(p) = \frac{N(E)}{E^2} \frac{c^3}{4 \pi }
\,,
\end{align}
%
where \(f(p)\) is calculated at the momentum \(p = E /c\).
Plugging this into the expression for the absorption coefficient we find
%
\begin{align}
\alpha _\nu = \frac{c^2}{8 \pi h \nu^3} \int \frac{4 \pi }{c^3} E^2 \dd{E} \frac{c^3}{4 \pi } \qty[\frac{N(E - h \nu )}{(E - h \nu )^2} - \frac{N(E)}{E^2}] P (E, \nu )
\,.
\end{align}

We will further assume that the particle energy \(E\) is much larger than the photon energy \(h \nu \); this is justified by the fact that we are not using the full machinery of QED which would be required if the calculation was nonclassical.

If this is the case, then we can expand the expression in a power series around \(E\), so that it reads
%
\begin{align}
\alpha _\nu  = - \frac{c^2}{8 \pi h \nu^3} \int E^2 P(E, \nu ) \pdv{}{E} \qty[ \frac{N(E)}{E^2}] h \nu
\,.
\end{align}

Let us restrict this result to a powerlaw electron distribution, so that \(N(E) = C E^{-P}\) for some \(P \in \mathbb{R}^{+}\).
If this is the case, then the expression inside the integral reads
%
\begin{align}
- E^2 \dv{}{E} \qty(\frac{N(E)}{E^2})
= - E^2 \dv{}{E} \qty[C E^{-P-2}]
= C E^2 (P+2) E^{-P-3} = (P+2) \frac{N(E)}{E}
\,,
\end{align}
%
so the absorption coefficient will be approximately
%
\begin{align}
\alpha _\nu = (P+2) \frac{c^2}{8 \pi \nu^2} \int \frac{N(E)}{E} P(E, \nu ) \dd{E}
\,.
\end{align}

We must insert the known expression for the power radiated by a single charge as \(P(E, \nu )\); this can be evaluated to finally find
%
\begin{align}
\alpha _\nu \propto \nu^{- (P+4) / 2}
\,.
\end{align}

Now, recall that in general the definition of the source function \(S_\nu \) is
%
\begin{align}
S_\nu = \frac{j_\nu }{\alpha _\nu } = \frac{P(\nu )}{4 \pi \alpha _\nu }
\,,
\end{align}
%
where \(P(\nu )\) is the total spectral power, and since we know that its dependence on \(\nu \) is as \(P(\nu ) \propto \nu^{- (P-1) / 2}\), we can calculate the general source function for a powerlaw distribution:
%
\begin{align}
S_\nu \propto \frac{\nu^{- (P-1 ) / 2}}{\nu^{- (P+4) / 2}} = \nu^{- \frac{P - 1 + P - 4}{2}} = \nu^{5/2}
\,,
\end{align}
%
which is \textbf{independent of} \(P\)!

Knowing the source function we can make certain predictions about the emission of a medium which is optically thick at least for the low-energy part of the spectrum.

We have shown at the beginning of the course that if the optical depth in a medium is large then the specific intensity \(I_\nu \) becomes close to the source function: \(I_\nu  \sim S_\nu \).
In the case of synchrotron emission we have seen that \(\alpha_\nu \sim \nu^{- (P+4) / 2}\), a \emph{decreasing} function of the frequency, so the absorption will be large at low energies and small at high energies.
The same will generally hold for the optical depth, since it is proportional to the absorption coefficient if the length (\(\sim\) size of the medium) is fixed.

On the other hand, if the medium is optically thin then \(I_\nu \sim j_\nu \sim P(\nu )\).

So, if we make a plot (let us make it log-log as is usually done) of the specific intensity \(I_\nu \) as a function of frequency \(\nu \) we will have a ``thick'' region at low \(\nu \) where the intensity increases as \(I_\nu \sim \nu^{5/2}\), and a ``thin'' region at high \(\nu \) where the intensity decreases as \(I_\nu \sim \nu^{- (P-1) / 2}\).

\end{document}


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Bremsstrahlung}

\marginpar{Saturday\\ 2020-8-22, \\ compiled \\ \today}

The German word comes from the words meaning ``braking'' and ``radiation''. It is also called ``free-free emission''.

The name is historical: this kind of radiation was first observed in the lab coming from the deceleration of charges which hit a target.
This is the radiation emitted by electrons when a force is exerted upon them, for instance the Coulomb force by an ion. So, this kind of radiation can occur in a plasma since there we have both free protons and electrons.

We could also have electron-electron electromagnetic interactions, but in real astrophysical plasmas, as we shall see, this is less relevant.

Suppose we have an ion with a positive charge \(Ze\), and an electron with a negative charge \(-e\). Between them we have the attractive electrostatic force, whose modulus is
%
\begin{align}
F = \frac{Z e^2}{r^2}
\,.
\end{align}

If the ion is a proton, then the acceleration on the electron and on the proton can be calculated by equating
%
\begin{align}
m_e a_e = m_p a_p = \frac{e^2}{r^2}
\,
\end{align}
%
in modulus.
So, the acceleration of the proton is about \(m_p / m_e \approx 1836\) times \emph{smaller} than that of the electron. This allows us to approximate the ion as a fixed source of force, neglecting its acceleration completely.

We know from the Larmor formula \eqref{eq:larmor} that the power emitted by an accelerating charge is proportional to its acceleration squared, so the power emitted by the ion is about \((m_p / m_e)^2 \approx \num{4e6}\) times smaller than that emitted by the electron.

Now, let us consider the case of the repulsion of two identical particles, such as two electrons, in the nonrelativistic dipole approximation (which is not completely justified, often electrons will be relativistic\dots however we will make it for simplicity).

If \(d\) is the dipole moment, then we know from the Larmor formula for the dipole \eqref{eq:larmor-dipole} that the power emitted is proportional to \(\ddot{d}^2\).
The dipole moment is
%
\begin{align}
\vec{d} = -e \sum _{i}  \vec{r}_i = - \frac{e}{m_e} \sum _{i} m_e \vec{r}_i
= - \frac{e}{m_e} \vec{r}_{CM}
\,.
\end{align}

We can then see that the dipole moment is proportional to the center of mass of the system.
Now, we are always implicitly assuming that the two charges we are treating are isolated, so they are not subject to any external forces: therefore, the acceleration of the center of mass is zero.

This means that \(\ddot{d} = 0\), so the power emitted is zero in the dipole approximation.
This is why \emph{a system of nonrelativistic identical charges does not radiate in the dipole approximation}.

Now, let us move to an actual description of bremsstrahlung radiation.
We will treat it approximately; a complete description, even accounting for quantum mechanics, can be given, however it is beyond the scope of this course.

We start off by making the small angle approximation: we assume that the electron moves fast enough, so that its trajectory looks like a straight line and the deviation due to the proton is only a small perturbation.

We define the position vector \(\vec{r}\) as the one connecting the ion to the electron, while \(\vec{v} = \vec{\dot{r}}\) is the velocity of the electron, and the impact parameter \(b = \min \abs{\vec{r}}\) measures the distance of closest approach of the particles.

The dipole moment is given by \(\vec{d} = -e \vec{r}\), whose second derivative will be \(\ddot{\vec{d}} = -e \vec{\dot{v}}\).
Let us move to frequency space:
%
\begin{align}
\frac{1}{2 \pi } \int_{-\infty}^{\infty } \ddot{\vec{d}} e^{i \omega t} \dd{t} &= \frac{1}{2 \pi } \int_{-\infty }^{\infty } (- e \dot{\vec{v}})  e^{i \omega t} \dd{t} \\
- \omega^2 \hat{d} (\omega ) &= - \frac{e}{2 \pi } \int_{-\infty }^{\infty} \dot{\vec{v}} e^{i \omega t} \dd{t}
\,.
\end{align}

Then, we can see that if we are able to determine the value of the integral on the right-hand side we can directly calculate the dipole moment, and with it the emitted power.
Evaluating it exactly is hard in general, however we can do so in two limiting cases, depending on the ``duration of the interaction''.
The timescale of the process, that is, the rough amount of time over which the electrostatic interaction between the two particles is significant, is of the order \(\tau \approx b / v\).
Then, we can try to simplify the problem by integrating only over a range of size \(\sim \tau \) around zero in the time domain, instead of going from \(- \infty \) to \(+ \infty \), since there we will find the largest contribution.

Now, our limiting cases refer to the frequency \(\omega \) of the emitted radiation.
If this frequency is very high, such that \(\omega \tau \gg 1\), then in the integral we will have a slowly-varying term \(\dot{v}\) times a quickly oscillating term \(\exp(i \omega \tau )\): thus, the value of the integral will be close to zero.

On the other hand, if \(\omega \tau \ll 1\), then the exponential will be \(\exp(i \omega \tau ) \approx 1\): therefore the integral will be
%
\begin{align}
\int_{-\infty}^{\infty } \dot{\vec{v}} e^{i \omega t}\dd{t} \approx
\int_{-\infty}^{\infty } \dot{\vec{v}} \dd{t} = \Delta \vec{v}
\,.
\end{align}

Therefore, the Fourier transform of the dipole moment will be given by
%
\begin{align}
\hat{d}(\omega ) =
\begin{cases}
    - \frac{e}{2 \pi } \frac{\Delta \vec{v}}{\omega^2} &\qquad \omega \tau \ll 1  \\
    0 &\qquad \omega \tau \gg 1\,.
\end{cases}
\end{align}

With this, we can compute the spectral distribution of the emitted radiation \eqref{eq:spectral-distribution-dipole}:
%
\begin{align}
\dv{w}{\omega }
&= \frac{8 \pi }{3 c^3 \omega^{4}}
\begin{cases}
    \frac{e^2}{4 \pi^2 } \frac{\abs{\Delta \vec{v}}^2}{\omega^4} &\qquad \omega \tau \ll 1  \\
    0 &\qquad \omega \tau \gg 1
\end{cases}
\\
&=
\begin{cases}
    \frac{2e^2}{3 \pi c^3 } \abs{\Delta \vec{v}}^2 &\qquad \omega \tau \ll 1  \\
    0 &\qquad \omega \tau \gg 1\,.
\end{cases}
\end{align}

Now, then, we need to calculate \(\Delta \vec{v}\) in order to find the spectral distribution of the power. However, we already have an interesting result: a \textbf{flat power spectrum} in both the high- and low-frequency regimes, at a certain value to be calculated for high frequencies, and at zero for low frequencies.
There will need to be some smooth connection between the two regions for \(\omega \tau \sim 1 \).

Now, as long as the interaction time is short, the electron ``flies by'' the ion, which then has little time to exert a force on it, and it will do so only in the region in which the electron is close, and in which \(\vec{F}\) is approximately perpendicular to \(\vec{v}\).

In general, we can compute the variation in velocity as
%
\begin{align}
\Delta \vec{v} = \int_{- \infty }^{\infty} \vec{a} \dd{t}
\,,
\end{align}
%
and since the normal (dominant) component of the acceleration is given by \(a_N = F_N / m_e \), we can approximate it as
%
\begin{align}
\abs{\Delta \vec{v}} \approx \int_{- \infty }^{\infty} \frac{F_N}{m_e} \dd{t}
= \int_{- \infty }^{\infty}
\frac{Ze^2}{m_e r^2} \frac{b}{r} \dd{t}
\,,
\end{align}
%
where the factor \(b/r = \cos \theta \) accounts for the fraction of the force which is indeed normal: \(\theta \) is the angle between the radial separation \(\vec{r}\) and the velocity \(\vec{v}\) (or \(\pi - \) this angle), so that \(F \cos \theta = F_N\).
We can then solve this using the fact that the particle moves linearly, so that \(r^2= b^2 + v^2 t^2\): substituting this in we find
%
\begin{align}
\abs{\Delta v} &= \int _{- \infty }^{\infty} \frac{Ze^2b}{m_e r^3} \dd{t}
= \frac{Z e^2 b}{m_e} \int  _{- \infty }^{\infty} \frac{ \dd{t}}{(b^2 + v^2 t^2)^{3/2}}  \\
&= \frac{Ze^2}{m_e bv } \underbrace{\int _{- \infty }^{\infty} \frac{ \dd{x}}{(1 + x^2)^{3/2}}}_{= 2} = \frac{2 Z e^2}{m_e bv}
\,.
\end{align}

We can then insert this into the expression for the spectral distribution of the signal, also using \(\tau \sim b/v\):
%
\begin{align}
\dv{w}{\omega }
&=
\begin{cases}
    \frac{8 Z^2e^6}{3 \pi c^3 b^2 v^2 m_e^2 }  &\qquad b \ll v/\omega   \\
    0 &\qquad b \gg v/\omega \,.
\end{cases}
\end{align}

\subsubsection{Bremsstrahlung in a plasma}

We found the spectral density for a single electron: now, we wish to compute it for the whole plasma, whose ion density is \(n_i\), and whose electron density is \(n_e\).
To simplify, we will assume that all the electrons have the same speed \(v\), but we will let their impact parameters \(b\) vary.

Let us consider this for a single ion, onto which many electrons will impact.
The flux of electrons will be given by \(n_e v\); so the number of particles crossing an annulus of radii \(b\), \(b + \dd{b}\) will be given by \(n_e v \dd{A}  =2 \pi b \dd{b} n_e v\).
Let us then integrate in \(\dd{b}\) to find the total emitted power. We will integrate from some minimum impact parameter \(b _{\text{min}}\) instead of from zero: this is needed to find a physical result, and it will be explained in more detail later. We find:
%
\begin{align}
\frac{ \dd{w}}{ \dd{t} \dd{\omega }} = \int_{b _{\text{min}}}^{\infty }
n_e v \dv{w}{\omega } 2 \pi b \dd{b}
= 2 \pi n_e v \int_{b _{\text{min}}}^{\infty} \dv{w}{\omega } b \dd{b}
\,,
\end{align}
%
where now we need to substitute our expression; however we only have a nonzero contribution in the low-frequency limit, or equivalently the small-\(b\) limit. So, our integrand will be zero asymptotically; before that it will go as \(1/b\). We account for this by only integrating up to some cutoff \(b _{\text{max}} \sim v/ \omega \), whose exact value must be determined by a more detailed analysis. We will use the rough estimate \(b _{\text{max}} = v / \omega \).
If we also account for the ion density \(n_i\) to find the power per unit frequency and volume, our integral can be expressed as:
%
\begin{align}
\frac{ \dd{w}}{ \dd{t} \dd{\omega } \dd{V}}
&= \frac{16 Z^2 e^{6}}{3 c^3 m_e^2 v } n_e n_i \int_{b _{\text{min}}}^{b _{\text{max}}} \frac{b}{b^2} \dd{b }  \\
&= \frac{16 Z^2 e^{6}}{3 c^3 m_e^2 v } n_e n_i
\log \qty( \frac{b _{\text{max}}}{b _{\text{min}}})
\,.
\end{align}

Now, what should the value of \(b _{\text{min}}\) be?
A first approximation we made is the small-angle one, which holds as long as \(\abs{\Delta \vec{v}} / \abs{\vec{v}}\) is small, less than unity. Suppose we are at the upper limit of this condition, when \(\abs{\Delta \vec{v}} \sim \abs{\vec{v}}\).

Inserting this into our expression for \(\abs{\Delta \vec{v}}\) we find the limit where \(b\) is so small --- the electron comes so close to the ion --- that the interaction is too large to be treated perturbatively:
%
\begin{align}
\abs{\Delta \vec{v}} \sim v \sim \frac{2 Z e^2}{m_e b v} \implies b = b _{\text{min}} = \frac{2 Z e^2}{m_e v^2}
\,.
\end{align}

A second line of reasoning comes from the quantum-mechanical uncertainty principle: \(\Delta x \Delta p \gtrsim \hbar\). If our \(\Delta x\) is \(b _{\text{min}}\), this means that we will have
%
\begin{align}
b \gtrsim b _{\text{min}} = \frac{\hbar}{m_e v}
\,.
\end{align}

Which of these is greater? To make it clearer, in natural units we are comparing (\(1/m_e v\) times) the quantities: \(1\) for the quantum mechanical threshold, and
%
\begin{align}
\underbrace{8 \pi \alpha}_{\approx \num{.18}} \frac{Z}{v}
\,
\end{align}
%
for the small-angle-approximation threshold. Since we are dealing with  nonrelativistic particles \(v\) will be small, so this can easily become the larger bound of the two, even for a hydrogen ion.

\todo[inline]{These seem to both be arguments as to why we cannot deal with small \(b\) with our approximations, however I'd expect small-\(b\) processes to occur\dots The quantum mechanical bound is more convincing than the other.}

Anyhow, these considerations are all heuristic, and what is typically done is to parametrize the uncertainty in this aspect with a so-called \textbf{Gaunt factor}
%
\begin{align}
g_{ff} (v, \omega ) = \frac{\sqrt{3}}{\pi } \log \qty(\frac{b _{\text{max}}}{ b _{\text{min}}})
\,,
\end{align}
%
where the prefactor is, I think, there for historic reasons. This will in general be a function of both \(v\) (inside \(b _{\text{min}}\)) and of \(\omega \) (inside of \(b _{\text{max}}\)). With it, we can write
%
\begin{align} \label{eq:single-velocity-bremsstrahlung-distribution}
\frac{ \dd{w}}{ \dd{t} \dd{V} \dd{\omega }}
&= \frac{16 \pi Z^2 e^6 \pi }{3 \sqrt{3} c^3 m_e^2 v} n_e n_i g_{ff} (v, \omega )
\,.
\end{align}

With a proper quantum-mechanical treatment one can find an exact expression for this factor, and in the literature there are several good approximations which are good in different regimes; also, the values are tabulated.
One can then assume that this is a known function.

\end{document}