% JuliaCon proceedings template
\documentclass{juliacon}
\setcounter{page}{1}

\usepackage{hyperref}
%\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\usepackage{graphicx}
%\usepackage[font=small,labelfont=bf]{caption}
\usepackage{subcaption}

\usepackage{tikz}
\usetikzlibrary{shapes.misc}
\tikzset{cross/.style={cross out, draw=black, thick, minimum size=2*(#1-\pgflinewidth), inner sep=0pt, outer sep=0pt},
%default radius will be 1pt.
cross/.default={3.5pt}}
\usepackage{pgfplots}
\pgfplotsset{samples=600}
\pgfplotsset{compat=newest}
\usepackage{amsmath}
%\usetikzlibrary{external}
%\tikzexternalize[prefix=tikz/]

\def\sinc{\mathrm{sinc}}
\def\lc{\lstinline}
\newcommand\todo[1]{\textcolor{red}{\textbf{TODO: #1}}}

\usepackage{siunitx}

\begin{document}

\input{header}

\maketitle

\begin{abstract}
\verb|FourierTools.jl| aims at simplifying work in Fourier
space without loosing efficiency.
First, we provide several convenient wrappers to ease the usage of the common Fourier processing operations.
Second, we implement array transformations like $\sinc$ interpolation, sub pixel shifting, convolution, rotation and shearing based on Fourier transforms.
Notably, those algorithms handle the Nyquist term in case of an even sized array correctly
under the condition of minimum mean-square slope.

\end{abstract}

\section{Introduction}
The fast Fourier transform (FFT) is a popular algorithm because it can evaluate the Discrete Fourier Transform (DFT)
\begin{equation}
    X_k = \sum_{n=0}^{N-1} x_n \exp\left(-i2 \pi \frac{kn}{N} \right)
    \label{eq:dft}
\end{equation}
and the Inverse Discrete Fourier Transform (IDFT)
\begin{equation}
    x_n = \frac1{N}\sum_{k=0}^{N-1} X_k \exp\left(i2 \pi \frac{kn}{N} \right)
    \label{eq:idft}
\end{equation}
in $\mathcal{O}(N \log N)$ operations, whereas the straightforward
evaluation takes $\mathcal{O}(N^2)$ operations.
The DFT transforms the signal to a different basis, the Fourier space which
has wide applications in signal processing.
In Julia \cite{bezanson2017julia} \verb|FFTW.jl| offers a convenient interface to
the \verb|FFTW| library \cite{FFTW05} which is among the fastest FFT libraries.
The interpretation of a signal in Fourier space is versatile and many operations can
be done using FFT operations.
In this work, we present several operations which can be calculated quickly
due to the good performance of FFTs.
Additionally, we also stick to a certain boundary handling in all algorithms.
All algorithms and examples are implemented using the Julia package \verb|FourierTools.jl|\footnote{The full source is available on GitHub: \url{https://github.com/bionanoimaging/FourierTools.jl/}}.

\section{Mathematical Background}
    Essential for the understanding of the algorithms presented here, is the Discrete Fourier series (DFS).
    Qualitatively, the DFS is a weighted sum of complex $\cos$ and $\sin$ terms.
    In our notation, the DFS of a signal defined in $[0, L)$ is defined as
    \begin{align}
        x_n &= \frac1{N}\sum_{k=0}^{N-1} X_k \cdot \exp\left(i2\pi \frac{k}{N} n \right) \\
        &= \frac1{N}\sum_{k=0}^{N-1} X_k \cdot \left[ \cos\left(i2\pi \frac{k}{N} n \right) + i \cdot  \sin\left(i2\pi \frac{k}{N} n \right) \right]\label{eq:DFS2}\\
        &=\frac1{N}\sum_{k=-\lfloor N/2\rfloor}^{\lceil N/2 -1 \rceil} X_k \cdot \left[ \cos\left(i2\pi \frac{k}{N} n \right) + i \cdot  \sin\left(i2\pi \frac{k}{N} n \right) \right]\label{eq:DFS}
    \end{align}
    which is the same as the IDFT definition in \autoref{eq:idft}.
    The brackets $\lfloor \cdot \rfloor$, $\lceil \cdot \rceil$ denote the floor, next lower integer number and the ceil, next upper inter number respectively. In fact, the coefficients $X_k$ are obtained with the DFT.
    \autoref{eq:DFS} can be derived from the $2\pi$ periodicity of $\sin$ and $\cos$.
    An inherent assumption of the DFS is that the signal $x_n$ is circular repeating.
    So whenever a DFT/IDFT is calculated, this property is included.
    The coefficients $X_K$ describe the strength of an individual complex
    exponential basis function.
    In general, coefficients $X_k$ are complex valued but for real input data $x_n$ it follows that
    \begin{equation}
        X_{N-k} = X_k^*.
        \label{eq:symmetry}
    \end{equation}
    This property is usually called \textit{Hermitian}.
%    \begin{figure}
%        \centering
%        \begin{tikzpicture}
%            \def\myradius{1.6}
 %           \def\mydiff{0.3}
 %           \draw[->] (-\myradius - \mydiff, 0) -- (+\myradius + \mydiff,0) node[right] {$\mathrm{Re}$};
%            \draw[->] (0, -\myradius - \mydiff) -- (0, +\myradius + \mydiff) node[right] {$\mathrm{Im}$};
%            \draw (0,0) circle(\myradius);
 %           \foreach \i in {0, 1, ..., 6}{
%                \pgfmathsetmacro{\anglee}{360/7 * \i}
%            }
%        \end{tikzpicture}
%        \caption{Caption}
%        \label{fig:dft}
%    \end{figure}

    As seen from \autoref{eq:DFS}, the maximum frequency of a signal described with the DFS is $-\lfloor N/2 \rfloor / L$ (the border of the summation interval of $k = -\lfloor N/2 \rfloor$)\footnote{You might wonder why the maximum frequency in \autoref{eq:DFS2} would be at $k=N-1$. However, this frequency is not Nyquist sampled\footnote{According to the Whittaker–Nyquist–Shannon
        theorem \cite{whittaker_e_t_1915_1428702}, \cite{Shannon1949}.} and leads to aliasing which results that those frequencies appear at lower frequencies.}.
    For a signal (over the range $[0, L)$) with 7 elements, the frequencies in the Fourier series would be
    \begin{equation}
        f_{\{k\}} = \left[-\frac{3}{7 L} , -\frac{2}{7 L},  -\frac{1}{7L}, \frac{0}{7L}, \frac{1}{7L}, \frac{2}{7L}, \frac{3}{7L} \right]
        \label{eq:7elems}
    \end{equation}
    and for one with 6 elements
    \begin{equation}
       f_{\{k\}} = \left[-\frac{4}{8 L} ,-\frac{3}{8 L} ,-\frac{2}{8 L} ,-\frac{1}{8 L} ,\frac{0}{8 L} ,\frac{1}{8 L} ,\frac{2}{8 L} ,\frac{3}{8 L} \right].
       \label{eq:6elems}
    \end{equation}
    The highest frequency $-\frac{4}{8 L}$ in \autoref{eq:6elems} does not fulfill Nyquist sampling and thus
    cannot be interpreted uniquely. As drawn in \autoref{fig:nyquist_sample}, comparing the solid green, red and blue curves, there are infinitely many periodic functions of one and the same coarsest frequency
     (varying in phase and amplitude) exhibiting same sampled values.
    \begin{figure}
        \centering
        \begin{tikzpicture}
            \begin{axis}[xmin=-0.5, xmax=7, ymax=1.2, ymin=-1.2, width=8cm, height=5cm,
                xtick = {0, 3.1415, 6.283}, xticklabels={0, $L/2$, $L$}]
                    \addplot[sharp plot, domain=-1:9, color=green, samples=400] {0.2 * cos(deg(x)};
                    \addplot[sharp plot, domain=-1:9, color=blue, samples=100] {0.5 * cos(deg(x) - 66.42};
                    \addplot[sharp plot, domain=-1:9, color=red, samples=100] {1 * cos(deg(x) - 78.46};
                    \addplot[sharp plot, domain=-1:9, color=orange, samples=100] {0.2*cos(deg( 3 *x))};
                    \draw[] (axis cs:-1, 0) -- (axis cs:8, 0);
                    \draw[dashed] (axis cs:0, -2) -- (axis cs:0, 2);
                    \draw[dashed] (axis cs:3.1415, -2) -- (axis cs:3.1415, 2);
                    \draw[dashed] (axis cs:6.283, -2) -- (axis cs:6.283, 2);

                   \draw (0,0.2) node[cross, thick] { };
                    \draw (3.1415,-0.2) node[cross, thick] { };
                     \draw (6.283,0.2) node[cross, thick] { };

            \end{axis}
        \end{tikzpicture}
        \caption{Periodic signal of period length $2\pi$ which has been sampled with $\pi$ spacing.}
        \label{fig:nyquist_sample}
    \end{figure}

    Coming back to \autoref{eq:7elems} we can think about what happens if we extend the Fourier spectrum by introducing additional frequencies.
    For example, we can symmetrically introduce higher frequencies, which we pretend to have measured but which yielded zero amplitude. That means, that we do not add new information to our measured information.
    However, calculating the IDFT we now evaluate the Fourier series on a different grid.
    By our introduction of higher frequencies with zero strength, we did not change the domain of the function but we simply evaluate the Fourier series at a denser spaced grid in real space. This evaluation changes the signal to a smother, \textit{interpolated} appearance.
    The whole procedure is called Sinc \textit{interpolation}.\\

    \subsection{Sinc Upsampling of Real--Valued Signals}
        Now we discuss how we apply $\sinc$ interpolation to of real--valued signals.
        Starting with an odd-sized signal is simple (see above) as we only have to pad with zeros.
        Starting with an even-sized signal is more difficult, which is demonstrated in the next parts.

        \subsubsection{Even $\rightarrow$ Odd}
            If we consider a real signal (with length $N=4$) $[x_0, x_1, x_2, x_3]$ we discuss how to interpolate
            this signal to a length of $N=5$ and $N=6$.
            The frequencies of the signal shall be called $[X_{-2}, X_{-1}, X_0, X_1]$.
            Interpolation to a length $N=5$, naively we would think that padding the frequencies to
            \begin{equation}
                Y_k = [X_{-2}, X_{-1}, X_0, X_1, 0]
            \end{equation}
            would be sufficient. However, because of \autoref{eq:symmetry} the resulting signal
            \begin{equation}
                y_n = \mathcal{F}^{-1}\left[X_{-2}, X_{-1}, X_0, X_1, 0\right]
            \end{equation}
            is not real anymore. In our opinion, a real interpolated signal
            should result in a real signal after interpolation.
            Therefore, we have to restore the symmetry property, resulting in a real signal.
            To achieve that, one way would be to equally split the real--valued Nyquist amplitude (the highest frequency) equally between two highest frequencies like
            \begin{equation}
                y_n = \mathcal{F}^{-1}\left[X_{-2} / 2, X_{-1}, X_0, X_1, X_{-2}/2 \right].
            \end{equation}
            This choice of interpolation warrants real values of $y_n$ and additionally, minimizes the mean-square slope\footnote{This has been derived by Steven G. Johnson: \url{https://discourse.julialang.org/t/sinc-interpolation-based-on-fft/52512/82}.
            There are also additional notes here: \url{https://math.mit.edu/~stevenj/fft-deriv.pdf}.}. Note, that since $X_{-2}$ is guaranteed to be real for real--valued signals, no complex conjugation is required in the split for one-dimensional Fourier-transformations.

        \subsubsection{Even $\rightarrow$ Even}
            To interpolate to $N=8$ we can then do
            \begin{equation}
                y_n = \mathcal{F}^{-1}\left[0, 0, X_{-2} / 2, X_{-1}, X_0, X_1, X_{-2}/2, 0 \right].
            \end{equation}
            The limit--frequency is split as before and all frequencies above the initial Nyquist amplitude are set to zero.

        \subsubsection{Odd $\rightarrow$ Odd}
            The odd case is straightforward, starting with the frequencies
            \begin{equation}
                X_k = [X_{-1}, X_0, X_1]
            \end{equation}
            going to a larger odd or even array by zero--padding, always fulfills the real property.
            \begin{equation}
                y_n = \mathcal{F}^{-1}[0, X_{-1}, X_0, X_1, 0]
            \end{equation}

        \subsubsection{Odd $\rightarrow$ Even}
            Also for the even case, the all new frequencies are set to zero including the new Nyquist frequency.
            \begin{equation}
                y_n = \mathcal{F}^{-1}\left[0, X_{-1}, X_0, X_1\right].
            \end{equation}

    \subsection{Sinc Downsampling}
        Instead of upsampling the signal, we can also remove higher frequency content.
        The result is a signal which in principle is free of aliasing, as opposed to for example binning of adjacent pixels, which would result in aliasing.

        \subsubsection{Even $\rightarrow$ Odd}
            We can simply remove the Nyquist amplitude.
            Starting from $[X_{-2}, X_{-1}, X_0, X_1]$ downsampling to $N=3$ is
            \begin{equation}
                Y_k = [X_{-1}, X_0, X_1].
            \end{equation}
        \subsubsection{Even $\rightarrow$ Even}
            However, going to $N=2$ we have to accumulate the two highest frequencies into the one remaining Nyquist amplitude:
            \begin{equation}
                Y_k = [X_{-1} + X_{1}, X_0].
            \end{equation}
            The qualitative meaning is, that since we only have a single negative Nyquist amplitude, this frequency is aliased (positive counter--part is missing) and this negative Nyquist amplitude stores the sum of both entries. Note that also here, no complex conjugation is applied to yield a real-valued Nyquist frequency, forcing the resulting signal $y_n$ to be real (see symmetry property \autoref{eq:symmetry}), .

        \subsubsection{Odd $\rightarrow$ Odd}
            We can downsample
            \begin{equation}
                Y_k = [X_{-2}, X_{-1}, X_0, X_1, X_2]
            \end{equation}
            to an odd--sized array in a straight--forward way by dropping the surplus frequencies:
            \begin{equation}
                Y_k = [X_{-1}, X_0, X_1].
            \end{equation}

       \subsubsection{Odd $\rightarrow$ Even}
           Going to an even sized signal is again achieved by accumulating the Nyquist amplitude entry:
            \begin{equation}
                Y_k = [X_{-1} + X_1, X_0].
            \end{equation}
        In summary, for upsampling only the source even--sized arrays and for downsampling only the destination even-sized arrays require a special treatment.
        It is worth stressing that the Nyquist-frequency data in even-sized signals has no unique interpretation neither for real nor for complex valued data many combinations of underlying amplitude and phase yield the same sampled signal. Therefore, first downsampling to an even-sized array followed by upsampling will in general be modifying this downsampled Nyquist--frequency, which can only be avoided if additional assumptions can be made. Ideally, the user should avoid data being present at this even-sized Nyquist frequency, to avoid ambiguities.
        Finally, real--valued signals should be interpolated by exploiting real--to--complex Fourier transform algorithms, as this comes with increased speed and reduced memory consumption. Yet the considerations presented above apply in an analogous way.
    \subsection{Sinc Interpolation in $N$ Dimensions}
        The multidimensional signal handling is in principle similar as for
        a 1D dataset. However, the Nyquist amplitude is not a single entry anymore but rather a $(N-1)$--dimensional hyperplane.
        \subsubsection{Odd Case}
            Assuming the real dataset $x_{i,j}$, we obtain its DFT as
            \begin{equation}
                X_{k, m} = \begin{bmatrix}
                   X_{-1, -1} &  X_{-1, 0} \\
                   X_{0,-1} & X_  {0, 0} \\
                \end{bmatrix}
            \end{equation}
            The interpolation to a $3\times 3$ array  $y_{\{i, j\}}$, needs the following form
            \begin{equation}
                y_{i, j} = \mathcal{F}^{-1}\begin{bmatrix}
                   X_{-1, -1}/2 &  X_{-1, 0}/2  & 0\\
                   X_{0,-1}/2   & X_{0, 0}      &  X_{0,-1}^*/2\\
                    0           &  X_{-1, 0}^*/2 &  X_{-1, -1}^*/2\\
                \end{bmatrix}
            \end{equation}
            Note, in $N>1$ dimensions we need to adhere to the \textit{Hermitian} property for the
            resulting signal to be real.
            For a 1D signal the Nyquist term is real and hence complex conjugation was not necessary but in $N>1$ complex conjugation seems to be required.\\
            However, due to the separability of the DFT, we can  interpolate each dimension separately.
            This allows us to treat any N dimensional signal as a collection of 1D signals, treating each dimension individually but simultaneously for all such 1D signals.
            The first interpolation along the first dimension would be
            \begin{equation}
                y^{1}_{i, j} = \mathcal{F}_{m}^{-1}\begin{bmatrix}
                   X^1_{-1, -1}/2 &  X^1_{-1, 0}/2 \\
                   X^1_{0,-1} & X^1_{0, 0} \\
                   X^1_{-1,-1}/2 & X^1_{-1, 0}/2 \\
                \end{bmatrix}
            \end{equation}
            and with the new coefficients $\mathcal{F} \left[y^{1}_{\{i, j\}} \right] =Y^{1}_{k, m} $
            If we interpolate this intermediate result along the second dimension we obtain
            \begin{equation}
                y^{1,2}_{k, m} = \mathcal{F}_{k}^{-1}\begin{bmatrix}
                   Y^1_{-1, -1} /2 &  Y^1_{-1, 0} & Y^1_{-1, -1} / 2\\
                   Y^1_{0,-1} /2& Y^1_{0, 0} & Y^1_{0, -1} / 2\\
                   Y^1_{-1,-1}/2 & Y^1_{-1, 0}& Y^1_{-1, -1} / 2\\
                \end{bmatrix}.
            \end{equation}
            The advantage of this method is the easier boundary handling since we only
            need to divide the $N-1$ dimensional slice and do not have to care about maintaining the
            Hermitian property (which is more difficult to program).
            In the even case, the Nyquist amplitude gets aliased and hence we would have
            to add the $(N-1)$ dimensional slices together in a flipped way.
            But again, doing that sequentially is simpler because there is no need to flip the data.

        \subsubsection{Downsampling}
            As described in the 1D case, for downsampling to an even-sized array we have to
            add the two highest frequencies to yield one new Nyquist amplitude. For $N$ dimensions this
            procedure works sequentially over the dimensions as described in the upsampling routine.

    \subsection{Convolution Theorem}
        The convolution theorem states that a convolution can be expressed as
        a point wise multiplication and three Fourier transforms,
        \begin{equation}
            u * v = \mathcal{F}^{-1}\left[\mathcal F[u] \cdot \mathcal{F}[v]\right]
        \end{equation}
        where $*$ is the convolution and $\cdot$ the point wise multiplication.
        This property can be proved with the usage of the DFT definition (see \autoref{eq:dft}).
        A FFT based convolution usually offers an improvement if the kernel $u$ is large.

    \subsection{Shift Theorem}
        The Fourier Shift theorem tells us that a shift of a signal can be calculated
        in Fourier space the point wise multiplication of the Fourier--transformed signal by a complex-valued phase ramp.
        This is directly derived from the DFT definition.
        The full form of the Fourier shift theorem is:
        \begin{equation}
            u(\mathbf x+ \Delta \mathbf x) = \mathcal{F}^{-1}\big[\mathcal{F}[u(\mathbf x)](\mathbf k) \cdot \exp(i \mathbf k \Delta \mathbf x) \big]
            \label{eq:shifting}
        \end{equation}
        where $\Delta x$ can be an arbitrary shift not restricted to integer values.
        \subsubsection{Handling of the Nyquist Amplitude}
            As before, separate handling of the Nyquist amplitude is essential, otherwise
            the real symmetry property might be violated leading to a complex--valued output for a real input.
            \autoref{eq:shifting} can be also interpreted as a convolution of the signal $u$
            with the kernel $\mathcal{F}^{-1}\left[\exp(i \mathbf k \Delta \mathbf x)\right]$.
            Hence, this kernel should be purely real in real space, otherwise the
            shifting kernel does mix real and imaginary part of the signal.
            For a real signal not observing this kernel--property would introduce an imaginary part, for a purely imaginary signal it would introduce real values, and for a complex--valued signal it would mix both of them. We thus need to ensure a real--valued kernel. The complex exponential in \autoref{eq:shifting} has the required Hermitian property except for its value at the Nyquist--frequency in case of even--sized arrays. In this case we only multiply by the real part of the phase factor! Note that this yield to a positive and negative shift operation not perfectly compensating each other. Another consequence of the undefined underlying signal. This effect can be avoided and a reversible shift can be guaranteed by setting this frequency of the kernel to the real value of 1.0 but this might be less accurate on average for the shift. Note that for shifts not being exactly half-pixels, the inverse shift can be modified such that it perfectly recreates the original signal, even if the first shift used the real part of the kernel at Nyquist--frequency, if the user knows that the operation is an inverse shift: We simply need to replace the real part with one over the real part.

        \subsection{Shifting in $N$ Dimensions}
            As for multidimensional $\sinc$ interpolation, we similarly can process the shifting in multiple dimensions sequentially.

    \subsection{Array Shearing}
        Since we can shift data by an arbitrary amount via the shift theorem,
        we can apply it with varying amounts of shifts between the $(N-1)$ dimensional hyperplanes of an $N$ dimensional array. If we vary the shifts linearly with position we obtain a sheared version of the input signal.

    \subsection{Array Rotation}
        A very simple image rotation algorithm is the "three shear algorithm" \cite{Paeth1986:15}.
        Its essence is that a rotation in two dimensions can be implemented by shearing an image first horizontally, then vertically and finally horizontally again.
        Mathematically, the rotation matrix (in 2D) with rotation angle $\theta$ can be separated into three shear matrices:
        \begin{equation}
            \begin{pmatrix}
                \cos(\theta) & -\sin(\theta)\\
                \sin(\theta) & \phantom{-}\cos(\theta)
            \end{pmatrix}
            =  \begin{pmatrix}
                1 & \alpha\\
                0 & 1
            \end{pmatrix}
            \cdot
             \begin{pmatrix}
                1 & 0\\
                \beta & 1
            \end{pmatrix}
            \cdot
             \begin{pmatrix}
                1 & \alpha\\
                0 & 1
            \end{pmatrix}
        \end{equation}
        The new parameters are $\alpha=-\tan\left(\frac{\theta}{2} \right)$ and $\beta=\sin(\theta)$\footnote{See also the more detailed notes of Tobin Fricke: \url{https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html}.}.
        Since shearing is a shift operation with different amounts along each line, the complete rotation
        can be fully expressed with Fourier transforms and element--wise multiplications. Note that care has to be taken to provide additional padding prior to shifting, as the Fourier--based shifts (and thus also the shears) are cyclic in nature.

\section{Examples}
    In this part we present the main contributions of \verb|FourierTools.jl| which
    include \lstinline{fftshift} alternatives, wrappers for \lstinline{fft}, $\mathrm{sinc}$ interpolation, convolutions and shifting, shearing and rotation.
    Our experiments additionally made use of the packages ArrayInterface.jl\cite{DifferentialEquations.jl-2017}, BenchmarkTools.jl\cite{BenchmarkTools.jl-2016} .

    \subsection{FFT Helpers}
        A very common pattern in signal processing is to apply various frequency filters
        which are multiplied by the signal in Fourier space.
\begin{lstlisting}[language=Julia, caption={Typical workflow to manipulate the data in Fourier space. Note that it would be also possible to \lc{ifftshift(mask)} instead of rearranging the data twice. However, this breaks apart once one has a non-straightforward accessing scheme instead of a plain array which is multiplied.},
label = {lst:example_render},
captionpos = b,]
function fft_filter(arr, mask)
    arr_ft = fftshift(fft(arr))
    arr_ft .*= mask
    ifft(ifftshift(arr_ft))
end

function ffts_filter(arr, mask)
    arr_ft = ffts(arr)
    arr_ft .*= mask
    iffts(arr_ft)
end
\end{lstlisting}

Our package provides \lc{ffts} and \lc{iffts} which automatically applies \lc{fftshift} and \lc{ifftshift} after respectively before calling \lc{fft}.
\lc{fftshift} and \lc{ifftshift} also copy and reshuffle the data. We instead make use of the package
\verb|ShiftedArrays.jl|\footnote{See \url{https://github.com/bionanoimaging/ShiftedArrays.jl}. Thanks
to Pietro Vertechi who supported us in implementing \lc{ShiftedArrays.fftshift} and \lc{ShiftedArrays.ifftshift}}
By keeping track of the total shifts we can avoid to copy the data in between using the \lc{fftshift} and \lc{ifftshift} but instead return the original data.

In \autoref{code:ffts} we can see the simple benchmark showing the computing time and the total
allocations needed.
\begin{lstlisting}[language=Julia, label={code:ffts}, caption={By using \lc{ffts} we can save a factor of 2 in memory}, captionpos=b]
julia> using FourierTools, BenchmarkTools, FFTW, IndexFunArrays
julia> N = (4000, 3001);
julia> arr = randn(ComplexF32, N); # 91.583 MiB
julia> mask = IndexFunArrays.rr(Float32, N) .< 50; # low pass
julia> @btime fft_filter($arr, $mask); # using fftshift
  604.883 ms (78 allocations: 366.34 MiB)
julia> @btime ffts_filter($arr, $mask); # using a fftshift_view
  596.754 ms (54 allocations: 183.17 MiB)
\end{lstlisting}

Additionally, to \lc{ffts}, inspired by DipImage\footnote{\url{https://diplib.org/}}, we also provide \lc{ft} which does both a shifting before and after calling \lc{fft}.
The schematic behavior of all different methods is drawn in \autoref{fig:freq_pos}.
    \begin{figure}[h]
        \centering
        \begin{tikzpicture}
            \begin{scope}[shift={(0,00)}]
                \draw[->] (2.3, 0.25) -- (3.2, 0.25) node[midway, above] {\lc{fft}};
                \draw[fill=red!50!white] (0, 0) rectangle (0.5, 0.5);
                \draw[fill=green!50!white] (3.5, 0) rectangle (4.0, 0.5);
                \foreach \i/\j/\k in {0/0/1,1/1/2,2/-2/3,3/-1/4}{
                    \draw (\i/2, 0) rectangle (\i/2+1/2, 1/2) node[midway] {\k};
                    \draw (3.5+\i/2, 0) rectangle (3.5+\i/2+1/2, 1/2)node[midway] {\j};
                }

            \end{scope}
            \begin{scope}[shift={(0,-1)}]
                \draw[->] (2.3, 0.25) -- (3.2, 0.25) node[midway, above] {\lc{ffts}};
                \draw[fill=red!50!white] (0, 0) rectangle (0.5, 0.5);
                \draw[fill=green!50!white] (4.5, 0) rectangle (5.0, 0.5);
                \foreach \i/\j/\k in {0/-2/1,1/-1/2,2/0/3,3/1/4}{
                    \draw (\i/2, 0) rectangle (\i/2+1/2, 1/2)node[midway] {\k};
                    \draw (3.5+\i/2, 0) rectangle (3.5+\i/2+1/2, 1/2)node[midway] {\j};
                }

            \end{scope}
            \begin{scope}[shift={(0,-2)}]
                            \draw[->] (2.3, 0.25) -- (3.2, 0.25) node[midway, above] {\lc{ft}};
                \draw[fill=red!50!white] (1, 0) rectangle (1.5, 0.5);
                \draw[fill=green!50!white] (4.5, 0) rectangle (5.0, 0.5);
                \foreach \i/\j/\k in {0/-2/1,1/-1/2,2/0/3,3/1/4}{
                    \draw (\i/2, 0) rectangle (\i/2+1/2, 1/2)node[midway] {\k};
                    \draw (3.5+\i/2, 0) rectangle (3.5+\i/2+1/2, 1/2)node[midway] {\j};
                }
            \end{scope}
            \draw[color=white] (1,-2.5) --(2,-2.5);
        \end{tikzpicture}
        \caption{Schematic description how the different methods interpret the center  position. The red box
            indicates the center index position in real space and the green box the center position of the 0 frequency. The corresponding inverse routines have the letter i prepended and perform the corresponding transformation from right to left. The intensity scaling is unmodified compared to the underlying fft routine.}
        \label{fig:freq_pos}
    \end{figure}

In \autoref{code:fft_ft_difference} the difference between those methods becomes clear,
if we transform a delta peak in real space.
\begin{lstlisting}[language=Julia, label={code:fft_ft_difference}, caption={Difference between \lc{fft} and \lc{ft}.}, captionpos=b]
julia> fft([1.0, 0.0, 0.0])
3-element Vector{ComplexF64}:
 1.0 + 0.0im
 1.0 + 0.0im
 1.0 + 0.0im

julia> fft([0.0, 1.0, 0.0])
3-element Vector{ComplexF64}:
  1.0 + 0.0im
 -0.5 - 0.8660254037844386im
 -0.5 + 0.8660254037844386im

julia> ft([0.0, 1.0, 0.0])
3-element ShiftedArrays.CircShiftedVector{ComplexF64,
                                          Vector{ComplexF64}}:
 1.0 + 0.0im
 1.0 + 0.0im
 1.0 + 0.0im
\end{lstlisting}

    \subsection{Convolution}
        The filter operation from above, is the same as a convolution of the \lc{arr} with \lc{ifft(mask)}.

        Similar to \lc{plan_fft} we offer a method called \lc{plan_conv}.
        The speedup in \autoref{code:conv} is mainly due to the FFTW planning flag \lc{FFTW.PATIENT}.
        For repeated use of \lc{conv} (e.g. an iterative algorithm) another option is
        \lc{plan_conv_buffer} which allocates buffers beforehand and has zero
        memory consumption during \lc{conv} calls.
\begin{lstlisting}[language=Julia, captionpos=b, caption={Convolution planning. One with memory allocations, the other without.}, label=code:conv]
julia> mask, pconv = plan_conv(arr, ComplexF32.(iffts(mask)),
                               flags=FFTW.PATIENT);

julia> @btime pconv(arr);
  394.561 ms (8 allocations: 366.33 MiB)

julia> @time mask, pconv_buffer = FourierTools.plan_conv_buffer(
        arr, ComplexF32.(iffts(mask)), flags=FFTW.PATIENT);
  0.862628 seconds (156 allocations: 1.073 GiB, 2.29% gc time)

julia> @btime pconv_buffer(arr);
  348.121 ms (0 allocations: 0 bytes)
\end{lstlisting}

    \subsection{Sinc Interpolation}

        The upsampled (since interpolated) data (dashed blue curve) in \autoref{fig:resampling} was generated with the source code in \autoref{code:resampling}. The input \lc{arr_low} (green crosses) interpreted as sampled according to Nyquist sampling and hence all information is already included in the low sampling.
        Therefore, a re-evaluation of the Fourier series does not generate new information but
        just evaluates the sampled data on a new grid with smaller spacing leading
        to a smoother (interpolated) appearance.

\begin{lstlisting}[language=Julia, label={code:resampling}, caption={Resampling of a Nyquist sampled signal.}, captionpos=b]
N_low = 64
x_min = 0.0
x_max = 8*2π

xs_low = range(x_min, x_max, length=N_low+1)[1:N_low]
xs_high = range(x_min, x_max, length=5000)[1:end-1]
f(x) = sin(0.5*x) + cos(x) + cos(2 * x) + sin(0.25*x)
arr_low = f.(xs_low)
arr_high = f.(xs_high)

N = 1000
xs_interp = range(x_min, x_max, length=N+1)[1:N]
arr_interp = resample(arr_low, N)
\end{lstlisting}

         \begin{figure}[h]
             \centering
             \includegraphics[width=.45\textwidth]{figures/resampling.pdf}
             \caption{The array with low sampling, $\sinc$ interpolation and high sampling all contain the same information.}
             \label{fig:resampling}
         \end{figure}

 %   In cases where the input signal is Nyquist sampled, Sinc interpolation can lead
 %   to results which look far superior to classic interpolation schemes

 %   \begin{figure}
 %       \centering
 %       \subcaptionbox{$\SI{32}{px}\times\SI{32}{px}$}{\includegraphics[width=.15\textwidth]{%%figures/jinc_small.png}}
   %     \subcaptionbox{\lc{ImageTransformations.imresize}}{\includegraphics[width=.15\textwid%th]{figures/jinc_imresize.png}}
%        \subcaptionbox{\lc{FourierTools.resample}}{\includegraphics[width=.15\textwidth]{figures/jinc_sinc.png}}
  %      \caption{Caption}
  %      \label{fig:sinc_interp_image}
   % \end{figure}

    \subsection{Subpixel Shifting of Signals}

    In \autoref{code:shifting} we show that we can shift a Nyquist sampled signal by an sub pixel amount.
    We evaluate the function \lc{f(x)} shifted by a distance of $0.25$, a quarter of its period. Since the original signal is periodic on the grid and was sampled according to the Nyquist limit, the full information
    of the signal was captured and hence an evaluation of the Fourier series provides identical results up to rounding errors.

\begin{lstlisting}[language=Julia, label={code:shifting}, caption={Shifting a Nyquist sampled signal by a sub pixel amount.}, captionpos=b]
julia> f(x) = sin(x * 2π / 5)
f (generic function with 1 method)

julia> x = 0:4
0:4

julia> transpose(f.(x))
1x5 transpose(::Vector{Float64}) with eltype Float64:
 0.0  0.951057  0.587785  -0.587785  -0.951057

julia> transpose(f.(x .- 0.25))
1x5 transpose(::Vector{Float64}) with eltype Float64:
 -0.309017  0.809017  0.809017  -0.309017  -1.0

julia> transpose(shift(f.(x), 0.25))
1x5 transpose(::Vector{Float64}) with eltype Float64:
 -0.309017  0.809017  0.809017  -0.309017  -1.0
\end{lstlisting}
Additionally, shifting based on Fourier transforms is also fully revertible, with the only exception being the even--sized Nyquist frequency as mentioned above.

\subsubsection{Handling the Nyquist Amplitude}
In \autoref{code:shifting2} we demonstrate that handling the Nyquist amplitude
in the phase ramp is crucial for correct output results.
Otherwise, shifting a real signal would result in a nonzero imaginary part.
For a real signal one could also take the real part at the end.

\begin{lstlisting}[language=Julia, label={code:shifting2}, caption={Fixing the Nyquist amplitude of the shifting exponential phase ramp.}, captionpos=b]
julia> x = [1.0, 0.0, 1.0, 0.0];

julia> exp_term = cispi.(2 .* 0.5 .*
                         fftfreq(size(x, 1))) # shift 0.5 indices
4-element Vector{ComplexF64}:
                1.0 + 0.0im
 0.7071067811865476 + 0.7071067811865476im
                0.0 - 1.0im
 0.7071067811865476 - 0.7071067811865476im

julia> ifft(exp_term) # real space kernel is not real
4-element Vector{ComplexF64}:
   0.6035533905932737 - 0.25im
 -0.10355339059327379 + 0.25im
 -0.10355339059327379 - 0.25im
   0.6035533905932737 + 0.25im

julia> ifft(fft(x) .* exp_term) # contains imaginary parts
4-element Vector{ComplexF64}:
 0.5 - 0.5im
 0.5 + 0.5im
 0.5 - 0.5im
 0.5 + 0.5im

julia> exp_term[div(size(x, 1), 2) + 1] = real(exp_term[div(size(x, 1), 2) + 1])
0.0

julia> ifft(exp_term) # real space kernel is real
4-element Vector{ComplexF64}:
   0.6035533905932737 + 0.0im
 -0.10355339059327379 + 0.0im
 -0.10355339059327379 + 0.0im
   0.6035533905932737 + 0.0im

julia> ifft(fft(x) .* exp_term) # real output
4-element Vector{ComplexF64}:
 0.5 + 0.0im
 0.5 + 0.0im
 0.5 + 0.0im
 0.5 + 0.0im
\end{lstlisting}

Also for complex--valued signals this modification is required, as the real and imaginary part would mix.
This is demonstrated in \autoref{code:shifting3} where we first shift again
by the na\"ive kernel. And as we can see, shifting the signal with the wrong kernel, can lead to completely wrong results.
% I disagree. I think it is matter of interpretation here.
% There is no right of wrong in this case.
\begin{lstlisting}[language=Julia, label={code:shifting3}, caption={Handling of Nyquist frequency is also for complex signals important.}, captionpos=b]
julia> x = [1.0 + 1.0im, 0, 1.0 + 1.0im, 0]
4-element Vector{ComplexF64}:
 1.0 + 1.0im
 0.0 + 0.0im
 1.0 + 1.0im
 0.0 + 0.0im

julia> ifft(fft(x) .* cispi.(2 .* 0.5 .*fftfreq(size(x, 1)))) # wrong because of Nyquist amplitude
4-element Vector{ComplexF64}:
 1.0 + 0.0im
 0.0 + 1.0im
 1.0 + 0.0im
 0.0 + 1.0im

julia> shift(x, 0.5) # Using the correct kernel
4-element Vector{ComplexF64}:
 0.5 + 0.5im
 0.5 + 0.5im
 0.5 + 0.5im
 0.5 + 0.5im
\end{lstlisting}

    \subsection{Shearing and Rotation}
        \autoref{fig:shearandrotation} shows the results of shearing and rotating an image.
        The image is provided by \verb|TestImages.jl|\footnote{\url{https://github.com/JuliaImages/TestImages.jl}}.
        Since we do not resize the image before, the shearing happens completely within the boundaries of the image.
        This circular property is induced by the circularity assumption of the DFT itself.
        The rotated image has even more complex patterns in the boundary regions.
        The reason is that the rotation is based on three shears and hence
        each shear introduces circular artifacts which are shifted again by following shearing.
        \begin{figure}[h]
            \subcaptionbox{Sheared by \SI{150}{px}.\label{fig:sheared}}{\includegraphics[width=.23\textwidth]{figures/img_sheared.png}}
            \subcaptionbox{Rotated by \SI{50}{\degree}.\label{fig:rotated}}{\includegraphics[width=.23\textwidth]{figures/img_rotated.png}}
            \caption{Mandril testimage.}
            \label{fig:shearandrotation}
        \end{figure}
        Fortunately, those artifacts are completely revertible\footnote{in some case the shifting exponential multiplies with $0$ which cannot be recovered.}.
        To rotate without such boundary artifacts, one has to pre-allocate a suitable array beforehand.

\section{Conclusion}
    \verb|FourierTools.jl| offers convenient functions to handle data in Fourier space.
    In additon to those functions, we also provide algorithms for different kinds of array manipulation.
    % Hier wird es total Cyptisch!
    The advantage of those algorithms is, that they do not add additional information
    to the signal but instead rely on the available information
    by reinterpreting the information Fourier space.

\section{Acknowledgement}
We want to thank all people participating in the Discourse discussion\footnote{\url{https://discourse.julialang.org/t/sinc-interpolation-based-on-fft/52512}} and contributing to the clarification of how the boundary is the best handled.
Especially, Steven G. Johnson pointed out the underlying mathematics.

\input{bib.tex}

\end{document}
% Inspired by the International Journal of Computer Applications template


\subsection{Integration by Parts (Reversing the Product Rule)}
When a $u$-substitution won't work because the integrand is the product of two functions that are not related by a derivative, integration by parts may be able to help us.
Starting from the product rule\footnote{This justification isn't completely rigorous because you can't always treat differentials like fractions, but the overall result is correct.},
\begin{align*}
	\dd{}{x}{(uv)} &= u\dd{v}{x} + v\dd{u}{x} \\
	\int{\dd{}{x}{(uv)}\d{x}} &= \int{u\dd{v}{x}\d{x}} + \int{v\dd{u}{x}\d{x}} \\
	\int{\d{(uv)}} &= \int{u\d{v}} + \int{v\d{u}} \\
	uv &= \int{u\d{v}} + \int{v\d{u}} \\
	\int{u\d{v}} &= uv - \int{v\d{u}}.
\end{align*}

Essentially, if we are integrating a function that is a product of something we can differentiate ($u$) and something we can integrate ($\d{v}$), then we can do integration by parts.
\begin{example}
	Solve the following indefinite integral:
	\begin{equation*}
		\int{x\sin{x}\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	Since $x$ and $\sin{x}$ don't have any parts that are related by a derivative, a $u$-substitution wouldn't be helpful.
	However, we know how to differentiate $x$ and integrate $\sin{x}$, so integration by parts is a good strategy to try.
	\begin{align*}
		u = x &\text{ and } \d{u} = \d{x} \\
		\d{v} = \sin{x}\d{x} &\text{ and } v =\footnotemark -\cos{x} \\
		\int{x\sin{x}\d{x}} &= -x\cos{x} - \int{-\cos{x}\d{x}} \\
		&= -x\cos{x} + \sin{x} + C.
	\end{align*}
\end{answer}
\footnotetext{We don't include the $+C$ when integrating $\d{v}$. We'll still include one at the end.}

\subsubsection{LIPET (How to choose $u$)}
The tricky part of integration by parts is choosing what should be $u$ and what should be $\d{v}$.
If in the previous example we had instead chosen $u=\sin{x}$, we would have gotten that $\int{x\sin{x}\d{x}} = \frac{x^2}{2}-\int{\frac{x^2}{2}\cos{x}\d{x}}$, which although true, doesn't give us an easier function to integrate.
Generally, we want to choose $u$ such that $\d{u}$ is a ``simplier'' function to integrate.
This can mean a lower-degree polynomial, or getting rid of functions like $\ln{x}$ and replacing them with polynomials.
You can remember the acronym \textbf{LIPET} as a guide to help you.
\begin{itemize}[align=left, leftmargin=0.66in]
	\item[\textbf{L}ogarithms]
	\item[\textbf{I}nverse Trig Functions]
	\item[\textbf{P}olynomials]
	\item[\textbf{E}xponentials]
	\item[\textbf{T}rig Functions]
\end{itemize}

\begin{example}
	Solve the following indefinite integral:
	\begin{equation*}
		\int{x^3\ln{x}\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	Following LIPET, we select $u = \ln{x}$.
	\begin{align*}
		u = \ln{x} &\text{ and } \d{u} = \frac{\d{x}}{x} \\
		\d{v} = x^3\d{x} &\text{ and } v = x^4/4 \\
		\int{x^3\ln{x}\d{x}} &= \frac{x^4\ln{x}}{4} - \int{\frac{x^3}{4}\d{x}} \\
		&= \frac{x^4\ln{x}}{4} - \frac{x^4}{16} + C.
	\end{align*}
\end{answer}

Sometimes, we might have to do integration by parts multiple times.
It might seem like integration by parts is leading us in circles, but this can actually be a good thing, allowing us to solve the integral if we're observant.
\begin{example}
	Solve the following indefinite integral:
	\begin{equation*}
		\int{e^{-x}\cos{x}\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	Following LIPET, $u=e^{-x}$.
	\begin{align*}
		u = e^{-x} &\text{ and } \d{u} = -e^{-x}\d{x} \\
		\d{v} = \cos{x}\d{x} &\text{ and } v = \sin{x} \\
		\int{e^{-x}\cos{x}\d{x}} &= e^{-x}\sin{x} + \int{e^{-x}\sin{x}\d{x}}.
	\end{align*}

	Following LIPET, $u = e^{-x}$ again.
	\begin{align*}
		u = e^{-x} &\text{ and } \d{u} = -e^{-x}\d{x} \\
		\d{v} = \sin{x}\d{x} &\text{ and } v = -\cos{x} \\
		e^{-x}\sin{x} + \int{e^{-x}\cos{x}\d{x}} &= e^{-x}\sin{x} - e^{-x}\cos{x} - \int{e^{-x}\cos{x}\d{x}}
	\end{align*}

	It might seem like we've hit a dead end, since we're back trying to integrate the exact same function we started with.
	However, if we rearrange our equations a little bit, we'll see that we're actually really close to a solution.
	\begin{align*}
		\int{e^{-x}\cos{x}\d{x}} &= e^{-x}\sin{x} - e^{-x}\cos{x} - \int{e^{-x}\cos{x}\d{x}} \\
		2\int{e^{-x}\cos{x}\d{x}} &=\footnotemark\hspace{5pt} e^{-x}\sin{x} - e^{-x}\cos{x} + C \\
		\int{e^{-x}\cos{x}\d{x}} &= \frac{e^{-x}\sin{x} - e^{-x}\cos{x}}{2} + C.
	\end{align*}
\end{answer}
\footnotetext{We need to remember to include a $+C$. Usually it would come from completely solving an integral, but here we just did some rearranging.}

\subsubsection{Inverse Trig Functions}
We can use tabular integration to integrate the inverse trig functions.
We'll also end up applying two different strategies: Integration by parts and $u$-substitution to solve one integral.
\begin{align*}
	u = \arcsin{x} &\text{ and } \d{u} = \frac{1}{\sqrt{1-x^2}} \\
	\d{v} = \d{x} &\text{ and } v = x \\
	\int{\arcsin{x}\d{x}} &= x\arcsin{x} - \int{\frac{x}{\sqrt{1-x^2}}\d{x}} \\
	u = 1-x^2 &\text{ and } x\d{x} = \frac{-1}{2}\d{u} \text{ (this is a different $u$ than before)} \\
	&= x\arcsin{x} + \frac{1}{2}\int{\frac{\d{u}}{\sqrt{u}}} \\
	&= x\arcsin{x} + \frac{1}{2}\left(\frac{\sqrt{u}}{1/2}\right) + C \\
	&= x\arcsin{x} + \sqrt{u} + C \\
	&= x\arcsin{x} + \sqrt{1-x^2} + C.
\end{align*}

We can do the exact same thing to find the integral of $\arccos{x}$.
\begin{equation*}
	\int{\arccos{x}\d{x}} = x\arccos{x} - \sqrt{1-x^2} + C.
\end{equation*}

The integration by parts into $u$-substitution idea even works for $\arctan{x}$.
\begin{align*}
	u = \arctan{x} &\text{ and } \d{u} = \frac{1}{1+x^2} \\
	\d{v} = \d{x} &\text{ and } v = x \\
	\int{\arctan{x}\d{x}} &= x\arctan{x} - \int{\frac{x}{1+x^2}\d{x}} \\
	u = 1+x^2 &\text{ and } x\d{x} = \frac{1}{2}\d{u} \text{ (this is a different $u$ than before)} \\
	&= x\arctan{x} - \frac{1}{2}\int{\frac{\d{u}}{u}} \\
	&= x\arctan{x} - \frac{1}{2}\ln{\abs{u}} + C \\
	&= x\arctan{x} - \frac{1}{2}\ln{\abs{1+x^2}} + C.
\end{align*}

We can do the same thing to find the integral of $\arccot{x}$.
\begin{equation*}
	\int{\arccot{x}\d{x}} = x\arccot{x} + \frac{1}{2}\ln{\abs{1+x^2}} + C.
\end{equation*}

The integrals of $\arcsec{x}$ and $\arccsc{x}$ involve a special type of function you probably haven't seen before, $\text{arccosh}{x}$.
However, if you're familiar with this function and it's derivative, you can still apply integration by parts to find their antiderivatives.
\begin{align*}
	\int{\arcsec{x}\d{x}} &= x\arcsec{x} - \text{arccosh}{x} + C \\
	\int{\arccsc{x}\d{x}} &= x\arccsc{x} + \text{arccosh}{x} + C.
\end{align*}

\subsubsection{Tabular Integration}
If we're trying to integrate a function like $x^4e^{-x}$ using integration by parts, we quickly realize that we'll need to perform integration by parts 4 times, choosing the same value for $\d{v}$ every time.
Since this would be rather tedious and repetitive, we can extract the essential elements of integration by parts into a table and perform integration by parts many times without needing to repeat ourselves.

In integration by parts, we essentially choose one function ($u$) to continually differentiate and another ($\d{v}$) to continually integrate.
We then alternate signs with each iteration of integration by parts, due to negative signs canceling each other out.
So, in our table, we'll have three columns, one where we'll continually differentiate a function, usually until its a constant, a column to keep track of the sign, and a final column that we'll continually integrate.
Then, all we'll have to do is multiply across columns and sum across rows to get our result from repeated integration by parts.

\begin{example}
	Solve the following indefinite integral:
	\begin{equation*}
		\int{x^4e^{-x}\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	Following LIPET, $x^4$ is our function to differentiate and $e^{-x}$ is our function to integrate.
	Setting up our table,
	\begin{figure}[H]
		\label{tabular}
		\centering
		\includegraphics[width = 0.5\textwidth]{./integrals/integration_by_parts/tabular.png}
	\end{figure}
	Multiplying across each arrow and summing,
	\begin{align*}
		\int{x^4e^{-x}\d{x}} &= x^4\left(-e^{-x}\right) + 4x^3(-1)e^{-x} + 12x^2\left(-e^{-x}\right) + 24x(-1)e^{-x} + 24\left(-e^{-x}\right) + C \\
		&= -x^4e^{-x} - 4x^3e^{-x} - 12x^2e^{-x} - 24xe^{-x} - 24e^{-x} + C \\
		&= -e^{-x}\left(x^4 + 4x^3 + 12x^2 + 24x + 24\right) + C.
	\end{align*}
\end{answer}


\section{Antiderivatives and Indefinite Integration}\label{sec:antider}

We have spent considerable time considering the derivatives of a function and their applications. In the following chapters, we are going to starting thinking in ``the other direction.'' That is, given a function $f(x)$, we are going to consider functions $F(x)$ such that $F'(x) = f(x)$. There are numerous reasons this will prove to be useful: these functions will help us compute areas, volumes, mass, force, pressure, work, and much more.

Given a function $y=f(x)$, a \textit{differential equation} is one that incorporates $y$, $x$, and the derivatives of $y$. For instance, a simple differential equation is: $$y' = 2x.$$

Solving a differential equation amounts to finding a function $y$ that satisfies the given equation. Take a moment and consider that equation; can you find a function $y$ such that $y' = 2x$?

Can you find another?

And yet another?

Hopefully one was able to come up with at least one solution: $y = x^2$. ``Finding another'' may have seemed impossible until one realizes that a function like $y=x^2+1$ also has a derivative of $2x$. Once that discovery is made, finding ``yet another'' is not difficult; the function $y = x^2 + 123,456,789$ also has a derivative of $2x$. The differential equation $y' = 2x$ has many solutions. This leads us to some definitions.
%\enlargethispage{3\baselineskip}

\begin{definition}{Antiderivatives and Indefinite Integrals}{antider}
{Let a function $f(x)$ be given. An \textbf{antiderivative} of $f(x)$ on an open interval $ I $ is a function $F(x)$ such that $\Fp(x) = f(x)$ for all $ x\in I $.\index{antiderivative}\index{indefinite integral}\index{integration!indefinite}\\

The set of all antiderivatives of $f(x)$ is referred to as the \textit{General Antiderivative}, ot the \textbf{(indefinite) integral of $f$}, denoted by $$\int f(x) \ dx.$$
}
\end{definition}

The process of finding the indefinite integral is called \dfont{integration} (or \dfont{integrating $f(x)$}).
Make a note about our definition: we refer to \textit{an} antiderivative of $f$, as opposed to \textit{the} antiderivative of $f$, since there is \textit{always} an infinite number of them. We often use upper-case letters to denote antiderivatives.

Knowing one antiderivative of $f$ allows us to find infinitely more, simply by adding a constant. Not only does this give us \textit{more} antiderivatives, it gives us \textit{all} of them.

\begin{theorem}{Antiderivative Forms}{antideriv_const}
{Let $F(x)$ and $G(x)$ be antiderivatives of $f(x)$. Then there exists a constant $C$ such that $$G(x) = F(x) + C.$$
}
\end{theorem}

To see why this is true, observe that if $ F $ and $ G $ are both antiderivatives of $ f $, then $ F'=G'=f $, so $ \frac{d}{dx}(G-F)=G'-F'=f-f=0$, so $ G-F $ is a constant function giving $ G-F=C $, so $ G=F+C$.

So, given a function $f$ and one of its antiderivatives $F$, we know \textit{all} antiderivatives of $f$ have the form $F(x) + C$ for some constant $C$. Using Definition \ref{def:antider}, we can say that $$\int f(x) \ dx = F(x) + C.$$

Let's analyze this indefinite integral notation.\index{integration!notation}

\begin{figure}
\begin{center}
\begin{tikzpicture}[scale=1.5,>=latex]
%\draw [thin,step=1cm] (0,0) grid  (3,3);
\draw  (2,2) node {$\displaystyle \int f(x)\ dx=F(x)+C$};

\draw [{\colorone}] (1,1) node [](a) {\scriptsize Integrand};
\draw [{\colorone},->] (a) -- (1.1,1.75);

\draw [{\colorone}] (0,3) node [text width=32pt,align=center] (b) {\scriptsize \centering Integration\\[-5pt] symbol};
\draw [{\colorone},->] (b) -- (.6,2.1);

\draw [{\colorone}] (2,3) node [text width=32pt,align=center] (c) {\scriptsize \centering Differential\\[-5pt] of $x$};
\draw [{\colorone},->] (c) -- (1.7,2.3);

\draw [{\colorone}] (2.5,1) node [text width=32pt,align=center] (c) {\scriptsize \centering One\\[-5pt] antiderivative};
\draw [{\colorone},->] (c) -- (2.5,1.75);

\draw [{\colorone}] (3.5,3) node [text width=35pt,align=center] (c) {\scriptsize \centering Constant of\\[-5pt] integration};
\draw [{\colorone},->] (c) -- (3.45,2.3);
\end{tikzpicture}

\caption{Understanding the indefinite integral notation.\label{fig:anti1}}
\end{center}
\end{figure}

%\includegraphics{figures/figanti1}
%\captionsetup{type=figure}%
%\caption{Understanding the indefinite integral notation.}\label{fig:anti1}

Figure \ref{fig:anti1} shows the typical notation of the indefinite integral. The integration symbol, $\int$, is in reality an ``elongated S,'' representing ``take the sum.'' We will later see how \textit{sums} and \textit{antiderivatives} are related.

The function we want to find an antiderivative of is called the \textit{integrand}. It contains the differential of the variable we are integrating with respect to. The $\int$ symbol and the differential $dx$ are not ``bookends'' with a function sandwiched in between; rather, the symbol $\int$ means ``find all antiderivatives of what follows,'' and the function $f(x)$ and $dx$ are multiplied together; the $dx$ does not ``just sit there.''

Let's practice using this notation.\\

\begin{example}{Evaluating indefinite integrals}{ex_anti2}{
Evaluate $\displaystyle \int \sin x\ dx.$}
\end{example}
\begin{solution}
{We are asked to find all functions $F(x)$ such that $\Fp(x) = \sin x$. Some thought will lead us to one solution: $F(x) = -\cos x$, because $\frac{d}{dx}(-\cos x) = \sin x$.

The indefinite integral of $\sin x$ is thus $-\cos x$, plus a constant of integration. So:
$$\int \sin x \ dx = -\cos x + C.$$
\vskip-\baselineskip
}
\end{solution}

{\bf Common mistakes:}
One habit students make with integrals is to \ifont{drop the dx} at the end of the integral.
This is required! Think of the integral as a set of parenthesis.
Both are required so it is clear where the integrand ends and what variable you are integrating with respect to.

Another common mistake is to \ifont{forget the +C} for indefinite integrals.

A commonly asked question is ``What happened to the $dx$?'' The unenlightened response is ``Don't worry about it. It just goes away.'' A full understanding includes the following.

This process of \textit{antidifferentiation} is really solving a \textit{differential} question. The integral $$\int \sin x\ dx$$ presents us with a differential, $dy = \sin x\ dx$. It is asking: ``What is $y$?'' We found lots of solutions, all of the form $y = -\cos x+C$.

%We can view integration in the following way.
Letting $dy = \sin x\ dx$,  rewrite
$$\int \sin x \ dx \quad \text{as}\quad \int  dy.$$
This is asking: ``What functions have a differential of the form $dy$?'' The answer is ``Functions of the form $y+C$, where $C$ is a constant.'' What is $y$? We have lots of choices, all differing by a constant; the simplest choice is $y = -\cos x$.

Understanding all of this is more important later as we try to find antiderivatives of more complicated functions. In this section, we will simply explore the rules of indefinite integration, and one can succeed for now with answering ``What happened to the $dx$?'' with ``It went away.''

Let's practice once more before stating integration rules.\\

\begin{example}{Evaluating indefinite integrals}{ex_anti3}{
Evaluate $\ds \int (3x^2 + 4x+5)\ dx$.}
\end{example}

\begin{solution}
{We seek a function $F(x)$ whose derivative is $3x^2+4x+5$. When taking derivatives, we can consider functions term--by--term, so we can likely do that here.

What functions have a derivative of $3x^2$? Some thought will lead us to a cubic, specifically $x^3+C_1$, where $C_1$ is a constant.

What functions have a derivative of $4x$? Here the $x$ term is raised to the first power, so we likely seek a quadratic. Some thought should lead us to $2x^2+C_2$, where $C_2$ is a constant.

Finally, what functions have a derivative of $5$? Functions of the form $5x+C_3$, where $C_3$ is a constant.

Our answer appears to be
$$\int (3x^2+4x+5)\ dx = x^3+C_1+2x^2+C_2+5x+C_3.$$ We do not need three separate constants of integration; combine them as one constant, giving the final answer of
$$\int (3x^2+4x+5)\ dx = x^3+2x^2+5x+C.$$

It is easy to verify our answer; take the derivative of $x^3+2x^3+5x+C$ and see we indeed get $3x^2+4x+5$.
}
\end{solution}

This final step of ``verifying our answer'' is important both practically and theoretically. In general, taking derivatives is easier than finding antiderivatives so checking our work is easy and vital as we learn.

We also see that taking the derivative of our answer returns the function in the integrand. Thus we can say that: $$\frac{d}{dx}\left(\int f(x)\ dx\right) = f(x).$$
Differentiation ``undoes'' the work done by antidifferentiation.

The table at the end of Chapter 4  gave a list of the derivatives of common functions we had learned at that point. We restate part of that list here to stress the relationship between derivatives and antiderivatives. This list will also be useful as a glossary of common antiderivatives as we learn.

%\newlength{\bh}%box height
%\newlength{\bd}%box depth
%\settoheight{\bh}{$\frac{d}{dx}$}
%\settodepth{\bd}{$\frac{d}{dx}$}

%\newcommand{\myrule}{\rule[-\bd]{0pt}{\bh+\bd}}%\ $\frac{d}{dx}$
%\newcommand{\myrule}{\rule[-4pt]{0pt}{13pt}}%\ $\frac{d}{dx}$

\begin{theorem}{Derivatives and Antiderivatives}{indef_alg}
\begin{minipage}[t]{.45\textwidth}
Common Differentiation Rules\rule[-2pt]{0pt}{1pt}
\begin{enumerate}
\item \myrule$\frac{d}{dx}\big(cf(x) \big) = c\cdot \fp(x)$
\item \myrule$\frac{d}{dx}\big(f(x)\pm g(x) \big) = $
\vskip .05\baselineskip
\myrule$\fp(x)\pm g'(x)$
\item $\frac{d}{dx}\big(C \big) = 0$\myrule
\item $\frac{d}{dx}\big(x \big) = 1$\myrule
\item $\frac{d}{dx}\big(x^n \big) = n\cdot x^{n-1}$\myrule
\item $\frac{d}{dx}\big(\sin x \big) = \cos x$\myrule
\item $\frac{d}{dx}\big(\cos x \big) = -\sin x$\myrule
\item $\frac{d}{dx}\big(\tan x \big) = \sec^2 x$\myrule
\item $\frac{d}{dx}\big(\csc x \big) = -\csc x\cot x$\myrule
\item $\frac{d}{dx}\big(\sec x \big) = \sec x\tan x$\myrule
\item $\frac{d}{dx}\big(\cot x \big) = -\csc^2 x$\myrule
\item $\frac{d}{dx}\big(e^ x \big) = e^x$\myrule
\item $\frac{d}{dx}\big(a^x \big) = \ln a\cdot a^x$\myrule
\item $\frac{d}{dx}\big(\ln x \big) =\frac{d}{dx}\big(\ln |x| \big) = \frac1 x$\myrule
\item $\frac{d}{dx}\big(\cosh x\big) = \sinh x$
\item $\frac{d}{dx}\big(\sinh x\big) = \cosh x$
\item $\frac{d}{dx}\big(\ln(\cosh x)\big) = \tanh x$
\item $\frac{d}{dx}\big(\ln(\sinh x)\big) = \coth x$
\item $\ds \frac{d}{dx}\big(\sin^{-1}(x)\big) = \frac{1}{\sqrt{1-x^2}}$ \label{invsin}
%\item $\ds \frac{d}{dx}\big(\sec^{-1}(x)\big) = \frac{1}{ x \sqrt{x^2-1}}$
\item $\ds \frac{d}{dx}\big(\tan^{-1}(x)\big) = \frac{1}{1+x^2}$ \label{invtan}
%\item $\frac{d}{dx}\big(\csch x\big) = -\csch x\coth x$
%\item $\frac{d}{dx}\big(\coth x\big) = -\csch^2x$

\end{enumerate}
\end{minipage}
\begin{minipage}[t]{.55\textwidth}
Common Indefinite Integral Rules\rule[-2pt]{0pt}{1pt}
\begin{enumerate}
\item \myrule$\int c\cdot f(x)\ dx = c\cdot \int f(x)\ dx$
\item \myrule$\int \big(f(x)\pm g(x)\big)\ dx =$
\vskip .05\baselineskip
\myrule$ \int f(x)\ dx\pm \int g(x)\ dx$
\item $\int 0\ dx = C$\myrule
\item $\int 1\ dx = \int dx = x+C$\myrule
\item $\int x^n\ dx =\frac{1}{n+1}x^{n+1}+ C$\quad {\scriptsize ($n\neq -1$)}\myrule
\item $\int \cos x\ dx = \sin x+C$\myrule
\item $\int \sin x\ dx = -\cos x+C$\myrule
\item $\int \sec^2 x\ dx = \tan x+C$\myrule
\item $\int \csc x\cot  x\ dx = -\csc x+C$\myrule
\item $\int \sec x\tan x\ dx = \sec x+C$\myrule
\item $\int \csc^2 x\ dx = -\cot x+C$\myrule
\item $\int e^x\ dx = e^x+C$\myrule
\item $\int a^x\ dx = \frac{1}{\ln a}\cdot a^x+C$\myrule
\item $\int \frac{1}x\ dx = \ln |x|+C$\myrule
\item $\ds\int \cosh x\ dx = \sinh x+C$ \myrule
\item $\ds\int \sinh x\ dx = \cosh x+C$ \myrule
\item $\ds\int \tanh x\ dx = \ln(\cosh x) +C$ \myrule
\item $\ds\int \coth x\ dx = \ln|\sinh x\,|+C$ \myrule
\item $\ds     \int \frac{1}{\sqrt{1-x^2}} dx = \sin^{-1}(x) +C $
%\item $\ds \frac{d}{dx}\big(\sec^{-1}(x)\big) = \frac{1}{ x \sqrt{x^2-1}}$
\item $\ds  \int \frac{1}{1+x^2} \; dx = \tan^{-1}(x) +C$
\end{enumerate}
\end{minipage}%\\
%
%\textit{\small (continued $\ldots$)}
\end{theorem}
%\addtocounter{theoremcounter}{-1}
%
%\theorem{thm:indef_algb}{Derivatives and Antiderivatives -- Continued}
%{\begin{minipage}[t]{.45\specialboxlength}
%Common Differentiation Rules\rule[-2pt]{0pt}{1pt}
%\begin{enumerate}\addtocounter{enumi}{5}
%%\item \myrule$\frac{d}{dx}\big(cf(x) \big) = c\cdot \fp(x)$
%%\item \myrule$\frac{d}{dx}\big(f(x)\pm g(x) \big) = $
%%\vskip .05\baselineskip
%%\myrule$\fp(x)\pm g'(x)$
%%\item $\frac{d}{dx}\big(C \big) = 0$\myrule
%%\item $\frac{d}{dx}\big(x \big) = 1$\myrule
%%\item $\frac{d}{dx}\big(x^n \big) = n\cdot x^{n-1}$\myrule
%\item $\frac{d}{dx}\big(\sin x \big) = \cos x$\myrule
%\item $\frac{d}{dx}\big(\cos x \big) = -\sin x$\myrule
%\item $\frac{d}{dx}\big(\tan x \big) = \sec^2 x$\myrule
%\item $\frac{d}{dx}\big(\csc x \big) = -\csc x\cot x$\myrule
%\item $\frac{d}{dx}\big(\sec x \big) = \sec x\tan x$\myrule
%\item $\frac{d}{dx}\big(\cot x \big) = -\csc^2 x$\myrule
%\item $\frac{d}{dx}\big(e^ x \big) = e^x$\myrule
%\item $\frac{d}{dx}\big(a^x \big) = \ln a\cdot a^x$\myrule
%\item $\frac{d}{dx}\big(\ln x \big) = \frac1 x$\myrule
%\end{enumerate}
%\end{minipage}
%\begin{minipage}[t]{.55\specialboxlength}
%Common Indefinite Integral Rules\rule[-2pt]{0pt}{1pt}
%\begin{enumerate}\addtocounter{enumi}{5}
%%\item \myrule$\int c\cdot f(x)\ dx = c\cdot \int f(x)\ dx$
%%\item \myrule$\int \big(f(x)\pm g(x)\big)\ dx =$
%%\vskip .05\baselineskip
%%\myrule$ \int f(x)\ dx\pm \int g(x)\ dx$
%%\item $\int 0\ dx = C$\myrule
%%\item $\int 1\ dx = \int dx = x+C$\myrule
%%\item $\int x^n\ dx =\frac{1}{n+1}x^{n+1}+ C$\quad {\scriptsize ($n\neq -1$)}\myrule
%\item $\int \cos x\ dx = \sin x+C$\myrule
%\item $\int \sin x\ dx = -\cos x+C$\myrule
%\item $\int \sec^2 x\ dx = \tan x+C$\myrule
%\item $\int \csc x\cot  x\ dx = -\csc x+C$\myrule
%\item $\int \sec x\tan x\ dx = \sec x+C$\myrule
%\item $\int \csc^2 x\ dx = -\cot x+C$\myrule
%\item $\int e^x\ dx = e^x+C$\myrule
%\item $\int a^x\ dx = \frac{1}{\ln a}\cdot a^x+C$\myrule
%\item $\int \frac{1}x\ dx = \ln |x|+C$\myrule
%\end{enumerate}
%\end{minipage}
%}

We highlight a few important points from Theorem \ref{thm:indef_alg}:
\begin{itemize}
	\item		Rule \#1 states $\int c\cdot f(x)\ dx = c\cdot \int f(x)\ dx$. This is the Constant Multiple Rule: \index{Constant Multiple Rule!of integration} we can temporarily ignore constants when finding antiderivatives, just as we did when computing derivatives (i.e., $\frac{d}{dx}\big(3x^2\big)$ is just as easy to compute as $\frac{d}{dx}\big(x^2\big)$). An example:
	$$\int 5\cos x\ dx = 5\cdot\int \cos x\ dx = 5\cdot (\sin x+C) = 5\sin x + C.$$
	In the last step we can consider the constant as also being multiplied by 5, but ``5 times a constant'' is still a constant, so we just write ``$C$\,''.
	\item		Rule \#2 is the Sum/Difference Rule:\index{integration!Sum/Difference Rule}\index{Sum/Difference Rule!of integration} we can split integrals apart when the integrand contains terms that are added/subtracted, as we did in Example \ref{ex_anti3}. So:
	\begin{align*}
	\int(3x^2+4x+5)\ dx &= \int 3x^2\ dx + \int 4x\ dx + \int 5\ dx \\
											&= 3\int x^2\ dx + 4\int x\ dx + \int 5 \ dx\\
											&= 3\cdot \frac13x^3 + 4\cdot \frac12x^2+5x+C\\
											&= x^3+2x^2+5x+C
	\end{align*}
	In practice we generally do not write out all these steps, but we demonstrate them here for completeness.
	\item		Rule \#5 is the Power Rule of indefinite integration.\index{integration!Power Rule}\index{Power Rule!integration} There are two important things to keep in mind:
		\begin{enumerate}
		\item		Notice the restriction that $n\neq -1$. This is important: $\int \frac{1}{x}\ dx \neq $ ``$\frac{1}{0}x^0+C$''; rather, see Rule \#14.
		\item		We are presenting antidifferentiation as the ``inverse operation'' of differentiation. Here is a useful quote to remember:
		\begin{quote}%\centering
		``Inverse operations do the opposite things in the opposite order.''
		\end{quote}
		When taking a derivative using the Power Rule, we \textbf{first} \textit{multiply} by the power, then \textbf{second} \textit{subtract} 1 from the power. To find the antiderivative, do the opposite things in the opposite order: \textbf{first} \textit{add} one to the power, then \textbf{second} \textit{divide} by the power.
			\end{enumerate}
		\item		Note that Rule \#14 incorporates the absolute value of $x$. The exercises will work the reader through why this is the case; for now, know the absolute value is important and cannot be ignored.
\end{itemize}

Note that we don't have properties to deal with products or quotients of functions, that is, in general
%Just like with derivatives, the following will \dfont{NOT} work:
$$\int f(x)\cdot g(x)\,dx\neq \int f(x)\,dx\int g(x)\,dx.$$
$$\int \frac{f(x)}{g(x)}\,dx\neq \frac{\int f(x)\,dx}{\int g(x)\,dx}.$$
With derivatives, we had the product and quotient rules to deal with these cases.
For integrals, we have no such rules, but we will learn a variety of different techniques to deal with these cases.

\begin{example}{Indefinite Integral}{IndefiniteIntegral2}
If $f'(x)=x^4+2x-8\sin x$ then what is $f(x)$?
\end{example}

\begin{solution}
The answer is:
$$\begin{array}{rcl}
\ds{f(x)=\int f'(x)\,dx}&=&\ds{\int \left(x^4+2x-8\sin x\right)\,dx}\\
\\
&=&\ds{ \int x^4 \,dx + 2\int x\,dx -8 \int \sin x\,dx }\\
\\
&=&\ds{\frac{x^5}{5}+x^2+8\cos x+C,}\\
\end{array}$$
where $C$ is a constant.
\end{solution}

\begin{example}{Indefinite Integral}{IndefiniteIntegral3}
Find the indefinite integral: $\ds\int 3x^2\,dx$.
\end{example}
\begin{solution}
	$$\begin{array}{rcl}
	\ds{\int 3x^2\,dx} & = & \ds{3\int x^2\,dx}\\
	\\
	&=&\ds{3\frac{x^3}{3}+C}\\
	\\
	&=&x^3+C\\
	\end{array} $$
\end{solution}

\begin{example}{Indefinite Integral}{IndefiniteIntegral4}
Find the general antiderivative of $\ds \frac{2}{\sqrt x}\,dx$.
\end{example}
\begin{solution}
	$$\begin{array}{rcl}
	\ds{\int \frac{2}{\sqrt x}\,dx} & = & \ds{2\int x^{-\frac{1}{2}}\,dx}\\
	\\
	&=&\ds{2\frac{x^{-\frac{1}{2}+1}}{-\frac{1}{2}+1}+C}\\
	\\
	&=&4\sqrt x+C\\
	\end{array}$$
\end{solution}

\begin{example}{Indefinite Integral}{IndefiniteIntegral5}
Find:  $\ds\int \left(\frac{1}{x}+e^{7x}+x^\pi+7\right)\,dx$.
\end{example}
\begin{solution}
	$$\begin{array}{rcl}
	\ds{\int \left(\frac{1}{x}+e^{7x}+x^\pi+7\right)\,dx} & = & \ds{\int \frac{1}{x}\,dx+\int e^{7x}\,dx+\int x^\pi\,dx+\int 7\,dx}\\
	\\
	&=& \ds{\ln|x|+\frac{1}{7}e^{7x}+\frac{x^{\pi+1}}{\pi+1}+7x+C}\\
	\end{array}$$
\end{solution}

\subsection*{Differential Equations and Initial Value Problems}
An equation involving derivatives where we want to solve for the original function is called a \dfont{differential equation}.
For example, $f'(x)=2x$ is a differential equation with general solution $f(x)=x^2+C$.
Some solutions (i.e., particular values of $C$) are shown below.
$$\includegraphics[width=3in]{images2/integral-curves}$$
As seen with integral curves, we may have an infinite family of solutions satisfying the differential equation.
However, if we were given a point (called an \ifont{initial value}) on the curve then we could determine $f(x)$ completely.
Such a problem is known as an \ifont{initial value problem}.

\begin{example}{Initial Value Problem}{Initial Value Problem}
If $f'(x)=2x$ and $f(0)=2$ then determine $f(x)$.
\end{example}

\begin{solution}
As previously stated, we have a solution of:
$$f(x)=x^2+C.$$
But $f(0)=2$ implies:
$$2=0^2+C\quad\to\quad C=2.$$
Therefore, $f(x)=x^2+2$ is the solution to the initial value problem.
\end{solution}

We have seen that the derivative of a position function is a velocity function, and the derivative of a velocity function describes  acceleration.\index{initial value problem} We can now go ``the other way:'' the antiderivative of an acceleration function gives a velocity function, etc. While there is just one derivative of a given function, there are infinite antiderivatives. Therefore we cannot ask ``What is \textit{the} velocity of an object whose acceleration is $-32$ft/s$^2$?'', since there is more than one answer.

As above, we can find \textit{the} answer if we provide more information with the question, in the form of an  initial value, a value of the function that one knows beforehand.\\

\begin{example}{Solving initial value problems}{ex_anti4}
The acceleration due to gravity of a falling object is $-32$ ft/s$^2$. At time $t=3$, a falling object had a velocity of $-10$ ft/s. Find the equation of the object's velocity.
\end{example}

\begin{solution}
{We want to know a velocity function, $v(t)$. We know two things:
	\begin{itemize}
		\item		The acceleration, i.e., $v\primeskip'(t)= -32$, and
		\item		the velocity at a specific time, i.e., $v(3) = -10$.
	\end{itemize}
Using the first piece of information, we know that $v(t)$ is an antiderivative of $v\primeskip'(t)=-32$. So we begin by finding the indefinite integral of $-32$:
		$$\int (-32)\ dt = -32t+C=v(t).$$
Now we use the fact that $v(3)=-10$ to find $C$:
\begin{align*}
	v(t) &= -32t+C \\
	v(3) &= -10 \\
	-32(3)+C &= -10\\
	C &= 86
\end{align*}

Thus $v(t)= -32t+86$. We can use this equation to understand the motion of the object: when $t=0$, the object had a velocity of $v(0) = 86$ ft/s. Since the velocity is positive, the object was moving upward.

When did the object begin moving down? Immediately after $v(t) = 0$:
$$-32t+86 = 0 \quad \Rightarrow\quad  t = \frac{43}{16}  \approx 2.69\text{s}.$$
Recognize that we are able to determine quite a bit about the path of the object knowing just its acceleration and its velocity at a single point in time.
}
\end{solution}

\begin{example}{Solving initial value problems}{ex_anti5}
 {
Find $f(t)$, given that $\fpp(t) = \cos t$, $\fp(0) = 3$ and $f(0) = 5$.}
\end{example}

\begin{solution}
{We start by finding $\fp(t)$, which is an antiderivative of $\fpp(t)$:
		$$\int \fpp(t)\ dt = \int \cos t\ dt = \sin t + C = \fp(t).$$

		So $\fp(t) = \sin t+C$ for the correct value of $C$. We are given that $\fp(0) = 3$, so:
		$$\fp(0) = 3 \quad \Rightarrow \quad \sin 0+C = 3 \quad \Rightarrow \quad C=3.$$
		Using the initial value, we have found $\fp(t) = \sin t+ 3.$

We now find $f(t)$ by integrating again.

%\enlargethispage{3\baselineskip}
$$f(t)=\int \fp(t) \ dt = \int (\sin t+3)\ dt = -\cos t + 3t + C.$$
We are given that $f(0) = 5$, so
\begin{align*}
-\cos 0 + 3(0) + C &= 5 \\
-1 + C &= 5\\
C &= 6
\end{align*}
 Thus $f(t) = -\cos t + 3t + 6$.
}
\end{solution}

This section introduced antiderivatives and the indefinite integral. We found they are needed when finding a function given information about its derivative(s). For instance, we found a position function given a velocity function.

In the next section, we will see how position and velocity are unexpectedly related by the areas of certain regions on a graph of the velocity function. Then, in Section \ref{sec:FTC}, we will see how areas and antiderivatives are closely tied together.


% -*- LaTeX -*-
% -*- coding: utf-8 -*-
%
% michael a.g. aïvázis
% orthologue
% (c) 1998-2019 all rights reserved
%

\section{Monte Carlo integration: a bit of theory}
\label{sec:montecarlo}

Suppose you have a function $f(x)$ that is sufficiently well behaved for
all $x$ in a region $\Omega \subset \mathbb{R}^{n}$ and you wish to compute the
definite integral
%
\begin{equation}
  I_{\Omega} (f)
  =
  \int_{\Omega} f
\label{eq:integral}
\end{equation}
%
The Monte Carlo integration method estimates the value of a definite integral by sampling
the function at points $x$ in $\Omega$ that are chosen at random with uniform
probability. Suppose that you have $N$ such points forming a sample $X_{N}$. The Monte
Carlo estimate for the integral is
%
\begin{equation}
  I_{\Omega} (f; X_{N})
  =
  \Omega \cdot \langle f \rangle
  =
  \Omega \, \frac{1}{N} \sum_{x \in X_{N}} f(x)
\label{eq:mc-estimate}
\end{equation}
%
where $\langle f \rangle$ is the sample mean of the function $f$ and we have used $\Omega$
as short-hand for the volume of the associated region. More details can be found in
\citep{hammersley,ueberhuber}; see \cite{weinzierl} for an excellent pedagogical
introduction to the subject.

One can show that the error in the estimate descreases as $1/\sqrt{N}$, a convergence rate that
is rather slow. It is possible to improve on this by being smart about how the region of
integration is sampled; see \cite{lepage-78,lepage-80,press}. On the bright side, the
convergence rate is {\em independent} of the dimensionality of the integral, making this method
very well suited for multi-dimensional integrals. Further, it is rather straightforward to
write a parallel implementation and compensate for the slow convergence by computing on a large
machine.

We are ready to recast \eqref{mc-estimate} in a form better suited for a computer
implementation, where we use pseudo-random number generators to create the sample set. Most
generators produce numbers between $0$ and $1$, so actual calculations require finding a box
$B_{\Omega}$ that contains $\Omega$, and building $n$-dimensional sampling points $x$ by
stretching and translating the unit interval to match the dimensions of $B_{\Omega}$. The
integration is then restricted to $\Omega$ by introducing a function
%
\begin{equation}
  \Theta_{\Omega}(x)
  =
  \left\{
  \begin{array}{ll}
  1 & x \in \Omega \\
  0 & {\rm otherwise}
  \end{array}
  \right.
\label{eq:theta}
\end{equation}
%
that takes the value $1$ inside $\Omega$ and vanishes identically outside. We can now recast
\eqref{integral} as
%
\begin{equation}
  I_{\Omega} (f)
  =
  \int_{B_{\Omega}} \Theta_{\Omega} \, f
\label{eq:integral-box}
\end{equation}
%
There are now two classes of points in the sample $X_{N}$: those interior to $\Omega$, and the
rest. Let $\tilde{N}$ be the number of points in $X_{N}$ that fall in $\Omega$; then
\eqref{mc-estimate} becomes:
%
\begin{equation}
  I_{\Omega} (f; X_{N})
  =
  \Omega \, \frac{1}{\tilde{N}} \sum_{x \in X_{\tilde{N}}} f(x)
\label{eq:mc-box}
\end{equation}
%
where $X_{\tilde{N}}$ is the subset of the sample in $\Omega$. Now, let $B$ be the volume
of the sampling box and observe that
%
\begin{equation}
  \Omega
  =
  \frac{\tilde{N}}{N} B
\label{eq:volume-estimate}
\end{equation}
%
is a good estimate of the volume of the integration region. Further, the sum over
$X_{\tilde{N}}$ in \eqref{mc-box} can be extended to $X_{N}$ as long as $f$ is
multiplied by $\Theta_{\Omega}$. We obtain
%
\begin{equation}
  I_{\Omega} (f; X_{N})
  =
  B \, \frac{1}{N} \sum_{x \in X_{N}} \Theta_{\Omega} \, f(x)
\label{eq:mc}
\end{equation}
%

In the remainder of this article, we will transform this rather innocuous expression into
a sequence of computer programs of increasing complexity and, hopefully, flexibility. The
goal is to construct a piece of software that will enable our end users to explore the
method with as little programming on their part as possible.

% end of file


\chapter{Dimensional analysis}

\section{Non-dimensionalisation}
Consider system (\ref{boussinesq}) and rescale spatial lengths with the transformation
\begin{equation}
\tilde{x}=\frac{x}{L},\quad \tilde{y}=\frac{y}{L},\quad \tilde{z}=\frac{z}{H},
\end{equation}
where $\tilde{x},\tilde{y},\tilde{z}$ are nondimensional numbers and $L$ and $H$
are scaling values in the horizontal and vertical respectively. These generally take
typical values for length scales so that $\tilde{x},\tilde{y},\tilde{z}$ are ${\cal O}(1)$.

The momentum equations in (\ref{boussinesq}) then become (for simplicity dropping $\bmF$, $\hat{\bmu}$
and assuming that $2 \bmOmega \times \bmu=f\bmk\times\bmu$ and $\nabla\cdot \tautens = \nu\nabla^2\bmu$),
\begin{subeqnarray*}
\frac{\pp u}{\pp t}
+ L^{-1}u\frac{\pp u}{\pp\tilde{x}} + L^{-1}v\frac{\pp u}{\pp\tilde{y}} + H^{-1}w\frac{\pp u}{\pp\tilde{z}}
-fv
&=& -L^{-1}\frac{\pp p}{\pp\tilde{x}} - L^{-1}g\frac{\pp \eta}{\pp\tilde{x}}\\&&\quad
+L^{-2}\nu\frac{\pp^2 u}{\pp\tilde{x}^2}+L^{-2}\nu\frac{\pp^2 u}{\pp\tilde{y}^2}+H^{-2}\nu\frac{\pp^2 u}{\pp\tilde{z}^2},\\
\frac{\pp v}{\pp t}
+ L^{-1}u\frac{\pp v}{\pp\tilde{x}} + L^{-1}v\frac{\pp v}{\pp\tilde{y}} + H^{-1}w\frac{\pp v}{\pp\tilde{z}}
+fu
&=& -L^{-1}\frac{\pp p}{\pp\tilde{y}} - L^{-1}g\frac{\pp \eta}{\pp\tilde{y}}\\&&\quad
+L^{-2}\nu\frac{\pp^2 v}{\pp\tilde{x}^2}+L^{-2}\nu\frac{\pp^2 v}{\pp\tilde{y}^2}+H^{-2}\nu\frac{\pp^2 v}{\pp\tilde{z}^2},\\
\frac{\pp w}{\pp t}
+ L^{-1}u\frac{\pp w}{\pp\tilde{x}} + L^{-1}v\frac{\pp w}{\pp\tilde{y}} + H^{-1}w\frac{\pp w}{\pp\tilde{z}}
&=& -H^{-1}\frac{\pp p}{\pp\tilde{z}} - \rho g\\&&\quad
+L^{-2}\nu\frac{\pp^2 w}{\pp\tilde{x}^2}+L^{-2}\nu\frac{\pp^2 w}{\pp\tilde{y}^2}+H^{-2}\nu\frac{\pp^2 w}{\pp\tilde{z}^2}.
\end{subeqnarray*}

Now suppose that we also have representative velocity scales in the horizontal and vertical ($U,W$) and
define
\begin{equation}
\tilde{u}=\frac{u}{U},\quad \tilde{v}=\frac{v}{U},\quad \tilde{w}=\frac{w}{W},
\end{equation}
then substituting we have
\begin{subeqnarray*}
U\frac{\pp \tilde{u}}{\pp t}
+ U^2L^{-1}\left(\tilde{u}\frac{\pp \tilde{u}}{\pp\tilde{x}} + \tilde{v}\frac{\pp \tilde{u}}{\pp\tilde{y}}\right) + UWH^{-1}\tilde{w}\frac{\pp \tilde{u}}{\pp\tilde{z}}
-Uf\tilde{v}
&=& -L^{-1}\frac{\pp p}{\pp\tilde{x}} - L^{-1}g\frac{\pp \eta}{\pp\tilde{x}}\\&&\quad
+UL^{-2}\nu\left(\frac{\pp^2 \tilde{u}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{u}}{\pp\tilde{y}^2}\right)+H^{-2}U\nu \frac{\pp^2 \tilde{u}}{\pp\tilde{z}^2},\\
U\frac{\pp \tilde{v}}{\pp t}
+ U^2L^{-1}\left(\tilde{u}\frac{\pp \tilde{v}}{\pp\tilde{x}} + \tilde{v}\frac{\pp \tilde{v}}{\pp\tilde{y}}\right) + UWH^{-1}\tilde{w}\frac{\pp \tilde{v}}{\pp\tilde{z}}
+Uf\tilde{u}
&=& -L^{-1}\frac{\pp p}{\pp\tilde{y}} - L^{-1}g\frac{\pp \eta}{\pp\tilde{y}}\\&&\quad
+UL^{-2}\nu\left(\frac{\pp^2 \tilde{v}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{v}}{\pp\tilde{y}^2}\right)+H^{-2}U\nu\frac{\pp^2 \tilde{v}}{\pp\tilde{z}^2},\\
W\frac{\pp \tilde{w}}{\pp t}
+ UWL^{-1}\left(\tilde{u}\frac{\pp \tilde{w}}{\pp\tilde{x}} + \tilde{v}\frac{\pp \tilde{w}}{\pp\tilde{y}}\right) + W^2H^{-1}\tilde{w}\frac{\pp \tilde{w}}{\pp\tilde{z}}
&=& -L^{-1}\frac{\pp p}{\pp\tilde{z}} -  \rho g\\&&\quad
+WL^{-2}\nu\left(\frac{\pp^2 \tilde{w}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{w}}{\pp\tilde{y}^2}\right)+H^{-2}W\nu\frac{\pp^2 \tilde{w}}{\pp\tilde{z}^2}.
\end{subeqnarray*}

Once representative length and velocity scales have been defined this automatically sets a temporal scale
\begin{equation*}
\tilde{t}=\frac{t}{T},\quad T=\frac{L}{U},
\end{equation*}
and the divergence-free continuity equation $\nabla\cdot\vec{u}=0$
suggests the scaling constraint
\begin{equation}
\frac{W}{H}=\frac{U}{L}.
\label{aspect99}
\end{equation}

Again, substituting and dividing through by $UT^{-1}$ in the horizontal and $WT^{-1}$
in the vertical, and making use of (\ref{aspect99}) yields
\begin{subeqnarray*}
\frac{\pp \tilde{u}}{\pp \tilde{t}}
+ \tilde{u}\frac{\pp \tilde{u}}{\pp\tilde{x}} + \tilde{v}\frac{\pp \tilde{u}}{\pp\tilde{y}} + \tilde{w}\frac{\pp \tilde{u}}{\pp\tilde{z}}
-\frac{Lf}{U}\tilde{v}
&=& -U^{-2}\frac{\pp p}{\pp\tilde{x}} - U^{-2}g\frac{\pp \eta}{\pp\tilde{x}}\\&&\quad
+U^{-1}L^{-1}\nu\left(\frac{\pp^2 \tilde{u}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{u}}{\pp\tilde{y}^2}\right)+H^{-2}LU^{-1}\nu \frac{\pp^2 \tilde{u}}{\pp\tilde{z}^2},\\
\frac{\pp \tilde{v}}{\pp \tilde{t}}
+ \tilde{u}\frac{\pp \tilde{v}}{\pp\tilde{x}} + \tilde{v}\frac{\pp \tilde{v}}{\pp\tilde{y}} + \tilde{w}\frac{\pp \tilde{v}}{\pp\tilde{z}}
+\frac{Lf}{U}\tilde{u}
&=& -U^{-2}\frac{\pp p}{\pp\tilde{y}} - U^{-2}g\frac{\pp \eta}{\pp\tilde{y}}\\&&\quad
+U^{-1}L^{-1}\nu\left(\frac{\pp^2 \tilde{v}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{v}}{\pp\tilde{y}^2}\right)+H^{-2}LU^{-1}\nu \frac{\pp^2 \tilde{v}}{\pp\tilde{z}^2},\\
\frac{\pp \tilde{w}}{\pp \tilde{t}}
+ \tilde{u}\frac{\pp \tilde{w}}{\pp\tilde{x}} + \tilde{v}\frac{\pp \tilde{w}}{\pp\tilde{y}} + \tilde{w}\frac{\pp \tilde{w}}{\pp\tilde{z}}
&=& -LH^{-1}U^{-1}W^{-1}\frac{\pp p}{\pp\tilde{z}} - LU^{-1}W^{-1}\rho g\\&&\quad
+U^{-1}L^{-1}\nu\left(\frac{\pp^2 \tilde{v}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{v}}{\pp\tilde{y}^2}\right)+H^{-2}LU^{-1}\nu \frac{\pp^2 \tilde{v}}{\pp\tilde{z}^2}.
\end{subeqnarray*}

The Coriolis term takes the form
\begin{equation*}
\frac{Lf}{U}\bmk\times\tilde{\bmu}=\Ro^{-1}\bmk\times\tilde{\bmu}.
\end{equation*}
The buoyancy term takes the form
\begin{equation*}
\frac{L}{UW}\rho g = \frac{L}{H}\frac{U}{W}\Fr^{-1}\rho = \frac{L^2}{H^2}\Fr^{-1}\rho = \Delta^2\Fr^{-1}\rho.
\end{equation*}
The viscous term takes the form
\begin{equation*}
U^{-1}L^{-1}\nu\left(\frac{\pp^2 \tilde{u}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{u}}{\pp\tilde{y}^2}\right)+H^{-2}LU^{-1}\nu \frac{\pp^2 \tilde{u}}{\pp\tilde{z}^2}
=\Re_H^{-1}\left(\frac{\pp^2 \tilde{u}}{\pp\tilde{x}^2}+\frac{\pp^2 \tilde{u}}{\pp\tilde{y}^2}\right) + \Re_V^{-1}\frac{\pp^2 \tilde{u}}{\pp\tilde{z}^2},
\end{equation*}
where $\Re_H=UL/\nu$ and $\Re_V=WH/\nu$.
%Note that is would have been
%trivial in the above to use a tensor of viscosity values rather than a
%constant.  -- COLIN: not sure what this means
Rescaling the free surface height in terms of the vertical
coordinate ($\tilde{\eta}=\eta/H$) yields a free surface term
\begin{equation*}
\frac{Hg}{U^2}\nabla_{\tilde{x}}\tilde{\eta}=\Fr^{-1}\nabla_{\tilde{x}}\tilde{\eta}.
\end{equation*}

Dropping tildes and using $p=U^{-2}p$ we have finally that
\begin{subeqnarray*}
\frac{\pp {u}}{\pp {t}}
+ \bmu\cdot\nabla u
-\Ro^{-1}{v}
&=& \frac{\pp p}{\pp {x}} - \Fr^{-1}\frac{\pp \eta}{\pp {x}}
+\Re_H^{-1}\left(\frac{\pp^2 {u}}{\pp {x}^2}+\frac{\pp^2 {u}}{\pp {y}^2}\right)+ \Re_V^{-1}\frac{\pp^2 {u}}{\pp {z}^2},\\
\frac{\pp {v}}{\pp {t}}
+ \bmu\cdot\nabla v
+\Ro^{-1}{u}
&=& \frac{\pp p}{\pp {y}} - \Fr^{-1}\frac{\pp \eta}{\pp {y}}
+\Re_H^{-1}\left(\frac{\pp^2 {v}}{\pp {x}^2}+\frac{\pp^2 {v}}{\pp {y}^2}\right)+ \Re_V^{-1}\frac{\pp^2 {v}}{\pp {z}^2},\\
\frac{\pp {w}}{\pp \tilde{t}}
+ \bmu\cdot\nabla w
&=& -\Delta^2 \frac{\pp p}{\pp {z}} - \Delta^2 \Fr^{-1}g
+\Re_H^{-1}\left(\frac{\pp^2 {v}}{\pp {x}^2}+\frac{\pp^2  {v}}{\pp {y}^2}\right)+ \Re_V^{-1}\frac{\pp^2 {v}}{\pp {z}^2}.
\end{subeqnarray*}

%\begin{thm}[Buckingham-Pi Theorem]
%Why nondimensionalising can help reduce parameter phase space ... Barenblatt ...
%\end{thm}

\section{Dimensionless parameters}
Table \ref{table:dimensionless} gives some common non-dimensional numbers.
The following expressions are also useful
\begin{equation*}
N^2=-\frac{g}{\rho_0}\ddt[z]{\rho} \approx \frac{g}{\rho_0}\frac{\Delta\rho}{H}=\frac{g'}{H}
\end{equation*}
\begin{equation*}
g' =\frac{\Delta\rho}{\rho_0}g
\end{equation*}

\begin{table}[t]
\begin{center}
\begin{tabular}{lll} \hline
Name        &   Description         &   Form        \\  \hline
            &                           &               \\
Aspect ratio   &   Ratio of length scales   &   $\Delta=L/H$  \\
Reynolds    &   inertial/viscous        &   $\Re=UL/\nu$    \\
Rossby      &   inertial/Coriolis\footnotemark[6]            &   $\Ro=U/fL$      \\
Froude      &   inertial/gravitational\footnotemark[7] &   $\Fr=U^2/gH$    \\
Froude (internal)   &   inertial/buoyancy   &   $\Fr_{\mathrm{internal}}=U^2/g'H=U/NH$   \\
Burger      &  $\Bu=\Ro/\Fr$    &   $\Bu = NH/fL$  \\
Richardson  &   buoyancy/inertial\footnotemark[8]       &   $\Ri=N^2/(U/L)^2$   \\
Richardson (gradient)   &   buoyancy/shear  &   $\Ri_{\mathrm{gradient}}=N^2/(\Delta U/\Delta L)^2$ \\
Stokes      &   viscous/gravitational   &   \\
Prandtl     &   viscosity/diffusivity   &  $\nu/\kappa$ \\
P\'ecl\'et  &     $\Pe=\Re\,\Pr$  &  $UL/\kappa$ \\
Ekman       &   viscous/Coriolis    &   $\Ek=2\nu_H/fL^2$   \\
Grashof     &    free convective/viscous    & $\Gr=g\Delta\rho L^3/\rho \nu^2$\\
Rayleigh    &     $Ra=\Gr\,\Pr$                 & $\Ra = g\beta(T-T_0)L^3/\alpha\nu$\\
            &                           &               \\  \hline
\label{table:dimensionless}
\end{tabular}
\end{center}

\caption{Summary of useful dimensionless numbers}
\label{tab:dimensionless}
\end{table}

\footnotetext[6]{Also ratio of rotational and convective timescales.}
\footnotetext[7]{This sometimes termed the first Froude number and is also often defined as the square root of this (the second Froude number).}
\footnotetext[8]{also $(\textrm{internal Froude number})^{-2}$ if $H=L$}


\subsection{Permutations}

Function to make row exchanges. Elimination with row exchanges:

\[A=LU\implies PA=LU\]

Works for any invertible $A$.

\[P=\text{identity with reordered rows (exchanges)}\]

Count of possible reorderings ($n\times n$ permutations): $n! = n(n-1)\cdots 3(2)(1)$.

\[\boxed{P^{-1}=P^T\text{ and }P^TP=I}\]

Defining a transpose, or flip over diagonal:

\[(A^T)_{ij}=A_{ji}\]

For symmetric matrices, transpose does not cause change; $A^T=A$. If two rectangular matrices $R^T$ and $R$
give a square matrix, then $R^TR$ is always symmetric.

\[\boxed{(R^TR)^T=R^TR^{TT}=R^TR}\]

\subsection{Vector Spaces and Subspaces}

Examples: $\R^2$ is all vectors in 2D space, $x-y$ plane: $\begin{bmatrix}3\\2\end{bmatrix}$.
$\R^3$ is all vectors with 3 components. All combinations of vectors in $\R^n$ yield a result in that space $\R^n$.

\[\boxed{\R^n\text{ is all column vectors with $n$ components.}}\]

The origin exists to allow for scalar multiplication and addition of vectors. Every vector space has a $\tb{0}$.

\subsubsection{Subspaces}

If a vector space is defined as 1st quadrant in $\R^2$, then multiplying by a negative scalar $k$
removes the result from that space, so it is not \textbf{closed} under that operation, so this is not a vector space.
Vector space must be closed under linear combinations. Thus, subspace in $\R^2$ is all multiples of that vector, a line and the line must go through $\tb{0}$.
Every subspace must contain $\tb{0}$.\newline

Subspaces of $\R^2$:
\begin{itemize}
    \item All of $\R^2$
    \item Any line through $\tb{0}_2$ or $L$
    \item Just $\tb{0}_2$ or $Z$
\end{itemize}

Similarly, for $\R^3$ can have $\R^3$, plane, line, $\tb{0}_3$.\newline

Given $A=\left[\begin{array}{ll}
    1 & 3 \\
    2 & 3 \\
    4 & 1\end{array}\right]$, all linear combinations of these columns form a subspace.
This is called \textbf{column space}, $C(A)$. This would form a plane in $\R^3$. Thus, the column space is a subspace.


\subsection{Introduction}
\begin{definition}
	A first order linear differential equation is given by $x' = a\p{t}x + f\p{t}$. If $f\p{t} = 0$, then the equation is said to be \textit{homogeneous}. Otherwise, it is said to be \textit{inhomogeneous}.
\end{definition}
\begin{minipage}[t]{0.5\textwidth}
	\textit{Examples:}
	\vspace{-\baselineskip}
	\begin{addmargin}{0.25in}
		\noskip
		\begin{flalign*}
			& x' = tx + t^2 & \\
			& x' = \p{\frac{\sin{t}}{t} + e^t}x + e^{t^2}
		\end{flalign*}
	\end{addmargin}
\end{minipage}%
\begin{minipage}[t]{0.5\textwidth}
	\textit{Non-examples:}
	\vspace{-\baselineskip}
	\begin{addmargin}{0.25in}
		\noskip
		\begin{flalign*}
		 	& x' = x^2 & \\
			& xx' = \sin{t} \\
			& x' = \sin{x}
		\end{flalign*}
	\end{addmargin}
\end{minipage}%

\subsection{Solutions to Homogeneous Linear Equations}
\normalskip
Homogeneous first order linear differential equations are of the form $x' = a\p{t}x$. Notice that it is separable, and that the general solution is given by \[ x\p{t} = Ce^{\int a\p{t} \, \diff{t}}, \text{ where } C \in \mathbb{R}. \]

\subsection{Solutions to Inhomogeneous Equations}
\subsubsection{Integrating Factors}
Consider the simple inhomogeneous equation \[ x' + x = A. \] The appearance of the $+$ sign and a derivative reminds us of the product rule. As the equation stands, we can't write it as a derivative of a product. However, if we multiply it by some function, we will be able to. The integrating factor for this particular differential equation is $e^x$. Notice that $\p{e^x x}' = e^x x' + e^x x$. Then
\begin{align*}
	e^x x' + e^x x &= Ae^x \\
	\p{e^x x}' &= Ae^x
\end{align*}
which is a separable equation in terms of the function $e^x x$. Using the substitution $u = e^x x$ will make this clear:
\begin{align*}
	u' &= Ae^x \\
	\int \diff{u} &= \int Ae^x \, \diff{x} \\
	u &= Ae^x + C \\
	e^x x &= Ae^x + C
\end{align*}
We now generalize this method to any first order linear ODE.
\par\bigskip
Consider a linear equation written in the form $x' + a\p{t}x = f\p{t}$. Our goal is to find some function $\mu\p{t}$ so that after multiplying both sides of the equation, the LHS can be written as the derivative of a product.
\[ \mu\p{t} x' + \mu\p{t} a\p{t}x = g'\p{t} \]
By inspection, it is easy to see that the LHS is the derivative of a function times the function $x$. Taking $g\p{t} = h\p{t}x$, we have
\begin{align*}
	\p{h\p{t}x}' &= \mu\p{t} x' + \mu\p{t} a\p{t}x \\
	h\p{t}x' + h'\p{t}x &= \mu\p{t} x' + \mu\p{t} a\p{t}x \\
	\implies h\p{t} = \mu\p{t}, \, h'\p{t} &= \mu'\p{t} = \mu\p{t} a\p{t} \\
	\mu\p{t} &= Ce^{\int a\p{t} \, \diff{t}}
\end{align*}
Notice that the value of $C$ does not matter, as long as it is non-zero. We take $\mu\p{t} = e^{\int a\p{t} \, \diff{t}}$, which yields
\begin{align*}
	\mu\p{t} x' + \mu\p{t} a\p{t}x &= \mu\p{t} f\p{t} \\
	\p{\mu\p{t} x}' &= \mu\p{t} f\p{t} \\
	\mu\p{t} x &= \int \mu\p{t} f\p{t} \, \diff{t} \\
	x &= \frac{1}{\mu\p{t}} \int \mu\p{t} f\p{t} \, \diff{t} \\
	x\p{t} &= e^{-\int a\p{t} \, \diff{t}} \int f\p{t} e^{\int a\p{t} \, \diff{t}} \, \diff{t}
\end{align*}
This formula is quite complicated and not worth memorizing. It is easier to find $\mu\p{t}$ and then work from there.
\par\bigskip
\begin{example}
	\noskip
	\begin{align*}
		x' + x\tan{t} &= 2t\cos{t} \\
		\mu = e^{\int \tan{t} \, \diff{t}} &= e^{\ln\abs{\sec{t}}} = \sec{t} \\
		x'\sec{t} + x\sec{t}\tan{t} &= 2t \\
		\p{x\sec{t}}' &= 2t \\
		x\sec{t} &= t^2 + C \\
		x &= t^2\cos{t} + C\cos{t}
	\end{align*}
\end{example}

\subsubsection{Variation of Parameters}
\normalskip
Consider the linear equation \[ y' = 3y + 6. \] Now, consider the associated homogeneous equation: \[ y'_h = 3y_h \implies y_h = Ae^{3t} \] The solution to the associated homogeneous equation is related to the solution of our original equation somehow. Once again, the $+$ sign is suggestive of the product rule, so we will guess that the solution is the product of $y_h\p{t}$ and another function $v\p{t}$. Let $y\p{t} = v\p{t}y_h\p{t}$. Then
\begin{align*}
	y' = v'y_h + vy'_h &= 3vy_h + 6 \\
	v' \cdot Ae^{3t} + \cancel{v \cdot 3Ae^{3t}} &= \cancel{3v \cdot Ae^{3t}} + 6 \\
	v' &= \frac{6}{A}e^{-3t} \\
	v &= -\frac{2}{A}e^{-3t} + C
\end{align*}
Thus, the general solution to the ODE is given by \[ y\p{t} = -2 + Ce^{3t}. \]
\par
We will now generalize this method to any first order linear ODE. Consider the equation \\ $y' = a\p{t}y + f\p{t}$ and its associated homogeneous equation: \[ y'_h = a\p{t}y_h \implies y_h = Ae^{\int a\p{t} \, \diff{t}} \]
Again, we assume that the solution to our ODE will be of the form $y\p{t} = v\p{t}y_h\p{t}$. Then
\begin{align*}
	y' = v'y_h + vy'_h &= a\p{t}vy_h + f\p{t} \\
	v' \cdot Ae^{\int a\p{t} \, \diff{t}} + \cancel{v \cdot Aa\p{t}e^{\int a\p{t} \, \diff{t}}} &= \cancel{v \cdot Aa\p{t}e^{\int a\p{t} \, \diff{t}}} + f\p{t} \\
	v' &= \frac{1}{A}f\p{t}e^{-\int a\p{t} \, \diff{t}} \\
	v &= \frac{1}{A} \int f\p{t}e^{-\int a\p{t} \, \diff{t}} \, \diff{t}
\end{align*}
which gives us the general solution \[ y\p{t} = e^{\int a\p{t} \, \diff{t}} \int f\p{t}e^{-\int a\p{t} \, \diff{t}} \, \diff{t}. \] Note that the solution from here and from using integrating factors differ by a negative sign on $a\p{t}$. This is simply because of the general form of each equations we started with: in the section of integrating factors we had $y' + a\p{t}y = f\p{t}$ whereas in this section, we used the form $y' = a\p{t}y + f\p{t}$. We can simply rewrite the second equation as $y' + \p{-a\p{t}}y = f\p{t}$, which eliminates the discrepancy.
\subsubsection{Structure of Solutions}
\begin{theorem}
	Let $y' - ay = f$ be a general first order linear differential equation. Suppose that $y_p\p{t}$ is a particular solution to the equation, and $y_h\p{t}$ is a particular solution to the associated homogeneous differential equation $y' - ay = 0$. Then every solution of $y' - ay = f$ is of the form \[ y\p{t} = y_p\p{t} + Cy_h\p{t}. \]
\end{theorem}
\begin{proof}
	Consider the differential equation $y' - ay = f$, and the function $y\p{t} = y_p\p{t} + Cy_h\p{t}$ with $y_p$ and $y_h$ be as described above. Then
	\begin{align*}
		y'\p{t} - ay\p{t} &= \p{y_p + Cy_h}' - a\p{y_p + Cy_h} \\
								  &= y_p' + Cy_h' - ay_p - Cay_h \\
								  &= \p{y_p' - ay_p} + C\p{y_h' - ay_h} \\
								  &= f + C \cdot 0 \\
		y'\p{t} - ay\p{t} &= f
	\end{align*}
	Thus, $y\p{t} = y_p\p{t} + Cy_h\p{t}$ satisfies the differential equation.
\end{proof}
\subsection{Mixing Problems}
\textit{Example:} \\
\begin{minipage}[t]{0.25\textwidth}
	\begin{tikzpicture}[baseline = (current bounding box.north)]
		\draw[->] (1, -0.25) -- (1, -0.75) node[pos = 0, above] {\begin{tabular}{c} 5 gal/min \\ 1 lb/gallon of salt \end{tabular}};
		\fill[fill=blue!20!white] (0, -1) rectangle (2, -2);
		\draw (0, -0.5) -- (0, -2) -- (2, -2) -- (2, -0.5);
		\node at (1, -1.5) {\begin{tabular}{c} 100~gal \\ Pure water \end{tabular}};
		\draw[->] (1, -2.25) -- (1, -2.75) node[pos = 1, below] {2 gal/min};
	\end{tikzpicture}
\end{minipage}%
\begin{minipage}[t]{0.75\textwidth}
	\begin{center}
		(We assume the liquid in the tank is homogeneous at all times.)
	\end{center}
	We want to model the amount of salt in the container at any time. Let $x\p{t}$ be the number of pounds of salt at time $t$. While we can't model $x\p{t}$ directly, we can easily model its rate of change. To do so, we have to look at how much salt flows in and out over time. The rate of salt flow is simply the rate at which the solution flows in or out multiplied by the concentration of salt. In other words, it's the amount of salt that comes with the solution per unit time. Thus, we can model the rate of change of $x\p{t}$ as follows:
\end{minipage}
\begin{align*}
		\frac{\diff{x}}{\diff{t}} &= \text{(inflow of salt)} - \text{(outflow of salt)} \\
		&= \p{5~\frac{\text{gal}}{\text{min}} \cdot 1~\frac{\text{lb}}{\text{gal}}} - \p{2~\frac{\text{gal}}{\text{min}} \cdot \frac{x}{100 + \p{5 - 3}t}~\frac{\text{lb}}{\text{gal}}} \\
		&= 5~\frac{\text{lb}}{\text{min}} - \frac{2x}{100 + 2t}~\frac{\text{lb}}{\text{min}} \\
		&= 5 - \frac{x}{50 + t}
\end{align*}
Notice that the outflow of salt is a function of $x$ and $t$, which is because the amount of salt isn't constant and the amount of solution in the tank changes over time (at a constant rate in this problem).
\par\bigskip
The differential equation is linear with the initial condition $x\p{0} = 0$, meaning we can solve it using any of the techniques discussed in this chapter. We will use variation of parameters (Section 3.3.2).
\begin{align*}
	\frac{\diff{x_h}}{\diff{t}} &= -\frac{x_h}{50 + t} \\
	\int\frac{\diff{x_h}}{x_h} &= -\int\frac{\diff{t}}{50 + t} \\
	\ln{\abs{x_h}} &= -\ln{\abs{50 + t}} \\
	x_h &= \frac{1}{50 + t} \qquad \text{(both quantities are positive)}
\end{align*}
Our solution will be in the form $x = vx_h$, so substituting into the differential equation yields
\begin{align*}
	v'x_h + vx_h' &= 5 - \frac{vx_h}{50 + t} \\
	\frac{v'}{50 + t} - \cancel{\frac{v}{\p{50 + t}^2}} &= 5 - \cancel{\frac{v}{\p{50 + t}^2}} \\
	\frac{\diff{v}}{\diff{t}} &= 5\p{50 + t} \\
	v &= 250t + \frac{5}{2}t^2 + C
\end{align*}
Thus, the general solution to the mixing problem is given by \[ x\p{t} = \frac{250t + \frac{5}{2}t^{2} + C}{50 + t}. \] With the initial condition $x\p{0} = 0$, we get \[ \frac{C}{50} = 0 \implies C = 0, \] so the solution to the mixing problem is \[ x\p{t} = \frac{250t + \frac{5}{2}t^2}{50 + t} = \frac{500t + t^2}{100 + 2t}. \] A common variation to this problem is the addition to one or more tanks. In those problems, the outflow of one tank may be the inflow of another, but the overall setup will be the same. The main difference is that there will be more differential equations to solve, but they will all be linear.