\section{Higher Order Linear ODEs}
\subsection{Second Order Linear ODEs with Constant Coefficients}
\begin{definition}
    A second order linear ODE with constant coefficient is an ODE of the form
    $$a\frac{\mathrm d^2y}{\mathrm dx^2}+b\frac{\mathrm dy}{\mathrm dx}+cy=f(x)$$
    where $a,b,c$ are constants with $a\neq 0$.
\end{definition}
\begin{definition}
    A linear differential operator $\mathscr{D}$ is a linear combination of (different orders of) differentiation operators.
\end{definition}
We know that the (arbitrary order) differentiation operator is linear, hence any linear differential operator is linear, which gives rise to the principle of superposition.
\begin{proposition}
    If $y_\alpha, y_\beta$ are solutions to $ay^{\prime\prime}+by^\prime+cy=f$, then $y_{\alpha}-y_\beta$ is a solution to $ay^{\prime\prime}+by^\prime+cy=0$.
\end{proposition}
\begin{proof}
    Consider the linear operator
    $$\mathscr{D}=a\frac{\mathrm d^2}{\mathrm dx^2}+b\frac{\mathrm d}{\mathrm dx}+c$$
    then $\mathscr{D}(y_\alpha-y_\beta)=\mathscr{D}(y_\alpha)-\mathscr{D}(y_\beta)=f(x)-f(x)=0$.
\end{proof}
We can extend the above proposition to any order of linear ODEs with constant coefficient in the obvious way.\\
So we can solve these solutions in the following steps, assuming $\mathscr{D}$ is defined as in the above proof:\\
First, we find complementary (linearly independent, defined below) solutions $y_1,y_2$ to $\mathscr{D}(y)=0$.\\
Then we find a particular solution $y_p$ to $\mathscr{D}(y)=f(x)$.\\
The general form of the solutions to $\mathscr{D}(y)=f(x)$ is $y_p+Ay_1+By_2$ where $A,B$ are constants.
\begin{definition}
    A set of functions $(f_i)_{i\in I}$ are linearly dependent if $\sum_ic_if_i(x)=0$ for some constants $c_i$ that are not all zero.
    The sum here is taken over some finite set of indices.\\
    Otherwise, they are linearly independent.
\end{definition}
Equivalently, if a function in the set of functions can be written as a linear combination of others, then the set is linearly dependent.\\
Consider a second-order linear differential operator.
We know that the a first order one has eigenfunction to be the exponential function.
Note that it is also the eigenfunction of a second-order one.
In fact, the exponential is the eigenfunction of any linear differential operator.
Consider the homogeneous equation $\mathscr{D}y=0$ where
$$\mathscr{D}=a\frac{\mathrm d^2}{\mathrm dx^2}+b\frac{\mathrm d}{\mathrm dx}+c$$
Plug in $y=e^{\lambda x}$ we have $a\lambda^2+b\lambda+c=0$ which we call \textit{characteristic equation} or \textit{auxiliary equation}.
From FTA, we have at least $1$ (complex) solutions.
Let $\lambda_1,\lambda_2$ be two solutions, then\\
Case 1: $\lambda_1\neq\lambda_2$.\\
Then $y_1=Ae^{\lambda_1x}$, $y_2=Be^{\lambda_2x}$ are both solutions for each $A,B$ constants.
It is easy to see that these two are linearly independent for $AB\neq 0$.
We can show that they form a basis for solution space,
\footnote{This will be discussed later.}
and any other solutions must be of the form $Ae^{\lambda_1x}+Be^{\lambda_2x}$ for $A,B$ constants.\\
Case 2: $\lambda_1=\lambda_2$.\\
In this case, $y_1,y_2$ cannot span the solution space, but there is some workaround.
\begin{example}
    $y^{\prime\prime}-4y^\prime+4y=0$, then $(y^\prime-2y)^\prime-2(y^\prime-2y)=0$ can give the solution.\\
    Or we can consider the slightly modifies equation $y^{\prime\prime}-4y^\prime+(4-\epsilon^2)y=0$ for some small enough $\epsilon$.
    This modified equation has the general solution $y_\epsilon=Ae^{\lambda_1x}+Be^{\lambda_2x}$ where $\lambda_{1,2}=2\pm\epsilon$, so
    $$y_\epsilon=e^{2x}(Ae^{\epsilon x}+Be^{-\epsilon x})=e^{2x}((A+B)+\epsilon x(A-B)+O(\epsilon^2))\to Cxe^{2x}+De^{2x}$$
    for some constants $C,D$ by clever (or not) choices of $A,B$.
    This gives a pair of linearly independent solutions.
\end{example}
\subsection{Second Order Linear ODEs with Non-constant Coefficients}
Again we are interested in the homogeneous ones due to the superposition principle.
Consider the equations in the form
$$y^{\prime\prime}+p(x)y^\prime+q(x)y=0$$
We shall use the method of reduction of order.
Given one solution $y_1$ to the equation, we shall find a second solution by looking for solutions of the form $y_2=vy_1$.\\
First, note that $y_2^\prime=v^\prime y_1+vy_1^\prime,y_2^{\prime\prime}=v^{\prime\prime}y_1+2v^\prime y_1^\prime +vy^{\prime\prime}$, so plugging it in we have
$$v^\prime(2y_1^\prime+py_1)+v^{\prime\prime}y_1=0$$
So it is a seperable first order equation in $v^\prime$ which we know how to solve, plugging it back gives the solution.
\subsection{Phase Space}
Consider the ODE $p_i(x)y^{(i)}=f(x)$ where the summation convention is used and $i$ is summed over $0,1,\ldots,n$.
So we can write $y^{(n)}$ as a combination of $y^{(i)}$ for $i\in\{0,1,2,\ldots,n-1\}$ and $f$ and $p_i$'s.
\begin{example}
    The damped oscillator has the DE
    $$m\ddot{y}=-ky-L\dot{y}$$
    The state of the system can be described by an $n$-dimensional solution vector
    $$\underline{y}=
    \begin{pmatrix}
        y\\
        y^\prime\\
        \vdots\\
        y^{(n-1)}
    \end{pmatrix}$$
    Going back to an undampted oscillator $y^{\prime\prime}+4y=0$ which has the general solution spanned by $y_1(x)=\cos{2x}, y_2(x)=\sin{2x}$, so the solution vectors are
    $$\underline{y_1}=
    \begin{pmatrix}
        \cos{2x}\\
        -2\sin{2x}
    \end{pmatrix},
    \underline{y_2}=
    \begin{pmatrix}
        \sin{2x}\\
        2\cos{2x}
    \end{pmatrix}$$
    Thus we can do 2D phase portrait of the solutions to observe the two vectors, where we can find that the trajectories coincide.
    Since $y_1,y_2$ are linearly independent, any point in phase space can be obtained from a linear combination of them.
    In general, $y_1,y_2,\ldots, y_n$ are linearly independent if their solution vectors are linearly independent in the phase space.
\end{example}
$n$ linearly independent solution vectors form a basis for the phase space of an $n^{th}$ order ODE.
Consider the initial conditions for a second order homogeneous ODE $y(0)=a,y^\prime(0)=b$.
If the general solution is formed by the linear combination of linearly independent functions $y_1,y_2$, then in order to find a solution that complies with the initial condition, we will be solving the linear system in $A,B$.
$$
\begin{cases}
    Ay_1(0)+By_2(0)=a\\
    Ay_1^\prime(0)+By_2^\prime(0)=b\\
\end{cases}
$$
So we obtain unique solutions if and only if
$$y_1(0)y_2^\prime(0)\neq y_2(0)y_1^\prime(0)$$
which is true if $\underline{y_1}(0),\underline{y_2}(0)$ are linearly independent.
\begin{definition}
    The Wronskian $W(x)$ is defined as
    $$W(x)=
    \begin{vmatrix}
        y_1&y_2&\dots&y_n\\
        y_1^\prime&y_2^\prime&\dots&y_n^\prime\\
        \vdots&\vdots&\ddots&\vdots\\
        y_1^{(n-1)}&y_2^{(n-1)}&\dots&y_n^{(n-1)}
    \end{vmatrix}
    $$
\end{definition}
So the solutions are linearly independent if $W(x)\neq 0$.
But does $W(x)=0$ necessarily imply linear dependence?
\begin{theorem}[Abel's Theorem]
    Consider a second order linear ODE $y^{\prime\prime}+py^\prime+qy=0$.
    If $p(x),q(x)$ are continuous on an interval $I$, then either $\forall x\in I, W(x)\neq 0$ or $\forall x\in I, W(x)=0$
\end{theorem}
\begin{proof}[Sketch of proof]
    Let $y_1,y_2$ be solutions to the ODE, and $\mathscr D$ be the differential operator in the left hand side of the equation, so
    $$y_2\mathscr{D}(y_1)=y_1\mathscr{D}(y_2)=0\implies y_2y_1^{\prime\prime}-y_1y_2^{\prime\prime}+(y_2y_1^\prime-y_1y_2^\prime)p\implies W^\prime+pW=0$$
    So we could integrate it back from $x_0$ to $x$ to get
    $$W(x)=W(x_0)\exp\left(-\int_{x_0}^xp(u)\,\mathrm du\right)$$
    which is called the Abel identity.
    Since $p$ is continuous on a closed interval, it is bounded and integrable, therefore the exponential function part is always defined and nonzero, so $W(x_0)=0$ if and only if $W(0)=0$.
\end{proof}
\begin{corollary}
    If $p(x)=0$, then the Wronskian is constant.
\end{corollary}
One can generalize the theorem above to $n^{th}$ order linear ODEs.
Indeed, for $\underline{y}^\prime+A(x)\underline{y}=\underline{0}$, we have $W^\prime+\operatorname{tr}(A(x))W=0$.\\
One practical application of Abel's identity can be used to find a second solution $y_2$ given one solution $y_1$.
This can be done by observing that
$$y_1y_2^\prime-y_2y_1^\prime=W(x)=W(x_0)\exp\left(-\int_{x_0}^xp(u)\,\mathrm du\right)$$
is both linear and of first order in $y_2$.\\
We cannot, of course, (analytically) solve all ODEs, but for some special types of them, it is sometimes possible.
\subsection{Special Types of ODEs}
\begin{definition}
    An ODE is called equidimensional if the differential operator is unaffected by a multiplicative rescaling.
    So $\mathscr{D}_x=\mathscr{D}_{\tilde{x}=\alpha x}$ where $\alpha$ is a constant.
    So the general form of a second order linear equidimensional is the following:
    $$ax^2\frac{\mathrm d^2y}{\mathrm dx^2}+bx\frac{\mathrm dy}{\mathrm dx}+cy=f(x)$$
\end{definition}
There are two methods to find $y_c$.\\
Method 1: Note that $y=x^k$ is an eigenfunction of the eigenvector of $x(\mathrm d/\mathrm dx)$.
To solve the homogeneous equation, we can plug the eigenfunction in and find out that $k$ satisfies
$$ak(k-1)+bk+c=0$$
If there are two roots $k_1,k_2$, then the general solution is $y_c=Ax^{k_1}+Bx^{k_2}$ where $A,B$ are constants.
Otherwise, we have at least one $y_c$, so we could use the method in previous sections to find the other.\\
Method 2: Use the substitution $z=\ln x$, so we will have
$$a\frac{\mathrm d^2y}{\mathrm dz^2}+(b-a)\frac{\mathrm dy}{\mathrm dz}+cy=0$$
which we know how to solve.\\
For the forced type of ODE, due to the superposition principle, we just (and will) need a particular solution $y_p$.
There are two methods to do it.\\
Method 1: Guesswork.
\begin{center}
    \begin{tabular}{c|c}
        Form of $f(x)$&Educated guess\\
        \hline
        $e^{kx}$&$Ae^{kx}$\\
        $\sin(kx),\cos(kx)$&$A\sin{kx}+B\cos{kx}$\\
        Polynomial&Polynomials
    \end{tabular}
\end{center}
We could plug the educated guesses in and solve for the coefficients.\\
Method 2: Method of parameters.
Given complementary functions $y_1,y_2$ and solution vectors $\underline{y_1},\underline{y_2}$.
Suppose that the solution vector $\underline{y_p}$ for $y_p$ satisfies
$$\underline{y_p}=u(x)\underline{y_1}+v(x)\underline{y_2}$$
So what we now try to do is to define two equations for $u^\prime,v^\prime$
$$
\begin{cases}
    y_p=uy_1+vy_2\\
    y_p^\prime=uy_1^\prime+vy_2^\prime
\end{cases}
$$
So by differentiating the first equation and comparing it with the other, we have $u^\prime y_1+v^\prime y_2=0$.
Now differentiate the second equation and plug back to the differential equation, we know that $u^\prime y_1^\prime+ v^\prime y_2^\prime =f(x)$.
Since the Wronskian is not zero due to definitions of $y_1,y_2$, we know that we can solve for $u^\prime,v^\prime$ and a unique solution is guaranteed.
Indeed, $u^\prime=-fy_2/W,v^\prime=fy_1/W$.
So
$$y_p=y_2\int_{x_0}^x\frac{y_1(t)f(t)}{W(t)}\,\mathrm dt-y_1\int_{x_0}^x\frac{y_2(t)f(t)}{W(t)}\,\mathrm dt$$
\begin{definition}
    A forced oscillating ODE is a linear second order ODE forced by oscillating force.
\end{definition}
This arises as many physical systems have a restoring force and damping (e.g. friction).
\begin{example}
    A wheel of mass $M$ is connected to a spring and a damper from above and a force $F(t)$ is applied from below, so $M\ddot{y}=F(t)-ky-L\dot{y}$, written in stardard form,
    $$\ddot{y}+\frac{L}{M}\dot{y}+\frac{k}{M}y=\frac{F(t)}{M}$$
    we redefine time $\tau=\sqrt{k/M}t$ and get
    $$y^{\prime\prime}+2Ky^\prime+y=f(\tau), K=\frac{L}{2\sqrt{kM}}$$
\end{example}
We can evaluate the unforced (free, homogeneous) solution where $f\equiv 0$, which we know how to solve in the general form $Ay_1+By_2$ where $A,B$ are constants.\\
Case 1: $K<1$, so we have complex roots of auxiliary equation, so we say the system is underdamped, so
$$y=e^{-K\tau}[A\sin(\sqrt{1-K^2}\tau)+B\cos(\sqrt{1-K^2}\tau)]$$
Case 2: $K=1$, which system we call it is a critically damped.
$$y=(A+B\tau)e^{-K\tau}$$
Case 3: $K>1$, where the system is overdamped.
$$y=Ae^{\lambda_1\tau}+Be^{\lambda_2\tau},\lambda_{1,2}=-K\pm\sqrt{K^2-1}$$
So the unforced response always decays exponentially as time goes to infinity.\\
As for forced response, if we have
$$\ddot{y}+\mu\dot{y}+\omega_0^2y=\sin{\omega t}$$
which by guessing we have the particular solution
$$y_p=\frac{\omega_0^2-\omega^2}{\omega_0^2-\omega^2+\mu^2\omega^2}\sin{\omega t}+\frac{-\mu\omega}{\omega_0^2-\omega^2+\mu^2\omega^2}\cos{\omega t}$$
For $\mu\neq 0$, we have finite complitude oscillations matching the forcing frequency.
Even when $\omega=\omega_0$, then taking the limit to get $-(\cos\omega t)/(\mu\omega)$, in which case we still have finite amplitude oscillations.\\
So in general, in a damped system, the unforced part gives the short time response whilst the particular solution gives the long time behaviour.\\
But if $\mu=0,\omega_0=\omega$, we call this is a resonance.
The forcing then matches the unforced responce, where the equation turns to
$$\ddot{y}+\omega_0 y=\sin\omega_0 t$$
We shall use detuning to solve the equation.
Consider the equation
$$\ddot{y}+\omega_0 y=\sin\omega t$$
So by guessing, we have $y_p=\sin\omega t/(\omega_0^2-\omega_2)$.
Due to linearity, $y_p+Ay_c$ works for any constant $A$, so
$$y_p=\frac{\sin\omega t-\sin\omega_0t}{\omega_0^2-\omega^2}=\frac{2}{\omega_0^2-\omega^2}\cos\left(\frac{\omega_0+\omega}{2}t\right)\sin\left(\frac{\omega-\omega_0}{2}t\right)$$
also solves the equation.
By letting $\omega_0\to\omega$, we have
$$y_p=-\frac{t\cos\omega_0t}{2\omega_0}$$
which indeed solves the problem.
\subsection{Dirac Delta (sorry; not sorry)}
\begin{definition}
    An impulse forcing is a type of forcing by a sudden change.
\end{definition}
\begin{example}
    A mountainbike riding from the road onto a side block.
    When they hits, then there is a sudden increase in altitude, which might be followed by some oscillation.
    When the time for this goes to zero, then it is considered as an (infinite) impulse.
    So the forced, damped oscillator has the equation
    $$M\ddot{y}=F(t)-ky-L\dot{y}$$
    Now if we add the impulse, and we integrate both sides and let the time elapsed during the hitting to the road block tend to $0$, we will want to define the impulse by
    $$I=\lim_{\epsilon\to 0}\int_{T-\epsilon}^{T+\epsilon}F(t)\,\mathrm dt=\lim_{\epsilon\to 0}M[\dot{y}]^{T+\epsilon}_{T-\epsilon}$$
\end{example}
We then introduce the Dirac delta function.
\footnote{And this is the end of the world.}
\begin{definition}
    Fix a family of functions $D(t;\epsilon)$ indexed by $\epsilon$ with the properties:\\
    1. For any $t\neq 0$, we have $\lim_{\epsilon\to0}D(t;\epsilon)=0$ for $t\neq 0$.\\
    2.
    $$\int_{\mathbb R}D(t;\epsilon)\,\mathrm dt=1$$
    We ``define'' the Dirac delta function by $\delta(t)=\lim_{\epsilon\to 0}D(t;\epsilon)$.
\end{definition}
\begin{proposition}
    1. $\forall t\neq 0,\delta(t)=0$.\\
    2. For any $a<0<b$,
    $$\int_a^b\delta(t)\,\mathrm dt=1$$
    3. We have the sampling property.
    For all functions $g(x)$ which are continuous at $x=0$, then
    $$\int_{\mathbb{R}}g(x)\delta(x)\,\mathrm dx=g(0)$$
    In general
    $$\int_a^bg(x)\delta(x-x_0)\,\mathrm dx=
    \begin{cases}
        g(x_0)\text{, if $x_0\in[a,b]$}\\
        0\text{, otherwise}
    \end{cases}$$
\end{proposition}
\begin{proof}
    Ahem.
\end{proof}
\begin{definition}
    The Heaviside step function $H(x)$ is defined by
    $$H(x)=\int_{-\infty}^x\delta(x)\,\mathrm dx=
    \begin{cases}
        1\text{, if $x>0$}\\
        0\text{, if $x<0$}
    \end{cases}$$
    And $H(0)$ is not defined.\\
    It is decreed that $H^\prime(x)=\delta(x)$ ``by FTC''.
    \footnote{I know, I know, stay calm, it is an applied course.}
\end{definition}
\begin{definition}
    The ramp function $r(x)$ is defined by
    $$r(x)=\int_{-\infty}^xH(x)\,\mathrm dx=
    \begin{cases}
        x\text{, when $x\ge 0$}\\
        0\text{, otherwise}
    \end{cases}$$
    By FTC, $r^\prime=H$.
    \footnote{Lol I guess?}
\end{definition}
Consider
$$y^{\prime\prime}+py^\prime+qy=\delta(x)$$
Since $\delta(x)=0$ for all $x\neq 0$, so $y^{\prime\prime}+py^\prime+qy=0$ for $x<0$ and $x>0$.
The highest order derivative inherits the discontinuity from the forcing.
But we would want $y$ to be continuous at $x=0$, so $\lim_{\epsilon\to 0}[y]^{\epsilon}_{-\epsilon}=0$.
And $y^\prime$ would have a jump near $0$, so
$$\lim_{\epsilon\to 0}[y^\prime]^\epsilon_{-\epsilon}=\lim_{\epsilon\to 0}\int_{-\epsilon}^\epsilon y^{\prime\prime}+py^\prime+qy\,\mathrm dx=1$$
We first solve the equations for $x<0$ and $x>0$, so we will have $4$ undetermined constants. and $2$ initial conditions, so we need $2$ more equations, which we can find from the two jump conditions as stated above.
\begin{example}
    Consider $y^{\prime\prime}-y=3\delta(x-\pi/2)$
    with $y=0$ at $x=0,\pi$.
    Note that $y^{\prime\prime}-y=0\implies y=A\sinh x+B\cosh x$
    Our initial conditions then implies
    $$y=
    \begin{cases}
        A\sinh x\text{, when $x<\pi/2$.}\\
        C\sinh (\pi-x)\text{, when $x>\pi/2$.}
    \end{cases}$$
    For constants $A,C$.
    So we put in the jump condition to have
    $3=\int_{\pi/2-\epsilon}^{\pi/2+\epsilon}y^{\prime\prime}-y\,\mathrm dx=[y^\prime]^{(\pi/2)^+}_{(\pi/2)^-}$
    Putting in the definitions to have
    $$-A\cosh(\pi/2)-c\cosh(\pi/2)=3$$
    Since we also have $0=[y]^{(\pi/2)^+}_{(\pi/2)^-}$, we can solve to get $A=C$, therefore
    $$A=C=\frac{-3}{2\cosh(\pi/2)}$$
    Thus
    $$y=
    \begin{cases}
        \frac{-3\sinh x}{2\cosh(\pi/2)}\text{, when $x\le\pi/2$.}\\
        \frac{-3\sinh (\pi-x)}{2\cosh(\pi/2)}\text{, when $x>\pi/2$.}
    \end{cases}$$
\end{example}
We can also have the forcing by Heaviside function.
Consider
$$y^{\prime\prime}+py^\prime+qy=H(x-x_0)$$
For $p,q$ continuous.
$y(x)$ satisfies $y^{\prime\prime}+py^\prime+qy=0$ for $x<x_0$ and $y^{\prime\prime}+py^\prime+qy=1$ for $x>x_0$
We can evaluate the equation on either sides of $x_0$, thus
$$[y^{\prime\prime}]^{x_0^+}_{x_0^-}+p[y^{\prime}]^{x_0^+}_{x_0^-}+q[y]^{x_0^+}_{x_0^-}=1$$
If $y^{\prime\prime}$ behaves like the Heaviside function (then $y^\prime$ bahaves like the ramp function), then $y^\prime,y$ are continuous, thus $[y^{\prime}]^{x_0^+}_{x_0^-}=[y]^{x_0^+}_{x_0^-}=0$ and $[y^{\prime\prime}]^{x_0^+}_{x_0^-}=1$, which is our jump conditions, which would be enough to find out the constants with the initial conditions.
\subsection{Higher-Order Discrete/Difference Equations}
\begin{definition}
    The general form of an $m^{th}$ order linear discrete equation with constant coefficient is
    $$a_my_{n+m}+a_{m-1}y_{n+m-1}+\cdots+a_0y_n=f(n)$$
\end{definition}
Turns out that they are closely related to higher order DEs, and we can solve them using the same principles.
\begin{definition}
    A difference operator $\mathcal D$ is such that $\mathcal D (y_n)=y_{n+1}$.
    It has eigenfunctions in the form $y_n=k^n$ for constant $k$ as $D(k^n)=k(k^n)=ky_n$.
\end{definition}
Note that our difference equation is linear in $y$, thus we can dissolve $y$ into sum of particular and complementary solutions $y_n=y_n^{(c)}+y_n^{(p)}$.
\begin{example}
    We want to solve $a_2y_{n+2}+a_1y_{n+1}+a_0y_n=f_n$.
    Consider the homogeneous equation with $f=0$, then we can put in $y_n=k^n$ to get $a_2k^2+a_1k+a_0=0$, so its solutions $k_{1,2}$ gives the general form of the complementary solution
    $$
    y_n^{(c)}=
    \begin{cases}
        Ak_1^n+Bk_2^n\text{, if $k_1\neq k_2$}\\
        Ak_1^n+Bnk_1^n\text{, if $k_1=k_2$}
    \end{cases}
    $$
    Note that these are all the complementary solutions since it has $2$ degrees of freedom and the sequence would be completely determined by its value at the first two initial values.\\
    We can use guessing works for particular solution
    \begin{center}
        \begin{tabular}{c|c}
            Form of $f_n$&Form of $y_n^{(p)}$\\
            \hline
            $k^n,k\neq k_{1,2}$&$Ak^n$\\
            $k_{1,2}^n$&$Ank_1^n+Bnk_2^n$\\
            Polynomial&Polynomials
        \end{tabular}
    \end{center}
\end{example}
The Fibonacci numbers are defined as $y_0=y_1=1,y_{n+1}=y_n+y_{n-1}$ for $n\ge 1$.
Note that it has the auxiliary equation $k^2-k-1=0$, which has roots $k_{1,2}=(1\pm\sqrt{5})/2$
So by plugging in our initial conditions, we obtain
$$y_n=\frac{1}{\sqrt{5}}(\phi^{n+1}-(-\phi^{-1})^{n+1})$$
where $\phi=(\sqrt{5}+1)/2$ is the golden ratio.
In particular, $y_{n+1}/y_n\to\phi$ as $n\to\infty$.
\subsection{Series Solutions}
Often, there are no analytic solutions to some particular ODE or it is very hard to obtain one.
In this case, we can try to solve the equation in the form of an infinite power series.
We can use the method of Frobenius.
Consider the ODE $py^{\prime\prime}+qy^\prime+ry=0$.
We will seek a series expansion at $x=x_0$ of a (local) solution around some point.
There are many choice of $x_0$.
If the series expansions of $q/p$ and $r/p$ converge locally at $x_0$, we say $x_0$ is an ordinary point.
Otherwise, we say it is a singular point.
There are two types of singular point:
If $x_0$ is a singular point but the equation can be written in the following way:
$$P(x-x_0)^2y^{\prime\prime}+Q(x-x_0)y^\prime+Ry=0$$
and $Q/P,R/P$ are analytic, then we say $x_0$ is a regular singular point.
Note that $Q/P=(x-x_0)q/p,R/P=(x-x_0)^2r/p$.
Otherwise it is called an irregular singular point.
\begin{example}
    1. We want to solve
    $$(1-x^2)y^{\prime\prime}-2xy^\prime+2y=0$$
    then $q/p=-2x/(1-x^2),r/p=2/(1-x^2)$, so $x=\pm 1$ are singularities.
    But $Q/P=(x-x_0)q/p=-2x/(1+x)$, so $x=1$ is regular.
    Similarly $x=-1$ is regular as well.\\
    2. Consider
    $$y^{\prime\prime}\sin x+y^\prime\cos x+2y=0$$
    Then the singularities are $n\pi,n\in\mathbb Z$, but since $(x-n\pi)/(\sin x)$ as $x\to n\pi$ tends to a limit, every of them is regular.\\
    3. We look into
    $$(1+\sqrt{x})y^{\prime\prime}-2xy^\prime+2y=0$$
    So $q/p=-2x/(1+\sqrt{x})$, one can find that the Taylor series at $0$ is undefined.
    Indeed, $0$ is an irregular singular point here.
\end{example}
\begin{theorem}[Fuch's Theorem]
    1. If $x=x_0$ is an ordinary point, then there are two linearly independent power series solutions of the form
    $$y=\sum_{n=0}^\infty a_n(x-x_0)^{n}$$
    locally near $x_0$.\\
    2. If $x=x_0$ is a regular singular point, then there is at least $1$ solution of the form
    $$y=\sum_{n=0}^\infty a_n(x-x_0)^{n+\sigma}$$
    where $\sigma$ is real and $a_0\neq 0$.
\end{theorem}
\begin{example}
    1. The equation $(1-x^2)y^{\prime\prime}-2xy^\prime+2y=0$ has singular points $\pm 1$ and they are both regular.
    We first find series solution about an ordinary point, say $x=0$.
    We try
    $$y=\sum_{n=0}^\infty a_n(x-0)^{n}$$
    So by plugging in,
    $$(1-x^2)\sum_{n=2}^\infty n(n-1)a_nx^{n-2}-2x\sum_{n=1}^\infty na_nx^{n-1}+2\sum_{n=0}^\infty a_nx^{n}=0$$
    From which we have $a_nn(n-1)-a_{n-2}(n-2)(n-3)-2a_{n-2}(n-2)+2a_{n-2}=0$ for $n\ge 2$.
    Just simplify to get $n(n-1)a_n=(n^2-3n)a_{n-2}$, so
    $$a_n=\frac{n-3}{n-1}a_{n-2}$$
    Consequently, $a_0,a_1$, which can be arbitrary constants, could be our initial condition.
    Note that $a_3=0$, hence $a_k=0$ for any odd $k\ge 3$.
    for even values of $n$, we have
    $$a_n=\frac{n-3}{n-1}a_{n-2}=\frac{n-3}{n-1}\frac{n-5}{n-3}a_{n-4}=\frac{n-5}{n-1}a_{n-4}=\cdots=\frac{n-2k-1}{n-1}a_{n-2k}$$
    Thus $a_{2k}=a_0/(1-n)$, so
    $$y=a_1x+a_0\left(1-x^2-\frac{x^4}{3}-\frac{x^6}{5}-\cdots\right)=a_1x+a_0\left( 1-\frac{x}{2}\ln\frac{1+x}{1-x} \right)$$
    2. Consider $4xy^{\prime\prime}+2(1-x^2)y^\prime-xy=0$, which has a regular singular point at $x=0$.
    We now try to expand the solution near it.
    We try $y=\sum_{n=0}^\infty a_nx^{n+\sigma}$ for $a_0\neq 0$ by Fuch's Theorem.
    So we can plug it in our equation to try and get a recurrence for the coefficients.
    Note first that, by multiplying $x$ to both sides
    $$\sum_{n=0}^\infty a_nx^{n+\sigma}(4(n+\sigma)(n+\sigma-1)+2(1-x^2)(n+\sigma)-x^2)=0$$
    Thus by comparing coefficients,
    $$2(n+\sigma)(2n+2\sigma-1)a_n=(2n+2\sigma-3)a_{n-2}$$
    which is our equivalence relation.\\
    To find $\sigma$, we will equate coefficient of lowest power of $x$.
    Set $n=0$, then we can equate the coefficient of $x^\sigma$, so $a_0(4\sigma(\sigma-1))+a_02\sigma=0\implies 2\sigma(2\sigma-1)a_0=0$.
    This is called the indicial equation.
    So we get $\sigma=0$ or $\sigma=1/2$.\\
    For $\sigma=0$, we again set $n=0$ to find that $a_0$ is arbitrary.
    Then consider $n=1$, we have $2a_1=0\implies a_1=0$.
    Our equivalence relation reduced to $2n(2n-1)a_n=(2n-3)a_{n-2}$, which means that $a_k=0$ for any odd $k$.
    For even $n=2k$, we can calculate a few values to get
    $$y=a_0\left(1+\frac{x^2}{4\cdot 3}+\frac{5x^4}{8\cdot 7\cdot 4\cdot 3}+\cdots\right)$$
    For $\sigma=1/2$, our recurrence relation reduced to $(2n+1)(2n)b_n=(2n-2)b_{n-2}$ for $n\ge 2$ where $b_n=a_n$ to avoid ambiguity.
    We can equate the coefficient in the lowest power to get $b_0$ being arbitrary, and by considering $n=1$ we have $b_1=0$, so $b_k=0$ for any odd $k$ as well.
    We can calculate $b_n$ for $n$ even and get
    $$y=b_0x^{1/2}\left( 1+\frac{x^2}{2\cdot 5}+\frac{5x^4}{2\cdot 5\cdot 4\cdot 9}+\cdots \right)$$
\end{example}
There are some special cases of the indicial equation.
Let $x_0$ be a regular singular point, and suppose $\operatorname{Re}(\sigma_1)\le\operatorname{Re}(\sigma_2)$ where $\sigma_{1,2}$ are the roots of the indicial equation.\\
Case 1: $\sigma_2-\sigma_1$ is a non-integer, then the two solutions are linearly independent.\\
Case 2: It is a nonzero integer.
In this case, it is possible, but no guarantee, that the solutions $y_1,y_2$ are linearly dependent, so we might need an extra term in the form $cy_1\ln(x-x_0)$ in $y_2$, where $c$ is a constant.\\
Case 3: It is $0$.
Hence $c\neq 0$, so we can set $c=1$, so we may add $y_1\ln(x-x_0)$ to $y_2$ in order to yield two linearly independent solutions.


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Trigonometric Integral}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Various exercises relating to the integration of trigonometric functions.
\end{abstract}
\maketitle

Compute the indefinite integrals below. Since there are many possible answers (which differ by constant values), use the given instructions if needed to choose which possible answer to use.

\begin{exercise}%[APEX0603TRIG05]
\[  \int \sin x\cos^4x\ dx = \answer{-\frac{1}{5} \cos ^5(x)}+C\]
(Your answer should not include any constant term.)
%
%
\end{exercise}

\begin{exercise}%[APEX0603TRIG08]
\[ \int \sin^3 x\cos^3 x\ dx = \answer{\frac{1}{6}\cos^6x-\frac{1}{4}\cos^4x}+C\]
(Your answer should not include any constant terms.)
%
%
\end{exercise}

\begin{exercise}%[APEX0603TRIG19]
\[ \int \tan^4x\sec^2x\ dx = \answer{\frac{\tan ^5(x)}{5}}+ C\]
(Add a constant to your answer if needed so that it equals $0$ at $x = 0$.)
%
%
\end{exercise}

\begin{exercise}%[APEX0603TRIG23]
\[ \int \tan^3x\sec^3x\ dx = \answer{\frac{\sec ^5(x)}{5}-\frac{\sec ^3(x)}{3}}+ C\]
(Add a constant to your answer if needed so that it equals $-2/15$ at $x = 0$.)
%
%
\end{exercise}

\begin{exercise}%[APEX0603TRIG10]
\[ \int \sin^2 x\cos^7 x\ dx = \answer{-\frac{1}{9} \sin ^{9}(x)+\frac{3 \sin^7(x)}{7}-\frac{3\sin ^5(x)}{5}+\frac{\sin ^3(x)}{3}}+ C\]
(Your answer should not include any constant terms.)
\begin{hint}
\[ (1-u^2)^3 = 1 - 3 u^2 + 3 u^4 - u^6. \]
\end{hint}
%
%
\end{exercise}

\begin{exercise}%[APEX0603TRIG13]
\[ \int \sin(5x)\cos(3x)\ dx = \answer{\frac{1}{2}\left(-\frac{1}{8}\cos(8x)-\frac{1}{2}\cos(2x)\right)}+ C\]
\begin{hint}
You can either use an integration by parts technique or you can use a trigonometric identity to simplfy the expression $\cos Ax \sin Bx$ (for constants $A$ and $B$) as a sum of simpler things.
\end{hint}
(Your answer should not include any constant terms.)
%
%
\end{exercise}

\begin{exercise}%[APEX0603TRIG11]
\[ \int \sin^2 x\cos^2 x\ dx = \answer{\frac{x}{8}-\frac{1}{32} \sin (4 x)}+ C \]
(Your answer should not include any constant terms and should equal $0$ at $x = 0$.)
\begin{hint}
Use power reduction formulas.
\end{hint}
%
%
\end{exercise}

\begin{exercise}
To fully evaluate the integral from Example \ref{trig:reduce_example}, it helps to identify the pattern.  Suppose that the power $7$ is replaced by an unknown positive constant $n > 2$. Carry out the calculation again with the unspecified exponent:
\[ \begin{aligned}
\int & \sec^{n} x \, dx  = \int \sec^{\answer{n-2}} x  \sec^2 x \, dx \\
& = \sec^{\answer{n-2}} x \tan x - \int \answer{(n-2)} (\sec x)^{\answer{n-3}} (\sec x \tan x) \tan x dx \\
& = \sec^{\answer{n-2}} x \tan x - \answer{(n-2)}  \int \sec^{n} x \, dx + \answer{n-2} \int \sec^{\answer{n-2}} x \, dx.
\end{aligned} \]
We conclude
\[ \int \sec^n x \, dx = \answer{\frac{1}{n-1}} \sec^{n-2} x \tan x + \answer{\frac{n-2}{n-1}} \int \sec^{n-2} x \, dx. \]
Using this formula several times in a row gives the result
\[ \begin{aligned}
\int &  \sec^7 x \, dx \\
 = & \frac{\sec^5 x \tan x}{6} + \frac{5}{6} \int \sec^5 x \, dx \\
 = & \frac{\sec^5 x \tan x}{6} + \frac{5}{6}  \left[ \frac{\sec^3 x \tan x}{4} + \frac{3}{4} \int \sec^3 x \, dx \right] \\
 = & \frac{\sec^5 x \tan x}{6} +  \frac{5 \sec^3 x \tan x}{4 \cdot 6} + \frac{5}{6} \cdot \frac{3}{4} \left[ \frac{\sec x \tan x}{2} + \frac{1}{2} \int \sec x \, dx \right] \\
 = & \frac{\sec^5 x \tan x}{6} + \frac{5 \sec^3 x \tan x}{24} + \frac{5 \sec x \tan x}{16} + \frac{5}{16} \ln |\sec x + \tan x| + C.
\end{aligned} \]
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[TrigInt001]

Compute the value of the integral
\[\int_{0}^{\frac{\pi}{4}} \sin^{3} 2x ~dx.\]
(Hints won't be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle \frac{1}{5}\)}
\choice[correct]{\(\displaystyle \frac{1}{3}\)}
\choice{\(\displaystyle \frac{1}{2}\)}
\choice{\(\displaystyle 1\)}
\choice{\(\displaystyle 2\)}
\choice{\(\displaystyle 3\)}
\end{multiplechoice}
\begin{feedback}
To simplify the calculation, begin with a substitution which replaces \(x\) with \(x/2\). The question reduces to computing
\[\frac{1}{2}\int_{0}^{\frac{\pi}{2}} \sin^{3} x ~dx.\]
This integral is compatible with the substitution \(u = \cos x\). \begin{hint} By the substitution formula, this means \(dx = - du / \sin x\), and one must also replace \(\sin^2 x\) by \(1 - u^2\). Furthermore, by virtue of the special angle formulas \(\cos 0 = 1\) and \(\cos \frac{\pi}{2} = 0\), the problem is reduced to computing the integral
\[-\frac{1}{2}\int_{1}^{0} (1 - u^2) ~du.\]
\begin{hint}
Carrying out this calculation in the usual way gives a final answer of \(\frac{1}{3}\).
\end{hint}
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[TrigInt010]

Compute the value of the integral
\[\int_{\frac{\pi}{6}}^{\frac{\pi}{2}} \tan^{-6} x  \sec^{5} x ~dx.\]
(Hints won't be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle \frac{17}{5}\)}
\choice{\(\displaystyle \frac{19}{5}\)}
\choice{\(\displaystyle \frac{23}{5}\)}
\choice{\(\displaystyle \frac{29}{5}\)}
\choice[correct]{\(\displaystyle \frac{31}{5}\)}
\choice{\(\displaystyle \frac{37}{5}\)}
\end{multiplechoice}
\begin{feedback}
Since the power of secant is odd and the power of tangent is even, try rewriting the integral in terms of sine and cosine. This gives
\[\int_{\frac{\pi}{6}}^{\frac{\pi}{2}} \sin^{-6} x  \cos x ~dx.\]
This integral is compatible with the substitution \(u = \sin x\). \begin{hint} By the substitution formula, this means \(dx = du / \cos x\). Furthermore, by virtue of the special angle formulas \(\sin \frac{\pi}{6} = \frac{1}{2}\) and \(\sin \frac{\pi}{2} = 1\), the problem is reduced to computing the integral
\[\int_{\frac{1}{2}}^{1} u^{-6} ~du.\] \begin{hint}
Carrying out this calculation in the usual way gives a final answer of \(\frac{31}{5}\). \end{hint} \end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[TrigInt020]

Compute the value of the integral
\[\int_{\frac{\pi}{6}}^{\frac{\pi}{2}} \sin^{-2} x  \cos^{3} x ~dx.\]
(Hints won't be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle \frac{1}{5}\)}
\choice{\(\displaystyle \frac{1}{3}\)}
\choice[correct]{\(\displaystyle \frac{1}{2}\)}
\choice{\(\displaystyle 1\)}
\choice{\(\displaystyle 2\)}
\choice{\(\displaystyle 3\)}
\end{multiplechoice}
\begin{feedback}
This integral is compatible with the substitution \(u = \sin x\). By the substitution formula, this means \(dx = du / \cos x\), and one must also replace \(\cos^2 x\) by \(1-u^2\). Furthermore, by virtue of the special angle formulas \(\sin \frac{\pi}{6} = \frac{1}{2}\) and \(\sin \frac{\pi}{2} = 1\), the problem is reduced to computing the integral \begin{hint}
\[\int_{\frac{1}{2}}^{1} u^{-2}(1-u^2) ~du.\] \begin{hint}
Carrying out this calculation in the usual way gives a final answer of \(\frac{1}{2}\). \end{hint} \end{hint}
\end{feedback}

\end{question}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2015C.10]

Compute the integral below.
\[ \int_0^{\frac{\pi}{8}} \tan^4 2x \sec^4 2x ~ dx \]
\begin{multiplechoice}
\choice{\(\displaystyle \frac{4}{9}\)}
\choice{\(\displaystyle \frac{7}{24}\)}
\choice{\(\displaystyle \frac{5}{14}\)}
\choice{\(\displaystyle \frac{9}{28}\)}
\choice[correct]{\(\displaystyle \frac{6}{35}\)}
\choice{\(\displaystyle \frac{1}{7}\)}
\end{multiplechoice}

\end{question}

\end{document}


\hypertarget{group__numpp__differentiation__forward__automatic}{}\section{Forward Automatic Differentiation}
\label{group__numpp__differentiation__forward__automatic}\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}}

First order automatic differentiation in forward mode.

\subsection*{Classes}
\begin{DoxyCompactItemize}
\item
class \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward$<$ T $>$}
\end{DoxyCompactItemize}
\begin{DoxyCompactItemize}
\item
{\footnotesize template$<$typename T $>$ }\\constexpr auto \hyperlink{group__numpp__differentiation__forward__automatic_ga2faff4e6f917370493a6c12f947e2573}{std\+::relu} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_gab45ba058e4fd16d0f9f4c26426e385f9}{std\+::exp} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_ga0f6011595edffe756a984de60358fe5f}{std\+::log} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_ga6f6f07250087b18ada9ea1f84418f24f}{std\+::sin} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_gaa5a06320f93037d94fda441546d7d690}{std\+::cos} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_ga9547d13665c214d44b017b140ccc1a3e}{std\+::atan} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_ga9c142d0a384bfa6e79bf6f614c98fe1d}{std\+::sqrt} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\item
{\footnotesize template$<$typename T $>$ }\\C\+O\+N\+S\+T\+E\+X\+PR auto \hyperlink{group__numpp__differentiation__forward__automatic_ga2db415f6b1d59e1f1679ff7bbf4c426b}{std\+::asin} (const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&x)
\end{DoxyCompactItemize}

\subsection{Detailed Description}
First order automatic differentiation in forward mode.

\begin{DoxyCode}
\textcolor{preprocessor}{#include"numpp/differentiation/automatic/forward.h"}
\end{DoxyCode}

You can use it almost in the same way as built-\/in arithmetic types, e.\+g. addition, multiplication and other similiar operations are provided.

It is advised to use auto whenever possible to ease the need of explicit namespace/class names providing.

You can use partial differentiation, but you have to specify the activity parameter in the constructor (equal to 0 for inactive node), example shown below.

/warning\+If you need derivatives of higher order it is advised to use symbolic differentiation module for it\textquotesingle{}s simplifying ability.

{\bfseries Example\+: Calculating $ \frac{\partial z}{\partial x} \cos(x+y)*\sin(y) + 3 $\+:}
\begin{DoxyCode}
\textcolor{comment}{//Active variable x with initial value of 6}
\hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp::differentiation::automatic::forward}
      \hyperlink{group__numpp__differentiation__symbolic_gac865497d2896f51d0cab2e9c64799a15}{x}\{6\};
\textcolor{comment}{//Inactive variable y with initial value of 4;}
\hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp::differentiation::automatic::forward} y\{4\};
\textcolor{comment}{//Calculate value and partial derivative into two seperate variables}
\textcolor{keyword}{auto} z = \hyperlink{group__numpp__differentiation__forward__automatic_gaa5a06320f93037d94fda441546d7d690}{std::cos}(\hyperlink{group__numpp__differentiation__symbolic_gac865497d2896f51d0cab2e9c64799a15}{x}+y)*\hyperlink{group__numpp__differentiation__forward__automatic_ga6f6f07250087b18ada9ea1f84418f24f}{std::sin}(y) + 3
std::cout << \textcolor{stringliteral}{"Function value: "} << z.value << std::endl;
std::cout << \textcolor{stringliteral}{"Function derivative wrt x: "} << z.derivative<< std::endl;
\end{DoxyCode}

\begin{DoxySeeAlso}{See also}
numpp\+\_\+symbolic\+\_\+differentiation \char`\"{}\+Symbolic Differentiation\char`\"{}
\end{DoxySeeAlso}

\subsection{Function Documentation}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_ga2db415f6b1d59e1f1679ff7bbf4c426b}\label{group__numpp__differentiation__forward__automatic_ga2db415f6b1d59e1f1679ff7bbf4c426b}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!asin@{asin}}
\index{asin@{asin}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{asin()}{asin()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::asin (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $\arcsin(x)$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the asin argument \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through asin function
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_ga9547d13665c214d44b017b140ccc1a3e}\label{group__numpp__differentiation__forward__automatic_ga9547d13665c214d44b017b140ccc1a3e}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!atan@{atan}}
\index{atan@{atan}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{atan()}{atan()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::atan (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $\arctan(x)$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the arctan argument \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through arctan function
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_gaa5a06320f93037d94fda441546d7d690}\label{group__numpp__differentiation__forward__automatic_gaa5a06320f93037d94fda441546d7d690}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!cos@{cos}}
\index{cos@{cos}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{cos()}{cos()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::cos (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $\cos(x)$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the cos argument \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through cos function
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_gab45ba058e4fd16d0f9f4c26426e385f9}\label{group__numpp__differentiation__forward__automatic_gab45ba058e4fd16d0f9f4c26426e385f9}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!exp@{exp}}
\index{exp@{exp}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{exp()}{exp()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::exp (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $e^x$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the exponent \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through exponential function
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_ga0f6011595edffe756a984de60358fe5f}\label{group__numpp__differentiation__forward__automatic_ga0f6011595edffe756a984de60358fe5f}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!log@{log}}
\index{log@{log}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{log()}{log()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::log (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $\ln(x)$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the logarithm argument \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through logarithm function
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_ga2faff4e6f917370493a6c12f947e2573}\label{group__numpp__differentiation__forward__automatic_ga2faff4e6f917370493a6c12f947e2573}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!relu@{relu}}
\index{relu@{relu}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{relu()}{relu()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
constexpr auto std\+::relu (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates rectified linear unit $\max(0,x)$ of a forward variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being passed to Re\+LU \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through Re\+LU
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_ga6f6f07250087b18ada9ea1f84418f24f}\label{group__numpp__differentiation__forward__automatic_ga6f6f07250087b18ada9ea1f84418f24f}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!sin@{sin}}
\index{sin@{sin}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{sin()}{sin()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::sin (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $\sin(x)$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the sin argument \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through sin function
\end{DoxyReturn}
\mbox{\Hypertarget{group__numpp__differentiation__forward__automatic_ga9c142d0a384bfa6e79bf6f614c98fe1d}\label{group__numpp__differentiation__forward__automatic_ga9c142d0a384bfa6e79bf6f614c98fe1d}}
\index{Forward Automatic Differentiation@{Forward Automatic Differentiation}!sqrt@{sqrt}}
\index{sqrt@{sqrt}!Forward Automatic Differentiation@{Forward Automatic Differentiation}}
\subsubsection{\texorpdfstring{sqrt()}{sqrt()}}
{\footnotesize\ttfamily template$<$typename T $>$ \\
C\+O\+N\+S\+T\+E\+X\+PR auto std\+::sqrt (\begin{DoxyParamCaption}\item[{const \hyperlink{classnumpp_1_1differentiation_1_1automatic_1_1forward}{numpp\+::differentiation\+::automatic\+::forward}$<$ T $>$ \&}]{x }\end{DoxyParamCaption})}

Calculates $\sqrt(x)$ of a given variable
\begin{DoxyParams}{Parameters}
{\em x} & forward node being the sqrt argument \\
\hline
\end{DoxyParams}
\begin{DoxyReturn}{Returns}
Node passed through sqrt function
\end{DoxyReturn}


\documentclass{book}
\usepackage{lmodern}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{xcolor}
\usepackage{mdframed}
\usepackage[top=3cm, bottom=3cm, inner=4cm, outer=3cm]{geometry}
\usepackage{hhline}
\usepackage{graphicx}

\DeclareMathOperator{\lcm}{lcm}
\DeclareMathOperator{\bigo}{\mathcal{O}}
\renewcommand{\mod}{\textrm{mod }}

\newenvironment{task}
  {\begin{mdframed}[backgroundcolor=lightgray]}
  {\end{mdframed}}

\begin{document}

\chapter{Solutions}

\section{Multiples of 3 and 5}

\begin{task}
If we list all the natural numbers below $10$ that are multiples of $3$ and $5$, we get $3$, $5$, $6$, and $9$. The sum of these multiples is $23$.\\
\\
Find the sum of all the multiples of $3$ and $5$ below $1000$.
\end{task}

The first problem is pretty much straightforward. It asks us to find the sum of all multiples of $3$
and $5$, but any multiple must not exceed $1000 - 1 = 999$. But wait, is this task really that simple?
The only trick is that when we calculate the multiples, some of the multiples of $3$ and $5$ are the
same, e.g. $15$, $30$, ... That means, that if we do not implement this task in smart way, some of the
multiples might get counted twice. Those numbers are exactly the multiples of the least common
multiple of $3$ and $5$, which is $15$. We denote that as $\lcm(3, 5) = 15$.\\

Let us take a look at naive solution:

\begin{verbatim}
#include<stdio.h>

int main()
{
    int sum = 0;
    for (int i = 1; i <= 333; i++)   // we iterate from 1 to (int)999/3 = 333
        sum += 3 * i;
    for (int i = 1; i <= 199; i++)   // we iterate from 1 to (int)999/5 = 199
        sum += 5 * i;
    for (int i = 1; i <= 66; i++)    // we iterate from 1 to (int)999/15 = 66
        sum -= 15 * i;               // we must subtract the twice counted numbers
    printf("%d\n", sum);

    return 0;
}
\end{verbatim}

Note that if you implement this program, it will print the value instantly, therefore you might pose the question, why is this a naive solution?
The reasons are two. Firstly, we use three for loops. That results in asymptotic complexity of $\bigo(m/x + m/y + m/ \lcm(x, y))$, where $x$ and $y$ are the numbers whose multipliers we want to sum and $m$ is the upper bound for all multiples. Now consider we are given extremely large numbers. Although the program runs in linear time, it will take more and more time to get through those three loops. We can clearly see, that we could get rid of three loops and make only one loop, as in next implementation. Wait until you see the real reason why this implementation is not as beautiful as it could be (not to mention it is tiresomly slow - oh, wait, linear is slow, can you do it faster, let’s say logarithmic? Hell no, CONSTANT!).

\begin{verbatim}
#include<stdio.h>

int main()
{
    int sum = 0;
    for (int i = 3; i < 1000; i++)
    {
        if (i % 3 == 0 || i % 5 == 0)
            sum += i;
    }
    printf("%d\n", sum);

    return 0;
}
\end{verbatim}

Note that this implementation does not require any subtraction of multiples of $15$, because we check every integer between $3$ and $999$ inclusively only once. We also expect this program to run slower then the three for one, because in this case we have to loop through $m - 2 = 998$ numbers and each time check whether that number is a multiple of $3$ or $5$, whereas in three for loop we only loop through $\lfloor(m - 1)/x\rfloor + \lfloor(m - 1)/y\rfloor + \lfloor(m - 1)/ \lcm(x, y)\rfloor = 333 + 199 + 66 = 598$ numbers and we do not need to do any checks.

The second reason, and here starts the real solution to the problem, is that what we are doing is actually computing a sum of three \textit{arithmetic series}. An arithmetic series is a sum of numbers that are computed using relation \[a_1;~~~a_i = a_1 + d(i-1),\] where $d$ denotes a difference or \textit{distance} between two numbers in series. In our case we have three arithmetic series, first given by \[a_1 = 3;~~a_i = 3 + 3(i-1) = 3i,\] the second by \[b_1 = 5;~~b_i = 5 + 5(i-1) = 5i\] and the third by \[c_1 = 15;~~c_i = 15 + 15(i-1) = 15i.\] We are actually subtracting the sum of $\{c_i\}_{i = 1}^{66}$ series.
Now, beautiful mathematics gives us a formula for computing sum of arithmetic series. The formula is \[\sum_{i=1}^n a_i = \frac{n}{2}(a_1 + a_n).\] Since in our case $a_n = 3n$, $b_n = 5n$, and $c_n = 15n$, the above equation simplifies into \[\sum_{i=1}^n a_i = a_1 \cdot \frac{n(n+1)}{2}.\] You can simply make your own calculations to see that formula is indeed correct. Let us now implement the program WITHOUT any \texttt{for} loops at all! We use a formula derived above and get \[\textrm{sum} = \sum_{i=1}^{333} 3i + \sum_{i=1}^{199} 5i - \sum_{i=1}^{66} 15i = 3\cdot\frac{333\cdot334}{2} + 5\cdot\frac{199\cdot200}{2} - 15\cdot\frac{66\cdot67}{2}.\]

\begin{verbatim}
#include<stdio.h>
#include "campovski.h"               // needed for LCM

int main()
{
    int x, y, m;                     // we let user input starting numbers
    scanf("%d %d %d, &x, &y, &m);    // (in our case "3 5 1000")
    m--;                             // number below m, m not included (1000-1)

    int z = lcm(x, y);
    int n1 = (int) m / x;            // calculate range of sum for x (3)
    int n2 = (int) m / y;            // calculate range of sum for y (5)
    int n3 = (int) m / z;            // calculate range of sum for LCM(x,y) (15)

    int sum = x * n1 * (n1 + 1) / 2;
    sum += y * n2 * (n2 + 1) / 2;
    sum -= z * n3 * (n3 + 1) / 2;

    printf("%d\n", sum);

    return 0;
}
\end{verbatim}

You might say taht you could do that faster with your own calculator. Well, the beauty of coding is finding mathematical background of given problem and optimizing the solution.\\

I tested all three of the above programs with number $x = 3$, $y = 5$, and $m \in \{10^3, 10^6, 10^9, 10^{12}\}$. The results are astounding (see table below). I also had to change \texttt{int}s to \texttt{long long}s.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c|c||}
\hhline{|t:=====:t|}
\textit{Algorithm} & $t(m = 10^3)$ & $t(m = 10^6)$ & $t(m=10^9)$ & $t(m=10^{12})$\\
\hhline{||=||=|=|=|=||}
Three \texttt{for} loops & $0.000002$ & $0.001528$ & $1.384336$ & $1321.064965$\\
\hhline{||-||-|-|-|-||}
One \texttt{for} loop & $0.000004$ & $0.003579$ & $3.020462$ & \\
\hhline{||-||-|-|-|-||}
Formula & $< 0.000001$ & $< 0.000001$ & $< 0.000001$ & $< 0.000001$\\
\hhline{|b:=====:b|}
\end{tabular}
\caption{Table shows running time (in seconds) of given algorithms in relation to $m$. Time was measured with help of \texttt{time.h}.}
\end{table}

Note, hot lovely are the running times of first two algorithms increasing with $m$. Thousand times greater problem, thousand times more time needed. The third algorithm obviously runs in constant time.

\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Even Fibonacci numbers}

\begin{task}
Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with $1$ and $2$, the first $10$ terms will be: \[1,2,3,5,8,13,21,34,55,89,\dotsc\]

By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.
\end{task}

Let us start with example from the task. The first ten terms of Fibonacci sequence, as the task states it, are $1$, $2$, $3$, $5$, $8$, $13$, $21$, $34$, $55$, and $89$. Even-valued terms, that do not exceed $34$ are $2$, $8$, and $34$. Therefore, the sum of the even-valued terms that do not exceed $34$ is $2+8+34 = 44$.\\

Before we get into programming, let us first discuss how terms of Fibonacci sequence are calculated. The first two terms, according to task, are $F_1 = 1$ and $F_2 = 2$. (Mind, that some sources state Fibonacci sequence as $1$, $1$, $2$, $3$, $5$, $8$, $13$,... or even $0$, $1$, $1$, $2$, $3$, $5$, $8$, $13$,...) The next term is acquired by adding previous two together. We end up with formula \[F_n = F_{n-1} + F_{n-2};~n \geq 3.\] This is recursive formula with starting values of $F_1 = 1$ and $F_2 = 2$. At this point you might think that we are going to recursion in order to calculate terms of Fibonacci sequence, but we are not. The reason is, that if we want to compute $n$-th term of Fibonacci sequence, we can use recursion. In our case, we do not know what term is the first that exceeds $4000000$, therefore we cannot use recursion. Besides, naive recursion (if we could use it) would calculate the same terms over and over again - see figure below. We could use a concept called \textit{memoization} and get rid of this problem, but as I said, we cannot use this technique in this situation.

\begin{figure}[h!]
\centering
\includegraphics[scale=1]{images/002-tree.pdf}
\caption{Recursion tree for Fibonacci sequence. Note, how terms keep repeating.}
\end{figure}

Since we cannot use recursion from $n$ to $1$, we are going to go from $1$ to $n$, summing the previous two terms together to obtain the next one. We are going to check whether that term is even and if it is, we are going to add it to sum.

\begin{verbatim}
#include<stdio.h>

int main()
{
    int x = 1;             // x represents F(n-2)
    int y = 2;             // y represents F(n-1)
    int tmp;               // tmp represents F(n)
    int sum = 0;
    while (x <= 4000000)
    {
        if (!(x % 2))      // check if x is even
            sum += x;
        tmp = x + y;       // F(n) = F(n-1) + F(n-2)
        x = y;             // F(n-1) becomes F(n-2) for next iteration
        y = tmp;           // F(n) becomes F(n-1) for next iteration
    }
    printf("%d\n", sum);

    return 0;
}
\end{verbatim}

There are probably few things I should explain. First is the condition in \texttt{if} clause. In C, as well as in some other languages, number $0$ of type \texttt{int}, beside of being a regular number, stands for $\textit{false}$ (mind, that C does NOT have boolean variables). Thus, we do not need to make comparison of \texttt{0 == 0}, but we can simply write \texttt{!0}, which evaluates to \texttt{1}.
Next thing is the way we compute terms for next iteration. First, we sum current terms and save the result in temporary variable \texttt{tmp}. Then we rewrite \texttt{x} and \texttt{y} with \texttt{y} and \texttt{tmp} respectively, thus shifting the sequence to the left. The asymptotic complexity of this algorithm is $\bigo(n)$, where $n$ is the number which terms must not exceed - in our case $n = 4000000$.\\

Remember, that I said that some sources state Fibonacci sequence as $1$, $1$, $2$, $3$, $5$, $8$, $13$, $21$, $44$,... Observing this sequence for long enough we can see, that even-valued terms are $F_{3i}$ for all $i \in \mathbb{N}$. Proving this is simple. The sum of two odd numbers is even and the sum of even and odd number is odd. Looking at the sequence we can see that this proves the statement by law of induction. Therefore, we can be sure that if we add every third term of Fibonacci sequence to our final sum, we will obtain the correct result. The following algorithm does exactly that. Each iteration we compute the next three terms of Fibonacci sequence and always add the third to final sum.

\begin{verbatim}
#include<stdio.h>

int main()
{
    int x = 1;             // x represents F(n-2) - always odd
    int y = 1;             // y represents F(n-1) - always odd
    int z = 2;             // z represents F(n) - the even term
    int sum = 0;

    while (z <= 4000000)
    {
        sum += z;          // always add z since it is always even
        x = y + z;         // F(n+1) = F(n-1) + F(n) (x becomes F(n-2))
        y = z + x;         // F(n+2) = F(n) + F(n+1) (y becomes F(n-1))
        z = x + y;         // F(n+3) = F(n+2) + F(n+2) (z becomes F(n))
    }

    printf("%d\n", sum);

    return 0;
}
\end{verbatim}

This algorithm still runs in $\bigo(n)$ but we are advancing three times faster than in previous algorithm, because we calculate three terms in each iteration of \texttt{while} loop and omitting the \texttt{if} clause, thus expecting this algorithm to run approximately three times faster does not sound stupid. The only problem is that we are also doing more summations, which will result in algorithm not being three times as fast as the previous one, but it should still be faster.\\

Now it is time for some maths. We already figured out that even-valued terms are $F_{3i}$. If we manage to rewrite the recusrion formula $F_n = F_{n-1} + F_{n-2}$ into $F_n = \alpha f_{n-3} + \beta F_{n-6}$, we will obtain a formula that uses only every third Fibonacci number and we will not have to bother with odd-valued terms. Let us try to do that.
\begin{align*}
F_n &= F_{n-1} + F_{n-2}\\
&= (F_{n-2} + F_{n-3}) + (F_{n-3} + F_{n-4})\\
&= \big((F_{n-3} + F_{n-4}) + F_{n-3}\big) + \big(F_{n-3} + (F_{n-5} + F_{n-6})\big)\\
&= 3F_{n-3} + (F_{n-4} + F_{n-5}) + F_{n-6}\\
&= 4F_{n-3} + F_{n-6}
\end{align*}
Easy peasy lemon squizy! Based on formula derived, we can implement next program.

\begin{verbatim}
#include<stdio.h>

int main()
{
    int x = 2;             // x represents F(n-6)
    int y = 8;             // y represents F(n-3)
    int tmp;               // tmp represents F(n)
    int sum = 0;

    while (x <= 4000000)
    {
        sum += x;
        tmp = 4 * y + x;   // F(n) = 4 * F(n-3) + F(n-6)
        x = y;             // F(n-3) becomes F(n-6) for next iteration
        y = tmp;           // F(n) becomes F(n-3) for next iteration
    }

    printf("%d\n", sum);

    return 0;
}
\end{verbatim}

Can you guess, how much faster, if faster, will this algorithm run compared to the second one? It still runs in $\bigo(n)$, we still advance by three terms each iteration... But hey, did you forget about the number of operations again?! I hope not. This time, we do one multiplication more than in second algorithm, but two summations less. Therefore, we expect this program to run faster than the second one, and therefore also faster than the first one. Actually, if we compare the first program to the last one, we can see, that the difference is one \texttt{if} clause more in the first one and one small multiplication more in the third one. We might actually hope that the last algorithm will run three times as fast as the first one.\\

I ran all three programs and measured the time with \texttt{time.h} but the precision of timing was too low to see much difference. The first program finished in average time of $0.000002$ seconds, and the other two finished in under $0.000001$ seconds. with $n = 9\,000\,000\,000\,000\,000\,000 \approx \texttt{LLONG\_MAX}$ ($9$ quintillion). I decided to make each algorithm run more times. The following table shows the results.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c||}
\hhline{|t:====:t|}
\textit{Algorithm} & $t(r = 10^3)$ & $t(r = 10^6)$ & $t(r = 10^9)$\\ \hhline{||=||=|=|=||}
\texttt{for} and \texttt{if} & $0.001498$ & $0.335005$ & $330.163120$\\ \hhline{||-||-|-|-||}
Naive three step & $0.000784$ & $0.183093$ & $181.040890$\\ \hhline{||-||-|-|-||}
Three step formula & $0.000215$ & $0.102709$ & $97.865098$\\ \hhline{|b:====:b|}
\end{tabular}
\caption{Table shows the running time (in seconds) of given algorithms in relation to $r$, where $r$ is the number of repetitions the core of program (the algorithm) ran. Time was measured with help of \texttt{time.h}.}
\end{table}

The results are just like we expected them to be. The \textit{three step formula} (last algorithm) is three times as fast as the \textit{\texttt{for} and \texttt{if}} (first algorithm). The \textit{naive three step} (second algorithm) is somewhere in the middle.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Largest prime factor}

\begin{task}
The prime factors of $13195$ are $5$, $7$, $13$, and $29$.\\
\\
What is the largest prime factor of the number $600851475143$?
\end{task}

There is a pretty straightforward solution to this problem that seemingly runs in $\bigo(n)$. What we can do is to loop through every number below $600851475143$ and check whether it is a prime and whether it divides $600851475143$. The most easily grasped solution with high running time. But why did I say it \textit{seemingly} runs in $\bigo(n)$. The answer is simple. We have to loop through $n$ numbers, for each check, whether it is a prime or not, that adds another $n$ in game. The total running time is then $\bigo(n^2)$. With $n = 600851475143$, my head starts burning. I just hope my computer will not. Let us implement this monstrous piece. I advise you not to try this at home. (Not sure should I do a backup first or not...)

\begin{verbatim}
#include<stdio.h>
#include<stdlib.h>
#define N 600851475143

int is_prime(long n);

int main()
{
    long largest = 1;
    for (long i = 2; i <= N; i++)
    {
        if (N % i == 0 && is_prime(i))
            largest = i;
    }

    printf("%li\n", largest);

    return 0;
}

int is_prime(long n)
{
    for (int i = 2; i < n; i++)
    {
        if (n % i == 0)
            return 0;
    }
    return 1;
}
\end{verbatim}

Mind two things: Although this is a not do at home algorithm, this might come in handy sometime in the future. Mind how we wrote \texttt{N \% i == 0} before \texttt{is\_prime(i)} in \texttt{if} clause of function \texttt{main}. Since we wrote it this way, the computer will first check whether \texttt{i} divides \texttt{N}. If not, it will not bother executing \texttt{is\_prime(i)}, since the \texttt{AND} clause will evaluate to \texttt{0} no matter the second argument. By doing that, we make sure, that we do not check every number for primality and therefore the running time of algorithm is reduced substantially. What we could also do is return $n$ when the for loop comes to it and \texttt{largest == 1}, since we did not find any other divisor of $n$, meaning $n$ is prime and we do not need to check its primality.

The second thing you should mind is that we do not need to check whether the new prime divisor is greater than the largest. You might have seen it by yourself why is so, but if not, here is the deal. The \texttt{i}, which is a possible prime divisor, increases and never decreases. Therefore, it always holds \texttt{i > largest}.\\

There are few things we could do better in implementation above and we would not even need to think hard. The first that comes to my mind is, that when we check for primality, we do not need to loop from $2$ to $n$. We only need to loop to $\sqrt{n}$. The reason is that if a number has a higher divisor then its square root, the result of division is smaller than number's square root and it is also a divisor. Therefore we can only check in range from $2$ to $\sqrt{n}$.\\

It is time for some maths. You have learned about \textit{sieve of Eratosthenes} in primary school for sure. The idea behind the sieve is to generate an array of numbers between $2$ and $n$. Then we start and cross out all multiples of $2$, except $2$, which is prime. After that, we cross out all multiples of next number that is not crossed, that is $3$. We continue by crossing the multiples of all numbers that are not crossed. By doing so, we end up with table of prime numbers. The array \texttt{eratosthenes} returns is the array of $0$s and $1$s. A $0$ on $i$-th place in array denotes $i$ is prime. For full implementation of Eratosthenes' sieve see appendix. The only problem is, that we cannot use it here, because we cannot store $600851475143$ bytes anywhere. We will have to dig further, but here is the implementation using sieve of Eratosthenes. It works for small numbers and it works fast.

\begin{verbatim}
#include<stdio.h>
#include "campovski.h"                  // needed for ERATOSTHENES
#define N 100003                        // some small number

int main()
{
    char* sieve = eratosthenes(N);      // get prime array
    int largest = 1;

    for (int i = 2; i <= N; i++)
    {
        if (!sieve[i] && N % i == 0)    // if i is prime, sieve[i] = 0
            largest = i;
    }

    printf("%d\n", largest);

    return 0;
}

\end{verbatim}

The idea for next algorithm comes directly from Eratosthenes but instead of creating a sieve, we completely divide out each of the divisor of $n$. By doing that we make sure that the next divisor will certainly be prime and therefore we do not need to check for divisor whether it is prime or not. By doing so we spare a lot of time, because every time we find a divisor $a$, we divide $n$ by $a$ as many times as we can, meaning we end up with smaller and smaller $n$, thus the calculation become faster and faster. Next implementation shows how that is done.

\begin{verbatim}
#include<stdio.h>
#include<stdlib.h>

int main()
{
    long long n = 600851475143;
    long long factor = 2;
    long long largest;

    while (n > 1)
    {
        if (!(n % factor))
        {
            largest = factor;
            do {
                n /= factor;
            } while (!(n % factor));
        }

        factor++;
    }

    printf("%lli\n", largest);

    return 0;
}
\end{verbatim}

This algorithm can be improved in two ways. It is obvious that $2$ is the only even-valued prime number. Therefore we can treat it on its own. By doing so, we can increase \texttt{factor} by 2 on each iteration and only traverse over odd numbers. The second improvement comes with realisation that every number can have at most one prime factor greater than $\sqrt{n}$. That leaves us with two options, either we found a factor lower then $\sqrt{n}$, divided it out and continued with next iteration, or we came to $\sqrt{n}$ and did not find a factor. In second option, we can stop checking since if there is no prime smaller then $n$ that would divide $n$, then the only numbers that divide $n$ are $1$ and $n$. That means $n$ is a prime and we return it. The next implementation does exactly that.

\begin{verbatim}
#include<stdio.h>
#include<math.h>

int main()
{
    long long n = 600851475143;
    long long largest = 1;

    if (!(n % 2))
    {
        largest = 2;
        do {
            n /= 2;
        } while (!(n % 2));
    }

    long long factor = 3;
    long long max_factor = (long long) sqrt(n);

    while (n > 1 && factor <= max_factor)
    {
        if (!(n % factor))
        {
            largest = factor;
            do {
                n /= factor;
            } while (!(n % factor));
            max_factor = (long long) sqrt(n);
        }

        factor += 2;
    }

    if (n == 1)
        printf("%lli\n", largest);
    else printf("%lli\n", n);

    return 0;
}
\end{verbatim}

I measured the running time of algorithms and the results were quite shocking. The first algorithm needed..., while the second and third one were surprisingly fast. The next table shows running times of algorithms with respect to $n$. Each program was run 1000 times to ensure smaller relative error, except for \textit{one step \texttt{factor}} on $n = 100123456789$.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c|c|c||}
\hhline{|t:======:t|}
\textit{Algorithm} & $t(n = 10^3)$ & $t(n = 10^6)$ & $t(n = 10^5 + 3)$ & $t(n = n_1)$ & $t(n= n_2)$\\ \hhline{||=||=|=|=|=|=||}
Monster & $0.012390$ & $8.448938$ & $1.754996$ &  &\\ \hhline{||-||-|-|-|-|-||}
Sieve & $0.009984$ & $3.659880$ & $0.454618$ & $-$ & $-$ \\ \hhline{||-||-|-|-|-|-||}
One step \texttt{factor} & $0.000478$ & $0.000634$ & $0.214148$ & $0.130969$ & $\sim116$ days\\ \hhline{||-||-|-|-|-|-||}
Two step + \texttt{sqrt} & $0.006801$ & $0.007498$ & $0.001555$ & $0.023457$ & $1.362939$ \\ \hhline{|b:======:b|}
\end{tabular}
\caption{Table shows the running time (in seconds) of given algorithms with respect to $n$, where $n$ is the number whose greatest prime factor we are trying to calculate. $n_1 = 600851475143$ and $n_2 = 100123456789$ ($n_2$ is prime). Time was measured with help of \texttt{time.h}.}
\end{table}

As you can see, running times are quite strange. For small numbers that are not prime the \texttt{one step \texttt{factor}} is faster than \texttt{two step + \texttt{sqrt}}, because square root takes too much time to evaluate. On the other hand, if $n$ happens to be prime, the \texttt{one step \texttt{factor}} takes a lot of time, because it has to search through all numbers below $n$ before realizing it could actually return an answer. Therefore, one can clearly see that \textit{two step + \texttt{sqrt}} is a way to go. The \textit{sieve} is quite fast, but not appropriate for this problem, since has such a specific implementation that it simply cannot compete. We included it just for broadening mind. Mind also, that we computed sieve only once and ran the rest of the program 1000 times. Places with $-$ denote we could not allocate such a huge sieve and empty space means we did not run the program because it would take too long to execute.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Largest palindrome product}

\begin{task}
A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is $9009 = 91 \cdot 99$.\\
\\
Find the largest palindrome made from the product of two 3-digit numbers.
\end{task}

This task might look a bit awkward at first. The first problem we encounter is how to smartly generate possible products and the second is how to check if the number is palindrome.

We tackle the first problem in this way. For generating products we use two \texttt{for} loops to generate each number in product and then multiply them together. Since we are not stupid, we know that the product $x \cdot y$ is the same as $y \cdot x$. That means, that not both numbers $x$ and $y$ must start their loop at $100$. Instead, the inner loop can start where $x$ has left.

The second problem is a bit trickier. If we were to check if string is palindrome, we could simply iterate over it and check whether the $i$-th character from start is the same as $i$-th character from the end. In case of numbers, we cannot directly iterate over them, but have to first convert them into \texttt{char} arrays. We use function \texttt{int\_to\_arr}, which is defined in \texttt{campovski.h}. Now we can do what we would have done with strings. The other way is to use an algorithm for reversing the number and then checking if original number is the same as reversed, but we are not going to do that. Some other time maybe. If you are eager to learn, you can google how to reverse a number in C and you will get plenty of hits.\\

Now we are good to go implement the program.

\begin{verbatim}
#include<stdio.h>
#include<math.h>
#include<stdlib.h>
#include "campovski.h"          // needed for INT_TO_ARR

char is_palindrome(int n);

int main()
{
    int largest = 10000;        // 10000 = 100 * 100

    for (int x = 100; x <= 999; x++)
    {
        for (int y = x; y <= 999; y++)  // evade double checking
        {
            if (is_palindrome(x*y) && x * y >= largest)
                largest = x * y;
        }
    }

    printf("%d\n", largest);

    return 0;
}

char is_palindrome(int n)
{
    char* n_arr = int_to_arr(n);
    int l = floor(log10(abs(n))) + 1;
    for (int i = 0; i < (int) l/2; i++)
    {
        if (n_arr[i] != n_arr[l-i-1])
            return 0;
    }

    return 1;
}
\end{verbatim}

Although this program seems alright, there are still some improvements to make. First, let us assume that the largest palindrome product will be a product of two large 3-digit numbers. Therefore, we can start with $x = 999$, decrement $x$ on each iteration and break the inner \texttt{for} loop when $y$ is so small that $x * y \leq \textrm{largest}$. By doing that we hope to spend less time for finding the largest palindrome product.

The next improvemt is not always possible, therefore we will still consider the above program as BEST! This improvement comes after doing some maths. We realize that $111111 = 143\cdot 777$ is palindromic and $999 \cdot 999 < 1000000$. That means that the largest palindrome that is a product of two 3-digit numbers has 6 digits. Let $p$ be a palindromic number with 6 digits. Then we can write
\begin{align*}
p &= 100000a + 10000b + 1000c + 100c + 10b+ a\\
&= 100001a + 10010b + 1100c\\
&= 11(9091a + 910b + 100c)
\end{align*}

We figured out, that $11$ must divide the 6-digit palindrome. Since $11$ is a prime, a theorem from number theory and abstract algebra tells us, that at least one of $x$ and $y$ must be divisible by $11$. Therefore, if $x$ is not divisible by $11$, $y$ has to be. Using the information derived and described above, we can implement next program.

\begin{verbatim}
#include<stdio.h>
#include<stdlib.h>
#include<math.h>
#include "campovski.h"         // needed for INT_TO_ARR

char is_palindrome(int n);

int main()
{
    int largest = 111111;       // 111111 = 143 * 777
    int y, dy;

    for (int x = 999; x >= 100; x--)
    {
        if (x % 11 == 0)       // 11 divides x, doesn't need to divide y
        {
            y = 999;           // we must check all y's
            dy = 1;            // and decrement by 1
        }
        else                   // 11 must divide y
        {
            y = 990;           // largest 3 digit number that y % 11 = 0
            dy = 11;           // decrement by 11
        }

        for (; y >= x; y -= dy)
        {
            if (x * y <= largest)    // don't even need to check if
                break;               // the product is palindrome
            if (is_palindrome(x*y))
                largest = x * y;
           // y -= dy;
        }
    }

    printf("%d\n", largest);

    return 0;
}

char is_palindrome(int n)
{
    char* n_arr = int_to_arr(n);
    int l = floor(log10(abs(n))) + 1;
    for (int i = 0; i < (int) l/2; i++)
    {
        if (n_arr[i] != n_arr[l-i-1])
            return 0;
    }

    return 1;
}
\end{verbatim}

I tested the program above for finding the largest palindrome product of two $n$-digit numbers, where $n \in \{3,4,5\}$. The table above shows running times.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c||}
\hhline{|t:===:t|}
\textit{Algorithm} & $t(n=3)$ & $t(n=4)$\\ \hhline{||=||=|=||}
Simple & $0.140$ & $5.852$ \\ \hhline{||-||-|-||}
Advanced & $0.086 / 1000$ & \\ \hhline{|b:===:b|}
\end{tabular}
\caption{Running time (in seconds) of two implementations for finding largest palindromic product of two $n$-digit numbers. The time was measured by \texttt{time} command in bash shell.}
\end{table}

As we assumed the second algorithm was much faster, but it could not be used in case of $n = 4$, since the product of two 4-digit numbers can either be a 7-digit or 8-digit number. We could of course break that down too and maybe find something interesting but we are not going to. You can do that for your homework.\\

From now on, we will be measuring running time with Unix \texttt{time} command rather than with \texttt{time.h}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Smallest multiple}

\begin{task}
$2520$ is the smallest number that can be divided by each of the numbers from $1$ to $10$ without any remainder.\\
\\
What is the smallest positive number that is evenly divisible by all of the numbers from $1$ to $20$?
\end{task}

This is the first task that actually requires us to know some maths. There is no naive implementation that would just pop up from nowhere. What this task actually requires is to solve the next system of linear congruences:
\begin{align*}
n &\equiv 0~(\mod 1)\\
n &\equiv 0~(\mod 2)\\
n &\equiv 0~(\mod 3)\\
&~~\vdots\\
n &\equiv 0~(\mod 20)
\end{align*}

The only thing we are lucky here is, that $n$ has to be congruent to $0$ by all moduli. The reason for that is, that if $x$ was to be congruent to some other numbers, we would have to use \textit{Chinese remainder theorem}. And since Chinese remainder theorem only works when moduli are pairwise coprime, meaning that $\gcd$ of each pair of moduli is $1$, we would have to disassemble the pairs of congruences where $\gcd$ of moduli would not be $1$ into more congruences whose moduli would be pairwise coprime. The reason why we do not need to use Chinese remainder theorem is fairly obvious. Since $n$ has to be congruent to $0$, that means that $n$ has to be a multiple of moduli. Because we are looking for the smallest number that satisfies these criteria, we are actually looking for least common multiple of moduli. Once we know that, the implementation is really easy, just like "Hello world!".\\

Let me just mention few things on how to implement the algorithm. First, since $1$ divides every number, we can omit the first congruence. Second, the following is true and can be easily proved: \[\lcm(x,y,z) = \lcm(\lcm(x,y), z) = \lcm(x, \lcm(y,z)).\] We can use that to compute $\lcm$ on-the-fly, meaning we can compute the $\lcm$ of previous $\lcm$ and next modulo and the result will be the $\lcm$ of all moduli.

\begin{verbatim}
#include<stdio.h>
#include "campovski.h"                // used for LCM

int main()
{
    int r = 20;                       // the highest modulo is 20
    long long n = 2;                  // 1 divides everything

    for (int i = 3; i < r; i++)
        n = lcm(n, i);                // compute lcm of moduli

    printf("%lli\n", n);

    return 0;
}
\end{verbatim}

This algorithm is faster than light. I ran it with different $r$s and the following table shows the results.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c||}
\hhline{|t:====:t|}
\textit{Algorithm} & $t(r = 20)$ & $t(r = 30)$ & $t(r = 40)$ \\ \hhline{||=||=|=|=||}
\texttt{lcm} & $0.676 \times 10^{-6}$ & $1.112 \times 10^{-6}$ & $1.508 \times 10^{-6}$ \\ \hhline{|b:====:b|}
\end{tabular}
\caption{Running time (in seconds) of the algorithm, measured by Unix \texttt{time} bash command.}
\end{table}

The limit on $r$ is $r = 43$. With $r >= 44$ we cannot store $n$ in \texttt{unsigned long long}. Note, that \textit{Project Euler Problem 5 overview} contains different algorithm, but that algorithm is highly impractical. The reason is, that the algorithm assumes it is given an array of prime numbers, which takes time to calculate. Furthermore, we cannot use our \texttt{eratosthenes} from \texttt{campovski.h} because that array contains a lot of zeros and uses a lot of important memory. We would need to implement another function that can compute large primes in a moment and return the array prime numbers. We might do that in the future, if we come across some high primality problem.

The algorithm provided by Project Euler is suitable for extremely large numbers, which you would need to store in \texttt{char} array but we are not going to go into that at this moment. If you want you can check the algorithm after you have succesfully solved the problem and checked the solution.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Sum square difference}

\begin{task}
The sum of the squares of the first ten natural numbers is \[1^2 + 2^2 + \dotsm + 10^2 = 385.\] The square of the sum of the first ten natural numbers is \[(1+2+ \dotsm + 10) ^2 + 55^2 = 3025.\] Hence the difference between the sum of the squares of the first ten natural numbers and the square of the sum is $3025 - 385 = 2640$.\\
\\
Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.
\end{task}

This task is really simple. Even a child in primary school can solve it. All you have to do is loop from numbers between $1$ and $100$, calculate their squares and add that number to final sum At the same time we add each number to \texttt{tmp} which is a normal sum of integers between $1$ and $100$. At the end we subtract the normal sum squared from the sum of squares and print a number that is greater than $0$. BAM, problem 6 done!

\begin{verbatim}
#include<stdio.h>

int main()
{
    int sum = 0;                            // sum of squares
    int tmp = 0;                            // sum of natural
    for (int i = 1; i <= 100; i++)
    {
        sum += i * i;
        tmp += i;
    }
    sum -= tmp * tmp;                       // calculate difference

    printf("%d\n", sum >= 0 ? sum : -sum);  // print nonnegative sum

    return 0;
}
\end{verbatim}

But wait, there must be something behind this problem. It cannot be that easy and the straightforward solution shall not return the answer instantly... Actually there is a better solution. The implementation above runs in $\bigo(n)$. But since this is a simple mathematical problem, there exists a solution that runs in $\bigo(1)$. Let me walk you to better solution.\\

In \textit{problem 1} we already discussed the concept of \textit{arithmetic series}. The sum of natural numbers between $1$ and $n$ is nothing else but an arithmetic series with $a_1 = 1$ and $d = 1$. So we derive the next formula: \[\sum_{i=1}^n i = \frac{n(n+1)}{2}.\] For the sum of squares, there also exists a formula, which we are not going to derive here but the derivation can easily be found online. The formula is \[\sum_{i=1}^n i^2 = \frac{n(n+1)(2n+1)}{6}.\] Once known it can be quite easily proved by the law of induction. Knowing this two formulas, we get a solution that runs in $\bigo(1)$.

\begin{verbatim}
#include<stdio.h>

int main()
{
    int n = 100;
    int sum_squares = n * (n+1) * (2*n+1) / 6;  // sum of squares
    int sum = n * (n+1) / 2;                    // sum of natural
    int diff = sum_squares - sum * sum;         // difference

    printf("%d\n", diff >= 0 ? diff : -diff);   // print nonnegative diff

    return 0;
}
\end{verbatim}

Mind, that we did not need the extra variable \texttt{diff}. We only did that to make the code more readable. Another thing we could do in some other situation is, that we could divide by 6 in \texttt{sum\_squares} at some other position and by that evade the number from growing over the result of that whole line operation, thus being able to calculate higher sums since we would not overflow. But in this case we cannot do that, since we do not know whether any of numbers $n$, $n+1$, and $2n+1$ is divisible by $6$. I am just saying this so you know that the order of operation matters and can sometimes get really useful. For example, if we knew that $n$ is even, we could write \texttt{int sum = n / 2 * (n+1);} and may be able to calculate sum for higher $n$, because we would first halve the $n$ and then multiply it by $n+1$. Similarly, if $n$ was even, we could write \texttt{int sum = (n+1) / 2 * n;}.

I ran the above programs and measured running time with Unix \texttt{time}. The results are in the following table.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c||}
\hhline{|t:====:t|}
\textit{Algorithm} & $t(n=10^2)$ & $t(n = 10^3)$ & $t(n=10^4)$\\ \hhline{||-||---||}
Brute force & $0.384 \times 10^{-6}$ & $2.964 \times 10^{-6}$ & $29.340 \times 10^{-6}$\\ \hhline{||-||---||}
Formulas & $0.016 \times 10^{-6}$ & $0.012 \times 10^{-6}$ & $0.024 \times 10^{-6}$ \\ \hhline{|b:====:b|}
\end{tabular}
\caption{Running time (in seconds) of programs. They were run $10^6$-times, that is why the time is then divided by $10^6$. $n$ is the upper range of the sum we are calculating (from $1$ to $n$).}
\end{table}

One can clearly see, that first program is nicely linear in $n$. The second runs in constant time... Actually it runs in some respect to the length of $n$, since the longer the number, the more time it needs to multiply. But that is still $\bigo(1)$ with respect to $n$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{10001st prime}

\begin{task}
By listing the first six prime numbers: $2$, $3$, $5$, $7$, $11$, and $13$, we can see that the 6th prime is $13$.\\
\\
What is the 10001st prime number?
\end{task}

We have already discussed a great deal about prime numbers in \textit{problem 3}. Therefore, we are not going to go in any details here. We are going to use \texttt{eratosthenes} from \texttt{campovski.h} which starts with array of $0$ and then places $1$ on $i$-th place if $i$ is NOT a prime number.

After acquiring the eratosthenes sieve, we loop through it and every time we find $0$ we add $1$ to count of prime numbers. When count reaches $10001$, we stop and print the index. If the returned index is \texttt{i = len(sieve) - 1}, the sieve made is not big enough, because we now that \texttt{(len(sieve) - 1} is not a prime number, since it is even.

\begin{verbatim}
#include<stdio.h>
#include "campovski.h"

int main()
{
    char* sieve = eratosthenes(1000001);  // generate huge sieve
    int count = 0;                        // count of primes

    int i;                                // we will need i outside the loop
    for (i = 2; i < 1000001; i++)
    {
        if (!sieve[i])                    // if sieve[i] = 0 then i is prime
            count++;                      // we found another prime
        if (count == 10001)               // we found 10001st prime
            break;
    }

    printf("%d\n", i);

    return 0;
}
\end{verbatim}

Let us take a look at the running times of above implementation with different $n$s, where $n$ means that we want to find $n$-th prime.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c|c||}
\hhline{|t:=====:t|}
\textit{Algorithm} & $t(n=10001)$ & $t(n = 10^5)$ & $t(n=10^6)$ & $t(n=10^7)$\\ \hhline{||-||----||}
Sieve walk & $0.048$ & $0.236$ & $2.700$ & $29.080$ \\ \hhline{|b:=====:b|}
\end{tabular}
\caption{Running time (in seconds) of programs. We used a sieve of size $10^6 + 1$ for $n=10001$, $10^7 + 1$ for $n = 10^5$, $10^8 + 1$ for $n = 10^6$. and $10^9 + 1$ for $n = 10^7$.}
\end{table}

The algorithm is lovely linear. We increase $n$ ten times, the running time increases ten times.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\setcounter{section}{24}
\section{1000-digit Fibonacci number}

\begin{task}
The Fibonacci sequence is defined by the recurrence relation:
\[F_n = F_{n-1} + F_{n-2},\textrm{ where $F_1 = 1$ and $F_2 = 1$}.\]
Hence the first $12$ terms will be:
\begin{align*}
F_1 &= 1\\
F_2 &= 1\\
F_3 &= 2\\
F_4 &= 3\\
F_5 &= 5\\
F_6 &= 6\\
F_7 &= 13\\
F_8 &= 21\\
F_9 &= 34\\
F_{10} &= 55\\
F_{11} &= 89\\
F_{12} &= 144
\end{align*}
The $12$th term, $F_{12}$, is the first term to contain three digits.\\
\\
What is the index of the first term in the Fibonacci sequence to contain $1000$ digits?
\end{task}

This is one of the problems that we are particularly not lucky to be programming in C. We cannot store the $1000$ digit number in any of number types. We will have to use array of \texttt{int}s (we could use \texttt{char}s, but it is really not necessary, since we do not need to spare with memory). The only possible solution to this problem is brute force, since there are no known rules for how Fibonacci numbers' lengths increase. In languages like Python we could write the solution to this problem very easily because we do not need to worry about storing such huge numbers. The solution in Python is this:

\begin{verbatim}
x = 1
y = 1
count = 2
while len(str(y)) != 1000:
    x, y = y, x + y
    count += 1
print count
\end{verbatim}

Simple as f***. Now, the solution in C is not much harder, we just have to be careful with arrays. Since we will not care about memory, we can allocate three $1000$ \texttt{int}s long arrays, one for $F_{n-1}$, one for $F_n$, and one for $F_{n+1}$. Then we loop over all the digits in $F_{n-1}$ and $F_n$ and calculate each digit of $F_{n+1}$ by piece. The only problem is, that we must take care of the situation where \texttt{F(n-1)[i] + F(n)[i] >= 10}. In this case, we have to remember to increment the next digit. Fairly simple, but quite painful. The next implementation does exactly that.

\begin{verbatim}
#include<stdio.h>
#include<stdlib.h>

int main()
{
    int* x = (int*) malloc(1000*sizeof(int));
    int* y = (int*) malloc(1000*sizeof(int));
    int* tmp = (int*) malloc(1000*sizeof(int));
    int inc = 0;                                  // increment on next step

    x[0] = 1;                                     // F(1) = 1
    y[0] = 1;                                     // F(2) = 1
    int count = 2;                                // we already know 2 numbers

    while (!y[999])                               // loop while 1000th digit is 0
    {
        for (int i = 0; i < 1000; i++)
        {
            int s = x[i] + y[i] + inc;            // sum of i-th digit + increment
            tmp[i] = s % 10;                      // digit of F(n+1)
            x[i] = y[i];                          // F(n) becomes F(n-1)
            y[i] = tmp[i];                        // F(n+1) becomes F(n)
            inc = (s >= 10) ? 1 : 0;              // if s>=10, we must inc on next iter
        }
        count++;
    }

    printf("%d\n", count);

    return 0;
}
\end{verbatim}

Now if we would want to spare with memory, we would have two options. The first thing we could do is change all \texttt{int} values to \texttt{short}s. That would spare a lot of the memory we used for variables. But furthermore, since we know that we will only be using really small numbers at a time (less then $9+9+1 = 19$), we could allocate a \texttt{char} array and spare even more. Many people think that \texttt{char} stands for \textit{character}. Well, it actually does, but the \texttt{char} type can normally store integral values, just as long as they do not exceed $2^8$. Instead of $4$ bytes (can store up to $2^{32}$) for an \texttt{int}, we could only use $1$ byte (can store up to $2^8$) for a \texttt{char}. The implementation is the same.

\begin{verbatim}
#include<stdio.h>
#include<stdlib.h>

int main()
{
    char* x = (char*) malloc(1000*sizeof(char));
    char* y = (char*) malloc(1000*sizeof(char));
    char* tmp = (char*) malloc(1000*sizeof(char));
    char inc = 0;

    x[0] = 1;
    y[0] = 1;
    int count = 2;

    while (!y[999])
    {
        for (int i = 0; i < 1000; i++)
        {
            char s = x[i] + y[i] + inc;
            tmp[i] = s % 10;
            x[i] = y[i];
            y[i] = tmp[i];
            inc = s >= 10 ? 1 : 0;
        }
        count++;
    }

    printf("%d\n", count);

    return 0;
}
\end{verbatim}

Let me take another note on \texttt{char}s. One \texttt{char} is 1 byte, 1 byte is 8 bits and 1 bit can store two values, \texttt{0} and \texttt{1}. Since we can store 8 bits in a byte, that means, we can store $2^8$ values in 1 byte. Therefore, if we only have less than $2^8 = 256$ different signs to store, we can use a \texttt{char}. We use so called ASCII table, which is nothing else but a bijection between symbols like characters, numbers, punctuation, etc. and numbers between $0$ and $255$. For example, a character \texttt{"A"} has an ASCII value of \texttt{65}. Therefore, if we write something like \texttt{char c = "A";}, what actually gets stored in \texttt{c} is the number \texttt{65}, which represents \texttt{"A"}. That means, that if we store \texttt{42} in \texttt{char}, the computer might understand that as a character with ASCII value of 42, that is \texttt{*}, but it can serve as a pure number for our needs, see what I mean? For full ASCII table go ahead and google it, there is really no point in creating one here.

Also, we will not be making any time comparison here, since we only have one algorithm, the running time of which is clearly $\bigo(n)$, where $n$ is the length of Fibonacci number, in our case $n = 1000$.\\

I measured the running times of above programs with Unix \texttt{time} bash command. The following table contains results.

\begin{table}[h!]
\centering
\begin{tabular}{||l||c|c|c||}
\hhline{|t:====:t|}
\textit{Algorithm} & $t(n=10^3)$ & $t(n=10^4)$ & $t(n=10^5)$\\ \hhline{||=||=|=|=||}
\texttt{int*} & $0.076$ & $4.004$ & $398.320$ \\ \hhline{||-||-|-|-||}
\texttt{char*} & $0.040$ & $4.064$ & $402.524$ \\ \hhline{|b:====:b|}
\end{tabular}
\caption{Running times of the implementations with respect to $n$, where $n$ is the desired length of Fibonacci number.}
\end{table}

\end{document}