![Callysto.ca Banner](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-top.jpg?raw=true)

# Fibonacci Numbers

The **Fibonacci** sequence is the set of numbers that starts out like this:

$$0,1,1,2,3,5,8,13,\ldots.$$

It's easy to recognize the pattern here. Each number is the sum of the previous two numbers in the sequence. Except, of course, the first two number, 0 and 1, which we put in there to get things started.

This sequence, or pattern of numbers, goes on forever.

These numbers are most commonly known as the **Fibonacci numbers**, after the Italian mathematician **L. Fibonacci** (c. 1200 C.E.). However, these numbers were actually first described hundreds of years before, by Indian mathematicians. The first such mathematician for whom we have written records was **Virahanka** (c. 700 C.E.).

It is difficult to overcome the usage of a name adopted hundreds of years ago, and so in what follows we will refer to the Virahanka-Fibonacci numbers as the "Fibonacci numbers."

**Exercise 1:** Check out the following article about the "so-called Fibonacci" numbers in ancient India: https://www.sciencedirect.com/science/article/pii/0315086085900217

For an excellent exposition about the motivation, poetry, and linguistics of these early mathematicians, and to learn about some fascinating properties of these numbers, check out the following video:

from IPython.display import YouTubeVideo
YouTubeVideo('LP253wHIoO8', start=2633)

It's convenient to label these numbers, so we write $F_0 = 0$, $F_1 = 1$ $F_2 = 2$ and so on. The list of numbers is thus  defined **recursively**  by the formula
$$ \qquad$$
$$ F_n = F_{n-1} + F_{n-2}.$$

We can check the few numbers in the Fibonacci sequence are obtained by that formula, by computing:

$$\begin{eqnarray*}
F_{2} &=&F_{1}+F_{0}=1+0=1 \\
F_{3} &=&F_{2}+F_{1}=1+1=2 \\
F_{4} &=&F_{3}+F_{2}=2+1=3 \\
F_{5} &=&F_{4}+F_{3}=3+2=5 \\
F_{6} &=&F_{5}+F_{4}=5+3=8 \\
F_{7} &=&F_{6}+F_{5}=8+5=13 \\
&&\vdots
\end{eqnarray*}
$$

Here is a list of the first 40 Fibonacci numbers:

$$
\begin{array}{rrrrrrrrrrrrrrr}
0 &  & 1 &  & 1 &  & 2 &  & 3 &  & 5 &  & 8 &  & 13 \\
&  &  &  &  &  &  &  &  &  &  &  &  &  &  \\
21 &  & 34 &  & 55 &  & 89 &  & 144 &  & 233 &  & 377 &  & 610 \\
&  &  &  &  &  &  &  &  &  &  &  &  &  &  \\
987 &  & 1597 &  & 2584 &  & 4181 &  & 6765 &  & 10946 &  & 17711 &  & 28657
\\
&  &  &  &  &  &  &  &  &  &  &  &  &  &  \\
46368 &  & 75025 &  & 121393 &  & 196418 &  & 317811 &  & 614229 &  & 832040
&  & 1346269 \\
&  &  &  &  &  &  &  &  &  &  &  &  &  &  \\
2178309 &  & 3524578 &  & 5702887 &  & 9227465 &  & 14930352 &  & 24157817 &
& 39088169 &  & 63245986%
\end{array}$$

**Exercise 2:** Write a code that computes the first N Fibonacci numbers, saves them into an array, and displays them on the screen.

**WAIT** -- before you read the next cell, try to do Exercise 2!

N = 20   # Set the size of the list we will compute

F=[0,1]  # The first two numbers in the list
for i in range(2, N):
    F.append(F[i-1]+F[i-2])  # append the next item on the list

print('First',N,'Fibonacci numbers:',F)

**For fun,** we can make a little widget to control how many numbers to print out.

from ipywidgets import interact

def printFib(N=10):
    F=[0,1]  # The first two numbers in the list
    for i in range(2, N):
        F.append(F[i-1]+F[i-2])  # append the next item on the list
    print(F)

interact(printFib, N=(10,100,10));

By moving the slider above, print out the first 100 Fibonacci numbers

As we can see, this sequence grows pretty fast. The Fibonacci numbers seem
to have one more digit after about every five terms in the sequence.

## How fast does it grow?

One of the ways to study the growth of a sequence is to look at ratios between consecutive terms. We  look at ratios of pairs of numbers in the Fibonacci sequence.

The first few values are
\begin{eqnarray}
F_2/F_1 &=& 1 \\
F_3/F_2 &=& 2/1 = 2 \\
F_4/F_3 &=& 3/2 = 1.5 \\
F_5/F_4 &=& 5/3 = 1.666... \\
F_6/F_5  &=& 8/5 = 1.6 \\
F_7/F_6  &=& 13/8 = 1.625
\end{eqnarray}

So the ratios are levelling out somewhere around 1.6. We observe that $1.6^5 \approx 10$, which is why after every five terms in the Fibonacci sequence, we get another digit. This tells us we have roughly  **exponential growth,** where $F_n$ grows about as quickly as the exponential function $(1.6)^n$.

We can check this computation in Python. We use $ ** $ to take a power, as in the following cell.

(1.6)**5

## The Golden Ratio

We can print out a bunch of these ratios, and plot them, just to see that they do. The easiest way to do this is with a bit of Python code. Perhaps you can try this yourself.

**Exercise 3** Write some code that computes the first N ratios $F_{n+1}/F_n$, save them it into an array, and displays them on the screen.

**WAIT!** Don't read any further until you try the exercises.

%matplotlib inline
from matplotlib.pyplot import *

N = 20
F = [0,1]
R = []
for i in range(2, N):
    F.append(F[i-1]+F[i-2])  # append the next item on the list
    R.append(F[i]/F[i-1])

figure(figsize=(10,6));
plot(R,'o')
title('The first '+str(N-2)+' Ratios $F_{n+1}/F_n$')
xlabel('$n$')
ylabel('$Ratio$');

print('The first', N-2, 'ratios are:',R)

We see the numbers are levelling out at the value 1.6108034...  This number may be familiar to you. It is called the **Golden Ratio.**

We can compute the exact value by observing the ratios satisfy a nice algebraic equation:
$$
\frac{F_{n+2}}{F_{n+1}}=\frac{F_{n+1}+F_{n}}{F_{n+1}}=1+\frac{F_{n}}{F_{n+1}}=1+\frac{1}{\frac{F_{n+1}}{F_{n}}},
$$
or more simply
$$\frac{F_{n+2}}{F_{n+1}}=1+\frac{1}{\frac{F_{n+1}}{F_{n}}}.$$

As $n$ gets larger and larger, the ratios $F_{n+2}/F_{n+1}$ and $F_{n+1}/F_{n}$ tend toward a final value, say $x$. This value must then solve the equation
$$x=1+\frac{1}{x}.$$

We rewrite this as a quadratic equation
$$x^2=x+1$$
which we solve from the quadratic formula
$$ x= \frac{1 \pm \sqrt{1+4}}{2} = \frac{1 \pm \sqrt{5}}{2}.$$
It is the positive solution $x= \frac{1 + \sqrt{5}}{2} = 1.6108034...$ which is called the Golden Ratio.

The **Golden ratio** comes up in art, geometry, and  Greek mythology as a perfect ratio that is pleasing to the eye (and to the gods).

For instance, the rectangle shown below is said to have the dimensions of the Golden ratio, because the big rectangle has the same shape as the smaller rectangle inside. Mathematically, we have the ratios of lengths
$$ \frac{a+b}{a} = \frac{a}{b}.$$

![Golden ratio rectangle](images/Golden2.png)

Writing $x = \frac{a}{b}$, the above equation simplifies to
$$ 1 + \frac{1}{x} = x,$$
which is the same quadratic equation we saw for the limit of ratios of Fibonacci numbers.

For more information about the Golden ratio see
https://en.wikipedia.org/wiki/Golden_ratio

## A Formula for the Fibonacci Sequence $F_n$

Let's give the Golden ratio a special name. In honour of the ancient Greeks who used it so much, we call it `phi:'
$$ \varphi = \frac{1 + \sqrt{5}}{2}. $$
We'll call the other quadratic root 'psi:'
$$ \psi = \frac{1 - \sqrt{5}}{2}. $$
This number $\psi$ is called the **conjugate** of $\varphi$ because it looks the same, except for the negative sign in front of the $\sqrt{5}$.

Here's something **amazing.** It turns out that we have a remarkable formula for the Fibonnaci numbers, in terms of these two Greek numbers. The formula says
$$F_n = \frac{\varphi^n - \psi^n}{\sqrt{5}}.$$

#### Wow!

Seems amazing. And it is handy because now we can compute, say, the thousandth term in the sequence, $F_{1000}$ directly, without having to compute all the other terms that come before.

But, whenever someone gives you a formula, you should check it!

**Exercise 4:** Write a piece of code to show that the formula above, with $\varphi,\psi$ does produce, say, the first 20 Fibonnaci numbers.

**WAIT!** Don't go on until you try writing a program yourself, to compute the Fibonacci numbers using only powers of $\varphi, \psi$.

## SOLUTION (don't peak!)

from numpy import *  ## We need this to define square roots
phi = (1 + sqrt(5))/2
psi = (1 - sqrt(5))/2
for n in range(20):
    print( (phi**n - psi**n)/sqrt(5) )

Looking at that computer output, it does seem to give Fibonacci numbers, with a bit of numerical error.

## Checking the Math

Doing math, though, we like exact answers and we want to know why. So WHY does this formula $(\phi^n - \psi^n)/\sqrt{5}$ give Fibonacci numbers?

Well, we can check, step by step.

For $n=0$, the formula gives
$$\frac{\varphi^0 - \psi^0}{\sqrt{5}} = \frac{1-1}{\sqrt{5}} = 0,$$ which is $F[0]$, the first Fibonacci number.

For $n=1$, the formula gives
$$\frac{\varphi^1 - \psi^1}{\sqrt{5}} =
\frac{\frac{1 + \sqrt{5}}{2} - \frac{1 -\sqrt{5}}{2} }{\sqrt{5}} = \frac{\sqrt{5}}{\sqrt{5}} = 1,$$ which is $F[1]$, the next Fibonacci number.

For $n=2$, it looks harder because we get the squares $\varphi^2, \psi^2$ in the formula. But then remember that both $\varphi$ and $\psi$ solve the quadratic $x^2 = x+1$, so we know $\varphi^2 = \phi +1$ and $\psi^2 = \psi +1$. So we can write
$$\frac{\phi^2 - \psi^2}{\sqrt{5}} = \frac{\phi + 1 - \psi -1}{\sqrt{5}} = \frac{\phi - \psi }{\sqrt{5}} = 1,$$
since we already calculated this in the $n=1$ step. So this really is $F[2]=1$.

For $n=3,4,5,\ldots$ again it might seem like it will be hard because of the higher powers. But multiplying the formulas $\varphi^2 = \varphi +1$ and $\psi^2 = \psi +1$ by powers of $\phi$ and $\psi$, we get

$$\begin{eqnarray*}
\varphi^2 &=& \varphi +1,\quad \varphi^3 = \varphi^2+\varphi
,\quad \varphi^4=\varphi^3+\varphi^2,\qquad \dots \qquad %
\varphi^{n+2}=\varphi^{n+1}+{\varphi}^n,\quad \text{and} \\
\psi^2 &=&\psi +1,\quad \psi^3=\psi^2+\psi ,\quad \psi^4=\psi^3+\psi^2,\qquad
\dots \qquad \psi^{n+2}=\psi^{n+1}+\psi^n.
\end{eqnarray*}$$

So, assuming we know the generating formula already for $n$ and $n+1$ we can write the next term as
$$\frac{\varphi^{n+2} - \psi^{n+2}}{\sqrt{5}} = \frac{\varphi^{n+1} +\varphi^n - \psi^{n+1} - \psi^n}{\sqrt{5}}
= \frac{\varphi^{n+1} - \psi^{n+1}}{\sqrt{5}} + \frac{\varphi^{n} - \psi^{n}}{\sqrt{5}} = F[n+1] + F[n] = F[n+2].$$

So we do get $\frac{\varphi^{n+2} - \psi^{n+2}}{\sqrt{5}} = F[n+2]$, and the formula holds for all numbers n.

This method of verifying the formula for all n, based on previous values of n, is an example of **mathematical induction.**

## Why did this work?

Well, from the Golden ratio, we have the formula $\varphi^2 = \varphi + 1$, which then gives the formula $\varphi^{n+2} = \varphi^{n+1} + \varphi^n$.  This looks a lot like the Fibonacci formula $$F[n+2] = F[n+1] + F[n].$$ Same powers of $\psi$.

If we take ANY linear combination of powers of $\varphi, \psi$, such as
$$f(n) = 3\varphi^n + 4\psi^n,$$
we will get a sequence that behaves like the Fibonacci sequence, with $f(n+2) = f(n+1) + f(n).$ To get the 'right' Fibonacci sequence, we just have to replace the 3 and 4 with the right coefficients.

## From sequences to functions

Wouldn't it be fun to extend Fibonacci numbers to a function, defined for all numbers $x$?

The problems is the function
$$F[x] = \frac{\varphi^x - \psi^x}{\sqrt{5}}$$
is not defined for values of $x$ other than integers.

The issue is the term $\psi^{x}=\left( \frac{1-\sqrt{5}}{2}\right) ^{x}$, which is the power of a negative number.
We don't really know how to define that. For instance, what is the square root of a negative number?

To
overcome this technical difficulty, we write

$$\psi ^{x}=\left( -\left( -\psi \right) \right) ^{x}=\left( -\left( \frac{%
\sqrt{5}-1}{2}\right) \right) ^{x}=\left( -1\right) ^{x}\left( \frac{\sqrt{5}%
-1}{2}\right) ^{x}. $$

Now the factor $\left( \frac{\sqrt{5}-1}{2} \right) ^{x}$ make sense since
the number inside the brackets is positive. We have localized the problem into the powers of $-1$ for the term $\left(
-1\right) ^{x}$. We would like to replace this term  by a
continuous function $m(x)$ such that it takes the values $\pm1$ on the integers. That is,

$$m(n) =1\quad \text{if }n\text{ is even }\quad\text{and}\quad m(n) =-1\quad \text{if }n\text{ is odd.} $$

The cosine function works. That is

$$m\left( x\right) =\cos \left( \pi x\right) \qquad \text{does the job.} $$
That is:
$$\cos \left( n\pi \right) =1\quad \text{if }n\text{ is even}\quad\text{ and}\quad %
\cos \left( n\pi \right) =-1\quad \text{if }n\text{ is odd.}$$

Why this is a **good** choice would lead us to complex numbers and more!

Hence, we obtain the following closed formula for our function $F[x]:$

$$\begin{eqnarray*}
F[x]  &=&\frac{{\varphi }^{x}-\left( -1\right) ^{x}\left( -\psi
\right) ^{x}}{{\varphi -\psi }}=\frac{1}{\sqrt{5}}\left( {\varphi }%
^{x}-\left( -1\right) ^{x}\left( -\psi \right) ^{x}\right)  \\
&=&\frac{1}{\sqrt{5}}\left( \left( \frac{1+\sqrt{5}}{2}\right) ^{x}-\cos
\left( \pi x\right) \left( \frac{\sqrt{5}-1}{2}\right) ^{x}\right) .
\end{eqnarray*}$$

Let's plot this function, and the Fibonacci sequence.

## A plot of the continuous Fibonacci function

%matplotlib inline
from numpy import *
from matplotlib.pyplot import *

phi=(1+5**(1/2))/2
psi=(5**(1/2)-1)/2

x = arange(0,10)
y = (pow(phi,x) - cos(pi*x)*pow(psi,x))/sqrt(5)
xx = linspace(0,10)
yy = (pow(phi,xx) - cos(pi*xx)*pow(psi,xx))/sqrt(5)

figure(figsize=(10,6));
plot(x,y,'o',xx,yy);
title('The continuous Fibonacci function')
xlabel('$x$')
ylabel('$Fib(x)$');

## A plot with negative values

Well, with this general definition, we can even include negative numbers for $x$ in the function.

Let's plot this too.

%matplotlib inline
from numpy import *
from matplotlib.pyplot import *

phi=(1+5**(1/2))/2
psi=(5**(1/2)-1)/2

x = arange(-10,10)
y = (pow(phi,x) - cos(pi*x)*pow(psi,x))/sqrt(5)
xx = linspace(-10,10,200)
yy = (pow(phi,xx) - cos(pi*xx)*pow(psi,xx))/sqrt(5)

figure(figsize=(10,6));
plot(x,y,'o',xx,yy);
title('The Fibonacci function, extended to negative values')
xlabel('$x$')
ylabel('$Fib(x)$');

So we see we can even get negative Fibonacci numbers!

## The Golden Ratio and Continued Fractions

We have found that the Golden ratio ${\varphi =}\frac{{1+}\sqrt{5}}{2}$
satisfies the identity

$$
{\varphi =1+}\frac{1}{{\varphi }}.
$$

Substituting for ${\varphi }$ on the denominator in the right, we obtain

$$
{\varphi =1+}\frac{1}{{1+}\frac{1}{{\varphi }}}.
$$

Substituting again for ${\varphi }$ on the denominator in the right, we
obtain

$$
{\varphi =1+}\frac{1}{{1+}\dfrac{1}{{1+}\frac{1}{{\varphi }}}}.
$$

Repeating this again,

$$
{\varphi =1+}\frac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\frac{1}{{\varphi }}}}}%
.$$

And again,

$$
{\varphi =1+}\frac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\frac{1}{%
{\varphi }}}}}}.
$$

And again,

$$
{\varphi =1+}\frac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1%
}{{1+}\frac{1}{{\varphi }}}}}}}.
$$

We see that this process can be $\textit{continued indefinitely}$. This results
in an $\textit{infinite expansion of a fraction}$. These type of expressions are known as
$\textbf{continued fractions}$:

$$
{\varphi =1+}\frac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1%
}{{1+}\dfrac{1}{{1+}\dfrac{1}{1+\dfrac{1}{{\vdots }}}}}}}}}.
$$

We can approximate continued fractions with the finite fractions obtained by
stopping the development at some point. In our case, we obtain the
approximates

$$
1,~1+1,~1+\frac{1}{1+1},~1+\frac{1}{1+\dfrac{1}{1+1}},~1+\frac{1}{1+\dfrac{1%
}{1+\dfrac{1}{1+1}}},~1+\frac{1}{1+\dfrac{1}{1+\dfrac{1}{1+\dfrac{1}{1+1}}}}%
,\dots
$$

Explicitly, these approximates are

$$
1,~2,~\frac{3}{2},~\frac{5}{3},~\frac{8}{5},~\frac{13}{8},\dots
$$

This looks like it is just the sequence of ratios $F_{n+1}/F_n$ we saw above!  How can we prove this is the case for all $n$?

We know that the sequence $R_{n} = F_{n+1}/F_n$ satisfies the recursive relation.

$$
R_{n}=\frac{F_{n+1}}{F_{n}}=1+\frac{F_{n-1}}{F_{n}}=1+\frac{1}{R_{n-1}}%
,\qquad \text{with}\qquad R_{1}=1.
$$

Then, we can generate all the terms in the sequence $R_{n}$ by staring with $%
R_{1}=1$, and then using the relation $R_{n+1}=1+\frac{1}{R_{n}}:$

$$
\begin{eqnarray*}
R_{1} &=&1 \\
R_{2} &=&1+\frac{1}{R_{1}}=1+\frac{1}{1}=2 \\
R_{3} &=&1+\frac{1}{R_{2}}=1+\frac{1}{1+R_{1}}=1+\frac{1}{1+1} \\
R_{4} &=&1+\frac{1}{R_{3}}=1+\frac{1}{1+\frac{1}{1+1}} \\
R_{5} &=&1+\frac{1}{R_{4}}=1+\frac{1}{1+\frac{1}{1+\frac{1}{1+1}}} \\
&&\vdots
\end{eqnarray*}
$$

This confirms that both the sequence of rations $R_{n}$ and the sequence of
approximations to the continuous fraction of ${\varphi }$ are the same
sequence. $\square $

In general, continued fractions are expressions of the form

$$
a_{0}+\frac{1}{a_{1}+\dfrac{1}{a_{2}+\dfrac{1}{a_{3}+\dots }}}
$$

where $a_{0}$ is an integer and $a_{1},a_{2},a_{3},\dots $ are positive
integers. These type of fractions are abbreviated by the notation

$$
\left[ a_{0};a_{1},a_{2},a_{3},\dots \right] =a_{0}+\frac{1}{a_{1}+\dfrac{1}{%
a_{2}+\dfrac{1}{a_{3}+\dots }}}.
$$

For example

$$
\begin{eqnarray*}
\left[ 1;1,1,2\right]  &=&1+\frac{1}{1+\dfrac{1}{1+\dfrac{1}{1+1}}}=\frac{8}{%
5} \\
&& \\
\left[ 1;1,1,1,1,\dots \right]  &=&{1+}\frac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{%
{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{{1+}\dfrac{1}{1+\dfrac{1}{{\vdots }%
}}}}}}}}={\varphi }
\end{eqnarray*}
$$

For more information of continued fractions, see
https://en.wikipedia.org/wiki/Continued_fraction

## Conclusion

### What have we learned?

- a **sequence** is an ordered list of numbers, which may go on forever.
- the **Fibonacci sequence** 0,1,1,2,3,5,8,13,... is a famous list of numbers, well-studied since antiquity.
- each number in this sequence is the sum of the two coming before it in the sequence.
- the sequence grows fast, increasing by a **factor** of about **10** for every **five** terms.
- the **ratio** of pairs of Fibonacci numbers converges to the **Golden ratio,** known since the ancient Greeks as the number
$$\varphi = \frac{1 + \sqrt{5}}{2} \approx 1.6108.$$
- the Fibonacci numbers can be computed directly as the difference of powers of $\varphi$ and its **conjugate,** $\psi = \frac{1 - \sqrt{5}}{2}.$ This is sometimes faster than computing the whole list of Fibonnaci numbers.
- this formula with powers of $\varphi, \psi$ is verified using **induction.**
- The Fibonacci numbers can be **extended** to a **continuous function** $Fib(x)$, defined for all real numbers $x$ (including negatives). It **oscillates** (wiggles) on the negative x-axis.
- The **Golden Ratio** can also be expressed a **continued fraction,** which is an infinite expansion of fractions with sub-fraction terms. Many interesting numbers come from interesting continued fraction forms.

[![Callysto.ca License](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-bottom.jpg?raw=true)](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


import numpy as np
import matplotlib.pyplot as plt

def computeCost(X, y, theta):

    #COMPUTECOST Compute cost for linear regression
    #   J = COMPUTECOST(X, y, theta) computes the cost of using theta as the
    #   parameter for linear regression to fit the data points in X and y

    # Initialize some useful values
    m = len(y) # number of training examples
    J = 0
	#	theta is an (n+1)-dimensional vector
	#	X is an m x (n+1)-dimensional matrix
	#	y is an m-dimensional vector
    s = (X.dot(theta) - np.transpose([y]))**2
    J = (1.0 / (2 * m)) * s.sum(axis=0)

    return J

def gradientDescent(X, y, theta, alpha, num_iters):

    # GRADIENTDESCENT Performs gradient descent to learn theta
    #   theta = GRADIENTDESENT(X, y, theta, alpha, num_iters) updates theta by
    #   taking num_iters gradient steps with learning rate alpha

    # Initialize some useful values
    m = len(y) # number of training examples
    J_history = np.zeros((num_iters, 1))
    for i in range(num_iters):
        theta = theta - alpha * (1.0 / m) * np.transpose(X).dot(X.dot(theta) - np.transpose([y]))
        J_history[i] = computeCost(X, y, theta)

    return theta

def plotData(x, y):
    # PLOTDATA Plots the data points x and y into a new figure
    #   PLOTDATA(x,y) plots the data points and gives the figure axes labels of
    #   population and profit.
    plt.plot(x, y, 'rx', markersize=10, label='Training data')
    plt.xlabel('Population of City in 10,000s')
    plt.ylabel('Profit in $10,000s')
    plt.show(block=False)  # prevents having to close the chart

def warmUpExercise(*args, **kwargs):
    return np.identity(5)

def featureNormalize(X):

    # FEATURENORMALIZE Normalizes the features in X
    #   FEATURENORMALIZE(X) returns a normalized version of X where
    #   the mean value of each feature is 0 and the standard deviation
    #   is 1. This is often a good preprocessing step to do when
    #   working with learning algorithms.
    X_norm = X
    mu = np.zeros((1, X.shape[1]))
    sigma = np.zeros((1, X.shape[1]))
    for i in range(X.shape[1]):
    	mu[:,i] = np.mean(X[:,i])
    	sigma[:,i] = np.std(X[:,i])
    	X_norm[:,i] = (X[:,i] - float(mu[:,i]))/float(sigma[:,i])

    return X_norm, mu, sigma

def gradientDescentMulti(X, y, theta, alpha, num_iters):

    # GRADIENTDESCENTMULTI Performs gradient descent to learn theta
    #   theta = GRADIENTDESCENTMULTI(x, y, theta, alpha, num_iters) updates theta by
    #   taking num_iters gradient steps with learning rate alpha

    # Initialize some useful values
    m = len(y) # number of training examples
    J_history = np.zeros((num_iters, 1))

    for i in range(num_iters):
        theta = theta - alpha*(1.0 / m) * np.transpose(X).dot(X.dot(theta) - np.transpose([y]))
        J_history[i] = computeCost(X, y, theta)

    return theta, J_history

def normalEqn(X, y):
    #   NORMALEQN(X,y) computes the closed-form solution to linear
    #   regression using the normal equations.
    theta = np.zeros((X.shape[1], 1))
    theta = np.linalg.pinv(np.transpose(X).dot(X)).dot(np.transpose(X).dot(y))

    return theta


from sympy import (sympify, factorial, var, cos, S, sin, Dummy, sqrt, pi, exp,
        I, latex, symbols)

def Plm(l, m, z):
    """
    Returns the associated Legendre polynomial P_{lm}(z).

    The Condon & Shortley (-1)^m factor is included.
    """
    l = sympify(l)
    m = sympify(m)
    z = sympify(z)
    if m >= 0:
        r = ((z**2-1)**l).diff(z, l+m)
        return (-1)**m * (1-z**2)**(m/2) * r / (2**l * factorial(l))
    else:
        m = -m
        r = ((z**2-1)**l).diff(z, l+m)
        return factorial(l-m)/factorial(l+m) * (1-z**2)**(m/2) * r / (2**l * factorial(l))

def Plm_cos(l, m, theta):
    """
    Returns the associated Legendre polynomial P_{lm}(cos(theta)).

    The Condon & Shortley (-1)^m factor is included.
    """
    l = sympify(l)
    m = sympify(m)
    theta = sympify(theta)
    z = Dummy("z")
    r = ((z**2-1)**l).diff(z, l+m).subs(z**2-1, -sin(theta)**2).subs(z, cos(theta))
    return (-1)**m * sin(theta)**m * r / (2**l * factorial(l))

def Ylm(l, m, theta, phi):
    """
    Returns the spherical harmonics Y_{lm}(theta, phi) using the Condon & Shortley convention.
    """
    l, m, theta, phi = sympify(l), sympify(m), sympify(theta), sympify(phi)
    return sqrt((2*l+1)/(4*pi) * factorial(l-m)/factorial(l+m)) * Plm_cos(l, m, theta) * exp(I*m*phi)

def Zlm(l, m, theta, phi):
    """
    Returns the real spherical harmonics Z_{lm}(theta, phi).
    """
    l, m, theta, phi = sympify(l), sympify(m), sympify(theta), sympify(phi)
    if m > 0:
        return sqrt((2*l+1)/(2*pi) * factorial(l-m)/factorial(l+m)) * Plm_cos(l, m, theta) * cos(m*phi)
    elif m < 0:
        m = -m
        return sqrt((2*l+1)/(2*pi) * factorial(l-m)/factorial(l+m)) * Plm_cos(l, m, theta) * sin(m*phi)
    elif m == 0:
        return sqrt((2*l+1)/(4*pi)) * Plm_cos(l, 0, theta)
    else:
        raise ValueError("Invalid m.")

def Zlm_xyz(l, m, x, y, z):
    """
    Returns the real spherical harmonics Z_{lm}(x, y, z).

    It is assumed x**2 + y**2 + z**2 == 1.
    """
    l, m, x, y, z = sympify(l), sympify(m), sympify(x), sympify(y), sympify(z)
    if m > 0:
        r = (x+I*y)**m
        r = r.as_real_imag()[0]
        return sqrt((2*l+1)/(2*pi) * factorial(l-m)/factorial(l+m)) * Plm(l, m, z) * r / sqrt(1-z**2)**m
    elif m < 0:
        m = -m
        r = (x+I*y)**m
        r = r.as_real_imag()[1]
        return sqrt((2*l+1)/(2*pi) * factorial(l-m)/factorial(l+m)) * Plm(l, m, z) * r / sqrt(1-z**2)**m
    elif m == 0:
        return sqrt((2*l+1)/(4*pi)) * Plm(l, 0, z)
    else:
        raise ValueError("Invalid m.")

var("theta phi")
x, y, z = symbols("x y z", real=True)
print "Spherical harmonics:"
print
print ".. math::"
print
for l in range(4):
    for m in range(-l, l+1):
        print r"    Y_{%d,%d}(\theta, \phi) =" % (l, m), \
            latex(Ylm(l, m, theta, phi))
        print

print
print "Real spherical harmonics:"
print
print ".. math::"
print
for l in range(4):
    for m in range(-l, l+1):
        print r"    Z_{%d,%d}(\theta, \phi) =" % (l, m), \
            latex(Zlm(l, m, theta, phi))
        print

print
print "Real spherical harmonics (using $x$, $y$ and $z$, assuming $x^2 + y^2 + z^2 = 1$):"
print
print ".. math::"
print
for l in range(4):
    for m in range(-l, l+1):
        print r"    Z_{%d,%d}(x, y, z) =" % (l, m), \
            latex(Zlm_xyz(l, m, x, y, z).simplify())
        print


########################################
# Overview
#
# This file defines loss functions for
# generalized linear models. These functions
# are the negative log likelihoods using the
# canonical links.
########################################
import numpy as np
from scipy.special import factorial, gamma

def SSE(Y, X, B):
    Y_Hat = np.matmul(X, B)
    Y_Hat.reshape([Y_Hat.shape[0],1])
    loss = np.sum( np.power(Y - Y_Hat, 2))

    return loss

########################################
# Mathematical background
# Step 1 Calculate eta
# Step 2 Inverse link
# step 3 Calculate negative log likelihood
########################################

def neg_ll_gaussian(Y, X, B):
    Y_Hat = np.matmul(X, B)
    Y_Hat.reshape([Y_Hat.shape[0],1])
    Y_Hat = Y_Hat # Inverse link

    n = X.shape[0]
    p = X.shape[1]
    sigma = np.sum(np.power(Y - Y_Hat, 2)) / (n-p)

    ll = -(n/2)*np.log(2*np.pi) - n*np.log(sigma) - (1/2*np.power(sigma, 2)) * np.sum(np.power(Y-Y_Hat,2))
    ll = -1 * ll

    return ll

def neg_ll_poisson(Y, X, B):
    Y_Hat = np.matmul(X, B)
    Y_Hat.reshape([Y_Hat.shape[0],1])
    Y_Hat = np.exp(Y_Hat) # Inverse link

    ll = Y * np.log(Y_Hat) - Y_Hat - np.log(factorial(Y))
    ll = np.sum(ll)
    ll = -1 * ll

    return ll

def neg_ll_bernoulli(Y, X, B):
    Y_Hat = np.matmul(X, B)
    Y_Hat.reshape([Y_Hat.shape[0],1])
    Y_Hat = np.exp(Y_Hat) / (1 + np.exp(Y_Hat)) # Inverse link

    ll = Y * np.log(Y_Hat) + (1 - Y) * np.log(1 - Y_Hat)
    ll = np.sum(ll)
    ll = -1 * ll

    return ll

def neg_ll_gamma(Y, X, B):
    Y_Hat = np.matmul(X, B)
    Y_Hat.reshape([Y_Hat.shape[0],1])
    Y_Hat = np.power(Y_Hat, -1) # Inverse link

    # Using method of moments estimate
    # MLE does not have closed form.
    # Don't want to write numerical method for MLE
    n = X.shape[0]
    p = X.shape[1]
    numerator = np.power(Y - Y_Hat, 2)
    denominator = np.power(Y_Hat, 2) * (n-p)
    phi = np.sum(numerator / denominator)

    ll = -1 * np.log(Y) - np.log(gamma(1/phi)) + (1/phi) * (np.log(Y) - np.log(Y_Hat) - np.log(phi)) - Y/(Y_Hat * phi)
    ll = np.sum(ll)
    ll = -1 * ll

    return ll


#--------#--------#--------#--------#--------#--------#--------#--------#--------#--------#
#   QR Algorithm for obtaining eigenvalues and eigenvectors from tridiagonal matrices     #
#                                      20/07/21                                           #
#--------#--------#--------#--------#--------#--------#--------#--------#--------#--------#

import numpy as np
import math
from functools import reduce

#Generates a rotation matrix
def generateRotationMatrix(shape, c_mat, s_mat, i_mat, j_mat):
    matrixG = np.array(np.zeros((shape,shape)))
    for i in range(0, shape):
        for j in range(0, shape):
            if (i == j) and (i == i_mat or j == j_mat): #Gkk = c for k == i,j
                matrixG[i,j] = c_mat
            elif (i == j) and (i != i_mat or j != j_mat): #Gkk = 1 for k != i,j
                matrixG[i,j] = 1
    matrixG[i_mat,j_mat] = s_mat
    matrixG[j_mat,i_mat] = -s_mat
    return matrixG

#Triangularization function using rotation matrix
def givensRotation(matrixA):
    QRResults = []
    GMatrices = []
    dimension = matrixA.shape[0]
    matrixAk = matrixA
    for i in range(1, dimension):
        if matrixAk[i,i-1] != 0:
            if abs(matrixAk[i-1,i-1]) > abs(matrixAk[i,i-1]): #Numerically stable
                tau = (-matrixAk[i,i-1])/matrixAk[i-1,i-1]
                c = 1/(math.sqrt(1+pow(tau,2)))
                s = c*tau
            else:
                tau = (-matrixAk[i-1,i-1])/matrixAk[i,i-1]
                s = 1/(math.sqrt(1+pow(tau,2)))
                c = s*tau
            G = generateRotationMatrix(dimension, c, s, i, i-1)
            GMatrices.append(G)
            matrixAk = np.matmul(G, matrixAk)
    matrixR = matrixAk #Upper triangular
    GMatricesTransp = [np.transpose(GMatrices[i]) for i in range(0, len(GMatrices))]
    matrixQ = reduce(np.matmul, GMatricesTransp) #np.matmul of all transposed G matrices
    QRResults = [matrixQ, matrixR]
    return QRResults

#QR Algorithm
def QRAlgorithm(A, desloc):
    Ak = A
    shape = A.shape[0] #Matrix A, k = 0
    Vk = np.identity(shape) #Matrix V, k = 0, V0 = I
    errval = 10e-6
    k = 0 #Number of iterations
    for m in range(1, shape):
        while abs(Ak[m,m-1]) > errval:
            if k == 0 or desloc == False:
                mi = 0
            else:
                dk = (Ak[m-1,m-1] - Ak[m,m])/2 #dk = (alpha_n-1 + alpha_n)/2
                if dk >= 0:
                    sign = 1
                else:
                    sign = -1
                mi = (Ak[m,m] + dk - (sign*np.sqrt(pow(dk,2)+pow(Ak[m,m-1],2))))
            QRArray = givensRotation((Ak - (np.identity(shape))*mi)) #QR Factorization
            Qk = QRArray[0]
            Rk = QRArray[1]
            Ak = (np.matmul(Rk,Qk) + (np.identity(shape))*mi) #Updates the matrix
            Vk = np.matmul(Vk,Qk) #Updates eigenvectors
            k+=1
    print("->" + str(k) + " Iteracoes do Algoritmo QR")
    results = [Ak,Vk]
    return results


import numpy as np

def inverse_iteration(A, x, shift=2):
	diff = 10000
	diffn = 0
	xn = None
	xo = x
	lamda = 0
	iters = 0
	A = A - shift * np.diag(np.diag(np.ones(A.shape)))
	while(diff > 10**(-12)):
		iters+=1
		yn = np.linalg.solve(A, xo)
		xn = yn / np.linalg.norm(yn, ord=2)
		lamda = np.linalg.norm(yn, ord=2)
		diff = np.linalg.norm((xn - xo), ord=2) / np.linalg.norm(xo, ord=2)
		xo = xn

	return 1/(lamda) + shift, xn, iters

if __name__ == '__main__':
	print("Inverse Iteration")
	print("-------------------\n\n")

	seeds = [1, 89, 98, 23, 88, 91, 101, 11, 17, 19]

	for i in range(10):
		A = np.array([[6, 2, 1], [2, 3, 1], [1, 1, 1]])
		np.random.seed(seeds[i])
		print('Seed:', seeds[i])
		x0 = np.random.random((3,))
		l, x, iters = inverse_iteration(A, x0)
		print("Computed Eigenvalue:", l)
		print("Computed Eigenvector:\n\t", x, "'\n")
		print("Number of iterations:", iters)
		print("\n\n")


# for activation functions of Neural Networks
import numpy as np

# activation function
def linear(x):
    """
    Performs linear function on the input elements.
    Linear function: f(x) = x

    :param x: {array}, shape {n_samples,}
            input to the function.
    :return: {array}, shape {n_samples,}
            output of linear function.
    """
    return x

def logistic(x):
    """
    Performs logistic function on the input elements.
    Logistic function: f(x) = 1/(1+exp(-x))

    :param x: {array}, shape {n_samples,}
            input to the function.
    :return: {array}, shape {n_samples,}
            output of logistic function.
    """
    return 1 / (1 + np.exp(-x))

def relu(x):
    """
    Performs Rectified Linear Unit function on the input elements.
    Rectified Linear Unit: f(x) = max(0,x)

    :param x: {array}, shape {n_samples,}
            input to the function.
    :return: {array}, shape {n_samples,}
            output of relu function.
    """
    return np.maximum(x, 0)

def tanh(x):
    """
    Performs Bipolar logistic function on the input elements.
    Bipolar logistic function: f(x) = tanh(x) = (exp(x)-exp(-x))/(exp(x)+exp(-x))
                                    = 2*(1/(1+exp(-2*x)) - 1
    :param x: {array}, shape {n_samples,}
            input to the function.
    :return: {array}, shape {n_samples,}
            output of tanh function.
    """
    return 2 * logistic(x) - np.ones(x.shape)

def rbf(x):
    """
    Performs Radial Basis function on the input elements.
    Radial Basis function: f(x) = exp(-(x - mean_x)**2/(2*var_x)) / sqrt(2*pi*var_x)

    :param x: {array}, shape {n_samples,}
            input to the function.
    :return: {array}, shape {n_samples,}
            output of rbf function.
    """
    mean_x = np.mean(x)
    var_x = np.var(x)
    z = (x - mean_x)
    return np.exp(-z**2 / (2 * var_x)) / np.sqrt(2 * np.pi * var_x)

activations = {"logistic": logistic, "relu": relu,
               "tanh": tanh, "rbf": rbf, "linear": linear}


"""
Last change: 30.07.2018
"""

# Some imports
import numpy as np
import math
from scipy import stats

class Standardization:

    def __init__(self):
        pass

    def standardize(self, x):

        """
        Standardization: x_std = (x - x_mean) / std_dev

        Inputs:
        - x: input vector (has to be a row vector)

        Returns:
        - x_norm: standardized vector of the inputed vector
        """

        mean = self.compute_mean(x)
        std_dev = self.compute_standard_deviation(x)

        x_norm = np.zeros_like(x)
        for i in range(x.shape[1]):

            x_norm[0, i] = (x[0, i] - mean) / std_dev

        return x_norm

    def compute_mean(self, x):

        """
         Computes mean of the inputed vector.

        Inputs:
         - x: input vector (has to be a row vector)

        Returns:
        - mean: mean value of the inputed vector
        """

        sm = np.sum(x)
        ln = x.shape[1]

        mean = sm / ln

        return mean

    def compute_standard_deviation(self, x):

        """
        Computes standard deviation of the inputed vector.

        Inputs:
        - x: input vector (has to be a row vector)

        Returns:
        - std_dev: standard deviation of the inputed vector
        """

        sm = np.sum(x)
        mean = self.compute_mean(x)

        summ = 0
        for i in range(x.shape[1]):
            summ = summ + (x[0, i] - mean)**2

        std_dev = math.sqrt(summ/x.shape[1])

        return std_dev

"""
# Test

# Instantiate class
obj = Normalization()

# Generate random vector
np.random.seed(5)
v = 10*np.random.random([1, 10])
print('Vector: ', v)

# Compute mean
mean = np.mean(v)
print('Mean: ', mean)

# Compute manual mean
mean_manual = obj.compute_mean(v)
print('Mean manual', mean_manual)

# Compute standard deviation
std_dev = np.std(v)
print('Standard deviation: ', std_dev)

# Compute manual standard deviation
std_dev_manual = obj.compute_standard_deviation(v)
print('Standard deviation manual', std_dev_manual)

# Standardize
standard = stats.zscore(v, axis=1)
print('Standardized: ', standard)

# Manual standardized
standard_manual = obj.standardize(v)
print('Standardized manual: ', standard_manual)
"""


'''
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Taking user input ~~~~~~~~~~~~~~~~~~~~~~~~~~~
while True:
    try:
        N = int(input("Enter the Number of Points: "))
        if N > 1: break
        else: print("Number of Points must be greater than 1")
    except ValueError:
        print("Enter a Natural Number greater than 1")
X, Y = [], []
for i in range(N):
    while True:
        try:
            x = float(input("Enter the X coordinate of the point "+str(i+1)+": "))
        except ValueError:
                print("Enter a Real Number")
                continue
        else: break
    X.append(x)
    while True:
        try:
            y = float(input("Enter the Y coordinate of the point "+str(i+1)+": "))
        except ValueError:
                print("Enter a Real Number")
                continue
        else: break
    Y.append(y)
print()
'''

# X = [0, 1, 2, 3, 4, 5]
# Y = [2.1, 7.7, 13.6, 27.2, 40.9, 61.1]

X = [3, 4, 5, 6, 7, 8]
Y = [0, 7, 17, 26, 35, 45]

N = len(X)  # Number of data points

# ~~~~~~~~~~~~~~~~~~~~~~~~ Method 1: Using for loop ~~~~~~~~~~~~~~~~~~~~~~~~

sumx = sumy = sumx2 = sumxy = 0 # Initial value of summation variables
for i in range(N):
    sumx += X[i]
    sumy += Y[i]
    sumx2 += X[i]**2
    sumxy += X[i]*Y[i]
meanx = sumx / N
meany = sumy / N

a1 = (N*sumxy - sumx*sumy) / (N*sumx2 - sumx**2)
a0 = meany - a1 * meanx

# ~~~~~~~~~~~~~~~~~ Method 2: Using numpy array, sum, mean ~~~~~~~~~~~~~~~~~

from numpy import array, mean

# sum(X**2) requires X to be a numpy array as python list cannot be squared
X = array(X, float)

# a1 = (sum(X*Y) - mean(X)*sum(Y))/(sum(X**2) - N*mean(X)**2)
# a0 = (mean(Y)*sum(X**2) - mean(X)*sum(X*Y))/(sum(X**2) - N*mean(X)**2)

a1 = (N*sum(X*Y) - sum(X)*sum(Y)) / (N*sum(X**2) - sum(X)**2)
a0 = mean(Y) - a1 * mean(X)

print("The straight line equation :")
s = "-" if a1<0 else "+"
a1 = -a1 if a1<0 else a1
print('y = %.3f %s %.3f x' %(a0, s, a1))

'''
y = a0 + a1 x + e    # where a0 = intercept, a1 = slope, y = mx+c
e = y - a0 - a1 x    # e = true value of y ~ approximate value of a0+a1x

Criteria for a best fit:
∑ (ei) = ∑ (yi - a0 - a1 xi)

Sr = ∑ (ei)^2 = ∑ (yi - a0 - a1 xi)^2

∂Sr/∂a0 = -2    ∑ (yi - a0 - a1 xi)
∂Sr/∂a1 = -2 xi ∑ (yi - a0 - a1 xi)

0 = ∑    yi - ∑ a0    - ∑ a1 xi
0 = ∑ xi yi - ∑ a0 xi - ∑ a1 xi^2

(∑ xi) a0 + (∑ xi^2) a1 = ∑ xi yi
     n a0 + (∑ xi  ) a1 = ∑    yi    # ∑ a0 = n a0

Cross Multiplication:
a1 = ( n ∑ xi yi - ∑ xi ∑ yi ) / ( n ∑ xi^2 - (∑ xi)^2 )

a0 = ( ∑ yi - (∑ xi) a1 ) / n
   = mean(y) - a1 mean(x)

or,
a1 = ( n ∑ xi yi - ∑ xi ∑ yi )/n / ( n ∑ xi^2 - (∑ xi)^2 )/n
   = ( ∑ xi yi - mean(x) ∑ yi ) / ( ∑ xi^2 - n mean(x)^2 )
a0 = ( ∑ xi^2 ∑ yi - ∑ xi ∑ xi yi )/n / ( n ∑ xi^2 - (∑ xi)^2 )/n
   = ( mean(y) ∑ xi^2 - mean(x) ∑ xi yi ) / ( ∑ xi^2 - n mean(x)^2 )

'''

2. Interpolation and Curve Fitting/5. Linear (Least Squares) Regression or Fit.py


import numpy as np

def l2_norm(x, y=None):
    if y is None:
        return np.linalg.norm(x)
    else:
        return np.linalg.norm(x - y)

def mean(matrix, axis=0):
    return np.average(matrix, axis=axis)

def get_centroid(matrix, centroid_method=mean):
    return centroid_method(matrix)

def get_distance(x, y, distance_metric=l2_norm):
    return distance_metric(x, y)

def rbf_kernel(x, y, sigma=1, norm=l2_norm):
    return np.exp(-(norm(x, y) ** 2) / (2 * (sigma ** 2)))

def kernel_matrix(x, y, kernel=rbf_kernel, tuner=1):
    m = x.shape[1]
    n = y.shape[1]
    mat = np.zeros((m, n))
    for i in range(m):
        for j in range(n):
            mat[i, j] = kernel(x[:, i], y[:, j], tuner)
    return mat


import math
import numpy as np

def cholensky(__A):
    _A = np.array(__A)
    # return np.linalg.cholesky(_A)
    n = list(np.shape(_A))[0]
    A = np.zeros([n, n], dtype='double')
    for i in range(n):
        for j in range(i+1):
            A[i, j] = _A[i, j]
    for j in range(n):
        A[j, j] = np.sqrt(A[j, j] - np.linalg.norm(A[j, 0:j])**2)
        for i in range(j+1, n):
            tmp = 0
            for k in range(j):
                tmp += A[i, k] * A[j, k]
            A[i, j] = (A[i, j] - tmp) / A[j, j]

    return A

def cho_solve(A, _b):
    n = list(np.shape(A))[0]
    b = np.array(_b)
    L = cholensky(A)
    Lt = np.transpose(L)

    y = np.zeros(n, dtype='double')
    for i in range(n):
        for j in range(n - 1 - i):
            b[i + j + 1] -= b[i] * L[i + j + 1, i] / L[i, i]
        y[i] = b[i] / L[i, i]

    x = np.zeros(n, dtype='double')
    for i in range(n):
        for j in range(n - 1 - i):
            y[n - 1 - (i + j + 1)] -= y[n-1 - i] * Lt[n - 1 - (i + j + 1), n - 1 - i] / Lt[n - 1 - i, n - 1 - i]
        x[n - 1 - i] = y[n - 1 - i] / Lt[n - 1 - i, n - 1 - i]
    return x

def hilbert(n):
    ret = np.zeros([n, n])
    for i in range(n):
        for j in range(n):
            ret[i][j] = 1.0 / (i + j + 1)

    return ret

def my_sum(cb, max_count, dt):
    sigma = dt(0.0)
    n = dt(1.0)
    while True:
        sigma1 = sigma + dt(1) / dt(n)
        cb(n, sigma)
        if sigma1 == sigma:
            break
        if n == max_count:
            break
        n += dt(1)
        sigma = sigma1
    return sigma

def newton(f, df, x0, get_lambda, lambda0, epsilon, cb):
    k = 0
    x = x0
    while True:
        s = f(x) / df(x)
        x1 = x - s
        i = 0
        y = f(x)
        la = 1
        while True:
            y1 = f(x1)
            if abs(y1) < abs(y):
                break
            la = get_lambda(lambda0, i)
            x1 = x - la * s
            i += 1
        cb(k, la, y, y1, x, x1)
        k += 1
        if abs(x1 - x) <= epsilon:
            break
        x = x1

def iteration_solver(A, b, x0, threshold, cb, method='jacobi', omega=1):
    x = x0
    iter_count = 0
    n = list(np.shape(b))[0]
    if method == 'jacobi':
        while True:
            y = np.zeros(n, dtype='double')
            for i in range(n):
                a = np.dot(A[i], x) - A[i, i] * x[i]
                new_xi = (b[i] - a) / A[i, i]
                y[i] = new_xi
            delta = np.linalg.norm(y - x)
            if delta <= threshold:
                break
            if delta > 1e10:
                break
            cb(iter_count, x, y - x)
            x = y
            iter_count += 1
        print('Total iter %d, delta %g' % (iter_count, delta))
        return x
    else:
        while True:
            y = np.array(x)
            for i in range(n):
                a = np.dot(A[i], x) - A[i, i] * x[i]
                new_xi = (1 - omega) * x[i] + omega * (b[i] - a) / A[i, i]
                x[i] = new_xi
            delta = np.linalg.norm(y - x)
            if delta <= threshold:
                break
            cb(iter_count, x, y - x)
            iter_count += 1
        print('Total iter %d, delta %g' % (iter_count, delta))
        return x

def power(A, x0, threshold, cb):
    u = x0
    last_lambda1 = 0
    i = 0
    while True:
        v = np.dot(A, u)
        lambda1 = np.max(v)
        u = v / lambda1
        delta = abs(last_lambda1 - lambda1)
        if delta < threshold:
            break
        cb(i, u, lambda1, delta)
        last_lambda1 = lambda1
        i += 1
    return lambda1, u

def fitting(pts, n):
    phi = []
    for i in range(n):
        phi.append(np.power(pts[0], i))

    A = np.zeros((n, n), dtype='double')
    for i in range(n):
        for j in range(n):
            A[i, j] = np.dot(phi[i], phi[j])

    b = np.zeros(n, dtype='double')
    for i in range(n):
        b[i] = np.dot(phi[i], pts[1])

    x = cho_solve(A, b)

    return x
    #  tt = 0
    # For each point
    #  for _ in range(list(np.shape(pts))[1]):
    #      A = np.zeros((n, n), dtype='double')
    #      for i in range(n):
    #          for j in range(n):
    #              A[i, j] = pts[0][i] * pts[0][j]
    #      tt += np.linalg.norm(np.dot(A, x) - b) ** 2
    #  d = np.sqrt(tt / list(np.shape(pts))[1])
    #  return x, d


import numpy as np
import matplotlib.pyplot as plt

def normal_pdf(x, mu, sigma):
    pdf = 1/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (x - mu)**2 / (2 * sigma**2))
    return pdf

fig = plt.figure(figsize=(6,9))

# Random sampling of a normal distribution
my_mu, my_sigma = 0, 0.1 # mean and standard deviation

bit_generators = [np.random.MT19937(), np.random.Philox(), np.random.SFC64()]
names = ['Mersenne Twister PRNG (MT19937)', 'Philox (4x64) PRNG (Philox)', 'Chris Doty-Humphrey\'s SFC PRNG (SFC64)']
indexes = [1,2,3]

for bit_generator, name, index in zip(bit_generators, names, indexes):
    sn = np.random.Generator(bit_generator).normal(loc = my_mu, scale = my_sigma, size = 10000)
    ax = fig.add_subplot(3, 1, index)
    ax.hist(sn, density=True, bins='auto', edgecolor='k', color='#c7ddf4', label=name)
    my_xn = np.linspace(my_mu - 4 * my_sigma, my_mu + 4 * my_sigma, 1000)
    my_yn = normal_pdf(x=my_xn, mu=my_mu, sigma=my_sigma)
    ax.plot(my_xn, my_yn, linewidth=2, linestyle='--', color='#ff464a', label ='Target Normal PDF')
    ax.set_ylim(0.0, 7.0)
    ax.set_xlim(my_mu - 6 * my_sigma, my_mu + 6 * my_sigma)
    ax.set_xlabel('x')
    ax.set_ylabel('Probability Density')
    ax.legend()

fig.tight_layout()


def approx_first_derivative_no_main_block(f,x,h):
    """
    Numerical differentiation by finite differences. Uses central point formula
    to approximate first derivative of function.
    Args:
        f (function): function definition.
        x (float): point where first derivative will be approximated
        h (float): step size for central differences. Tipically less than 1
    Returns:
        df (float): approximation to first_derivative.
    """
    df = (f(x+h) - f(x-h))/(2.0*h)
    return df

def approx_second_derivative_no_main_block(f,x,h):
    """
    Numerical differentiation by finite differences. Uses central point formula
    to approximate second derivative of function.
    Args:
        f (function): function definition.
        x (float): point where second derivative will be approximated
        h (float): step size for central differences. Tipically less than 1
    Returns:
        ddf (float): approximation to second_derivative.
    """
    ddf =(f(x+h) - 2.0*f(x) + f(x-h))/h**2
    return ddf

## Python libraries
import numpy as np

## Parameters
f = np.arctan
x = 0.9
h = 1e-6

## Main code
res_first_d = approx_first_derivative_no_main_block(f,x,h)
res_second_d = approx_second_derivative_no_main_block(f,x,h)

print(res_first_d)
print(res_second_d)


from numpy.random import rand
from math import sqrt
from sys import argv

def distance(array1, array2):
	assert len(array1) == len(array2), "Arrays must be of the same dimension"
	return sqrt(sum([(array2[i] - array1[i])**2 for i in range(len(array1))]))

if __name__ == '__main__':
	dim = int(argv[1])
	assert dim >= 2, "Dimension must be greater than or equal to two"
	iterations = int(argv[2]) if len(argv) >= 3 else 10000
	assert iterations >= 10, "Iterations must be greater than or equal to 10"
	sum_dist = 0
	for i in range(iterations):
		array1 = rand(dim)
		array2 = rand(dim)
		sum_dist += distance(array1, array2)
	estimate = round(sum_dist / iterations, 5)
	print(f"With {iterations} iterations, the estimated distance between two points in a(n) {dim}-dimensional cube is {estimate}")


import math
import numpy as np
from PIL import Image

def basic_sigmoid(x):
    s = 1/(1+math.exp(-x))
    return s
print(1, basic_sigmoid(10))

x = np.array([1,2,3])
print(2, x+3)

def sigmoid(x):
    s = 1/(1+np.exp(-x))
    return s
print(3, sigmoid(x))

def sigmoid_derivative(x):
    s = 1/(1+np.exp(-x))
    ds = s*(1-s)
    return ds
print(4, sigmoid_derivative(x))

def image2vector(image):
    v = image.reshape((image.shape[0] * image.shape[1] * image.shape[2]), 1)
    return v
image = np.array(Image.open("./icon_sample.JPG"))
print(5, image2vector(image))

def normalizeRows(x):
    x_norm = np.linalg.norm(x, axis=1, keepdims=True)
    x = x/x_norm
    return x
x = np.array([
    [0,3,4], [1,6,4]
])
print(6, normalizeRows(x))

def softmax(x):
    x_exp = np.exp(x)
    x_sum = np.sum(x_exp, axis=1, keepdims=True)
    s = x_exp/x_sum
    return s
x = np.array([
    [1,2,3],
    [4,5,6],
    [7,8,9]
])
print(7, softmax(x))


# -*- coding: utf-8 -*-
#
import math

import scipy.special
import sympy

def integrate_monomial_over_unit_simplex(k, symbolic=False):
    """The integrals of monomials over the standard triangle and tetrahedron are
    given by

    \\int_T x_0^k0 * x1^k1 = (k0!*k1!) / (2+k0+k1)!,
    \\int_T x_0^k0 * x1^k1 * x2^k2 = (k0!*k1!*k2!) / (4+k0+k1+k2)!,

    see, e.g.,
    A set of symmetric quadrature rules on triangles and tetrahedra,
    Linbo Zhang, Tao Cui and Hui Liu,
    Journal of Computational Mathematics,
    Vol. 27, No. 1 (January 2009), pp. 89-96,
    <https://www.jstor.org/stable/43693493>.

    See, e.g., <https://math.stackexchange.com/q/207073/36678> for a formula in
    all dimensions.
    """
    if symbolic:
        return sympy.prod([sympy.gamma(kk + 1) for kk in k]) / sympy.gamma(
            sum(k) + len(k) + 1
        )
    # exp-log to account for large values in numerator and denominator
    # import scipy.special
    return math.exp(
        math.fsum([scipy.special.gammaln(kk + 1) for kk in k])
        - scipy.special.gammaln(sum([kk + 1 for kk in k]) + 1)
    )


import numpy as np
import pylab
import matplotlib.pyplot as plt

def lagranz_intrepolation(X,Y,x):
   z =  0
   for j in range(len(Y)):
       p1 = 1
       p2 = 1
       for i in range(len(X)):
           if i == j :
             continue
           else:
             p1 = p1 * (x - X[i])
             p2 = p2 * (X[j] - X[i])
       z = z + Y[j]/p2*p1
   return z

def newton_interpolation(x, y, u):
    g = y[:]
    s = g[0]
    for i in range(len(y) - 1):
        g = [(g[j + 1] - g[j]) / (x[j + i + 1] - x[j]) for j in range(len(g) - 1)]
        s += g[0] * product(u - x[j] for j in range(i + 1))
    return s

def product(a):
    p = 1
    for i in a: p *= i
    return p

def main():
    X = np.array([0,np.pi/6,np.pi/4,np.pi/3,np.pi/2])
    Y = 4*np.sin(X)**2
    x_in = np.sin(np.pi/7)
    y_true = 4*np.sin(x_in)**2
    xnew = np.linspace(np.min(X),np.max(X),100)
    Y_lagranz = np.array([lagranz_intrepolation(X,Y,i) for i in xnew])
    Y_newton = np.array([newton_interpolation(X,Y,i) for i in xnew])
    y_newtwon = newton_interpolation(X,Y,x_in)
    y_lagranz = lagranz_intrepolation(X,Y,x_in)
    print("In newtwon : " + str(y_true-y_newtwon))
    print("In lagranz :" + str(y_true-y_lagranz))
    plt.plot(X,Y,'o')
    plt.plot(xnew,Y_lagranz,'g')
    plt.plot(xnew,Y_newton,'b')
    plt.grid(True)
    pylab.show()
main()


# Z-Test Assignement

"""
A sample of 400 male students is found to have a mean height of 171.38cms. Can it be reasonably regarded as a sample from a large population with mean height 171.17 and standard deviation 3.30cms
"""
print("Assignment 1")
import numpy as np
n=400
sMean=171.38
pMean=171.17
pStd=3.30

zstatistics=(sMean - pMean)/(pStd/np.sqrt(n))
print("Zstatistics :",zstatistics)

from scipy.stats import norm

zcritical_l=norm.ppf(q=0.05/2)
zcritical_u=-zcritical_l
print("Critical Values are :",zcritical_l,zcritical_u)

if zstatistics<zcritical_l or zstatistics>zcritical_u:
    print("Reject the Null Hypothesis")
    print("sample is not from the large population")
else:
    print("Fail to reject the Null Hypothesis")
    print("sample is from the large population")

    print("\n")

"""
2.A sample of 900 items has mean 3.4 and standard deviation 2.61. can the sample be regarded as drawn from a population with mean 3.25 at 1 percent level of significance
"""
print("Assignment 2")
n=900
sMean=3.4
pMean=3.25
pStd=2.61

zstatistics=(sMean - pMean)/(pStd/np.sqrt(n))
print("Zstatistics :",zstatistics)

from scipy.stats import norm

zcritical_l=norm.ppf(q=0.01/2)
zcritical_u=-zcritical_l
print("Critical Values are :",zcritical_l,zcritical_u)

if zstatistics<zcritical_l or zstatistics>zcritical_u:
    print("Reject the Null Hypothesis")
    print("sample is not from the large population")
else:
    print("Fail to reject the Null Hypothesis")
    print("sample is from the large population")


#%%
import numpy as np

#%%
# Example 1
x = np.arange(1.0, 8 + 1, 1.0)
y = np.array([6.9, 10.8, 9.3, 7.8, -0.7, -9.2, -22.1, -37.7])

x_mean = np.mean(x)
y_mean = np.mean(y)

b1 = np.dot(y_mean - y, x_mean - x) / np.sum(np.power(x_mean - x, 2))
b0 = y_mean - b1 * x_mean
sigma_squared = (
    1 / x.shape[0] * np.sum([(yi - b0 - b1 * xi) ** 2 for (xi, yi) in zip(x, y)])
)
print(f"b0 = {b0}")
print(f"b1 = {b1}")
print(f"sigma_squared = {sigma_squared}")

# %%
# Example 2
x = np.array([1.0, 3.0, 6.0, 10.0])
y = np.array([1.1, 2.9, 6.1, 10.1])

x_mean = np.mean(x)
y_mean = np.mean(y)

b1 = np.dot(y_mean - y, x_mean - x) / np.sum(np.power(x_mean - x, 2))
b0 = y_mean - b1 * x_mean
sigma_squared = (
    1 / x.shape[0] * np.sum([(yi - b0 - b1 * xi) ** 2 for (xi, yi) in zip(x, y)])
)
print(f"b0 = {b0}")
print(f"b1 = {b1}")
print(f"sigma_squared = {sigma_squared}")

# %%
# Example 3
T_denominator = np.sqrt(np.sum([(yi - b0 - b1 * xi) ** 2 for (xi, yi) in zip(x, y)]))
print(T_denominator)

sigma_prime = np.sqrt(
    (
        1
        / (x.shape[0] - 2)
        * np.sum([(yi - b0 - b1 * xi) ** 2 for (xi, yi) in zip(x, y)])
    )
)
sx = np.sqrt(np.sum([(xi - x_mean) ** 2 for xi in x]))
b0_distribution_denominator = sigma_prime * np.sqrt(
    1 / x.shape[0] + x_mean ** 2 / sx ** 2
)
print(b0_distribution_denominator)
print(T_denominator / b0_distribution_denominator)

T = (b0 - 0.448) / T_denominator
print(T)
print(T * T_denominator / b0_distribution_denominator)


import numpy as np

def identity(z):
    """Identity function...
    Args:
        z (np.array)

    Returns:
        f(z) = z (np.array)
    """
    return z

def dfdz_identity(z):
    """Derivative of the Identity function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = 1.0 (np.array)
    """
    return np.ones_like(z)

def sigmoid(z):
    """Sigmoid function...
    Args:
        z (np.array)

    Returns:
        f(z) = 1 / (1 + exp(-z)) (np.array)
    """
    return 1.0 / (1.0 + np.exp(-z))

def dfdz_sigmoid(z):
    """Derivative of the Sigmoid function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = f(z) * (1 - f(z)) (np.array)
    """
    return sigmoid(z) * (1.0 - sigmoid(z))

def logistic(z):
    """Logistic function...
    Args:
        z (np.array)

    Returns:
        f(z) = 1 / (1 + exp(-z)) (np.array)
    """
    return sigmoid(z)

def dfdz_logistic(z):
    """Derivative of the Logistic function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = f(z) * (1 - f(z)) (np.array)
    """
    return sigmoid(z) * (1.0 - sigmoid(z))

def tanh(z):
    """Hyperbolic tangent function...
    Args:
        z (np.array)

    Returns:
        f(z) = 2.0 / (1.0 + np.exp(-2.0 * z)) - 1.0 (np.array)
    """
    return 2.0 / (1.0 + np.exp(-2.0 * z)) - 1.0

def dfdz_tanh(z):
    """Derivative of the hyperbolic tangent function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = 1.0 - np.square(tanh(z)) (np.array)
    """
    return 1.0 - np.square(tanh(z))

def softsign(z):
    """Softsign function...
    Args:
        z (np.array)

    Returns:
        f(z) = z / (1.0 + np.abs(z)) (np.array)
    """
    return z / (1.0 + np.abs(z))

def dfdz_softsign(z):
    """Derivative of the softsign function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = None (np.array)
    """
    raise RuntimeError('not implemented...')

def ReLU(z):
    """Rectified linear unit function...
    Args:
        z (np.array)

    Returns:
        f(z) = np.max(0, z) (np.array)
    """
    return z * (z > 0)

def dfdz_ReLU(z):
    """Derivative of the rectified linear unit function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = 1 if x > 0 else 0 (np.array)
    """
    return (z > 0)

def LReLU(z):
    """Leaky rectified linear unit function...
    Args:
        z (np.array)

    Returns:
        f(z) = z if z > 0 else 0.01 * z (np.array)
    """
    return PReLU(z, 0.01)

def dfdz_LReLU(z):
    """Derivative of the leaky rectified linear unit function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = 1 if x > 0 else 0.01 (np.array)
    """
    return dfdz_PReLU(z, 0.01)

def PReLU(z, alpha):
    """Parametric rectified linear unit function...
    Args:
        z (np.array)

    Returns:
        f(z) = z if z > 0 else alpha * z (np.array)
    """
    return z * (z > 0) + alpha * z * (z <= 0)

def dfdz_PReLU(z, alpha):
    """Derivative of the parametric rectified linear unit function...
    Args:
        z (np.array)

    Returns:
        df(z)/dz = 1 if x > 0 else alpha (np.array)
    """
    return 1.0 * (z > 0) + alpha * (z <= 0)


import numpy as np
import matplotlib.pyplot as plt

def sigmoid(val):
    return 1/(1 + np.exp(-val))

def stable_coeff(rho, psi):
    r = sigmoid(rho)
    theta = np.pi * sigmoid(psi)

    a_1 = -2*r*np.cos(theta)
    a_2 = r**2
    return a_1, a_2

def roots_polynomial(a_1, a_2):
    delta = a_1**2 - 4 * a_2
    delta = delta.astype(np.complex)
    root_1 = (-a_1 + np.sqrt(delta))/2
    root_2 = (-a_1 - np.sqrt(delta))/2
    idx_real = delta > 0
    return root_1, root_2, idx_real

if __name__ == '__main__':

    N = 100000
    rho = np.random.randn(N)*1
    psi = np.random.randn(N)*1

    a_1, a_2 = stable_coeff(rho, psi)
    r_1, r_2, idx_real = roots_polynomial(a_1, a_2)

    fig, ax = plt.subplots()
    ax.plot(a_1, a_2, '*')
    ax.plot(a_1[idx_real], a_2[idx_real], 'k*')
    ax.set_xlabel('a_1')
    ax.set_ylabel('a_2')
    ax.set_xlim([-2, 2])
    ax.set_ylim([-2, 2])

    fig, ax = plt.subplots()
    ax.plot(np.real(r_1), np.imag(r_1), 'r*')
    ax.plot(np.real(r_2), np.imag(r_2), 'r*')
    ax.plot(np.real(r_1)[idx_real], np.imag(r_1)[idx_real], 'k*')
    ax.plot(np.real(r_2)[idx_real], np.imag(r_2)[idx_real], 'k*')
    ax.set_xlim([-1.2, 1.2])
    ax.set_ylim([-1.2, 1.2])

    perc_real = np.sum(idx_real) / N *100
    print(f"Real poles in {perc_real:.1f} cases")


from sympy import *

ax, bx, cx, dx, ay, by, cy, dy, t = symbols('ax bx cx dx ay by cy dy t')
speed_t = diff((3*ax*t**2 + 2*bx*t + cx)**2 + (3*ay*t**2 + 2*by*t + cy)**2, t )
print('speed by t expression: ')
print(speed_t)
print('\n')

constant_speed_eq = integrate(speed_t**2, (t, 0, 1))
print('constant speed equation: ')
print(constant_speed_eq)
print('\n')

# p0 => t = 0; p1 => t = 1
p0x, p0y, p1x, p1y, p0dx, p0dy, p1dy_to_dx = symbols('p0x p0y p1x p1y p0dx p0dy p1dy_to_dx')

point0_x_eq = dx - p0x
point0_y_eq = dy - p0y
point1_x_eq = ax + bx + cx + dx - p1x
point1_y_eq = ay + by + cy + dy - p1y

point0_dx_eq = cx - p0dx
point0_dy_eq = cy - p0dy

point1_dy_to_dx_eq = 3*ay + 2*by + cy - p1dy_to_dx*(3*ax + 2*bx + cx)

solution = solve([constant_speed_eq, point0_x_eq, point0_y_eq, point1_x_eq, point1_y_eq, point0_dx_eq, point0_dy_eq, point1_dy_to_dx_eq], (ax, bx, cx, dx, ay, by, cy, dy))

print('solution: ')
print(solution)
print('\n')


# -*- coding: utf-8 -*-
import numpy as np

def cos_sim(a, b):
	"""Takes 2 vectors a, b and returns the cosine similarity according
	to the definition of the dot product
	"""
	dot_product = np.dot(a, b)
	norm_a = np.linalg.norm(a)
	norm_b = np.linalg.norm(b)
	return dot_product / (norm_a * norm_b)

# the counts we computed above
sentence_m = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0])
sentence_h = np.array([0, 0, 1, 1, 1, 1, 0, 0, 0])
sentence_w = np.array([0, 0, 0, 1, 0, 0, 1, 1, 1])

# We should expect sentence_m and sentence_h to be more similar
print(cos_sim(sentence_m, sentence_m)) # 1.0
print(cos_sim(sentence_m, sentence_h)) # 0.5
print(cos_sim(sentence_m, sentence_w)) # 0.25


from numpy import any, array, array_str, concatenate, dot, intersect1d
from numpy import nonzero, ravel, setdiff1d, zeros
from numpy.linalg import lstsq, norm

def active_constraints(A, b, x):
    """Return a set of active constraints
    """
    dx = b - dot(A, x)
    if any(dx > 0):
        raise ValueError('x is infeasible')
    return nonzero(dx == 0)[0]

def get_starting_point(A, b):
    raise NotImplementedError('You need to provide x_0')

def get_dual_vars(G, f, A, x, W):
    """Compute lagrange multiplies for W"""
    mu, res, rank, s = lstsq(A[W, :].T, dot(G, x) + f)
    return mu

def quadprog(G, f, A, b, x_0=None, max_iter=1e6, tol=1e-4, verbose=True):
    """Solve convex QP via active set method

    min 0.5 * x^T * G * x + f^T * x
    s.t. A * x >= b

    """
    # Compute a feasible starting point
    if x_0 is None:
        x = get_starting_point(A, b)
    else:
        x = x_0.copy()
    m, n, k = x.size, A.shape[0], 0
    # Set initial active-set of contraints
    W = active_constraints(A, b, x)
    while k < max_iter:
       f_obj = dot(x.T, dot(G, x)) + dot(f.T, x)
       if verbose:
           print 'Iter: {0}\tObj: {1}\tW: {2:5}\tx: {3}'.format(k,
               array_str(ravel(f_obj), precision=3), array_str(W),
               array_str(ravel(x), precision=3))
       p = solve_qp_eq(G, f, A[W, :], b[W], x)
       if norm(p) <= tol * m**2:
           lmbda = get_dual_vars(G, f, A, x, W)
           if all(lmbda >= 0):
               # Global minimizer was found
               if verbose:
                   print 'Optimization finished\n'
               break
           else:
               # Remove most-violated constraint
               j = lmbda.argmin()
               W = setdiff1d(W, [W[j]])
       else:
           alpha, j = step_size(A, b, x, p, W)
           x = x + alpha*p
           if j is not None:
               # Add a blocking constraint
               W = concatenate((W, [j]), axis=0)
       k += 1
    if k == max_iter and verbose:
        print 'Max number of iterations reached'
    return x

def solve_qp_eq(G, f, C, d, x):
    """Solve QP with equality constraint"""
    m, n = x.size, C.shape[0]
    A = concatenate((concatenate((G, C.T), axis=1),
                     concatenate((C, zeros((n, n))), axis=1)), axis=0)
    b = concatenate((f + dot(G, x), dot(C, x) - d), axis=0)
    lmbda, res, rank, s = lstsq(A, b)
    return -lmbda[:m, :]

def step_size(A, b, x, p, W):
    """Compute minimum step size such that x + p is still in the polyhedron"""
    num, den, alpha_min, block = b - dot(A, x), dot(A, p), 1, None
    alpha = num / den
    J = intersect1d(setdiff1d(range(A.shape[0]), W, assume_unique=True),
                    nonzero(den < 0)[0])
    if J.size != 0:
        alpha_min = alpha[J].min()
    alpha_hat = min(1, alpha_min)
    # Check blocking constraints
    if alpha_hat < 0:
        raise ValueError('Got a negative alpha (step length)')
    elif alpha_hat < 1:
        block = nonzero(alpha_hat == alpha)[0][0]
    return alpha_hat, block

def hw5_1b():
    """
    Solve the following qp with active set method
    min x_1^2 + 2*x_2^2 - 2*x_1 - 6*x_2 - 2*x_1*x_2
    s.t. 0.5*x_1 + 0.5*x_2 <= 1; -1*x_1 + 2*x_2 <= 2; x_1>=0; x_2>= 0

    Choose x_0 inside, in the boundary and as extrem point.
    """
    G = array([[2, -2],[-2, 4]])
    F = array([[-2],[-6]])
    A = -1 * array([[0.5, 0.5], [-1, 2], [-1, 0], [0, -1]])
    b = -1 * array([[1],[2],[0],[0]])
    x_seed = [array([[1], [0.5]]), array([[0], [0]]), array([[0], [0.5]])]
    for x_0 in x_seed:
        x = quadprog(G, F, A, b, x_0)

if __name__ == '__main__':
    hw5_1b()


import numpy as np
import matplotlib.pyplot as plt
from welib.tools.signal import *
from numpy.random import seed; seed(0)

# --- Parameters
dt    = 1
n     = 10000
coeff = 0.95 # 1:full corr, 0: no-corr
nMax  = 180
# --- Create a correlated time series
tvec = np.arange(0,n)*dt
ts   = correlated_signal(coeff, n)
# --- Compute correlation coefficient
R, tau = correlation(ts, nMax=nMax, dt=dt)

# --- Plot
fig,axes = plt.subplots(2, 1, sharey=False, figsize=(6.4,4.8)) # (6.4,4.8)
fig.subplots_adjust(left=0.12, right=0.95, top=0.95, bottom=0.11, hspace=0.20, wspace=0.20)
ax=axes[0]
# Plot time series
ax.plot(tvec,ts)
ax.set_xlabel('t [s]')
ax.set_ylabel('u [m/s]')
ax.tick_params(direction='in')
# Plot correlation
ax=axes[1]
ax.plot(tau,  R              ,'-o', label='Computed')
ax.plot(tau, coeff**(tau/dt) ,'--' ,label=r'Theoretical -  c$^{\tau/dt}$') # analytical coeff^n trend
ax.set_xlabel(r'$\tau$ [s]')
ax.set_ylabel(r'$R(\tau)$ [-]')
ax.set_title('Signal - Correlation coefficient')
ax.legend()

if __name__=='__main__':
    plt.show()
if __name__ == '__test__':
    pass
if __name__=="__export__":
    from welib.tools.repo import export_figs_callback
    export_figs_callback(__file__)


import numpy as np
import matplotlib.pyplot as plt

# Plot 2D points
def displaypoints2d(points):
  plt.figure(0)
  plt.plot(points[0,:],points[1,:], '.b')
  plt.xlabel('Screen X')
  plt.ylabel('Screen Y')

# Plot 3D points
def displaypoints3d(points):
  fig = plt.figure(1)
  ax = fig.add_subplot(111, projection='3d')
  ax.scatter(points[0,:], points[1,:], points[2,:], 'b')
  ax.set_xlabel("World X")
  ax.set_ylabel("World Y")
  ax.set_zlabel("World Z")

def cart2hom(points):
  """ Transforms from cartesian to homogeneous coordinates.

  Args:
    points: a np array of points in cartesian coordinates

  Returns:
    points_hom: a np array of points in homogeneous coordinates
  """

  #
  # You code here
  #

  # add an additional component and fill it with ones
  return np.concatenate((points, np.ones((1, points.shape[1]))))

def hom2cart(points):
  """ Transforms from homogeneous to cartesian coordinates.

  Args:
    points: a np array of points in homogenous coordinates

  Returns:
    points_hom: a np array of points in cartesian coordinates
  """

  #
  # You code here
  #

  # Perform perspective division
  res = points[:-1,:]/points[-1,:]
  return res

def gettranslation(v):
  """ Returns translation matrix T in homogeneous coordinates for translation by v.

  Args:
    v: 3d translation vector

  Returns:
    T: translation matrix in homogeneous coordinates
  """

  #
  # You code here
  #

  T = np.eye(4)
  T[0:3,3] = v
  return T

def getxrotation(d):
  """ Returns rotation matrix Rx in homogeneous coordinates for a rotation of d degrees around the x axis.

  Args:
    d: degrees of the rotation

  Returns:
    Rx: rotation matrix
  """

  #
  # You code here
  #

  r = np.radians(d)
  Rx = np.eye(4)
  Rx[1:3,1:3] = np.array([[np.cos(r), -np.sin(r)],[np.sin(r),np.cos(r)]])
  return Rx

def getyrotation(d):
  """ Returns rotation matrix Ry in homogeneous coordinates for a rotation of d degrees around the y axis.

  Args:
    d: degrees of the rotation

  Returns:
    Ry: rotation matrix
  """

  #
  # You code here
  #

  r = np.radians(d)
  Ry = np.eye(4)
  Ry[0:3:2, 0:3:2] = np.array([[np.cos(r),np.sin(r)],[-np.sin(r),np.cos(r)]])
  return Ry

def getzrotation(d):
  """ Returns rotation matrix Rz in homogeneous coordinates for a rotation of d degrees around the z axis.

  Args:
    d: degrees of the rotation

  Returns:
    Rz: rotation matrix
  """

  #
  # You code here
  #

  r = np.radians(d)
  Rz = np.eye(4)
  Rz[0:2,0:2] = np.array([[np.cos(r),-np.sin(r)],[np.sin(r),np.cos(r)]])
  return Rz

def getcentralprojection(principal, focal):
  """ Returns the (3 x 4) matrix L that projects homogeneous camera coordinates on homogeneous
  image coordinates depending on the principal point and focal length.

  Args:
    principal: the principal point, 2d vector
    focal: focal length

  Returns:
    L: central projection matrix
  """

  #
  # You code here
  #

  # l1-image_formation-v1 Seite 35

  # [[f,0,px,0],
  #  [0,f,py,0],
  #  [0,0,1,0]]

  return np.array([[focal, 0.0, principal[0], 0.0],
                    [0.0, focal, principal[1], 0.0],
                    [0.0, 0.0, 1.0, 0.0]])

def getfullprojection(T, Rx, Ry, Rz, L):
  """ Returns full projection matrix P and full extrinsic transformation matrix M.

  Args:
    T: translation matrix
    Rx: rotation matrix for rotation around the x-axis
    Ry: rotation matrix for rotation around the y-axis
    Rz: rotation matrix for rotation around the z-axis
    L: central projection matrix

  Returns:
    P: projection matrix
    M: matrix that summarizes extrinsic transformations
  """

  #
  # You code here
  #

  # l1-image_formation-v1 Seite 39

  # Model transformations
  M = Rz.dot(Rx.dot(Ry.dot(T)))
  # Full projection matrix
  P = L.dot(M)
  return P,M

def projectpoints(P, X):
  """ Apply full projection matrix P to 3D points X in cartesian coordinates.

  Args:
    P: projection matrix
    X: 3d points in cartesian coordinates

  Returns:
    x: 2d points in cartesian coordinates
  """

  #
  # You code here
  #

  return hom2cart(P.dot(cart2hom(X)))

def loadpoints():
  """ Load 2D points from obj2d.npy.

  Returns:
    x: np array of points loaded from obj2d.npy
  """

  #
  # You code here
  #

  # Load arrays from ``.npy``
  x = np.load('data/obj2d.npy')
  return x

def loadz():
  """ Load z-coordinates from zs.npy.

  Returns:
    z: np array containing the z-coordinates
  """

  #
  # You code here
  #
  z = np.load('data/zs.npy')
  return z

def invertprojection(L, P2d, z):
  """
  Invert just the projection L of cartesian image coordinates P2d with z-coordinates z.

  Args:
    L: central projection matrix
    P2d: 2d image coordinates of the projected points
    z: z-components of the homogeneous image coordinates

  Returns:
    P3d: 3d cartesian camera coordinates of the points
  """

  #
  # You code here
  #

  # Extract camera intrinsics
  # l1-image_formation-v Seite 37
  K = L[:,0:3]
  # Account for unknown scale
  # l1-image_formation-v1 Seite 24
  P3d = z*(np.linalg.solve(K,cart2hom(P2d)))
  return P3d

def inverttransformation(M, P3d):
  """ Invert just the model transformation in homogeneous coordinates
  for the 3D points P3d in cartesian coordinates.

  Args:
    M: matrix summarizing the extrinsic transformations
    P3d: 3d points in cartesian coordinates

  Returns:
    X: 3d points after the extrinsic transformations have been reverted
  """

  #
  # You code here
  #

  # extrinsic transformations
  # l1-image_formation-v1 Seite 29

  X = np.linalg.solve(M,cart2hom(P3d))
  return X

def p3multiplecoice():
  '''
  Change the order of the transformations (translation and rotation).
  Check if they are commutative. Make a comment in your code.
  Return 0, 1 or 2:
  0: The transformations do not commute.
  1: Only rotations commute with each other.
  2: All transformations commute.
  '''

  return -1


#------------------------------------------------------------------------------------------------------------#
#Chapter 2 - Visualization with hierarchical clustering and t-SNE

#------------------------------------------------------------------------------------------------------------#
#Hierarchical clustering of the grain data

# Perform the necessary imports
from scipy.cluster.hierarchy import linkage, dendrogram
import matplotlib.pyplot as plt

# Calculate the linkage: mergings
mergings = linkage(samples, method='complete')

# Plot the dendrogram, using varieties as labels
dendrogram(mergings,
           labels=varieties,
           leaf_rotation=90,
           leaf_font_size=6,
)
plt.show()

#------------------------------------------------------------------------------------------------------------#

#Hierarchies of stocks
# Import normalize
from sklearn.preprocessing import normalize

# Normalize the movements: normalized_movements
normalized_movements = normalize(movements)

# Calculate the linkage: mergings
mergings = linkage(normalized_movements, method='complete')

# Plot the dendrogram
dendrogram(mergings,
           labels=companies,
           leaf_rotation=90,
           leaf_font_size=6,
)

plt.show()

#------------------------------------------------------------------------------------------------------------#
#Different linkage, different hierarchical clustering!
# Perform the necessary imports
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import dendrogram, linkage

# Calculate the linkage: mergings
mergings = linkage(samples,method='single')

# Plot the dendrogram
dendrogram(mergings,
           labels=country_names,
           leaf_rotation=90,
           leaf_font_size=6,
)
plt.xlabel('European Nations')
plt.ylabel('Number of votes')
plt.title('EuroVision 2017 Denogram ')
plt.legend(loc='upper right')
plt.show()

#------------------------------------------------------------------------------------------------------------#

#Extracting the cluster labels
# Perform the necessary imports
import pandas as pd
from scipy.cluster.hierarchy import fcluster

# Use fcluster to extract labels: labels
labels = fcluster(mergings,6,criterion='distance')

# Create a DataFrame with labels and varieties as columns: df
df = pd.DataFrame({'labels': labels, 'varieties': varieties})

# Create crosstab: ct
ct = pd.crosstab(df['labels'],df['varieties'])

# Display ct
print(ct)

#------------------------------------------------------------------------------------------------------------#
#t-SNE visualization of grain dataset
# Import TSNE
from sklearn.manifold import TSNE

# Create a TSNE instance: model
model = TSNE(learning_rate=200)

# Apply fit_transform to samples: tsne_features
tsne_features = model.fit_transform(samples)

# Select the 0th feature: xs
xs = tsne_features[:,0]

# Select the 1st feature: ys
ys = tsne_features[:,1]

# Scatter plot, coloring by variety_numbers
plt.scatter(xs,ys,c=variety_numbers)
plt.show()

#------------------------------------------------------------------------------------------------------------#
#A t-SNE map of the stock market

# Import TSNE
from sklearn.manifold import TSNE

# Create a TSNE instance: model
model = TSNE(learning_rate=50)

# Apply fit_transform to normalized_movements: tsne_features
tsne_features = model.fit_transform(normalized_movements)

# Select the 0th feature: xs
xs = tsne_features[:,0]

# Select the 1th feature: ys
ys = tsne_features[:,1]

# Scatter plot
plt.scatter(xs,ys,alpha=0.5)

# Annotate the points
for x, y, company in zip(xs, ys, companies):
    plt.annotate(company, (x, y), fontsize=5, alpha=0.75)
plt.show()

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

#------------------------------------------------------------------------------------------------------------#

Unsupervised Learning in Python/Chapter 2 - Visualization with hierarchical clustering and t-SNE.py


# -*- coding: utf-8 -*-
"""problem 2 class 4 offline

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/10n45tcpjVcRcO65S-6hJK2_trgI091yp
"""

# libraries
import random
import math
import numpy as np
import matplotlib.pyplot as plt
random.seed(10)

def integral(N):
  A= 0
  B = 8
  # lists for storing the values
  y1_values = []
  y1_square_values =[]

  y2_values =[]
  y2_square_values =[]

  count_1 = 0
  count_2 = 0

  for i in range(1,N+1):
    # x random value generation
    x = random.uniform(A,B)
    # calculating first area value
    if x <=4 :
      y1 = math.sqrt(4 * x)
      y1_values.append(y1)
      y1_square_values.append(y1*y1)
      count_1 += 1
    else:
      y2 = 8 - x
      y2_values.append(y2)
      y2_square_values.append(y2*y2)
      count_2 += 1

  # calculate 0 to 4 area first part
  y1_average = sum(y1_values) / len(y1_values)
  y1_square_average = sum(y1_square_values) / len(y1_square_values)
  area_1 = y1_average * ( 4-0 )
  error_1 = ( (4-0) / math.sqrt(count_1) ) * math.sqrt(y1_square_average - y1_average**2)

  # calculate 4 to 8 area second part
  y2_average = sum(y2_values) / len(y2_values)
  y2_square_average = sum(y2_square_values) / len(y2_square_values)
  area_2 = y2_average * ( 8-4 )
  error_2 = ( (8-4) / math.sqrt(count_2) ) * math.sqrt(y2_square_average - y2_average**2)

  #total area and error
  integral_value = area_1 + area_2
  error_estimate = error_1 + error_2

  print("For number of sample points: ", N)
  print("Integral Value: ", integral_value)
  print("Estimated Error: ", error_estimate)
  print("")

N = [100,1000,5000,10000]
for i in N:
  integral(i)

Assignment 03 - Monte Carlo Intregral/problem2.py


"""
You have a list of points in the plane. Return the area of the largest triangle that can be formed by any 3 of the points.

Example:

    Input: points = [[0,0],[0,1],[1,0],[0,2],[2,0]]
    Output: 2
    Explanation: The five points are show in the figure below. The red triangle is the largest.

    https://s3-lc-upload.s3.amazonaws.com/uploads/2018/04/04/1027.png

Notes:
    1. 3 <= points.length <= 50.
    2. No points will be duplicated.
    3. -50 <= points[i][j] <= 50.
    4. Answers within 10^(-6) of the true value will be accepted as correct.
"""

import itertools
import numpy as np

class Solution:
    def largestTriangleArea1(self, points):
        def area(A, B, C):
            x_1, y_1 = A
            x_2, y_2 = B
            x_3, y_3 = C
            return 0.5 * abs(x_2 * y_3 - x_3 * y_2 - x_1 * y_3 + x_3 * y_1 + x_1 * y_2 - x_2 * y_1)

        return max(area(*selected) for selected in itertools.combinations(points, 3))

    def largestTriangleArea2(self, points):
        return max(
            abs(np.linalg.det(np.hstack((np.array(selected), np.ones((3, 1)))))) * 0.5 for selected in itertools.combinations(points, 3)
        )


#regla de simpson compuesta

import numpy as np
import matplotlib.pyplot as plt

def f(x):
    return (1/4)*x*(10-x)+ np.sin(np.pi*x)
a=0
b=5
N=5
h=(b-a)/N

def simpson(f,a,b,N, h):
    h = h / 2
    Xi=np.array([a+h*i for i in range(2*N+1)])
    Yi=f(Xi)
    suma1=sum(np.array([Yi[2*i] for i in range(1,N)]))
    suma2=sum(np.array([Yi[2*i+1] for i in range(N)]))
    simpson= (h/3)*(f(a)+2*suma1+4*suma2+f(b))
    print(simpson)

    return simpson

print("La aprox. de la integral por la regla de simpson compuesta es: ", simpson(f,a,b,N))

Xi= np.array([a+h*i for i in range(N)]) #para calcular la funcion
Yi= f(Xi)

xi=np.linspace(a,b,100) #Para graficar muchos puntos para f(x) y sea suave
yi=f(xi)

plt.title("Funcion y funcion escalon usada para integrar")
plt.xlabel("x")
plt.ylabel("y")
plt.step(Xi,Yi,'r') #funcion escalon
plt.plot(xi,yi,'b') #funcion f(x)

for j in range(len(Xi)):
    plt.plot(Xi[j], Yi[j],'go') #puntos (xi,yi)

plt.savefig("EjReglaDeSimpson.png")
plt.show()

N = 10
x = [1 * i for i in range(2*N+1)]
n1 = len(x)
n2 = len([x[2*i] for i in range(1,N)])
n3 = len([x[2*i+1] for i in range(N)])


import enum
import numpy as np
import random
import matplotlib.pyplot as plt

# 3. Calculate the logistic map function for l = 0.5, 2.5, 3.5, 4
#    Xn+1 = l * Xn (1 - Xn)
# 4. Compare the same statistical features of your RW (2) and the four
#    different logistic map time series (3)

LENGTH = 100
l = np.asarray([0.5, 2.5, 3.5, 4])
l_len = len(l)

means = np.zeros((l_len, 1))
variances = np.zeros((l_len, 1))
strd_deviations = np.zeros((l_len, 1))

ensemble = np.zeros(LENGTH)
ensemble_mean = 0
ensemble_strd_dev = 0

# Temporary storage for calculations
tmp_arr = np.zeros((l_len, LENGTH))

X = np.zeros((l_len, LENGTH))

for i in range(l_len):
    # Initial value
    X[i][0] = 0.3
    for n in range(LENGTH - 1):
        X[i][n+1] = l[i] * X[i][n] * (1 - X[i][n])

    means[i] = np.mean(X[i])

    # Standard deviation
    tmp_arr[i] = np.subtract(X[i], means[i])
    tmp_arr[i] = tmp_arr[i] * tmp_arr[i]
    variances[i] = np.sum(tmp_arr[i]) / LENGTH

    strd_deviations[i] = np.sqrt(np.sum(tmp_arr[i]) / (LENGTH - 1))

print ("Means")
print (means)
print ("Variances")
print (variances)
print ("Standard deviations")
print (strd_deviations)

# Calculate ensemble
ensemble = np.sum(X, axis=0)
ensemble = ensemble / l_len

# Ensemble time-average mean
ensemble_mean = np.mean(ensemble)

# Ensemble standard deviation
ensemble_tmp = np.subtract(ensemble, ensemble_mean)
ensemble_tmp = ensemble_tmp * ensemble_tmp
ensemble_strd_dev = np.sqrt(np.sum(ensemble_tmp) / (LENGTH - 1))

print ("Ensemble mean: {}".format(ensemble_mean))
print ("Ensemble standard deviation: {}".format(ensemble_strd_dev))

# Logistic map functions
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2)
ax1.plot(X[0])
ax1.axhline(means[0], label='time-average', color='r', linestyle='-')
ax1.set_title("l = {}".format(l[0]))
ax2.plot(X[1])
ax2.set_title("l = {}".format(l[1]))
ax2.axhline(means[1], label='time-average', color='r', linestyle='-')
ax3.plot(X[2])
ax3.set_title("l = {}".format(l[2]))
ax3.axhline(means[2], label='time-average', color='r', linestyle='-')
ax4.plot(X[3])
ax4.set_title("l = {}".format(l[3]))
ax4.axhline(means[3], label='time-average', color='r', linestyle='-')
plt.show()

# Ensemble
# Time-average mean
plt.axhline(ensemble_mean, label='time-average', color='r', linestyle='-')
# Ensemble average
plt.plot(ensemble, color='b', label='Ensemble')
plt.legend()
plt.show()

flatX = X.flatten()
n, bins, patches = plt.hist(flatX, 25, facecolor='b')
plt.grid(True)
plt.show()