# Chapter 7: Cross-Correlations, Fourier Transform, and Wavelet Transform
prepared by Gilbert Chua

Gaining a deeper undersanding of time series dynamics and classifying them, we look at time series forecasting through another lens.

In this chapter, we will take a different approach to how we analzye time series that is complementary to forecasting. Previously, methods of explaining such as the Granger Causality, Simplex Mapping, and Convergent Cross Mapping, focused on the time domain - the values of the time series when measured over time or in its phase space. While both are useful for many tasks, it can be often useful to transform these time domain measurements to unearth patterns which are difficult to tease out. Specifically, we want to look at the frequency domain to both analyze the dynamics and perform pre-processing techniques that may be used to modify real-world datasets.

We will be analyzing the dynamics of time series not exactly to make forecasts, but to understand them in terms of their frequencies in complement to the previous methods of causality and explainability presented.

We introduce three techniques:

    1) Cross-correlations

    2) Fourier Transform

    3) Wavelet Transform

and test their use on the Jena Climate Dataset (2009-2016) along with a handful of other datasets.

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import pandas as pd
from datetime import date, timedelta
import scipy as scp
import random
from statsmodels.tsa.stattools import acf, adfuller, ccf, ccovf
from itertools import repeat
import pywt
import os
import warnings
warnings.filterwarnings('ignore')

## Cross-correlation Between Two Signals

Phenomenon captured by time series may not happen at the exact same time with some lag $h$ between them. A simple example of this is Sonar technology. The time series associated with the response from the sound waves being reflected comes at some lag compared to the time series of the device emitting the initial sound waves. It is this lag we want to measure when we use the cross-correlation function (CCF). This function was discussed previously in Chapter 3 - Vector Autoregressive Methods. We will explore this concept in this chapter and adopt the same definition.

Let $\mathbf\Gamma_h$ be the covariance matrix at lag $h$, $\mathbf D$ be a $N\times N$ diagonal matrix containing the standard deviations of $y_{i,t}$ for $i=1, ..., N$. The lagged correlation matrix of $\mathbf{y}_t$ is defined as

$$\boldsymbol\rho_h = \mathbf D^{-1}\mathbf\Gamma_h\mathbf D^{-1}$$

The $(i,j)$th element of $\boldsymbol\rho_h$ is the correlation coefficient between $y_{i,t}$ and $y_{j,t-h}$:

$$\rho_{i,j}(h) = \dfrac{Cov\left[y_{i,t}, y_{j,t-h} \right]}{\sigma_{i,t} \sigma_{j,t}}$$

where $Cov[y_{i,t}, y_{j,t-h}]$ is the covariance of two time series $y_{i,t}$ and $y_{j,t-h}$ at time $t$ and lag $h$.

The values of the correlation coefficient from the cross-correlation function are interpreted as such:

|Correlation Coefficient| Interpretation |
|:----|:----|
|$\rho_{i,j}(0)\neq0$|$y_{i,t}$ and $y_{j,t}$ are *contemporaneously linearly correlated*|
|$\rho_{i,j}(h)=\rho_{j,i}(h)=0$ for all $h\geq0$|$y_{i,t}$ and $y_{j,t}$ share *no linear relationship*|
|$\rho_{i,j}(h)=0$ and $\rho_{j,i}(h)=0$ for all $h>0$|$y_{i,t}$ and $y_{j,t}$ are said to be linearly *uncoupled*|
|$\rho_{i,j}(h)=0$ for all $h>0$, but $\rho_{j,i}(q)\neq0$ for at least some $q>0$|There is a *unidirectional (linear) relationship* between $y_{i,t}$ and $y_{j,t}$, where $y_{i,t}$ does not depend on $y_{j,t}$, but $y_{j,t}$ depends on (some) lagged values of $y_{i,t}$|
|$\rho_{i,j}(h)\neq0$ for at least some $h>0$ and $\rho_{j,i}(q)\neq0$ for at least some $q>0$|There is a *bi-directional (feedback) linear relationship* between $y_{i,t}$ and $y_{j,t}$|

### Noisy Signals

To demonstrate this, we take the case of two simple yet noisy time series.

We want to measure the shift between two periodic signals Two sine waves are generated that are identical in both amplitude and frequency but one is shifted forward in time. By taking the maximum of the cross-correlation, we should be able to retrieve how much one of the time series leads (or lags!) the other.

def sine_wave(amp=1, freq=1, sample_rate=200, duration=5,
              plot=True, shift=0, noise=0):
    x = np.linspace(0, duration, sample_rate*duration)
    frequencies = x * freq
    y = amp*np.sin((2 * np.pi * frequencies) + shift) + noise
    if plot:
        plt.subplots(figsize=(15, 2))
        plt.plot(x, y)
        plt.show()
    return x, y

samp_rate = 500  # Sampling Frequency in Hertz
dur = 5  # Duration in Seconds

amp1 = 1  # Amplitude of Sine Wave 1
freq1 = 1  # Frequency of Sine Wave 1

amp2 = 1  # Amplitude of Sine Wave 2
freq2 = 1  # Frequency of Sine Wave 2

phase_shift = np.pi/2  # Phase shift
noise = np.random.uniform(-1, 1, samp_rate*dur)
x1, y1 = sine_wave(amp1, freq1, samp_rate, dur, noise=noise,
                   plot=False)
noise = np.random.uniform(-0.3, 0.3, samp_rate*dur)
x2, y2 = sine_wave(amp2, freq2, samp_rate, dur, noise=noise,
                   shift=phase_shift, plot=False)

plt.subplots(figsize=(15, 2))
plt.plot(np.linspace(0, dur, samp_rate*dur), y1, label='$y1$')
plt.plot(np.linspace(0, dur, samp_rate*dur), y2, label='$y2$')
plt.ylabel('Value')
plt.xlabel('Time (in $s$)')
plt.legend();

plt.subplots(figsize=(15, 2))

ccf_12 = ccf(y1, y2, unbiased=False)
plt.ylabel('Correlation Coefficient')
plt.xlabel('Lag $h$')
plt.stem(np.linspace(0, dur, samp_rate*dur), ccf_12)
peak = np.argmax(ccf_12)/samp_rate
plt.axvline(peak, c='orange')
print('Maximum at:', peak, '(in $s$)')

We find that the maximum correlation between $y_2$ and $y_1$ occurs at ~0.247 seconds, approximately 1/4 of the frequency. This is very close to the theoretical value of 0.25 seconds corresponding to a phase shift of $\frac{\pi}{2}$. Using only direct measurements from the data, albeit synthetic, we are able to retrieve a reasonable estimate of the lag and delay between them.

By measuring the cross-correlation between these two signals we can find how far apart we need to adjust the time series to maximize their linear correlation. It is important to note that unlike the different causality measures discussed (Granger, Convergent Cross-mapping, etc.), cross-correlation does **not** give insights into causality - rather it only gives correlations.

### Example 1: Cross-Correlations in the Jena Climate Data

Let us look at how cross-correlations can be used to unearth relationships in real data. We will again use the Jena Climate Dataset to analyze the average temperature. This time, we will look at the relationship between daily temperature and daily wind speed.

df = pd.read_csv('../data/jena_climate_2009_2016.csv')
display(df.head())
data = df.iloc[:, 1:].astype(float).to_numpy()
temp = data[:, 1][::144]  # Temperature (in degrees Celsius) at one sample/day
wind = data[:, 10][::144]  # Wind Speed (in m/s) at one sample/day
dur = len(temp)
samp_rate = 1

plt.subplots(figsize=(15, 2))
plt.plot(range(len(temp)), temp, label='Temperature')
plt.plot(range(len(wind)), wind, label='Wind Velocity')
plt.legend();

We rescale the graphs so that we can easily visualize them.

plt.subplots(figsize=(15, 2))
plt.plot(range(len(temp)), (temp-np.mean(temp)) /
         np.std(temp), label='Temperature')
plt.plot(range(len(wind)), (wind-np.mean(wind)) /
         np.std(wind), label='Wind Velocity')
plt.ylabel('Value')
plt.xlabel('Time (in days)')
plt.legend();

From previous chapters, we know that these time series are stationary. With stationarity, we can proceed with applying the CCF to these two time series.

plt.subplots(figsize=(15, 2))

ccf_12 = ccf(temp, wind, unbiased=False)
plt.ylabel('Correlation Coefficient')
plt.xlabel('Lag $h$')
plt.stem(np.linspace(0, dur, samp_rate*dur), ccf_12)
peak = np.argmax(ccf_12)/samp_rate
plt.axvline(peak, c='r')
print('Maximum at:', peak, '(in $s$)')

We find that the shift maximizing the correlation between the daily values of temeperature and wind speed are 178 samples. An interesting result is that the correlation at $s=0$ is at a minimum. Both of these suggest that temperature and wind speed are periodic functions mirror each other in that when one is maximized, the other is minimized. We estimate then that the wind speed leads the temperature values by 178 days.

It should be noted that unlike Granger Causality which also draws relationships between multiple time series, cross-correlation does not provide us with the causality between the two. At best, cross-correlation shows the synchronocity between two time series.

### Example 2: Cross-Correlations Between Precipitation and Flow in the Sudbury River

We apply cross-correlation analysis on the daily stream flow of a river and the precipitation in its surrounding area. The data will be coming from the United States Geological Survey's (USGS) Real-time Water Data for the Nation and covers the Subdury River at Saxonville, Massachusetts for the year 2000. It comprises of the daily mean flow of water measured by the the USGS station. Understanding the causes and dynamics of flow in this specific river can have profound implications on managing the integity and diversity of its ecosystem.

flow_df = pd.read_csv('../data/cc/flowsud_2000.txt', sep='\t')
flow_df.columns = ['USGS', 'site_no', 'datetime', 'flow', '-']

flow = flow_df['flow']

rain_df = pd.read_csv('../data/cc/weather_2000.txt')
rain_df['Prcp'] = rain_df['Prcp'].replace('T', 0.0005).astype(float)

rain = rain_df['Prcp']

dur = len(rain)
samp_rate = 1

plt.subplots(figsize=(15, 2))
plt.plot(flow, label='Stream Flow')
plt.plot(rain, label='Precipitation')
plt.legend();

Once again, we rescale the graphs so that we can easily visualize them.

plt.subplots(figsize=(15, 2))
plt.plot((rain-np.mean(rain))/np.std(rain), label='Preciptation (exaggerated)')
plt.plot((flow-np.mean(flow))/np.std(flow), label='Stream Flow')
plt.xlabel('Day of the Year')
plt.legend();

We perform the Augmented Dickey-Fuller test on the two time series to figure out if they are stationary. If they are indeed stationary, we can proceed to performing the cross-correlation.

result = adfuller(flow)
print('ADF Statistic: %f' % result[0])
print('p-value: %f' % result[1])
print('Critical Values:')
for key, value in result[4].items():
    print('\t%s: %.3f' % (key, value))

result = adfuller(rain)
print('ADF Statistic: %f' % result[0])
print('p-value: %f' % result[1])
print('Critical Values:')
for key, value in result[4].items():
    print('\t%s: %.3f' % (key, value))

Both time series are stationary and we proceed with applying the CCF.

plt.subplots(figsize=(15, 2))

ccf_12 = ccf(flow, rain, unbiased=False)
plt.ylabel('Correlation Coefficient')
plt.xlabel('Lag $h$')
plt.stem(np.linspace(0, dur, samp_rate*dur), ccf_12)
peak = np.argmax(ccf_12)/samp_rate
plt.axvline(peak, c='r')
print('Maximum at:', peak, '(in $days$)')

We find that the maximum correlation happens when precipitation leads stream flow by 1 day. While cross-correlation alone cannot give us causality, understanding of the measurements involved can be used to argue such. In the case of rainfall and precipitation, there is established understanding that precipitation leads to increases in river flow. We can conclude with some confidence that precipitation will cause an increase in river flow after 1 day.

To supplement our analysis, we move into the frequency domain with the Fourier Transform by first understanding the Fourier Series.

## The Fourier Series:

Any periodic signal can be broken down into a summation of sine and cosine waves. A periodic time series $X(t)$ can be broken down into fundamental frequencies given by the Fourier Series:

$\begin{align} y(t) = \frac{a_0}{2}  + \sum_{n=1}^{\infty}{a_n cos(nt)) + b_n sin(nt)} \end{align}$

Integrating both sides by $\int+{-\pi}^{\pi} cos(mt)dt$ where $m$ is an integer:

$\begin{align} \int_{-\pi}^{\pi}{y(t)cos(mt)dt} = \frac{1}{2} a_0 + \sum_{n=1}^{\infty}{\int_{-\pi}^{\pi}a_n cos(nt)cos(mt)dt + \sum_{n=1}^{\infty}\int_{-\pi}^{\pi}b_n sin(nt) cos(mt)dt} \end{align}$

Via orthogonality between $cos$ and $sin$ over $[-\pi,\pi]$:

$\begin{align} \int_{-\pi}^{\pi}{cos(nt)cos(mt)dt} = 0,   m\neq n \end{align}$
$\begin{align} \int_{-\pi}^{\pi}{sin(nt)cos(mt)dt} = 0,   all m,n \end{align}$
$\begin{align} \int_{-\pi}^{\pi}{cos(nt)sin(mt)dt} = 0,   m\neq n \end{align}$

Equation 4 reduces to:

$\begin{align} \int_{-\pi}^{\pi}{y(t)cos(mt)dt} = a_m \int_{-\pi}^{\pi}cos^{2}(mt)dt \end{align}$

Solving for this a_m, we find that:

$\begin{align} a_m = \frac{1}{\pi}\int_{-\pi}^{\pi}{y(t) cos(mt)dt} \end{align}$

Doing the same for b_m, we find the similar coefficient:

$\begin{align} b_m = \frac{1}{\pi}\int_{-\pi}^{\pi}{y(t) sin(mt)dt} \end{align}$

for $n =0,1,2...,$ where:

$n = 0$ is the constant component of the signal

$n = 1$ is the \textit{fundamental} corresponding to a sine/cosine wave whose period matches $y(t)$'s period exactly

$n = 2$ is the first harmonic corresponding to a sine/cosine whose period matches $\frac{1}{2}$ of $y(t)$'s period

$n = 3$ is the second harmonic corresponding to a sine/cosine whose period matches $\frac{1}{3}$ of $y(t)$'s period

One useful quantity to look at is the energy of a signal as it tells us how "much" of a signal there is. We define a single coefficient $c_n^{2}$ to describe the "energy" of a signal at a specific frequency where $c_n$ is given by:

$\begin{align}c_n = \left[\frac{1}{2}(a^2_n + b^2_n)\right]^{\frac{1}{2}}\end{align}$

The square of these coefficients ($c_n^{2}$) corresponds to the energy of a certain frequency $n$.

## The Fourier Transform:

Transforming a signal from the time domain into the frequency ($\omega$) domain using this method is called the \textbf{Fourier Transform (FT)}. The FT is given by:

\begin{align} \hat{y}(\omega) = \int_{-\infty}^{\infty}{y(t)e^{-2\pi i \omega t}d\omega} \end{align}

To better understand the FT, let us take a look at some of its applications.

### FT of a Pure Signal

samp_rate = 500  # Sampling Frequency in Hertz
dur = 5  # Duration in Seconds
amp = 10  # Amplitude of Sine Wave
freq = 5  # Frequency of Sine Wave
x, y = sine_wave(amp, freq, samp_rate, dur, plot=True)

Let's work with ideal conditions first. Let's apply the Fourier Transform on the single sine wave with frequency of 5 Hz.

y_ft = scp.fft.fft(y)
freqs = scp.fft.fftfreq(x.shape[-1])*samp_rate
plt.subplots(figsize=(5, 5))
plt.plot(freqs, y_ft.real**2)
plt.ylabel('Energy')
plt.xlabel('Frequency (in Hz)')
print('Peak at:', np.argmax(y_ft.real**2)/dur, 'Hz')

We see two peaks when the frequency is 5 and -5.

The left-hand side corresponds to negative frequencies. Physically, negative frequencies are not very meaningful but their presence is important in calculating for the Fourier Transform. By finding where the energy peaks, we can find frequencies that best describe the time series.

Additionally, the FT is a perfectly (to numerical precision) reversible process. By knowing the energies of each frequency, we can retrieve the original signal by performing an \textbf{Inverse Fourier Transform (IFT)}.

plt.subplots(figsize=(15, 2))
plt.plot(x, scp.fft.ifft(y_ft).real)
plt.xlabel('Frequency')
print("Is close:", np.allclose(scp.fft.ifft(y_ft), y))

### FT on Mixed Signals

An important use of the FT is analyzing signals with mixed frequencies. We can perform the above exercise on 2 combined waves (or even an arbitrary number of waves with differing frequencies!) to find which frequencies are present in our signal.

samp_rate = 500  # Sampling Frequency in Hertz
dur = 5  # Duration in Seconds

amp1 = 1  # Amplitude of Sine Wave 1
freq1 = 5  # Frequency of Sine Wave 1

amp2 = 1  # Amplitude of Sine Wave 2
freq2 = 17  # Frequency of Sine Wave 2

x1, y1 = sine_wave(amp1, freq1, samp_rate, dur, plot=False)
x2, y2 = sine_wave(amp2, freq2, samp_rate, dur, plot=False)
y = y1 + y2  # Simply overlapping the two signals
plt.subplots(figsize=(15, 2))
plt.xlabel('Time')
plt.plot(x1, y);

Eyeballing the graph, it is challenging to determine its exact frequency make up.

y_ft = scp.fft.fft(y)
freqs = scp.fft.fftfreq(x1.shape[-1])*samp_rate

plt.subplots(figsize=(5, 5))
plt.plot(freqs, y_ft.real**2)
plt.xlabel('Frequency (in Hz)')
peaks = scp.signal.find_peaks(y_ft.real**2)
peak_vals = peaks[0][:len(peaks[0])//2]/dur
print('Peaks at:', peak_vals)

The FT makes it very easy to find the frequencies of the signal even if they are mixed.

## Noise Filtering

Real time series data are never free from noise making it tricky to find the real dynamics of whatever it is measuring. We may end up trying to model noise and end up overlooking the real behavior of the time series.

Thankfully, we can manipulate time series data in the frequency domain to eliminate one of the most types of noise - white noise. White noise pervades time series data and by definition, has components across all frequencies. Usually, a time series with white noise is represented as:

\begin{align} y(t) = s(t) + v(t) \end{align}

where $s(t)$ is the true signal we want to retrieve from $y(t)$ and $v(t)$ is white noise we want to eliminate. We simulate this below by adding white noise to the signal.

### Filtering by Energy Level

White noise is characterized by having energy levels almost equal for all frequencies. Such a signal can be generate by simply using a random number generator.

noise = np.random.uniform(-1,1, len(y))
plt.subplots(figsize=(15, 2))
plt.plot(x1, noise);

Taking the FT of the white noise, we find that indeed, for almost all the frequencies, there are positive energy values for them all. This means that for all frequencies, the white noise will have some contribution which can affect our analysis of the time series.

noise_ft = scp.fft.fft(noise)
freqs = scp.fft.fftfreq(x1.shape[-1])*samp_rate
peaks = scp.signal.find_peaks(noise_ft.real**2)

plt.subplots(figsize=(5, 5))
plt.plot(freqs, noise_ft.real**2)
plt.ylabel('Energy')
plt.xlabel('Frequency');

The addiiton of white noise to the mixed frequency signal makes its actual behavior less observable. The white noise has "corrupted" the signal and presents a challenge we find when making any measurement. How can we sift out the original mixed signal when the data we receive is noisy?

y_noisy = y+noise
plt.subplots(figsize=(15, 2))
plt.plot(x1, y_noisy);

The resulting FT of the noisy signal results in multiple peaks being found. This poses a problem if we are trying to find the appropriate periodic trends in our data. Luckily, the noise appears to be present only at low energy levels. Should measurements have been done reasonably well in collecting this data, noise can be managed as they only have low energy levels.

y_ft = scp.fft.fft(y_noisy)
freqs = scp.fft.fftfreq(x1.shape[-1])*samp_rate
peaks = scp.signal.find_peaks(y_ft.real**2)
peak_vals = peaks[0][:len(peaks[0])//2]/dur
print('Peaks at:', peak_vals)

plt.subplots(figsize=(5, 5))
plt.plot(freqs, y_ft.real**2)
plt.ylabel('Energy')
plt.xlabel('Frequency');

In crude fashion, we can set a hard threshold that removes the coefficients of frequencies below a certain energy level . One way to accomplish this is by setting the cut-off as a multiple of the standard deviation. The Fourier Transform allows us to make such

\textit{Conversely, we can perform the same method to eliminate high energy signals or even select those in a certain band}

threshold = 3*np.std(y_ft.real)
peaks = scp.signal.find_peaks(y_ft.real, threshold=threshold)
peak_vals = peaks[0][:len(peaks[0])//2]/dur
print('Peaks at:', peak_vals)

By visual inspection and setting an arbitrary threshold of 3x the standard deviation, we are able to retrieve a clearer signal of the dominant frequencies, albeit with the likely addiiton of some stray frequencies. Obtaining the exact signal from a noisy one can be very difficult but with judicious application and an understanding of the underlying systme, filtering by energy levels can be a quick and effective way to modify your time series.

y_ft_denoised = [0 if x < threshold else x for x in y_ft]
plt.subplots(figsize=(15, 2))
y_denoised = scp.fft.ifft(y_ft_denoised)
plt.plot(x1, y_denoised.real);

### Filtering by Frequency

hf_noise = sine_wave(amp, 70, samp_rate, dur, plot=False)[1]

y_hf_noisy = y+hf_noise
plt.subplots(figsize=(15, 2))
plt.plot(x, y_hf_noisy);

Observing the FT of a signal with high frequency noise, the noise dominates the energy spectrum and masks the underlying signal.

y_ft_hf = scp.fft.fft(y_hf_noisy)
freqs = scp.fft.fftfreq(x.shape[-1])*samp_rate
plt.plot(freqs, y_ft_hf.real**2)
peaks = scp.signal.find_peaks(y_ft_hf.real)
peak_vals = peaks[0][:len(peaks[0])//2]/dur
plt.ylabel('Energy')
plt.xlabel('Frequency')
print('Peaks at:', peak_vals)

While the loewr frequency peaks of the signal can still be retrieved, the extreme noise will make it untenable for most forecasting purposes. These high frequencies* can be eliminated to retrieve the original signal. Guided by our desire to observe the signals that exhibit lower frequencies, we can force the energies of all frequencies above a desired threshold to 0 and perform the IFT to obtain the

\textbf{*}\textit{We can do the same for low frequencies and those in a "band".}

freqs = scp.fft.fftfreq(x.shape[-1])*samp_rate
freqs

f_thresh = 20

freqs = scp.fft.fftfreq(x.shape[-1])*samp_rate
y_ft_hf_filtered = y_ft_hf.copy()
y_ft_hf_filtered[(abs(freqs.real) > f_thresh)] = 0

plt.plot(freqs, y_ft_hf_filtered.real**2)
plt.ylabel('Energy')
plt.xlabel('Frequency');
peaks = scp.signal.find_peaks(y_ft_hf_filtered.real)
peak_vals = peaks[0][:len(peaks[0])//2]/dur
print('Peaks at:', peak_vals)

plt.subplots(figsize=(15, 2))
y_lf = scp.fft.ifft(y_ft_hf_filtered).real
plt.plot(x1, y_lf.real);

### Example 3: Filtering via FT in the Jena Climate Dataset

Looking at climate information, we can glean insights into the seasonality of climate patterns. Let's examine the mean daily temperature of the Jena climate dataset and apply FT to examine its seasonality.

df = pd.read_csv('../data/jena_climate_2009_2016.csv')
display(df.head())
df['Date Time'] = pd.to_datetime(
    df['Date Time'], dayfirst=True, format="%d.%m.%Y %H:%M:%S")
df['Date'] = df['Date Time'].dt.date
mean_temps = df.groupby('Date')['T (degC)'].mean().values
plt.subplots(figsize=(15, 2))
plt.plot(temp)

There is a very strong trend that can be seen every year. Suppose we want to examine only this trend as we are interested in the general dynamics of the system. To highlight that aspect of the time series, we filter out frequencies corresponding to greater than 1 year by setting their energies to 0.

y_ft = scp.fft.fft(temp)
y_ft_copy = y_ft.copy()
freqs = scp.fft.fftfreq(len(temp), d=1/365)
y_ft_copy[(abs(freqs) >= 1)] = 0
plt.plot(freqs, abs(y_ft_copy))

By removing all frequencies greater than 1 year, we obtain only the lower frequency values. With these values, we can have a better look at the yearly dynamics of daily temperature without interference from inter-year dynamics. Should forecasts on actual values of the temperature need to be made, the value sof the frequencies removed can simply be added back.

y_hf = scp.fft.ifft(y_ft_copy)
plt.subplots(figsize=(15, 2))
# plt.title('Removing Yearly Cycle')
plt.plot(range(len(temp)), y_hf.real)

## Wavelet Transform

A wavelet transform (WT) allows you to measure how "much"of a certain type of wavelet there exists in a given signal. While both FT and WT both transform a time series into the frequency domain, a key difference is that WT gives you information about the time domain as well.

This transformation is achieved by imagining a specified wavelet sliding it across the entire signal through time. This wavelet that we scan is called the "mother wavelet". In addition to sliding the mother Wavelet across the span of the time series, its size, dependent on a scale factor, is varied as well while scanning. The scaling transformation applied on the mother Wavelet can be thought of as "stretching" it only without changing its frequency.

The wavelet transform of a time series $y(t)$ is given by:

$\begin{align} WT(y(t)) = \frac{1}{\sqrt{|\sigma|}}\int_{-\infty}^{\infty}y(t)\psi \left( \frac{t-h}{\sigma} \right)dt \end{align}$

where $\sigma$ is the scale, $h$ is the time lag/translation, and $\psi$ is the mother wavelet.

From the WT, we obtaing sets of coefficients corresponding to different time lags and scales. The greater the magnitude of the coefficients, the greater overlap there is between the scaled, time-shifted mother wavelet and the time series.

def rescale(arr, scale=2):
    n = len(arr)
    return np.interp(np.linspace(0, n, scale*n+1), np.arange(n), arr)

fig, ax = plt.subplots(figsize=(15, 2))
wav1 = pywt.ContinuousWavelet('gaus1')
int_psi1, x = pywt.integrate_wavelet(wav1)
for s in range(5):
    ax.plot(rescale(int_psi1, scale=s), label='Scale: '+ str(s))
    ax.legend()
    ax.set_title('Gaussian Mother Wavelet')

Suppose we have a signal of repearing Gaussian waves. Applying the WT on this signal using a mother wavelet corresponding to that of a Gaussian wavelet, we can pinpoint where the time series exhibits a Gaussian-like pattern.

s1 = list(scp.signal.gaussian(250, std=20))
s1 = np.tile(s1, 4)
plt.subplots(figsize=(15, 2))
plt.xlim(left=0, right=1000)
plt.ylabel('Value')
plt.xlabel('Time (t)')
plt.plot(s1)
plt.show()

coeffs, freqs = pywt.cwt(s1, range(1,250), wavelet='gaus1')
plt.subplots(figsize=(18.75, 2))
plt.xlim(left=0, right=1000)
plt.ylabel('Scale (s)')
plt.xlabel('Time (t)')
plt.imshow(coeffs, aspect='auto')
plt.colorbar();

By observing where the magnitudes of the WT are greatest, we can pinpoint the time and scale wherein there is Gaussian-like behavior in our time series. The WT is also useful in sifting out our signal from noise. We apply white noise once more to our Gaussian pulses and observe the WT's results.

s1 = list(scp.signal.gaussian(250, std=20))
noise = np.random.uniform(-1.5,1.5,4*len(s1))
s1 = np.tile(s1, 4) + noise
plt.subplots(figsize=(15, 2))
plt.xlim(left=0, right=1000)
plt.ylabel('Scale (s)')
plt.xlabel('Time (t)')
plt.plot(s1)
plt.show()

coeffs, freqs = pywt.cwt(s1, range(1,300), wavelet='gaus1')
plt.subplots(figsize=(18.75, 2))
plt.xlim(left=0, right=1000)
plt.ylabel('Scale (s)')
plt.xlabel('Time (t)')
plt.imshow(coeffs, aspect='auto')
plt.colorbar();

Regions of high magnitudes for our WT coefficients are still clearly visible albeit with minior interference from the white noise at smaller scales.

In general, choosing the mother wavelet depends on what kind of behavior you want or expect in your time series. Another popular mother wavelet used to identify sine waves in time series is the "Morlet" wavelet. This wavelet is a sine wave tapered by a Gaussian wave.

fig, ax = plt.subplots(figsize=(15, 2))
wav2 = pywt.ContinuousWavelet('morl')
int_psi2, x = pywt.integrate_wavelet(wav2)
for s in range(5):
    ax.plot(rescale(int_psi2, scale=s), label='Scale: '+ str(s))
    ax.set_title('Morlet Mother Wavelet')
    ax.legend()

Applying the Morlet wavelet on a mixed frequency sinusoidal signal, we can find that it is able to differentiate between the two frequencies as $s$ is varied.

sine5 = sine_wave(freq=5, duration =5, plot=False)[1]
sine10 = sine_wave(freq=10, plot=False)[1]
sine_5_10 = sine5+sine10
plt.subplots(figsize=(15, 2))
plt.plot(sine_5_10)
plt.show()
plt.subplots(figsize=(18.75, 2))
plt.xlim(left=0, right=1000)
plt.ylabel('Scale (s)')
plt.xlabel('Time (t)')
plt.imshow(coeffs, aspect='auto')
plt.colorbar();

### Example: 4 Using Accelerometer and Gyroscopic Data

The following code is taken from:

https://towardsdatascience.com/multiple-time-series-classification-by-using-continuous-wavelet-transformation-d29df97c0442

The following data contains time signals of accelerometer and gyroscopic data attached to people performing different activities. Throughout the duration of their activity, measurements were taken from the devices and recorded as time series. Each time series depicts a different activity.

def load_y_data(y_path):
    y = np.loadtxt(y_path, dtype=np.int32).reshape(-1, 1)
    # change labels range from 1-6 t 0-5, this enables a sparse_categorical_crossentropy loss function
    return y - 1

def load_X_data(X_path):
    X_signal_paths = [X_path + file for file in os.listdir(X_path)]
    X_signals = [np.loadtxt(path, dtype=np.float32) for path in X_signal_paths]
    return np.transpose(np.array(X_signals), (1, 2, 0))

PATH = '../data/cwt/'
LABEL_NAMES = ["Walking", "Walking upstairs",
               "Walking downstairs", "Sitting", "Standing", "Laying"]

# load X data
X_train = load_X_data(PATH + 'train/Inertial Signals/')
X_test = load_X_data(PATH + 'test/Inertial Signals/')
# load y label
y_train = load_y_data(PATH + 'train/y_train.txt')
y_test = load_y_data(PATH + 'test/y_test.txt')

Quite evidently, we find that walking has a different "look" from that of simply laying down. Humans can easily determine this at a glance but can be difficult for machines to differentiate between the two signals. The WT allows for the creation of images that contain temporal and frequency information providing a compact and meaningful representation of these time series.

fig, ax = plt.subplots(ncols=2, figsize=(15, 5))
pd.DataFrame(X_train[79]).plot(ax=ax[0], title='Walking')
pd.DataFrame(X_train[51]).plot(ax=ax[1], title='Laying');

def split_indices_per_label(y):
    indicies_per_label = [[] for x in range(0, 6)]
    # loop over the six labels
    for i in range(6):
        indicies_per_label[i] = np.where(y == i)[0]
    return indicies_per_label

def plot_cwt_coeffs_per_label(X, label_indicies, label_names, signal, sample, scales, wavelet):

    fig, axs = plt.subplots(nrows=2, ncols=3, sharex=True,
                            sharey=True, figsize=(12, 5))

    for ax, indices, name in zip(axs.flat, label_indicies, label_names):
        coeffs, freqs = pywt.cwt(
            X[indices[sample], :, signal], scales, wavelet=wavelet)
        ax.imshow(coeffs, cmap='coolwarm', aspect='auto')
        ax.set_title(name)
        ax.spines['right'].set_visible(False)
        ax.spines['top'].set_visible(False)
        ax.set_ylabel('Scale')
        ax.set_xlabel('Time')
    plt.tight_layout()

train_labels_indicies = split_indices_per_label(y_train)

# signal indicies: 0 = body acc x, 1 = body acc y, 2 = body acc z, 3 = body gyro x, 4 = body gyro y, 5 = body gyro z, 6 = total acc x, 7 = total acc y, 8 = total acc z
signal = 3  # signal index
sample = 1  # sample index of each label indicies list
scales = np.arange(1, 65)  # range of scales
wavelet = 'morl'  # mother wavelet

plot_cwt_coeffs_per_label(X_train, train_labels_indicies, LABEL_NAMES, signal,
                          sample, scales, wavelet)

for sample in range(2, 7):
    plot_cwt_coeffs_per_label(X_train, train_labels_indicies, LABEL_NAMES,
                              signal, sample, scales, wavelet)

Looking at the plots for each activity, we see can see some patterns between the images. By feeding these results into a convolutional neural network, we can create a classifier to classify time series into activities.

## Summary

To summarize, using these techniques can extract valuable information about time series which are not readily available.

Cross-correlations can give insights into the sychronicity of data. With careful analysis, these can even be used as a step towards a more holistic view of causality in conjunction with Granger Causality and Convergent Cross Mapping.

Fourier Transformation allows us to view the time series in the frequency domain, allowing operations previously untenable in the time domain. Filtering is a common task where FT is useful as it lets us easily manipulate the data to remove noise or highlight aspects of interest within the time series.

Wavelet Transformation lets us decompose time series more finely than the FT since we can define the "mother" wavelet to be our wavefunction of interest. By characterizing time series, we can quite literally, paint a picture of the time series that can be used to pick out specific waveforms in the time series.

In the context of forecasting, all three techniques can be used to inform the modeler how to prepare their data. With cross-correlations, the best time lagged time series can be used to provide better information about the target. The Fourier Transform can be applied to denoise the data and remove certain trends. Wavelet Transforms can be used to classify time series allowing the modeler to include their classification as a feature for forecasting or creating separate models for each class of time series. Again, these techniques serve as complements to predictive methods and can be used as feature engineering techniques to improve time series forecasing.

The following chapter represents the culmination of all the forecasting concepts from previous sections. There is no better way to test your time series capabilities than with a competition. The next chapter discusses the best techniques which were utilized in the recently concluded M5 Competition. In it, the best models under some constraints, are pushed to the limit to perform time series forecasting.

## References

The contents of this notebook are complied from the following references:

 *  Chapters 1-3 of Priestley, M.B. (1981). Spectral Analysis and Time Series, Vols. 1 and 2, Academic Press, New York.

 * <a href='https://towardsdatascience.com/multiple-time-series-classification-by-using-continuous-wavelet-transformation-d29df97c0442'>Multiple Time Series Classification by Using Continuous Wavelet Transformation</a>

 * <a href='https://pywavelets.readthedocs.io/en/latest/'>PyWavelets Documentation</a>

 * <a href='https://www.mathworks.com/help/wavelet/gs/interpreting-continuous-wavelet-coefficients.html'>Interpreting Continuous Wavelet Coefficients</a>

 *
Ngui, W. K., Leong, M. S., Hee, L. M., & Abdelrhman, A. M. (2013). Wavelet Analysis: Mother Wavelet Selection Methods. Applied Mechanics and Materials, 393, 953–958. https://doi.org/10.4028/www.scientific.net/amm.393.953

import numpy,import scipy,from statsmodels


# packages, rich is terminal output formatting
import numpy as np
import math
import rich
from rich import print, pretty
import matplotlib.pyplot as plt
pretty.install()

# GMB Model
# log(x_T) ~ N(log(x_S)+(mu-sigma^2/2)(T-S),sigma^2(T-S))

def SimulateGBM(S0, r, sd, T, paths, steps, reduce_variance = True):
    steps = int(steps)
    dt = T/steps
    Z = np.random.normal(0, 1, paths//2 * steps).reshape((paths//2, steps))
    # Z_inv = np.random.normal(0, 1, paths//2 * steps).reshape((paths//2, steps))
    if reduce_variance:
      Z_inv = -Z
    else:
      Z_inv = np.random.normal(0, 1, paths//2 * steps).reshape((paths//2, steps))
    dWt = math.sqrt(dt) * Z
    dWt_inv = math.sqrt(dt) * Z_inv
    dWt = np.concatenate((dWt, dWt_inv), axis=0)
    St = np.zeros((paths, steps + 1))
    St[:, 0] = S0
    for i in range (1, steps + 1):
        St[:, i] = St[:, i - 1]*np.exp((r - 1/2*np.power(sd, 2))*dt + sd*dWt[:, i-1])

    return St[:,1:]
    #return St

######## Test ##############################
S0_value = 36
r_value = 0.06
sd_value = 0.2
T_value = 1
paths_value = 100
steps_value = 50

Test_GBM = SimulateGBM(S0=S0_value, r=r_value, sd=sd_value, T=T_value, paths=paths_value,
steps=steps_value)

########## Plot the Data ################

Time_steps = np.arange(1, steps_value+1)

#plt.plot(Time_steps, Test_GBM.T)
#plt.show()

######################################################################


import numpy as np
''' Look up: https://docs.scipy.org/doc/numpy/reference/routines.random.html '''

def random_integer():
	np.random.seed(1337)

	d = np.random.randint(10, 20, size = (2, 3)) # integers in [low, high) as 2D array

	print(d)
	print('*' * 50)
	return d

def all_info(input_array):
	shape = input_array.shape
	print('Shape:', shape)
	print('Number of objects in the array:', input_array.size)
	print('Type:', input_array.dtype)
	print('Dimentionality:', len(shape))
	print('The array has', shape[0] , 'rows.')
	print('The array has', shape[1] , 'columns.')
	print('*' * 50)

def sum_all_elements(array):
	print('Sum of all elements: ', array.sum())
	return array.sum()

def sum_specific_dir(d, dir):
	if dir == 0:
		sum_dir = d.sum(axis = dir)
		print('Iterated over rows')
		print('Sum of each column: ', sum_dir)
	if dir == 1:
		sum_dir = d.sum(axis = dir)
		print('Iterated over columns')
		print('Sum of each row: ', sum_dir)

	return sum_dir

def find_min(array):
	minimum = array.min()
	return minimum

def find_max(array):
	maximum = array.min()
	return maximum

def find_mean(array):
	mean_value = array.mean()
	return mean_value

if __name__ == "__main__":

	print("Random integers")
	d = random_integer()
	all_info(d)
	sum_of_all = sum_all_elements(d)

	sum_column = sum_specific_dir(d, dir = 0) # Iteration over rows gives sum of for each column
	min_sum_columns = find_min(sum_column)
	print('Minimum value of the sums for columns:', min_sum_columns)

	sum_rows = sum_specific_dir(d, dir = 1) # Iteration over columns gives sum of for each rows
	max_sum_rows = find_max(sum_rows)
	print('Max value of the sums for rows:',max_sum_rows)

	print('Mean value of all values in the array:', find_mean(d))


from scipy.spatial.distance import pdist, squareform
from scipy import exp
from scipy.linalg import eigh
import numpy as np

# Implementing a kernel principal component analysis in Python

def rbf_kernel_pca1(X, gamma, n_components):
    """
    RBF kernel PCA implementation.

    Parameters
    ------------
    X: {NumPy ndarray}, shape = [n_examples, n_features]

    gamma: float
      Tuning parameter of the RBF kernel

    n_components: int
      Number of principal components to return

    Returns
    ------------
     X_pc: {NumPy ndarray}, shape = [n_examples, k_features]
       Projected dataset

    """
    # Calculate pairwise squared Euclidean distances
    # in the MxN dimensional dataset.
    sq_dists = pdist(X, 'sqeuclidean')

    # Convert pairwise distances into a square matrix.
    mat_sq_dists = squareform(sq_dists)

    # Compute the symmetric kernel matrix.
    K = exp(-gamma * mat_sq_dists)

    # Center the kernel matrix.
    N = K.shape[0]
    one_n = np.ones((N, N)) / N
    K = K - one_n.dot(K) - K.dot(one_n) + one_n.dot(K).dot(one_n)

    # Obtaining eigenpairs from the centered kernel matrix
    # scipy.linalg.eigh returns them in ascending order
    eigvals, eigvecs = eigh(K)
    eigvals, eigvecs = eigvals[::-1], eigvecs[:, ::-1]

    # Collect the top k eigenvectors (projected examples)
    X_pc = np.column_stack([eigvecs[:, i]
                            for i in range(n_components)])

    return X_pc

def rbf_kernel_pca2(X, gamma, n_components):
    """
    RBF kernel PCA implementation.

    Parameters
    ------------
    X: {NumPy ndarray}, shape = [n_examples, n_features]

    gamma: float
      Tuning parameter of the RBF kernel

    n_components: int
      Number of principal components to return

    Returns
    ------------
     alphas: {NumPy ndarray}, shape = [n_examples, k_features]
       Projected dataset

     lambdas: list
       Eigenvalues

    """
    # Calculate pairwise squared Euclidean distances
    # in the MxN dimensional dataset.
    sq_dists = pdist(X, 'sqeuclidean')

    # Convert pairwise distances into a square matrix.
    mat_sq_dists = squareform(sq_dists)

    # Compute the symmetric kernel matrix.
    K = exp(-gamma * mat_sq_dists)

    # Center the kernel matrix.
    N = K.shape[0]
    one_n = np.ones((N, N)) / N
    K = K - one_n.dot(K) - K.dot(one_n) + one_n.dot(K).dot(one_n)

    # Obtaining eigenpairs from the centered kernel matrix
    # scipy.linalg.eigh returns them in ascending order
    eigvals, eigvecs = eigh(K)
    eigvals, eigvecs = eigvals[::-1], eigvecs[:, ::-1]

    # Collect the top k eigenvectors (projected examples)
    alphas = np.column_stack([eigvecs[:, i]
                              for i in range(n_components)])

    # Collect the corresponding eigenvalues
    lambdas = [eigvals[i] for i in range(n_components)]

    return alphas, lambdas


from basic_operations import Basic_Operations
import numpy as np

class Operations_One_Matrix(Basic_Operations):
    """This class performs basic operations with a matrix
    """

    def __init__(self, matrix1=None, matrix2=None):
        super().__init__()
        self.matrix1 = self.check_type_matrix(matrix1)
        self.matrix2 = self.check_type_matrix(matrix2)

    def scalar(self, number):
        """scalar : multiply a matrix by a scalar

        Args:
            number: number to multiply by a matrix

        Return : result of the operation between matrices of type <<numpy.ndarray>>.
                If an error occurs, it returns a message indicating what the error is.
        """
        result = self.matrix1 * number
        return result

    def determinant(self):
        """determinant:  calculates the determinant of a matrix
        Return : an integer
        """
        determinat = np.linalg.det(self.matrix1)
        return determinat

    def shape(self):
        """shape: checks the dimensions of a matrix

        Return: Tuple of array dimensions.
        """
        dimensions = self.matrix1.shape
        return dimensions

    def inverse(self):
        """inverse : calculate the inverse of a matrix

        Return : result of the operation between matrices of type <<numpy.ndarray>>.
                If an error occurs, it returns a message indicating what the error is.
        """
        try:
            if self.determinant() != 0 and (self.shape()[0] == self.shape()[1]):
                inverse = np.linalg.inv(self.matrix1)
            else:
                raise Exception

        except Exception as e:
            return f"Error: {e}"
        return inverse

    def transpose(self):
        """transpose: Invert or permute the axes of a matrix
        Return: returns the modified array with the matrix transpose
        """
        try:
            transpose = np.transpose(self.matrix1)
            return transpose

        except Exception as e:
            return f"Error: {e}"


"""
This file contains the implementation of the Newton-Cotes rules
"""

import numpy as np

def rectangle(x, f):
    """
    Compute a 1D definite integral using the rectangle (midpoint) rule
    Parameters
    ----------
    f : function
        User defined function.
    x : numpy array
        Integration domain.
    Returns
    -------
    I : float
        Integration result.
    """
    a = x[0]
    b = x[1]
    ya = f((a+b)/2)
    I = (b-a) * ya
    return I

def trapz(x, f):
    """
    Compute a 1D definite integral using the trapezoidal rule
    Parameters
    ----------
    f : function
        User defined function.
    x : numpy array
        Integration domain.
    Returns
    -------
    I : float
        Integration result.
    """
    a = x[0]
    b = x[1]
    ya = f(a)
    yb = f(b)
    I = (b-a) * (ya + yb) / 2
    return I

def simpson(x, f):
    """
    Compute a 1D definite integral using Simpson's rule.
    Parameters
    ----------
    f : function
        User defined function.
    x : numpy array
        Integration domain.
    Returns
    -------
    I : float
        Integration result.
    """
    a = x[0]
    b = x[1]
    ya = f(a)
    yb = f((a+b)/2)
    yc = f(b)
    I = (b-a) * (ya + 4 * yb + yc) / 6
    return I

def simpson3_8(x, f):
    """
    Compute a 1D definite integral using the 3/8 Simpson's rule.
    Parameters
    ----------
    f : function
        User defined function.
    x : numpy array
        Integration domain.
    Returns
    -------
    I : float
        Integration result.
    """
    a = x[0]
    b = x[1]
    ya = f(a)
    yb = f((2*a+  b)/3)
    yc = f((  a+2*b)/3)
    yd = f(b)
    I = (b-a) * (ya + 3 * (yb + yc) + yd) / 8
    return I

def boole(x, f):
    """
    Compute a 1D definite integral using Boole's rule.
    Parameters
    ----------
    f : function
        User defined function.
    x : numpy array
        Integration domain.
    Returns
    -------
    I : float
        Integration result.
    """
    a = x[0]
    b = x[1]
    ya = f(a)
    yb = f((3*a+  b)/4)
    yc = f((  a+  b)/2)
    yd = f((  a+3*b)/4)
    ye = f(b)
    I = (b-a) * (7 * (ya + ye) + 32 * (yb + yd) + 12 * yc) * 2 / 45
    return I


import numpy as np
from copy import deepcopy

class PermutationHandler():
    def __init__(self, n, k=None, random_seed=0): #n choose k
        if k is None: k = n
        self.n = n
        self.k = k
        self.n_perms = np.math.factorial(n)//np.math.factorial(n-k)
        self.random_seed = random_seed
        self.rng = np.random.RandomState(seed=random_seed)

    def generate_random_permutation(self):
        n, k = self.n, self.k
        item = []
        a = list(range(n))
        for i in range(k):
            entry_idx = self.rng.randint(n-i)
            item.append(a[entry_idx])
            del a[entry_idx]
        return tuple(item)

    def get_generator(self, count=None):
        if count is None:
            for idx in range(self.n_perms):
                yield self.index_to_permutation(idx)
        else:
            for idx in range(count):
                yield self.generate_random_permutation()

    def index_to_permutation(self, idx):
        assert idx < self.n_perms, "Index must be less than {}".format(self.n_perms)
        n_perms = self.n_perms
        n, k = self.n, self.k
        a = list(range(n))
        item = []
        for i in range(k):
            stride = n_perms // (n-i)
            entry_index = int(idx // stride)
            idx = idx - entry_index * stride
            item.append(a[entry_index])
            del a[entry_index]
            n_perms = n_perms//(n-i)
        return tuple(item)

    def permutation_to_index(self, item):
        n_perms = self.n_perms
        n, k = self.n, self.k
        a = list(range(n))
        idx = 0
        for i in range(len(item)):
            idx = idx + n_perms//(n-i) * a.index(item[i])
            a.remove(item[i])
            n_perms = n_perms//(n-i)
        return int(idx)

    def mutate(self, item, pos=None):
        item = list(item)
        n, k = self.n, self.k
        if pos is None: pos = self.rng.randint(k)
        perturbation = self.rng.randint(n-1) + 1
        existing_val = item[pos]
        new_val = (existing_val + perturbation) % n
        try:
            new_val_idx = item.index(new_val)
        except:
            new_val_idx = None
        if new_val_idx is not None:
            item[new_val_idx] = existing_val
        item[pos] = new_val
        return tuple(item)

    def crossover(self, item1, item2):
        item1, item2 = list(item1), list(item2)
        n, k = self.n, self.k
        pos = 1 + self.rng.randint(k-1)
        citem1 = item1[:pos] + [entry for entry in item2 if entry not in item1[:pos]][-k+pos:]
        citem2 = item2[:pos] + [entry for entry in item1 if entry not in item2[:pos]][-k+pos:]
        citem3 = [entry for entry in item2 if entry not in item1[pos:]][:pos] + item1[pos:]
        citem4 = [entry for entry in item1 if entry not in item2[pos:]][:pos] + item2[pos:]
        return tuple(citem1), tuple(citem2), tuple(citem3), tuple(citem4)


# Multivariate Linear Regression
# Includes data normalization

import numpy as np
from sklearn import datasets, linear_model

def partial_derivative(theta, trainX, trainY, trainSize):
    return trainX.T.dot((trainX.dot(theta) - trainY)) / trainSize

def normalize(data):
    mean = np.mean(data, axis=0)
    stdDeviation = np.std(data, axis=0)
    return (data - mean) / stdDeviation

def fit(trainX, trainY, learningRate=0.1, threashold=1E-5):
    trainSize, featureSize = trainX.shape
    theta = np.array(np.ones((featureSize, 1)), dtype=float) # Initialize theta
    derivative = partial_derivative(theta, trainX, trainY, trainSize)
    while abs(derivative).max() > threashold: # While gradient doesn't converge
        theta -= learningRate * derivative # Perform gradient descent
        derivative = partial_derivative(theta, trainX, trainY, trainSize) # Update partial derivative
    return theta

def standard_fit(trainX, trainY):
    trainX = trainX[:, 1:] # Remove the extra bias term
    model = linear_model.LinearRegression()
    model.fit(trainX, trainY)
    return np.append(model.intercept_, model.coef_)[np.newaxis].T # Returns the hypothesis in a column vector

def accuracy(theta, testX, testY):
    # Returns the R Squared value of the model
    explained = testX.dot(theta)
    mean = np.sum(testY) / len(testY)
    totSumSquares = np.sum((testY - mean) ** 2)
    resSumSquares = np.sum((testY - explained) ** 2)
    accuracy = 1 - resSumSquares / totSumSquares
    return accuracy

dataset = datasets.load_diabetes()
data, target = dataset.data, dataset.target
dataSize = data.shape[0] # Number of rows (entries)
data = normalize(data) # Normalize the data
data = np.hstack((np.ones((dataSize, 1)), data)) # Add an extra bias term
target = target[np.newaxis].T # Transpose target vector into column vector
separation = int(dataSize * 0.8) # Percentage of training data set and testing data set
trainX, trainY = data[:separation, :], target[:separation]
testX, testY = data[separation:, :], target[separation:]

hypothesis = standard_fit(trainX, trainY)
print('Standard Fitted Model: {}'.format(hypothesis.T))
acc = accuracy(hypothesis, testX, testY)
print('Standard Fitted Accuracy: {:.4f}\n'.format(acc))

hypothesis = fit(trainX, trainY, 0.1, 1E-6)
print('Fitted Model: {}'.format(hypothesis.T))
acc = accuracy(hypothesis, testX, testY)
print('Fitted Accuracy: {:.4f}'.format(acc))


'''
    Problem 69 : Totient maximum
'''
from time import time
from numba import jit
import numpy as np

@jit
def getPrimes(n):
    q, r = divmod(n, 2)
    r = q + r
    seive = np.ones(r)
    seive[0] = 0

    lim = int(n**0.5 / 2) + 1

    for i in range(1, lim + 1):
        p = 2 * i + 1
        if seive[i]:
            sp = 2 * i * (i + 1)
            while (sp < r):
                seive[sp] = 0
                sp = sp + p

    def_prime = np.asarray([2])
    seived_primes = np.asarray(np.nonzero(seive)).flatten() * 2 + 1
    primes = np.concatenate((def_prime, seived_primes))
    return primes

@jit
def get_phi(n):
    if n in primes_list:
        return n - 1

    res = n
    for p in primes_list:
        r = n % p
        if r == 0:
            while r == 0:
                n = n // p
                r = n % p

            res = res // p
            res = res * (p - 1)
        elif p * p > n:
            break

    if n > 1:
        res = res // n
        res = res * (n - 1)

    return res

if __name__ == "__main__":
    t0 = time()
    N = 1000000
    primes_list = getPrimes(int(N**0.5))
    print("Primes computed in {:.3f} secs".format(time() - t0))
    a, r = 0, 0
    for i in range(2, N + 1):
        phi = get_phi(i)
        # print(i," ", phi)
        r1 = i / phi
        if r1 > r:
            a, r = i, r1

    print("Result : ", a, " , ratio :", r)
    print("Done in {:.3f} secs".format(time() - t0))


import numpy as np
import pandas as pd
import math
import matplotlib.pyplot as plt

# loads the dataset
data = pd.read_csv('../dataset/3D_spatial_network.csv')
N = len(data)
# N = 100000

# splits the dataset into training and testing
train_set_size = int(0.7 * N)
test_set_size = int(0.3 * N)

def rms_calc(w1, w2, w0):
    '''
    Calculates the RMS Error given w1, w2, w0
    '''
    rms_error = 0.0
    for data_index in range(test_set_size):

        index = train_set_size + data_index - 1
        y = data.iloc[index, 3]
        x1 = data.iloc[index, 1]
        x2 = data.iloc[index, 2]

        error = abs(y - ((w1 * x1) + (w2 * x2) + (w0)))
        rms_error += (error * error)
        # adding the square of error to rms_error

    # dividing the Rms error with dataset size
    rms_error /= test_set_size
    rms_error = math.sqrt(rms_error)
    print("RMS Error:", rms_error)
    return rms_error

def r2_error(w1, w2, w0):
    '''
    Calculates the R2 error given w1, w2, w0
    '''
    # t_mean of the target variable
    mean = np.mean(data.iloc[train_set_size:, 3])
    tss = 0
    rss = 0

    for data_index in range(test_set_size):

        index = train_set_size + data_index - 1
        y = data.iloc[index, 3]
        x1 = data.iloc[index, 1]
        x2 = data.iloc[index, 2]

        tss += ((y - mean) * (y - mean))
        rss += math.pow((y - ((w1 * x1) + (w2 * x2) + (w0))), 2)

    # calculates the R2 value
    r2 = 1 - (rss / tss)
    print("r2 error: ", r2)
    return r2

# Equation: y = w1x1 + w2x2 + w0
# loads the training dataset target values
Y = data.iloc[0: train_set_size, 3]
Y = Y.to_numpy()

X = (data[0: train_set_size].to_numpy())[:, [0, 1, 2]]
X[:, 0] = 1

temp1 = np.dot(X.T, X)
temp2 = np.linalg.inv(temp1)
temp3 = np.dot(X.T, Y)

thetha = np.dot(temp2, temp3)
w0 = thetha[0]
w1 = thetha[1]
w2 = thetha[2]

print(thetha)
print("RMSE Error: ", rms_calc(w1, w2, w0))
print("R2 Error: ", r2_error(w1, w2, w0))


"""
The purpose of this module is to perform a single Euler step on a step
function with the WENO method.

The initial conditions are

    q(t=0, x) = 1_{ x > 0 }

That is, q = 1 if x > 0, and q = 0 otherwise.

Required Modules:
-----------------

    * sympy - symbolic toolbox for Python.
"""

from weno_diff import central_diff1, central_diff2, ConstructIntegratedF, LinearReconstruct

import sympy

# Quick accessors

dx = sympy.Symbol("dx", real=True )    # spatial resolution
dt = sympy.Symbol("dt", real=True )    # time step
nu = sympy.Symbol("nu", real=True )    # CFL number
u  = sympy.Symbol("u" , real=True )    # advection speed

dt = nu*dx/u

qim5 = 0
qim4 = 0
qim3 = 0
qim2 = 0
qim1 = 0
qi   = 1
qip1 = 1
qip2 = 1
qip3 = 1
qip4 = 1

# Expansion of f:
#
#    F \approx f_i - (u*dt)/2 u_{i,x} + (u*dt)^2/3 u_{i,xx}
#
Fim3 = sympy.simplify( ConstructIntegratedF( [ qim5, qim4, qim3, qim2, qim1 ], u, dx, dt ) )
Fim2 = sympy.simplify( ConstructIntegratedF( [ qim4, qim3, qim2, qim1, qi   ], u, dx, dt ) )
Fim1 = sympy.simplify( ConstructIntegratedF( [ qim3, qim2, qim1, qi  , qip1 ], u, dx, dt ) )
Fi   = sympy.simplify( ConstructIntegratedF( [ qim2, qim1, qi  , qip1, qip2 ], u, dx, dt ) )
Fip1 = sympy.simplify( ConstructIntegratedF( [ qim1, qi  , qip1, qip2, qip3 ], u, dx, dt ) )
Fip2 = sympy.simplify( ConstructIntegratedF( [ qi  , qip1, qip2, qip3, qip4 ], u, dx, dt ) )

Fimh = sympy.simplify( LinearReconstruct( [Fim3, Fim2, Fim1, Fi, Fip1] ) )
Fiph = sympy.simplify( LinearReconstruct( [Fim2, Fim1, Fi, Fip1, Fip2] ) )

# Update is q_i^{n+1} = q_i^n - dt ( Fiph - Fim2 ) / dx
stability_polynomial = sympy.collect( sympy.expand(1 - dt*(Fiph - Fimh)/dx), nu )

print('Stability polynomial for Taylor PIF-WENO is ')
sympy.pretty_print( stability_polynomial )
print('Stability polynomial for Taylor PIF-WENO is ')
print( stability_polynomial )

# Try this with a simpler scheme that has a known solution:
#forward_euler = sympy.collect( 1 - (u*dt)*( qi - qim1 ) / dx, exp(-I*k*dx) )
#print('Stability polynomial for Forward Euler is ')
#sympy.pretty_print( forward_euler )


import numpy as np
#import time

A = np.matrix('1 2 -2 1; 2 5 -2 3; -2 -2 5 3; 1 3 3 2')
b = np.matrix('4; 7; -1; 0')

def lu_decomposition(A):
    # if A.shape[0]!=A.shape[1]: implement exception

    dimension = A.shape[0]

    L = np.zeros(shape=(dimension,dimension))
    U = np.zeros(shape=(dimension,dimension))

    for i in range(dimension):
        L[i,i] = 1.0

    for i in range(dimension):
        for j in range(dimension):
            sum = 0.0

            if j >= i:
                for k in range(i+1):
                    sum += L[i,k] * U[k,j]

                U[i,j] = A[i,j] - sum
            else:
                for k in range(j+1):
                    sum += L[i,k] * U[k,j]

                L[i,j] = (A[i,j] - sum)/U[j,j]

    return L,U

def solveReverse(L, b):
    dimension = L.shape[0]

    y = np.zeros(shape = (dimension,1))

    for i in range(dimension):
        sum = 0

        for j in range(dimension):
            if j<i:
                sum += L[i][j]*y[j]

        y[i] = b[i] - sum

    return y

def solveNormal(U,y):
    dimension = U.shape[0]

    x = np.zeros(shape = (dimension,1))

    for i in range(dimension-1,-1,-1): #range([start], stop[, step])
        sum = 0

        for j in range(0,dimension):
            if not(j == i):
                sum += U[i][j] * x[j]

        x[i] = (y[i] - sum)/ U[i][i]

    return x

#a=time.clock()

l,u = lu_decomposition(A)
y = solveReverse(l,b)
x = solveNormal(u,y)

#b = time.clock()

print "matriz L \n",l
print "matriz U \n",u
print "L * U \n",np.dot(l,u)
print "\nvetor y \n",y
print "\nvetor x \n",x

#print 'delta', b-a


import random
import numpy as np
import matplotlib.pyplot as plt

random.seed(2)
f = lambda x: x**2
a = 0.0

b = 3.0
NumSteps = 1000000
XIntegral=[]
YIntegral=[]
XRectangle=[]
YRectangle=[]

ymin = f(a)
ymax = ymin
for i in range(NumSteps):
    x = a + (b - a) * float(i) / NumSteps
    y = f(x)
    if y < ymin: ymin = y
    if y > ymax: ymax = y

A = (b - a) * (ymax - ymin)
N = 1000000
M = 0
for k in range(N):
    x = a + (b - a) * random.random()
    y = ymin + (ymax - ymin) * random.random()
    if y <= f(x):
            M += 1
            XIntegral.append(x)
            YIntegral.append(y)
    else:
            XRectangle.append(x)
            YRectangle.append(y)
NumericalIntegral = M / N * A
print ("Numerical integration = " + str(NumericalIntegral))

XLin=np.linspace(a,b)
YLin=[]
for x in XLin:
    YLin.append(f(x))

plt.axis   ([0, b, 0, f(b)])
plt.plot   (XLin,YLin, color="red" , linewidth="4")
plt.scatter(XIntegral, YIntegral, color="blue", marker   =".")
plt.scatter(XRectangle, YRectangle, color="yellow", marker   =".")
plt.title  ("Numerical Integration using Monte Carlo method")
plt.show()


# Implementation of cross-entropy error function using numpy.
# Compare the random weight results with the closed form solution (0,4,4).

# Code Flow:
    # 1. Import all relevant libraries.
    # 2. Generate sample data from std. normal distribution.
    # 3. Create random weights for plotting.
    # 4. Define sigmoid function.
    # 5. Feedforward calculation.
    # 6. Cross-entropy error function.
    # 7. Print cross-entropy error from random weights.
    # 8. Closed-form solution.
    # 9. Print cross-entropy error from closed form solution.

# 1. Import all relevante libraries:
import numpy as np

# 2. Create input data:
N = 100 # No. of samples
D = 2 # No. of features

# Random data generation:
X = np.random.randn(N,D)

# Center the first 50 points at (-2,-2)
X[:50,:] = X[:50,:] - 2*np.ones((50,D))

# Center the last 50 points at (2, 2)
X[50:,:] = X[50:,:] + 2*np.ones((50,D))

# Labels: first 50 are 0, last 50 are 1
T = np.array([0]*50 + [1]*50) # Random labels

# Bias term: Add a column of ones
ones = np.ones((N, 1))
Xb = np.concatenate((ones, X), axis=1)

# 3. Create random weights:
w = np.random.randn(D + 1)

# 4. Sigmoid function
def sigmoid(z):
    return 1/(1 + np.exp(-z))

# 5. Feedforward:
z = Xb.dot(w)
Y = sigmoid(z)

# 6. Cross-entropy error function:
# Section 3B from jupyter notebook
def cross_entropy(T, Y):
    E = 0
    for i in range(len(T)):
        if T[i] == 1:
            E -= np.log(Y[i])
        else:
            E -= np.log(1 - Y[i])
    return E

# 7. Print cross-entropy error from random weights:
print(cross_entropy(T, Y))

# 8. Closed-form solution:
# Section 3A from jupyter notebook
w = np.array([0, 4, 4])

# calculate the model output
z = Xb.dot(w)
Y = sigmoid(z)

# 9. Print cross-entropy error from closed form solution:
print(cross_entropy(T, Y))
print('As we expect the error is much lower with our closed form solution!')

2.Logistic Regression/1.Code - Using Theory/1.Cross_Entropy_Function.py


import torch
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import DataLoader,random_split,Subset
from sklearn.model_selection import LeaveOneOut,KFold
from pathlib import Path
from dataset import data_generator,RegressionDataset
from opt import parse_args

opt=parse_args()

# Using RMSE loss as loss function
def RMSE_loss(yhat,y):
    mse=((yhat - y) ** 2).sum() / yhat.data.nelement()
    return torch.sqrt(mse)

# Perform Linear Regression AX=b
def linear_regression(x,weight):
    outputs=[]
    for i in x:
        out=i.view([1,len(i)]).mm(weight)
        outputs.append(out.numpy())
    return outputs

# Get the best weight using QR Decomposition
"""def get_best_weight(x,y,_lambda=0):
    Q,R=torch.qr(x)
    pseudo_inv=R.pinverse(_lambda).mm(Q.t())
    weight=torch.mm(pseudo_inv,y)
    return weight"""

# Get the best weight by the formulat mention in the course
def get_best_weight(x,y,_lambda=0):
    x_t=x.t()

    if _lambda==0:
        pseudo_inv=torch.inverse(x_t.mm(x))
    else:   # if regularization is needed
        mat=x_t.mm(x)
        I=torch.eye(mat.size(0))
        pseudo_inv=torch.inverse(mat+_lambda*I)

    weight=torch.mm(pseudo_inv.mm(x_t),y)
    return weight

# get polynomial features when polynomial regression with degree>1
def get_poly_features(x,degree):
    poly_x=[]
    for each_x in x:
        poly_term=[each_x**n for n in range(1,degree+1)]
        poly_x.append(poly_term)
    return poly_x

# draw the fitting plot
def draw_fitting_plot(train_set,range,w_1,deg_1,legend,title,fname,w_2=None,deg_2=None,w_3=None,deg_3=None,w_4=None,deg_4=None):
    train_X=[]
    train_Y=[]

    for i,(inputs,labels) in enumerate(train_set):
        train_X.append(inputs)
        train_Y.append(labels)

    train_X=np.asarray(train_X)
    train_Y=np.asarray(train_Y)
    plt.scatter(train_X,train_Y,facecolors='none',edgecolors='b')

    X=np.linspace(range[0],range[1],100)

    deg_1_X=get_poly_features(X,deg_1)
    deg_1_X=np.hstack((deg_1_X,np.ones((len(deg_1_X),1))))
    deg_1_X=torch.FloatTensor(deg_1_X)
    w_1=torch.FloatTensor(w_1)
    deg_1_Y=linear_regression(deg_1_X,w_1)
    deg_1_Y=torch.FloatTensor(deg_1_Y)
    deg_1_Y=torch.flatten(deg_1_Y)
    plt.plot(X,deg_1_Y,'r',linewidth=0.5)

    if w_2 is not None:
        deg_2_X=get_poly_features(X,deg_2)
        deg_2_X=np.hstack((deg_2_X,np.ones((len(deg_2_X),1))))
        deg_2_X=torch.FloatTensor(deg_2_X)
        w_2=torch.FloatTensor(w_2)
        deg_2_Y=linear_regression(deg_2_X,w_2)
        deg_2_Y=torch.FloatTensor(deg_2_Y)
        deg_2_Y=torch.flatten(deg_2_Y)
        plt.plot(X,deg_2_Y,'g',linewidth=0.5)

    if w_3 is not None:
        deg_3_X=get_poly_features(X,deg_3)
        deg_3_X=np.hstack((deg_3_X,np.ones((len(deg_3_X),1))))
        deg_3_X=torch.FloatTensor(deg_3_X)
        w_3=torch.FloatTensor(w_3)
        deg_3_Y=linear_regression(deg_3_X,w_3)
        deg_3_Y=torch.FloatTensor(deg_3_Y)
        deg_3_Y=torch.flatten(deg_3_Y)
        plt.plot(X,deg_3_Y,'deepskyblue',linewidth=0.5)

    if w_4 is not None:
        deg_4_X=get_poly_features(X,deg_4)
        deg_4_X=np.hstack((deg_4_X,np.ones((len(deg_4_X),1))))
        deg_4_X=torch.FloatTensor(deg_4_X)
        w_4=torch.FloatTensor(w_4)
        deg_4_Y=linear_regression(deg_4_X,w_4)
        deg_4_Y=torch.FloatTensor(deg_4_Y)
        deg_4_Y=torch.flatten(deg_4_Y)
        plt.plot(X,deg_4_Y,'darkviolet',linewidth=0.5)

    plt.legend(legend,fontsize=16)
    plt.xlabel('x',fontsize=14)
    plt.ylabel('y',fontsize=14)
    plt.title(title,fontsize=14)
    plt.savefig(fname)
    plt.clf()

# Testing
def test(test_set,degree,weight):
    test_X=[]
    test_Y=[]

    for i,(inputs,labels) in enumerate(test_set):
        test_X.append(inputs)
        test_Y.append(labels)

    test_X=get_poly_features(test_X,degree)
    test_X=np.hstack((test_X,np.ones((len(test_X),1))))
    test_X=torch.FloatTensor(test_X)
    test_Y=torch.FloatTensor(test_Y)
    test_Y=test_Y.view([test_Y.size(0),1])

    weight=torch.FloatTensor(weight)

    outputs=linear_regression(test_X,weight)
    outputs=torch.FloatTensor(outputs)
    outputs=outputs.view([outputs.size(0),1])

    loss=RMSE_loss(outputs,test_Y)
    avg_test_loss=loss/test_X.size(0)

    print(f'testing_loss: {avg_test_loss:.4f}\n')

# Training when the cross validation is five folds
def kf_train(dataset,degree=1,_lambda=0):
    kf=KFold(n_splits=5,random_state=None,shuffle=False)

    total_training_loss=0.0
    total_valid_loss=0.0
    total_weight=np.empty((degree+1,1))

    for train_idx,valid_idx in kf.split(dataset):   # Split the data to train and valid
        train_set=Subset(dataset,train_idx)
        valid_set=Subset(dataset,valid_idx)

        train_X=[]
        train_Y=[]

        ##########          training          ##########
        for i,(inputs,labels) in enumerate(train_set):
            train_X.append(inputs)
            train_Y.append(labels)

        train_X=get_poly_features(train_X,degree)
        train_X=np.hstack((train_X,np.ones((len(train_X),1))))
        train_X=torch.FloatTensor(train_X)
        train_Y=torch.FloatTensor(train_Y)
        train_Y=train_Y.view([train_Y.size(0),1])

        weight=get_best_weight(train_X,train_Y,_lambda)
        total_weight+=weight.numpy()

        outputs=linear_regression(train_X,weight)
        outputs=torch.FloatTensor(outputs)

        loss=RMSE_loss(outputs,train_Y)
        total_training_loss+=loss/train_X.size(0)

        valid_X=[]
        valid_Y=[]

        ##########          validation          ##########
        for i,(inputs,labels) in enumerate(valid_set):
            valid_X.append(inputs)
            valid_Y.append(labels)

        valid_X=get_poly_features(valid_X,degree)
        valid_X=np.hstack((valid_X,np.ones((len(valid_X),1))))
        valid_X=torch.FloatTensor(valid_X)
        valid_Y=torch.FloatTensor(valid_Y)
        valid_Y=valid_Y.view([valid_Y.size(0),1])

        outputs=linear_regression(valid_X,weight)
        outputs=torch.FloatTensor(outputs)

        loss=RMSE_loss(outputs,valid_Y)
        total_valid_loss+=loss/valid_X.size(0)

    best_weight=total_weight/dataset.__len__()
    avg_training_loss=total_training_loss/dataset.__len__()
    avg_valid_loss=total_valid_loss/dataset.__len__()

    print(f'training_loss: {avg_training_loss:.4f}\tvalid_loss: {avg_valid_loss:.4f}')

    return best_weight

# Training when the cross validation is leave one out
def loo_train(dataset,degree=1,_lambda=0):
    loo=LeaveOneOut()

    total_training_loss=0.0
    total_valid_loss=0.0
    total_weight=np.zeros((degree+1,1))

    for train_idx,valid_idx in loo.split(dataset):  # split the data to train and valid
        train_set=Subset(dataset,train_idx)
        valid_set=Subset(dataset,valid_idx)

        train_X=[]
        train_Y=[]

        ##########          training          ##########
        for i,(inputs,labels) in enumerate(train_set):
            train_X.append(inputs)
            train_Y.append(labels)

        train_X=get_poly_features(train_X,degree)
        train_X=np.hstack((train_X,np.ones((len(train_X),1))))
        train_X=torch.FloatTensor(train_X)
        train_Y=torch.FloatTensor(train_Y)
        train_Y=train_Y.view([train_Y.size(0),1])

        weight=get_best_weight(train_X,train_Y,_lambda)
        total_weight+=weight.numpy()

        outputs=linear_regression(train_X,weight)
        outputs=torch.FloatTensor(outputs)
        outputs=outputs.view([outputs.size(0),1])

        loss=RMSE_loss(outputs,train_Y)
        total_training_loss+=loss/train_X.size(0)

        valid_X=[]
        valid_Y=[]

        ##########          validation          ##########
        for i,(inputs,labels) in enumerate(valid_set):
            valid_X.append(inputs)
            valid_Y.append(labels)

        valid_X=get_poly_features(valid_X,degree)
        valid_X=np.hstack((valid_X,np.ones((len(valid_X),1))))
        valid_X=torch.FloatTensor(valid_X)
        valid_Y=torch.FloatTensor(valid_Y)
        valid_Y=valid_Y.view([valid_Y.size(0),1])

        outputs=linear_regression(valid_X,weight)
        outputs=torch.FloatTensor(outputs)
        outputs=outputs.view([outputs.size(0),1])

        loss=RMSE_loss(outputs,valid_Y)
        total_valid_loss+=loss

    best_weight=total_weight/dataset.__len__()
    avg_training_loss=total_training_loss/dataset.__len__()
    avg_valid_loss=total_valid_loss/dataset.__len__()

    print(f'training_loss: {avg_training_loss:.4f}\tvalid_loss: {avg_valid_loss:.4f}')
    return best_weight

def main():

    ##########          Linear Regression with degree=1,5,10,14          ##########
    if Path('./data/data.csv').is_file()==False:
        data_generator('linear','./data/data.csv',20)

    data_set=RegressionDataset('./data/data.csv')
    total_sz=data_set.__len__()
    test_sz=int(0.25*total_sz)
    train_set,test_set=random_split(data_set,[total_sz-test_sz,test_sz])

    origin_train_set=train_set
    origin_test_set=test_set

    print("Regression Linear Leave One Out Degree 1 20 Data Points")
    weight_loo_deg1=loo_train(train_set,1)
    test(test_set,1,weight_loo_deg1)

    print("Regression Linear Five Fold Degree 1 20 Data Points")
    weight_kf_deg1=kf_train(train_set,1)
    test(test_set,1,weight_kf_deg1)

    print("Regression Linear Leave One Out Degree 5 20 Data Points")
    weight_loo_deg5=loo_train(train_set,5)
    test(test_set,5,weight_loo_deg5)

    print("Regression Linear Five Fold Degree 5 20 Data Points")
    weight_kf_deg5=kf_train(train_set,5)
    test(test_set,5,weight_kf_deg5)

    print("Regression Linear Leave One Out Degree 10 20 Data Points")
    weight_loo_deg10=loo_train(train_set,10)
    test(test_set,10,weight_loo_deg10)

    print("Regression Linear Five Fold Degree 10 20 Data Points")
    weight_kf_deg10=kf_train(train_set,10)
    test(test_set,10,weight_kf_deg10)

    print("Regression Linear Leave One Out Degree 14 20 Data Points")
    weight_loo_deg14=loo_train(train_set,14)
    test(test_set,14,weight_loo_deg14)

    print("Regression Linear Five Fold Degree 14 20 Data Points")
    weight_kf_deg14=kf_train(train_set,14)
    test(test_set,14,weight_kf_deg14)

    draw_fitting_plot(train_set,[-3,3],weight_loo_deg1,1,['degree=1','degree=5','degree=10','degree=14'],'Linear Data Live One Out Curve','./figure/linear-loo.jpg',weight_loo_deg5,5,weight_loo_deg10,10,weight_loo_deg14,14)
    draw_fitting_plot(train_set,[-3,3],weight_kf_deg1,1,['degree=1','degree=5','degree=10','degree=14'],'Linear Data Five Folds Curve','./figure/linear-kf.jpg',weight_kf_deg5,5,weight_kf_deg10,10,weight_kf_deg14,14)

    ##########          Linear Regression on Sine Curve Data with degree=5,10,14          ##########
    if Path('./data/sin_data.csv').is_file()==False:
        data_generator('sin','./data/sin_data.csv',20)

    data_set=RegressionDataset('./data/sin_data.csv')
    total_sz=data_set.__len__()
    test_sz=int(0.25*total_sz)
    train_set,test_set=random_split(data_set,[total_sz-test_sz,test_sz])

    print('Regression Sine Leave One Out Degree 5 20 Data Points')
    weight_loo_deg5=loo_train(train_set,5)
    test(test_set,5,weight_loo_deg5)

    print('Regression Sine Five Fold Degree 5 20 Data Points')
    weight_kf_deg5=kf_train(train_set,5)
    test(test_set,5,weight_kf_deg5)

    print('Regression Sine Leave One Out Degree 10 20 Data Points')
    weight_loo_deg10=loo_train(train_set,10)
    test(test_set,10,weight_loo_deg10)

    print('Regression Sine Five Fold Degree 10 20 Data Points')
    weight_kf_deg10=kf_train(train_set,10)
    test(test_set,10,weight_kf_deg10)

    print('Regression Sine Leave One Out Degree 14 20 Data Points')
    weight_loo_deg14=loo_train(train_set,14)
    test(test_set,14,weight_loo_deg14)

    print('Regression Sine Five Fold Degree 14 20 Data Points')
    weight_kf_deg14=kf_train(train_set,14)
    test(test_set,14,weight_kf_deg14)

    draw_fitting_plot(train_set,[0,1],weight_loo_deg5,5,['degree=5','degree=10','degree=14'],'Sine Data Live One Out Curve','./figure/sine-loo.jpg',weight_loo_deg10,10,weight_loo_deg14,14)
    draw_fitting_plot(train_set,[0,1],weight_kf_deg5,5,['degree=5','degree=10','degree=14'],'Sine Data Five Folds Curve','./figure/sine-kf.jpg',weight_kf_deg10,10,weight_kf_deg14,14)

    ##########          Linear Regression on Different Training Data Size          ##########
    if Path('./data/data_320.csv').is_file()==False:
        data_generator('linear','./data/data_320.csv',320)

    data_set=RegressionDataset('./data/data_320.csv')
    total_sz=data_set.__len__()
    delete_sz=260
    use_set,delete_set=random_split(data_set,[total_sz-delete_sz,delete_sz])

    total_sz=use_set.__len__()
    test_sz=int(0.25*total_sz)
    train_set,test_set=random_split(use_set,[total_sz-test_sz,test_sz])

    print('Regression Linear Leave One Out Degree 14 60 Data Points')
    weight_loo_data60=loo_train(train_set,14)
    test(test_set,14,weight_loo_data60)

    print('Regression Linear Five Fold Degree 14 60 Data Points')
    weight_kf_data60=kf_train(train_set,14)
    test(test_set,14,weight_kf_data60)

    total_sz=data_set.__len__()
    delete_sz=160
    use_set,delete_set=random_split(data_set,[total_sz-delete_sz,delete_sz])

    total_sz=use_set.__len__()
    test_sz=int(0.25*total_sz)
    train_set,test_set=random_split(use_set,[total_sz-test_sz,test_sz])

    print('Regression Linear Leave One Out Degree 14 160 Data Points')
    weight_loo_data160=loo_train(train_set,14)
    test(test_set,14,weight_loo_data160)

    print('Regression Linear Five Fold Degree 14 160 Data Points')
    weight_kf_data160=kf_train(train_set,14)
    test(test_set,14,weight_kf_data160)

    total_sz=data_set.__len__()
    delete_sz=0
    use_set,delete_set=random_split(data_set,[total_sz-delete_sz,delete_sz])

    total_sz=use_set.__len__()
    test_sz=int(0.25*total_sz)
    train_set,test_set=random_split(use_set,[total_sz-test_sz,test_sz])

    print('Regression Linear Leave One Out Degree 14 320 Data Points')
    weight_loo_data320=loo_train(train_set,14)
    test(test_set,14,weight_loo_data320)

    print('Regression Linear Five Fold Degree 14 320 Data Points')
    weight_kf_data320=kf_train(train_set,14)
    test(test_set,14,weight_kf_data320)

    draw_fitting_plot(origin_train_set,[-3,3],weight_loo_data60,14,['m=60','m=160','m=320'],'Linear Data Different m Live One Out Curve','./figure/data-m-loo.jpg',weight_loo_data160,14,weight_loo_data320,14)
    draw_fitting_plot(origin_train_set,[-3,3],weight_kf_data60,14,['m=60','m=160','m=320'],'Linear Data Different m Five Folds Curve','./figure/data-m-kf.jpg',weight_kf_data160,14,weight_kf_data320,14)

    ##########          Linear Regression with Regularization Term lambda          ##########
    print('Regularization 0.001/m Linear Five Fold Degree 14 20 Data Points')
    _lambda=0.001/20
    weight_kf_0001l=kf_train(origin_train_set,14,_lambda)
    test(origin_test_set,14,weight_kf_0001l)

    print('Regularization 1/m Linear Five Fold Degree 14 20 Data Points')
    _lambda=float(1)/20
    weight_kf_1l=kf_train(origin_train_set,14,_lambda)
    test(origin_test_set,14,weight_kf_1l)

    print('Regularization 1000/m Linear Five Fold Degree 14 20 Data Points')
    _lambda=float(1000)/20
    weight_kf_1000l=kf_train(origin_train_set,14,_lambda)
    test(origin_test_set,14,weight_kf_1000l)

    draw_fitting_plot(origin_train_set,[-3,3],weight_kf_0001l,14,['0.001/m','1/m','1000/m'],'Linear Data Five Fold with Regularization Curve','./figure/regularization-kf.jpg',weight_kf_1l,14,weight_kf_1000l,14)

if __name__=='__main__':
    main()


'''
solve Sudoku puzzles with backtracking algorithm.
(projecteuler.net, problem no. 96)
'''

import numpy as np, sys

def check(a, i, val):
    """
    Check if 'val' at position 'i' in 9x9 matrix 'a' satisfies sudoku rules.
    """
    i0=3*(i//3)
    mask=np.zeros((9,9))
    mask[i[0],:] = 1
    mask[:,i[1]]=1
    mask[i0[0]:i0[0]+3,i0[1]:i0[1]+3] = 1
    return np.all(a[mask==1]!=val)

def solve(a):
    """
    Uses backtracking to solve sudoku matrix 'a'. Returns True if solved successfully.
    """
    if a.all(): return True     # if no cell is empty, puzzle is solved
    i = np.argwhere(a==0)[0]
    for val in np.arange(1,10): # try each value and check conditions,
        if check(a,i,val):      # if satisfied, continue solving recursively
            a[i[0],i[1]] = val
            ok = solve(a)
            if ok: return True
            a[i[0],i[1]] = 0
    return False

if __name__ == '__main__':

    with open(sys.argv[1], 'r') as f:
        inp = f.readlines()
    raw = [''.join(inp[i:i+9]) for i in range(1,len(inp)+1,10)]
    res =[]
    for item in raw:
        a = np.array([list(map(int,i)) for i in item.strip().split('\n')],dtype=np.int64)
        solve(a)
        res.append(a)
    # ProjectEuler answer format
    res = sum([int(''.join([str(a[0,0]),str(a[0,1]),str(a[0,2])])) for a in res])
    print(res)


import numpy as np

print("\nMétodo de Newton-Raphson")

# Definindo a função
def f(x):
    return (x*np.log10(x)) - 1

# Definindo a derivada
def der(x):
    dx = 1e-10
    return (f(x+dx) - f(x)) / dx

# Implementação do método numérico
def newraph(tol, x):
    x_0 = x
    dif = 1
    iter = 0
    while dif > tol:
        x_1 = x_0 - (f(x_0)/der(x_0))
        dif = abs(x_1 - x_0)
        x_0 = x_1
        iter += 1
    print("\nRaiz:", x_1)
    print("Quantidade de iterações:", iter)
    print("x_1 - x_0 = ", dif, "\n")

# Atribuição dos parâmetros
tol =  1e-4
a_inicial = 2
b_inicial = 3
x_inicial = 3

newraph(tol, x_inicial)


""" Orbital Mechanics for Engineering Students Example 1.16
Question:
Expand the function sin(t+h) in a Taylor series around t=1.
Plot the Taylor series of orders 1 to 4 and compare them with
sin(1+h) for -2<h<2.
Written by: J.X.J. Bannwarth
"""
import matplotlib.pyplot as plt
from sympy import symbols, diff, sin, cos
from math import factorial
from numpy import array, linspace, dot, zeros
import numpy as np

# Title
print("Orbital Mechanics for Engineering Students Example 1.16")

# Equation to approximate
t = symbols('t')
g = sin(t)

# Linearisation point
tLin = 1.

# Step size
hs = linspace(-2.,2.,50)
gTrue = np.sin(tLin + hs)

# Find the taylor coefficients
orders = range(5)
taylorCoefsEqn = [diff(g, t, order)/factorial(order) for order in orders]
taylorCoefs = array([coef.evalf(subs={t:tLin}) for coef in taylorCoefsEqn])

# Calculate the terms of the Taylor series polynomials
termApprox = zeros((hs.shape[0],len(orders)))
for idx, h in enumerate(hs):
    powers = array([h**order for order in orders])
    termApprox[idx,:] = powers * taylorCoefs

# Calculate the sum of the terms
gApprox = zeros((hs.shape[0],len(orders)))
gApprox[:,0] = termApprox[:,0]
for idx in range(1,gApprox.shape[1]):
    gApprox[:,idx] = termApprox[:,0:idx+1].sum(axis=1)

# Plot the results
plt.figure()
plt.grid()
plt.plot(hs, gTrue, label=f"sin(1+h)")
for idx, order in enumerate(orders):
    plt.plot(hs, gApprox[:,idx], label=f"$p_{order}$")
plt.xlabel("$h$ (-)")
plt.ylabel("Value (-)")
plt.title("Example 1.16 Plots of zeroth- to fourth-order\nTaylor series expansions of sin(1-h)")
plt.legend()
plt.show()


import argparse
from numpy import sqrt

def n_prime_number(n: int) -> list:
	'''
	Return a list of the prime nums,
	 which contains precisely n elements.
	 Default Eratosthene algo

	n -- len of the list
	'''
	prime_list = []
	i = 2
	while len(prime_list) != n:
		for j in prime_list:
			if i % j == 0:
				break
		else:
			prime_list.append(i)

		i+=1

	return prime_list

if __name__ == "__main__":

	# creating parser
	parser = argparse.ArgumentParser(description="showing prime numbers")
	parser.add_argument(
		'--show-all',
		'-a',
		action="store_true",
		help="showing also all previous \
		prime numbers before N"
	)
	parser.add_argument(
		'--file',
		 '-f',
		 action="store",
		 type=argparse.FileType("w"),
		 help="file for output"
	)
	parser.add_argument(
		'number',
		type=int
	)

	namespace = parser.parse_args()

	output_line = "the {}(th or st, nd, rd or smth else) prime number is {}".format(
	namespace.number, n_prime_number(namespace.number)[-1])
	if namespace.show_all is None:
		print(output_line)

	if namespace.show_all:
		print("first n prime numbers == {}".format(n_prime_number(namespace.number)))
		if namespace.file:
			print(output_line, file=namespace.file)

	if namespace.file:
		print("Just check my message in {}".format(namespace.file))
		print(output_line, file=namespace.file)


# -*- coding: utf-8 -*-
"""
In the introduction of his MOOC "SMAC" (Statistical Mechanics: Algorithms and
Computations - https://www.coursera.org/learn/statistical-mechanics), Werner
Krauth propose a simple method to compute pi using a direct sampling
Monte Carlo simulation. A program is proposed in Python, in a version which
allows to do many runs of the function direct_pi(N). The code is written in a
style close to pseudocode used for algorithms, or classical coding style used
in C, Fortran,...

It is possible to write the function in a more "pythonic" way, or to use the
numpy numerical library, to improve compactness and efficiency.

Function direct_pi_DV(N) use pure python with list comprehension to eliminate
the for loop. The sum is directly made on the boolean comparison results to count
the number of true trials.

Function direct_pi_DV_np(N) use the numpy library to vectorize the loop, directly
square values and sum the array elements over the smaller axis. Again the sum
is directly made on the boolean comparisons.

Finally, in order to compare efficiency, the execution times of the three
versions have been measured using the timeit library.

Here is value obtain for a sample run :
direct_pi       : 3.5209695600005944 s
direct_pi_DV    : 4.000994963998892 s
direct_pi_Dv_np : 0.19237353700009407 s

The use ot the numpy library clearly improve the computer speed performance by
a factor about 20.
"""
import random, timeit
import numpy as np

def direct_pi(N):
    n_hits = 0
    for i in range(N):
        x, y = random.uniform(-1.0, 1.0), random.uniform(-1.0, 1.0)
        if x ** 2 + y ** 2 < 1.0:
            n_hits += 1
    return n_hits

def direct_pi_DV(N):
    return sum((random.uniform(-1,1)**2 + random.uniform(-1,1)**2) < 1 for i in range(N))

def direct_pi_DV_np(N):
    return np.sum((np.random.uniform(-1,1,(N,2))**2).sum(1)<1)

n_runs = 1000
n_trials = 4000

# running :
for run in range(n_runs):
    print(run, 4.0 * direct_pi(n_trials) / n_trials)

for run in range(n_runs):
    print(run, 4.0 * direct_pi_DV(n_trials) / n_trials)

for run in range(n_runs):
    print(run, 4.0 * direct_pi_DV_np(n_trials) / n_trials)

# timing three versions :
print(timeit.timeit('direct_pi('+str(n_trials)+')', "from __main__ import direct_pi", number=n_runs))
print(timeit.timeit('direct_pi_DV('+str(n_trials)+')', "from __main__ import direct_pi_DV", number=n_runs))
print(timeit.timeit('direct_pi_DV_np('+str(n_trials)+')', "from __main__ import direct_pi_DV_np", number=n_runs))


# -*- coding: utf-8 -*-
"""
Created on Fri Aug 30 15:36:39 2019

@author: Simón Cometto
"""

import numpy as np

def biseccion(f, a, b, tolerancia=False, iteraciones=False):
    ''' Encuentra el cero de f entre a y b mediante el método bisección
        Parámetros:
            ->  f: una función que, dado x, devuelve el valor de f(x)
            ->  a y b: números reales iniciales para el método
            ->  tolerancia: el menor intérvalo entre el cual se encuentra el cero
            ->  iteraciones: la cantidad de iteraciones a realizar en el algoritmo'''

    #Si a la función no se le pasa un parámetro para detener las iteraciones devuelve False
    if not (tolerancia or iteraciones):
        return False

    if (f(a)*f(b))<0: #Hay un cero en el medio
        #El algoritmo se ejecuta hasta llegar a un intervalo de tolerancia menor o igual a "tolerancia"
        if(tolerancia):
            #Mientras el intérvalo es mayor a la tolerancia
            while(abs(a-b)>tolerancia):
                #Encuentro el punto medio entre a y b
                c =  a + (abs(b-a)*0.5)

                #Si f(a) y f(c) tienen signo distinto, entonces hubo un cruce por cero entre a y b
                if(f(a)*f(c)<0):
                    b = c
                #Sino el cero se encuentra entre b y c
                else:
                    a = c
            return c
        elif iteraciones:
            for i in range(iteraciones):
                c = a + (abs(b - a) * 0.5)

                # Si f(a) y f(c) tienen signo distinto, entonces hubo un cruce por cero entre a y b
                if (f(a) * f(c) < 0):
                    b = c
                # Sino el cero se encuentra entre b y c
                else:
                    a = c
            return c
    else:    #No se sabe si hay cero en el medio
        pass
        #Ver que devolver en caso de que no haya un cero en el medio, o como implementarlo

def punto_fijo(g, x_i, n):
    ''' Encuentra el cero de f a partir de g.
        Parámetros:
            ->  g: surge de despejar x de f
            ->  x_i: número real inicial para el método
            ->  n: la cantidad de iteraciones'''
    for i in range(n):
        x_i = g(x_i)
    return x_i

if __name__ == "__main__":
    f = lambda x: 1*(x**2) - x - 2

    a = float(input("Ingrese a: "))
    b = float(input("Ingrese b: "))
    i = int(input("Ingrese la cantidad de iteraciones: "))

    cero_b = biseccion(f, a, b, iteraciones=i)
    g = lambda x: (x + 2) ** 0.5
    cero_pf = punto_fijo(g, b, i)
    print("Resultado método de bisección:  ", cero_b)
    print("Resultado método de punto fijo: ", cero_pf)


import numpy as np
import matplotlib.pyplot as plt

from ivp_solvers import IVP, IVPSolver

class PredatorPrey1(IVP):
    """
    Predator-prey taken from example 16.8 (pg. 496) of "A First Course in Numerical Methods" by Ascher and Greif
    """
    tf = 100
    y0 = np.array([80., 30.])
    yf = np.array([94.04588719, 38.11498521])  # For running tests

    def dy_dt(self, _, y):
        return np.array([
            0.25 * y[0] - 0.01 * y[0] * y[1],
            -y[1] + 0.01 * y[0] * y[1]
        ])

def plot_example(t, y, show=True):
    plt.plot(t, y)

    if show:
        plt.show()

def run_example(problem: IVP, solver: IVPSolver, plot=True, show=True):
    with Timer():
        t, y = solver.solve_problem(problem)

    if plot:
        plot_example(t, y, show=show)

if __name__ == '__main__':
    from runge_kutta.explicit import Midpoint
    from ivp_solvers import CustomFixedStepRKSolver, FixedEuler, FixedRK2, FixedRK4, FixedThreeEigthsRK4
    from shared_utils.tests import PerfTimerNS as Timer

    run_example(PredatorPrey1(), FixedEuler(step_size=0.01), show=False)
    run_example(PredatorPrey1(), CustomFixedStepRKSolver(0.1, Midpoint()), show=False)
    run_example(PredatorPrey1(), FixedRK2(step_size=0.1), show=False)
    run_example(PredatorPrey1(), FixedRK4(step_size=0.1), show=False)
    run_example(PredatorPrey1(), FixedThreeEigthsRK4(step_size=0.1), show=True)

    _t, _y = FixedThreeEigthsRK4(step_size=0.01).solve_problem(PredatorPrey1())

    plt.show()


import matplotlib.pyplot as plt
import numpy as np

x = np.array(list(range(0, 11)))
y = np.array(list(map(lambda n: n * n, x)))

def compute_cost_function(m, t0, t1, x, y):
    return 1 / 2 / m * sum([(t0 + t1 * np.asarray([x[i]]) - y[i])**2 for i in range(m)])

def gradient_descent(x, y, alpha=0.01, ep=0.0001, max_iter=1500):
    converged = False
    iter = 0
    m = x.shape[0]    # number of samples

    # initial theta
    t0 = 0
    t1 = 0

    # total error, J(theta)
    J = compute_cost_function(m, t0, t1, x, y)
    # print('J=', J);
    # Iterate Loop
    num_iter = 0
    while not converged:
        # for each training sample, compute the gradient (d/d_theta j(theta))
        grad0 = 1.0 / m * sum([(t0 + t1 * np.asarray([x[i]]) - y[i]) for i in range(m)])
        grad1 = 1.0 / m * sum([(t0 + t1 * np.asarray([x[i]]) - y[i]) * np.asarray([x[i]]) for i in range(m)])

        # update the theta_temp
        temp0 = t0 - alpha * grad0
        temp1 = t1 - alpha * grad1

        # update theta
        t0 = temp0
        t1 = temp1

        # mean squared error
        e = compute_cost_function(m, t0, t1, x, y)
        # print ('J = ', e)
        J = e    # update error
        iter += 1    # update iter

        if iter == max_iter:
            print('Max interactions exceeded!')
            converged = True

    return t0, t1

def graph_points(data_x, data_y):
    for x, y in zip(data_x, data_y):
        plt.scatter(x, y, c='r')

def graph_line(data_x, m, b):
    line = [(m * x) + b for x in data_x]
    plt.plot(data_x, line, c='b')

def squared_error(y_points, regression_line):
    return sum((regression_line - y_points)**2)

def coeff_of_determination(y_points, regression_line):
    y_mean_line = [np.mean(y_points) for y in regression_line]
    squared_error_regr = squared_error(y_points, regression_line)
    squared_error_y_mean = squared_error(y_points, y_mean_line)
    return 1 - (squared_error_regr / squared_error_y_mean)

graph_points(x, y)
b, m = gradient_descent(x, y)
print(m, b)
graph_line(x, m, b)
print(coeff_of_determination(y, m * x + b))
plt.show()


import numpy as np
# import scipy as sp
# import sklearn
# import matplotlib.pyplot as plt

class Lab1(object):
    def solver(self, A, b):
        return np.dot(np.linalg.inv(A), b)

    def fitting(self, x, y):
        xmatr = np.column_stack((x, np.ones(x.shape)))
        coeff = np.dot(np.linalg.pinv(xmatr), y)
        return coeff

    def naive5(self, X, A, Y):
        # Calculate the matrix with $(i,j$)-th entry as  $\mathbf{x}_i^\top A \mathbf{y}_j$ by looping over the rows of $X,Y$.
        qf = np.zeros((X.shape[0], Y.shape[0]))
        for i in range(X.shape[0]):
            for j in range(Y.shape[0]):
                qf[i, j] = np.dot(np.dot(X[i], A), Y[j])
        return qf

    def matrix5(self, X, A, Y):
        # Repeat part (a), but using only matrix operations (no loops!).
        return np.dot(np.dot(X, A), Y.T)

    def naive6(self, X, A):
        # Calculate a vector with $i$-th component $\mathbf{x}_i^\top A \mathbf{x}_i$ by looping over the rows of $X$.
        qf = np.zeros(X.shape[0])
        for i in range(X.shape[0]):
            qf[i] = np.dot(np.dot(X[i], A), X[i])
        return qf

    def matrix6(self, X, A):
        # Repeat part (a) using matrix operations (no loops!).
        return np.sum(np.dot(X, A)*X, axis=1)


import math
import numpy as np

def read_file(name):
    # Open file
    file = open(name, 'r')

    # Return content
    return file.read()

def exercise_1_generator(size, alphabet, probability):
    # Result
    results = ""

    # Loop to generate size-long text
    for _ in range(size):
        results += np.random.choice(alphabet, p=probability)

    # Return prepared text
    return results

def entropy(dictionary):
    # Entropy sum
    entropy_result = 0.0

    # Loop to analyze
    for key, value in dictionary.items():
        entropy_result += value * math.log(value, 2)

    # Return result
    return -entropy_result

def analyze_content(content):
    # Letters dictionary
    letters = {}

    # Letters counter to prepare probability of single letter
    counter = 0

    # Loop to analyze content
    for _, letter in enumerate(content):
        cardinality = letters.get(letter, 0)
        letters.update({letter: cardinality + 1})
        counter += 1

    # Loop to change cardinality to probability
    for letter in letters:
        letters.update({letter: letters.get(letter) / counter})

    # Return letters dictionary
    return letters

def analyze_characters(content, row, separator=''):
    # Results - dictionary with letters
    letters = {}

    # Last characters
    last_characters = []

    # Counter
    counter = 0

    # Loop to iterate
    for char in content:
        if len(last_characters) == row:
            # Fetch dictionary
            selected_dictionary = letters.get(separator.join(last_characters), {})

            # Update total counter
            total = selected_dictionary.get('--TOTAL--', 0)
            selected_dictionary.update({'--TOTAL--': total + 1})

            # Update counter this letter
            cardinality = selected_dictionary.get(char, 0)
            selected_dictionary.update({char: cardinality + 1})

            # Update selected dictionary
            letters.update({separator.join(last_characters): selected_dictionary})

            # Update counter
            counter += 1

        # Append char to list
        last_characters.append(char)

        # Check length - if greater than row -> delete first char (FIFO)
        if len(last_characters) > row:
            del (last_characters[0])

    # Return letters dictionary with counter
    return letters, counter

def cardinality_to_probability(dictionary, counter):
    key_dictionary = {}

    # Loop to change cardinality to probability
    for key, value_dict in dictionary.items():
        # Fetch cardinality from special key
        cardinality = value_dict.pop('--TOTAL--')

        # Loop to iterate - values in dictionary
        for value_key, value in value_dict.items():
            # Set value
            value_dict.update({value_key: value / cardinality})

        # Probability of key
        key_dictionary[key] = cardinality / counter

        # Update dictionary
        dictionary.update({key: value_dict})

    # Return updated dictionary
    return dictionary, key_dictionary

def conditional_entropy(dictionary, keys):
    # Entropy
    entropy_value = 0.0

    # Sum over keys - get all values
    for key, probability_x in keys.items():
        for value in dictionary.get(key).values():
            entropy_value += probability_x * value * math.log(value, 2)

    # Return calculated entropy
    return -entropy_value

def main():
    #
    # Exercise 1 - Random, all letters with probability 1/37
    #

    letters = list('qwertyuiopasdfghjklzxcvbnm0123456789 ')
    probability = [1/37 for _ in letters]
    size = 10_000
    text = exercise_1_generator(size=size, alphabet=letters, probability=probability)
    letters_dictionary = analyze_content(text)
    entropy_result = entropy(letters_dictionary)

    print('Exercise 1 - Raw, random text')
    print('\tEntropy =>', entropy_result)

    #
    # Exercise 1 - Analyze content from sample Wiki
    #

    file_name = '../Exercise_1/norm_wiki_sample.txt'
    content = read_file(file_name)
    size = 10_000
    letters_dictionary = analyze_content(content)
    text = exercise_1_generator(size=size, alphabet=list(letters_dictionary.keys()), probability=list(letters_dictionary.values()))
    letters_dictionary = analyze_content(text)
    entropy_result = entropy(letters_dictionary)

    print('Exercise 1 - Based on text')
    print('\tEntropy =>', entropy_result)

    #
    # Exercise 2
    #

    files = ['norm_wiki_en.txt', 'norm_wiki_la.txt', 'sample0.txt', 'sample1.txt', 'sample2.txt', 'sample3.txt', 'sample4.txt', 'sample5.txt']
    rows = [1, 2, 3, 4, 5]

    for file_name in files:
        print('File ' + str(file_name))
        content_char = read_file(file_name)
        content_words = content_char.split()

        print('\tContent [characters] =>', entropy(analyze_content(content_char)))
        print('\tContent [words] =>', entropy(analyze_content(content_words)))

        for row in rows:
            words, counter = analyze_characters(content_char, row, separator='')
            dictionary, keys = cardinality_to_probability(words, counter)
            print('\t- chars @ ' + str(row) + ' =>', conditional_entropy(dictionary, keys))

            words, counter = analyze_characters(content_words, row, separator=' ')
            dictionary, keys = cardinality_to_probability(words, counter)
            print('\t- words @ ' + str(row) + ' =>', conditional_entropy(dictionary, keys))

        print('-------------------')

if __name__ == '__main__':
    main()


from lam.eigen.eigenvalue import *
import json
from lam.readtext.readtext import readtext
import sympy as sp
import sympy.core.numbers as nu
import numpy as np
class MyEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.integer):
            return int(obj)
        elif isinstance(obj, np.floating):
            return float(obj)
        elif isinstance(obj, np.ndarray):
            return obj.tolist()
        elif isinstance(obj,sp.core.numbers.Integer):
            return str(obj)
        else:
            return str(obj)
def getlambdamat(mat: sp.MutableDenseMatrix):#得到矩阵λE-A
    lambdamatrix = -mat
    lambdamatrix1 = sp.eye(mat.shape[0])
    x = sp.symbols("lambda")
    lambdamatrix1 = lambdamatrix1 * x
    lambdamatrix = lambdamatrix + lambdamatrix1
    return lambdamatrix
def getlambdamatvalue(mat: sp.MutableDenseMatrix):#得到列表[λ1E-A,λ2E-A,.....λiE-A]
    lambdamatrix = -mat
    lambdamatrix1 = sp.eye(mat.shape[0])
    eigenvalues=list(mat.eigenvals().keys())
    list0=[]
    for i in range(len(eigenvalues)):
        lambdamatrix2=lambdamatrix1*eigenvalues[i]
        lambdamatrix3 = lambdamatrix + lambdamatrix2
        list0.append(lambdamatrix3)
    return list0
def slveigengetcharpoly(a:str):#特征多项式
    eigenSolver = EigenSolver(readtext(a))
    p = eigenSolver.getCharpoly()
    p=sp.latex(p)
    json_str = json.dumps(p)
    return json_str
def slveigenvalue(a:str):#特征值
    eigenSolver = EigenSolver(readtext(a))
    p = eigenSolver.getEigenvalues()
    p = sp.latex(p)
    json_str = json.dumps(p)
    return json_str
def slveigenvectors(a:str):#特征向量
    eigenSolver = EigenSolver(readtext(a))
    eigenvectors = eigenSolver.getEigenvectors()
    eigenvectors_0 = []
    for i in range(len(eigenvectors)):
        p = []
        for j in range(len(eigenvectors[i][2])):
            p.append(sp.latex(eigenvectors[i][2][j]))
        m = [sp.latex(eigenvectors[i][0]), sp.latex(eigenvectors[i][1]), p]
        eigenvectors_0.append(m)
    eigenvectors=eigenvectors_0
    return eigenvectors
def slveigenCourse(a:str):#（特征值求解过程）
    eigenSolver=EigenSolver(readtext(a))
    p=eigenSolver.get_course()
    matrix=p['matrix']
    lambdamat = getlambdamat(matrix)
    lambdamatvalue = getlambdamatvalue(matrix)
    eigenvectors=p['eigenvectors']
    charpoly=p['charpoly']
    matrix=sp.latex(matrix)
    charpoly=sp.latex(charpoly)
    eigenvectors_0=[]
    for i in range(len(eigenvectors)):
        q = []
        for j in range(len(eigenvectors[i][2])):
            q.append(sp.latex(eigenvectors[i][2][j]))
        m = (sp.latex(eigenvectors[i][0]), sp.latex(eigenvectors[i][1]), q)
        eigenvectors_0.append(m)
    p['matrix']=matrix
    p['eigenvectors']=eigenvectors_0
    p['charpoly']=charpoly
    print(p['charpoly'])
    lambdamat = sp.latex(lambdamat)
    for i in range(len(lambdamatvalue)):
        lambdamatvalue[i] = sp.latex(lambdamatvalue[i])
    p.update({'lambdamat': lambdamat})
    p.update({'lambdamatvalue':lambdamatvalue})
    return p

if __name__ == '__main__':
    slveigenCourse("[[3,2,4],[2,0,2],[4,2,3]]")