### A Pluto.jl notebook ###
# v0.14.1

using Markdown
using InteractiveUtils

# ╔═╡ 7c62a2c4-9159-11eb-1b54-452017ea90d9
using LinearAlgebra, SparseArrays

# ╔═╡ 7ad16586-9162-11eb-1d94-5188eb8d4c5d
using DifferentialEquations, Plots

# ╔═╡ 44df6004-9158-11eb-1a63-651d7dca957b
md"# Heat equation with Robin boundary conditions

Consider a one-dimensional rod $\Omega = (0,L)$ and time $t \in (0,T).$
"

# ╔═╡ aff71d6e-9158-11eb-0559-cb7334f4a624
L = 0.1 # Length of rod

# ╔═╡ 3771dab8-9159-11eb-3a69-cd9b94a10399
Tf = 60.0 # Final simulation time

# ╔═╡ 44dbf81c-9159-11eb-2920-27ee60a7db9a
md"The rod has the physical properties
- thermal conductivity $\lambda$,
- specific heat capacity $c$,
- mass density $\rho$.

The diffusivity constant is given by $\alpha = \frac{\lambda}{c \rho}$. The heat equation is noted as

$\dot{\vartheta}(t,x) = \alpha \frac{\partial^2}{\partial x^2} \vartheta(t,x)$

for $(t,x) \in (0,T) \times \Omega$ with initial condition

$\vartheta(0,x) = \vartheta_{0}(x)$

for $x \in \overline{\Omega}$."

# ╔═╡ 86a40dc8-9158-11eb-16a3-1f776d6bde4e
begin
	λ = 45.0;    # Thermal conductivity
	ρ = 7800.0;  # Mass density
	cap = 480.0; # Specific heat capacitivity
end

# ╔═╡ 9e8ee0a2-9158-11eb-0210-4513221b7823
α = λ/(cap * ρ)   # Diffusivity

# ╔═╡ a4f1e1e2-9158-11eb-0754-a70399a341ba
md"## Boundary conditions

On both sides of the rod, heat transfer

$\phi_{t}(t,x) = - h ~ (\vartheta(t,x) - \vartheta_{amb})$

and heat radiation

$\phi_{r}(t,x) = - \epsilon \varrho ~ (\vartheta(t,x)^4 - \vartheta_{amb}^4)$

is assumed to operate. Parameter $\vartheta_{amb}$ describes the ambient temperature, $h>0$ is called the heat transfer coefficient, $\epsilon \in (0,1)$ is called emissivity and $\varrho\approx 5.67 \cdot 10^{-8}$ is known as Stefan-Boltzmann constant. For simplicity, $k=\epsilon \cdot \varrho$.

The sum of $\phi_{t}$ and $\phi_{r}$

$\phi_{out}(t,x) = \phi_{t}(t,x) + \phi_{r}(t,x) = - h ~ (\vartheta(t,x) - \vartheta_{amb}) - k ~ (\vartheta(t,x)^4 - \vartheta_{amb}^4)$

is the flux from the rod to the environment. Thus, the natural (Robin) boundary conditions are noted as

$\lambda ~ \left.\frac{\partial}{\partial x} \vartheta(\cdot, x) \right\rvert_{x = 0} \cdot \vec{n} ~=~ \lambda ~  \left. \frac{\partial}{\partial x} \vartheta(\cdot, x) \right\rvert_{x = L} \cdot \vec{n} ~=~ \phi_{out}(t,x)$

with outer [normal vector](https://en.wikipedia.org/wiki/Normal_(geometry)) $\vec{n}$ on the left or right boundary. Here, the normal vector $\vec{n} = -1$ on the left boundary at $x=0$, and  $\vec{n} = 1$ on the right boundary at $x=L$."

# ╔═╡ 7c9e9a4a-9159-11eb-3dda-59d2aa15f32a
begin
	h = 10.0; # Heat transfer coefficient
	ϵ = 0.6;  # Emissivity
	sb = 5.67*10^(-8) # Stefan-Boltzmann constant
	k = ϵ * sb; 	  # Radiation coefficient
end

# ╔═╡ c5da798e-9160-11eb-1d16-75ce792ef2fe
θamb = 298.0 # Ambient temperature in Kelvin

# ╔═╡ 7c80218c-9159-11eb-301a-95a428fb76d9
md"### Spatial approximation

The one-dimensional rod is discretized as a one-dimensional grid with N points and the finite discretization is noted by

$\Delta x = \frac{L}{N-1}.$

The resulting grid points have the position $x^{0} = 0$, $x^{n} = n~\Delta x$ and $x^{N-1} = L$. The second order derivative is approximated using the [Taylor series](https://en.wikipedia.org/wiki/Taylor_series) as

$\left. \frac{\partial^2 f(x)}{\partial x^2} \right\rvert_{x = \tilde{x}} \approx \frac{1}{\Delta x^2} \left[ f(\tilde{x} - \Delta x) - 2 f(\tilde{x}) + f(\tilde{x} + \Delta x)  \right].$

This [Finite Difference](https://en.wikipedia.org/wiki/Finite_difference_method) scheme is written in matrix-vector form as

$\frac{\partial^2 \vartheta(t,x)}{\partial x^2} \approx \frac{1}{\Delta x^2}
\begin{pmatrix}
-2 & 1 & 0 & \cdots &  & 0 \\
1 & -2 & 1 & 0  & \cdots & \vdots \\
0 & 1 & -2 & 1 & \ddots &  \\
 & & \ddots & \ddots & \ddots &  & \\
& & & 1 & -2 & 1 \\
& & &  & 1 & -2 \\
\end{pmatrix}
~
\begin{pmatrix}
\vartheta(t, x^{0}) \\
\vartheta(t, x^{1}) \\
\vdots \\
\vartheta(t, x^{N-1}) \\
\end{pmatrix}.$
"

# ╔═╡ 3487893a-915c-11eb-3a04-1fdd31009413
N = 101 # Number of grid elements

# ╔═╡ 39663578-915c-11eb-108c-2718b706729e
Δx = L/(N-1) # Finite discretization

# ╔═╡ 3468092a-915c-11eb-1719-d1b9b04b68e0
# Diffusion matrix
M = spdiagm(-1 => ones(N-1), 0 => -2*ones(N), 1 => ones(N-1));

# ╔═╡ 344c1bb6-915c-11eb-2be7-35f340f4caa8
Matrix(M)[1:5,1:5]

# ╔═╡ 342fe676-915c-11eb-3f22-1d20d5b502ed
md"### Approximated boundary conditions

The spatial approximation of the heat equation

$\frac{\partial^2 \vartheta(t,x)}{\partial x^2} \approx \frac{1}{\Delta x^2} \left( \vartheta(\cdot, x^{n-1}) - 2 ~ \vartheta(\cdot, x^{n}) + \vartheta(\cdot, x^{n+1}) \right)$

can not be evaluated directly at the grid points $x^{0}$ and $x^{N-1}$ because they depend on values from the not-existing grid points $x^{-1}$ and $x^{N}$. Therefore, the Neumann boundary condition is used to find $\vartheta(\cdot, x^{-1})$ and $\vartheta(\cdot, x^{N})$.

The Robin boundary condition states that the heat flux at both sides is described by

$\lambda \left. \frac{\partial \vartheta(\cdot,x)}{\partial x}  \right\rvert_{x = 0} \cdot \vec{n} = -1 \cdot \lambda \frac{\vartheta(\cdot, x^{1}) - \vartheta(\cdot, x^{-1})}{2 \Delta x}   = \phi_{out}(t,x^{0})$

on the left boundary and

$\lambda \left. \frac{\partial \vartheta(\cdot,x)}{\partial x}  \right\rvert_{x = L} \cdot \vec{n} = \lambda \frac{\vartheta(\cdot, x^{N}) - \vartheta(\cdot, x^{N-2})}{2 \Delta x}  = \phi_{out}(t,x^{N-1})$

on the right boundary. Reformulating both equations lead to

$\vartheta(\cdot, x^{-1}) = \vartheta(\cdot, x^{1}) + 2 \Delta x ~ \phi_{out}(t,x^{0})$

and

$\vartheta(\cdot, x^{N}) = \vartheta(\cdot, x^{N-2})  + 2 \Delta x ~ \phi_{out}(t,x^{N-1}).$

On the left boundary one yields

$\left. \frac{\partial^2 \vartheta(t,x)}{\partial x^2} \right\rvert_{x = 0} \approx \frac{1}{\Delta x^2} \left( - 2 ~ \vartheta(\cdot, x^{0}) + 2 ~ \vartheta(\cdot, x^{1}) \right) + \frac{2}{\Delta x} \phi_{out}(t,x^{0})$

and analog on the right boundary

$\left. \frac{\partial^2 \vartheta(t,x)}{\partial x^2} \right\rvert_{x = 0} \approx \frac{1}{\Delta x^2} \left(2 ~ \vartheta(\cdot, x^{N-2}) - 2 ~ \vartheta(\cdot, x^{N-1}) \right) + \frac{2}{\Delta x} \phi_{out}(t,x^{N-1}).$

The diffusion matrix and the approximated flux are noted as

$M =
\begin{pmatrix}
-2 & 2 & 0 & \cdots &  & 0 \\
1 & -2 & 1 & 0  & \cdots & \vdots \\
0 & 1 & -2 & 1 & \ddots &  \\
 & & \ddots & \ddots & \ddots &  & \\
& & & 1 & -2 & 1 \\
& & &  & 2 & -2 \\
\end{pmatrix}, \quad

\Phi_{out}(t) =
\begin{pmatrix}
\phi_{out}(t, x^{0}) \\
0 \\
\vdots \\
\\
0 \\
\phi_{out}(t, x^{N-1}) \\
\end{pmatrix}.$
"

# ╔═╡ 3414b70c-915c-11eb-2606-4d1ad425d3ab
# First row
M[1,2] = 2;

# ╔═╡ 33cd51a0-915c-11eb-3c62-efc367e08126
# Last row
M[end,end-1] = 2;

# ╔═╡ 33af0164-915c-11eb-30f0-313d926c2dae
md"### Heat equation as ODE

After the spatial approximation the heat equation has a form of

$\dot{\theta}(t) =  \frac{\alpha}{\Delta x^2} ~ M ~ \theta(t) + \frac{2 \alpha}{\Delta x} ~ \Phi_{out}(t)$

with M as diffusion matrix, temperature vector $\quad \theta(t) = \left( \vartheta(t, x^{0}), \cdots, \vartheta(t, x^{N-1}) \right)^{\top}$. This is an ordinary differential equation (ODE) that can be solved with common solvers like forward Euler method or Runge-Kutta scheme."

# ╔═╡ 328a1ed0-9160-11eb-0892-7d0a9d05a54d
# Heat Equation as ODE
function heat_eq!(dθ, θ, p, t)

	N = size(θ)
	Φout = zeros(N);

	Φout[1] = -h * (θ[1] - θamb) - k*(θ[1]^4 - θamb^4)

	Φout[end] = -h * (θ[end] - θamb) - k*(θ[end]^4 - θamb^4)

    dθ .= (1/Δx^2) * α * M * θ + (2/Δx) * α * Φout
end

# ╔═╡ 66d5aee4-9161-11eb-1ef1-2193d9c2f6a2
md"### Initial Conditions

For simplicity, the initial data of the heat equation is assumed as $\vartheta_{0}(x)= 1000$ Kelvin for all $x \in \left[0,L\right]$. This function is approximated to gain the initial conditions of the ODE

$\theta(0) = \left( 10^3, \cdots , 10^3 \right)^{\top}.$
"

# ╔═╡ 91a272be-9162-11eb-1046-8dd9e33653e5
θ₀ = 10^3 * ones(N)

# ╔═╡ 1571f3f2-9162-11eb-0b63-ff4e25a07572
md"The upper limit of the sampling time has to

$\Delta t < \frac{1}{2} \frac{\Delta x^2}{\alpha}.$

to guarantee numerical stability."

# ╔═╡ f85a2d4a-9161-11eb-0e0f-851c0750832a
Δt_ul = (0.5*Δx^2)/α # upper limit of sampling time

# ╔═╡ 5aa78f74-9162-11eb-1bd4-c96f2b90e586
Δt = 10^(-2) # Sampling time

# ╔═╡ 66ba8b22-9162-11eb-13b0-b14dfcead641
md"## Simulation"

# ╔═╡ 861018de-9162-11eb-2a01-2354acd868e2
tspan = (0.0, Tf)

# ╔═╡ e0e033e8-9162-11eb-3fe6-05565991ce77
# 1-dimensional grid
xspan = 0 : Δx : L

# ╔═╡ 01b373f0-9163-11eb-3f35-3f645825720f
prob = ODEProblem( heat_eq!, θ₀, tspan ) # ODE Problem

# ╔═╡ 0103888c-9163-11eb-16ae-371cd6d04c45
sol = solve(prob,Euler(),dt=Δt,progress=true, save_everystep=false, save_start=true) # Solving the ODE

# ╔═╡ 08dcb90e-9163-11eb-2ff0-7129add62934
plot(xspan, sol.u[2], xlabel = "Position x", ylabel="Temperature", legend=false)

# ╔═╡ Cell order:
# ╟─44df6004-9158-11eb-1a63-651d7dca957b
# ╠═aff71d6e-9158-11eb-0559-cb7334f4a624
# ╠═3771dab8-9159-11eb-3a69-cd9b94a10399
# ╟─44dbf81c-9159-11eb-2920-27ee60a7db9a
# ╠═86a40dc8-9158-11eb-16a3-1f776d6bde4e
# ╠═9e8ee0a2-9158-11eb-0210-4513221b7823
# ╟─a4f1e1e2-9158-11eb-0754-a70399a341ba
# ╠═7c9e9a4a-9159-11eb-3dda-59d2aa15f32a
# ╠═c5da798e-9160-11eb-1d16-75ce792ef2fe
# ╟─7c80218c-9159-11eb-301a-95a428fb76d9
# ╠═7c62a2c4-9159-11eb-1b54-452017ea90d9
# ╠═3487893a-915c-11eb-3a04-1fdd31009413
# ╠═39663578-915c-11eb-108c-2718b706729e
# ╠═3468092a-915c-11eb-1719-d1b9b04b68e0
# ╠═344c1bb6-915c-11eb-2be7-35f340f4caa8
# ╟─342fe676-915c-11eb-3f22-1d20d5b502ed
# ╠═3414b70c-915c-11eb-2606-4d1ad425d3ab
# ╠═33cd51a0-915c-11eb-3c62-efc367e08126
# ╟─33af0164-915c-11eb-30f0-313d926c2dae
# ╠═328a1ed0-9160-11eb-0892-7d0a9d05a54d
# ╟─66d5aee4-9161-11eb-1ef1-2193d9c2f6a2
# ╠═91a272be-9162-11eb-1046-8dd9e33653e5
# ╟─1571f3f2-9162-11eb-0b63-ff4e25a07572
# ╠═f85a2d4a-9161-11eb-0e0f-851c0750832a
# ╠═5aa78f74-9162-11eb-1bd4-c96f2b90e586
# ╠═66ba8b22-9162-11eb-13b0-b14dfcead641
# ╠═7ad16586-9162-11eb-1d94-5188eb8d4c5d
# ╠═861018de-9162-11eb-2a01-2354acd868e2
# ╠═e0e033e8-9162-11eb-3fe6-05565991ce77
# ╠═01b373f0-9163-11eb-3f35-3f645825720f
# ╠═0103888c-9163-11eb-16ae-371cd6d04c45
# ╠═08dcb90e-9163-11eb-2ff0-7129add62934


# sample from a double well in 1D using HMC
using Printf
using Random
using LinearAlgebra
using ForwardDiff
using QuadGK
using BasicMD
using Statistics

include("potentials.jl")

β = 5.0;
x₀ = [-1.0];
M = 1.0;
seed = 100;
Δt = 2e-1;
n_iters = 10^4; # number of samples
nΔt = 10^1; # number of Verlet steps per HMC iteration
n_save_iters=10;

V = x->DoubleWell(x);
cfg = ForwardDiff.GradientConfig(V, x₀);
gradV! = (gradV, x)-> ForwardDiff.gradient!(gradV, V, x, cfg);

# second moment and energy obsrevables
f₁ = x-> x[1]^2;
f₂ = x-> V(x[1]);
observables = (f₁, f₂);

sampler = HMC(V, gradV!, β, M, Δt, nΔt);

Random.seed!(100);
Xvals,_ = sample_trajectory(x₀, sampler, options=MDOptions(n_iters=n_iters,n_save_iters=n_save_iters));
f₁_estimate = mean(f₁.(Xvals));
f₂_estimate = mean(f₂.(Xvals));
@printf("f₁ estimate with %d samples: %g\n",length(Xvals), f₁_estimate);
@printf("f₂ estimate with %d samples: %g\n", length(Xvals), f₂_estimate);
Random.seed!(100);
observable_samples = sample_observables(x₀, sampler, observables, options=MDOptions(n_iters=n_iters,n_save_iters=n_save_iters));
@printf("f₁ estimate with %d samples: %g\n",length(observable_samples), mean(observable_samples[1,:]));
@printf("f₂ estimate with %d samples: %g\n", length(observable_samples), mean(observable_samples[2,:]));


"""
    gammaHSIC(X::Vector{T}, Y::Vector{T})

"""

function gammaHSIC(X::Vector{T}, Y::Vector{T}; randomSubSet = 100, kernelSize = -1) where {T<:Real}

	M = length(X)

	# get kernel sizes
	if kernelSize == -1
		sigx = estimateKernelSize(X, sampleSize = randomSubSet)
		sigy = estimateKernelSize(Y, sampleSize = randomSubSet)
	else
		sigx = kernelSize
		sigy = kernelSize
	end

	bone = ones(T, M, 1)
	H = eye(M) - 1/M * ones(T, M,M)

	K = rbfDotProduct(X, X, sigx)
	L = rbfDotProduct(Y, Y, sigy)

	# NOTE: these are slightly biased estimates of centred Gram matrices
	Kc = H*K*H
	Lc = H*L*H

	# NOTE: we fit Gamma to testStat*M
	testStat = 1/M * sum(sum(Kc'.*Lc))

	varHSIC = (1/6 * Kc.*Lc).^2
	varHSIC = 1/M/(M-1)* (  sum(sum(varHSIC)) - sum(diag(varHSIC))  ) # second subtracted term is bias correction
	varHSIC = 72*(M-4)*(M-5)/M/(M-1)/(M-2)/(M-3) * varHSIC

	K = K-diagm(0 => diag(K))
	L = L-diagm(0 => diag(L))

	muX = 1/M/(M-1)*bone'*(K*bone)
	muY = 1/M/(M-1)*bone'*(L*bone)

	mHSIC  = 1/M * ( 1 .+muX*muY - muX - muY )

	al = mHSIC^2 / varHSIC
	bet = varHSIC*M ./ mHSIC

	return 1 - gammacdf(al[1], bet[1], testStat)

end


# I.Y. Kim and O.L. de Weck.  Adaptive weighted-sum method for bi-objective optimization
# Pareto front generation. Structural and Multidisciplinary Optimization.
# Vol. 29, Num. 2, pp. 149–158, 2005.
# https://link.springer.com/article/10.1007/s00158-004-0465-1

using NSGAIII, PyPlot

function plot_pop(P)
    clf()
    p = plot(map(x -> x.y[1], P), map(x -> x.y[2], P), "bo", markersize=1)
    !isinteractive() && show()
    sleep(0.2)
end

const d = RealCoding(8, [-3, -3], [3, 3])
z1(x1, x2) = -(3(1-x1)^2 * exp(-x1^2 - (x2+1)^2) - 10(x1/5 - x1^3 - x2^5) * exp(-x1^2-x2^2) -3exp(-(x1+2)^2 - x2^2) + 0.5(2x1 + x2))
z2(x1, x2) = -(3(1+x2)^2 * exp(-x2^2 - (1-x1)^2) - 10(-x2/5 + x2^3 + x1^5) * exp(-x1^2-x2^2) - 3exp(-(2-x2)^2 - x1^2))
z(x) = begin
    x1, x2 = decode(x, d)
    z1(x1, x2), z2(x1, x2)
end
nsga(300, 20, ()->bitrand(d.nbbitstotal), z, 10, fplot = plot_pop)


 @printf("\nTesting linear operators\n")

######## Conv ############
n,m = 5, 6
h = randn(m)
op = Conv(Float64,(n,),h)
x1 = randn(n)
y1 = test_op(op, x1, randn(n+m-1), verb)
y2 = conv(x1,h)

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
op = Conv(x1,h)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

########## DCT ############
n = 4
op = DCT(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, randn(n), verb)
y2 = dct(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
op = DCT((n,))
op = DCT(n,n)
op = DCT(Complex{Float64}, n,n)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == true
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

m = 10
op = DCT(n,m)
x1 = randn(n,m)

@test norm(op'*(op*x1) - x1) <= 1e-12
@test diag_AAc(op) == 1.
@test diag_AcA(op) == 1.

######### IDCT ############
n = 4
op = IDCT(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, randn(n), verb)
y2 = idct(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
op = IDCT((n,))
op = IDCT(n,n)
op = IDCT(Complex{Float64}, n,n)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == true
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

m = 10
op = IDCT(n,m)
x1 = randn(n,m)

@test norm(op'*(op*x1) - x1) <= 1e-12
@test diag_AAc(op) == 1.
@test diag_AcA(op) == 1.

######## DFT ############
# seems like there is an object called DFT in Base julia 0.7 (however in 1.0 was rm)
n,m = 4,7

op = AbstractOperators.DFT(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = fft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Complex{Float64},(n,))
x1 = randn(n)+im*randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = fft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = fft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Complex{Float64},(n,))
x1 = randn(n)+im*randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = fft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Float64,(n,),1)
x1 = randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = fft(x1,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Complex{Float64},(n,),1)
x1 = randn(n)+im*randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = fft(x1,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Float64,(n,m))
x1 = randn(n,m)
y1 = test_op(op, x1, fft(randn(n,m)), verb)
y2 = fft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Complex{Float64},(n,m))
x1 = randn(n,m)+im*randn(n,m)
y1 = test_op(op, x1, fft(randn(n,m)), verb)
y2 = fft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Float64,(m,n),1)
x1 = randn(m,n)
y1 = test_op(op, x1, fft(randn(m,n)), verb)
y2 = fft(x1,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.DFT(Complex{Float64},(n,m),2)
x1 = randn(n,m)+im*randn(n,m)
y1 = test_op(op, x1, fft(randn(n,m)), verb)
y2 = fft(x1,2)

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
op = AbstractOperators.DFT((n,))
op = AbstractOperators.DFT(n,n)
op = AbstractOperators.DFT(Complex{Float64}, n,n)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == false
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

m = 10
op = AbstractOperators.DFT(n,m)
x1 = randn(n,m)
y1 = op*x1
@test norm(op'*(op*x1) - diag_AcA(op)*x1) <= 1e-12
@test norm(op*(op'*y1) - diag_AAc(op)*y1) <= 1e-12

######### IDFT ############
n,m = 5,6

op = IDFT(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = ifft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Complex{Float64},(n,))
x1 = randn(n)+im*randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = ifft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Float64,(n,),1)
x1 = randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = ifft(x1,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Complex{Float64},(n,),1)
x1 = randn(n)+im*randn(n)
y1 = test_op(op, x1, fft(randn(n)), verb)
y2 = ifft(x1,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = AbstractOperators.IDFT(Float64,(n,m))
x1 = randn(n,m)
y1 = test_op(op, x1, fft(randn(n,m)), verb)
y2 = ifft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Complex{Float64},(n,m))
x1 = randn(n,m)+im*randn(n,m)
y1 = test_op(op, x1, fft(randn(n,m)), verb)
y2 = ifft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Float64,(m,n),1)
x1 = randn(m,n)
y1 = test_op(op, x1, fft(randn(m,n)), verb)
y2 = ifft(x1,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Complex{Float64},(m,n),2)
x1 = randn(m,n)+im*randn(m,n)
y1 = test_op(op, x1, fft(randn(m,n)), verb)
y2 = ifft(x1,2)

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 4,19,5
op = IDFT(Complex{Float64},(n,m,l),2)
x1 = fft(randn(n,m,l),2)
y1 = test_op(op, x1, ifft(x1,2), verb)
y2 = ifft(x1,2)

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 4,18,5
op = IDFT(Complex{Float64},(n,m,l),(1,2))
x1 = fft(randn(n,m,l),(1,2))
y1 = test_op(op, x1, ifft(x1,(1,2)), verb)
y2 = ifft(x1,(1,2))

@test all(norm.(y1 .- y2) .<= 1e-12)

op = IDFT(Complex{Float64},(n,m,l),(3,2))
x1 = fft(randn(n,m,l),(3,2))
y1 = test_op(op, x1, ifft(x1,(3,2)), verb)
y2 = ifft(x1,(3,2))

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
op = IDFT((n,))
op = IDFT(n,n)
op = IDFT(Complex{Float64}, n,n)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == false
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

m = 10
op = IDFT(n,m)
x1 = randn(n,m)
y1 = op*x1
@test norm(op'*(op*x1) - diag_AcA(op)*x1) <= 1e-12
@test norm(op*(op'*y1) - diag_AAc(op)*y1) <= 1e-12

####### RDFT ############
n = 4
op = RDFT(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, rfft(x1), verb)
y2 = rfft(x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 4,8,5
op = RDFT(Float64,(n,m,l),2)
x1 = randn(n,m,l)
y1 = test_op(op, x1, rfft(x1,2), verb)
y2 = rfft(x1,2)

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
op = RDFT((n,))
op = RDFT(n,n)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == false

####### IRDFT ############
n = 12
op = IRDFT(Complex{Float64},(div(n,2)+1,),n)
x1 = rfft(randn(n))
y1 = test_op(op, x1,irfft(randn(div(n,2)+1),n), verb)
y2 = irfft(x1,n)

@test all(norm.(y1 .- y2) .<= 1e-12)

n = 11
op = IRDFT(Complex{Float64},(div(n,2)+1,),n)
x1 = rfft(randn(n))
y1 = test_op(op, x1,irfft(randn(div(n,2)+1),n), verb)
y2 = irfft(x1,n)

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 4,19,5
op = IRDFT(Complex{Float64},(n,div(m,2)+1,l),m,2)
x1 = rfft(randn(n,m,l),2)
y1 = test_op(op, x1, irfft(x1,m,2), verb)
y2 = irfft(x1,m,2)

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 4,18,5
op = IRDFT(Complex{Float64},(n,div(m,2)+1,l),m,2)
x1 = rfft(randn(n,m,l),2)
y1 = test_op(op, x1, irfft(x1,m,2), verb)
y2 = irfft(x1,m,2)

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 5,18,5
op = IRDFT(Complex{Float64},(div(n,2)+1,m,l),n,1)
x1 = rfft(randn(n,m,l),1)
y1 = test_op(op, x1, irfft(x1,n,1), verb)
y2 = irfft(x1,n,1)

@test all(norm.(y1 .- y2) .<= 1e-12)

## other constructors
op = IRDFT((10,),19)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == false

########## DiagOp ############
n = 4
d = randn(n)
op = DiagOp(Float64,(n,),d)
x1 = randn(n)
y1 = test_op(op, x1, randn(n), verb)
y2 = d.*x1

@test all(norm.(y1 .- y2) .<= 1e-12)

n = 4
d = randn(n)+im*randn(n)
op = DiagOp(Float64,(n,),d)
x1 = randn(n)
y1 = test_op(op, x1, randn(n).+im*randn(n), verb)
y2 = d.*x1

@test all(norm.(y1 .- y2) .<= 1e-12)

n = 4
d = pi
op = DiagOp(Float64,(n,),d)
x1 = randn(n)
y1 = test_op(op, x1, randn(n), verb)
y2 = d.*x1

@test all(norm.(y1 .- y2) .<= 1e-12)

n = 4
d = im
op = DiagOp(Float64,(n,),d)
x1 = randn(n)
y1 = test_op(op, x1, randn(n)+im*randn(n), verb)
y2 = d.*x1

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
d = randn(4)
op = DiagOp(d)

d = randn(4).+im
op = DiagOp(d)

n = 4
d = pi
op = DiagOp((n,), d)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == true
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == false
@test is_invertible(DiagOp(ones(10)))       == true
@test is_invertible(DiagOp([ones(5);0]))    == false
@test is_full_row_rank(op)    == true
@test is_full_row_rank(DiagOp([ones(5);0]))    == false
@test is_full_column_rank(op) == true
@test is_full_column_rank(DiagOp([ones(5);0]))    == false

@test diag(op) == d
@test norm(op'*(op*x1) .- diag_AcA(op).*x1) <= 1e-12
@test norm(op*(op'*x1) .- diag_AAc(op).*x1) <= 1e-12

n = 4
d = pi
op = DiagOp((n,), d)
x1 = randn(n)

@test diag(op) == d
@test norm(op'*(op*x1) .- diag_AcA(op).*x1) <= 1e-12
@test norm(op*(op'*x1) .- diag_AAc(op).*x1) <= 1e-12

########## Eye ############
n = 4
op = Eye(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, randn(n), verb)

@test all(norm.(y1 .- x1) .<= 1e-12)

# other constructors
op = Eye(Float64, (n,))
op = Eye((n,))
op = Eye(n)
op = Eye(x1)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == true
@test is_diagonal(op)         == true
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == true
@test is_invertible(op)       == true
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

@test diag(op) == 1
@test diag_AcA(op) == 1
@test diag_AAc(op) == 1

######## Filt ############
n,m = 15,2
b,a = [1.;0.;1.;0.;0.], [1.;1.;1.]
op = Filt(Float64,(n,),b,a)
x1 = randn(n)
y1 = test_op(op, x1, randn(n), verb)
y2 = filt(b, a, x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

h = randn(10)
op = Filt(Float64,(n,m),h)
x1 = randn(n,m)
y1 = test_op(op, x1, randn(n,m), verb)
y2 = filt(h, [1.], x1)

@test all(norm.(y1 .- y2) .<= 1e-12)

# other constructors
Filt(n,  b, a)
Filt((n,m),  b, a)
Filt(n,  h)
Filt((n,),  h)
Filt(x1, b, a)
Filt(x1, b)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

###### FiniteDiff ############
n= 10
op = FiniteDiff(Float64,(n,))
x1 = randn(n)
y1 = test_op(op, x1, randn(n-1), verb)
y1 = op*collect(range(0,stop=1,length=n))
@test all(norm.(y1 .- 1/9) .<= 1e-12)

I1, J1, V1 = SparseArrays.spdiagm_internal(0 => ones(n-1))
I2, J2, V2 = SparseArrays.spdiagm_internal(1 => ones(n-1))
B = -sparse(I1,J1,V1,n-1,n)+sparse(I2,J2,V2,n-1,n)

@test norm(B*x1-op*x1) <= 1e-8

n,m= 10,5
op = FiniteDiff(Float64,(n,m))
x1 = randn(n,m)
y1 = test_op(op, x1, randn(n-1,m), verb)
y1 = op*repeat(collect(range(0,stop=1,length=n)),1,m)
@test all(norm.(y1 .- 1/9) .<= 1e-12)

n,m= 10,5
op = FiniteDiff(Float64,(n,m),2)
x1 = randn(n,m)
y1 = test_op(op, x1, randn(n,m-1), verb)
y1 = op*repeat(collect(range(0,stop=1,length=n)),1,m)
@test all(norm.(y1) .<= 1e-12)

n,m,l= 10,5,7
op = FiniteDiff(Float64,(n,m,l))
x1 = randn(n,m,l)
y1 = test_op(op, x1, randn(n-1,m,l), verb)
y1 = op*reshape(repeat(collect(range(0,stop=1,length=n)),1,m*l),n,m,l)
@test all(norm.(y1 .- 1/9) .<= 1e-12)

n,m,l= 10,5,7
op = FiniteDiff(Float64,(n,m,l),2)
x1 = randn(n,m,l)
y1 = test_op(op, x1, randn(n,m-1,l), verb)
y1 = op*reshape(repeat(collect(range(0,stop=1,length=n)),1,m*l),n,m,l)
@test all(norm.(y1) .<= 1e-12)

n,m,l= 10,5,7
op = FiniteDiff(Float64,(n,m,l),3)
x1 = randn(n,m,l)
y1 = test_op(op, x1, randn(n,m,l-1), verb)
y1 = op*reshape(repeat(collect(range(0,stop=1,length=n)),1,m*l),n,m,l)
@test all(norm.(y1) .<= 1e-12)

n,m,l,i = 5,6,2,3
op = FiniteDiff(Float64,(n,m,l,i),1)
x1 = randn(n,m,l,i)
y1 = test_op(op, x1, randn(n-1,m,l,i), verb)
y1 = op*reshape(repeat(collect(range(0,stop=1,length=n)),1,m*l*i),n,m,l,i)
@test all(norm.(y1 .- 1/(n-1)) .<= 1e-12)

n,m,l,i = 5,6,2,3
op = FiniteDiff(Float64,(n,m,l,i),4)
x1 = randn(n,m,l,i)
y1 = test_op(op, x1, randn(n,m,l,i-1), verb)
y1 = op*reshape(repeat(collect(range(0,stop=1,length=n)),1,m*l*i),n,m,l,i)
@test norm(y1) <= 1e-12

@test_throws ErrorException FiniteDiff(Float64,(n,m,l), 4)

## other constructors
FiniteDiff((n,m))
FiniteDiff(x1)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == false

########## GetIndex ############
n,m = 5,4
k = 3
op = GetIndex(Float64,(n,),(1:k,))
x1 = randn(n)
y1 = test_op(op, x1, randn(k), verb)

@test all(norm.(y1 .- x1[1:k]) .<= 1e-12)

n,m = 5,4
k = 3
op = GetIndex(Float64,(n,m),(1:k,:))
x1 = randn(n,m)
y1 = test_op(op, x1, randn(k,m), verb)

@test all(norm.(y1 .- x1[1:k,:]) .<= 1e-12)

n,m = 5,4
op = GetIndex(Float64,(n,m),(:,2))
x1 = randn(n,m)
y1 = test_op(op, x1, randn(n), verb)

@test all(norm.(y1 .- x1[:,2]) .<= 1e-12)

n,m,l = 5,4,3
op = GetIndex(Float64,(n,m,l),(1:3,2,:))
x1 = randn(n,m,l)
y1 = test_op(op, x1, randn(3,3), verb)

@test all(norm.(y1 .- x1[1:3,2,:]) .<= 1e-12)

# other constructors
GetIndex((n,m), (1:k,:))
GetIndex(x1, (1:k,:))

@test_throws ErrorException GetIndex(Float64,(n,m),(1:k,:,:))
op = GetIndex(Float64,(n,m),(1:n,1:m))
@test typeof(op) <: Eye

op = GetIndex(Float64,(n,),(1:k,))

##properties
@test is_sliced(op)           == true
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == true
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == false

@test diag_AAc(op) == 1

####### MatrixOp ############

# real matrix, real input
n,m = 5,4
A = randn(n,m)
op = MatrixOp(Float64,(m,),A)
x1 = randn(m)
y1 = test_op(op, x1, randn(n), verb)
y2 = A*x1

# real matrix, complex input
n,m = 5,4
A = randn(n,m)
op = MatrixOp(Complex{Float64},(m,),A)
x1 = randn(m)+im.*randn(m)
y1 = test_op(op, x1, randn(n)+im*randn(n), verb)
y2 = A*x1

# complex matrix, complex input
n,m = 5,4
A = randn(n,m)+im*randn(n,m)
op = MatrixOp(Complex{Float64},(m,),A)
x1 = randn(m)+im.*randn(m)
y1 = test_op(op, x1, randn(n)+im*randn(n), verb)
y2 = A*x1

# complex matrix, real input
n,m = 5,4
A = randn(n,m)+im*randn(n,m)
op = MatrixOp(Float64,(m,),A)
x1 = randn(m)
y1 = test_op(op, x1, randn(n)+im*randn(n), verb)
y2 = A*x1

@test all(norm.(y1 .- y2) .<= 1e-12)

# complex matrix, real matrix input
c = 3
op = MatrixOp(Float64,(m,c),A)
@test_throws ErrorException MatrixOp(Float64,(m,c,3),A)
@test_throws MethodError MatrixOp(Float64,(m,c),randn(n,m,2))
x1 = randn(m,c)
y1 = test_op(op, x1, randn(n,c).+randn(n,c), verb)
y2 = A*x1

# other constructors
op = MatrixOp(A)
op = MatrixOp(Float64, A)
op = MatrixOp(A, c)
op = MatrixOp(Float64, A, c)

op = convert(LinearOperator,A)
op = convert(LinearOperator,A,c)
op = convert(LinearOperator, Complex{Float64}, size(x1), A)

##properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(MatrixOp(randn(Random.seed!(0),3,4)))    == true
@test is_full_column_rank(MatrixOp(randn(Random.seed!(0),3,4))) == false

##### LMatrixOp ############

n,m = 5,6
b = randn(m)
op = LMatrixOp(Float64,(n,m),b)
x1 = randn(n,m)
y1 = test_op(op, x1, randn(n), verb)
y2 = x1*b

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m = 5,6
b = randn(m)+im*randn(m)
op = LMatrixOp(Complex{Float64},(n,m),b)
x1 = randn(n,m)+im*randn(n,m)
y1 = test_op(op, x1, randn(n)+im*randn(n), verb)
y2 = x1*b

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 5,6,7
b = randn(m,l)
op = LMatrixOp(Float64,(n,m),b)
x1 = randn(n,m)
y1 = test_op(op, x1, randn(n,l), verb)
y2 = x1*b

@test all(norm.(y1 .- y2) .<= 1e-12)

n,m,l = 5,6,7
b = randn(m,l)+im*randn(m,l)
op = LMatrixOp(Complex{Float64},(n,m),b)
x1 = randn(n,m)+im*randn(n,m)
y1 = test_op(op, x1, randn(n,l)+im*randn(n,l), verb)
y2 = x1*b

@test all(norm.(y1 .- y2) .<= 1e-12)

## other constructors
op = LMatrixOp(b,n)

##properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == false
@test is_full_column_rank(op) == false

######### MyLinOp ############

n,m = 5,4;
A = randn(n,m);
op = MyLinOp(Float64, (m,),(n,), (y,x) -> mul!(y,A,x), (y,x) -> mul!(y,A',x))
x1 = randn(m)
y1 = test_op(op, x1, randn(n), verb)
y2 = A*x1

# other constructors
op = MyLinOp(Float64, (m,), Float64, (n,), (y,x) -> mul!(y,A,x), (y,x) -> mul!(y,A',x))

######### MIMOFilt ############
m,n = 10,2
b = [[1.;0.;1.;0.;0.],[1.;0.;1.;0.;0.]]
a = [[1.;1.;1.],[2.;2.;2.]]
op = MIMOFilt(Float64, (m,n), b, a)

x1 = randn(m,n)
y1 = test_op(op, x1, randn(m,1), verb)
y2 = filt(b[1],a[1],x1[:,1])+filt(b[2],a[2],x1[:,2])

@test all(norm.(y1 .- y2) .<= 1e-12)

m,n = 10,3; #time samples, number of inputs
b = [[1.;0.;1.],[1.;0.;1.],[1.;0.;1.],[1.;0.;1.],[1.;0.;1.],[1.;0.;1.], ];
a = [[1.;1.;1.],[2.;2.;2.],[      3.],[      4.],[      5.],[      6.], ];
op = MIMOFilt(Float64, (m,n), b, a)

x1 = randn(m,n)
y1 = test_op(op, x1, randn(m,2), verb)
y2 = [filt(b[1],a[1],x1[:,1])+filt(b[2],a[2],x1[:,2])+filt(b[3],a[3],x1[:,3]) filt(b[4],a[4],x1[:,1])+filt(b[5],a[5],x1[:,2])+filt(b[6],a[6],x1[:,3])]

@test all(norm.(y1 .- y2) .<= 1e-12)

m,n = 10,3
b = [randn(10),randn(5),randn(10),randn(2),randn(10),randn(10)]
a = [[1.],[1.],[1.],[1.],[1.],[1.]]
op = MIMOFilt(Float64, (m,n), b, a)

x1 = randn(m,n)
y1 = test_op(op, x1, randn(m,2), verb)
y2 = [filt(b[1],a[1],x1[:,1])+filt(b[2],a[2],x1[:,2])+filt(b[3],a[3],x1[:,3]) filt(b[4],a[4],x1[:,1])+filt(b[5],a[5],x1[:,2])+filt(b[6],a[6],x1[:,3])]

@test all(norm.(y1 .- y2) .<= 1e-12)

## other constructors
MIMOFilt((10,3),  b, a)
MIMOFilt((10,3),  b)
MIMOFilt(x1,  b, a)
MIMOFilt(x1,  b)

#errors
@test_throws ErrorException MIMOFilt(Float64, (10,3,2) ,b,a)
a2 = [[1.0f0],[1.0f0],[1.0f0],[1.0f0],[1.0f0],[1.0f0]]
b2 = convert.(Array{Float32,1},b)
@test_throws ErrorException MIMOFilt(Float64, (m,n),b2,a2)
@test_throws ErrorException MIMOFilt(Float64, (m,n),b,a[1:end-1])
push!(a2,[1.0f0])
push!(b2,randn(Float32,10))
@test_throws ErrorException MIMOFilt(Float32, (m,n),b2,a2)
a[1][1] = 0.
@test_throws ErrorException MIMOFilt(Float64, (m,n) ,b,a)

b = [randn(10),randn(5),randn(10),randn(2),randn(10),randn(10)]
a = [[1.],[1.],[1.],[1.],[1.],[1.]]
op = MIMOFilt(Float64, (m,n), b, a)

##properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

######## Variation ############

n,m = 10,5
op = Variation(Float64,(n,m))
x1 = randn(n,m)
y1 = test_op(op, x1, randn(m*n,2), verb)

y1 = op*repeat(collect(range(0,stop=1,length=n)),1,m)
@test all(norm.(y1[:,1] .- 1/(n-1) ) .<= 1e-12)
@test all(norm.(y1[:,2] ) .<= 1e-12)

Dx = spdiagm(0 => ones(n), -1 => -ones(n-1))
Dx[1,1],Dx[1,2] = -1,1
Dy = spdiagm(0 => ones(m), -1 => -ones(m-1))
Dy[1,1],Dy[1,2] = -1,1

Dxx = kron(sparse(I,m,m),Dx)
Dyy = kron(Dy,sparse(I,n,n))
TV = [Dxx;Dyy]

x1 = randn(n,m)
@test norm(op*x1-reshape(TV*(x1[:]),n*m,2))<1e-12

n,m,l = 10,5,3
op = Variation(Float64,(n,m,l))
x1 = randn(n,m,l)
y1 = test_op(op, x1, randn(m*n*l,3), verb)

y1 = op*reshape(repeat(collect(range(0,stop=1,length=n)),1,m*l),n,m,l)
@test all(norm.(y1[:,1] .- 1/(n-1) ) .<= 1e-12)
@test all(norm.(y1[:,2] ) .<= 1e-12)
@test all(norm.(y1[:,3] ) .<= 1e-12)

### other constructors
Variation(Float64, n,m)
Variation((n,m))
Variation(n,m)
Variation(x1)

##errors
@test_throws ErrorException Variation(Float64,(n,))

###properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == false
@test is_full_column_rank(op) == false

######## Xcorr ############
n,m = 5, 6
h = randn(m)
op = Xcorr(Float64,(n,),h)
x1 = randn(n)
y1 = test_op(op, x1, randn(n+m), verb)
y2 = xcorr(x1, h; padmode=:longest)

@test all(norm.(y1 .- y2) .<= 1e-12)
# other constructors
op = Xcorr(x1,h)

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == true
@test is_full_column_rank(op) == true

######## ZeroPad ############
n = (3,)
z = (5,)
op = ZeroPad(Float64,n,z)
x1 = randn(n)
y1 = test_op(op, x1, randn(n.+z), verb)
@test all(norm.(y1 .- [x1;zeros(5)] ) .<= 1e-12)

n = (3,2)
z = (5,3)
op = ZeroPad(Float64,n,z)
x1 = randn(n)
y1 = test_op(op, x1, randn(n.+z), verb)
y2 = zeros(n.+z)
y2[1:n[1],1:n[2]] = x1
@test all(norm.(y1 .- y2 ) .<= 1e-12)

n = (3,2,2)
z = (5,3,1)
op = ZeroPad(Float64,n,z)
x1 = randn(n)
y1 = test_op(op, x1, randn(n.+z), verb)
y2 = zeros(n.+z)
y2[1:n[1],1:n[2],1:n[3]] = x1
@test all(norm.(y1 .- y2 ) .<= 1e-12)

# other constructors
ZeroPad(n, z...)
ZeroPad(Float64, n, z...)
ZeroPad(n, z...)
ZeroPad(x1, z)
ZeroPad(x1, z...)

#errors
@test_throws ErrorException ZeroPad(Float64,n,(1,2))
@test_throws ErrorException ZeroPad(Float64,n,(1,-2,3))

#properties
@test is_linear(op)           == true
@test is_null(op)             == false
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == true
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == false
@test is_full_column_rank(op) == true

diag_AcA(op) == 1

########## Zeros ############
n = (3,4)
D = Float64
m = (5,2)
C = Complex{Float64}
op = Zeros(D,n,C,m)
x1 = randn(n)
y1 = test_op(op, x1, randn(m)+im*randn(m), verb)

#properties
@test is_linear(op)           == true
@test is_null(op)             == true
@test is_eye(op)              == false
@test is_diagonal(op)         == false
@test is_AcA_diagonal(op)     == false
@test is_AAc_diagonal(op)     == false
@test is_orthogonal(op)       == false
@test is_invertible(op)       == false
@test is_full_row_rank(op)    == false
@test is_full_column_rank(op) == false


using HypothesisTests, Base.Test
using HypothesisTests: default_tail

# www.uni-siegen.de/phil/sozialwissenschaften/soziologie/mitarbeiter/ludwig-mayerhofer/statistik/statistik_downloads/statistik_ii_7.pdf
u5 = [620, 5350, 7220]
u250 = [3580, 4180, 5690]
u2500 = [3600, 3820, 3840, 3850, 4160, 5300, 6900, 7120, 9370]
more = [3920, 4520, 4760, 8560, 10350]
t = HypothesisTests.KruskalWallisTest(u5, u250, u2500, more)

@test t.n_i == [length(u5), length(u250), length(u2500), length(more)]
@test t.df == 3
@test t.R_i == [31, 25, 88, 66]
@test t.H ≈ 1.5803174603174597
@test t.tie_adjustment == 1
@test pvalue(t) ≈ 0.6638608922384397
@test default_tail(t) == :right
show(IOBuffer(), t)

# http://www.brightstat.com/index.php?option=com_content&task=view&id=41&Itemid=1&limit=1&limitstart=2
city1 = [68, 93, 123, 83, 108, 122]
city2 = [119, 116, 101, 103, 113, 84]
city3 = [70, 68, 54, 73, 81, 68]
city4 = [61, 54, 59, 67, 59, 70]
t = HypothesisTests.KruskalWallisTest(city1, city2, city3, city4)

@test t.n_i == [length(city1), length(city2), length(city3), length(city4)]
@test t.df == 3
@test t.R_i == [104, 113, 53, 30]
@test t.H ≈ 16.028783253379856
@test t.tie_adjustment ≈ 0.9969565217391304
@test pvalue(t) ≈ 0.0011186794961869423
show(IOBuffer(), t)

# example with non-integer rank sum
t1 = [1.2,1.9,2.1]
t2 = [3.4,1.9,5.6]
t3 = [1.3,4.4,9.9]
t = KruskalWallisTest(t1, t2, t3)
@test t.n_i == [length(t1), length(t2), length(t3)]
@test t.df == 2
@test t.R_i ==  [9.5, 17.5, 18.0]
@test t.H ≈ 2.039215686274513
@test t.tie_adjustment ≈ 0.9916666666666667
@test pvalue(t) ≈ 0.3607363776845705
show(IOBuffer(), t)


### A Pluto.jl notebook ###
# v0.16.0

using Markdown
using InteractiveUtils

# ╔═╡ 5bb6cf00-9b75-4e74-bc75-6a059bc1b32d
# Učita se paket SymPy
using SymPy

# ╔═╡ efc1cf20-8276-11eb-39ba-bd62a4342e68
md"
# Integrali racionalnih funkcija
"

# ╔═╡ d52c1d09-9bee-4359-8e2c-a969a2f32ea3
# Ova naredba daje popis svih djelova paketa, uglavnom su to funkcije.
# varinfo(SymPy)

# ╔═╡ 8153dabd-5be6-4308-b7c5-1f4ed54722ab
begin
	# Izračunajmo nekoliko integrala.
	x=Sym("x")
	integrate(x^2*sin(x))
end

# ╔═╡ 10b872ad-f070-404c-9ec8-19f0bc2032c5
integrate(exp(-x^2))

# ╔═╡ 056c397f-7303-464e-9ec6-8cf893bbc98c
# Primjer rekurzivne formule. Ovo traje malo duže
integrate(1/(1+x^2)^13)

# ╔═╡ b29b5f68-8978-4b4b-9b29-4840a4097978
# Kraća rekurzivna formula
integrate(1/(1+x^2)^3)

# ╔═╡ 232dcad5-1984-4421-9c1a-001406096217
methods(diff)

# ╔═╡ a2819cd4-d0b3-476a-ab8c-40838db72d54
# Derivacija
diff(x^2*exp(sin(1/x)))

# ╔═╡ 74af42dd-9768-4e91-aa3b-b26c69d6fb68
# Racionalna funkcija trigonometrijskih funkcija (univerzalna trigonometrijska supstitucija)
integrate(1/((2+cos(x))*sin(x)))

# ╔═╡ de024987-9c44-4199-9085-b259980e94a5
md"""
#### Zadatak 1.5.e)
"""

# ╔═╡ 62c5d76c-9eef-40a9-b8c9-446720733f80
integrate(1/(4*sin(x)+3*cos(x)+5))

# ╔═╡ 379bc6e9-7a38-4f3e-8e4a-2ff80917880b
integrate(1/(sin(x)*(2+cos(x)-2*sin(x))))

# ╔═╡ 28dd8649-ed7c-47e5-b7ab-69e99dc0ce86
# Ovo traje beskonačno?
# integrate(cos(x)^3/(sin(x)^2+sin(x)))

# ╔═╡ 12904f05-67c3-4d8a-aefd-e9645e6ef0eb
# Ovo je isti zadatak kao gore, ali traje kratko. Koristi se jednostavnija supstitucija.
integrate((1-sin(x)^2)*cos(x)/(sin(x)^2+sin(x)))

# ╔═╡ bf1829c9-b2fb-4e3b-b92d-38a93513678b
# Ovo traje jako dugo i ne uspije izračunati nego vrati polazni integral
# integrate( (2*tan(x)+3)/(sin(x)^2+2*cos(x)^2))

# ╔═╡ 3732f5f7-ed44-4254-8b16-d9d860d2c04f
md"""
Uz supstituciju $t=\tan(x)$ integral postaje jednostavan:
"""

# ╔═╡ 232927bc-ac81-4225-801f-9da5e5842d6d
begin
	t=Sym("t")
	I₁=integrate( (2*t+3)/(t^2+2))
end

# ╔═╡ 13e53f50-8c7d-403c-85c5-848bf2ccbe94
begin
	# Vratimo supstituciju natrag
	I₂=convert(Function,I₁)
	I₂(tan(x))
end

# ╔═╡ d9ada9db-2651-4895-876b-704e4ed05bcb
md"""
#### Zadatak 1.6
"""

# ╔═╡ f2eb749b-51d6-4875-8e14-1e0f688cdfc7
# Ovo traje beskonačno!
# I₄=integrate((1+sinh(x))/((2+cosh(x))*(3+sinh(x))),x)
# Program treba našu pomoć!!!

# ╔═╡ 244208a5-b153-424b-8305-5b6002fd8186
md"""
__Univerzalna hiperbolna supstitucija__

$$\begin{aligned}
t&=\tanh(\frac{x}{2}), \quad x=2\mathop{\mathrm{atanh}}(t),\quad  dx=\displaystyle\frac{2}{1-t^2}dt,\\
\sinh(x)&=\displaystyle\frac{2t}{1-t^2}, \quad \cosh(x)=\displaystyle\frac{1+t^2}{1-t^2}.\end{aligned}$$

"""

# ╔═╡ 78acc71b-b314-4799-b633-4ab5b7958496
I₄=(1+sinh(x))/((2+cosh(x))*(3+sinh(x)))

# ╔═╡ 463c1ac7-e503-4507-8aca-d671acf04d36
# supstitucija za sinh(x)
I₅=subs(I₄,sinh(x),(2*t)/(1-t^2))

# ╔═╡ 1b47139d-6c06-4227-91d2-1556d2f3a9d2
# Supstitucija za cosh(x)
I₆=subs(I₅,cosh(x),(1+t^2)/(1-t^2))

# ╔═╡ 646c21be-c589-43b6-b648-3d0dea8353db
# Pomnožimo s dx
I₇=I₆*2/(1-t^2)

# ╔═╡ 34320cf9-d978-46ca-ace2-ea638cc602d7
# Integriramo racionalnu funkciju
I₈=integrate(I₇,t)

# ╔═╡ 6e0e6146-fefa-4967-938e-1755a0a04607
# Vratimo supstituciju natrag
I₉=subs(I₈,t,tanh(x/2))

# ╔═╡ a817a9d1-f9d2-4fb9-ad16-2477a0a6dea6

# ╔═╡ 00000000-0000-0000-0000-000000000001
PLUTO_PROJECT_TOML_CONTENTS = """
[deps]
SymPy = "24249f21-da20-56a4-8eb1-6a02cf4ae2e6"

[compat]
SymPy = "~1.0.52"
"""

# ╔═╡ 00000000-0000-0000-0000-000000000002
PLUTO_MANIFEST_TOML_CONTENTS = """
# This file is machine-generated - editing it directly is not advised

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "a325370b9dd0e6bf5656a6f1a7ae80755f8ccc46"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.7.2"

[[CommonEq]]
git-tree-sha1 = "d1beba82ceee6dc0fce8cb6b80bf600bbde66381"
uuid = "3709ef60-1bee-4518-9f2f-acd86f176c50"
version = "0.2.0"

[[CommonSolve]]
git-tree-sha1 = "68a0743f578349ada8bc911a5cbd5a2ef6ed6d1f"
uuid = "38540f10-b2f7-11e9-35d8-d573e4eb0ff2"
version = "0.2.0"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "31d0151f5716b655421d9d75b7fa74cc4e744df2"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.39.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[Conda]]
deps = ["JSON", "VersionParsing"]
git-tree-sha1 = "299304989a5e6473d985212c28928899c74e9421"
uuid = "8f4d0f93-b110-5947-807f-2305c1781a2d"
version = "1.5.2"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "a32185f5428d3986f47c2ab78b1f216d5e6cc96f"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.5"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[IrrationalConstants]]
git-tree-sha1 = "f76424439413893a832026ca355fe273e93bce94"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.1.0"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "8076680b162ada2a031f707ac7b4953e30667a37"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.2"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[LogExpFunctions]]
deps = ["ChainRulesCore", "DocStringExtensions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "34dc30f868e368f8a17b728a1238f3fcda43931a"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.3"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "5a5bc6bf062f0f95e62d0fe0a2d99699fed82dd9"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.8"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[OpenLibm_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "05823500-19ac-5b8b-9628-191a04bc5112"

[[OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[Parsers]]
deps = ["Dates"]
git-tree-sha1 = "a8709b968a1ea6abc2dc1967cb1db6ac9a00dfb6"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.0.5"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.2"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[PyCall]]
deps = ["Conda", "Dates", "Libdl", "LinearAlgebra", "MacroTools", "Serialization", "VersionParsing"]
git-tree-sha1 = "169bb8ea6b1b143c5cf57df6d34d022a7b60c6db"
uuid = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
version = "1.92.3"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[RecipesBase]]
git-tree-sha1 = "44a75aa7a527910ee3d1751d1f0e4148698add9e"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.1.2"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[SpecialFunctions]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
git-tree-sha1 = "793793f1df98e3d7d554b65a107e9c9a6399a6ed"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "1.7.0"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[SymPy]]
deps = ["CommonEq", "CommonSolve", "LinearAlgebra", "Markdown", "PyCall", "RecipesBase", "SpecialFunctions"]
git-tree-sha1 = "1ef257ecbcab8058595a68ca36a6844b41babcbd"
uuid = "24249f21-da20-56a4-8eb1-6a02cf4ae2e6"
version = "1.0.52"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[VersionParsing]]
git-tree-sha1 = "80229be1f670524750d905f8fc8148e5a8c4537f"
uuid = "81def892-9a0e-5fdd-b105-ffc91e053289"
version = "1.2.0"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"
"""

# ╔═╡ Cell order:
# ╟─efc1cf20-8276-11eb-39ba-bd62a4342e68
# ╠═5bb6cf00-9b75-4e74-bc75-6a059bc1b32d
# ╠═d52c1d09-9bee-4359-8e2c-a969a2f32ea3
# ╠═8153dabd-5be6-4308-b7c5-1f4ed54722ab
# ╠═10b872ad-f070-404c-9ec8-19f0bc2032c5
# ╠═056c397f-7303-464e-9ec6-8cf893bbc98c
# ╠═b29b5f68-8978-4b4b-9b29-4840a4097978
# ╠═232dcad5-1984-4421-9c1a-001406096217
# ╠═a2819cd4-d0b3-476a-ab8c-40838db72d54
# ╠═74af42dd-9768-4e91-aa3b-b26c69d6fb68
# ╟─de024987-9c44-4199-9085-b259980e94a5
# ╠═62c5d76c-9eef-40a9-b8c9-446720733f80
# ╠═379bc6e9-7a38-4f3e-8e4a-2ff80917880b
# ╠═28dd8649-ed7c-47e5-b7ab-69e99dc0ce86
# ╠═12904f05-67c3-4d8a-aefd-e9645e6ef0eb
# ╠═bf1829c9-b2fb-4e3b-b92d-38a93513678b
# ╟─3732f5f7-ed44-4254-8b16-d9d860d2c04f
# ╠═232927bc-ac81-4225-801f-9da5e5842d6d
# ╠═13e53f50-8c7d-403c-85c5-848bf2ccbe94
# ╟─d9ada9db-2651-4895-876b-704e4ed05bcb
# ╠═f2eb749b-51d6-4875-8e14-1e0f688cdfc7
# ╟─244208a5-b153-424b-8305-5b6002fd8186
# ╠═78acc71b-b314-4799-b633-4ab5b7958496
# ╠═463c1ac7-e503-4507-8aca-d671acf04d36
# ╠═1b47139d-6c06-4227-91d2-1556d2f3a9d2
# ╠═646c21be-c589-43b6-b648-3d0dea8353db
# ╠═34320cf9-d978-46ca-ace2-ea638cc602d7
# ╠═6e0e6146-fefa-4967-938e-1755a0a04607
# ╠═a817a9d1-f9d2-4fb9-ad16-2477a0a6dea6
# ╟─00000000-0000-0000-0000-000000000001
# ╟─00000000-0000-0000-0000-000000000002


"""
    eom_logistic2(dx, x, p, n) -> function

Equations of motions for a system consisting of two coupled logistic maps where
X unidirectionally influences Y.

The parameter `c` controls how strong the dynamical forcing is. Parameters `r₁`
and `r₂` are set to the chaotic regime by default

# References
1. Diego, D., Agasøster Haaga, K., & Hannisdal, B. (2018, November 1).
Transfer entropy computation using the Perron-Frobenius operator.
Eprint ArXiv:1811.01677. Retrieved from
https://ui.adsabs.harvard.edu/#abs/2018arXiv181101677D
"""
function eom_logistic2(dx, x, p, n)
    c, r₁, r₂, σ = (p...,)
    ξ = rand() # random number from flat distribution on [0, 1]
    x, y = x[1], x[2]
    f_xy = (y +  (c*(x + σ*ξ)/2) ) / (1 + (c/2)*(1+σ))

    dx[1] = r₁ * x * (1 - x)
    dx[2] = r₂ * (f_xy) * (1 - f_xy)
    return
end

function logistic2(u₀, c, r₁, r₂, σ)
    p = [c, r₁, r₂, σ]
    DiscreteDynamicalSystem(eom_logistic2, u₀, p)
end

"""
    logistic2(;u₀ = rand(2), c = 0.1, σ = 0.05,
        r₁ = 3.78, r₂ = 3.66) -> DiscreteDynamicalSystem

Initialise a system consisting of two coupled logistic maps where X
unidirectionally influences Y. By default, the parameters `r₁` and `r₂` are set
to values yielding chaotic behaviour.

The parameter `c` controls how strong the dynamical forcing is. If `σ > 0`,
dynamical noise masking the influence of  `x` on `y` equivalent to
``\\sigma \\cdot \\xi`` is added at each iteration. Here,``\\xi`` is a draw from a
flat distribution on ``[0, 1]``. Thus, setting `σ = 0.05` is equivalent to
add dynamical noise corresponding to a maximum of ``5 \\%`` of the possible
range of values of the logistic map.

The equations of motion are

```math
\\begin{aligned}
dx &= r_1 x(1 - x) \\
dy &= r_2 f(x,y)(1 - f(x,y)),
\\end{aligned}
```
with
```math
\\begin{aligned}
f(x,y) = \\dfrac{y + \\frac{c(x \\xi )}{2}}{1 + \\frac{c}{2}(1+ \\sigma )}
\\end{aligned}
```

where

# References

1. Diego, D., Agasøster Haaga, K., & Hannisdal, B. (2018, November 1).
Transfer entropy computation using the Perron-Frobenius operator.
Eprint ArXiv:1811.01677. Retrieved from
https://ui.adsabs.harvard.edu/#abs/2018arXiv181101677D

"""
logistic2(;u₀ = rand(2), c = 0.1, r₁ = 3.78, r₂ = 3.66, σ = 0.05) =
    logistic2(u₀, c, r₁, r₂, σ)


#
#Simple example of a Poisson problem:
#
#           y
#             |
#             |           u_y = 0
#          Ly ----------------------------------
#             |                                |
#             |                                |
#             |                                |
#       u = 0 |    u - u_xx - u_yy = f         | u = 0
#             |                                |
#             |                                |
#             |                                |
#           -----------------------------------|-----  x
#                         u_y = 0               Lx
#
#Here,
#
#      u = sin(omegax x) * cos(omegay y)
#      f = ( 1 + omegax^2 + omegay^2 ) * sin(omegax x) * cos(omegay y)
#      omegax = nx pi / Lx
#      omegay = ny pi / Ly
#

using FinElt
using FinElt.PlanarPoisson
using Printf

include("params.jl")

const omegax = 3 * pi / Lx
const omegay = pi / Ly

function exact_u(x)
    return sin(omegax*x[1]) * cos(omegay*x[2])
end

function f(x)
    c = 1 + omegax^2 + omegay^2
    return c * exact_u(x)
end

ell_f(v, z) = P1SourceTimesFunc!(v, z, f)
lin_functionals = [ ("Omega", ell_f) ]

maxerr = zeros(refinements+1)
@printf("%10s  %12s  %8s  %8s\n\n",
        "N", "max error", "rate", "seconds")
for k = 0:refinements
    start = time()
    mesh = read_msh_file("rect$k.msh")
    vp = VariationalProblem(mesh, ["Left", "Right"])
    add_bilin_form!(vp, "Omega", grad_dot_grad!)
    add_bilin_form!(vp, "Omega", func_times_func!)
    add_lin_functnl!(vp, "Omega", source_times_func!, f)
    # Implicit zero boundary conditions.
    A, b = assembled_linear_system(vp)
    ufree = A \ b
    uh = complete_soln(ufree, vp)
    u = get_nodal_vals(exact_u, mesh)
    finish = time()
    maxerr[k+1] = maximum(abs.(uh-u))
    N = length(ufree)
    if k == 0
        @printf("%10d  %12.4e\n", N, maxerr[k+1])
    else
        rate = log2(maxerr[k]/maxerr[k+1])
        elapsed = finish - start
        @printf("%10d  %12.4e  %8.4f  %8.4f\n",
                N, maxerr[k+1], rate, elapsed)
    end
end


using LinearAlgebraicRepresentation
using SparseArrays
Lar = LinearAlgebraicRepresentation
L = Lar
using ViewerGL
GL = ViewerGL

function twocubegrids(n,m,p)
    V,(VV,EV,FV,CV) = Lar.cuboidGrid([n,m,p],true)
    mybox = (V,CV,FV,EV)

    twocubs = Lar.Struct([mybox, L.t(.3,.4,.5), L.r(pi/5,0,0), L.r(0,0,pi/12), mybox])

    V,CV,FV,EV = Lar.struct2lar(twocubs)
    GL.VIEW([ GL.GLGrid(V,FV, GL.COLORS[1]) ]);

    cop_EV = Lar.coboundary_0(EV::Lar.Cells);
    cop_EW = convert(Lar.ChainOp, cop_EV);
    cop_FE = Lar.coboundary_1(V, FV::Lar.Cells, EV::Lar.Cells);
    W = convert(Lar.Points, V');

    V, copEV, copFE, copCF = Lar.Arrangement.spatial_arrangement( W, cop_EW, cop_FE)

    EV = Lar.cop2lar(copEV)
    FE = [findnz(copFE[k,:])[1] for k=1:size(copFE,1)]
    FV = [collect(Set(cat(EV[e] for e in FE[f]))) for f=1:length(FE)]
    FV = convert(Lar.Cells, FV)
    W = convert(Lar.Points, V')
    WW = [[k] for k=1:size(W,2)]

    #GL.VIEW(GL.numbering(.2)((W,[WW,EV])));

    V,CVs,FVs,EVs = Lar.pols2tria(W, copEV, copFE, copCF)
    GL.VIEW(GL.GLExplode(V,FVs,1.5,1.5,1.5,99,1));
    GL.VIEW(GL.GLExplode(V,EVs,1.5,1.5,1.5,99,1));
    GL.VIEW(GL.GLExplode(V,CVs,5,5,5,99,1));

    return V, copEV, copFE, copCF
end

twocubegrids(5,5,5);


# Starting example file for Julia number types

# Julia defines a set of specific sub-types for numbers
# Signed integers: Int, Int8, Int16, Int32, Int64, BigInt
a = Int16(1)
println(typeof(a))

b = Int(2000)
println(typeof(b))

# Unsigned integers: UInt, UInt8, UInt16, UInt32, UInt64
c = UInt16(1)
println(typeof(c))

d = UInt(2000)
println(typeof(d))

# typemax() and typemin() will provide max and min values
println(typemax(Int8))
println(typemax(Int32))
println(typemax(Int64))
println(typemin(Int64))

# Use the WORDSIZE property to see what type of system this is
println(Sys.WORD_SIZE)

# Trying to assign a number too large for the type will fail and give an error
# CANNOT do: a = UInt(300)

# Special values represent Infinity and not-a-number
println(1.0 / 0.0)
println(1.0 / Inf)
println(0.0 / 0.0)

# zero() and one() functions produce values for a given type
println(zero(Float64))
println(one(UInt32))


# morlet wavelet
function morlet(F::T, fs::T; dtype=Float32) where {T}
    wtime = range(-1, stop=1, length=Int(2 * fs))
    s = 6 / ( 2 * pi * F)
    wavelet = exp.(2im * pi * F .* wtime) .* exp.(-wtime.^2 ./ (2*s^2));
    return wavelet |> Vector{Complex{dtype}}
end

"""
function dwt(data, fs, frange; reflection=false, wavelet=morlet)

Arguments:
- data
- fs
- frange

Keyword Arguments:
- reflection
- wavelet

Return result:

Note:
discrete wavelet transformation by convolution between the fft results of
original values and the wavelet function.
"""
function dwt(data::Vector{D}, fs::T, frange::Vector{T};
             reflection::Bool=false, wavelet::Function=morlet) where {D, T}
    data_fft = data
    if reflection
        data_flip = reverse(data_fft, dims=1)
        data_fft = [data_flip;data_fft;data_flip]
    end

    data_fft = data_fft |> Vector{Complex{Float32}}
    result = zeros(Complex{Float32}, size(data,1), length(frange))

    for (fidx, ftarget) in enumerate(frange)
        w = wavelet(ftarget, fs)
        if reflection
            result[:, fidx] = DSP.conv(data_fft, w)[Int(fs)+size(data,1):end-Int(fs)-size(data, 1)]
        else
            result[:, fidx] = DSP.conv(data_fft, w)[Int(fs):end-Int(fs)]
        end
    end

    result
end


using QuasiGeostrophy, LinearAlgebra, Test, FFTW, BenchmarkTools, Plots

# boiler plate definitions
Ωxy = S¹(0,2π) × S¹(0,2π)
Nx = 2^6; Ny = 2^6;
fourier_grid = FourierGrid((Nx, Ny), Ωxy)
x, y = fourier_grid.grid
kx, ky = fourier_grid.wavenumbers
u = @. 1 * sin(x) * sin(y) + 0im
v = @. 1 * cos(x) * cos(y) + 0im
û = copy(u)
v̂ = copy(v)

FFTW.set_num_threads(Threads.nthreads())
P = plan_fft(u) # typeof(iP) <: AbstractFFTs.ScaledPlan
iP = plan_ifft(u)

uv = P * (u .* v)
norm(uv - convolve(P * u, P * v, P, iP)) / norm(uv)

## Define Parameter
const Re  = 100.0
const Δt = 1.0 / Re
const G  = 4.0

## Define forcing
forcing = @. G * sin(5 * x) * sin(5 * y) + 0im
f̂ = copy(forcing)
mul!(f̂, P, forcing)

## Define Operators
∂x = FourierOperator(im .* kx)
∂y = FourierOperator(im .* ky)
Δ = ∂x^2 + ∂y^2
Δ⁻¹ = inv(Δ)

function get_pressure(u_rhs, v_rhs, ∂x, ∂y, G)
    return G(∂x(u_rhs) + ∂y(v_rhs))
end

function closure_rhs(∂x, ∂y, Δ⁻¹, P, iP, forcing)
    function rhs(state)
        u = state[1]
        v = state[2]
        uu = convolve(u, u, P, iP)
        uv = convolve(u, v, P, iP)
        vv = convolve(v, v, P, iP)
        u_rhs = - ∂x(uu) - ∂y(uv) +  forcing
        v_rhs = - ∂x(uv) - ∂y(vv)
        p = get_pressure(u_rhs, v_rhs, ∂x, ∂y, Δ⁻¹)
        u_rhs -= ∂x(p)
        v_rhs -= ∂y(p)
        return [u_rhs, v_rhs]
    end
end

rhs = closure_rhs(∂x, ∂y, Δ⁻¹, P, iP, forcing)

function closure_lhs(Δ, Re, Δt)
    function lhs()
        u_lhs = 1 + -(Δt / Re) * Δ
        return [u_lhs, u_lhs]
    end
end

lhs = closure_lhs(Δ, Re, Δt)
operators = lhs()
lhs_operator = [inv(operator) for operator in operators]

function imex_step!(state, lhs_operator, rhs, Δt)
    rhs_state = rhs(state)
    current_state = copy(state)
    for i in eachindex(state)
        tmp = lhs_operator[i](current_state[i] .+ Δt .* rhs_state[i])
        @. state[i] =  tmp
    end
end

##
contourf(x[:], y[:], real.(forcing)', linewidth = 0, color = :thermometer)
##
u = @. 1 * sin(x) * sin(y) + 0im
v = @. 1 * cos(x) * cos(y) + 0im
û = copy(u)
v̂ = copy(v)
mul!(û, P, u)
mul!(v̂, P, v)
# û, v̂ = copy(good_start)
state = [û, v̂]
## Checking Incompressibility
rhs_tmp = rhs(state)
maximum(abs.(∂x(rhs_tmp[1]) + ∂y(rhs_tmp[2])))
##
for i in 1:20000
    imex_step!(state, lhs_operator, rhs, Δt)
    if (i%1000) == 0
        println(i)
        ω̂ = ∂x(v̂) - ∂y(û)
        mul!(u, iP, ω̂)
        local p1 = contourf(x[:],y[:], real.(u)', linewidth = 0, color = :thermometer)
        display(p1)
        println("checking conergence to steady state")
        println(norm(rhs(state) + (1/Re) * Δ.(state)) / (norm(rhs(state))))
        sleep(0.01)
    end
end
# good_start = copy(state)
## check steady state
# include(pwd() * "/sandbox/two_d_navier_stokes.jl")


module ProblemTest
using LinearAlgebra
P1 = (sol = [[1,0], [0, 1]], vsol = -1,  A = [1 1], b = [1], c = [-1, -1], isoptimal = true, isbounded = true, isfeasible = true)
P2 = (sol = [[1,0], [0, 1]], vsol = -1, A = [1 1; 1 1], b = [1, 1], c = [-1, -1], isoptimal = true, isbounded = true, isfeasible = true)
function kmstar(n::Int, ::Type{T} = Int) where T
    xstar = zeros(T, 2*n)
    xstar[n] = 100^(n - 1)
    rest = [100^(k - 1) for k in 1:n]
    rest[end] = 0
    xstar[n+1:end] = rest
    return xstar
end
function km(n::Int, ::Type{T} = Int) where T
    c = [10^(n - i) for i in 1:n]
    b = [100^(i - 1) for i in 1:n]
    A = zeros(T, n, n)
    for i in 1:n
        A[i, 1:i] = 2*[10^(i - j) for j in 1:i]
    end
    A[:, :] -= I
    A = [A I]
    c = [c; zeros(T, n)]
    return A, b, -c, -100^(n - 1), kmstar(n)
end
PKM = [
    ((A, b, c, vstar, xstar) = km(i);
        (sol = [xstar],
        vsol = vstar,
            A = A,
            b = b,
            c = c,
            isoptimal = true,
            isbounded = true,
            isfeasible = true
        )
    )
    for i in 1:10
]
#Problem List
PL = [P1, P2, PKM...]
end


mutable struct LinearRegressor
    _w::AbstractArray{Float64,1}
    _var::Float64
end

function fitting(
    regressor::LinearRegressor,
    Phi::AbstractArray{Float64,2},
    t::AbstractArray{Float64,1},
)
    # Phi is the design matrix of size (n_samples, n_feature)
    # t is the target vector of size (sample_size, 1)
    # `pinv(Phi)` = (Phi^T * Phi)^(-1) * Phi^T
    w = pinv(Phi) * t
    error = Phi * w - t
    var = sum(error .^ 2) / size(t)[1]

    regressor._w = w
    regressor._var = var
end

function predict(
    regressor::LinearRegressor,
    phi::AbstractArray{Float64,1},
    return_std::Bool,
)
    # phi is the transformed vector of size n_features
    y = transpose(phi) * regressor._w
    std = regressor._var

    if return_std == true
        return y, std
    else
        return y
    end
end

function predict(
    regressor::LinearRegressor,
    Phi::AbstractArray{Float64,2},
    return_std::Bool,
)
    # Phi is the design matrix [phi(x_1), phi(x_2), ,,, phi(x_N)]^T of size (n_samples, n_features)
    y = Phi * regressor._w
    std = regressor._var

    if return_std == true
        return y, std
    else
        return y
    end
end


### A Pluto.jl notebook ###
# v0.14.1

using Markdown
using InteractiveUtils

# ╔═╡ b37139ed-ca54-4a70-a4aa-190fc4ff223a
using PlutoUI

# ╔═╡ 48988162-68d1-11eb-22dc-2bbc0e41043b
using Test

# ╔═╡ cb0ae3b2-42b8-485b-9c04-c1190e5eed94
PlutoUI.TableOfContents(indent=true, depth=4, aside=true)

# ╔═╡ b6e6f16a-68ce-11eb-320d-6b292123f85a
md"""
### Functions in julia
"""

# ╔═╡ d8456146-68ce-11eb-2872-3d66933b3385
begin
	DT = Float64

	function δ(a::DT, b::DT, c::DT)::DT
    	b * b - 4 * a * c
	end
end

# ╔═╡ fbec986e-68ce-11eb-1598-619db8bca4cc
function root(a::DT, b::DT, c::DT)::NamedTuple{(:x1, :x2),Tuple{DT, DT}}
    Δ = δ(a, b, c)
    Δ < 0 && throw(ArgumentError("No solution in ℜ"))

	x₁ = (-b + √ Δ) / 2a
   	x₂ = (-b - √ Δ) / 2a
    (x1 = x₁, x2 = x₂) # named tuple
end

# ╔═╡ 3750513e-68cf-11eb-0e76-7169551337af
with_terminal() do
	println(typeof(δ(1., 1., 1.)))
end

# ╔═╡ 50a85e6a-68cf-11eb-0e5e-2f2cbbfcce73
with_terminal() do
	r = root(2., -3., 1.)
	println("r: $(r) / typeof(r): $(typeof(r))")
end

# ╔═╡ 80512f66-68cf-11eb-3f58-498f113425d4
with_terminal() do
	r = root(1., -2., 1.)
	println("r: $(r) / typeof(r): $(typeof(r))")
end

# ╔═╡ 94a3ce9c-68cf-11eb-1a26-e57d02e78f19
with_terminal() do
	try
    	println(root(1., -1., 1.))
	catch ex
    	println("Intercepted: ", string(ex))
	end
end

# ╔═╡ a5f7141a-68cf-11eb-3dd1-9b04e8b62317
md"""
#### Function composition
"""

# ╔═╡ b2f4f754-68cf-11eb-3d27-dd9df6feb65f
function f(x::Real)::Real  ## Real is an abstract type
    2. * x
end

# ╔═╡ c06c4d8a-68cf-11eb-133e-ed1e6ae4c889
function g(x::Real)::Real
    x - 5.
end

# ╔═╡ cadfceca-68cf-11eb-1b1b-7fcc66d4c632
(g ∘ f)(4.)  # type: \circ<tab>: g(f(4.)) = 2 * 4. - 5, = 3

# ╔═╡ d97425bc-68cf-11eb-20c0-2dcaf9ede5f4
(f ∘ g)(7.)

# ╔═╡ e0553894-68cf-11eb-0e1f-5be65d986ed8
ary = [x for x in 0:5:100]

# ╔═╡ e4d32200-68cf-11eb-2ff7-0b227d2de510
nary = map(x -> 3x + 2, ary) |>
    x -> filter(isodd, x)

# ╔═╡ f250750e-68cf-11eb-3e4e-b5fcbcdf2320
map(x -> 3x + 2, ary) |>
    x -> filter(isodd, x) |>
    x -> reduce(+, x, init=0)

# ╔═╡ fcf27c16-68cf-11eb-3ae6-9b5a3d98c6ab
reduce(+, [x for x in 1:10], init=0)

# ╔═╡ 07a84580-68d0-11eb-0dcc-2dfc25b93d8b
reduce(+, 1:10, init=11)

# ╔═╡ 0b7b3b18-68d0-11eb-0e84-57cbc7e344d1
md"""
#### Function and Closure
"""

# ╔═╡ 1c25a7be-68d0-11eb-29f3-4dd237a2ef2e
adder = function(α::Number=2)
    x -> x + α
end

# ╔═╡ 44e600ba-68d0-11eb-122f-e9973916ffc8
add2 = adder()   # ≡ ad2(x) = x + 2

# ╔═╡ 53b8de8a-68d0-11eb-34e0-c1bbb3824ede
add2(2)

# ╔═╡ 7174059e-68d0-11eb-01fe-e712691890a4
begin
	add3 = adder(3)
	add3(10)
end

# ╔═╡ 840923b0-68d0-11eb-039f-05cd7abb2c04
map(add2, 1:10)

# ╔═╡ 8c579696-68d0-11eb-1321-ad2e8290b3bd
counter = function(; init=0)
    cnt = init

    inc() = cnt += 1
	dec() = cnt -= 1
	reset() = cnt = 0

	## Declare public:
    ()->(inc, dec, reset)
end

# ╔═╡ 2542f116-68d1-11eb-0eb2-0fa4398666ee
begin
	xc = counter()

	@test xc.inc() == 1
	@test xc.inc() == 2
	@test xc.inc() == 3

	@test xc.dec() == 2

	@test xc.reset() == 0
end

# ╔═╡ 6d948d44-68d1-11eb-283f-95c7209184f3
begin
	yc = counter(init=100)

	@test yc.inc() == 101
	@test yc.inc() == 102
	@test yc.inc() == 103

	@test yc.dec() == 102

	@test yc.reset() == 0
end

# ╔═╡ 8d553214-68d1-11eb-29d7-d791c9dff9f4
counter_ng = function(;init=0)
    cnt = init

	# closure
    λ = function(action=:inc)
        if action == :inc
            cnt += 1
        elseif action == :dec
            cnt -= 1
        elseif action == :reset
            cnt = 0
        else
            throw(ArgumentError("action $(action) not yet supported..."))
        end
    end

    λ
end

# ╔═╡ b2a41b98-68d1-11eb-21b3-9b48c657591d
begin
	ix = counter_ng()

	@test ix(:inc) == 1
	@test ix(:inc) == 2
	@test ix(:inc) == 3

	@test ix(:dec) == 2
	@test ix(:dec) == 1

	@test ix(:reset) == 0
end

# ╔═╡ eadfe19a-68d1-11eb-05c0-43915ae2d1b1
begin
	jy = counter_ng(init=100)

	@test jy(:inc) == 101
	@test jy(:inc) == 102
	@test jy(:inc) == 103

	@test jy(:dec) == 102
	@test_throws ArgumentError jy(:rec)

	@test jy(:reset) == 0
end

# ╔═╡ 3dc862b0-a58a-4731-a3c8-4eb44ded1443
html"""
<style>
  main {
        max-width: calc(800px + 25px + 6px);
  }
  .plutoui-toc.aside {
    background: linen;
  }
  h3, h4 {
        background: wheat;
  }
</style>
"""

# ╔═╡ Cell order:
# ╠═b37139ed-ca54-4a70-a4aa-190fc4ff223a
# ╠═cb0ae3b2-42b8-485b-9c04-c1190e5eed94
# ╟─b6e6f16a-68ce-11eb-320d-6b292123f85a
# ╠═d8456146-68ce-11eb-2872-3d66933b3385
# ╠═fbec986e-68ce-11eb-1598-619db8bca4cc
# ╠═48988162-68d1-11eb-22dc-2bbc0e41043b
# ╠═3750513e-68cf-11eb-0e76-7169551337af
# ╠═50a85e6a-68cf-11eb-0e5e-2f2cbbfcce73
# ╠═80512f66-68cf-11eb-3f58-498f113425d4
# ╠═94a3ce9c-68cf-11eb-1a26-e57d02e78f19
# ╟─a5f7141a-68cf-11eb-3dd1-9b04e8b62317
# ╠═b2f4f754-68cf-11eb-3d27-dd9df6feb65f
# ╠═c06c4d8a-68cf-11eb-133e-ed1e6ae4c889
# ╠═cadfceca-68cf-11eb-1b1b-7fcc66d4c632
# ╠═d97425bc-68cf-11eb-20c0-2dcaf9ede5f4
# ╠═e0553894-68cf-11eb-0e1f-5be65d986ed8
# ╠═e4d32200-68cf-11eb-2ff7-0b227d2de510
# ╠═f250750e-68cf-11eb-3e4e-b5fcbcdf2320
# ╠═fcf27c16-68cf-11eb-3ae6-9b5a3d98c6ab
# ╠═07a84580-68d0-11eb-0dcc-2dfc25b93d8b
# ╟─0b7b3b18-68d0-11eb-0e84-57cbc7e344d1
# ╠═1c25a7be-68d0-11eb-29f3-4dd237a2ef2e
# ╠═44e600ba-68d0-11eb-122f-e9973916ffc8
# ╠═53b8de8a-68d0-11eb-34e0-c1bbb3824ede
# ╠═7174059e-68d0-11eb-01fe-e712691890a4
# ╠═840923b0-68d0-11eb-039f-05cd7abb2c04
# ╠═8c579696-68d0-11eb-1321-ad2e8290b3bd
# ╠═2542f116-68d1-11eb-0eb2-0fa4398666ee
# ╠═6d948d44-68d1-11eb-283f-95c7209184f3
# ╠═8d553214-68d1-11eb-29d7-d791c9dff9f4
# ╠═b2a41b98-68d1-11eb-21b3-9b48c657591d
# ╠═eadfe19a-68d1-11eb-05c0-43915ae2d1b1
# ╟─3dc862b0-a58a-4731-a3c8-4eb44ded1443


##
#
# PiBBP tests
#

using PiBBP
using Base.Test

#
# Particular digits
#

@test digpi(0)       == 2
@test digpi(1000000) == 6

#
# It actually computes fractional part of $\pi$.
#

@test pi - (3 + sum(digpi(n) / 16^(n+1) for n in 0:14)) ≈ 0
@test pi - (3 + sum(digpi(n) / big(16)^(n+1) for n in 0:64)) ≈ 0


export fac, stirlingNumber

caching_fac = [1]
function fac(n)
    if n == 0
        return 1
    elseif size(caching_fac, 1) >= n
        return caching_fac[n]
    end
    fac = caching_fac[size(caching_fac, 1)]
    for i = size(caching_fac, 1)+1:n
        append!(caching_fac, fac * i)
    end
    return caching_fac[n]
end

function stirlingNumber(m, k)
    if k > m || k < 0 || (m > 0 && k == 0)
        return 0
    elseif m == 0 && k == 0
        return 1
    else
        return k * stirlingNumber(m - 1, k) + stirlingNumber(m - 1, k - 1)
    end
end


using CGMethod1D, FastGaussQuadrature
using Test

include("test_interpolation.jl")
include("test_quadrature.jl")


Pkg.add("RDatasets")
Pkg.add("Query")
Pkg.add("StatPlots")
Pkg.add("GR")

using RDatasets, Plots, Query, DataFrames

iris = dataset("datasets", "iris")

function SplitDFbySpecies(iris, species)
    df = @from i in iris begin
         @where i.Species == species
         @select {i.PetalLength, i.PetalWidth}
         @collect DataFrame
    end
    return df
end

function AttributeDescription(df, name)
    println(name, ":\n")
    colnames = names(df)
    ncols = size(colnames)[1]
    for i in 1:ncols
        println(colnames[i], ":")
        println(" mean: ", mean(df[i]))
        println(" var:  ", var(df[i]))
    end
    println()
end

function dfmean(df::DataFrame)
    return [mean(i) for i in df.columns]
end

function dfcov(df)
    cols = df.columns
    ncols = size(cols)[1]
    ndatas = size(cols[1])[1]
    array = zeros(ncols, ncols)
    mean = dfmean(df)
    for i in 1:ncols
        for j in 1:ncols
            array[i, j] = 1/ndatas * (cols[i] .- mean[i])' * (cols[j] .- mean[j])
        end
    end
    return array
end

function scale(df, iris)
    ncols = length(df.columns)
    means = dfmean(iris[[:PetalLength, :PetalWidth]])
    cov = dfcov(iris[[:PetalLength, :PetalWidth]])
    sigmas = [sqrt(cov[i, i]) for i in 1:ncols]

    scaler(x, mu, sigma) = (x .- mu) / sigma
    array = zeros(size(df))
    for i in 1:ncols
        array[:, i] = scaler(df[i], means[i], sigmas[i])
    end
    return array
end

setosa = SplitDFbySpecies(iris, "setosa")
versicolor = SplitDFbySpecies(iris, "versicolor")
virginica = SplitDFbySpecies(iris, "virginica")

gr()
scatter(setosa[1], setosa[2]; xlabel="Petal Length", ylabel="Petal Width",
        xlims=(1, 7), ylims=(-1, 4), m=(0.5, :s), label="setosa")
scatter!(versicolor[1], versicolor[2]; m=(0.5, :c), label="versicolor")
scatter!(virginica[1], virginica[2]; m=(0.5, :v), label="virginica")
savefig("figures/unscaled.png")

setosa_scaled = scale(setosa, iris)
versicolor_scaled = scale(versicolor, iris)
virginica_scaled = scale(virginica, iris)

scatter(setosa_scaled[:, 1], setosa_scaled[:, 2]; xlabel="Petal Length", ylabel="Petal Width",
        xlims=(-1.5, 1.5), ylims=(-1.5, 1.5), m=(0.5, :s), label="setosa")
scatter!(versicolor_scaled[:, 1], versicolor_scaled[:, 2]; m=(0.5, :c), label="versicolor")
scatter!(virginica_scaled[:, 1], virginica_scaled[:, 2]; m=(0.5, :v), label="virginica")
savefig("figures/scaled.png")


using Test

@testset "Neumann0 Boundary" begin
    N = 100
    d_order = 2
    approx_order = 2
    x = collect(1:1.0:N).^2
    A = DerivativeOperator{Float64}(2,2,1.0,N,:Neumann0,:Neumann0)
    boundary_points = A.boundary_point_count
    res = A*x
    @test res[boundary_points[1] + 1: N - boundary_points[2]] ≈ 2.0*ones(N - sum(boundary_points)) atol=10.0^approx_order
    FD = DerivativeOperator{Float64}(1,2,1.0,N,:Neumann0,:Neumann0)
    first_deriv = FD*res
    @test first_deriv[1] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0
    @test first_deriv[end] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0

    N = 1000
    d_order = 4
    approx_order = 10
    y = collect(1:1.0:N).^4 - 2*collect(1:1.0:N).^3 + collect(1:1.0:N).^2;
    A = DerivativeOperator{Float64}(d_order,approx_order,1.0,N,:Neumann0,:Neumann0)
    boundary_points = A.boundary_point_count
    res = A*y
    @test res[boundary_points[1] + 1: N - boundary_points[2]] ≈ 24.0*ones(N - sum(boundary_points)) atol=10.0^-1; # Float64 is less stable
    FD = DerivativeOperator{Float64}(1,2,1.0,N,:Neumann0,:Neumann0)
    first_deriv = FD*res
    @test first_deriv[1] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0
    @test first_deriv[end] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0

    A = DerivativeOperator{BigFloat}(d_order,approx_order,one(BigFloat),N,:Neumann0,:Neumann0)
    y = convert(Array{BigFloat, 1}, y)
    res = A*y
    @test res[boundary_points[1] + 1: N - boundary_points[2]] ≈ 24.0*ones(N - sum(boundary_points)) atol=10.0^-approx_order;
    FD = DerivativeOperator{BigFloat}(1,2,one(BigFloat),N,:Neumann0,:Neumann0)
    first_deriv = FD*res
    @test first_deriv[1] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0
    @test first_deriv[end] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0
end

@testset "General Neumann Boundary Condition" begin
    N = 100
    h_inv = 1/(N-1)
    d_order = 2
    approx_order = 2
    x = 0:h_inv:π
    A = DerivativeOperator{Float64}(2,2,h_inv,N,:Neumann,:Neumann;BC=(1,-1))
    boundary_points = A.boundary_point_count
    res = A*sin.(x)
    @test res ≈ cos.(x) atol=10.0^approx_order
    FD = DerivativeOperator{Float64}(1,2,h_inv,N,:None,:None)
    first_deriv_res = FD*res
    @test first_deriv_res[1] ≈ -1.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0
    @test first_deriv_res[end] ≈ +1.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0

    N = 1000
    h_inv = 1/(N-1)
    d_order = 2
    approx_order = 2
    x = 0:h_inv:1
    A = DerivativeOperator{Float64}(d_order,approx_order,h_inv,N,:Neumann,:Neumann;BC=(1,-1))
    boundary_points = A.boundary_point_count
    res = A*(x.*(1.0.-x))
    @test res ≈ -2*ones(size(x)) atol=10.0^-approx_order

    # A = DerivativeOperator{BigFloat}(d_order,approx_order,N,:Neumann0,:Neumann0)
    # y = convert(Array{BigFloat, 1}, y)
    # res = A*y
    # @test res[boundary_points + 1: N - boundary_points] ≈ 24.0*ones(N - 2*boundary_points) atol=10.0^-approx_order;
    # FD = DerivativeOperator{BigFloat}(1,2,N,:Neumann0,:Neumann0)
    # first_deriv = FD*res
    # @test first_deriv[1] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0
    # @test first_deriv[end] ≈ 0.0 atol=10.0^-1 ## Derivative at edges in Neumann 0 is 0

end


function step_size(x0)
    h = sqrt.( eps.( abs.(x0) ) )
    return h
end

function jacobian_fw!(ode_fcn, x0, jac::Matrix, h_tmp, f_tmp)
    # f: R^n -> R^m
    # g_ij = df_i / dx_j

    h_tmp .= sqrt.( eps.( abs.(x0) ) ) # n-by-1
    return jacobian_fw!(ode_fcn, x0, h_tmp, jac, f_tmp)
end

function jacobian_fw!(ode_fcn, x0, h, jac::Matrix, f_tmp)
    # f: R^n -> R^m
    # g_ij = df_i / dx_j

    f0    = ode_fcn(x0)                  # m-by-1
    copyto!(f_tmp, f0)
    return jacobian_fw!(ode_fcn, x0, h, f_tmp, jac)   # m-by-n
end

function jacobian_fw!(ode_fcn, x0, h, f0, jac::Matrix)
    # f: R^n -> R^m
    # g_ij = df_i / dx_j

    x1 = copy(x0)
    for kk = 1:length(x0)
        x1[kk]     += h[kk]
        f1          = ode_fcn(x1)
        jac[:, kk]  = (f1 .- f0) ./ h[kk]
        x1[kk]     -= h[kk]
    end
    return jac
end

function time_numdiff_fw!(ode_fcn, t0, dfdt_tmp)
    h = sqrt(eps(abs(t0)))
    return time_numdiff_fw!(ode_fcn, t0, h, dfdt_tmp)
end

function time_numdiff_fw!(ode_fcn, t0, h, dfdt_tmp)
    f0 = ode_fcn(t0)
    return time_numdiff_fw!(ode_fcn, t0, h, f0, dfdt_tmp)
end

function time_numdiff_fw!(ode_fcn, t0, h, f0, dfdt_tmp)
    f1 = ode_fcn(t0 + h)
    dfdt_tmp .= (f1 .- f0) ./ h
    return dfdt_tmp
end


#Linear Kalman filter with control input example
#This example closely follows that given on "Greg Czerniak's Website". Namely
#the canonball example on this page: http://greg.czerniak.info/guides/kalman1/
#Basically we shoot a canonball into the air at a given angle of elevation.
#Using observations of the canonball's velocity and position we're going to
#track the position of the canonball.

#To use the Kalman Filter we need three things:
#1) Some observations
#2) The parameters of the Kalman Filter model describing the system process and
#the observation model. In this example we also need the control input
#parameters.
#3) An initial guess of the state.
#

#Let's import the modules required to execute the Kalman Filter
using StateSpace
using Distributions
using Gadfly
using Colors

################################################################################
#Section: Generate noisy Observations
#-------------------------------------------------------------------------------
#First we're going to provide some parameter values required to model the motion
#of the canonball.

#Set the Parameters
elevation_angle = 45.0 #Angle (measured anti-clockwise) between the ground (zero degrees) and the direction the canon ball is fired
muzzle_speed = 100.0 #Speed at which the canonball leaves the muzzle
initial_velocity = [muzzle_speed*cos(deg2rad(elevation_angle)), muzzle_speed*sin(deg2rad(elevation_angle))] #initial x and y components of the velocity
gravAcc = 9.81 #gravitational acceleration
initial_location = [0.0, 0.0] # initial position of the canonball
Δt = 0.1 #time between each measurement

#Functions describing the position of canonball
x_pos(x0::Float64, Vx::Float64, t::Float64) = x0 + Vx*t
y_pos(y0::Float64, Vy::Float64, t::Float64, g::Float64) = y0 + Vy*t - (g * t^2)/2
#Function to describe the evolution of the velocity in the vertical direction
velocityY(Vy::Float64, t::Float64, g::Float64) = Vy - g * t

#Give variances of the observation noise for the position and velocity
x_pos_var = 200.0
y_pos_var = 200.0
Vx_var = 1.0
Vy_var = 1.0

#Set the number of observations and preallocate vectors to store true and noisy
#measurement values
numObs = 145
x_pos_true = Vector{Float64}(numObs)
x_pos_obs = Vector{Float64}(numObs)
y_pos_true = Vector{Float64}(numObs)
y_pos_obs = Vector{Float64}(numObs)

Vx_true = Vector{Float64}(numObs)
Vx_obs = Vector{Float64}(numObs)
Vy_true = Vector{Float64}(numObs)
Vy_obs = Vector{Float64}(numObs)

#Generate the data (true values and noisy observations)
for i in 1:numObs
    x_pos_true[i] = x_pos(initial_location[1], initial_velocity[1], (i-1)*Δt)
    y_pos_true[i] = y_pos(initial_location[2], initial_velocity[2], (i-1)*Δt, gravAcc)
    Vx_true[i] = initial_velocity[1]
    Vy_true[i] = velocityY(initial_velocity[2], (i-1)*Δt, gravAcc)

    x_pos_obs[i] = x_pos_true[i] + randn() * sqrt(x_pos_var)
    y_pos_obs[i] = y_pos_true[i] + randn() * sqrt(y_pos_var)
    Vx_obs[i] = Vx_true[i] + randn() * sqrt(Vx_var)
    Vy_obs[i] = Vy_true[i] + randn() * sqrt(Vy_var)
end
#Create the observations vector for the Kalman filter
observations = [x_pos_obs Vx_obs y_pos_obs Vy_obs]'
#End Section: Generate noisy Observations
################################################################################

################################################################################
#Section: Describe Kalman Filter parameters
#-------------------------------------------------------------------------------

#Describe the system parameters
process_matrix = [[1.0, Δt, 0.0, 0.0] [0.0, 1.0, 0.0, 0.0] [0.0, 0.0, 1.0, Δt] [0.0, 0.0, 0.0, 1.0]]'
process_covariance = 0.01*eye(4)
observation_matrix = eye(4)
observation_covariance = 0.2*eye(4)
control_matrix = [[0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 0.0, 0.0] [0.0, 0.0, 1.0, 0.0] [0.0, 0.0, 0.0, 1.0]]
control_input = [0.0, 0.0, -(gravAcc * Δt^2)/2, -(gravAcc * Δt)]

#Create an instance of the LKF with the control inputs
linCISMM = LinearGaussianCISSM(process_matrix, process_covariance, observation_matrix, observation_covariance, control_matrix, control_input)
#End Section: Describe Kalman Filter parameters
################################################################################

################################################################################
#Section: Set Initial Guess
#-------------------------------------------------------------------------------
initial_guess_state = [0.0, initial_velocity[1], 500.0, initial_velocity[2]]
initial_guess_covariance = eye(4)
initial_guess = MvNormal(initial_guess_state, initial_guess_covariance)
#End Section: Set Initial Guess
################################################################################

################################################################################
#Section: Execute Kalman Filter
#-------------------------------------------------------------------------------
filtered_state = filter(linCISMM, observations, initial_guess)
#End Section: Execute Kalman Filter
################################################################################

################################################################################
#Section: Plot Filtered results
#-------------------------------------------------------------------------------
#Here we are plotting the filtered results with Gadfly. See the Gadfly
#documentation for information about how plotting works if you are unfamiliar.
#Website: http://gadflyjl.org/
#Plot results
x_filt = Vector{Float64}(numObs)
y_filt = Vector{Float64}(numObs)
for i in 1:numObs
    current_state = filtered_state.state[i]
    x_filt[i] = current_state.μ[1]
    y_filt[i] = current_state.μ[3]
end

n = 3
getColors = distinguishable_colors(n, Color[LCHab(70, 60, 240)],
                                   transform=c -> deuteranopic(c, 0.5),
                                   lchoices=Float64[65, 70, 75, 80],
                                   cchoices=Float64[0, 50, 60, 70],
                                   hchoices=linspace(0, 330, 24))

cannonball_plot = plot(
    layer(x=x_pos_true, y=y_pos_true, Geom.line, Theme(default_color=getColors[3])),
    layer(x=[initial_guess_state[1]; x_filt], y=[initial_guess_state[3]; y_filt], Geom.line, Theme(default_color=getColors[1])),
    layer(x=x_pos_obs, y=y_pos_obs, Geom.point, Theme(default_color=getColors[2])),
    Guide.xlabel("X position"), Guide.ylabel("Y position"),
    Guide.manual_color_key("Colour Key",["Filtered Estimate", "Measurements","True Value "],[getColors[1],getColors[2],getColors[3]]),
    Guide.title("Measurement of a Canonball in Flight")
    )
#End Section: Plot Filtered results
################################################################################


"""
    jensen_alpha(asset_returns, benchmark_returns; risk_free=0.0)

Jensen's alpha, or simply alpha, is a risk-adjusted excess-performance measure. It indicates the average return of an investment above or below that predicted by the capital asset pricing model (CAPM). The measure adjusts the excess returns such that the risk is identical for the investment and benchmark.

# Arguments
- `asset_returns`:      Vector of asset returns.
- `benchmark_returns`:  Vector of benchmark returns (e.g. market portfolio returns for CAPM beta).
- `risk_free`:          Optional vector or scalar value denoting the risk-free return (must have same frequency as the provided returns, e.g. daily).

# Returns
Jensen's alpha measure.

# Sources
- Bacon, Carl (2008). Practical Portfolio Performance Measurement and Attribution, 2nd Edition, John Wiley & Sons Ltd. Page 72.
"""
function jensen_alpha(asset_returns, benchmark_returns; risk_free=0.0)
    (α, β) = capm(asset_returns, benchmark_returns; risk_free)
    α
end

"""
    modified_jensen(asset_returns, benchmark_returns; risk_free=0.0)

Divides Jensen’s alpha by the systematic risk, which measures the systematic risk-adjusted return per unit of systematic risk. See also `jensen_alpha`

# Arguments
- `asset_returns`:      Vector of asset returns.
- `benchmark_returns`:  Vector of benchmark returns (e.g. market portfolio returns for CAPM beta).
- `risk_free`:          Optional vector or scalar value denoting the risk-free return (must have same frequency as the provided returns, e.g. daily).

# Returns
Modified Jensen's alpha measure.

# Sources
- Bacon, Carl (2008). Practical Portfolio Performance Measurement and Attribution, 2nd Edition, John Wiley & Sons Ltd. Page 77.
"""
function modified_jensen(asset_returns, benchmark_returns; risk_free=0.0)
    (α, β) = capm(asset_returns, benchmark_returns; risk_free)
    α / β
end


# =========================================================================== #
# Activation Functions
# =========================================================================== #

abstract type AbstractActivationFunction
end

struct TanhActivationFunction <: AbstractActivationFunction
end

function (act_fcn::TanhActivationFunction)(x)
    return tanh(x)
end

function differentiate(act_fcn::TanhActivationFunction, x)
    return one(typeof(x)) - act_fcn(x)^2
end

function eval(act_fcn::TanhActivationFunction, x)
    return act_fcn(x)
end

struct ReluActivationFunction <: AbstractActivationFunction
end

function (act_fcn::ReluActivationFunction)(x)
    T = typeof(x)
    z = zero(T)
    return (x < z) * z + (x >= z) * x
end

function differentiate(act_fcn::ReluActivationFunction, x)
    T = typeof(x)
    z = zero(T)
    return (x < z) * z + (x >= z) * one(T)
end

function eval(act_fcn::ReluActivationFunction, x)
    return act_fcn(x)
end

struct IdentityActivationFunction <: AbstractActivationFunction
end

function (act_fcn::IdentityActivationFunction)(x)
    return x
end

function differentiate(act_fcn::IdentityActivationFunction, x)
    return one(typeof(x))
end

function eval(act_fcn::IdentityActivationFunction, x)
    return act_fcn(x)
end


"""
Return the rotation angle between two
vectors of any number of dimensions.

Note that if at least one of the vector is
the zero vector, then it returns 0.
by convention.

Arguments
----------
:Type{RotationAngle}
    must be type ``RotationAngle``

vector_1:AbstractArray
    vector 1

vector_2:AbstractArray
    vector 2

tol_near_zero=1e-7:AbstractFloat
    tolerance for being close to zero
"""
function gage(::Type{RotationAngle}, vector_1::AbstractArray, vector_2::AbstractArray,
    tol_near_zero::AbstractFloat=1e-7)
    @debug @assert length(vector_1) == length(vector_2)

    if nearly_zero(norm(vector_1,2), tol_near_zero)
        return 0.
    elseif nearly_zero(norm(vector_2,2), tol_near_zero)
        return 0.
    else
        #= developer's note: (Yuhang Wang 04/17/2016)
            Sometimes, the normalized dot product is > 1 or < -1 due to
            error in floating point division.
            Without proper handling, Julia will report domain error.
        =#
        normalized_dot = dot(vector_1/norm(vector_1,2), vector_2/norm(vector_2,2))
        if normalized_dot > 1.0
            return acos(floor(normalized_dot))
        elseif normalized_dot < -1.0
            return acos(ceil(normalized_dot))
        else
            return acos(normalized_dot)
        end
    end
end


# given a simplex and a face returns:
# +v if face is the v-th face of the simplex oriented according to the simplex
# -v if face is the v-th face of the simplex oriented oppositely to the simplex
# 0 is face is not a face of the simplex
function relorientation(face, simplex)

    v = setdiff(simplex, face)
    length(v) == 1 || return 0

    # find the position of the missing vertex
    v = v[1]
    i = Base.findfirst(simplex, v)
    s = (-1)^(i-1)

    # remove that vertex from the simplex
    face2 = Array{Int}(length(simplex)-1)
    for j in 1 : i-1
        face2[j] = simplex[j]
    end
    for j in i : length(simplex)-1
        face2[j] = simplex[j+1]
    end

    # get the permutation that maps face to face2
    #p = indexin(face, face2)
    p = [ findfirst(face2,v) for v in face ]

    return s * levicivita(p) * i
end

"""
    getcommonedge(cell1, cell2) -> e1, e2, edge

Returns in edge the common vertices of cell1 and cell2. e1 contains the index
of the vertex of cell1 opposite to this common edge, and with a plus or minus
sign depending on whether the orientation of the common edge is along or
against the internal orientation of cell1. Similar for e2.
"""
function getcommonedge(cell1, cell2)
    isct = intersect(cell1, cell2)
    relorientation(isct, cell1), relorientation(isct, cell2), isct
end


using LinearAlgebra: size
using Test
using SemiLagrangian

# include("mpoly/runtests.jl")
include("test_interpolation.jl")
include("test_lagrange.jl")
include("test_hermite.jl")
include("test_splinelu.jl")
include("test_bspline.jl")
include("testfftbig.jl")
include("test_util.jl")
include("test_mesh.jl")
include("test_advection.jl")
# include("test_poisson1d.jl")
include("test_poisson.jl")
include("test_poisson2d.jl")
# include("test_rotation1d.jl")
include("test_rotation.jl")
# include("test_translation1d.jl")
include("test_translation.jl")
include("test_swirling.jl")
include("test_quasigeostrophic.jl")


using Test, ApproxInference

const VecI  = AbstractVector # Input  Vector
const VecIO = AbstractVector # In/Out Vector
const MatI  = AbstractMatrix # Input  Matrix
const MatIO = AbstractMatrix # In/Out Matrix

print_head(s::String) = println("\033[1m\033[32mTesting Task\033[0m \033[1m\033[33m$s\033[0m")
print_body(s::String) = println("             $s")

##############################
#   NIST Dataset: BoxBOD     #
##############################
@objective struct BoxBOD
    x::Vector{Float64}; y::Vector{Float64}

    BoxBOD() = new([1., 2., 3., 5., 7., 10.], [109., 149., 149., 191., 213., 224.])
end

function ApproxInference.get_residual!(r::VecIO, θ::VecI, f::BoxBOD)
    x, y = f.x, f.y
    @inbounds θ1 = θ[1]
    @inbounds θ2 = θ[2]
    @simd for i in eachindex(x)
        @inbounds r[i] = y[i] - θ1 * (1.0 - exp(-θ2 * x[i]))
    end
    return nothing
end

function ApproxInference.get_jacobian!(J::MatIO, θ::VecI, f::BoxBOD)
    x = f.x
    @inbounds θ1 = θ[1]
    @inbounds θ2 = θ[2]
    @inbounds for i in eachindex(x)
        tmp = exp(-θ2 * x[i])
        J[i,1] = 1.0 - tmp
        J[i,2] = θ1 * x[i] * tmp
    end
    return nothing
end

include("./test-BoxBOD-LM.jl")
include("./test-BoxBOD-VI.jl")

##############################
#   NIST Dataset: Eckerle4   #
##############################
@objective struct Eckerle4
    x::Vector{Float64}; y::Vector{Float64}

    Eckerle4() = new(
        # x vector
        [400.0, 405.0, 410.0, 415.0, 420.0, 425.0, 430.0,
         435.0, 436.5, 438.0, 439.5, 441.0, 442.5, 444.0,
         445.5, 447.0, 448.5, 450.0, 451.5, 453.0, 454.5,
         456.0, 457.5, 459.0, 460.5, 462.0, 463.5, 465.0,
         470.0, 475.0, 480.0, 485.0, 490.0, 495.0, 500.0],
        # y vector
        [1.575000e-04, 1.699000e-04, 2.350000e-04, 3.102000e-04, 4.917000e-04,
         8.710000e-04, 1.741800e-03, 4.640000e-03, 6.589500e-03, 9.730200e-03,
         1.490020e-02, 2.373100e-02, 4.016830e-02, 7.125590e-02, 1.264458e-01,
         2.073413e-01, 2.902366e-01, 3.445623e-01, 3.698049e-01, 3.668534e-01,
         3.106727e-01, 2.078154e-01, 1.164354e-01, 6.167640e-02, 3.372000e-02,
         1.940230e-02, 1.178310e-02, 7.435700e-03, 2.273200e-03, 8.800000e-04,
         4.579000e-04, 2.345000e-04, 1.586000e-04, 1.143000e-04, 7.100000e-05]
    )
end

function ApproxInference.get_residual!(r::VecIO, θ::VecI, f::Eckerle4)
    x, y = f.x, f.y
    @inbounds θ1 = θ[1]
    @inbounds θ2 = θ[2]
    @inbounds θ3 = θ[3]
    @simd for i in eachindex(x)
        @inbounds r[i] = y[i] - θ1 * exp(-0.5 * abs2((x[i] - θ3) / θ2)) / θ2
    end
    return nothing
end

function ApproxInference.get_jacobian!(J::MatIO, θ::VecI, f::Eckerle4)
    x = f.x
    @inbounds θ1 = θ[1]
    @inbounds θ2 = θ[2]
    @inbounds θ3 = θ[3]

    θ2² = abs2(θ2)
    θ2³ = θ2 * θ2 * θ2

    @inbounds for i in eachindex(x)
        tmp1 = abs2(x[i] - θ3) / θ2²
        tmp2 = exp(-0.5 * tmp1)
        J[i,1] = tmp2 / θ2
        J[i,2] = θ1 * tmp2 * (tmp1 - 1.0) / θ2²
        J[i,3] = θ1 * (x[i] - θ3) * tmp2 / θ2³
    end
    return nothing
end

include("./test-Eckerle4-LM.jl")
include("./test-Eckerle4-VI.jl")