### A Pluto.jl notebook ###
# v0.11.14

using Markdown
using InteractiveUtils

# This Pluto notebook uses @bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of @bind gives bound variables a default value (instead of an error).
macro bind(def, element)
    quote
        local el = $(esc(element))
        global $(esc(def)) = Core.applicable(Base.get, el) ? Base.get(el) : missing
        el
    end
end

# ╔═╡ 49e8e6f6-0483-11eb-0a74-d94aa536c9ab
using LinearAlgebra, Plots, Zygote, PlutoUI

# ╔═╡ fccca5fc-0597-11eb-1033-a5cc3b71d854
md"""
# Constrained convex optimization

**Michiel Stock**
STMO

## Equality constraints

Consider the following optimization problem:

$$\min_{\mathbf{x}} f(\mathbf{x})$$
$$\text{subject to } g(\mathbf{x})=0\,.$$

For every point $\mathbf{x}$ on the surface $g(\mathbf{x})=0$, the gradient $\nabla g(\mathbf{x})$ is normal to this surface. This can be shown by considering a point $\mathbf{x}+\boldsymbol{\epsilon}$, also on the surface. If we make a Taylor expansion around $\mathbf{x}$, we have

$$g(\mathbf{x}+\boldsymbol{\epsilon})\approx g(\mathbf{x}) + \boldsymbol{\epsilon}^\top\nabla g(\mathbf{x})\,.$$

Given that both $\mathbf{x}$ and $\mathbf{x}+\boldsymbol{\epsilon}$ lie on the surface it follows that $g(\mathbf{x}+\boldsymbol{\epsilon})= g(\mathbf{x})$. In the limit that $||\boldsymbol{\epsilon}||\rightarrow 0$ we have that $\boldsymbol{\epsilon}^\top\nabla g(\mathbf{x})=0$. Because $\boldsymbol{\epsilon}$ is parallel to the surface $g(\mathbf{x})$, it follows that $\nabla g(\mathbf{x})$ is normal to the surface.

We seek a point $\mathbf{x}^\star$ on the surface such that $f(\mathbf{x})$ is minimized. For such a point, it should hold that the gradient w.r.t. $f$ should be parallel to $\nabla g$. Otherwise, it would be possible to give a small 'nudge' to $\mathbf{x}^\star$ in the direction of $\nabla f$ to decrease the function value, which would indicate that $\mathbf{x}^\star$ is not a minimizer. This figures below illustrate this point.

$$\nabla f(\mathbf{x}^\star) + \nu \nabla g (\mathbf{x}^\star)=0\,,$$
with $\nu\neq 0$ called the *Lagrange multiplier*. The constrained minimization problem can also be represented by a *Lagrangian*:
$$L(\mathbf{x}, \nu) 	\equiv f(\mathbf{x}) + \nu g(\mathbf{x})\,.$$
The constrained stationary condition is obtained by setting $\nabla_\mathbf{x} L(\mathbf{x}, \nu) =0$, the condition $\partial  L(\mathbf{x}, \nu)/\partial \nu=0$ leads to the constraint equation $g(\mathbf{x})=0$.

"""

# ╔═╡ 5e4eabba-0483-11eb-09a4-31ae6c582d3e
f((x1, x2)) = 2x1^2 +1.4x1*x2 + x2^2 - 0.3x1 + x1

# ╔═╡ 470048ec-0598-11eb-0d48-67357ed1f12c
md"In addition to a $g(\mathbf{x})$, we also implement a parametric version where $g(\mathbf{x})=0$ for plotting purposes."

# ╔═╡ 5503c842-0591-11eb-0104-359b9ebd5e98
md"Show the countour of $f(\mathbf{x})$:"

# ╔═╡ 47067fc8-058c-11eb-224c-6fe2e470107f
@bind show_f_contour CheckBox()

# ╔═╡ c369c65e-0598-11eb-0a90-dd4db070c99b
show_f_contour

# ╔═╡ 60328136-0591-11eb-1929-b7515a8c03d9
md"Show the countour of $g(\mathbf{x})$:"

# ╔═╡ d10b14c2-058c-11eb-3a43-7bca5d042969
@bind show_g_contour CheckBox()

# ╔═╡ ba835a96-0598-11eb-1d54-9365e3ea337f
show_g_contour

# ╔═╡ 681a031a-0591-11eb-2fec-8d94eca758f8
md"Show the function $g(\mathbf{x})=0$:"

# ╔═╡ 13467b42-058d-11eb-3023-b9d3f145335b
@bind show_g0_constraint CheckBox()

# ╔═╡ ca227ac2-0598-11eb-3a39-f1a90769ac52
show_g0_constraint

# ╔═╡ d8276f26-0598-11eb-1626-95f87cf6de22
md"Show the gradients:"

# ╔═╡ 60c41848-0592-11eb-39ef-7103ae834990
@bind show_gradients CheckBox()

# ╔═╡ d16723ca-0598-11eb-02f4-f1c5d45095a5
show_gradients

# ╔═╡ 95dc760a-058e-11eb-2149-898b2776079f
@bind t Slider(0:0.01:2π, default=4.3)

# ╔═╡ 41d8c8d0-0591-11eb-2cd7-3b53f91ef589
t

# ╔═╡ aef30ee8-0597-11eb-2203-ed0f9b14f0e4
t

# ╔═╡ 00a009a2-0599-11eb-341f-3364198e1753
md"We plot the objective value on the equality constraint."

# ╔═╡ 13b4ad4a-0599-11eb-0043-6bfc0eb23389
md"At the minimizer, the gradients are parallel."

# ╔═╡ 7e77e628-0707-11eb-1b2f-27b5c5819260
# compute angle between two vectors
angle(u, v) = acos(dot(u, v) / (norm(u) * norm(v)))

# ╔═╡ 358f21ca-0599-11eb-3bfb-0bb12a1465bb
md"""
## Inequality constraints

The same argument can be made for inequality constraints, i.e. solving

$$\min_{\mathbf{x}} f(\mathbf{x})$$
$$\text{subject to } g(\mathbf{x})\leq0\,.$$

Here, two situations can arise:

- **Inactive constraint**: the minimizer of $f$ lies in the region where $g(\mathbf{x}) < 0$. This corresponds to a Lagrange multiplier $\nu=0$. Note that the solution would be the same if the constraint was not present.
- **Active constraint**: the minimizer of $f$ lies in the region where $g(\mathbf{x}) > 0$. The solution of the constrained problem will lie on the bound where $g(\mathbf{x})=0$, similar to the equality-constrained problem and corresponds to a Lagrange multiplier $\nu>0$.

For both cases, the product $\nu g(\mathbf{x})=0$, the solution should thus satisfy the following conditions:
$$g(\mathbf{x}) \leq 0$$
$$\nu \geq 0$$
$$\nu g(\mathbf{x})=0\,.$$
These are called the *Karush-Kuhn-Tucker* conditions.

It is relatively straightforward to extend this framework towards multiple constraints (equality and inequality) by using several Lagrange multipliers.
"""

# ╔═╡ 2d7ae208-0599-11eb-1a91-277b764ad45e
md"Change the location of the constraint"

# ╔═╡ c98b4f9a-0592-11eb-2034-5f05e792cbd2
@bind R Slider(10:0.5:25, default=16)

# ╔═╡ bd7b55ae-0599-11eb-1056-ed3682fe8160
g((x1, x2)) = (x1-10)^2 + (x2-17)^2 - R^2

# ╔═╡ c10838a2-0599-11eb-32e3-1579a926f9b7
g0(t) = [R*cos(t) + 10, R*sin(t) + 17]

# ╔═╡ d2c4ffe2-0485-11eb-0e20-dd52446fa3a6
x = g0(t)

# ╔═╡ 6e89822c-070b-11eb-06f3-8b8e20aaa57d
f(x)

# ╔═╡ e944567a-0596-11eb-2cf1-75f196bab4d1
f'(x) / norm(f'(x))

# ╔═╡ 02673cec-0597-11eb-2db4-e9f7d9047ac0
g'(x) / norm(g'(x))

# ╔═╡ c04a0ffe-0707-11eb-2a74-b3b258b4aec6
angle(f'(x), g'(x))  # close to 0 or pi if stationary point

# ╔═╡ f47ce686-0485-11eb-2a33-4bd0b150b18b
g(x)

# ╔═╡ b0cf9a40-058f-11eb-13dc-b7c35328ef2a
begin
	plot(t -> f(g0(t)), 0:0.01:2π, label="f(x) where g(x)=0")
	xlabel!("t")
	ylabel!("objective")
	scatter!([t], [f(g0(t))], color=:purple, label="x")
end

# ╔═╡ be35a06c-0596-11eb-2988-61a79c7e6a87
R

# ╔═╡ a08e7230-0591-11eb-3956-fb6c2d756494
fconstr(x) = g(x) ≤ 0.0 ? f(x) : NaN

# ╔═╡ 81629936-0593-11eb-3fca-afa68e2ace3d
@bind t_approx Slider(0.1:0.05:10.0, default=1.0)

# ╔═╡ a1402cbc-0593-11eb-33b4-09a0c654eefd
t_approx

# ╔═╡ cc4209ee-0593-11eb-3762-e76539e2a455
Î₋(u) = u < 0.0 ? -(1/t_approx)* log(-u) : Inf

# ╔═╡ a398740e-0593-11eb-2f81-c92861e080b8
begin
	plot(-3:0.001:2, Î₋, label="logarithmic barier (t=$t_approx)")
	plot!([-3, 0, 0], [0, 0, 5], ls=:dash, label="indicator function")
	ylims!(-2, 5)
	xlabel!("u")
end

# ╔═╡ dce25666-0594-11eb-24f9-85eba4949d1e
fsoft(x) = g(x) ≤ 0.0 ? t_approx * f(x) + Î₋(g(x)) : NaN

# ╔═╡ 701efa2a-0596-11eb-2592-39eced2fb875
@bind show_soft_grads CheckBox()

# ╔═╡ 7c5d381a-0596-11eb-1dfe-0b7074093cd6
show_soft_grads

# ╔═╡ 7c934678-0599-11eb-1e6b-1f2c9003509a
md"## Utilities"

# ╔═╡ 865bbb8e-0483-11eb-0622-b9a8a6ba5d4c
x1min, x1max = -8.0, 15.0

# ╔═╡ 9736f388-0483-11eb-069a-0d5ed87efb73
x2min, x2max = -8.0, 12.0

# ╔═╡ f39f0106-0483-11eb-3617-5fb04388489e
grads(f, x) = 0.1f'(x) |> df -> Tuple([e] for e in df)

# ╔═╡ 828cbe72-0483-11eb-33b2-7f4f37db889f
begin
	plot(colorbar=false)
	show_f_contour && contourf!(x1min:0.1:x1max, x2min:0.1:x2max, (x1, x2) -> f((x1, x2)), label="f(x)", colorbar=true, color=:Blues, aspect=:equal)
	show_g_contour && contour!(x1min:0.1:x1max, x2min:0.1:x2max, (x1, x2) -> g((x1, x2)), color=:Reds)
	show_g0_constraint && plot!([g0(t)[1] for t in 0:0.01:2π], [g0(t)[2] for t in 0:0.01:2π], lw=2, color=:red, label="g(x)=0")
	show_g0_constraint && scatter!([x[1]], [x[2]], label="x", color=:purple)
	scatter!([0], [0], label="x* (no constraint)", color=:green)
	show_gradients && quiver!([x[1]], [x[2]], quiver=grads(f, x), color=:blue)
	show_gradients && quiver!([x[1]], [x[2]], quiver=grads(g, x), color=:orange)
	xlims!(x1min, x1max)
	ylims!(x2min, x2max)
	xlabel!("x1")
	ylabel!("x2")
end

# ╔═╡ c2640938-0591-11eb-058c-5949d1e316d3
begin
	contourf(x1min:0.01:x1max, x2min:0.01:x2max, (x1, x2) -> fconstr((x1, x2)), 			color=:Blues)
	show_g_contour && contour!(x1min:0.1:x1max, x2min:0.1:x2max, (x1, x2) -> g((x1, x2)), color=:Reds)
	contour!(x1min:0.1:x1max, x2min:0.1:x2max, (x1, x2) -> f((x1, x2)), 			color=:Blues)
	scatter!([x[1]], [x[2]], label="x", color=:purple)
	scatter!([0], [0], label="x* (no constraint)", color=:green)
	show_g0_constraint && plot!([g0(t)[1] for t in 0:0.01:2π], [g0(t)[2] for t in 0:0.01:2π], lw=2, color=:red, label="g(x)=0")
	show_gradients && quiver!([x[1]], [x[2]], quiver=grads(f, x), color=:blue)
	show_gradients && quiver!([x[1]], [x[2]], quiver=grads(g, x), color=:orange)
	xlims!(x1min, x1max)
	ylims!(x2min, x2max)
end

# ╔═╡ 52bacf14-0595-11eb-33c9-dd5c40125360
∇fsoft(x) = g(x) ≤ 0.0 ? fsoft'(x) : [0.0, 0.0]

# ╔═╡ 0b93638a-0595-11eb-3096-af3cc7202aa5
begin
	contourf(x1min:0.01:x1max, x2min:0.01:x2max, (x1, x2) -> fconstr((x1, x2)), 			color=:Blues)

	contour!(x1min:0.1:x1max, x2min:0.1:x2max, (x1, x2) -> f((x1, x2)), 			color=:Blues)
	show_g_contour && contour!(x1min:0.1:x1max, x2min:0.1:x2max, (x1, x2) -> fsoft((x1, x2)), color=:Reds)
	scatter!([x[1]], [x[2]], label="x", color=:purple)
	scatter!([0], [0], label="x* (no constraint)", color=:green)
	show_g0_constraint && plot!([g0(t)[1] for t in 0:0.01:2π], [g0(t)[2] for t in 0:0.01:2π], lw=2, color=:red, label="g(x)=0")
	show_soft_grads && quiver!(x1min:2:x1max, (x2min:2:x2max)', quiver=(x1, x2)->-0.05.*∇fsoft((x1, x2)), color=:pink)
	xlims!(x1min, x1max)
	ylims!(x2min, x2max)
end

# ╔═╡ 0a5783e2-0596-11eb-1e38-a5da40325159

# ╔═╡ Cell order:
# ╠═49e8e6f6-0483-11eb-0a74-d94aa536c9ab
# ╟─fccca5fc-0597-11eb-1033-a5cc3b71d854
# ╠═5e4eabba-0483-11eb-09a4-31ae6c582d3e
# ╠═bd7b55ae-0599-11eb-1056-ed3682fe8160
# ╟─470048ec-0598-11eb-0d48-67357ed1f12c
# ╠═c10838a2-0599-11eb-32e3-1579a926f9b7
# ╠═41d8c8d0-0591-11eb-2cd7-3b53f91ef589
# ╠═d2c4ffe2-0485-11eb-0e20-dd52446fa3a6
# ╠═6e89822c-070b-11eb-06f3-8b8e20aaa57d
# ╠═f47ce686-0485-11eb-2a33-4bd0b150b18b
# ╟─5503c842-0591-11eb-0104-359b9ebd5e98
# ╟─47067fc8-058c-11eb-224c-6fe2e470107f
# ╟─c369c65e-0598-11eb-0a90-dd4db070c99b
# ╟─60328136-0591-11eb-1929-b7515a8c03d9
# ╟─d10b14c2-058c-11eb-3a43-7bca5d042969
# ╟─ba835a96-0598-11eb-1d54-9365e3ea337f
# ╟─681a031a-0591-11eb-2fec-8d94eca758f8
# ╟─13467b42-058d-11eb-3023-b9d3f145335b
# ╟─ca227ac2-0598-11eb-3a39-f1a90769ac52
# ╟─d8276f26-0598-11eb-1626-95f87cf6de22
# ╟─60c41848-0592-11eb-39ef-7103ae834990
# ╟─d16723ca-0598-11eb-02f4-f1c5d45095a5
# ╟─828cbe72-0483-11eb-33b2-7f4f37db889f
# ╠═95dc760a-058e-11eb-2149-898b2776079f
# ╠═aef30ee8-0597-11eb-2203-ed0f9b14f0e4
# ╟─00a009a2-0599-11eb-341f-3364198e1753
# ╟─b0cf9a40-058f-11eb-13dc-b7c35328ef2a
# ╟─13b4ad4a-0599-11eb-0043-6bfc0eb23389
# ╠═e944567a-0596-11eb-2cf1-75f196bab4d1
# ╠═02673cec-0597-11eb-2db4-e9f7d9047ac0
# ╠═7e77e628-0707-11eb-1b2f-27b5c5819260
# ╠═c04a0ffe-0707-11eb-2a74-b3b258b4aec6
# ╟─358f21ca-0599-11eb-3bfb-0bb12a1465bb
# ╟─2d7ae208-0599-11eb-1a91-277b764ad45e
# ╠═c98b4f9a-0592-11eb-2034-5f05e792cbd2
# ╠═be35a06c-0596-11eb-2988-61a79c7e6a87
# ╠═a08e7230-0591-11eb-3956-fb6c2d756494
# ╟─c2640938-0591-11eb-058c-5949d1e316d3
# ╟─81629936-0593-11eb-3fca-afa68e2ace3d
# ╠═a1402cbc-0593-11eb-33b4-09a0c654eefd
# ╠═cc4209ee-0593-11eb-3762-e76539e2a455
# ╟─a398740e-0593-11eb-2f81-c92861e080b8
# ╠═dce25666-0594-11eb-24f9-85eba4949d1e
# ╟─701efa2a-0596-11eb-2592-39eced2fb875
# ╠═7c5d381a-0596-11eb-1dfe-0b7074093cd6
# ╟─0b93638a-0595-11eb-3096-af3cc7202aa5
# ╟─7c934678-0599-11eb-1e6b-1f2c9003509a
# ╠═865bbb8e-0483-11eb-0622-b9a8a6ba5d4c
# ╠═9736f388-0483-11eb-069a-0d5ed87efb73
# ╠═f39f0106-0483-11eb-3617-5fb04388489e
# ╠═52bacf14-0595-11eb-33c9-dd5c40125360
# ╠═0a5783e2-0596-11eb-1e38-a5da40325159


using SafeTestsets

@safetestset "Harmonic Oscillator                                                             " begin include("harmonic_oscillator_tests.jl") end
@safetestset "Kubo Oscillator                                                                 " begin include("kubo_oscillator_tests.jl") end
@safetestset "Lorenz Attractor                                                                " begin include("lorenz_attractor_tests.jl") end
@safetestset "Lotka-Volterra 2D                                                               " begin include("lotka_volterra_2d_tests.jl") end
@safetestset "Lotka-Volterra 2D with singular Lagrangian                                      " begin include("lotka_volterra_2d_singular_tests.jl") end
@safetestset "Lotka-Volterra 2D with symmetric Lagrangian                                     " begin include("lotka_volterra_2d_symmetric_tests.jl") end
@safetestset "Lotka-Volterra 2D with symmetric Lagrangian with gauge terms                    " begin include("lotka_volterra_2d_gauge_tests.jl") end
@safetestset "Lotka-Volterra 3D                                                               " begin include("lotka_volterra_3d_tests.jl") end
@safetestset "Lotka-Volterra 4D                                                               " begin include("lotka_volterra_4d_tests.jl") end
@safetestset "Lotka-Volterra 4D (Lagrangian)                                                  " begin include("lotka_volterra_4d_lagrangian_tests.jl") end
@safetestset "Point Vortices                                                                  " begin include("point_vortices_tests.jl") end


using OrdinaryDiffEq, DifferentialEquations

tspan = (0.0f0, 50.0f0)

"""
Simple SIR disease model from:
    https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology

Note: this does not include birth and death rates
"""
function sir_system(du, u, p, t;
                     U_func_time=U_func_time_trivial,
                     U_func_space=U_func_space_trivial)
    Ft = U_func_time(t)
    Fs = U_func_space(u)
    beta, gamma = p
    S, I, R = u

    N = sum([S,I,R])
    tmp = beta*I*S/N
    du[1] = -tmp
    du[2] = tmp - gamma*I
    du[3] = gamma*I

    # This forcing function must keep the overall population constant
    du[1] += -Ft[1]
    du[3] += Ft[1]
    # du .+= Ft .+ Fs
end

# Generate data, with a time component
p = [0.5, 0.2]
u0 = [999.0, 1.0, 0.0]
ts = range(tspan[1], tspan[2], length=5001)

function solve_sir_system(;U_func_time=U_func_time_trivial,
                              U_func_space=U_func_space_trivial)
    prob = ODEProblem((du, u, p, t)->sir_system(du, u, p, t,
                                U_func_time=U_func_time,
                                U_func_space=U_func_space),
                    u0, tspan, p)
    sol = solve(prob, Tsit5(), saveat=ts);#, adaptive=true,
        #dt=1e-5);#, abstol=1e-12, reltol=1e-5);
    # sol = solve(prob, AB4(), saveat=ts, dt=1e-5);
    return sol
end

#####
##### Turing function for Bayesian parameter estimation
#####
# Unforced version for use with Turing.jl
function sir_system(u, p, t)
    u = convert.(eltype(p),u)
    beta, gamma = p
    S, I, R = u

    N = sum([S,I,R])
    tmp = beta*I*S/N
    du = [-tmp; tmp - gamma*I; gamma*I]
    return du
end

#####
##### True model in SINDy syntax
#####
a, b = p
a = a / sum(u0)
#      S  I  R  c SS SI SR II IR RR
 A = [[0  0  0  0  0 -a  0  0  0  0];
      [0 -b  0  0  0  a  0  0  0  0];
      [0  b  0  0  0  0  0  0  0  0]]
n = size(A, 1)
sindy_library = Dict(
    "cross_terms"=>2,
    "constant"=>nothing
);
core_dyn_true = sindyModel(ts, A,
                            convert_string2function(sindy_library),
                            ["S", "I", "R"])

# Actually export
export ts, solve_sir_system, sir_system, core_dyn_true
    # ,lorenz_grad_residual


using Statistics
using StatsBase

# Absolute energy of the time series, the sum over the squared values
abs_energy(x::AbstractArray{T} where T<:Real) = mapreduce(x -> x * x, +, x) #using reduce(map()) is faster but with more mem allocation

# Highest absolute value of the time series x
absolute_maximum(x::AbstractArray{T} where T<:Real) = maximum(abs, x)

# Sum over the abs value of consecutive changes in the series x
function absolute_sum_of_changes(x::AbstractArray{T} where T<:Real)
    diff = Real[]

    for i in 1:length(x)-1
        push!(diff,x[i+1] - x[i])
    end

    return sum(map(abs,diff))
end

"""
Descriptive statistics on the autocorrelation of the time series x.
Calculates the value of an aggregation function fₐ over
    the autocorrelation for different lags.
Input:  x -> time series to calculate the feature of
        param -> dictionaries {"f_agg":x, "maxlag":n},
            with x, a string, the name of the aggregation funcion(e.g. mean, variance, ...),
            and n, an integer, the maximal number of lags to consider
"""
#TODO functions that calls function must have parameter of type "Function" not "String"
function agg_autocorrelation(x::AbstractArray{T} where T<:Real, f::Function, maxlag::Int)

    R::AbstractArray = zeros(maxlag)
    μ = mean(x)
    σ²= var(x)

    for i in 1:maxlag
        aux = 0.

        for t in 1:(length(x) - i)
            aux += (x[t] - μ) * (x[t+i] - μ)
        end

        R[i] = ( 1 / ( (length(x) - i) * σ² ) ) * aux
    end

    return f(R)

end

"""
Implements a vectorized approximate entropy algorithm.
Input: x-> time series to calculate the feature of
       m-> length of compared run of data (int)
       r-> filtering level, must be positive (float)
"""

"""
function approximate_entropy(x::AbstractArray{T} where T<:Real, m::Int, r::Real)
    dist(x_i, x_j) =

    n = length(x)

    x_re = Array([[ x[j] for j in i:(i+m-1) ] for i in 1:(n-m+1)])

    # GO AHEAD
end
"""

# augmented_dickey_fuller TODO

# Calculates the autocorrelation of the specified lag.
function autocorrelation(x::AbstractArray{T} where T<:Real, lag::Int)
    n = length(x)
    μ = mean(x)
    σ²= var(x)

    sum = 0.
    for t in 1:(n-lag)
        sum += (x[t] - μ) * (x[t+lag] - μ)
    end

    return (1 / ((n-lag) * σ²)) * sum
end

"""
Returns the correlation from first digit distribution when compared
 to the Newcomb-Benford's Law fistribution
"""
function benford_correlation(x::AbstractArray{T} where T<:Real)
    # Take first digit from data
    x = Array([last(digits(trunc(Int, abs(x[i])))) for i in 1:length(x)])

    # Benford distribution
    P = Array([log10(1 + 1/n) for n in 1:9])

    # Data distribution
    D = Array([count(i -> i==n, x)/length(x) for n in 1:9])

    # Return the correlation between benford distribution and data distribution
    return cor(P,D)
end

"""
First bins the values of x into max_bins equidistant bins.
Then calculates the value of binned entropy
"""
function binned_entropy(x::AbstractArray{T} where T<:Real, max_bins::Int)
    # Create equidistant max_bins
    edges = LinRange(minimum(x), maximum(x), max_bins)
    # Fit function requires column vector
    edges = collect(permutedims(edges))

    # Create histogram with the calculated bins
    h = fit(Histogram, vec(x), vec(edges))
    hist = h.weights

    # Vector pₖ with percentages of samples in bin k
    p = hist / length(x)

    # Where the percentage is 0, set to 1 to avoid Nan values?
    p[p.==0] .= 1

    # Return Entropy = - ∑ᵐᵢ₌₁ pᵢ * log(pᵢ)
    return - reduce(+, map(x->x*log(x), p))
end

# Uses c3 statistics to measure non linearity in the time series
function c3(x::AbstractArray{T} where T<:Real, lag::Int)
    n = length(x)
    if 2*lag ≥ n
        return 0
    end

    sum=0.
    for i in 1:(n - 2*lag)
        sum += x[i + 2*lag] * x[i + lag] * x[i]
    end

    return (1 / (n - 2*lag)) * sum
end

"""
First takes a corridor given by the quatiles ql and qh of
the distribution of x. Then calculates the average, absolute
value of consecutive changes of the series x inside this corridor.
"""
function change_quantiles(x::AbstractArray{T} where T<:Real, ql::Float64, qh::Float64, f_agg::Function)
        # Compute the quantiles
        q = quantile(vec(x), [ql, qh], sorted=true)

        # Corridor between the 2 quantiles
        corr = Real[]
        for i in 1:length(x)
            if x[i] >= q[1] && x[i] <= q[2]
                push!(corr, x[i])
            end
        end

        # Return the aggregation function of corridor
        return f_agg(corr)
end

#This function calculator is an estimate for a time series complexity
function cid_ce(x::AbstractArray{T} where T<:Real, normalize::Bool)
    n = length(x)

    if normalize
        s = std(x)
        if s != 0
            x = (x .- mean(x)) / s
        else
            return 0
        end
    end

    sum = 0.
    for i in 2:n
        sum += (x[i] - x[i-1])^2
    end

    return sqrt(sum)
end

# Returns the percentage of values in x that are higher than t
count_above(x::AbstractArray{T} where T<:Real, t::Float64) = count(i -> i > t, x) / length(x)

# Returns the number of values in x that are higher than the mean of x
count_above_mean(x::AbstractArray{T} where T<:Real) = count(i -> i > mean(x), x)

# Returns the percentage of values in x that are lower than t
count_below(x::AbstractArray{T} where T<:Real, t::Float64) = count(i -> i < t, x) / length(x)

# Returns the number of values in x that are lower than the mean of x
count_below_mean(x::AbstractArray{T} where T<:Real) = count(i -> i < mean(x), x)

"""
Return the first location of the maximum value of x.
The position is calculated relatively to the length of x
"""
first_location_of_maximum(x::AbstractArray{T} where T<:Real) = findmax(x)[2] / length(x)

"""
Return the first location of the minimum value of x.
The position is calculated relatively to the length of x
"""
first_location_of_minimum(x::AbstractArray{T} where T<:Real) = findmin(x)[2] / length(x)

# Checks if any value in x occurs more than once
has_duplicate(x::AbstractArray{T} where T<:Real) = return length(x) != length(unique(x))

# Checks if the maximum value of x is observed more than once
has_duplicate_max(x::AbstractArray{T} where T<:Real) = if count(i -> i == maximum(x), x) > 1 return true else return false end

# Checks if the minimum value of x is observed more than once
has_duplicate_min(x::AbstractArray{T} where T<:Real) = if count(i -> i == minimum(x), x) > 1 return true else return false end

"""
Does time series have large standard deviation?
Return boolean denoting if the standard deviation of x
is higher than 'r' times the range.
"""
large_standard_deviation(x::AbstractArray{T} where T<:Real, r::Float64) = return std(x) > r * (maximum(x) - minimum(x))

# Returns the relative last location of tha maximum value of x
last_location_of_maximum(x::AbstractArray{T} where T<:Real) = findmax(reverse(x))[2] / length(x)

# Returns the relative last location of tha minimum value of x
last_location_of_minimum(x::AbstractArray{T} where T<:Real) = findmin(reverse(x))[2] / length(x)

# Returns the length of the longest consecutive subsequence in x that is bigger than the mean of x
function longest_strike_above_mean(x::AbstractArray{T} where T<:Real)
    aux = x.>mean(x)
    max_strike = 0
    i=1

    while i <= length(aux)
        if aux[i] == 1
            c = 1
            i+=1
            while i <= length(aux)
                if aux[i] == 1
                    c+=1
                else
                    break
                end
                i+=1
            end
            if c > max_strike
                max_strike = c
            end
        else
            i+=1
        end
    end

    return max_strike
end

# Returns the length of the longest consecutive subsequence in x that is lower than the mean of x
function longest_strike_below_mean(x::AbstractArray{T} where T<:Real)
    aux = x.<mean(x)
    max_strike = 0
    i=1

    while i <= length(aux)
        if aux[i] == 1
            c = 1
            i+=1
            while i <= length(aux)
                if aux[i] == 1
                    c+=1
                else
                    break
                end
                i+=1
            end
            if c > max_strike
                max_strike = c
            end
        else
            i+=1
        end
    end

    return max_strike
end

# Average over first differences
function mean_abs_change(x::AbstractArray{T} where T<:Real)
    diff = Real[]

    for i in 1:length(x)-1
        push!(diff,x[i+1] - x[i])
    end

    return mean(map(abs, diff))
end

# Average over time sereis differences
mean_change(x::AbstractArray{T} where T<:Real) = (x[end] - x[1]) / (length(x) - 1)

# Calculates the arithmetic mean of the n absolute maximum values of the time series
function mean_n_absolute_max(x::AbstractArray{T} where T<:Real, n::Int)
    aux = sort(map(abs, x))
    return mean(aux[length(aux) - (n - 1) : end])
end

# Returns the mean value of a central approximation of the secondo derivative
mean_second_derivative_central(x::AbstractArray{T} where T<:Real) = (x[end] - x[end - 1] - x[2] - x[1]) / (2 * (length(x) - 2))

# Calculates the number of crossing of x on m.
function number_crossing_m(x::AbstractArray{T} where T<:Real, m::Float64)
    positive = x .> m
    diff = Int[]

    for i in 1:length(x) - 1
        push!(diff, positive[i+1] - positive[i])
    end

    return count(i -> i != 0, diff)
end

# TODO Gabriele
# # Returns the percentage of non-unique data points.
# function percentage_of_reoccurring_datapoints_to_all_datapoints(x::AbstractArray{T} where T<:Real)
#     value_counts = countmap(x)
#     reoccurring_values = 0
#
#     for item in value_counts
#         if item[2] > 1
#             reoccurring_values += 1
#         end
#     end
#
#     return reoccurring_values / length(x)
# end

# TODO Gabriele
# # Returns the percentage of values that are present in the time series more than once
# function percentage_of_reoccurring_values_to_all_values(x::AbstractArray{T} where T<:Real)
#     value_counts = countmap(x)
#     reoccurring_values = 0
#     unique_values = 0
#
#     for item in value_counts
#         if item[2] > 1
#             reoccurring_values += 1
#         else
#             unique_values += 1
#         end
#     end
#
#     return reoccurring_values / unique_values
# end

# Count observed values within the interval [min, max)
range_count(x::AbstractArray{T} where T<:Real, min::Float64, max::Float64) = count(i -> i>=min && i<max, x)

# Ratio of values that are mroe than r * std(x) away from the mean of x
function ratio_beyond_r_sigma(x::AbstractArray{T} where T<:Real, r::Float64)
    μ = mean(x)
    r_sigma = r * std(x)

    aux = map(abs, map(x -> x - μ, x))

    aux = Array([aux[i] > r_sigma ? aux[i] : 0 for i in 1:length(aux)])

    return sum(aux) / length(x)
end

# TODO Gabriele
# """
# Returns a factow which is 1 if all values in the time series occur only once,
# and below one if this is not the case
# """
# function ratio_value_number_to_time_series_length(x::AbstractArray{T} where T<:Real)
#     value_counts = countmap(x)
#     unique = Real[]
#
#     for item in value_counts
#         if item[2] == 1
#             push!(unique, item[2])
#         end
#     end
#
#     return length(unique) / length(x)
# end

# Returns the root mean square of the time series
root_mean_square(x::AbstractArray{T} where T<:Real) = sqrt(mean(map(x -> x^2, x)))

# Calculate and return sample entropy of x
function sample_entropy(x::AbstractArray{T} where T<:Real)
    m = 2
    tolerance = 0.2 * std(x)

    xmi = Array([x[i : i+m-1] for i in 1:length(x)-m])
    xmj = Array([x[i : i+m-1] for i in 1:length(x)-m+1])

    xmi = maximum(collect(hcat(xmi...)'), dims=2)
    xmj = maximum(collect(hcat(xmj...)'), dims=2)

    B = sum( Array([sum(x -> x <= tolerance, map(abs, xmi_i .- xmj)) - 1 for xmi_i in xmi]))

    m += 1
    xm = Array([x[i:i+m-1] for i in 1:length(x)-m+1])
    xm = maximum(collect(hcat(xm...)'), dims=2)

    A = sum( Array([sum(x -> x <= tolerance, map(abs, xm_i .- xm)) - 1 for xm_i in xm]))

    return -log(A/B)
end

# # Returns the sum of all data points, that are present in the time series more than once
# function sum_of_reoccurring_data_points(x::AbstractArray{T} where T<:Real)
#     value_counts = countmap(x)
#     reoc = Real[]
#
#     for item in value_counts
#         if item[2] > 1
#             for i in 1:item[2]
#                 push!(reoc, item[1])
#             end
#         end
#     end
#
#     return sum(reoc)
# end

function window_slice(x::AbstractArray{T} where T<:Real;
        from::Union{Missing,Int}=missing,
        to::Union{Missing,Int}=missing)
        ismissing(from) && ismissing(to) && return x
        ismissing(to) && return x[from:end]
        ismissing(from) && return x[1:to]
        return x[from:to]
end

function paa(x::AbstractArray{T} where T <: Real;
        n_chunks::Union{Missing,Int}=missing,
        f::Function=mean,
        kwargs...)
    if ismissing(n_chunks) return x
    else
        N = length(x)
        @assert 0 ≤ n_chunks && n_chunks ≤ N "The number of chunks must be in [0,$(N)]"

        y = Array{Float64}(undef, n_chunks) # TODO Float64?
        for i in 1:n_chunks
            l = Int(ceil((N*(i-1)/n_chunks) + 1))
            h = Int(ceil(N*i/n_chunks))
            y[i] = f(x[l:h]; kwargs...)
        end
        return y
    end
end


### A Pluto.jl notebook ###
# v0.12.7

using Markdown
using InteractiveUtils

# ╔═╡ 6fd616f6-1ea1-11eb-3814-8bfb4a096c49
using Pkg, DrWatson

# ╔═╡ 6ffe2628-1ea1-11eb-24ea-57f985146a72
begin
	@quickactivate "StatisticsWithJuliaPlutoNotebooks"
	using Distributions, Random, StatsBase, DataFrames, Plots
	Random.seed!(1)
end;

# ╔═╡ ed174bc4-1ea0-11eb-1e2f-a32874cec549
md"## Listing 3.18"

# ╔═╡ 70179b62-1ea1-11eb-13ce-7fb4d54bad4b
begin
	L, K, n  = 500, [450, 400, 250, 100, 50], 30
	hyperDists = [Hypergeometric(k,L-k,n) for k in K]
	xGrid = 0:1:n
	pmfs = [ pdf.(dist, xGrid) for dist in hyperDists ]
	labels = "Successes = " .* string.(K)
end

# ╔═╡ bfe8d6f0-2163-11eb-22e7-9b978bc1d407
begin
	bar( xGrid, pmfs,
		alpha=0.8, c=[:orange :purple :green :red :blue ],
		label=hcat(labels...), ylims=(0,0.25),
		xlabel="x", ylabel="Probability", legend=:top)
end

# ╔═╡ 70182cd8-1ea1-11eb-094d-8d0d49cf15f3
md"## End of listing 3.18"

# ╔═╡ Cell order:
# ╟─ed174bc4-1ea0-11eb-1e2f-a32874cec549
# ╠═6fd616f6-1ea1-11eb-3814-8bfb4a096c49
# ╠═6ffe2628-1ea1-11eb-24ea-57f985146a72
# ╠═70179b62-1ea1-11eb-13ce-7fb4d54bad4b
# ╠═bfe8d6f0-2163-11eb-22e7-9b978bc1d407
# ╟─70182cd8-1ea1-11eb-094d-8d0d49cf15f3


##
##  p o l y n o m . j l  Polynomials
##

function pval(p::Vector, x::Number)
    local n = length(p)
    if n == 0
        return NaN
    elseif n == 1
        return p[1]
    else
        y = p[1]
        for i in 2:n
            y = y * x + p[i]
        end
    end
    return y
end

function horner(p::Vector, x::Number)
    local n = length(p)
    if n == 0
        return NaN, NaN
    elseif n == 1
        return p[1], 0
    else
        y = p[1]; dy = 0
        for i in 2:n
            dy = dy * x + y
            y  =  y * x + p[i]
        end
    end
    return y, dy
end

function pzero{T<:Real}(p::Vector{T}, x0::T)
    local x = x0, tol = 2*eps(x)
    px, dpx = horner(p, x)
    df = -px/dpx
    niter = 1
    while abs(df) >= tol && niter <= 100
        x += df
        px, dpx = horner(p, x)
        df = -px/dpx
        niter += 1
    end
    if niter > 100
        warn("Number of iterations exceeded.")
        x = NaN
    end
    return x + df
end

function pfit()
    error("Polynomial fit not yet implemented.")
end


"""
    log1mexp(x)

Compute `log(1 - exp(x))` accurately. See
"**Accurately Computing log(1 - exp(.)) – Assessed by Rmpfr**" by Martin Mächler
(2012) for details:
https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
"""
function log1mexp(x::Real)
  if x > -log(2)
    log(-expm1(x))
  else
    log1p(-exp(x))
  end
end

"""
    logexpm1(x)

Compute `log(exp(x) - 1)` accurately. Thresholds were obtained similarly to
"**Accurately Computing log(1 - exp(.)) – Assessed by Rmpfr**" by Martin Mächler
(2012):
https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
"""
function logexpm1(x::Real)
  if x >= 37
    x
  elseif x >= 19
    x - exp(-x)
  else
    log(expm1(x))
  end
end

"""
    log1pexp(x)

Compute `log(1 + exp(x))` accurately. See
"**Accurately Computing log(1 - exp(.)) – Assessed by Rmpfr**" by Martin Mächler
(2012) for details:
https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
"""
function log1pexp(x::Real)
  if x <= -37
    exp(x)
  elseif x <= 18
    log1p(exp(x))
  elseif x <= 33.3
    x + exp(-x)
  else
    x
  end
end


using Random

#Generate a matrix of given rank and ratio kappa between
#largest and smallest nonzero singular values
function generate_matrix(
    m :: Int,       #Number of rows
    n :: Int,       #Number of columns
    r :: Int,       #Rank
    κ :: Float64    #Condition number
)
    #Compute Rotations
    u, _ = qr(randn(m, m))
    vt, _ = qr(randn(n, n))

    #Compute singular values
    sig = sort(rand(r), rev=true) |> λ -> λ./maximum(λ)
    a = (1 - 1/κ)/(1 - minimum(sig))
    b = 1 - a
    σ =  a.*sig .+ b
    (min(m, n) > r) && append!(σ, zeros(min(m, n) - r))

    #Return Matrix
    return u*diagm(m, n, 0 => σ)*vt
end

# Generate a set of isotropic vector
function isotropic_vector(n::Int)
    return randn(n)
end

function isotropic_vector!(x::AbstractVector)
    randn!(x)
    return nothing
end

function randomized_trace(A::AbstractMatrix, nsamples::Int)
    n = size(A, 1)
    w = zeros(n)
    x = zeros(n)
    estimator = 0.0
    # TODO: use compensated summation technique for large nsamples
    factor = 1.0/nsamples

    for i=1:nsamples
        isotropic_vector!(w)
        x .= A*w
        estimator += factor*(w'*x)
    end

    return estimator
end


export CnoidalWaveSerreGreenNaghdi
using Elliptic

"""
    CnoidalWaveSerreGreenNaghdi(param; P)

Compute the Serre-Green-Naghdi cnoidal wave with prescribed `h₀<h₁<h₂`.
`h_1` is the minimum, `h_2` is the maximum of the wave.
As `h₀ -> h₁`, the cnoidal wave converges towards the solitary wave.
See for instance Gavrilyuk, Nkonga, Shyue and Truskinovsky, doi:10.1088/1361-6544/ab95ac

# Arguments
- `param :: NamedTuple`: parameters of the problem containing `h₀<h₁<h₂` and dimensionless parameters `ϵ` and `μ`, and number of collocation points `N`.
- `P :: Int`: (keyword, optional, default = 2) the number of periods of the cnoidal wave in the constructed mesh.

# Return values
`(η,u,v,mesh,param)` with
- `η :: Vector{Float64}`: surface deformation;
- `u :: Vector{Float64}`: layer-averaged velocity;
- `v :: Vector{Float64}`: tangential velocity;
- `mesh :: Mesh`: mesh collocation points;
- `param :: NamedTuple`: useful parameters
"""
function CnoidalWaveSerreGreenNaghdi(
                param :: NamedTuple;
                P = 2 :: Int)

        ϵ = param.ϵ
        μ = param.μ

        h₀=param.h₀
        h₁=param.h₁
        h₂=param.h₂
        c = sqrt(h₀*h₁*h₂)
        @info string("The velocity is c=",c)
        m = sqrt((h₂-h₁)/(h₂-h₀))
        κ = sqrt(3*(h₂-h₀))/(2*c)/sqrt(μ)
        λ = Elliptic.K(m^2)/κ
        @info string("The period is 2*λ=",2*λ)
        mesh = Mesh((L=P*λ,N=param.N))
        formula = h₁ .-1 .+ (h₂-h₁)*(Jacobi.cn.(κ*mesh.x,m^2).^2)

        a₀ = h₀
        a₁ = h₂ - h₀
        formula2 = a₀ .-1 .+ a₁*(Jacobi.dn.(κ*mesh.x,m^2).^2)
        H₀= a₀+a₁*Elliptic.E(m^2)/Elliptic.K(m^2)
        u2 = c*(1 ./ H₀ .- 1 ./(1 .+ formula2))
        param = (h₀=h₀,h₁=h₁,h₂=h₂,a₀=a₀,a₁=a₁,H₀=H₀,c=c,λ=λ,m=m,κ=κ)

        η = formula/ϵ
        h = 1 .+ ϵ*η
        u = c*η./h
	k = mesh.k
        Dx=  1im * k
	F₀ = sqrt(μ)*Dx
        DxF(v) = real.(ifft(F₀ .* fft(v)))
	v = u - 1/3 ./h .* (DxF(h.^3 .*DxF(u)))
        #
        # h2 = 1 .+ ϵ*formula2
        # v2 = u2 - 1/3 ./h2 .* (DxF(h2.^3 .*DxF(u2)))

        return (η,u,v,mesh,param)

end


import QuadGK:quadgk
import Roots:find_zeros

" Compute M₀ by solving F(m) = 0 "
function mag(β, mass)

    F(m) = begin
        g(x, n, m) = (1 / π) * (exp(β * m * cos(x)) * cos(n * x))
        bessel0(x) = g(x, 0, m)
        bessel1(x) = g(x, 1, m)
        mass * quadgk(bessel1, 0, π)[1] / quadgk(bessel0, 0, π)[1] - m
    end

    find_zero(F, (0, mass))
end

#-

function Norm(f::Array{Float64,2}, delta1, delta2)
   return delta1 * sum(delta2 * sum(real(f), dims=1))
end

struct HMF

    b    :: Float64
    m    :: Float64
    ϵ    :: Float64

    function HMF( mass, T, ϵ )

        b = 1 / T
        m = mag(b, mass)
        w = sqrt(m)
        new( b, m, ϵ )
    end

end

function distribution( mesh1 :: UniformMesh,
                       mesh2 :: UniformMesh,
		       hmf   :: HMF)
    b  = hmf.b
    m  = hmf.m
    delta1 = mesh1.step
    delta2 = mesh2.step
    x  = mesh1.points
    v  = mesh2.points

    f  = transpose(exp.(-b * (v.^2 / 2))) .* exp.(b * m * cos.(x))
    a  = mass / Norm(real(f), delta1, delta2)
    @. a * transpose(exp(-b * (v^2) / 2)) * exp(b*m*cos(x)) * (1+ϵ*cos(x))

end


# # The Passport Problem

# Let's now apply what we have learnt to solve a real modelling problem.

# The [Passport Index Dataset](https://github.com/ilyankou/passport-index-dataset)
# lists travel visa requirements for 199 countries, in `.csv` format. Our task
# is to find out the minimum number of passports required to visit all
# countries without  requiring a visa.

# ## Data handling

# Read the following file into a dataframe:

const DATA_FILENAME = joinpath(@__DIR__, "data", "passport-index-matrix.csv")

passport_data = nothing  # Replace this with Code

# In this dataset, the first column represents a passport (=from) and each
# remaining column represents a foreign country (=to).

# The values in each cell are as follows:
# * 3 = visa-free travel
# * 2 = eTA is required
# * 1 = visa can be obtained on arrival
# * 0 = visa is required
# * -1 is for all instances where passport and destination are the same

# Extract the list of country names

countries = nothing  # Replace this with code

# Let's check that you found all 199 countries

@assert length(countries) == 199

# The values we are interested in are -1 and 3. Modify the dataframe so that
# the -1 and 3 are `1` (true), and all others are `0` (false).

nothing  # Replace this with code

# The values in the dataframe should now represent:
#  * 1 = no visa required for travel
#  * 0 = visa required for travel

# ## Visualization

# Produce a visualization of the passport data.  For example, you might want to
# plot a histogram of the number of countries that have visa-free travel.

nothing

# ## Modelling

# We can formulate the passport problem in math as follows:
# ```math
# \begin{aligned}
# \min && \sum_{i \in I} x_i \\
# \text{s.t.} && \sum_{i \in I} a_{i,j} \cdot x_i \geq 1 && \forall j \in I \\
# && x_i \in \{0,1\} && \forall i \in I
# \end{aligned}
# ```
# where $x_i$ is a binary variable indicating if we want to select passport $i$,
# and $a_{i,j}$ is 1 if there is visa-free travel between countries $i$ and $j$,
# and 0 otherwise.

# Formulate and solve this problem with JuMP.

nothing


function lglnodes(N,a,b)

# %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
# %
# % lglnodes.m
# %
# % Computes the Legendre-Gauss-Lobatto nodes, weights and the LGL Vandermonde
# % matrix. The LGL nodes are the zeros of (1-x^2)*P'_N(x). Useful for numerical
# % integration and spectral methods.
# %
# % Reference on LGL nodes and weights:
# %   C. Canuto, M. Y. Hussaini, A. Quarteroni, T. A. Tang, "Spectral Methods
# %   in Fluid Dynamics," Section 2.3. Springer-Verlag 1987
# %
# % Written by Greg von Winckel - 04/17/2004
# % Contact: gregvw@chtm.unm.edu
# %
# %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#
# % Truncation + 1
	N1=N+1;

	#% Use the Chebyshev-Gauss-Lobatto nodes as the first guess
	x=cos.(pi*collect(0:N)/N);

	# The Legendre Vandermonde Matrix
	P=zeros(N1,N1);

	# 	% Compute P_(N) using the recursion relation
	# 	% Compute its first and second derivatives and
	# 	% update x using the Newton-Raphson method.

	xold=2;

	while maximum(abs.(x .- xold))>eps()

	    xold=x;

	    P[:,1] .= 1;    P[:,2] = x;

	    for k=2:N
		P[:,k+1]=( (2*k-1)*x.*P[:,k]-(k-1)*P[:,k-1] )/k;
	    end

	    x=xold-( x.*P[:,N1]-P[:,N] )./( N1*P[:,N1] );

	end

	w=2 ./ (N*N1*P[:,N1].^2)/2 .* (b-a);
	x = (x .+ 1)*(b-a)/2.0 .+ a
	return x,w,P
end


"""
    to_matrix(points)

Convert collection of d-dimensional points to d×n matrix
"""
function to_matrix(points)
    dim = length(points[1])
    T = eltype(points[1])
    n = length(points)
    result = zeros(T, (dim, n))

    for (i, p) in enumerate(points)
        length(p) == dim || throw(ArgumentError("points must have the same length"))
        result[:, i] .= p
    end

    return result
end

"""
    distances(points, metric=Euclidean(1e-12))

Return distance matrix calculated from `points` with `metric`.
"""
function distances(points, metric=Euclidean(1e-12))
    points_mat = to_matrix(points)
    dists = pairwise(metric, points_mat; dims=2)
    return dists
end

"""
    radius(dists)
    radius(points[, metric=Euclidean(1e-12)])

Calculate the radius of the space. This is used for default `thresholds`.
"""
function radius(dists::AbstractMatrix)
    return minimum(maximum(abs, dists[:, i]) for i in 1:size(dists, 1))
end
function radius(dists::SparseMatrixCSC)
    return maximum(dists)
end
function radius(points, metric=Euclidean(1e-12))
    radius = Inf
    for p in points
        p_max = 0.0
        for q in points
            p == q && continue
            p_max = max(p_max, metric(SVector(p), SVector(q)))
        end
        radius = min(p_max, radius)
    end
    return radius
end


using GaussQuadrature
using SpecialFunctions: gamma
using Printf

#T = Float32
T = Float64
#T = BigFloat

println("\nFloating point data type is ", T)
println("\teps = ", eps(T))

npts = Dict(
    Float32 => Dict(  "Legendre"  => 10,
                      "Chebyshev" => 10,
                      "Jacobi"    => 10,
                      "Laguerre"  => 10,
                      "Hermite"   => 10,
                      "Logweight" => 10 ),
    Float64 => Dict(  "Legendre"  => 20,
                      "Chebyshev" => 12,
                      "Jacobi"    => 12,
                      "Laguerre"  => 20,
                      "Hermite"   => 16,
                      "Logweight" => 16 ),
    BigFloat => Dict( "Legendre" => 100,
                      "Chebyshev" => 60,
                      "Jacobi"    => 60,
                      "Laguerre"  => 600,
                      "Hermite"   => 80,
                      "Logweight" => 70 )
)

const half = one(T) / 2

function variant(endpt)
    if endpt == neither
        return "Default    "
    elseif endpt == left
        return "Left Radau "
    elseif endpt == right
        return "Right Radau"
    elseif endpt == both
        return "Lobatto    "
    else
        error("Unknown endpt")
    end
end

function table(f, rule, name, ans::T, n::Integer, endpts) where {T}
    println("\nTesting ", name, " rule with ", n, " points:")
    for endpt in endpts
        x, w = rule(n, endpt)
        integral = zero(typeof(x[1]))
        for j = 1:n
            integral += w[j] * f(x[j])
        end
        relerr = ( integral - ans ) / ans
        @printf("\t%s  %12.2e\n", variant(endpt), relerr)
    end
end

Beta(x::T, y::T) where {T} = gamma(x) * gamma(y) / gamma(x+y)

legendrefunc(x::T)  where {T} = one(T) / ( one(T) + x^2 )

jacobifunc(x::T, c::T, alpha::T, beta::T) where {T} = (
             (x+c)^convert(T,-alpha-beta-2) )

jacobiintegral(alpha::T, beta::T, c::T) where {T} = ( 2^(alpha+beta+1)
      * Beta(1+alpha,1+beta) / ( (c-1)^(1+alpha) * (c+1)^(1+beta) ) )

function laguerrefunc(x::T, alpha::T) where {T}
    if x > eps(T)
        r = -expm1(-x) / x
    else
        r = one(T)
    end
    return r^alpha
end

laguerreintegral(alpha::T) where {T} = Beta(one(T), 1+alpha)

hermiteintegral(a::T) where {T} = sqrt(convert(T, pi)) * exp(a^2)

function logweightintegral(::Type{T}) where {T}
    three = convert(T, 3)
    return (π/three)^2 * 2sqrt(three)
end

endpts = [neither, left, right, both]
table(legendrefunc, (n, endpt) -> legendre(T, n, endpt),
      "Legendre", convert(T, pi)/2, npts[T]["Legendre"], endpts)

c = convert(T, 3.0)
table(x -> jacobifunc(x, c, -half, -half),
      (n, endpt) -> chebyshev(T, n, 1, endpt), "Chebyshev (kind=1)",
      jacobiintegral(-half, -half, c), npts[T]["Chebyshev"], endpts)

table(x -> jacobifunc(x, c, half, half),
      (n, endpt) -> chebyshev(T, n, 2, endpt), "Chebyshev (kind=2)",
      jacobiintegral(half, half, c), npts[T]["Chebyshev"], endpts)

alpha = convert(T, 2)  / 5
beta  = convert(T, -1) / 5
table(x -> jacobifunc(x, c, alpha, beta),
      (n, endpt) -> jacobi(n, alpha, beta, endpt), "Jacobi",
      jacobiintegral(alpha, beta, c), npts[T]["Jacobi"], endpts)

table(x -> laguerrefunc(x, alpha),
      (n, endpt) -> laguerre(n, alpha, endpt), "Laguerre",
      laguerreintegral(alpha), npts[T]["Laguerre"], [neither, left])

a = convert(T, 6) / 5
table(x -> exp(2*a*x), (n, endpt) -> hermite(T, n), "Hermite",
      hermiteintegral(a), npts[T]["Hermite"], [neither])

half = convert(T, 1//2)
table(x -> (1-x^2)/(1+x^3), (n, endpt) -> logweight(n, -half, endpt),
      "Logweight", logweightintegral(T), npts[T]["Logweight"], endpts)


using JuMP
using Clp

m = Model(Clp.Optimizer)

@variable(m, x₁ ≥ 0)
@variable(m, x₂ ≥ 0)

@objective(m, Max, 3000x₁  + 5000x₂)
@constraint(m, 3x₁ + 2x₂ ≤ 18)
@constraint(m, x₁ ≤ 4)
@constraint(m, x₂ ≤ 6)

print(m)

optimize!(m)

status = termination_status(m)

println("Solution status: ", status)

println("Objective value: ", objective_value(m))
println("x₁ = ", value(x₁))
println("x₂ = ", value(x₂))


"""
Archimedes' Approximation of Pi

## References

- https://itech.fgcu.edu/faculty/clindsey/mhf4404/archimedes/archimedes.html
- https://betterexplained.com/articles/prehistoric-calculus-discovering-pi/
"""
module ArchimedesPi

#=

"If an angle of a triangle be bisected and the straight line cutting the angle cut the base also,
the segments of the base will have the same ratio as the remaining sides of the triangle."

-- Euclid's Elements

Let $r = 1$, $OSide_{n} = a$, $ISide_{n} = b$, $OSide_{2n} = x$, $ISide_{2n} = y$, we have

$$
(x / 2) / (a / 2 - x / 2) = 1 / (a / b)  =>
x = a * b / (a + b)
$$

and

$$
(y / 2) / (x / 2) = (b / 2) / y  =>
2 (y ^ 2) = x * b
$$

So the perimeter of circumscribing polygon of 2n sides is (the harmonic mean)

$$
O_{2n} = 2n * OSide_{2n} = 2n / (1 / a + 1 / b) = \frac{2}{1 / I_{n} + 1 / O_{n}}
$$

and the perimeter of inscribing polygon of 2n sides is (the geometric mean)

$$
I_{2n} = 2n * ISide_{2n}
       = 2n * \sqrt{x * b} / \sqrt{2}
       = n * \sqrt{2} * \sqrt{x * b}
       = \sqrt{2} * \sqrt{(OSide_{2n} * n) * (ISide_{n} * n)}
       = \sqrt{2} * \sqrt{(O_{2n} / 2) * I_{n}}
       = \sqrt{I_{n} * O_{2n}}
$$
=#

function calculate_pi(n_digits::Int=100)::BigFloat
    prec = Int(ceil(log2(10) * (n_digits + 1)))  # +1 because pi = 3.(...)
    setprecision(prec * 2) do
        # Starts from a hexagon
        outer = big"2.0" * sqrt(big"3.0")
        inner = big"3.0"
        df = outer - inner

        while true
            outer = big"2.0" / (one(BigFloat) / inner + one(BigFloat) / outer)
            inner = sqrt(inner * outer)

            df = begin
                new_df = outer - inner
                if new_df >= df
                    break
                end
                new_df
            end
        end

        pi_archimedes = (outer + inner) / big"2.0"
        # Still uses the default rounding, intentionally
        setprecision(prec)
        pi_archimedes + big(0)
    end
end

#=
Verifies the answer:

    pi_archimedes = calculate_pi()
    setprecision(precision(pi_archimedes))  # Warning: Set the global BigFloat precision!
    println(pi_archimedes - big(pi))
=#

end # module


#############################################################################
# log_determinant.jl
# given V = {1, 2, ..., n}, M ∈ ℜ^{n×n}, F(S) = logdet(M_S)
#############################################################################

import Base.logdet

export logdet
export sign, monotonicity, modularity, evaluate

mutable struct LogDeterminantAtom <: SubmodFunc
  head::Symbol
  id_hash::UInt64
  children::Tuple{AbstractMatrix, CombiSet}
  size::Tuple{Int, Int}
  matrix::AbstractMatrix
  setvariables::Array{CombiSet}

  function LogDeterminantAtom(matrix::AbstractMatrix, S::CombiSet)
    if size(matrix)[1] != S.cardinality
      error("Cannot define a log determinant function when the number of volumns of the matrix is different from the size of the set variable.")
    else
      children = (matrix, S)
      setvariables = get_sv(S)
      return new(:logdet, hash(children), children, (1, 1), matrix, setvariables)
    end
  end
end

logdet(matrix::AbstractMatrix, S::CombiSet) = LogDeterminantAtom(matrix, S)

function sign(F::LogDeterminantAtom)
  return NoSign()
end

function monotonicity(F::LogDeterminantAtom)
  return (NoMonotonicity(), )
end

function modularity(F::LogDeterminantAtom)
  if isposdef(F.matrix)
    return SubModularity()
  elseif isposdef(-F.matrix)
    return SuperModularity()
  else
    return NotDetermined()
  end
end

function evaluate(F::LogDeterminantAtom)
  set = get_elements(F.children[2])
  if length(set) == 0
    return 0
  end
  matrix = F.matrix[]
  return logdet(F.matrix[set, set])
end


"""
**Adam Optimizer**
```julia
    Adam(;α=0.001, β₁=0.9, β₂=0.999, ϵ=10e-8)
```

Algorithm:
```math
\\begin{align*}
    m_t =& \\beta_1 m_{t-1} + (1-\\beta_1)g_t\\\\
    v_t =& \\beta_2 v_{t-1} + (1-\\beta_2)g_t^2\\\\
    \\hat{m}_t =& \\frac{m_t}{1-\\beta_1^t}\\\\
    \\hat{v}_t =& \\frac{v_t}{1-\\beta_2^t}\\\\
    \\Delta x_t =& \\frac{\\alpha}{\\sqrt{\\hat{v}_t}+\\epsilon}\\hat{m}_t\\\\
\\end{align*}
```
[Algorithm Reference](https://arxiv.org/abs/1412.6980)
"""
mutable struct Adam <: Optimizer
    opt_type::String
    t::Int64
    ϵ::Float64
    α::Float64
    β₁::Float64
    β₂::Float64
    m_t::AbstractArray
    v_t::AbstractArray
end

function Adam(;α::Real=0.001, β₁::Real=0.9, β₂::Real=0.999, ϵ::Real=10e-8)
    @assert α > 0.0 "α must be greater than 0"
    @assert β₁ > 0.0 "β₁ must be greater than 0"
    @assert β₂ > 0.0 "β₂ must be greater than 0"
    @assert ϵ > 0.0 "ϵ must be greater than 0"

    Adam("Adam", 0, ϵ, α, β₁, β₂, [], [])
end

params(opt::Adam) = "ϵ=$(opt.ϵ), α=$(opt.α), β₁=$(opt.β₁), β₂=$(opt.β₂)"

function update(opt::Adam, g_t::AbstractArray{T}) where {T<:Real}
    # resize biased moment estimates if first iteration
    if opt.t == 0
        opt.m_t = zero(g_t)
        opt.v_t = zero(g_t)
    end

    # update timestep
    opt.t += 1

    # update biased first moment estimate
    opt.m_t = opt.β₁ * opt.m_t + (one(T) - opt.β₁) * g_t

    # update biased second raw moment estimate
    opt.v_t = opt.β₂ * opt.v_t + (one(T) - opt.β₂) * ((g_t) .^2)

    # compute bias corrected first moment estimate
    m̂_t = opt.m_t / (one(T) - opt.β₁^opt.t)

    # compute bias corrected second raw moment estimate
    v̂_t = opt.v_t / (one(T) - opt.β₂^opt.t)

    # apply update
    ρ = opt.α * m̂_t ./ (sqrt.(v̂_t .+ opt.ϵ))

    return ρ
end


using LinearAlgebra
using Statistics

function computegeometriccenter(points)
    x = [points[i][1] for i in 1:length(points)]
    y = [points[i][2] for i in 1:length(points)]
    z = [points[i][3] for i in 1:length(points)]

    return [mean(x), mean(y), mean(z)]
end

function computesurfacenormal(vectors)
    v1 = vectors[2] - vectors[1]
    v2 = vectors[3] - vectors[1]
    return 0.5*cross(v1, v2)
end

function computemagnitude(vector)
    return sqrt(dot(vector, vector))
end


"""
 softmax(X::AbstractMatrix{<:Real})
Return the softmax computed in a numerically stable way:
``σ(x) = exp.(x) ./ sum(exp.(x))``
Implementation taken from NNlib.jl.
"""
function softmax(X::AbstractMatrix{<:Real})
    S = copyto!(similar(X, eltype(X)), X)
    return softmax!(S)
end

"""
 softmax!(X::AbstractMatrix{<:Real})
Return the softmax computed in a numerically stable way:
``σ(x) = exp.(x) ./ sum(exp.(x))`` and store the result in the
input matrix
Implementation taken from NNlib.jl.
"""
function softmax!(X::AbstractMatrix{<:Real})
    max_ = maximum(X, dims=2)
    X .= exp.(X .- max_)
    X ./= sum(X, dims=2)
    return X
end


# # Labelled Graphs
# This example demonstrates how to define new C-Sets from existing C-Sets via the example of adding labels to a graph. We treat labels as members of an arbitrary FinSet of labels rather than a data attribute for pedagogical reasons. When you think of graphs where the labels are numbers, colors, or values of some kind, you would want to make them attributes. The motivation for this example is to be the simplest extension to the theory of graphs that you could possibly make.

using Catlab, Catlab.Theories
using Catlab.CategoricalAlgebra
using Catlab.Graphs
using Catlab.Graphics
using Catlab.Graphics.Graphviz
using Colors
draw(g) = to_graphviz(g, node_labels=true, edge_labels=true)

# We start with the theory of graphs, which is copied from `Catlab.Graphs.BasicGraphs`. The two objects are the edges and vertices and we have two functions `src,tgt` that assign to every edge the source vertex and target vertex respectively. Functors from this category to Set (diagrams in Set of this shape) are category theoretic graphs (directed multigraphs).
@present TheoryGraph(FreeSchema) begin
  V::Ob
  E::Ob
  src::Hom(E,V)
  tgt::Hom(E,V)
end

to_graphviz(Catlab.Graphs.BasicGraphs.TheoryGraph)

# To the theory of graphs we want to add a set of labels `L` and map that assigns to every vertex to its label in `L`.
@present TheoryLGraph <: TheoryGraph begin
  L::Ob
  label::Hom(V,L)
end

# Catlab will automatically generate all the data structure and algorithms (storing, mutating, serializing, etc.) our `LGraphs` for us. This snippet declares that the Julia type `LGraph` should be composed of objects of the functor category `TheoryLGraph → Skel(FinSet)`, where `Skel(FinSet)` is the subcategory of Set containing finite sets `1:n`. We want our Julia type `LGraph` to inherit from the Graphs.jl type `AbstractGraph` so that we can run graph algorithms on it. And we want the generated data structures to make an index of the maps `src`, `tgt`, and `label` so that looking up the in and outneighbors of vertex is fast and accessing all vertices by label is also fast.

@acset_type LGraph(TheoryLGraph, index=[:src,:tgt,:label]) <: AbstractGraph

# We need to tell Catlab how to convert our `LGraph` type into the normal `Graph` type by taking just the edges and vertices. This could be computed with Functorial Data Migration, but that is left for another sketch.
to_graph(g::LGraph) = begin
  h = Graphs.Graph(nparts(g,:V))
  for e in edges(g)
    add_edge!(h, g[e, :src], g[e,:tgt])
  end
  return h
end

# Graphviz doesn't automatically know how we want to draw the labels, so we have to explicitly provide code that converts them to colors on the vertices. Note that we aren't calling these colored graphs, because that would imply some connectivity constraints on which vertices are allowed to be colored with the same colors. These labels are arbitrary, but we use color to visually encode them.
GraphvizGraphs.to_graphviz(g::LGraph; kw...) =
  to_graphviz(GraphvizGraphs.to_graphviz_property_graph(g; kw...))

function GraphvizGraphs.to_graphviz_property_graph(g::LGraph; kw...)
  h = to_graph(g)
  pg = GraphvizGraphs.to_graphviz_property_graph(h; kw...)
  vcolors = hex.(range(colorant"#0021A5", stop=colorant"#FA4616", length=nparts(g, :L)))
  for v in vertices(g)
    l = g[v, :label]
    set_vprops!(pg, v, Dict(:color => "#$(vcolors[l])"))
  end
  pg
end

draw(G::LGraph) = to_graphviz(G, node_labels=true, edge_labels=true)

# Now we can start making some `LGraph` instances.
G = @acset LGraph begin
  V = 4
  E = 4
  L = 4
  src = [1,1,2,3]
  tgt = [2,3,4,4]
  label = [1,2,3,4]
end

draw(G)

# The graph `G` has a 1-1 map between vertices and labels. That isn't necessary.
H = @acset LGraph begin
  V = 4
  E = 4
  L = 3
  src = [1,1,2,3]
  tgt = [2,3,4,4]
  label = [1,2,2,3]
end

draw(H)

# We can look at some homomorphisms from G to H by their action on the labels or on the vertices.
homsᵥ(G,H) = map(α -> α[:V], homomorphisms(G, H))
homsₗ(G,H) = map(α -> α[:L], homomorphisms(G, H))

# αₗ: G→ H
homsₗ(G,H)

# αᵥ: G→ H
homsᵥ(G,H)

# Note that if we reverse the direction of our homomorphism search, we get fewer matches even though the two `LGraph`s are isomorphic as graphs. The fact that in `H` vertex two and three are the same label means we have to send them to the same vertex in `G`.
homsᵥ(H,G)

# We can build some bigger examples like A.
A = @acset LGraph begin
  V = 6
  E = 7
  L = 3
  src = [1,1,2,3,2,5,4]
  tgt = [2,3,4,4,5,6,6]
  label = [1,2,2,3,2,3]
end
draw(A)

# and B.
B = @acset LGraph begin
  V = 6
  E = 7
  L = 4
  src = [1,1,2,3,2,5,4]
  tgt = [2,3,4,4,5,6,6]
  label = [1,2,4,3,2,3]
end
draw(B)

# The morphisms from A to B and B to A are also different, showing how the labels affect the structure in this category.
homsᵥ(A,B)

# There are more morphisms from B to A than A to B.
homsᵥ(B,A)

# There are two automorphisms on A
homsᵥ(A,A)

# And two automorphisms on B
homsᵥ(B,B)

# But if we forget about the labels and look at the automorphisms of the underlying graph, we get more automorphisms!
A₀ = to_graph(A)
homsᵥ(A₀, A₀)

# ## Limits and Composition by Multiplication
# Catlab has an implementation of limits for any C-Sets over any schema. So, we can just ask about labelled graphs. Notice that we get more distinct colors in the product than in either initial graph. This is because the labels of the product are pairs of labels from the factors. If `G` has `n` colors and `H` has `m` colors `G×H` will have `n×m` colors.
draw(apex(product(G,G)))

# The graph above looks weirdly disconnected and probably wasn't what you expected to see as the product. When we compose with products, we often want to add the reflexive edges in order to get the expected notion of product.
add_loops!(G::LGraph) = begin
  for v in parts(G,:V)
    add_edge!(G, v,v)
  end
  return G
end
add_loops(G::LGraph) = add_loops!(copy(G))

Gᵣ = add_loops(G)
P = apex(product(Gᵣ, G))
draw(apex(product(Gᵣ, G)))

# We can look at the shape of commuting triangle, which is our favorite 3-vertex graph.
T = @acset LGraph begin
  V = 3
  E = 3
  L = 3
  src = [1,1,2]
  tgt = [2,3,3]
  label = [1,2,3]
end
Tᵣ = add_loops(T)
draw(Tᵣ)

E = @acset LGraph begin
  V = 2
  E = 1
  L = 2
  src = [1]
  tgt = [2]
  label = [1,2]
end
Eᵣ = add_loops(E)
draw(Eᵣ)

# We can draw the product of the edge graph and the triangle graph to get the shape of a triangular prism. You can view this product as extruding `Tᵣ` along `Eᵣ`. Catlab provides a `ReflexiveGraph` as a type that handles these self-loops more intelligently than we are here. Graphviz struggles with the layout here because the product graph will include edges that are a step in both directions. This [blog post](https://www.algebraicjulia.org/blog/post/2021/04/cset-graphs-3/) does a good job explaining products in differnt kinds of graph categories.
draw(apex(product(Tᵣ,Eᵣ)))
legs(product(Tᵣ, Eᵣ))[1][:V] |> collect

# Another limit is the pullback. If you have a cospan, which is a diagram of the shape `X ⟶ A ⟵ Y`, you can pull back one arrow along the other by solving a system of equations.
PB₂₂ = pullback(homomorphisms(Tᵣ,Eᵣ)[2],homomorphisms(Tᵣ,Eᵣ)[2]);
draw(apex(PB₂₂))

# Note that the pullback depends not only on X,A,Y but also on the two arrows. You can play around with the choice of morphisms to gain an intuition of how the pullback depends on the morphisms.
PB₂₃ = pullback(homomorphisms(Tᵣ,Eᵣ)[2],homomorphisms(Tᵣ,Eᵣ)[3]);
draw(apex(PB₂₃))

# By constructions, the pullback is always a subobject (monic homomorphism) into the product. Catlab can enumerate all such monoic homs.
homomorphisms(apex(PB₂₂), apex(product(Tᵣ,Tᵣ)), monic=true) |> length

# ## Colimits and Composition by Glueing
# The dual concept to limits is colimits and if limits have vibes of taking all pairs that satisfy certain constraints, colimits have the vibes of designers just gluing stuff together to make it work.

# In order to illustrate this we will be gluing triangles together to make a mesh. We start by defining the point `X`, which is the shape of the boundary along which we will glue and the morphism `ℓ₁`, which is the place in `T` that we consider as the boundary.
X = @acset LGraph begin
  E = 0
  V = 1
  L = 3
  label=[2]
end
ℓ₁ = ACSetTransformation(X,T, V=[2],L=1:3)
draw(X)

# We have to check that the morphism is valid before we go and compute out pushout.
is_natural(ℓ₁)
P = pushout(ℓ₁, ℓ₁)
draw(apex(P))

# Now we want to repeat the gluing process to get a bigger mesh. So we are going to need a bigger interface.
I = @acset LGraph begin
  V = 2
  L = 3
  label = [1,1]
end
draw(I)

# We have to specify how this interface embeds into both of the things we want to glue. In this case we are gluing a copy of `P` onto itself.
ll = ACSetTransformation(I, apex(P), V=[3,5], L =[3,1,2])
is_natural(ll)
lr = ACSetTransformation(I, apex(P), V=[1,4], L =[1,3,2])
is_natural(lr)
P₂ = pushout(ll, lr);
draw(apex(P₂))


@testset "Group properties" begin

    σ1 = Permutation([1, 2, 3])
    σ2 = Permutation([2, 3, 1])
    σ3 = Permutation([3, 1, 2])
    τ1 = Permutation([1, 3, 2])
    τ2 = Permutation([3, 2, 1])
    τ3 = Permutation([2, 1, 3])
    cycles = [σ1, σ2, σ3]
    transpositions = [τ1, τ2, τ3]

    @testset "Existence of identity" begin
        for π = [cycles..., transpositions...]
            @test σ1 * π == π * σ1 == π
        end
    end

    @testset "Existence of the inverse" begin
        @test σ2 * σ3 == σ3 * σ2 == σ1
        for τ = transpositions
            @test τ * τ == σ1
        end
    end

    @testset "Closeness under multiplication" begin
        @test σ2 * σ2 == σ3
        @test σ3 * σ3 == σ2
        @test τ1 * τ2 == τ2 * τ3 == τ3 * τ1 == σ2
        @test τ3 * τ2 == τ2 * τ1 == τ1 * τ3 == σ3
    end

    @testset "Associativity" begin
        p_group = [cycles..., transpositions...]
        for π1 = p_group, π2 = p_group, π3 = p_group
            @test (π1 * π2) * π3 == π1 * (π2 * π3)
        end
    end

end


#=
Compute the running median of a sequence of numbers. That is, given a stream of numbers, print out the median of the list so far on each new element.

Recall that the median of an even-numbered list is the average of the two middle numbers.

For example, given the sequence [2, 1, 5, 7, 2, 0, 5], your algorithm should print out:

2
1.5
2
3.5
2
2
2
=#

using Test
include("Solutions/problem33_running_median.jl")

@test print_running_median([2, 1, 5, 7, 2, 0, 5], store = true) == [2, 1.5, 2, 3.5, 2, 2.0, 2]


using PyPlot

L = 1.0
γ0 = 0.5
γL = -0.5
x = range(0, stop=L, length=201)
u(x, c) = ( ( L - x ) * γ0 + x * γL ) / L + (c/2) * x * ( L - x )

figure(1)
cvals = collect(6:-3:-6)
plot(x, u.(x,cvals'))
grid(true)
xlabel(L"x")
legend([latexstring("c=$(cvals[k])") for k = 1:length(cvals)])
savefig("bvp1d_const_rhs.pdf")


@doc raw"""
```
t = find_threshold(Balanced(), histogram, edges)
```
In balanced histogram thresholding, one interprets a  bin as a  physical weight
with a mass equal to its occupancy count. The balanced histogram method involves
iterating the following three steps: (1) choose the midpoint bin index as a
"pivot",  (2) compute the combined weight to the left and right of the pivot bin
and (3) remove the leftmost bin if the left side is the heaviest, and the
rightmost bin otherwise. The algorithm stops when only a single bin remains.
The last bin determines the sought-after threshold.

# Output

Returns a real number `t` in `edges`. The `edges` parameter represents an
`AbstractRange` which specifies the intervals associated with the histogram bins.

# Details
Let ``f_n`` (``n = 1 \ldots N``) denote the number of observations in the ``n``th
bin of the histogram. The balanced histogram method constructs a sequence
of nested intervals

```math
[1,N] \cap \mathbb{Z} \supset I_2 \supset I_3 \supset \ldots \supset I_{N-1},
```
where for ``k = 2 \ldots N-1``
```math
I_k = \begin{cases}
   I_{k-1} \setminus \{\min \left( I_{k-1} \right) \} &\text{if } \sum_{n = \min \left( I_{k-1} \right)}^{I_m}f_n \gt   \sum_{n =  I_m + 1}^{ \max \left( I_{k-1} \right)} f_n, \\
   I_{k-1} \setminus \{\max \left( I_{k-1} \right) \} &\text{otherwise},
\end{cases}
```
and ``I_m = \lfloor \frac{1}{2}\left(  \min \left( I_{k-1} \right) +  \max \left( I_{k-1} \right) \right) \rfloor ``.
The final interval ``I_{N-1}`` consists of a single element which is the bin index
corresponding to the desired threshold.

If one interprets a bin as a physical weight with a mass equal to its occupancy
count, then each step of the algorithm can be conceptualised as removing the
leftmost or rightmost bin to "balance" the resulting histogram on a pivot. The
pivot is defined to be the midpoint between the start and end points of the
interval under consideration.

If it turns out that the single element in ``I_{N-1}`` equals ``1`` or ``N`` then
the original histogram must have a single peak and the algorithm has failed to
find a suitable threshold. In this case the algorithm will fall back to using
the `UnimodalRosin` method to select the threshold.

# Arguments

The function arguments are described in more detail below.

##  `histogram`

An `AbstractArray` storing the frequency distribution.

##  `edges`

An `AbstractRange` specifying how the intervals for the frequency distribution
are divided.

# Example

Compute the threshold for the "cameraman" image in the `TestImages` package.

```julia

using TestImages, ImageContrastAdjustment, HistogramThresholding

img = testimage("cameraman")
edges, counts = build_histogram(img, 256)
#=
  The `counts` array stores at index 0 the frequencies that were below the
  first bin edge. Since we are seeking a threshold over the interval
  partitioned by `edges` we need to discard the first bin in `counts`
  so that the dimensions of `edges` and `counts` match.
=#
t = find_threshold(Balanced(), counts[1:end], edges)
```

# Reference

1. “BI-LEVEL IMAGE THRESHOLDING - A Fast Method”, Proceedings of the First International Conference on Bio-inspired Systems and Signal Processing, 2008. Available: [10.5220/0001064300700076](https://doi.org/10.5220/0001064300700076)
"""
function find_threshold(algorithm::Balanced, histogram::AbstractArray, edges::AbstractRange)
    # set initial start/middle/end points and weigths
    Iₛ = 1
    Iₑ = length(histogram)
    Iₘ = round(Int, (Iₛ + Iₑ) / 2)
    Wₗ = 0
    Wᵣ = 0
    for i = 1:Iₘ
        Wₗ += histogram[i]
    end
    for i = Iₘ:Iₑ
        Wᵣ += histogram[i]
    end
    while Iₛ < Iₑ
        if Wₗ < Wᵣ
            Wᵣ -= histogram[Iₑ]
            Iₑ -= 1
            if (Iₛ + Iₑ) / 2 < Iₘ
                Wₗ -= histogram[Iₘ]
                Wᵣ += histogram[Iₘ]
                Iₘ -= 1
            end
        else
            Wₗ -= histogram[Iₛ]
            Iₛ += 1
            if (Iₛ + Iₑ) / 2 > Iₘ
                Wₗ += histogram[Iₘ + 1]
                Wᵣ -= histogram[Iₘ + 1]
                Iₘ += 1
            end
        end
    end
    if Iₘ == 1 || Iₘ == length(histogram)
        @warn "Failed to threshold. Falling back to `UnimodalRosin` method."
        return find_threshold(UnimodalRosin(), histogram, edges)
    else
        return edges[Iₘ]
    end
end


""" Julia program to find the Length of Longest Increasing Subsequence

In this problem, given an array we have to find the length of the longest increasing subsequence that array can make.
The problem can be solved using Dynamic Programming
"""

function length_longest_increasing_subsequence(arr, n)
    max_len = 0
    # Initialize the dp array with the 1 as value, as the maximum length
    # at each point is atleast 1, by including that value in the sequence
    dp = ones(Int, n)

    """ Now Lets Fill the dp array in Bottom-Up manner
    Compare Each i'th element to its previous elements from 0 to i-1,
    If arr[i] > arr[j](where j = 0 to i-1), then it qualifies for increasing subsequence and
    If dp[i] < dp[j] + 1, then that subsequence  qualifies for being the longest one"""
    for i in 1:n
        for j in 1:(i-1)
            if(arr[i] > arr[j] && dp[i] < dp[j] + 1)
                dp[i] = dp[j] + 1
            end
        end
    end

    # Now Find the largest element in the dp array
    max_len = findmax(dp)[1]
    return max_len
end

print("What is the length of the array? ")
n = readline()
n = parse(Int, n)
if (n <= 0)
    println("No numbers present in the array!!!")
    exit()
end
arr = Int[]
print("Enter the numbers: ")
arr = [parse(Int, num) for num in split(readline())]
res = length_longest_increasing_subsequence(arr, n)
print("The length of the longest increasing subsequence of the given array is $res")

"""
Time Complexity - O(n^2), where 'n' is the size of the array
Space Complexity - O(n)

SAMPLE INPUT AND OUTPUT

SAMPLE I
What is the length of the array? 5
Enter the numbers: 5 4 3 2 1
The length of the longest increasing subsequence of the given array is 1

"""


#############################################################################
#############################################################################
#
# This file implements polynomial GCD
#
#############################################################################
#############################################################################

"""
The extended euclid algorithm for polynomials modulo prime.
"""

function extended_euclid_alg(a::PolynomialModP, b::PolynomialModP)
    prime = a.mod
    old_r, r = a, b
    old_s, s = one(PolynomialModP,prime), zero(PolynomialModP,prime)
    old_t, t = zero(PolynomialModP,prime), one(PolynomialModP,prime)

    while !iszero(r)

        q = divide(old_r, r) |> first

        old_r, r = r, old_r - q*r

        old_s, s = s, old_s - q*s

        old_t, t = t, old_t - q*t

    end
    g, s, t = old_r, old_s, old_t
    @assert s*a + t*b - g == 0
    return g, s, t
end

"""
The GCD of two polynomials modulo prime.
"""

gcd(a::PolynomialModP, b::PolynomialModP) = extended_euclid_alg(a,b) |> first


# function linsolve_tridiagonal(left_diag::Vector, main_diag::Vector, right_diag::Vector, rhs::Vector)
function linsolve_tridiagonal(left_diag, main_diag, right_diag, rhs)
    n_row = length(main_diag)
    n = length(left_diag)
    if n+1 != n_row
        error("Invalid dimension for left diag")
    end
    if length(right_diag) != n
        error("Invalid dimension for right diag")
    end

    x = zeros(n_row)

    a = copy(main_diag)
    r = copy(rhs)

    # fw elimination
    @inbounds for ii = 2:n_row
        a[ii] -= left_diag[ii-1] * right_diag[ii-1] / a[ii-1]
        r[ii] -= left_diag[ii-1] * r[ii-1] / a[ii-1]
    end

    # bw elimination
    x[n_row] = r[n_row] / a[n_row]
    @inbounds for ii = (n_row-1):-1:1
        x[ii] = (r[ii] - x[ii+1] * right_diag[ii]) / a[ii]
    end

    return x
end


function sma{T,N}(ta::TimeArray{T,N}, n::Int)
  tstamps = ta.timestamp[n:end]
  vals    = zeros(length(ta) - (n-1))
  cname   = ["sma$n"]
  for i in 1:length(ta) - (n-1)
    vals[i] =  mean(ta.values[i:i+(n-1)])
  end
  TimeArray(tstamps, vals, cname, ta.meta)
end

function ema{T,N}(ta::TimeArray{T,N}, n::Int; wilder=false)

  if  wilder
    k  = 1/n
  else
    k = 2/(n+1)
  end

  tstamps = ta.timestamp[n:end]
  vals    = ones(length(ta))
  vals[n] = sma(ta, n).values[1] # seed with first value an sma value
  for i = n+1:length(ta)
    vals[i] =  ta.values[i] * k + vals[i-1] * (1-k)
  end
  cname   = ["ema$n"]
  TimeArray(tstamps, vals[n:length(ta)], cname, ta.meta)
end

# Array dispatch for use by ta algorithms

function sma(a::Array{Float64,1}, n::Int)
  vals    = zeros(length(a) - (n-1))
  for i in 1:length(a) - (n-1)
    vals[i] =  mean(a[i:i+(n-1)])
  end
  vals
end

function ema(a::Array{Float64,1}, n::Int; wilder=false)

  if  wilder
    k  = 1/n
  else
    k = 2/(n+1)
  end

  vals    = ones(length(a))
  vals[n] = sma(a, n)[1] # seed with first value an sma value

  for i = n+1:length(a)
    vals[i] =  a[i] * k + vals[i-1] * (1-k)
  end
  vals[n:end]
end


using NLsolve

poly(x, p) = sum(p[n] * x^(n-1) for n in 1:length(p))
∂poly(x, p) = sum(n * p[n+1] * x^(n-1) for n in 1:length(p)-1)

"""
        fit_cubic(p1, p2, s1, s2)

Return the coeficients c such that f(x) = c₁ + c₂x + c₃x² + c₄x³ is a cubic polynomial that passes through the points p1 = (x1, y1) and p2 = (x2, y2) with slope s1 at p1 and slope s2 at p2.
"""
function fit_cubic(p1, p2, s1, s2)
    x1, y1 = p1
    x2, y2 = p2

    function f!(F, c)
        F[1] = poly(x1, c) - y1
        F[2] = poly(x2, c) - y2
        F[3] = ∂poly(x1, c) - s1
        F[4] = ∂poly(x2, c) - s2
    end

    results = nlsolve(f!, zeros(4))
    return results.zero
end


using JuMP
using Clp
m = Model(Clp.Optimizer)
@variable(m, x11 ≥ 0)
@variable(m, x12 ≥ 0)
@variable(m, x21  ≥ 0)
@variable(m, x22 ≥ 0)
@variable(m, x31 ≥ 0)
@variable(m, x32 ≥ 0)
@objective(m, Min, 80x11 + 100x21 + 102x31 + 251x12 + 108x22 + 68x32)
@constraint(m, x11 + x12 ≤ 1000)
@constraint(m, x21 + x22 ≤ 1500)
@constraint(m, x31 + x32 ≤ 1200)
@constraint(m, x11 + x21 + x31 == 2300)
@constraint(m, x12 + x22 + x32 == 1400)
print(m)
optimize!(m)
status = termination_status(m)
println("Solution status: ", status)
println("Objective value: ", objective_value(m))
println("x11: ", value(x11))
println("x12: ", value(x12))
println("x21 ", value(x21))
println("x22 ", value(x22))
println("x31", value(x31))
println("x32 ", value(x32))


# Unit 01 - Linear Classifiers and Generalizations

function perceptron(set, T=1000)
    n = length(set); d = length(set[1][1])
    θ = zeros(d); θ₀ = 0;
    for t in 1:T
        ϵ = 0
        for i in 1:n
            x = set[i][1]; y = set[i][2]
            if (y*(θ' * x + θ₀) <= 0)
                θ  = θ + y * x
                θ₀ = θ₀ + y
                ϵ += 1
            end
        end
        if (ϵ == 0)
            return (θ,θ₀,ϵ,t)
        end
    end
    return (θ,θ₀,ϵ,T)
end

T = 5000
set = [([7,8],-1),([4,2],-1),([2,7],1),([3,-2],1),([-3,-2],1)]
(θ,θ₀,ϵ,t) = perceptron(set,T)

using Plots, LinearAlgebra

θₙ = θ/norm(θ)
θ₀ₙ = θ₀/norm(θ) *  θₙ
x⁺ = vcat([([tuple[1][1] tuple[1][2]])  for tuple in set if tuple[2] == 1]...)
x⁻ = vcat([([tuple[1][1] tuple[1][2]])  for tuple in set if tuple[2] == -1]...)

scatter(x⁺[:,1],x⁺[:,2], label="+1")
scatter!(x⁻[:,1],x⁻[:,2], label="-1")
#plot!([0,θ[1]],[0,θ[2]])
plot!([-θ₀ₙ[1],θₙ[1]-θ₀ₙ[1]],[-θ₀ₙ[2],θₙ[2]-θ₀ₙ[2]], arrow=1., label = "theta" )

set = [([3,3],1),([4,2],1),([2,7],-1),([3,-2],-1)]

function perceptronOrigin(set, T=30)
    println("*** New Lookup ***")
    n  = length(set); d = length(set[1][1])
    θ  = zeros(d)
    ϵt = 0
    for t in 1:T
        ϵ = 0
        for i in 1:n
            x = set[i][1]; y = set[i][2]
            error = false
            θpre = θ
            if (y*(θ' * x) <= 0)
                θ  = θ + y * x
                ϵ += 1; error = true; ϵt += 1
            end
            println("θpre: $θpre x: $x y: $y error: $error θpost=$θ")
        end
        if (ϵ == 0)
            return (θ,ϵ,t, ϵt)
        end
    end
    return (θ,ϵ,T,ϵt)
end

set = [([-1,-1],1),([1,0],-1),([-1,1.5],1)]
perceptronOrigin(set)

set = [([1,0],-1),([-1,1.5],1),([-1,-1],1)]
perceptronOrigin(set)

set = [([-1,-1],1),([1,0],-1),([-1,10],1)]
perceptronOrigin(set)

set = [([1,0],-1),([-1,10],1),([-1,-1],1)]
perceptronOrigin(set)

function perceptronFull(set; T=30, θ = zeros(length(set[1])), θ₀ = 0)
    println("*** New Classifier Lookup ***")
    println("Data: $set")
    println("Parameters: T = $T, θ=$θ, θ₀=$θ₀")
    n = length(set); d = length(set[1][1])
    for t in 1:T
        ϵ = 0
        for i in 1:n
            x = set[i][1]; y = set[i][2]
            error = false
            θpre = θ
            θ₀pre = θ₀
            if (y*(θ' * x + θ₀) <= 0)
                θ  = θ + y * x
                θ₀ = θ₀ + y
                ϵ += 1
                error = true
            end
            println("θpre: $θpre θ₀pre: $θ₀pre x: $x y: $y error: $error θpost=$θ θ₀post=$θ₀")
        end
        if (ϵ == 0)
            return (θ,θ₀,ϵ,T)
        end
    end
    return (θ,θ₀,ϵ,T)
end

set = [([-4,2],1),([-2,1],1),([-1,-1],-1),([2,2],-1),([1,-2],-1)]
(θ,θ₀,ϵ,T) = perceptronFull(set)

## Homework 1

### Tab 2

scatter([-1,1],[1,-1],label="positive")
scatter!([1,2],[1,2],label="negative")

## Homework 2

### Tab 1

λ = 0.5
x = [1,0]
y = 1

θ =
loss(θ,y=1,x=[1,0],λ=0.5) = max(0,1-y*(θ' * x)) + λ/2 * norm(θ)^2

loss([0,0])
loss([1,0])
loss([0.5,0])
loss([-1,0])
loss([1,-1])

max(0,1-1*([1,0]' * [1,0]))
max(0,1-1*([0.5,0]' * [1,0]))

norm()

### Tab 3

cos(0)
cos(π)
cos(2*π)
cos(3*π)
cos(4*π)

function setFactory(d,labels=ones(d))
    out = Tuple{Array{Int64,1},eltype(labels)}[] # or just out=[]
    #out = []
    for i in 1:d
      x = zeros(d)
      x[i] = cos(i*π)
      push!(out,(x,labels[i]))
    end
    return out
end

set = setFactory(20)
using Random
shuffle!(set)
(θ,ϵ,t,ϵt) = perceptronOrigin(set,400)

randLabel = rand(20)
set = setFactory(20,randLabel)
(θ,ϵ,t,ϵt) = perceptronOrigin(set,400)
shuffle!(set)
(θ,ϵ,t,ϵt) = perceptronOrigin(set,400)

randLabel = rand(3)
set = setFactory(3,randLabel)
(θ,ϵ,t,ϵt) = perceptronOrigin(set,400)

set = setFactory(3)
(θ,ϵ,t,ϵt) = perceptronOrigin(set,400)

set=setFactory(2)
(θ,ϵ,t,ϵt) = perceptronOrigin(set,400)

Unit 01 - Linear Classifiers and Generalizations/Unit 01 - Linear Classifiers and Generalizations.jl


function _nancov(x::AbstractVector, y::AbstractVector, corrected::Bool, μᵪ::Number, μᵧ::Number)
    # Calculate covariance
    σᵪᵧ = ∅ = zero(promote_type(typeof(μᵪ), typeof(μᵧ), Int))
    n = 0
    @turbo for i ∈ indices((x,y))
            δᵪ = x[i] - μᵪ
            δᵧ = y[i] - μᵧ
            δ² = δᵪ * δᵧ
            notnan = δ²==δ²
            n += notnan
            σᵪᵧ += ifelse(notnan, δ², ∅)
    end
    σᵪᵧ = σᵪᵧ / (n-corrected)
    return σᵪᵧ
end

"""
```julia
nancov(x::AbstractVector, y::AbstractVector; corrected::Bool=true)
```
Compute the covariance between the vectors `x` and `y`.
As `Statistics.cov`, but ignoring `NaN`s.

If `corrected` is `true` as is the default, _Bessel's correction_ will be applied,
such that the sum is scaled by `n-1` rather than `n`, where `n = length(x)`.
"""
function nancov(x::AbstractVector, y::AbstractVector; corrected::Bool=true)
    # Check lengths
    nᵪ = length(x)
    nᵧ = length(y)
    @assert nᵪ == nᵧ

    μᵪ = _nanmean(x,:)
    μᵧ = _nanmean(y,:)
    σᵪᵧ = _nancov(x, y, corrected, μᵪ, μᵧ)
    return σᵪᵧ
end

"""
```julia
nancov(X::AbstractMatrix; dims::Int=1, corrected::Bool=true)
```
Compute the covariance matrix of the matrix `X`, along dimension `dims`.
As `Statistics.cov`, but ignoring `NaN`s.

If `corrected` is `true` as is the default, _Bessel's correction_ will be applied,
such that the sum is scaled by `n-1` rather than `n`, where `n = length(x)`.
"""
function nancov(X::AbstractMatrix; dims::Int=1, corrected::Bool=true)
    Tₒ = Base.promote_op(/, eltype(X), Int)
    n = size(X, dims)
    m = size(X, mod(dims,2)+1)
    Σ = similar(X, Tₒ, (m, m))
    # Only two dimensions are possible, so handle each manually
    if dims == 1
        # Precalculate means for each column
        μ = ntuple(m) do d
            nanmean(view(X,:,d))
        end
        # Fill covariance matrix symmetrically
        @inbounds for i = 1:m
            for j = 1:i
                σᵢⱼ = _nancov(view(X,:,i), view(X,:,j), corrected, μ[i], μ[j])
                Σ[i,j] = Σ[j,i] = σᵢⱼ
            end
        end
    elseif dims == 2
        # Precalculate means for each row
        μ = ntuple(m) do d
            nanmean(view(X,d,:))
        end
        # Fill covariance matrix symmetrically
        @inbounds for i = 1:m
            for j = 1:i
                σᵢⱼ = _nancov(view(X,i,:), view(X,j,:), corrected, μ[i], μ[j])
                Σ[i,j] = Σ[j,i] = σᵢⱼ
            end
        end
    else
        throw("Dimension not in range")
    end
    return Σ
end
export nancov

"""
```julia
nancor(x::AbstractVector, y::AbstractVector)
```
Compute the (Pearson's product-moment) correlation between the vectors `x` and `y`.
As `Statistics.cor`, but ignoring `NaN`s.

Equivalent to `nancov(x,y) / (nanstd(x) * nanstd(y))`.
"""
function nancor(x::AbstractVector, y::AbstractVector; corrected::Bool=true)
    # Check lengths
    nᵪ = length(x)
    nᵧ = length(y)
    @assert nᵪ == nᵧ

    μᵪ = nanmean(x)
    μᵧ = nanmean(y)
    σᵪ = nanstd(x, mean=μᵪ, corrected=corrected)
    σᵧ = nanstd(y, mean=μᵧ, corrected=corrected)
    σᵪᵧ = _nancov(x, y, corrected, μᵪ, μᵧ)
    ρᵪᵧ = σᵪᵧ / (σᵪ * σᵧ)

    return ρᵪᵧ
end

"""
```julia
nancor(X::AbstractMatrix; dims::Int=1)
```
Compute the (Pearson's product-moment) correlation matrix of the matrix `X`,
along dimension `dims`. As `Statistics.cor`, but ignoring `NaN`s.
"""
function nancor(X::AbstractMatrix; dims::Int=1, corrected::Bool=true)
    Tₒ = Base.promote_op(/, eltype(X), Int)
    n = size(X, dims)
    m = size(X, mod(dims,2)+1)
    Ρ = similar(X, Tₒ, (m, m))
    # Diagonal must be unity
    @inbounds for i = 1:m
        Ρ[i,i] = one(Tₒ)
    end
    # Only two dimensions are possible, so handle each manually
    if dims == 1
        # Precalculate means and standard deviations
        μ = ntuple(m) do d
            nanmean(view(X,:,d))
        end
        σ = ntuple(m) do d
            nanstd(view(X,:,d), mean=μ[d], corrected=corrected)
        end
        # Fill off-diagonals symmetrically
        @inbounds for i = 1:m
            for j = 1:i
                σᵢⱼ = _nancov(view(X,:,i), view(X,:,j), corrected, μ[i], μ[j])
                Ρ[i,j] = Ρ[j,i] = σᵢⱼ / (σ[i] * σ[j])
            end
        end
    elseif dims == 2
        # Precalculate means and standard deviations
        μ = ntuple(m) do d
            nanmean(view(X,d,:))
        end
        σ = ntuple(m) do d
            nanstd(view(X,d,:), mean=μ[d], corrected=corrected)
        end
        @inbounds for i = 1:m
            for j = 1:i-1
                σᵢⱼ = _nancov(view(X,i,:), view(X,j,:), corrected, μ[i], μ[j])
                Ρ[i,j] = Ρ[j,i] = σᵢⱼ / (σ[i] * σ[j])
            end
        end
    else
        throw("Dimension not in range")
    end
    return Ρ
end
export nancor


using Optim
using GR
using Plots

gr()

f(x) =  (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2

start = [-5.0,4.0]
iter = 80

GD_res = optimize(f, start, GradientDescent(),
               Optim.Options(g_tol = 1e-40,
                             iterations = iter,
                             store_trace = true,
                             extended_trace = true,
                             show_trace = false))

CGD_res = optimize(f, start, ConjugateGradient(),
               Optim.Options(g_tol = 1e-40,
                             iterations = iter,
                             store_trace = true,
                             extended_trace = true,
                             show_trace = false))

SA_res = optimize(f, start, SimulatedAnnealing(),
               Optim.Options(g_tol = 1e-40,
                             iterations = iter,
                             store_trace = true,
                             extended_trace = true,
                             show_trace = false))

l = iter + 1

GD_trace = Array{Float64}(undef, l, 2)
CGD_trace = Array{Float64}(undef, l, 2)
SA_trace = Array{Float64}(undef, l, 2)

for i in 1:l
    x1 = Optim.x_trace(GD_res)[i][1]
    x2 = Optim.x_trace(GD_res)[i][2]
    GD_trace[i,1] = x1
    GD_trace[i,2] = x2
end

for i in 1:l
    x1 = Optim.x_trace(CGD_res)[i][1]
    x2 = Optim.x_trace(CGD_res)[i][2]
    CGD_trace[i,1] = x1
    CGD_trace[i,2] = x2
end

for i in 1:l
    x1 = Optim.x_trace(SA_res)[i][1]
    x2 = Optim.x_trace(SA_res)[i][2]
    SA_trace[i,1] = x1
    SA_trace[i,2] = x2
end

contour(-5:0.005:5,-5:0.005:5,f)

plot!([GD_trace[:,1]],
    [GD_trace[:,2]],
    lab = "Gradient Descent",
    zcolor = reverse(GD_trace, dims = 1), m = (4, 0.9, :reds, Plots.stroke(0)),
    cbar = false,
    w = 1.5,
    linecolor = :red)

plot!([CGD_trace[:,1]],
    [CGD_trace[:,2]],
    lab = "Conjugated GD",
    zcolor = reverse(CGD_trace, dims = 1), m = (4, 0.9, :greens, Plots.stroke(0)),
    cbar = false,
    w = 1.5,
    linecolor = :green)

plot!([SA_trace[:,1]],
    [SA_trace[:,2]],
    lab = "Simulated Annealing",
    zcolor = reverse(SA_trace, dims = 1), m = (4, 0.9, :blues, Plots.stroke(0)),
    cbar = false,
    w = 1.5,
    linecolor = :blue)


# indicator of nonnegative orthant

"""
  IndNonnegative()

Returns the indicator function the nonnegative orthant, that is

  `g(x) = 0 if x ⩾ 0, +∞ otherwise`
"""

immutable IndNonnegative <: IndicatorConvexCone end

function (f::IndNonnegative){R <: Real}(x::AbstractArray{R})
  for k in eachindex(x)
    if x[k] < 0
      return +Inf
    end
  end
  return 0.0
end

function prox!{R <: Real}(f::IndNonnegative, x::AbstractArray{R}, y::AbstractArray{R}, gamma::Real=1.0)
  for k in eachindex(x)
    if x[k] < 0
      y[k] = zero(R)
    else
      y[k] = x[k]
    end
  end
  return 0.0
end

fun_name(f::IndNonnegative) = "indicator of the Nonnegative cone"
fun_dom(f::IndNonnegative) = "AbstractArray{Real}"
fun_expr(f::IndNonnegative) = "x ↦ 0 if all(0 ⩽ x), +∞ otherwise"
fun_params(f::IndNonnegative) = "none"

function prox_naive{R <: Real}(f::IndNonnegative, x::AbstractArray{R}, gamma::Real=1.0)
  y = max.(zero(R), x)
  return y, 0.0
end


# Lotka-Volterra, np = 2
id = :Sahlodin2011c

pL = [2.98; 0.98]
pU = [3.02; 1.02]
Sahlodin2011c_x0(p::Vector{T}) where T = T[1.2*one(T); 1.1*one(T)]
function Sahlodin2011c_f!(du::Vector{T}, u::Vector{T}, p::Vector{T}, t) where T
    du[1] = p[1]*u[1]*(1.0 - u[2])
    du[2] = p[2]*u[2]*(u[1] - 1.0)
    return
end
tspan = (0.0, 2.0)

prob = ODERelaxProb(Sahlodin2011c_f!, tspan, Sahlodin2011c_x0, pL, pU)
url = "https://www.sciencedirect.com/science/article/abs/pii/S0168927411000316"
source = "Sahlodin2011"
desc = "Lotka-Volterra, np = 2"


#   Unit examples of the FourierAnalysis Package for julia language
#   v 0.0.1 - last update 24th of September 2019
#
#   MIT License
#   Copyright (c) 2019, Marco Congedo, CNRS, Grenobe, France:
#   https://sites.google.com/site/marcocongedo/home

# ? CONTENTS :
#   This example show hot to create and plot tapering window for FFT.

#   ~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~  #
#                                                                             #
#   ~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~~¤~  #

############ plot tapering windows #############
using FourierAnalysis, Plots

# using the standard plot function
tapers=[TaperKind(i) for i=1:8]
X=zeros(t, 8)
for i=1:8 X[:, i] = taper(tapers[i], t).y end
labels=[string(tapers[i]) for i=1:8]
plot(X; labels=labels)

# discrete prolate spheroid sequences
H=taper(slepian, t, α=4, n=7)
# This is a special plot function declared in the `plots.jl unit`
plot(H, title="Slepian multi-tapering (dpss)")

################################################


"""
    NeuralNetworkKernel()

Neural network kernel function.

```math
    κ(x, y) =  asin(x' * y / sqrt[(1 + x' * x) * (1 + y' * y)])
```
# Significance
Neal (1996) pursued the limits of large models, and showed that a Bayesian neural network
becomes a Gaussian process with a **neural network kernel** as the number of units
approaches infinity. Here, we give the neural network kernel for single hidden layer
Bayesian neural network with erf (Error Function) as activation function.

# References:
- [GPML Pg 105](http://www.gaussianprocess.org/gpml/chapters/RW4.pdf)
- [Neal(1996)](https://www.cs.toronto.edu/~radford/bnn.book.html)
- [Andrew Gordon's Thesis Pg 45](http://www.cs.cmu.edu/~andrewgw/andrewgwthesis.pdf)
"""
struct NeuralNetworkKernel <: BaseKernel end

function (κ::NeuralNetworkKernel)(x, y)
    return asin(dot(x, y) / sqrt((1 + sum(abs2, x)) * (1 + sum(abs2, y))))
end

Base.show(io::IO, κ::NeuralNetworkKernel) = print(io, "Neural Network Kernel")


import Base: isequal, ==, show, *, length, >
using SymPy

"A plane geometric object."
abstract type GeoObject end
abstract type GeoShape <: GeoObject end

show(io::IO, objs::Vector{T}) where T <: GeoObject = print(io, "[", join(objs, ", "), "]")

"""
    Point(x, y)

A point at coordinate `(x, y)`.

# Examples
```julia
A = Point(0,0); B = Point(1, 3); C = Point(4,2)
```
"""
struct Point <: GeoObject
    "x cooridnate"
    x::Sym
    "y cooridnate"
    y::Sym

    Point(x, y) = new(Sym(x), Sym(y))
end

"""
# Function

    Point()

A point at coordinate `(0, 0)`.
"""
Point() = Point(0, 0)

"Check if two points are at the same coordinate."
(==)(p1::Point, p2::Point) = p1.x==p2.x && p1.y==p2.y

show(io::IO, pt::Point) = print(io, "Point($(pt.x), $(pt.y))")

"""
    Triangle(A, B, C)

A triangle with vertices `A, B, C`.

# Examples
```julia
A = Point(0,0); B = Point(1, 3); C = Point(4,2)
tri = Triangle(A, B, C)
```
"""
struct Triangle <: GeoShape
    "Vertex A"
    A::Point
    "Vertex B"
    B::Point
    "Vertex C"
    C::Point
end
show(io::IO, tri::Triangle) = print(io, "Triangle($(tri.A), $(tri.B), $(tri.C))")

"""
# Function

    Triangle()

A triangle with vertices `(0, 0), (0, 1), (1,1)`.
"""
Triangle() = Triangle(Point(0, 0), Point(0, 1), Point(1,1))

"""
    Triangle(ax, ay, bx, by, cx, cy)

A triangle with vertices `(ax, ay), (bx, by), (cx, cy)`.

# Examples
```julia
tri = Triangle(0, 0, 1, 3, 4, 2)
```
"""
Triangle(ax, ay, bx, by, cx, cy) = Triangle(Point(ax, ay), Point(bx, by), Point(cx, cy))

"Check if two triangles are the same."
(==)(t1::Triangle, t2::Triangle) = vertices(t1) == vertices(t2)

"Get the list of vertices of the triangle `tri`."
function vertices(tri::Triangle)
    [tri.A, tri.B, tri.C]
end

"""
    Edge(src, dst)

An edge from point `src` to point `dst`.

# Examples
```julia
e = Edge(Point(0, 0), Point(1, 3))
```
"""
struct Edge <: GeoShape
    "Starting point of an edge."
    src::Point
    "Ending point of an edge."
    dst::Point
end

"Check if two edges are at the same."
(==)(e1::Edge, e2::Edge) = e1.src==e2.src && e1.dst==e2.dst

"Get the list of edges of the triangle `tri`."
function edges(tri::Triangle)
    elist = Edge[]
    pts = vertices(tri)
    for i in 1:length(pts)-1
        push!(elist, Edge(pts[i], pts[i+1]))
    end
    push!(elist, Edge(pts[length(pts)], pts[1]))
    elist
end

"Mulitpliy the lengths of two edges."
(*)(e1::Edge, e2::Edge) = length(e1) * length(e2)

"Mulitpliy a number and an edges length."
(*)(s1::Sym, e2::Edge) = s1 * length(e2)

"Compare lengths of two edges."
(>)(e1::Edge, e2::Edge) = length(e1) > length(e2)

"Find the length of an edge."
length(e::Edge) = distance(e.src, e.dst)

"""
    Circle(c, r)

A circle with center at `c` and radius `r`.

# Examples
```julia
circle = Circle(Point(0, 0), 1)
```
"""
struct Circle <: GeoShape
    center::Point
    radius::Number
end
show(io::IO, c::Circle) = print(io, "Circle($(c.center), $(c.radius))")

"Check if two circles are the same."
(==)(c1::Circle, c2::Circle) = c1.center == c2.center && c1.radius == c2.radius

"Get the center of a a circle."
center(c::Circle) = c.center


# Functions related to sterographic projection to/from a three-dimensional
# unit sphere centered at (0,0) = 0+0im

# function stereo(z::Number)::Vector{Float64}
#     if isinf(z)
#         return [0, 0, 1]
#     end

#     X, Y = reim(z)

#     x = 2X
#     y = 2Y
#     z = X^2 + Y^2 - 1

#     d = 1 / (1 + X^2 + Y^2)

#     return d * [x, y, z]
# end

"""
    stereo
Stereographic projection between the complex plane and a three-dimensional unit sphere
centered at `[0,0,0]`. The north pole, `[0,0,1]`, corresponds to complex infinity and the
south pole, `[0,0,-1]`, corresponds to `0+0im`.

For a complex number `z`, `stereo(z)` maps `z` to the sphere. This may also be invoked as
`stereo(x,y)`.

For a (unit) three-dimensional vector `v`, `stereo(v)` returns the complex number by projecting
`v` to the complex plane. This may also be invoked as `stereo(x,y,z)`.
"""
function stereo(X::Real, Y::Real)::Vector{Float64}
    if isinf(X) || isinf(Y)
        return [0, 0, 1]
    end

    x = 2X
    y = 2Y
    z = X^2 + Y^2 - 1

    d = 1 / (1 + X^2 + Y^2)

    return d * [x, y, z]
end

stereo(z::Number) = stereo(reim(z)...)

function stereo(x::Real, y::Real, z::Real)::Complex
    if z == 1
        return Inf + im * Inf
    end
    X = x / (1 - z)
    Y = y / (1 - z)
    return X + im * Y
end

function stereo(v::Vector{T})::Complex where {T<:Real}
    stereo(v...)
end

"""
    LFTQ
Create a linear fractional transformation from a 3-by-3 unitary matrix.

Given a 3-by-3 real matrix `Q` with `Q*Q'` equal to the identity and `det(Q)` equal to `1`,
create a `LFT` that maps a complex number `v` to
`stereo(Q*stereo(v))`.
"""
function LFTQ(Q::AbstractMatrix)::LFT
    Q = Matrix(Q)
    zz = [0 + 0im, 1 + 0im, 0 + im]

    uu = stereo.(zz)
    uu = [Q * u for u in uu]

    ww = [stereo(u) for u in uu]

    LFT(zz[1], ww[1], zz[2], ww[2], zz[3], ww[3])

end

export stereo, LFTQ


@add_problem sdp_and_exp function sdp_and_exp_log_det_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
    x = Variable(2, 2)
    p = maximize(logdet(x), [x[1, 1] == 1, x[2, 2] == 1])
    if test
        @test vexity(p) == ConvexVexity()
    end
    handle_problem!(p)
    if test
        @test p.optval ≈ 0 atol=10atol rtol=rtol
        @test evaluate(logdet(x)) ≈ 0 atol=10atol rtol=rtol
    end
end


# Simulate n random numbers generated as follows: half of the random
# numbers are drawn from the standard univariate normal, and half of
# the random numbers are drawn from a univariate normal with zero mean
# and a standard deviation of 3. The return value is a vector of
# floats of length n.
function normdatasim(n::Int)

  # Check input "n".
  if n <= 0
    throw(ArgumentError("Argument \"n\" should be positive"))
  end

  # Generate the random numbers.
  n1 = round(Int,0.5*n);
  n2 = round(Int,0.2*n);
  n3 = n - n1 - n2;
  return vcat(randn(n1),4*randn(n2),6*randn(n3))
end

# Simulate n random numbers generated as follows: 50% of the random
# numbers are drawn from the standard univariate normal; 20% are
# drawn from a t distribution with 4 degrees of freedom; and the
# remaining 30% are drawn from a t distribution with 6 degrees of
# freedom. The return value is a vector of floats of length n.
function normtmixdatasim(n::Int)

  # Check input "n".
  if n <= 0
    throw(ArgumentError("Argument \"n\" should be positive"))
  end

  # Generate the random numbers.
  n1 = round(Int,0.5*n);
  n2 = round(Int,0.2*n);
  n3 = n - n1 - n2;
  return vcat(randn(n1),rand(TDist(4),n2),rand(TDist(6),n3))
end


"""
    Return an offset vector `p`, where p[i] is the prior
    probability of any hypothesis such that sum_{j=1}^m γj = i,
    according to Womack (2015)'s proposal
"""
function ph0(m, ζ)
    p = OffsetArray([zeros(m); 1.0], 0:m)
    for l = m-1:-1:0
        for j = 1:m-l
            p[l] += ζ * p[l + j] * binomial(l + j, l)
        end
    end
    p /= sum(p)
    for l = 0:m-1
        p[l] /= binomial(m, l)
    end
    return p
end

function γcode(γ)
    code = 1
    @inbounds for j = 2:length(γ)
        code += γ[j] * 2^(j - 2)
    end
    code
end

function γvector(J, code)
    γ = zeros(Int, J)
    code -= 1
    for j = J:-1:2
        if (code ÷ 2^(j-2)) > 0
            code -= 2^(j-2)
            γ[j] = 1
        end
    end
    return γ
end

"Return the vector γ associated with code `code`, given `J`"
γvec(J, code) = γvector(J, code)[2:J]

"Return the vector γ associated with code `code`, given `J`, as a string"
γstr(J, code) = string(γvec(J, code)...)


using DifferentialEquations
using SimpleDiffEq
using DiffEqCallbacks
using Random
using Distributions
using Plots

function sir_ode!(du,u,p,t)
    (S,I,R,C) = u
    (β,c,γ) = p
    N = S+I+R
    infection = β*c*I/N*S
    recovery = γ*I
    @inbounds begin
        du[1] = -infection
        du[2] = infection - recovery
        du[3] = recovery
        du[4] = infection
    end
    nothing
end;

tmax = 40.0
δt = 1.0
tspan = (0.0,tmax)
obstimes = 1.0:δt:tmax;
u0 = [990.0,10.0,0.0,0.0]; # S,I.R,C
p = [0.05,10.0,0.25]; # β,c,γ

prob_ode = ODEProblem(sir_ode!,u0,tspan,p)
sol_ode_cumulative = solve(prob_ode,Tsit5(),saveat=δt);

out = Array(sol_ode_cumulative)
C = out[4,:];

X = C[2:end] .- C[1:(end-1)];

Random.seed!(1234);

Y = rand.(Poisson.(X));

bar(obstimes,Y)
plot!(obstimes,X)

S = out[1,:]
Cpred = 990.0 .- S
Cdiff = Cpred .- C
plot(obstimes,Cdiff[2:end])

affect!(integrator) = integrator.u[4] = 0.0
cb_zero = PresetTimeCallback(obstimes,affect!);

sol_ode_cb = solve(prob_ode,Tsit5(),saveat=δt,callback=cb_zero);

X_cb = sol_ode_cb(obstimes)[4,:];

Random.seed!(1234);

Y_cb = rand.(Poisson.(X_cb));

X_diff_cb = X_cb .- X
plot(obstimes,X_diff_cb)

Y_diff_cb = Y_cb .- Y
plot(obstimes,Y_diff_cb)

function sir_dde!(du,u,h,p,t)
    (S,I,R,C) = u
    (β,c,γ) = p
    N = S+I+R
    infection = β*c*I/N*S
    recovery = γ*I
    e = oneunit(t)
    history = h(p, t-e)*inv(e)
    @inbounds begin
        du[1] = -infection
        du[2] = infection - recovery
        du[3] = recovery
        du[4] = infection - history[4]
    end
    nothing
end;

function sir_history(p, t; idxs = 5)
    zero(t)
end;

prob_dde = DDEProblem(DDEFunction(sir_dde!),
        u0,
        sir_history,
        tspan,
        p;
        constant_lags = [1.0]);

sol_dde = solve(prob_dde,MethodOfSteps(Tsit5()));

X_dde = sol_dde(obstimes)[4,:];

Random.seed!(1234)
Y_dde = rand.(Poisson.(X_dde));

X_diff_dde = X_dde .- X
plot(X_diff_dde)

Y_diff_dde = Y_dde .- Y
plot(obstimes, Y_diff_dde)