### A Pluto.jl notebook ###
# v0.12.11

using Markdown
using InteractiveUtils

# ╔═╡ 8dbcf370-1dec-11eb-0a2d-e967ec1f2fde
md" ### Nomenclatura"

# ╔═╡ c4a8db60-1dec-11eb-0c39-73a8aac79092
md"Seja $X$ uma matriz com dimensões $n x m$ com $n$ linhas e $m$ colunas que contém nossos exemplos. $x_i^{(j)}$ corresponde ao elemento de $X$ em que $i$ é uma linha da matriz (exemplo) e $j$ uma coluna da matriz (atributo), cujo acesso na linguagem representamos por x[i,j]. Para pegarmos um vetor linha desta matriz (exemplo) representamos formalmente por $X_i$ e na linguagem por x[i,:]."

# ╔═╡ cd532e00-1dec-11eb-06d9-9b7c76e9c1ba
md"Seja $\theta$ um vetor linha com $m$ elementos tal que $m$ corresponde ao número de atributos mais um. $\theta = [\theta^{(0)},\theta^{(1)}, ...,\theta^{(m)}]$ em que cada elemento corresponde ao j-ésimo parâmetro associado ao j-ésimo atributo. Na nossa linguagem acessamos um elemento por $\theta[i]$ e o vetor por $\theta$"

# ╔═╡ dc365e10-1dec-11eb-0e4f-95624ee468ea
md"Seja $y$ nosso vetor alvo em que representamos o i-ésimo elemento por $y_i$. Na nossa linguagem representamos por $y[i]$."

# ╔═╡ 571a5ea0-1dee-11eb-0eaf-bf746a6faa86
md"### Algumas libs importantes usadas neste experimento"

# ╔═╡ 317bbe60-1ded-11eb-2faa-c3e47c0df176
begin
   import Statistics: mean,std
   import BenchmarkTools: @benchmark, @btime
   import StaticArrays: @SMatrix
   import Plots: plot,plot!,scatter,scatter!
   import StatsBase: zscore
   μ = mean
end

# ╔═╡ f9af3a70-1dec-11eb-34f0-91b7fc591b87
md" ### Hipótese e modelo linear"

# ╔═╡ 78d2c0e0-1f98-11eb-06bc-b51c3b4b9f35
md"A seguir descrevemos a hipótese que consiste da aplicação de um modelo (conjunto de parâmetros) em um exemplo. Note que a hipótese sempre se refere à aplicação de um único exemplo e não ao conjunto de dados todo. "

# ╔═╡ 2e7709e0-1ded-11eb-2abd-d770bf1afe8a
md"$H_{\theta}(X_i) = \sum_{j}^{m}{x_{i}^{(j)} \theta^{(j)}} = X_i\theta^{T}$"

# ╔═╡ b05f9930-1f97-11eb-0083-07ebd21313e0
md"Exemplificando o produto de vetores, nossa hipótese faria o seguinte cálculo para um exemplo $X_i = \begin{bmatrix} 1 & 500 \end{bmatrix}$ e um vetor de parâmetros $\theta^T = \begin{bmatrix}
    0.5 \\
    1.0
\end{bmatrix}$:"

# ╔═╡ cbf998e0-1f96-11eb-0a8b-a3bf631bfe02
md"e.g. $H_{\theta}(X_i) = \begin{bmatrix}
    1 & 500
  \end{bmatrix}\begin{bmatrix}
    0.5 \\
    1.0
\end{bmatrix} = 500.5$"

# ╔═╡ 4cc76c80-1f98-11eb-0c70-29aec0b92ac4
md"Note que o resultado é um escalar, pois estamos estimando um valor contínuo (problema de regressão)."

# ╔═╡ c0b094a0-1f98-11eb-2efb-07f680447b12
md"### Implementação a hipótese"

# ╔═╡ be0deb50-1ded-11eb-06bb-33e94217e815
H(χ,θ) = χ*Θ'

# ╔═╡ 2f5a6500-1ded-11eb-11dd-bd88bf102b98
md" ### Função de custo"

# ╔═╡ 30972ac0-1ded-11eb-1007-1d3833dfc264

md"$J(\theta)= \frac{1}{2n}\sum_{i=1}^{n}{(H_{\theta}(X_i) - y_i)²}$"

# ╔═╡ bba6d590-1e1d-11eb-2df6-67bd6c5d4777
md" A seguir estou calculando a função de custo sem loop, apenas por matrizes.Desta maneira, geramos em uma multiplicação de matrizes, um vetor de predições com a mesma dimensão da saída. Após isto, subtraímos elemento a elemento. Após isto, elevamos cada elemento ao quadrado e depois tiramos uma média. Explicado na aula anterior."

# ╔═╡ 9837a3d0-1ded-11eb-3c49-17a45387e46e
J(Θ,χ,γ) = .5μ((χ*Θ' .- γ).^2)

# ╔═╡ 2ebaad22-1dee-11eb-2fb1-f51c49ec13fb
md"### Gradiente descendente"

# ╔═╡ d1cf31b0-1e17-11eb-22d1-916c9a331700
md"A seguir, mostramos o cálculo do gradiente. Como podemos notar, o expoente 2 desce e é eliminado pela constante 2 do denominador. Adicionalemente, vemos que é uma subtração do gradiente, pois estamos minimizando a função de custo. Por isto, estamos caminhando no sentido contrário ao do gradiente. É importante também ressaltar que o viés já está embutido na matriz $X$ na forma de uma coluna com valores 1. Desta forma, o cálculo do gradiente é o mesmo para o parâmetro do viés ($θ^{(0)}$) e os demais."

# ╔═╡ cbf654c0-1dfa-11eb-0994-85a67ded79b0
md"$θ^{(j)} = θ^{(j)} - α \frac{1}{n} \sum_{i=1}^{n}{(H_θ(X_i) - y_i)x_{i}^{(j)}}$"

# ╔═╡ 46484b80-1e1d-11eb-2bdc-e50b24f6b530
md"Entretanto, podemos condensar esta fórmula usando notação de vetores e matrizes. Lembrando que $\theta$ é um vetor agora e os cálculos são feitos simultaneamente para cada eleemnto deste. Podemos observar que o índice da coluna some, pois são processadas automaticamente. A seguir, veja a fórmula:"

# ╔═╡ f1089b10-1e1d-11eb-120e-cdbd993d72dc
md"$θ = θ - α \frac{1}{n} \sum_{i=1}^{n}{(H_θ(X_i) - y_i) \circ X_{i}}$"

# ╔═╡ 79bc38d0-1e1f-11eb-066c-2929340c479e
md"Cabe observar que $\circ$ é o produto de hadamard."

# ╔═╡ f5d33090-1e1f-11eb-3f25-d10f2616803d
md"### Algoritmo "

# ╔═╡ 7a22eab0-1def-11eb-2672-495fb9cf19f8
function train(θ,χ,γ;verbose=false,α = 1e-3,ϵ=1e-5,τ=1e3)

	e₂  = J(θ,χ,γ)
	e₁  = e₂ + .1
	#Δ₁,Δ₂ = e₁ - e₂ + 2ϵ,e₁ - e₂
	💻 = [[e₂ θ]]
	i   = 1
	while e₂ < e₁ && e₂ > ϵ && i < τ
	#while Δ₁ > Δ₂  && e₂ > ϵ && i < τ

		∇     = μ((χ*θ' .- γ).* χ,dims=1)
		θ    -= α .* ∇

		e₁,e₂ = e₂,J(θ,χ,γ)
		#Δ₁,Δ₂ = Δ₂,e₁ - e₂
		verbose && if (i%10==0) append!(💻,[[e₂ θ]]) end

		i += 1
	end
	θ,vcat(💻...)
end

# ╔═╡ 1f0e51f0-1df4-11eb-0feb-b53552fa63c9
md"### Testes de unitários"

# ╔═╡ 58dd7720-1e09-11eb-1904-af5f9991ece5
md" Sempre que for desenvolver suas funções, realize uma bateria de testes para cada uma antes mesmo de executar todo o seu programa. Isso vai lhe poupar muito tempo para uma futura depuração de código. Assim, para cada função, faça pelo menos um teste. Elabore casos bons de entrada para suas funções de forma que cubra bem o espaço de possibilidades. Seus casos devem ser bem simples para que num futuro, se der erro, você consiga rapidamente resolver."

# ╔═╡ 259ac040-1e39-11eb-3670-f5c5519a1978
md"### Padronizando os Dados "

# ╔═╡ 246cc792-1e39-11eb-3c20-f1623d8ea737
md"O objetivo de padronizar os dados é acelerar a convergência do método. Aqui utilizaremos a padronização zscore. Entretanto, para amostras pequenas, a padronização não funciona muito bem."

# ╔═╡ c467ca00-1e08-11eb-3d41-75a38532652b
md"### Verificando a curva de erro"

# ╔═╡ d1a88a00-1f8f-11eb-36b7-f39dabee0657
md"### Veificando o comportamento dos parâmetros"

# ╔═╡ b405694e-1f8f-11eb-0fad-5d0c3527d206
md"### Verificando o modelo"

# ╔═╡ d497051e-1f94-11eb-2ccb-a7d3e52570d1
md"Como podemos observar, o gradiente jogou o θ₁ para zero (viés, responsável pela translação) e ajustou o θ₂ (responsável pela rotação). Cabe ressaltar que estamos trabalhando com os dados padronizados. É possível, no entanto, despadronizar e obter predições no campo dos números originais."

# ╔═╡ 37c52e40-1e48-11eb-1e13-7ba1bebfabc4
md"### Depuração "

# ╔═╡ 44997b82-1e48-11eb-1821-f7f3aecd8093
md" Costumo colocar alguma variáveis soltas para que eu veja se está tudo correto. "

# ╔═╡ a419b220-1e4b-11eb-1299-295bf39df3f4
md"### Solução direta por álgebra linear"

# ╔═╡ a39cb9f0-1e4b-11eb-296e-bf19e5438416
train(X,Y) = inv(X'X)X'Y

# ╔═╡ 628624c0-1dfa-11eb-2afb-137bf77669e1
begin
	Θ1 = [0. 4.]
	χ =  [1 50.;1 60.;1 100.; 1 200.]
	γ =  [200  ;   240  ;   400; 800]
	@assert J(Θ1,χ,γ) == 0.0
	@assert train(Θ1,χ,γ)[1] == [0. 4.] # iniciando com a solução, o θ deve continuar igual
end

# ╔═╡ 4c632fee-1e39-11eb-00c6-c3f2a0774005
begin
	χₚ = hcat(ones((4,1)), zscore(χ[:,2:end])) # padronizamos só o que não é viés
	γₚ = zscore(γ)
end

# ╔═╡ d8ab5330-1e47-11eb-06b3-af3d2ee97c7b
χₚ

# ╔═╡ 86be2920-1e48-11eb-10f1-6740a2459951
γₚ

# ╔═╡ 74604cb2-1e05-11eb-19ef-09fd04dc52e3
(θᵏ,r) = train([4 5],χₚ,γₚ,verbose=true,α = 1e-3,ϵ=1e-5,τ=1e5);

# ╔═╡ b7913600-1e0c-11eb-362e-0b4b9ef9d1aa
plot(1:length(r[:,1]),r[:,1],xlabel="iterações",ylabel="J(θ)",title="Análise de erros",label="Erro")

# ╔═╡ fa0bd460-1e23-11eb-131f-f31bec373c07
begin
	p = plot(1:length(r[:,2]),r[:,2],xlabel="iterações",ylabel="θ₁,θ₂",title="Análise de parâmetros",label="θ₁")
	plot!(p,1:length(r[:,3]),r[:,3],xlabel="iterações",label="θ₂")
end

# ╔═╡ cc73d620-1f8f-11eb-357d-b19b50c19280
begin
	f(x) = θᵏ[1] + θᵏ[2]*x # o mesmo que H(X) lá do início
	plot(χₚ[:,2],f,xlabel="m² (padronizado)",ylabel="Preço (padronizado)",title="Análise do da predição do modelo",label="θ₁ + θ₂x")
	scatter!(χₚ[:,2],γₚ,label="Dados Coletados")
end

# ╔═╡ 41505b10-1e34-11eb-06db-750eb3ab11cf
r[end,:]

# ╔═╡ 29b04780-1e44-11eb-3bcf-8b5f121d157f
J(θᵏ,χₚ,γₚ)

# ╔═╡ d0de29e0-1e36-11eb-398c-5f0b55ea281e
θᵏ

# ╔═╡ 70775a70-1e4c-11eb-3cec-11fd54e9b637
train(χₚ,γₚ)

# ╔═╡ Cell order:
# ╟─8dbcf370-1dec-11eb-0a2d-e967ec1f2fde
# ╟─c4a8db60-1dec-11eb-0c39-73a8aac79092
# ╟─cd532e00-1dec-11eb-06d9-9b7c76e9c1ba
# ╟─dc365e10-1dec-11eb-0e4f-95624ee468ea
# ╟─571a5ea0-1dee-11eb-0eaf-bf746a6faa86
# ╠═317bbe60-1ded-11eb-2faa-c3e47c0df176
# ╟─f9af3a70-1dec-11eb-34f0-91b7fc591b87
# ╟─78d2c0e0-1f98-11eb-06bc-b51c3b4b9f35
# ╟─2e7709e0-1ded-11eb-2abd-d770bf1afe8a
# ╟─b05f9930-1f97-11eb-0083-07ebd21313e0
# ╟─cbf998e0-1f96-11eb-0a8b-a3bf631bfe02
# ╟─4cc76c80-1f98-11eb-0c70-29aec0b92ac4
# ╟─c0b094a0-1f98-11eb-2efb-07f680447b12
# ╠═be0deb50-1ded-11eb-06bb-33e94217e815
# ╟─2f5a6500-1ded-11eb-11dd-bd88bf102b98
# ╟─30972ac0-1ded-11eb-1007-1d3833dfc264
# ╟─bba6d590-1e1d-11eb-2df6-67bd6c5d4777
# ╠═9837a3d0-1ded-11eb-3c49-17a45387e46e
# ╟─2ebaad22-1dee-11eb-2fb1-f51c49ec13fb
# ╟─d1cf31b0-1e17-11eb-22d1-916c9a331700
# ╟─cbf654c0-1dfa-11eb-0994-85a67ded79b0
# ╟─46484b80-1e1d-11eb-2bdc-e50b24f6b530
# ╟─f1089b10-1e1d-11eb-120e-cdbd993d72dc
# ╟─79bc38d0-1e1f-11eb-066c-2929340c479e
# ╟─f5d33090-1e1f-11eb-3f25-d10f2616803d
# ╠═7a22eab0-1def-11eb-2672-495fb9cf19f8
# ╟─1f0e51f0-1df4-11eb-0feb-b53552fa63c9
# ╟─58dd7720-1e09-11eb-1904-af5f9991ece5
# ╠═628624c0-1dfa-11eb-2afb-137bf77669e1
# ╟─259ac040-1e39-11eb-3670-f5c5519a1978
# ╟─246cc792-1e39-11eb-3c20-f1623d8ea737
# ╠═4c632fee-1e39-11eb-00c6-c3f2a0774005
# ╟─c467ca00-1e08-11eb-3d41-75a38532652b
# ╠═74604cb2-1e05-11eb-19ef-09fd04dc52e3
# ╟─b7913600-1e0c-11eb-362e-0b4b9ef9d1aa
# ╟─d1a88a00-1f8f-11eb-36b7-f39dabee0657
# ╠═fa0bd460-1e23-11eb-131f-f31bec373c07
# ╟─b405694e-1f8f-11eb-0fad-5d0c3527d206
# ╟─d497051e-1f94-11eb-2ccb-a7d3e52570d1
# ╠═cc73d620-1f8f-11eb-357d-b19b50c19280
# ╟─37c52e40-1e48-11eb-1e13-7ba1bebfabc4
# ╟─44997b82-1e48-11eb-1821-f7f3aecd8093
# ╠═41505b10-1e34-11eb-06db-750eb3ab11cf
# ╠═29b04780-1e44-11eb-3bcf-8b5f121d157f
# ╠═d0de29e0-1e36-11eb-398c-5f0b55ea281e
# ╠═d8ab5330-1e47-11eb-06b3-af3d2ee97c7b
# ╠═86be2920-1e48-11eb-10f1-6740a2459951
# ╟─a419b220-1e4b-11eb-1299-295bf39df3f4
# ╠═a39cb9f0-1e4b-11eb-296e-bf19e5438416
# ╠═70775a70-1e4c-11eb-3cec-11fd54e9b637


function arnoldi(a::AbstractArray{T,2},b::AbstractArray{T,1},m::S = length(b)) where {T <: AbstractFloat, S <: Integer} # Exact

    n = length(b)
    if m > n
        error("'m' must be no larger than the length of 'b' ")
    end

    q = zeros(n,m)
    h = zeros(m+1,m)

    β = norm(b)
    qk = copy(b)

    @views for k = 1:m
        q[:,k] .= qk/β
        qk .= a*q[:,k]
        for j = 1:k
            h[j,k] = qk'q[:,j]
            qk .-= h[j,k]*q[:,j]
        end
        β = norm(qk)
        h[k+1,k] = β
        if β < eps(T)
            return q[:,1:k], h[1:k,1:k], k
        end
    end

    return q, h[1:m,1:m], m

end

function arnoldi(f::Function,x::Array{T,1},r::Array{T,1},m::S=length(x)) where {T <: AbstractFloat, S <: Integer} #Exact

    n = length(x)
    if m > n
        error("'m' must be no larger than the length of 'b' ")
    end

    q = zeros(n,m)
    h = zeros(m+1,m)

    β = norm(r)
    qk = copy(r)

    @views for k = 1:m
        q[:,k] = qk/β
        jacvec!(qk,f,x,q[:,k])
        for j = 1:k
            h[j,k] = qk'q[:,j]
            qk -= h[j,k]*q[:,j]
        end
        β = norm(qk)
        h[k+1,k] = β
        if β < eps(T)
            return q[:,1:k], h[1:k,1:k], k
        end
    end

    return q, h[1:m,:], m

end

function arnoldi(a::AbstractArray{T,2},b::AbstractArray{T,1},tol::T,m::S=length(b)) where {T <: AbstractFloat, S <: Integer} #Inexact

    n = length(b)
    if m > n
        error("'m' must be no larger than the length of 'b' ")
    end

    q = zeros(n,m)
    h = zeros(m+1,m)

    β = norm(b)
    qk = copy(b)

    @views for k = 1:m
        q[:,k] .= qk/β
        qk .= a*q[:,k]
        for j = 1:k
            h[j,k] = qk'q[:,j]
            qk .-= h[j,k]*q[:,j]
        end
        β = norm(qk)
        h[k+1,k] = β
        if β < tol
            return q[1:n,1:k], h[1:k,1:k], k
        end
    end

    return q[1:n,1:m], h[1:m,1:m], m

end

function arnoldi(f::Function,x::Array{T,1},r::Array{T,1},tol::T,m::S=length(x)) where {T <: AbstractFloat, S <: Integer} # Inexact

    n = length(x)
    if m > n
        error("'m' must be no larger than the length of 'b' ")
    end

    q = zeros(n,m)
    h = zeros(m+1,m)

    β = norm(r)
    qk = copy(r)

    @views for k = 1:m
        q[:,k] = qk/β
        jacvec!(qk,f,x,q[:,k])
        for j = 1:k
            h[j,k] = qk'q[:,j]
            qk -= h[j,k]*q[:,j]
        end
        β = norm(qk)
        h[k+1,k] = β
        if β < tol
            return q[1:n,1:k], h[1:k,1:k], k
        end
    end

    return q[1:n,1:m], h[1:m,1:m], m

end

function arnoldi_inplace(f::Function,x::Array{T,1},r::Array{T,1},tol::T,m::S=length(x)) where {T <: AbstractFloat, S <: Integer} # Inexact

    n = length(x)
    if m > n
        error("'m' must be no larger than the length of 'b' ")
    end

    q = zeros(n,m)
    h = zeros(m+1,m)

    β = norm(r)
    qk = copy(r)

    @views for k = 1:m
        q[:,k] = qk/β
        jacvec_inplace!(qk,f,x,q[:,k])
        for j = 1:k
            h[j,k] = qk'q[:,j]
            qk -= h[j,k]*q[:,j]
        end
        β = norm(qk)
        h[k+1,k] = β
        if β < tol
            return q[1:n,1:k], h[1:k,1:k], k
        end
    end

    return q[1:n,1:m], h[1:m,1:m], m

end


using StatisticalRethinking

n_samples = 10000
a3d = ones(n_samples,1,1)
w = 6; l = 3; n = w +l
p = [0.5]
for i in 2:n_samples
  p_new = rand(Normal(p[i-1], 0.1), 1)[1]
  if  p_new < 0
    p_new = abs(p_new)
  end
  if p_new > 1
    p_new = 2 - p_new
  end
  q0 = pdf(Binomial(n, p[i-1]), w)
  q1 = pdf(Binomial(n, p_new), w)
  append!(p, [rand(Uniform(0, 1), 1)[1] < q1/q0 ? p_new : p[i-1]])
end

a3d[:, 1, 1] = p
chns = MCMCChains.Chains(a3d, ["toss"])

MCMCChains.describe(chns)

plot(chns)

w = 6; n = 9; x = 0:0.01:1
density(chns, lab="Samples")
plot!( x, pdf.(Beta( w+1 , n-w+1 ) , x ), lab="Conjugate solution")

# This file was generated using Literate.jl, https://github.com/fredrikekre/Literate.jl


# ---
# title: 888. Fair Candy Swap
# id: problem888
# author: Indigo
# date: 2021-02-01
# difficulty: Easy
# categories: Array
# link: <https://leetcode.com/problems/fair-candy-swap/description/>
# hidden: true
# ---
#
# Alice and Bob have candy bars of different sizes: `A[i]` is the size of the
# `i`-th bar of candy that Alice has, and `B[j]` is the size of the `j`-th bar
# of candy that Bob has.
#
# Since they are friends, they would like to exchange one candy bar each so that
# after the exchange, they both have the same total amount of candy.  ( _The
# total amount of candy  a person has is the sum of the sizes of candy bars they
# have._)
#
# Return an integer array `ans` where `ans[0]` is the size of the candy bar that
# Alice must exchange, and `ans[1]` is the size of the candy bar that Bob must
# exchange.
#
# If there are multiple answers, you may return any one of them.  It is
# guaranteed an answer exists.
#
#
#
# **Example 1:**
#
#
#
#     Input: A = [1,1], B = [2,2]
#     Output: [1,2]
#
#
# **Example 2:**
#
#
#
#     Input: A = [1,2], B = [2,3]
#     Output: [1,2]
#
#
# **Example 3:**
#
#
#
#     Input: A = [2], B = [1,3]
#     Output: [2,3]
#
#
# **Example 4:**
#
#
#
#     Input: A = [1,2,5], B = [2,4]
#     Output: [5,4]
#
#
#
#
# **Note:**
#
#   * `1 <= A.length <= 10000`
#   * `1 <= B.length <= 10000`
#   * `1 <= A[i] <= 100000`
#   * `1 <= B[i] <= 100000`
#   * It is guaranteed that Alice and Bob have different total amounts of candy.
#   * It is guaranteed there exists an answer.
#
#
## @lc code=start
using LeetCode

function fair_candy_swap(A::Vector{Int}, B::Vector{Int})
    set_a, set_b = Set(A), Set(B)
    dif = (sum(A) - sum(B)) ÷ 2
    for num_a in set_a
        ((num_a - dif) in set_b) && return [num_a, num_a - dif]
    end
end
## @lc code=end


# Sum square difference

LIMIT = 100

function solve()
    return sum(1:LIMIT)^2 - sum(n^2 for n in 1:LIMIT)
end

print(Int(solve()))


"""
2-D Gradient

    gradient2(a::Array, dx::Float64=1, dy::Float=1)
    :param a: matrix
    :param dx: stepsize
    :param dy: stepsize

    :return dxdu: derivatives along x
    :return dydv: derivatives along y
"""
function gradient2(a::Array, dx::Float64=1., dy::Float64=1.)
    m,n = size(a);
    dxdu = zeros(m,n);
    dydv = zeros(m,n);

    for i=1:m
        dxdu[i,:] = gradient(vec(a[i,:]), dx)
    end

    for i=1:n
        dydv[:,i] = gradient(a[:,i], dy)
    end

    return dxdu, dydv
end

"""
Creates Rectangular Grid in 2-D space

    meshgrid(a::LinSpace,b::LinSpace)
"""
function meshgrid(a::LinSpace,b::LinSpace)
    grid_a = [i for i in a, j in b]';
    grid_b = [j for i in a, j in b]';

    return grid_a, grid_b
end

"""
Linear interpolation

    approx(xd, yd, xi)
    :param xd: x samples
    :param yd: response samples
    :param xi: new x samples
"""
function approx(xd, yd, xi)
    nd = length(xd);
    ni = length(xi);

    yi = zeros(ni);
    for i in 1:ni
        if (xi[i] <= xd[1])
            t = (xi[i]-xd[1]) / (xd[2] - xd[1]);
            yi[i] = (1.0 - t) * yd[1] + t * yd[2];
        elseif (xd[nd] <= xi[i])
            t = (xi[i] - xd[nd-1]) / (xd[nd] - xd[nd-1]);
            yi[i] = (1.0 - t) * yd[nd-1] + t * yd[nd];
        else
            for k in 2:nd
                if (xd[k-1] <= xi[i] && xi[i] <= xd[k])
                    t = (xi[i] - xd[k-1]) / (xd[k] - xd[k-1]);
                    yi[i] = (1.0 - t) * yd[k-1] + t * yd[k];
                    break
                end
            end
        end
    end

    return yi
end

"""
Trapezoidal Integration

    trapz(x, y, dim=1)
    :param x: vector of time samples
    :param y: array of response samples
    :param dim: dimension along which to integrate
"""
function trapz(x::Array{Float64, 1}, y::Array{Float64}, dim::Integer=1)
    perm = [dim:max(ndims(y),dim); 1:dim-1];
    y = permutedims(y, perm);
    if ndims(y) == 1
        m = 1;
    else
        m = size(y,1);
    end

    if m == 1
        M = length(y);
        out = sum(diff(x).*(y[1:M-1] + y[2:M])/2.);
    else
        out = diff(x).' * (y[1:m-1,:] + y[2:m,:])/2.;
        siz = size(y); siz = collect(siz); siz[1] = 1;
        out = reshape(out, tuple(siz...));
        out = ipermutedims(out, perm);
        ind = find(collect(size(out)).==1);
        out = squeeze(out,ind[1]);
        if length(out) == 1;
            out = out[1];
        end
    end

    return out
end

"""
Cumulative Trapezoidal Integration

    cumtrapz(x, y, dim=1)
    :param x: vector describing time samples
    :param y: array describing response
    :param dim: dimension to integrate over
"""
function cumtrapz(x::Array{Float64, 1}, y::Array{Float64}, dim::Integer=1)
    perm = [dim:max(length(size(y)),dim); 1:dim-1];
    y = permutedims(y, perm);
    if ndims(y) == 1
        n = 1;
        m = length(y);
    else
        m, n = size(y);
    end

    if n == 1
        dt = diff(x)/2.0;
        z = [0; cumsum(dt.*(y[1:(m-1)] + y[2:m]))];
    else
        dt = repmat(diff(x)/2.0,1,n);
        z = [zeros(1,n); cumsum(dt.*(y[1:(m-1), :] + y[2:m, :]),1)];
        z = ipermutedims(z, perm);
    end

    return z
end

"""
Cumulative Trapezoidal Integration using midpoint

    cumtrapzmid(x, y, c)
    :param x: time samples
    :param y: resposne samples
    :param c: midpoint
"""
function cumtrapzmid(x, y, c)
    a = length(x);
    mid = round(Integer, a/2);

    # case < mid
    fn = zeros(a);
    tmpx = x[(mid-1):-1:1];
    tmpy = y[(mid-1):-1:1];
    tmp = c + cumtrapz(tmpx, tmpy);
    fn[1:(mid-1)] = reverse(tmp);

    # case >= mid
    fn[mid:a] = c + cumtrapz(x[mid:a],y[mid:a]);

    return fn

end

"""
Multivariate Normal random number generation

    mvnrand(mu, C, n)
    :param mu: mean vector
    :param C: covariance matrix
    :param n: number of samples
"""
function mvnrand(mu, C, n)
    tmp = cholfact(C, :U, Val{true});
    R = tmp[:U];
    R = Array(R);
    R = R[:, tmp.piv];
    retval = randn(n, size(R,1)) * R;
    retval += transpose(repmat(mu, 1, n));
    return transpose(retval)
end

"""
Linear interpolation when response contains flat regions

    interp1_flat(x, y, xx)
    :param x: time samples
    :param y: response samples
    :param xx: new time samples
"""
function interp1_flat(x,y,xx)
    flat = find(diff(x).<=0);
    n = length(flat);

    if n==0
        tmp = interpolate((x,), y, Gridded(Linear()))
        yy = tmp[xx];
    else
        yy = zeros(size(xx));
        i1 = 1;
        if flat[1] == 1
            i2 = 1;
            j = xx.==x[i2];
            yy[j] = minimum(y[i2:i2+1]);
        else
            i2 = flat[1];
            j = (xx.>=x[i1]) & (xx.<=x[i2]);
            tmp = interpolate((x[i1:i2],), y[i1:i2], Gridded(Linear()))
            yy[j] = tmp[xx[j]];
            i1 = copy(i2);
        end
        for k = 2:n
            i2 = flat[k];
            if i2 > i1+1
                j = (xx.>=x[i1]) & (xx.<=x[i2]);
                yi = interpolate((x[i1+1:i2],), y[i1+1:i2], Gridded(Linear()))
                yy[j] = tmp[xx[j]];
            end
            j = xx.==x[i2];
            yy[j] = minimum(y[i2:i2+1]);
            i1 = copy(i2);
        end
        i2 = length(x);
        j = (xx.>=x[i1]) & (xx.<=x[i2]);
        if i1+1 == i2
            yy[j] = y[i2];
        else
            tmp = interpolate((x[i1+1:i2],), y[i1+1:i2], Gridded(Linear()))
            yy[j] = tmp[xx[j]];
        end
    end

    return yy
end


ENV["GKSwstype"] = "100" #src

# # Generalized Linear Models
#
# Generalized Linear Models (GLMs) are a family of statistical models where the prediction
# is based on a function of a linear predictor $x_^T\beta$ where $x$ is a vector of features
# and $\beta$ is the parameter vector.
#
# One of the big assumptions of GLMs is that there is a strictly increasing *link function*
# $g$ that "links" the mean of the distribution $E(Y|X) = \mu$ to the linear predictor
# $x^T\beta$.  In other words, our prediction of $y$ given $x$ is $g^{-1}(x^T\beta)$.

# The two most common GLMs are linear and logistic regression.

# The GLM package
#
#

# ## Linear Regression

# ## Logistic Regression


"""
shoot(phi,xspan,lval,lder,rval,rder,init)

Use the shooting method to solve a two-point boundary value problem. The ODE is
u'' = `phi`(x,u,u') for x in `xspan`. Specify a function value or derivative at
the left endpoint using `lval` and `lder`, respectively, and similarly for the
right endpoint  using `rval` and `rder`. (Use an empty array to denote an
unknown quantity.) The value `init` is an initial guess for whichever value is
missing at the left endpoint.

Return vectors for the nodes, the values of u, and the values of u'.
    """
function shoot(phi,xspan,lval,lder,rval,rder,init)

    # Tolerances for IVP solver and rootfinder.
    ivp_opt = 1e-6
    optim_opt = 1e-5

    # Evaluate the difference between computed and target values at x=b.
    function objective(s)
        # Combine s with the known left endpoint value.
        v_init = isempty(lder) ?  [ lval; s[1] ]  :  [ s[1]; lder ]

        # ODE posed as a first-order equation in 2 variables.
        function shootivp(v,p,x)
            [ v[2]; phi(x,v[1],v[2]) ]
        end

        IVP = ODEProblem(shootivp,v_init,xspan)
        sol = solve(IVP,abstol=ivp_opt,reltol=ivp_opt)
        x = sol.t;  v = sol;

        return isempty(rder) ? v[1,end] - rval  :  v[2,end] - rder
    end

    # Find the unknown quantity at x=a by rootfinding.
    x = [];  v = [];   # the values will be overwritten
    s = nlsolve(objective,[init],xtol=optim_opt).zero

    # Don't need to solve the IVP again. It was done within the
    # objective function already.
    u = v[1,:]            # solution
    dudx = v[2,:]         # derivative

    return x,u,dudx
end

"""
    diffmat2(n,xspan)

Compute 2nd-order-accurate differentiation matrices on `n`+1 points in the
interval `xspan`. Return a vector of nodes, and the matrices for the first
and second derivatives.
"""
function diffmat2(n,xspan)
    a,b = xspan
    h = (b-a)/n
    x = [ a + i*h for i=0:n ]   # nodes

    # Define most of Dx by its diagonals.
    dp = fill(0.5/h,n)        # superdiagonal
    dm = fill(-0.5/h,n)       # subdiagonal
    Dx = diagm(-1=>dm,1=>dp)

    # Fix first and last rows.
    Dx[1,1:3] = [-1.5,2,-0.5]/h
    Dx[n+1,n-1:n+1] = [0.5,-2,1.5]/h

    # Define most of Dxx by its diagonals.
    d0 =  fill(-2/h^2,n+1)    # main diagonal
    dp =  ones(n)/h^2         # superdiagonal and subdiagonal
    Dxx = diagm(-1=>dp,0=>d0,1=>dp)

    # Fix first and last rows.
    Dxx[1,1:4] = [2,-5,4,-1]/h^2
    Dxx[n+1,n-2:n+1] = [-1,4,-5,2]/h^2

    return x,Dx,Dxx
end

"""
    diffcheb(n,xspan)

Compute Chebyshev differentiation matrices on `n`+1 points in the
interval `xspan`. Return a vector of nodes, and the matrices for the first
and second derivatives.
"""
function diffcheb(n,xspan)
    x = [ -cos( k*pi/n ) for k=0:n ]    # nodes in [-1,1]
    Dx = zeros(n+1,n+1)
    c = [2; ones(n-1); 2];    # endpoint factors

    # Off-diagonal entries
    Dx = [ (-1)^(i+j)*c[i+1]/(c[j+1]*(x[i+1]-x[j+1])) for i=0:n, j=0:n ]

    # Diagonal entries
    Dx[isinf.(Dx)] .= 0              # fix divisions by zero on diagonal
    s = sum(Dx,dims=2)
    Dx -= diagm(0=>s[:,1])           # "negative sum trick"

    # Transplant to [a,b]
    a,b = xspan
    x = @. a + (b-a)*(x+1)/2
    Dx = 2*Dx/(b-a)

    # Second derivative
    Dxx = Dx^2

    return x,Dx,Dxx
end

"""
     bvplin(p,q,r,xspan,lval,rval,n)

Use finite differences to solve a linear bopundary value problem. The ODE is
u''+`p`(x)u'+`q`(x)u = `r`(x) on the interval `xspan`, with endpoint function
values given as `lval` and `rval`. There will be `n`+1 equally spaced nodes,
including the endpoints.

Return vectors of the nodes and the solution values.
"""
function bvplin(p,q,r,xspan,lval,rval,n)
    x,Dx,Dxx = diffmat2(n,xspan)

    P = diagm(0=>p.(x))
    Q = diagm(0=>q.(x))
    L = Dxx + P*Dx + Q     # ODE expressed at the nodes

    # Replace first and last rows using boundary conditions.
    I = Diagonal(ones(n+1))
    A = [ I[[1],:]; L[2:n,:]; I[[n+1],:] ]
    b = [ lval; r.(x[2:n]); rval ]

    # Solve the system.
    u = A\b

    return x,u
end

"""
    bvp(phi,xspan,lval,lder,rval,rder,init)

Use finite differences to solve a two-point boundary value problem. The ODE is
u'' = `phi`(x,u,u') for x in `xspan`. Specify a function value or derivative at
the left endpoint using `lval` and `lder`, respectively, and similarly for the
right endpoint  using `rval` and `rder`. (Use an empty array to denote an
unknown quantity.) The value `init` is an initial guess for whichever value is
missing at the left endpoint.

Return vectors for the nodes and the values of u.
"""
function bvp(phi,xspan,lval,lder,rval,rder,init)
    n = length(init) - 1
    x,Dx,Dxx = diffmat2(n,xspan)
    h = x[2]-x[1]

    function residual(u)
        # Compute the difference between u'' and phi(x,u,u') at the
        # interior nodes and appends the error at the boundaries.
        dudx = Dx*u                   # discrete u'
        d2udx2 = Dxx*u                # discrete u''
        f = d2udx2 - phi.(x,u,dudx)

        # Replace first and last values by boundary conditions.
        f[1] = isempty(lder) ? (u[1] - lval)/h^2 : (dudx[1] - lder)/h
        f[n+1] = isempty(rder) ? (u[n+1] - rval)/h^2 : (dudx[n+1] - rder)/h
        return f
    end

    u = levenberg(residual,init)
    return x,u[:,end]
end

"""
    fem(c,s,f,a,b,n)

Use a piecewise linear finite element method to solve a two-point boundary
value problem. The ODE is (`c`(x)u')' + `s`(x)u = `f`(x) on the interval
[`a`,`b`], and the boundary values are zero. The discretization uses `n` equal
subintervals.

Return vectors for the nodes and the values of u.
"""
function fem(c,s,f,a,b,n)
    # Define the grid.
    h = (b-a)/n
    x = @. a + h*(0:n)

    # Templates for the subinterval matrix and vector contributions.
    Ke = [1 -1; -1 1]
    Me = (1/6)*[2 1; 1 2]
    fe = (1/2)*[1; 1]

    # Evaluate coefficent functions and find average values.
    cval = c.(x);   cbar = (cval[1:n]+cval[2:n+1]) / 2;
    sval = s.(x);   sbar = (sval[1:n]+sval[2:n+1]) / 2;
    fval = f.(x);   fbar = (fval[1:n]+fval[2:n+1]) / 2;

    # Assemble global system, one interval at a time.
    K = zeros(n-1,n-1);  M = zeros(n-1,n-1);  f = zeros(n-1);
    K[1,1] = cbar[1]/h;  M[1,1] = sbar[1]*h/3;  f[1] = fbar[1]*h/2;
    K[n-1,n-1] = cbar[n]/h;  M[n-1,n-1] = sbar[n]*h/3;  f[n-1] = fbar[n]*h/2;
    for k = 2:n-1
      K[k-1:k,k-1:k] += (cbar[k]/h) * Ke
      M[k-1:k,k-1:k] += (sbar[k]*h) * Me
      f[k-1:k] += (fbar[k]*h) * fe
    end

    # Solve system for the interior values.
    u = (K+M) \ f
    u = [0; u; 0]      # put the boundary values into the result

    return x,u
end


println(5 + π)

println(typeof(π)) # Echo the type of pi
println(typeof(100)) # Echo the type of 100
println(typeof(6.0000)) # Echo the type of 6.0000

println(Sys.WORD_SIZE) # This variable indicated whether the target system is 32-bit arch or 64-bit arch

# %% typeof
hexX = 0x1
println("typeof(hexX) ~~> ", typeof(hexX))

hexX = 0x123
println("typeof(hexX) ~~> ", typeof(hexX))

hexX = 0x123456789
println("typeof(hexX) ~~> ", typeof(hexX))

x = 0b10
println("typeof(x) ~~> ", typeof(x))

y = 0o010
println("typeof(y) ~~> ", typeof(y))

for T in [Int8, Int16, Int32, Int64, Int128, UInt8, UInt16, UInt32, UInt64, UInt128]
    println("* $(lpad(T,7)): [$(typemin(T)),$(typemax(T))]")
end
# %% End

# %% Overflow behavior
x = typemax(Int64)
println("x ~~> ", x)
x = x + 1
println("x ~~> ", x)
# x + 1 == typemax(Int64)
println(x + 1 == typemax(Int64))
# %% End

# %% Overflow resolution
println(10^19)
println(big(10)^19)
# %% End

# %% Float
myFloat = 1.777
println("typeof(myFLoat) ~~> ", typeof(myFloat))
println("typeof(Float32(myFloat)) ~~> ", typeof(Float32(myFloat)))
# %% End

# %% Positive and Negetive zero
println("0.0 == -0.0 ~~> ", 0.0 == -0.0)
println("bitstring(0.0) ~~> ", bitstring(0.0))
println("bitstring(-0.0) ~~> ", bitstring(-0.0))
# %% End

# %% Arbitrary Precision Arithmetic
println("BigInt(typemax(Int64)) + 1 ~~> ", BigInt(typemax(Int64)) + 1)
println((big"123456789012345678901234567890" + 1))
println(typeof(big"123456789012345678901234567890" + 1))
parse(BigInt, "123456789012345678901234567890")
println(string(big"2"^200, base = 16))
# %% End

# %% Numeric Literal Coefficients
x = 3
println("x ~~> ", x)
y = 2(x - 1)^2 - 3(x - 1) + 1
println("y ~~> ", y)
println("√16 ~~> ", √16)
# %% End


using Distributions, Random, Plots; pyplot()
Random.seed!(4)

function simulateMM1DoobGillespie(lambda,mu,Q0,T)
    t, Q = 0.0 , Q0
    tValues, qValues = [0.0], [Q0]
    while t<T
        if Q == 0
            t += rand(Exponential(1/lambda))
            Q = 1
        else
            t += rand(Exponential(1/(lambda+mu)))
            Q += 2(rand() < lambda/(lambda+mu)) -1
        end
        push!(tValues,t)
        push!(qValues,Q)
    end
    return [tValues, qValues]
end

function stichSteps(epochs,q)
    n = length(epochs)
    newEpochs  = [ epochs[1] ]
    newQ = [ q[1] ]
    for i in 2:n
        push!(newEpochs,epochs[i])
        push!(newQ,q[i-1])
        push!(newEpochs,epochs[i])
        push!(newQ,q[i])
    end
    return [newEpochs, newQ]
end

lambda, mu = 0.7, 1.0
Tplot, Testimation = 200, 10^7
Q0 = 20

eL,qL = simulateMM1DoobGillespie(lambda, mu ,Q0, Testimation)
meanQueueLength = (eL[2:end]-eL[1:end-1])'*qL[1:end-1]/last(eL)
rho = lambda/mu
println("Estimated mean queue length: ", meanQueueLength )
println("Theoretical mean queue length: ", rho/(1-rho) )

epochs, qValues = simulateMM1DoobGillespie(lambda, mu, Q0,Tplot)
epochsForPlot, qForPlot = stichSteps(epochs,qValues)
plot(epochsForPlot,qForPlot,
	c=:blue, xlims=(0,Tplot), ylims=(0,25), xlabel="Time",
	ylabel="Customers in queue", legend=:none)


## exponential.jl : implementation of exponential covariance function

## Exponential ##
struct Exponential{T} <: IsotropicCovarianceStructure{T}
    λ::T
    σ::T
    p::T

    function Exponential{T}(λ::T, σ::T, p::T) where T
        λ > 0 || throw(DomainError(λ, "correlation length λ of exponential covariance cannot be negative or zero"))
        σ > 0 || throw(DomainError(σ, "marginal standard deviation σ of exponential covariance cannot be negative or zero"))
        p >= 1 || throw(DomainError(p, "in p-norm, p must be greater than or equal to 1"))
        isinf(p) && throw(DomainError(p, "in p-norm, p cannot be infinity"))

        new{T}(λ, σ, p)
    end
end

"""
    Exponential(λ, [σ = 1], [p = 2])

Exponential covariance structure with correlation length `λ`, (optional) marginal standard deviation `σ` and (optional) `p`-norm defined as

``C(x, y) = σ \\exp\\left(-\\displaystyle\\frac{ρ}{λ}\\right)``

with ``ρ = ||x - y||_p``.

# Examples
```jldoctest
julia> Exponential(0.1)
exponential (λ=0.1, σ=1.0, p=2.0)

julia> Exponential(1.0, σ=2)
exponential (λ=1.0, σ=2.0, p=2.0)

```
See also: [`Linear`](@ref), [`Spherical`](@ref), [`Whittle`](@ref), [`Gaussian`](@ref), [`SquaredExponential`](@ref), [`Matern`](@ref)
"""
Exponential(λ::Real; σ::Real=1.0, p::Real=2) = Exponential{promote_type(typeof(λ),typeof(σ),typeof(p))}(promote(λ, σ, p)...)

# evaluate exponential covariance
apply(e::Exponential, x::Real) = exp(-x / e.λ)

# short name
shortname(::Exponential) = "exponential"


using Random, Distributions
Random.seed!(0)

lambda = 0.1
dist = Exponential(1/lambda)
actualMedian = median(dist)

M = 10^3
N = 10^4
nRange = 2:2:10
alpha = 0.05

for n in nRange
    coverageCount = 0
    for _ in 1:M
        sampleData = rand(dist, n)
        bootstrapSampleMeans = [median(rand(sampleData, n)) for _ in 1:N]
        L = quantile(bootstrapSampleMeans, alpha/2)
        U = quantile(bootstrapSampleMeans, 1-alpha/2)
        coverageCount += L < actualMedian && actualMedian < U
    end
    println("n = ",n,"\t coverage = ", coverageCount/M)
end


module FiniteDifferenceDerivatives

using ArrayViews

include("specialized.jl")

export fdd!, fdd, fddmatrix, fddat

# returns a sparse matrix D such that D*u is a der-derivative of specified order
function fddmatrix{T<:Number}(x::AbstractVector{T},der::Int,order::Int)
    npts = length(x)
    m = zeros(T,npts,npts)
    f = eye(T,npts)
    for i = 1:npts
        m[:,i]=fdd(f[:,i],x,der,order)
    end
    return sparse(m)
end

# compute the der-derivative using order-point scheme
function fdd!{T<:Number}(df::AbstractVector{T},f::AbstractVector{T},x::AbstractVector{T},der::Int,order::Int)
    npts = length(x)
    if order < der
        error("Order can not be smaller than der")
    elseif npts < order
        error("Number of mesh points shouldn't be smaller than order")
    end

    # specialized implementation for lower orders
    if der == 1 && order == 3
        fdd13!(df,f,x)
    elseif der == 2 && order == 3
        fdd23!(df,f,x)
    elseif der == 1
        fdd1n!(df,f,x,order)
    else
        # proceed with general implementation

        c = Array(T,order,der+1)

        for N = 1:npts
            # N1 is the leftmost point of the stencil
            N1    = min(max(1,N-div(order-1,2)),npts-order+1)
            df[N] = fddcore(f,x,der,x[N],N1,order,c)
        end
    end

end

function fdd{T<:Number}(f::AbstractVector{T},x::AbstractVector{T},der::Int,order::Int)
    df = Array(T,length(f))
    fdd!(df,f,x,der,order)
    return df
end

function fddcore{T<:Number}(f::AbstractVector{T},
                            x::AbstractVector{T},
                            der::Int,
                            x0::Real,
                            N1::Int,
                            order::Int,
                            c::Matrix{T})

    generatec!(c,x0,x,N1)

    dfN = zero(T)
    @simd for j = 1:order
        @inbounds dfN += c[j,der+1]*f[N1-1+j]
    end
    return dfN
end

function fddat{T<:Number}(f::AbstractVector{T},
                          x::AbstractVector{T},
                          der::Int,
                          x0::Real)

    if length(f) != length(x)
        error("x and f must have the same size")
    end

    N1 = 1
    order = length(x)
    c = Array(T,order,der+1)
    return fddcore(f,x,der,x0,N1,order,c)
end

# generate the coefficients c
function generatec!{T}(c::Matrix{T},x0::Real,x::AbstractVector{T},N1::Int)
    order = size(c,1)
    der   = size(c,2)-1

    if N1 < 1 || N1+order-1 > length(x)
        error("N1=$N1 and order=$order are out of bounds")
    end

    c1 = one(T)
    @inbounds c4 = x[N1] - x0
    c[:] = zero(T)
    c[1] = one(T)
    for i=1:order-1
        mn = min(i,der)
        c2 = one(T)
        c5 = c4
        @inbounds c4 = x[i+N1] - x0
        j = 0
        while j <= i-1
            @inbounds c3 = x[i+N1] - x[j+N1]
            c2 = c2*c3
            if j == i-1
                s = mn
                while s >= 1
                    @inbounds c[i+1,s+1] = c1*(s*c[i,s] - c5*c[i,s+1])/c2
                    s-=1
                end
                @inbounds c[i+1,1] = -c1*c5*c[i,1]/c2
            end
            s = mn
            while s >= 1
                @inbounds c[j+1,s+1] = (c4*c[j+1,s+1] - s*c[j+1,s])/c3
                s-=1
            end
            @inbounds c[j+1,1] = c4*c[j+1,1]/c3
            j+=1
        end
        c1 = c2
    end
end

end # module


"""
`solve(P::Problem, O::Options=DefaultOptions)::Tuple{Vector, Vector}`

Solve the problem `P` using options `O`.

See `Problem` and `Options` for more info about how to state a problem and change the
default options.
"""
function solve(P::Problem, O::Options=DefaultOptions)::Tuple{Vector, Vector}

    # Unpack the data structures
    @unpack α, β, γ, δ, p, q, a, b, A, y₀, z₀ = P
    @unpack n = O

    # Compute the step
    h = (b - a) / n

    # Prepare the result arrays
    y = Vector{Float64}(undef, n+1); z = copy(y)

    # Put the boundary values first
    y[1] = y₀
    z[1] = z₀

    # Compute the rest
    for i in 1:n
        x = a + i * h
        yᵢ₊₁(x, zᵢ₊₁) = (y[i] + h * (β(A) * zᵢ₊₁ + p(x, zᵢ₊₁))) / (1 - h * α(A))
        z[i+1] = find_zero(
            (zᵢ₊₁) -> -zᵢ₊₁ + z[i] + h * (γ(A) * yᵢ₊₁(x, zᵢ₊₁) + δ(A) * zᵢ₊₁ + q(x, zᵢ₊₁)),
            z[i],
        )
        y[i+1] = yᵢ₊₁(x, z[i+1])
    end

    return y, z

end


struct Point2D{T<:Real}
    x::T
    y::T
end

struct Point3D{T<:Real}
    x::T
    y::T
    z::T
end

"""
    euclidean2DRounded(i1::Int,i2::Int, map::Dict{Int, Point2D})

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xy-plane between the two points rounded to the nearest integer

# Arguments
- `i1::Int`: an index of a point in the map
- `i2::Int`: another index of a point in the map
- `map::Dict{Int, Point2D}`: the map of indexes to points
"""
function euclidean2DRounded(i1::Int,i2::Int, map::Dict{Int, Point2D})
    return euclidean2DRounded(map[i1],map[i2])
end

"""
    euclidean2DRounded(p1::Point2D,p2::Point2D)

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xy-plane between the two points rounded to the nearest integer

# Arguments
- `p1::Point2D`: a point to compare
- `p2::Point2D`: another point to compare
"""
function euclidean2DRounded(p1::Point2D,p2::Point2D)
    return round(sqrt(((p1.x - p2.x)^2)+((p1.y - p2.y)^2)))
end

"""
    euclidean2D(i1::Int,i2::Int, map::Dict{Int, Point2D})

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xy-plane between the two points

# Arguments
- `i1::Int`: an index of a point in the map
- `i2::Int`: another index of a point in the map
- `map::Dict{Int, Point2D}`: the map of indexes to points
"""
function euclidean2D(i1::Int,i2::Int, map::Dict{Int, Point2D})
    return euclidean2D(map[i1],map[i2])
end

"""
    euclidean2D(p1::Point2D,p2::Point2D)

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xy-plane between the two points

# Arguments
- `p1::Point2D`: a point to compare
- `p2::Point2D`: another point to compare
"""
function euclidean2D(p1::Point2D,p2::Point2D)
    return sqrt(((p1.x - p2.x)^2)+((p1.y - p2.y)^2))
end

"""
    euclidean3DRounded(i1::Int,i2::Int, map::Dict{Int, Point3D})

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xyz-plane between the two points rounded to the nearest integer

# Arguments
- `i1::Int`: an index of a point in the map
- `i2::Int`: another index of a point in the map
- `map::Dict{Int, Point3D}`: the map of indexes to points
"""
function euclidean3DRounded(i1::Int,i2::Int, map::Dict{Int, Point3D})
    return euclidean3DRounded(map[i1],map[i2])
end

"""
    euclidean3DRounded(p1::Point3D,p2::Point3D)

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xyz-plane between the two points rounded to the nearest integer

# Arguments
- `p1::Point3D`: a point to compare
- `p2::Point3D`: another point to compare
"""
function euclidean3DRounded(p1::Point3D,p2::Point3D)
    return round(sqrt(((p1.x - p2.x)^2)+((p1.y - p2.y)^2)+((p1.z - p2.z)^2)))
end

"""
    euclidean3D(i1::Int,i2::Int, map::Dict{Int, Point3D})

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xyz-plane between the two points

# Arguments
- `i1::Int`: an index of a point in the map
- `i2::Int`: another index of a point in the map
- `map::Dict{Int, Point3D}`: the map of indexes to points
"""
function euclidean3D(i1::Int,i2::Int, map::Dict{Int, Point3D})
    return euclidean3D(map[i1],map[i2])
end

"""
    euclidean3D(p1::Point3D,p2::Point3D)

Takes in a map of indexes to points and two indexes
Returns the euclidean distance on the xyz-plane between the two points

# Arguments
- `p1::Point3D`: a point to compare
- `p2::Point3D`: another point to compare
"""
function euclidean3D(p1::Point3D,p2::Point3D)
    return sqrt(((p1.x - p2.x)^2)+((p1.y - p2.y)^2)+((p1.z - p2.z)^2))
end


@testset "Norms" begin
    x = [2.0im, 3.0 - 1im, 5.0 + 2.0im]
    y = [-2.0im, 3.0 - 1im, 5.0 + 2.0im]
    Norm = HC.InfNorm()
    @test Norm isa HC.AbstractNorm
    @test Norm(x) ≈ abs(5.0 + 2.0im)
    @test Norm(x, x) ≈ 0.0
    @test Norm(x, y) ≈ abs(4im)

    WeightedNorm = HC.WeightedNorm(Norm, x)
    @test HC.weights(WeightedNorm) == ones(3)
    WeightedNorm .= 2.0
    @test HC.weights(WeightedNorm) == [2.0, 2, 2]
    WeightedNorm[1] = 4.0
    @test WeightedNorm isa HC.AbstractNorm
    @test WeightedNorm(x) ≈ 0.5 * abs(5 + 2.0im)
    @test WeightedNorm(x, x) ≈ 0.0
    @test WeightedNorm(x, y) ≈ 0.25 * abs(4im)

    Norm = HC.EuclideanNorm()
    @test Norm isa HC.AbstractNorm

    @test Norm(x) ≈ LinearAlgebra.norm(x, 2)
    @test Norm(x, x) ≈ 0.0
    @test Norm(x, y) ≈ abs(4im)

    WeightedNorm = HC.WeightedNorm([4.0, 2.0, 2.0], Norm)
    @test WeightedNorm isa HC.AbstractNorm
    @test WeightedNorm(x) ≈ sqrt(abs2(x[1]) / 16 + abs2(x[2]) / 4 + abs2(x[3]) / 4)
    @test WeightedNorm(x, x) ≈ 0.0
    @test WeightedNorm(x, y) ≈ sqrt(abs2(4im) / 16)
end


using DifferentialEquations
using SimpleDiffEq
using Distributions
using Random
using DataFrames
using StatsPlots
using BenchmarkTools

function sir_sde!(du,u,p,t)
    (S,I,R) = u
    (β,c,γ,δt) = p
    N = S+I+R
    ifrac = β*c*I/N*S*δt
    rfrac = γ*I*δt
    ifrac_noise = sqrt(ifrac)*rand(Normal(0,1))
    rfrac_noise = sqrt(rfrac)*rand(Normal(0,1))
    @inbounds begin
        du[1] = S-(ifrac+ifrac_noise)
        du[2] = I+(ifrac+ifrac_noise) - (rfrac + rfrac_noise)
        du[3] = R+(rfrac+rfrac_noise)
    end
    for i in 1:3
        if du[i] < 0 du[i]=0 end
    end
    nothing
end;

δt = 0.1
nsteps = 400
tmax = nsteps*δt
tspan = (0.0,nsteps)
t = 0.0:δt:tmax;

u0 = [990.0,10.0,0.0]; # S,I,R

p = [0.05,10.0,0.25,δt]; # β,c,γ,δt

Random.seed!(1234);

prob_sde = DiscreteProblem(sir_sde!,u0,tspan,p)

sol_sde = solve(prob_sde,solver=FunctionMap);

df_sde = DataFrame(sol_sde')
df_sde[!,:t] = t;

@df df_sde plot(:t,
    [:x1 :x2 :x3],
    label=["S" "I" "R"],
    xlabel="Time",
    ylabel="Number")

@benchmark solve(prob_sde,solver=FunctionMap)

include(joinpath(@__DIR__,"tutorials","appendix.jl"))
appendix()


using ML
using Base.Test

x,y = rand(10),rand(10)

k = GaussianKernel(σ=0.3)
@test k(x,y) ≈ exp(-norm(x-y)^2/(2*k.σ^2))

k = PolynomialKernel(α=10.0,c=1.5,ξ=2.5)
@test k(x,y) ≈ (k.α*dot(x,y) + k.c)^k.ξ

x_train = linspace(0,1,30)[:,:]'
y_train = sin(vec(x_train))
x_test = linspace(0,1,100)[:,:]'
y_test = sin(vec(x_test))

for k in ( GaussianKernel(σ=0.5),
		   PolynomialKernel(α=5.,c=0.1,ξ=4.),
		   ExponentialKernel(σ=10) )
    m = KRR(x_train,y_train,k=k,λ=1e-8)
    fit!(m)
    @test mae(m(x_test),y_test) < 1e-4
end


export prime_plane

"""
`prime_plane(n)` returns a finite projective plane of order `n` provided
`n` is prime.
"""
function prime_plane(p::Int)
    @assert isprime(p) "$p is not prime"
    alist = [ [a,b,1] for a=0:p-1 for b=0:p-1 ]
    blist = [ [a,1,0] for a=0:p-1 ]
    clist = [ [1,0,0] ]

    list = [alist; blist; clist ]

    n = length(list)

    A = zeros(Int,n,n)
    for i=1:n
        for j=1:n
            d = list[i]' * list[j]
            A[i,j] = mod(d[1],p) == 0
        end
    end
    return A
end


# Lets check normalization
using GeometryTypes
using Makie
using Colors
using JuliennedArrays

export spherical_plot, complex_spherical_plot

function sampleSphere(n::Int)
    map(normalize, Slices(randn(3,n),1))
end

function complexImage(vals::Array{Complex{Float64}})
    max = maximum(abs.(vals))
    map(x -> HSV(angle(x)*360/(2π),(abs(x)/max)^.5,1),vals)
end

function spherical_plot(fun::Function, n::Int)
    # n sets the resolution
    θ = [0;(0.5:n-0.5)/n;1]
    φ = [(0:2n-2)*2/(2n-1);2]
    x = [cospi(φ)*sinpi(θ) for θ in θ, φ in φ]
    y = [sinpi(φ)*sinpi(θ) for θ in θ, φ in φ]
    z = [cospi(θ) for θ in θ, φ in φ]
    vals = map(((x,y,z) -> fun( [x,y,z])), x,y,z);
    s = Makie.surface(x, y, z, color = complexImage(vals))
end

function comptosphere(z::Complex)
    # solve for the
    x = real(z)
    y = imag(z)
    t = 2/(1+abs(z)^2)
    [t*x,t*y,1-t]
end

function complex_spherical_plot(fun::Function, n::Int)
    # plots a function of the complex plane on the riemann sphere
    function spheretocomp(r::Array)
        # solve for the complex plane point
        x = r[1]
        y = r[2]
        t = 1-r[3]
        z = (x + im*y)/t
    end
    spherical_plot((x->fun(spheretocomp(x))), n)
end


# # Implement your own numerical methods to solve
#
# $$
# y'(t) = 1 - y(t),  t \in [0,5],  y(0) = 0.
# $$

# --

# ## Explicit Euler

euler(f, t, y, h) = t + h, y + h * f(t, y)

# ## Runge-Kutta 2nd order

rk2(f, t, y, h) = begin
    ỹ = y + h / 2 * f(t, y)
    t + h, y + h * f(t + h / 2, ỹ)
end

# ---

# ## Runge-Kutta 4th order

function rk4(f, t, y, dt)

    y₁ = dt * f(t, y)
    y₂ = dt * f(t + dt / 2, y + y₁ / 2)
    y₃ = dt * f(t + dt / 2, y + y₂ / 2)
    y₄ = dt * f(t + dt, y + y₃)

    t + dt, y + (y₁ + 2 * y₂ + 2 * y₃ + y₄) / 6

end

# ---

# ## Solve function

function dsolve(f, method, t₀, y₀, h, nsteps)

    t = zeros(Float64, nsteps)
    y = similar(t)

    t[1] = t₀
    y[1] = y₀

    for i = 2:nsteps
        t[i], y[i] = method(f, t[i-1], y[i-1], h)
    end

    t, y

end

# ---

# ## Plot solutions

using Plots

nsteps, tfinal = 7, 5.0
t₀, x₀ = 0.0, 0.0
dt = tfinal / (nsteps - 1)
f(t, x) = 1 - x

t, y_euler = dsolve(f, euler, t₀, x₀, dt, nsteps)

t, y_rk2 = dsolve(f, rk2, t₀, x₀, dt, nsteps)

t, y_rk4 = dsolve(f, rk4, t₀, x₀, dt, nsteps)

# ---

plot(t, y_euler; marker = :o, label = "Euler")
plot!(t, y_rk2; marker = :d, label = "RK2")
plot!(t, y_rk4; marker = :p, label = "RK4")
plot!(t -> 1 - exp(-t); line = 3, label = "true solution")
savefig("dsolve.png") #hide
# ![dsolve](dsolve.png)

# ---

# ## DifferentialEquations.jl

using DifferentialEquations

f(y, p, t) = 1.0 - y
y₀, t = 0.0, (0.0, 5.0)

prob = ODEProblem(f, y₀, t)

sol_euler = solve(prob, Euler(), dt = 1.0)
sol = solve(prob)

# ---

plot(sol_euler, label = "Euler")
plot!(sol, label = "default")
plot!(1:0.1:5, t -> 1.0 - exp(-t), lw = 3, ls = :dash, label = "True Solution!")

savefig("diffeq.png"); nothing #hide

# ![diffeq](diffeq.png)

# ---

# `sol.t` is the array of time points that the solution was saved at

sol.t

# `sol.u` is the array of solution values

sol.u

# ---

function lorenz(du, u, p, t)
    du[1] = 10.0 * (u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end

u0 = [1.0; 0.0; 0.0]
tspan = (0.0, 100.0)
prob = ODEProblem(lorenz, u0, tspan)

sol = solve(prob)

# ---

plot(sol, vars = (1, 2, 3))

savefig("lorenz.png"); nothing #hide

# ![lorenz](lorenz.png)

# ---

using ParameterizedFunctions

lotka_volterra = @ode_def begin
  d🐁 = α*🐁  - β*🐁*🐈
  d🐈 = -γ*🐈 + δ*🐁*🐈
end α β γ δ

u0 = [1.0, 1.0] # Initial condition

tspan = (0.0, 10.0) # Simulation interval
tsteps = 0.0:0.1:10.0 # intermediary points

p = [1.5, 1.0, 3.0, 1.0] # equation parameters: p = [α, β, δ, γ]

prob = ODEProblem(lotka_volterra, u0, tspan, p)
sol = solve(prob)

# ---

# # Type-Dispatch Programming
#
# - Centered around implementing the generic template of the algorithm not around building representations of data.
# - The data type choose how to efficiently implement the algorithm.
# - With this feature share and reuse code is very easy
#
# [JuliaCon 2019 | The Unreasonable Effectiveness of Multiple Dispatch | Stefan Karpinski](https://youtu.be/kc9HwsxE1OY)

# ---

# Simple gravity pendulum

using DifferentialEquations, Plots

g = 9.79 # Gravitational constants
L = 1.00 # Length of the pendulum

#Initial Conditions
u₀ = [0, π / 60] # Initial speed and initial angle
tspan = (0.0, 6.3) # time domain

#Define the problem
function simplependulum(du, u, p, t)
    θ = u[1]
    dθ = u[2]
    du[1] = dθ
    du[2] = -(g/L)*θ
end

prob = ODEProblem(simplependulum, u₀, tspan)
sol = solve(prob, Tsit5(), reltol = 1e-6)

# ---

# Analytic and computed solution
u = u₀[2] .* cos.(sqrt(g / L) .* sol.t)

scatter(sol.t, getindex.(sol.u, 2), label = "Numerical")
plot!(sol.t, u, label = "Analytic")
savefig("pendulum1.svg"); nothing # hide

# ![](pendulum1.svg)

# ---

# [Numbers with Uncertainties](http://tutorials.juliadiffeq.org/html/type_handling/02-uncertainties.html)

using Measurements

g = 9.79 ± 0.02; # Gravitational constants
L = 1.00 ± 0.01; # Length of the pendulum

#Initial Conditions
u₀ = [0 ± 0, π / 60 ± 0.01] # Initial speed and initial angle

#Define the problem
function simplependulum(du, u, p, t)
    θ = u[1]
    dθ = u[2]
    du[1] = dθ
    du[2] = -(g/L)*θ
end

#Pass to solvers
prob = ODEProblem(simplependulum, u₀, tspan)
sol = solve(prob, Tsit5(), reltol = 1e-6);
nothing # hide

#md # ---

# Analytic solution
u = u₀[2] .* cos.(sqrt(g / L) .* sol.t)

plot(sol.t, getindex.(sol.u, 2), label = "Numerical")
plot!(sol.t, u, label = "Analytic")
savefig("pendulum2.svg"); nothing # hide

# ![](pendulum2.svg)

# ---


### A Pluto.jl notebook ###
# v0.16.1

using Markdown
using InteractiveUtils

# This Pluto notebook uses @bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of @bind gives bound variables a default value (instead of an error).
macro bind(def, element)
    quote
        local el = $(esc(element))
        global $(esc(def)) = Core.applicable(Base.get, el) ? Base.get(el) : missing
        el
    end
end

# ╔═╡ 94f8e29e-ef91-11eb-1ae9-29bc46fa505a
begin
	using Pkg
    Pkg.activate(joinpath(Pkg.devdir(), "MLCourse"))
	using Plots, DataFrames, Random, CSV, MLJ, MLJLinearModels
end

# ╔═╡ 12d5824c-0873-49a8-a5d8-f93c73b633ae
using PlutoUI; PlutoUI.TableOfContents()

# ╔═╡ 20c5c7bc-664f-4c04-8215-8f3a9a2095c9
begin
    using MLCourse
    MLCourse.list_notebooks(@__FILE__)
end

# ╔═╡ 8217895b-b120-4b08-b18f-d921dfdddf10
md"# Linear Regression

## Wind speed prediction with one predictor
"

# ╔═╡ 9f84bcc5-e5ab-4935-9076-c19e9bd668e8
weather = CSV.read(joinpath(@__DIR__, "..", "data", "weather2015-2018.csv"), DataFrame);

# ╔═╡ 34e527f2-ef80-4cb6-be3a-bee055eca125
begin
    training_set1 = (X = (LUZ_pressure = weather.LUZ_pressure[1:end-5],),
                     y = weather.LUZ_wind_peak[6:end])
    m1 = machine(LinearRegressor(), training_set1.X, training_set1.y) |> fit!
end;

# ╔═╡ 006fc1eb-50d5-4206-8c87-53e873f158f4
begin
    scatter(training_set1.X.LUZ_pressure[1:10:end],
            training_set1.y[1:10:end], label = "data")
    plot!(fitted_linear_func(m1), label = "linear fit", w = 3)
end

# ╔═╡ e4712ebe-f395-418b-abcc-e10ada4b05c2
md"Let us inspect the results.
First we look at the fitted parameters.
We find that there is a negative correlation between the pressure in Luzern
and the wind speed 5 hours later in Luzern.
"

# ╔═╡ 8c9ae8f7-81b2-4d60-a8eb-ded5364fe0cc
fitted_params(m1)

# ╔═╡ f4f890b6-0ad4-4155-9321-15d673e15489
md"Next we predict the mean for different pressure values the distribution of
    wind speeds. In the probabilistic interpretation of supervised learning,
    standard linear regression finds conditional normal distributions with
    input-dependent mean ``\mu = \hat y = \theta_0 + \theta_1 x`` and
    constant standard deviation σ.
"

# ╔═╡ 9b62c374-c26e-4990-8ffc-790928e62e88
predict(m1, (LUZ_pressure = [930., 960., 990.],))

# ╔═╡ 7923a0a8-3033-4dde-91e8-22bf540c9866
md"We use the root-mean-squared-error (`rmse`) = ``\sqrt{\frac1n\sum_{i=1}^n(y_i - \hat y_i)^2}`` to evaluate our training error. To compute the test error, we use hourly data from 2019 to 2020.
"

# ╔═╡ 57f352dc-55ee-4e14-b68d-698938a97d92
rmse(predict(m1, training_set1.X), training_set1.y)

# ╔═╡ b0de002f-3f39-4378-8d68-5c4606e488b7
begin
    weather_test = CSV.read(joinpath(@__DIR__, "..", "data", "weather2019-2020.csv"), DataFrame);
    test_set1 = (X = (LUZ_pressure = weather_test.LUZ_pressure[1:end-5],),
                     y = weather_test.LUZ_wind_peak[6:end])
end

# ╔═╡ ce536f60-68b3-4901-bd6a-c96378054b12
rmse(predict(m1, test_set1.X), test_set1.y)

# ╔═╡ c65b81bd-395f-4461-a73b-3535903cb2d7
md"## Multiple Linear Regression

In multiple linear regression there are multiple predictors ``x_1, x_2, \ldots, x_p``
and the response is ``\hat y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p
`` (we use here ``\beta`` instead of ``\theta`` for the parameters).
With ``p = 2`` predictors we can visualize linear regression as the plane
that is closest to the data. Use the sliders below to get a feeling for the
parameters. You can also change the viewing angle by clicking and dragging in
the figure."

# ╔═╡ 51c9ea74-3110-4536-a4af-7cc73b45a4a6
md"β₀ = $(@bind β₀ Slider(-1:.02:1, default = .4, show_value = true))

β₁ = $(@bind β₁ Slider(-1:.02:1, default = .2, show_value = true))

β₂ = $(@bind β₂ Slider(-1:.02:1, default = -.6, show_value = true))
"

# ╔═╡ 0f544053-1b7a-48d6-b18b-72092b124305
begin
    Random.seed!(3)
	X = DataFrame(X1 = randn(20), X2 = randn(20))
    f0(X1, X2, β₀, β₁, β₂) = β₀ + β₁ * X1 + β₂ * X2
    f0(β₀, β₁, β₂) = (X1, X2) -> f0(X1, X2, β₀, β₁, β₂)
	data_generator(X1, X2; β₀, β₁, β₂, σ = 0.8) = f0(X1, X2, β₀, β₁, β₂) + σ * randn()
	y = data_generator.(X.X1, X.X2, β₀ = .4, β₁ = .5, β₂ = -.6)
end;

# ╔═╡ d541a8cd-5aa4-4c2d-bfdf-5e6297bb65a8
begin
    plotly()
    p1 = scatter3d(X.X1, X.X2, y, markersize = 1,
                   xlims = (-3, 3), xlabel = "X1",
                   ylims = (-3, 3), ylabel = "X2",
                   zlims = (-4, 4), zlabel = "y", label = "data")
    wireframe!(-3:.1:3, -3:.1:3, f0(β₀, β₁, β₂),
               label = "function", title = "data & function", color = :green)
    plot_residuals!(X.X1, X.X2, y, f0(β₀, β₁, β₂))
    p2 = contour(-1:.1:1, -1:.1:1, (β₀, β₁) -> mean((β₀ .+ β₁ .* X.X1 .+ β₂ .* X.X2 .- y).^2), levels = 100, ylabel = "β₁", cbar = false, title = "loss")
    scatter!([β₀], [β₁], label = nothing)
    p3 = contour(-1:.1:1, -1:.1:1, (β₀, β₂) -> mean((β₀ .+ β₁ .* X.X1 .+ β₂ .* X.X2 .- y).^2), levels = 100, xlabel = "β₀", ylabel = "β₂")
    scatter!([β₀], [β₂], label = "current loss")
    plot(p1, plot(p2, p3, layout = (2, 1)), layout = (1, 2),
         size = (700, 400), legend = false)
end

# ╔═╡ da6462d8-3343-41d8-82dd-48770176d4ba
md"## Wind speed prediction with multiple predictors"

# ╔═╡ 753ec309-1363-485d-a2bd-b9fa100d9058
m2 = machine(LinearRegressor(), select(weather[1:end-5,:], Not([:LUZ_wind_peak, :time])),
             weather.LUZ_wind_peak[6:end]) |> fit!;

# ╔═╡ 618ef3c7-0fda-4970-88e8-1dac195545de
sort!(DataFrame(predictor = names(select(weather, Not([:LUZ_wind_peak, :time]))),
                value = fitted_params(m2).coefs), :value)

# ╔═╡ 2d25fbb6-dc9b-40ad-bdce-4c952cdad077
rmse(predict(m2, select(weather[1:end-5,:], Not([:LUZ_wind_peak, :time]))),
     weather.LUZ_wind_peak[6:end])

# ╔═╡ c9f10ace-3299-45fb-b98d-023a35dd405a
rmse(predict(m2, select(weather_test[1:end-5,:], Not([:LUZ_wind_peak, :time]))),
     weather_test.LUZ_wind_peak[6:end])

# ╔═╡ 99a371b2-5158-4c42-8f50-329352b6c1f2
md"# Error Decomposition

"

# ╔═╡ f10b7cad-eda3-4ec9-99ee-d43ed013a057
begin
    f(x) = sin(2x) + 2*(x - .5)^3 - .5x
    conditional_generator(x; n = 50) = f.(x) .+ .2*randn(n)
end;

# ╔═╡ 05354df5-a803-422f-87a3-1c56a34e8a48
f̂(x) = 0.1 + x

# ╔═╡ 9e61b4c3-1a9f-41a7-9882-25ed797a7b8d
expected_error(f, x) = mean((conditional_generator(x, n = 10^6) .- f(x)).^2);

# ╔═╡ c6a59b85-d031-4ad4-9e24-691494d08cde
expected_error(f̂, .1)

# ╔═╡ e50b8196-e804-473a-b3b5-e22fdb9d2f45
(f(.1) - f̂(.1))^2

# ╔═╡ f413ea94-36ca-4afc-8ca8-9a7e88101980
expected_error(f, .1)

# ╔═╡ dbf7fc72-bfd0-4c57-a1a9-fb5881e16e7e
let x = rand(100), grid = 0:.05:1
    gr()
    p1 = scatter(x, vcat(conditional_generator.(x, n = 1)...), label = "samples")
    plot!(f, label = "f")
    plot!(f̂, label = "f̂")
    p2 = plot(grid, expected_error.(f̂, grid), label = "expected error f̂", w = 3)
    plot!(grid, (f.(grid) .- f̂.(grid)).^2, label = "reducible error", w = 3)
    hline!([.2^2], label = "irreducible error", ylims = (0, .15), w = 3, xlabel = "x")
    plot(p1, p2, layout = (2, 1), legend = :right, ylabel = "y")
end

# ╔═╡ ad5b293d-c0f4-4693-84f4-88308639a501
md"# Logistic Regression

## Preparing the spam data

The text in our spam data set is already preprocessed. But we do not yet have
a format similar to our weather prediction data set with a fixed number ``p`` of
predictors for each email. In this section we create a very simple feature
representation of our emails:
1. We create a lexicon of words that are neither very frequent nor very rare.
2. For each email we count how often every word in this lexicon appears.
3. Our feature matrix will consist of ``n`` rows (one for each email) and ``p``
    predictors (one for each word in the lexicon) with ``x_{ij}`` measuring how
    often word ``j`` appears in document ``i``, normalized by the number of
    lexicon words in each email (such that the elements in every row sum to 1).
"

# ╔═╡ 210b977d-7136-407f-a1c9-eeea869d0312
begin
    spamdata = CSV.read(joinpath(@__DIR__, "..", "data", "spam.csv"), DataFrame)
    dropmissing!(spamdata) # remove entries without any text (missing values).
end

# ╔═╡ 72969aca-b203-4d83-8923-74e523aa1c01
import TextAnalysis: Corpus, StringDocument, DocumentTermMatrix, lexicon,
                     update_lexicon!, tf

# ╔═╡ 4cbb3057-01f4-4e80-9029-4e80d6c9e5e6
md"In the next cell we create the full lexicon of words appearing in the first
2000 emails. Each lexicon entry is of the form `\"word\" => count`."

# ╔═╡ c50c529f-d393-4854-b5ed-91e90d557d12
begin
    crps = Corpus(StringDocument.(spamdata.text[1:2000]))
    update_lexicon!(crps)
    lexicon(crps)
end

# ╔═╡ 72b50cee-d436-42ce-add9-07b0c012cb31
md"Now we select only those words of the full lexicon that appear at least 100
times and at most 10^3 times. These numbers are pulled out of thin air (like
all the design choses of this very crude feature engineering).
"

# ╔═╡ bf4110a9-31a4-48a3-bd6d-85c404d0e72d
begin
    small_lex = Dict(k => lexicon(crps)[k]
                     for k in findall(x -> 100 <= last(x) <= 10^3, lexicon(crps)))
    m = DocumentTermMatrix(crps, small_lex)
end

# ╔═╡ 534681d5-71d8-402a-b455-f491cfbb353e
begin
    spam_or_ham = coerce(String.(spamdata.label[1:2000]), Binary)
    normalized_word_counts = float.(DataFrame(tf(m), :auto))
end

# ╔═╡ ec1c2ea5-29ce-4371-be49-08798305ff50
Markdown.parse("Here we go: now we have a matrix of size
 $(join(size(normalized_word_counts), " x ")) as input and a vector of binary label as
 output. We will be able to use this as input in multiple logistic regression.
 For future usage we save this preprocessed representation of the spam data to
 a file.")

# ╔═╡ 681cb7b9-f041-4aea-907e-4d85135c005a
CSV.write(joinpath(dirname(pathof(MLCourse)), "..", "data", "spam_preprocessed.csv"),
          [normalized_word_counts DataFrame(spam_or_ham = spam_or_ham)])

# ╔═╡ f7117513-283f-4e32-a2a1-3594c794c94d
md"## Multiple Logistic Regression

In the top row of the figure below we see in two different ways (once as a 3D
plot and once as a contour plot) the probability of class A for the selected
parameter values. The bottom row shows samples (large points, red = class A)
obtained with this probability distribution and predictions (small points)
at decision threshold 0.5.
Play with the parameters to get a feeling for how they affect the probability
and the samples.

θ₀ = $(@bind θ₀ Slider(-3:3, default = 0, show_value = true))

θ₁ = $(@bind θ₁ Slider(-8:8, default = 3, show_value = true))

θ₂ = $(@bind θ₂ Slider(-8:8, default = 0, show_value = true))
"

# ╔═╡ fd4165dc-c3e3-4c4c-9605-167b5b4416da
md"## Confusion Matrix, ROC and AUC"

# ╔═╡ 7738c156-8e1b-4723-9818-fba364822171
md"s = $(@bind s Slider(-4:.1:4, default = 0, show_value = true))

seed = $(@bind seed Slider(1:100, show_value = true))

threshold = $(@bind threshold Slider(.01:.01:.99, default = 0.5, show_value = true))
"

# ╔═╡ 0fcfd7d2-6ea3-4c75-bad3-7d0fdd6fde11
begin
    logistic(x) = 1/(1 + exp(-x))
    logodds(p) = log(p/(1-p))
    function error_rates(x, y, t)
        P = sum(y)
        N = length(y) - P
        pos_pred = y[(logistic.(x) .> t)]
        TP = sum(pos_pred)
        FP = sum(1 .- pos_pred)
        FP/N, TP/P
    end
end;

# ╔═╡ 4f89ceab-297f-4c2c-9029-8d2d7fad084f
let f(x1, x2) = logistic(θ₀ + θ₁ * x1 + θ₂ * x2)
    p1 = wireframe(-3:.1:3, -3:.1:3, f, zlims = (0, 1))
    p2 = contour(-3:.1:3, -3:.1:3, f, contour_labels = true, levels = 20, cbar = false)
	plotly()
    samples = (X1 = 6 * rand(200) .- 3, X2 = 6 * rand(200) .- 3)
    labels = f.(samples.X1, samples.X2) .> rand(200)
    xgrid = MLCourse.grid(-3:.2:3, -3:.2:3, names = (:X1, :X2))
    scatter(xgrid.X1, xgrid.X2, color = (f.(xgrid.X1, xgrid.X2) .> .5) .+ 1,
            markersize = 2, markerstrokewidth = 0, label = nothing)
    p3 = scatter!(samples.X1, samples.X2, color = labels .+ 1, xlabel = "X1")
    plot(p1, p2, plot(), p3, layout = (2, 2), size = (700, 600),
         ylabel = "X2", legend = false)
end

# ╔═╡ 285c6bfc-5f29-46e0-a2c1-8abbec74501b
begin
    Random.seed!(seed)
    auc_samples_x = 2 * randn(200)
end;

# ╔═╡ c98524b5-d6b3-469c-82a1-7d231cc792d6
begin
    auc_samples_y = logistic.(2.0^s * auc_samples_x) .> rand(200)
    auc = [error_rates(2.0^s * auc_samples_x, auc_samples_y, t)
           for t in .01:.01:.99]
    push!(auc, (0., 0.))
    prepend!(auc, [(1., 1.)])
end;

# ╔═╡ 3336ab15-9e9b-44af-a7d5-1d6472241e62
let
    gr()
    p1 = scatter(auc_samples_x, auc_samples_y, markershape = :vline, label = nothing, color = :black)
    plot!(x -> logistic(2.0^s * x), color = :blue, label = nothing, xlims = (-8, 8))
    vline!([1/(2.0^s) * logodds(threshold)], w = 3, color = :red,
           label = nothing, xlabel = "x", ylabel = "y")
    p2 = plot(first.(auc), last.(auc), title = "ROC", label = nothing)
    fp, tp = auc[floor(Int, threshold * 100)]
    scatter!([fp], [tp], color = :red, xlims = (-.01, 1.01), ylims = (-.01, 1.01),
            labels = nothing, ylabel = "true positive rate",
            xlabel = "false positive rate")
    plot(p1, p2, size = (700, 400))
end

# ╔═╡ 62ad57e5-1366-4635-859b-ccdab2efd3b8
md"## Multiple Logistic Regression on the spam data"

# ╔═╡ 29e1d9ff-4375-455a-a69b-8dd0c2cac57d
m3 = fit!(machine(LogisticClassifier(penalty = :none),
                  normalized_word_counts,
                  spam_or_ham));

# ╔═╡ 1d1a24c6-c166-49a2-aa21-7acf50b55a66
predict(m3, normalized_word_counts)

# ╔═╡ 32bafa9e-a35e-4f54-9857-d269b47f95c3
confusion_matrix(predict_mode(m3, normalized_word_counts), spam_or_ham)

# ╔═╡ 4e4f4adf-364f-49b9-9391-5050a4c1286a
md"With our simple features, logistic regression can classify the training data
almost always correctly. Let us see how well this works for test data.
"

# ╔═╡ 50c035e6-b892-4157-a52f-824578366977
begin
    test_crps = Corpus(StringDocument.(spamdata.text[2001:4000]))
    test_input = float.(DataFrame(tf(DocumentTermMatrix(test_crps, small_lex)), :auto))
    test_labels = coerce(String.(spamdata.label[2001:4000]), Binary)
    confusion_matrix(predict_mode(m3, test_input), test_labels)
end

# ╔═╡ 21b66582-3fda-401c-9421-73ae2f455a75
predict_mode(m3, normalized_word_counts)

# ╔═╡ ba4b5683-5932-415e-8772-8b3eef5eb63d
md"We save also the test data for future usage."

# ╔═╡ a30578dd-aecb-46eb-b947-f009282cf2fc
md"Let us evaluate the fit in terms of commonly used losses for binary classification."

# ╔═╡ 8ed39cdc-e99e-48ff-9973-66df41aa0f78
function losses(machine, input, response)
    (loglikelihood = -sum(log_loss(predict(machine, input), response)),
     misclassification_rate = mean(predict_mode(machine, input) .!= response),
     accuracy = accuracy(predict_mode(machine, input), response),
     auc = MLJ.auc(predict(machine, input), response)
	)
end;

# ╔═╡ dd463687-b73d-4e70-b2cf-97a56a0ad409
losses(m3, normalized_word_counts, spam_or_ham)

# ╔═╡ 57dcadc0-2da2-4521-aeaf-6fd01f4bd82b
spam_or_ham

# ╔═╡ 935adbcd-48ab-4a6f-907c-b04137ca3abe
losses(m3, test_input, test_labels)

# ╔═╡ 8b0451bf-59b0-4e71-be84-549e23b5bfe7
md"""# Exercises

## Conceptual

1. Suppose we have a data set with three predictors, ``X_1`` = Final Grade, ``X_2`` = IQ, ``X_3`` = Level (1 for College and 0 for High School).  The response is starting salary after graduation (in thousands of dollars). Suppose we use least squares to fit the model, and get ``\hat\beta_0 = 25, \hat\beta_1 = 2, \hat\beta_2 = 0.07, \hat\beta_3 = 15``.
   - Which answer is correct, and why?
      - For a fixed value of IQ and Final Grade, high school graduates earn more, on average, than college graduates.
      - For a fixed value of IQ and Final Grade, college graduates earn more, on average, than high school graduates.
   - Predict the salary of a college graduate with IQ of 110 and a Final Grade of 4.0.
2. Suppose we collect data for a group of students in a machine learning class with variables ``X_1 =`` hours studied, ``X_2 =`` grade in statistics class, and ``Y =`` receive a 6 in the machine learning class. We fit a logistic regression and produce estimated coefficients, ``\hat{\beta}_0 = -6``, ``\hat{\beta}_1 = 0.025``, ``\hat{\beta}_2 = 1``.
   - Estimate the probability that a student who studies for 75 hours and had a 4 in the statistics class gets a 6 in the machine learning class.
   - How many hours would the above student need to study to have a 50% chance of getting an 6 in the machine learning class?
3. In this exercise we will derive the loss function implicitly defined by maximum likelihood estimation of the parameters in a classification setting with multiple classes. Remember that the input ``f(x)`` of the softmax function ``s`` is a vector-valued function. Here we assume a linear function ``f`` and write the ``i``th component of this function as ``f_i(x) = \theta_{i0} + \theta_{i1}x_1 + \cdots + \theta_{ip}x_p``. Note that each component ``i`` has now its own parameters ``\theta_{i0}`` to ``\theta_{ip}``. Using matrix multiplication we can also write ``f(x) = \theta x`` where ``\theta = \left(\begin{array}{ccc}\theta_{10} & \cdots & \theta_{1p}\\\vdots & \ddots & \cdots\\\theta_{K0} & \cdots & \theta_{Kp}\end{array}\right)`` is a ``K\times(p+1)`` dimensional matrix and ``x = (1, x_1, x_2, \ldots, x_p)`` is a column vector of length ``p+1``.
    - Write the log-likelihood function for a classification problem with ``K`` classes. *Hint*: to simplify the notation we can use the convention ``s_y(f(x)) = P(y|x)`` to write the conditional probability of class ``y`` given input ``x``. This convention makes sense when the classes are identified by the integers ``1, 2, \ldots, K``; in this case ``s_y(f(x))`` is the ``y``th component of ``s(f(x))``. Otherwise we would could specify a mapping from classes ``C_1, C_2, \ldots, C_K`` to the integers ``1, 2, \ldots, K`` for this convention to make sense.
    - Assume now ``K = 3`` and ``p = 2``. Explicitly write the loss function for the training set ``\mathcal D = ((x_1 = (0, 0), y_1 = C), (x_2 = (3, 0), y_2 = A), (x_3 = (0, 2), y_3 = B))``.
    - Assume ``K = 2`` and ``p = 1`` and set ``\theta_{20} = 0`` and ``\theta_{21} = 0``. Show that we recover standard logistic regression in this case. *Hint*: show that ``s_1(f(x)) = \sigma(f_1(x))`` and ``s_2(f(x)) = 1 - \sigma(f_1(x))``, where ``s`` is the softmax function and ``\sigma(x) = 1/(1 + e^{-x})`` is the logistic function.
    - Show that one can always set ``\theta_{K0}, \theta_{K1}, \ldots, \theta_{Kp}`` to zero. *Hint* Show that the softmax function with the transformed parameters ``\tilde\theta_{ij}=\theta_{ij} - \theta_{Kj}`` has the same value as the softmax function in the original parameters.

## Applied
1. In the multiple linear regression of the weather data set above we used all
   available predictors. We do not know if all of them are relevant. In this exercise our aim is to find models with fewer predictors and quantify the loss in prediction accuracy.
    - Systematically search for the model with at most 2 predictors that has the lowest test rmse. *Hint* write a function `train_and_evaluate` that takes the training and the test data as input as well as an array of two predictors; remember that `data[:, [\"A\", \"B\"]]` returns a sub-dataframe with columns \"A\" and \"B\". This function should fit a `LinearRegressor` on the training set with those two predictors and return the test rmse for the two predictors. To get a list of all pairs of predictors you can use something like `predictors = setdiff(names(train), ["time", "LUZ_wind_peak"]); predictor_pairs = [[p1, p2] for p1 in predictors, p2 in predictors if p1 != p2 && p1 > p2]`
    - How much higher is the test error compared to the fit with all available predictors?
    - How many models did you have to fit to find your result above?
    - How many models would you have to fit to find the best model with at most 5 predictors? *Hint* the function `binomial` may be useful.
2. In this exercise we perform linear classification of the MNIST handwritten digits
   dataset.
   - Load the MNIST data set with `using OpenML; mnist = OpenML.load(554) |> DataFrame; dropmissing!(mnist);`
   - Usually the first 60'000 images are taken as training set, but for this exercise I recommend to use fewer rows, e.g. the first 5000.
   - Scale the input values to the interval [0, 1) with `mnist[:, 1:784] ./= 255`
   - Fit a `MLJLinearModels.MultinomialClassifier(penalty = :none)` to the data. Be patient! This can take a few minutes.
   - Compute the misclassification rate and the confusion matrix on the training set.
   - Use as test data rows 60001 to 70000 and compute the misclassification rate
     and the confusion matrix on this test set.
   - Plot some of the wrongly classified training and test images.
     Are they also difficult for you to classify?
"""

# ╔═╡ 7f08fcaa-000d-422d-80b4-e58a2f489d74
MLCourse.footer()

# ╔═╡ Cell order:
# ╟─12d5824c-0873-49a8-a5d8-f93c73b633ae
# ╠═94f8e29e-ef91-11eb-1ae9-29bc46fa505a
# ╟─8217895b-b120-4b08-b18f-d921dfdddf10
# ╠═9f84bcc5-e5ab-4935-9076-c19e9bd668e8
# ╠═34e527f2-ef80-4cb6-be3a-bee055eca125
# ╠═006fc1eb-50d5-4206-8c87-53e873f158f4
# ╟─e4712ebe-f395-418b-abcc-e10ada4b05c2
# ╠═8c9ae8f7-81b2-4d60-a8eb-ded5364fe0cc
# ╟─f4f890b6-0ad4-4155-9321-15d673e15489
# ╠═9b62c374-c26e-4990-8ffc-790928e62e88
# ╟─7923a0a8-3033-4dde-91e8-22bf540c9866
# ╠═57f352dc-55ee-4e14-b68d-698938a97d92
# ╠═b0de002f-3f39-4378-8d68-5c4606e488b7
# ╠═ce536f60-68b3-4901-bd6a-c96378054b12
# ╟─c65b81bd-395f-4461-a73b-3535903cb2d7
# ╟─51c9ea74-3110-4536-a4af-7cc73b45a4a6
# ╟─d541a8cd-5aa4-4c2d-bfdf-5e6297bb65a8
# ╟─0f544053-1b7a-48d6-b18b-72092b124305
# ╟─da6462d8-3343-41d8-82dd-48770176d4ba
# ╠═753ec309-1363-485d-a2bd-b9fa100d9058
# ╠═618ef3c7-0fda-4970-88e8-1dac195545de
# ╠═2d25fbb6-dc9b-40ad-bdce-4c952cdad077
# ╠═c9f10ace-3299-45fb-b98d-023a35dd405a
# ╟─99a371b2-5158-4c42-8f50-329352b6c1f2
# ╠═f10b7cad-eda3-4ec9-99ee-d43ed013a057
# ╠═05354df5-a803-422f-87a3-1c56a34e8a48
# ╠═9e61b4c3-1a9f-41a7-9882-25ed797a7b8d
# ╠═c6a59b85-d031-4ad4-9e24-691494d08cde
# ╠═e50b8196-e804-473a-b3b5-e22fdb9d2f45
# ╠═f413ea94-36ca-4afc-8ca8-9a7e88101980
# ╟─dbf7fc72-bfd0-4c57-a1a9-fb5881e16e7e
# ╟─ad5b293d-c0f4-4693-84f4-88308639a501
# ╠═210b977d-7136-407f-a1c9-eeea869d0312
# ╠═72969aca-b203-4d83-8923-74e523aa1c01
# ╟─4cbb3057-01f4-4e80-9029-4e80d6c9e5e6
# ╠═c50c529f-d393-4854-b5ed-91e90d557d12
# ╟─72b50cee-d436-42ce-add9-07b0c012cb31
# ╠═bf4110a9-31a4-48a3-bd6d-85c404d0e72d
# ╠═534681d5-71d8-402a-b455-f491cfbb353e
# ╟─ec1c2ea5-29ce-4371-be49-08798305ff50
# ╠═681cb7b9-f041-4aea-907e-4d85135c005a
# ╟─f7117513-283f-4e32-a2a1-3594c794c94d
# ╟─4f89ceab-297f-4c2c-9029-8d2d7fad084f
# ╟─fd4165dc-c3e3-4c4c-9605-167b5b4416da
# ╟─7738c156-8e1b-4723-9818-fba364822171
# ╠═0fcfd7d2-6ea3-4c75-bad3-7d0fdd6fde11
# ╟─3336ab15-9e9b-44af-a7d5-1d6472241e62
# ╟─285c6bfc-5f29-46e0-a2c1-8abbec74501b
# ╟─c98524b5-d6b3-469c-82a1-7d231cc792d6
# ╟─62ad57e5-1366-4635-859b-ccdab2efd3b8
# ╠═29e1d9ff-4375-455a-a69b-8dd0c2cac57d
# ╠═1d1a24c6-c166-49a2-aa21-7acf50b55a66
# ╠═32bafa9e-a35e-4f54-9857-d269b47f95c3
# ╟─4e4f4adf-364f-49b9-9391-5050a4c1286a
# ╠═50c035e6-b892-4157-a52f-824578366977
# ╠═21b66582-3fda-401c-9421-73ae2f455a75
# ╟─ba4b5683-5932-415e-8772-8b3eef5eb63d
# ╟─a30578dd-aecb-46eb-b947-f009282cf2fc
# ╠═8ed39cdc-e99e-48ff-9973-66df41aa0f78
# ╠═dd463687-b73d-4e70-b2cf-97a56a0ad409
# ╠═57dcadc0-2da2-4521-aeaf-6fd01f4bd82b
# ╠═935adbcd-48ab-4a6f-907c-b04137ca3abe
# ╟─8b0451bf-59b0-4e71-be84-549e23b5bfe7
# ╟─20c5c7bc-664f-4c04-8215-8f3a9a2095c9
# ╟─7f08fcaa-000d-422d-80b4-e58a2f489d74


using UnsafeArrays

export Maxima,
       Minima

export peakprom

struct Maxima; end
struct Minima; end

for (comps, Extrema) in (((:>, minimum, max), Maxima),
                       ((:<, maximum, min), Minima))

    comp, argcomp1, argcomp2 = comps

    @eval begin
        function peakprom(x::AbstractVector{T}, ::$Extrema, w=1, minprom::T=zero(T)) where T
            if ($Extrema) === Maxima
                m = maxima(x, w)
            else
                m = minima(x, w)
            end

            M = lastindex(m)
            lbegin = firstindex(x)
            lend = lastindex(x)
            proms = similar(x,M)

            @inbounds for i in eachindex(m)
                lb, rb = lbegin, lend
                lcan, rcan = zero(T), zero(T)

                # Find left bound
                for ii in (i-1):-1:1
                    if ($comp)(x[m[ii]], x[m[i]])
                        lb = m[ii]
                        break
                    end
                end

                # Find left bound
                for ii in (i+1):M
                    if ($comp)(x[m[ii]], x[m[i]])
                        rb = m[ii]
                        break
                    end
                end

                lcan = ($argcomp1)(uview(x, lb:(m[i] - 1))) # Slice from lower bound to the index prior to the current extrema
                rcan = ($argcomp1)(uview(x, (m[i] + 1):rb)) # Slice corollary upper side

                proms[i] = abs(x[m[i]] - ($argcomp2)(lcan, rcan))
            end

            if minprom != zero(T)
                matched = findall(x -> x >= minprom, proms)
                return (m[matched], proms[matched])
            else
                return (m, proms)
            end
        end

    end
end

@doc """
    peakprom(x, ::Maxima[, w=1, minprom=0])

Find all local maxima and maxima prominences in `x` matching the conditions `w` and `minprom`.
`w` sets the minimum allowed distance between maxima. `minprom` sets the minimum prominence
(inclusive) of returned maxima.

Peak prominence is calculated as the distance between the current maxima and the highest of
the minimums of the lower and upper bounds. Bounds extend from the next index from the
current maxima to the next maxima higher than the current maxima, or the end of the signal,
which ever comes first.

# Examples
```jldoctest
julia> x = rand(1000);

julia> ma, pa = peakprom(x, Maxima());

julia> mi, pi = peakprom(-x, Minima());

julia> @assert (mi == ma) && (pa == pi)

```

See also: [`maxima`](@ref), [`minima`](@ref)
"""
peakprom(x, ::Maxima)

@doc """
    peakprom(x, ::Minima[, w=1, minprom=0])

Find all local minima and minima prominences in `x` matching the conditions `w` and `minprom`.
`w` sets the minimum allowed distance between minima. `minprom` sets the minimum prominence
(inclusive) of returned minima.

Peak prominence is calculated as the distance between the current minima and the lowest of
the maximums of the lower and upper bounds. Bounds extend from the next index from the
current minima to the next minima lower than the current minima, or the end of the signal,
which ever comes first.

See also: [`maxima`](@ref), [`minima`](@ref)
"""
peakprom(x, ::Minima)


# Classical Multidimensional Scaling

## convert Gram matrix to Distance matrix

function gram2dmat!{DT}(D::AbstractMatrix{DT}, G::AbstractMatrix)
    # argument checking
    m = size(G, 1)
    n = size(G, 2)
    m == n || error("D should be a square matrix.")
    size(D) == (m, n) ||
        throw(DimensionMismatch("Sizes of D and G do not match."))

    # implementation
    for j = 1:n
        for i = 1:j-1
            @inbounds D[i,j] = D[j,i]
        end
        D[j,j] = zero(DT)
        for i = j+1:n
            @inbounds D[i,j] = sqrt(G[i,i] + G[j,j] - 2 * G[i,j])
        end
    end
    return D
end

gram2dmat{T<:Real}(G::AbstractMatrix{T}) = gram2dmat!(similar(G, Base.momenttype(T)), G)

## convert Distance matrix to Gram matrix

function dmat2gram!{GT}(G::AbstractMatrix{GT}, D::AbstractMatrix)
    # argument checking
    n = Compat.LinAlg.checksquare(D)
    size(G) == (n, n) ||
        throw(DimensionMismatch("Sizes of G and D do not match."))

    # implementation
    u = zeros(GT, n)
    s = 0.0
    for j = 1:n
        s += (u[j] = sum(abs2, view(D,:,j)) / n)
    end
    s /= n

    for j = 1:n
        for i = 1:j-1
            @inbounds G[i,j] = G[j,i]
        end
        for i = j:n
            @inbounds G[i,j] = (u[i] + u[j] - abs2(D[i,j]) - s) / 2
        end
    end
    return G
end

dmat2gram{T<:Real}(D::AbstractMatrix{T}) = dmat2gram!(similar(D, Base.momenttype(T)), D)

## classical MDS

function classical_mds{T<:Real}(D::AbstractMatrix{T}, p::Int;
        dowarn::Bool=true)

    n = size(D, 1)
    m = min(p, n) #Actual number of eigenpairs wanted

    G = dmat2gram(D)

    #Get m largest eigenpairs
    E = eigfact!(Symmetric(G))

    #Sometimes dmat2gram produces a negative definite matrix, and the sign just
    #needs to be flipped. The heuristic to check for this robustly is to check
    #if there is a negative eigenvalue of magnitude larger than the largest
    #positive eigenvalue, and flip the sign of eigenvalues if necessary.

    mineig, maxeig = extrema(E[:values])
    if mineig < 0 && abs(mineig) > abs(maxeig)
        #do flip
        ord = sortperm(E.values)
        v = -E[:values][ord[1:m]]
    else
        ord = sortperm(E.values; rev=true)
        v = E[:values][ord[1:m]]
    end

    for i = 1:m
        if v[i] > 0
            v[i] = √v[i]
        else #Keeping all remaining eigenpairs would not change solution (if 0)
             #or make the answer _worse_ (if <0).
             #The least squares solution would want to throw all these away.
            dowarn && warn("Gramian has only $(i-1) positive eigenvalue(s)")
            m = i-1
            ord = ord[1:m]
            v = v[1:m]
            break
        end
    end

    #Check if the last considered eigenvalue is degenerate
    if m>0
        nevalsmore = sum(abs.(E[:values][ord[m+1:end]] .- v[m]^2) .< n*eps())
        nevals = sum(abs.(E[:values] .- v[m]^2) .< n*eps())
        if nevalsmore > 1
            dowarn && warn("The last eigenpair is degenerate with $(nevals-1) others; $nevalsmore were ignored. Answer is not unique")
        end
    end
    U = E[:vectors][:, ord[1:m]]
    scale!(U, v)

    #Add trailing zero coordinates if dimension of embedding space (p) exceeds
    #number of eigenpairs used (m)
    if m < p
        U = [U zeros(n, p-m)]
    end

    U' #Return each coordinate in a column
end


#' ## Functions
#'
#'
#' In the last notebook, we talked about modeling data with functions. A **function** is one of the most fundamental concepts in computing (and also in mathematics).
#'
#' A function is a piece of a program that receives **input arguments**, processes them by doing certain calculations on them, and returns **outputs**.
#'
#' For example, we might have a function `g` that takes a number as an input and returns the square of that number as an output. How can we define this function `g` on a computer? Julia gives us a few different ways to do this.
#' ### Defining functions
#' Firstly, we could write `g` as follows:

#+ results="hidden"
g(x) = x^2
#+
#'
g(2), g(3.5) # This is a tuple.
#'
g(3.0)
#' Alternatively, we could declare this function using the `function` and `end` keywords:
#+ results="hidden"
function g1(x)
    x^2
end
#+
g1(2), g1(3.5)
#' The third way we could have declared this function is as an "anonymous" or
#' "lambda" function. "Anonymous" functions are functions that truly don't need
#' names! For example, we could have declared a function that squares its input
#' as
(x->x^2)(2)
#' Now that we've done that, we can't access the function `x -> x^2` again
#' because we have no name to call! That seems a little silly, doesn't it?
#'
#' Actually, there are times where functions without names are especially
#' handy.  Most commonly they're used arguments to "higher-order" functions:
#' these are functions which take _other functions_ as arguments. For example:
map(sqrt, [1, 2, 3])
#'
map(x->x^2, [1, 2, 3])
#' Of course, you can also assign anonymous function a name, but in general
#' it's better style to use the non-anonymous equivalents above if you actually
#' want to use a name.
g2 = x -> x^2
g2(3.5), g2("I ♡ Julia ") # Use \heartsuit + TAB to get the ♡ character

#' ## An important sigmoidal function
#' A particular function that is used a lot in machine learning is a so-called
#' "sigmoidal" function (meaning a function that is S-shaped, i.e. the graph
#' of the function looks like an `S`).
#'
#' $$\sigma(x) = \frac{1}{1 + e^{-x}}$$
#'
#' The sigmoid function that we will use is given the name $\sigma$, and is
#' defined by the following mathematical expression:
#'
#' $$\sigma(x) := \frac{1}{1 + \exp(-x)}$$

#' #### Exercise 1
#'
#' Use the first syntax given above to define the function `σ` in Julia.
#' Note that Julia actually allows us to use the symbol σ as a variable name!
#' To do so, type `\sigma<TAB>` in the code cell.

#' #### Solution
# Short form
σ(x) = 1/(1 + exp(-x))
σ(1)
#+ results="hidden"
# Long form
function sigma(x)
    1/(1 + exp(-x))
end
#+
sigma(1)

#' ## Plotting functions

#' Let's draw the function σ to see what it looks like. Throughout this course,
#' we'll use the Julia package `Plots.jl` for all of the graphics. This package
#' provides a flexible syntax for plotting, in which options to change
#' attributes like the width of the lines used in the figure are given as named
#' keyword arguments.
#'
#' In addition, it allows us to use different "backends", which are the other
#' libraries that actually carry out the plotting following the instructions
#' from `Plots.jl`.
using Plots
#gr()
# plotly interferes mathjax, tpl file needs to be fixed
# see templates/julia_html.tpl
plotly()

pl = plot(sigma, -5, 5, legend=(0.1,0.9), label="σ")
# add horizontal lines at 0 and 1, with dashed style and linewidth 3
hline!(pl, [0,1], ls=:dash, lw=3, label=false)
# add a vertical line at 0
vline!(pl, [0], ls=:dash, lw=3, label=false)

#' We can think of $\sigma$ as a smooth version of a step or threshold function
#' (often called a "Heaviside" function). To see this, let's modify the
#' steepness of the jump in $\sigma$ and compare it to the Heaviside function;
#' we'll see how all this works in more detail later:
#+ results="hidden"
function heaviside(x)
    x < 0 ? 0.0 : 1.0
end
#+

#' Try manipulating the value of `w` between 0 to :
w = 10.0
plot(x->sigma(w*x), -5, 5, label="σ", lw=2, legend=(0.1,0.9))
plot!(heaviside, ls=:dash, label="step")
# This particular function takes any real number as input, and gives an output
# between $0$ and $1$. It is continuous and smooth.
#' Try a smaller $w$.
w = 1.0
plot(x->sigma(w*x), -5, 5, label="σ", lw=2, legend=(0.1,0.9))
plot!(heaviside, ls=:dash, label="step")
#' #### Exercise 2
#' Declare the sigmoid function above as an anonymous function with a different
#' name.
σx = x -> 1/(1+exp(-x))
σx(1)
#' ### Mutating functions: `...!`
#' To generate our plot of σ above, we used some functions that end with `!`.
#'  What does a `!` at the end of a function name mean in Julia?
#'
#' Functions that change or modify their inputs are called
#' **mutating functions**. But wait, don't all functions do that?
#' No, actually. Functions typically take *inputs* and use those *inputs* to
#' generate *outputs*, but the inputs themselves usually don't actually get
#' changed by a function. For example, copy and execute the following code:
v1 = [9, 4, 7, 11]
v2 = sort(v1)
#' `v2` is a sorted version of `v1`, but after calling `sort`, `v1` is still
#' unsorted.
v1
#+ results="hidden"
sort!(v1)
#' Look at the values in `v1` now!
v1
#' This time, the original vector itself was changed (mutated), and is now
#' sorted. Unlike `sort`, `sort!` is a mutating function. Did the `!` make
#' `sort!` mutating? Well, no, not really. In Julia, `!` indicates mutating
#' functions by convention. When the author of `sort!` wrote `sort!`, they added
#' a `!` to let you to know that `sort!` is mutating, but the `!` isn't what
#' makes a function mutating or non-mutating in the first place.

#' #### Exercise
#' Some of our plotting commands end with `!`. Copy and execute the following code:
r = -5:0.1:5
g(x) = x^2
h(x) = x^3
plot(r, g, label="g")
plot!(r, h, label="h")
#' Then change the code slightly to remove the `!` after `plot!(r, h)`.
#' How does this change your output? What do you think it means to add `!`
#' after plotting commands?
#' #### Solution
r = -5:0.1:5
g(x) = x^2
h(x) = x^3
p1 = plot(r, g, label="g")
#'
p2 = plot(r, h, label="h")
#' ```julia
#' plot(r, g)
#' plot!(r, h)
#' ```
#' creates an overlay of `g` and `h`, whereas
#'
#' ```julia
#' plot(r, g)
#' plot(r, h)
#' ```
#'
#' creates one plot for `g` and one for `h`.
#'
#' When we add a `!` after a plotting command, we are mutating or updating an
#' *existing* plot.

#' ## Pointwise application of functions, `f.(x, y)` and `x .+ y` - broadcasting
#' We saw in a previous notebook that we needed to add `.` after the names of
#' some functions, as in
#'
#' ```julia
#' green_amount = mean(Float64.(green.(apple)))
#' ```
#'
#' What are those extra `.`s really doing?
#'
#' When we add a `.` after a function's name, we are telling Julia that we want
#' to "**broadcast**" that function over the inputs passed to the function.
#' This means that we want to apply that function *element-wise* over the
#' inputs; in other words, it will apply the function to each element of the
#' input, and return an array with the newly-calculated values.

#' For example, copy and execute the following code:
g.(r)
#' Since the function `g` squares it's input, this squares all the elements of
#' the range `r`.
#' What happens if instead we just call `g` on `r` via
# g(r)

#' You should see an error message after calling `g(r)`, which says that Julia
#' cannot multiply two vectors. When we call `g(r)`, we ask Julia to multiply
#' `r` by `r`. When we call `g.(r)`, we ask Julia to multiply *each element*
#' in `r` by itself.

#' Try this and see what happens.
x = [1 2 3; 4 5 6]
f(t) = sqrt(t) + 5
f.(x)

#' #### Exercise 3
#'
#' Copy and execute the following code to get the type of the object
#' `numbers = [1, 2, "three", 4.0]`:
#+ results="hidden"
numbers = [1, 2, "three", 4.0]
#' What is the type of `numbers`?
typeof(numbers)
#' Its type is `Array{Any,1}`. This means that `numbers` is a
#' 1-dimensional array that has elements of abstract type `Any`. We can think
#' of the `Any` type as a classification that includes *all* concrete types,
#' for example, `Float64`, `Int32`, `Bool`, `String`, `Char`, etc.

#' #### Exercise 4
#'
#' Broadcast `typeof` over `numbers` to see what the types of the elements
#' stored inside `numbers` are.
typeof.(numbers)

#' Note: Alternatively, we could have looked at the `typeof` each element in
#' the array `numbers` using a `for` loop:
for n in numbers
    println(typeof(n))
end

#' #### Exercise 5
#'
#' Write a `for` loop that applies `g` to each of the elements of `r` and
#' prints the results. Verify that the numbers printed by this `for` loop are
#' equal to the entries of the output of `g.(r)`.
r = -5:0.1:5
for x in r
    #println(g(x)) # uncomment to print
end

#' We can check this via
broadcasted_g = g.(r)
for (x, bg) = zip(r, broadcasted_g)
    # println(g(x) == bg) # uncomment to print
end

#' A better way
answers_match = true
for (x, bg) = zip(r, broadcasted_g)
    if (g(x) != bg)
        answers_match = false
    end
end
answers_match ? println("Our `for` loop worked!") : println("Our `for` loop does not work")

#' #### Exercise 6
#'
#' Define a range `xs` between -5 and 5 with steps of 0.5.
#' Apply the $\sigma$ function pointwise to this range and define `ys` as the
#' result.
#' What does the result look like? Plot these as points and join them with
#' lines.
#'
#' Make the plot interactive where you can vary the step size. Fix the range of
#' the plot in the `x` and `y` directions using the functions `xlims!` and
#' `ylims!`.
#'
#' Learn about plot optinos by `plotattr(:Series)` and `plotattr("markershape")`
xs = -5:0.5:5
ys = sigma.(xs)
pl = plot(xs, ys; legend=(0.1,0.9), markershape=:circle)

#' Try manipulating `stepsize` between 0 and 1:
stepsize = 0.5
xs = -5.0:stepsize:5.0
ys = σ.(xs)

scatter(xs, ys)
plot!(xs, ys)

xlims!(-5, 5)
ylims!(0, 1)


# private helper for union
function _union(A::Multiset{S}, B::Multiset{S}) where {S}
    M = Multiset{S}()
    for (x, v) in A.data
        M[x] = max(v, B[x])
    end
    for (y, w) in B.data
        M[y] = max(A[y], w)
    end
    return M
end

function type_convert(T, A::Multiset)
    M = Multiset{T}()
    for (x, v) in A.data
        xx = convert(T, x)
        M[xx] = v
    end
    return M
end

"""
`union(A,B)` for multisets creates a new multiset in which the
multiplicity of `x` is `max(A[x],B[x])`. This may be invoked as
`A ∪ B`.
"""
function union(A::Multiset{S}, B::Multiset{T}) where {S,T}
    if S == T
        return _union(A, B)
    end
    ST = typejoin(S, T)
    AA = type_convert(ST, A)
    BB = type_convert(ST, B)
    return _union(AA, BB)
end

"""
`A+B` for multisets is the disjoint union, i.e., a new multiset in which the
multiplicity of `x` is `A[x]+B[x]`.
"""
function (+)(A::Multiset{S}, B::Multiset{T}) where {S,T}
    ST = typejoin(S, T)
    M = Multiset{ST}()
    for (x, v) in A.data
        push!(M, x, v)
    end
    for (x, v) in B.data
        push!(M, x, v)
    end
    return M
end

# private helper for A-B
function _minus(A::Multiset{S}, B::Multiset{S}) where {S}
    M = Multiset{S}()
    for (x, v) in A.data
        M.data[x] = max(v - B[x], 0)
    end
    return M
end

"""
`A-B` for multisets is the multiset difference, i.e., a new multiset
in which the multiplicity of `x` is `A[x]-B[x]` unless this goes
below `0`, in which case the multiplicity is 0.
"""
function (-)(A::Multiset{S}, B::Multiset{T}) where {S,T}
    if S == T
        return _minus(A, B)
    end
    ST = typejoin(S, T)
    AA = type_convert(ST, A)
    BB = type_convert(ST, B)
    return _minus(AA, BB)
end

# private helper for intersect
function _intersect(A::Multiset{S}, B::Multiset{S}) where {S}
    M = Multiset{S}()
    for (x, v) in A.data
        push!(M, x, min(v, B[x]))
    end
    return M
end

"""
`intersect(A,B)` for multisets creates a new multiset in which the
multiplicity of `x` is `min(A[x],B[x])`.
This may be abbreviated `A ∩ B`.
"""
function intersect(A::Multiset{S}, B::Multiset{T}) where {S,T}
    if S == T
        return _intersect(A, B)
    end
    ST = typejoin(S, T)
    AA = type_convert(ST, A)
    BB = type_convert(ST, B)
    return _intersect(AA, BB)
end

"""
`A*B` for the Cartesian product of multisets `A` and `B`.
"""
function (*)(A::Multiset{S}, B::Multiset{T}) where {S,T}
    ST = Tuple{S,T}
    M = Multiset{ST}()
    for (a, v) in A.data
        for (b, w) in B.data
            M.data[(a, b)] = v * w
        end
    end
    return M
end

"""
`n*A` is the scalar multiple of a multiset in which the multiplicity of
`x` is `n*A[x]`. Of course, we require `n >= 0`.
"""
function (*)(n::Int, A::Multiset{T}) where {T}
    @assert n >= 0 "Scalar multiplication of a multiset must be by a nonnegative integer"
    M = Multiset{T}()
    for (x, v) in A.data
        M.data[x] = n * v
    end
    return M
end

# private helper for issubset
function _sub(A::Multiset{S}, B::Multiset{S}) where {S}
    for (x, v) in A.data
        if v > B[x]
            return false
        end
    end
    return true
end

function issubset(A::Multiset{S}, B::Multiset{T}) where {S,T}
    if S == T
        return _sub(A, B)
    end
    ST = typejoin(S, T)
    AA = type_convert(ST, A)
    BB = type_convert(ST, B)
    return _sub(AA, BB)
end

(==)(A::Multiset, B::Multiset) = (length(A) == length(B)) && issubset(A, B)

## mixing Multisets with other AbstractSets and the like

union(A::Multiset, B::T) where {T} = union(A, Multiset(B))
union(B::Set, A::Multiset) = union(A, Multiset(B))
union(B::BitSet, A::Multiset) = union(A, Multiset(B))

intersect(A::Multiset, B::T) where {T} = intersect(A, Multiset(B))
intersect(B::Set, A::Multiset) = intersect(A, Multiset(B))
intersect(B::BitSet, A::Multiset) = intersect(A, Multiset(B))

(+)(A::Multiset, B::T) where {T} = A + Multiset(B)
(+)(B::T, A::Multiset) where {T} = A + Multiset(B)

(-)(A::Multiset, B::T) where {T} = A - Multiset(B)
(-)(B::T, A::Multiset) where {T} = Multiset(B) - A

(*)(A::Multiset, B::T) where {T} = A * Multiset(B)
(*)(A::T, B::Multiset) where {T} = Multiset(A) * B


using OrdinaryDiffEq, DiffEqSensitivity, DiffEqFlux, Flux
using ComponentArrays, LinearAlgebra, GalacticOptim, Test

const nknots = 10
const h = 1.0/(nknots+1)
x = range(0, step=h, length=nknots)
u0 = sin.(π*x)

@inline function f(du,u,p,t)
  du .= zero(eltype(u))
  u₃ = @view u[3:end]
  u₂ = @view u[2:end-1]
  u₁ = @view u[1:end-2]
  @. du[2:end-1] = p.k*((u₃ - 2*u₂ + u₁)/(h^2.0))
  nothing
end

p_true = ComponentArray(k=0.42)
jac_proto = Tridiagonal(similar(u0,nknots-1), similar(u0), similar(u0, nknots-1))
prob = ODEProblem(ODEFunction(f,jac_prototype=jac_proto), u0, (0.0,1.0), p_true)
@time sol_true = solve(prob, Rodas4P(), saveat=0.1)

function loss(prob0, p)
  prob = remake(prob0, p=p)
  sol = solve(prob, Rodas4P(autodiff=false), saveat=0.1, sensealg=ForwardDiffSensitivity())
  sum((sol .- sol_true).^2)
end

p0 = ComponentArray(k=1.0)
res = DiffEqFlux.sciml_train(p -> loss(prob,p), p0, ADAM(0.01), GalacticOptim.AutoZygote(), maxiters=100)
@test res.u.k ≈ 0.42461977305259074 rtol=1e-1


#Constants and setup
using OrdinaryDiffEq
initial = [0.01, 0.01, 0.01, 0.01]
tspan = (0.,100.)

#Define the problem
function double_pendulum_hamiltonian(udot,u,p,t)
    α  = u[1]
    lα = u[2]
    β  = u[3]
    lβ = u[4]
    udot .=
    [2(lα-(1+cos(β))lβ)/(3-cos(2β)),
    -2sin(α) - sin(α+β),
    2(-(1+cos(β))lα + (3+2cos(β))lβ)/(3-cos(2β)),
    -sin(α+β) - 2sin(β)*(((lα-lβ)lβ)/(3-cos(2β))) + 2sin(2β)*((lα^2 - 2(1+cos(β))lα*lβ + (3+2cos(β))lβ^2)/(3-cos(2β))^2)]
end

#Pass to solvers
poincare = ODEProblem(double_pendulum_hamiltonian, initial, tspan)

sol = solve(poincare, Tsit5())

using Plots; gr()
plot(sol, vars=[(0,3),(0,4)], leg=false, plotdensity=10000)

plot(sol, vars=(3,4), leg=false)

f = (t) -> sol(t,idxs=4)

using Optim
opt = optimize(f,18.0,22.0)

println(opt.minimizer)
println(opt.minimum)

f = (t) -> -sol(first(t),idxs=4)
opt2 = optimize(f,0.0,22.0)

plot(sol, vars=(0,4), plotdensity=10000)
scatter!([opt.minimizer],[opt.minimum],label="Local Min")
scatter!([opt2.minimizer],[-opt2.minimum],label="Local Max")

f = (t) -> -sol(first(t),idxs=4)
opt = optimize(f,[20.0],BFGS())

import NLopt, ForwardDiff

count = 0 # keep track of # function evaluations

function g(t::Vector, grad::Vector)
  if length(grad) > 0
    #use ForwardDiff for the gradients
    grad[1] = ForwardDiff.derivative((t)->sol(first(t),idxs=4),t)
  end
  sol(first(t),idxs=4)
end
opt = NLopt.Opt(:GN_ORIG_DIRECT_L, 1)
NLopt.lower_bounds!(opt, [0.0])
NLopt.upper_bounds!(opt, [40.0])
NLopt.xtol_rel!(opt,1e-8)
NLopt.min_objective!(opt, g)
(minf,minx,ret) = NLopt.optimize(opt,[20.0])
println(minf," ",minx," ",ret)
NLopt.max_objective!(opt, g)
(maxf,maxx,ret) = NLopt.optimize(opt,[20.0])
println(maxf," ",maxx," ",ret)

plot(sol, vars=(0,4), plotdensity=10000)
scatter!([minx],[minf],label="Global Min")
scatter!([maxx],[maxf],label="Global Max")

using DiffEqTutorials
DiffEqTutorials.tutorial_footer(WEAVE_ARGS[:folder],WEAVE_ARGS[:file])