### A Pluto.jl notebook ###
# v0.14.2

using Markdown
using InteractiveUtils

# ╔═╡ 17ada06e-a49a-11eb-29f8-35891d4afe6c
begin
	using Random
	using Plots
	using Base.Threads
	using Distributions
	using DataFrames
end

# ╔═╡ d79fcb85-28ca-4e32-bc4d-8c442b771cd6
md"""
# Random Election Upsets Riddler

This notebook solves the [Riddler Classic problem from April 23rd 2021](https://fivethirtyeight.com/features/can-you-cut-the-perfect-pancake/), which asks us to consider a large nation of voters who independently submit their vote between two candidates (A and B) by flipping identical fair coins. We're asked: if 80% of votes are tallied on election day (regular votes), while the remaining 20% (early votes) are tallied later, what is the probability of an upset -- i.e. what is the probability that the election day outcome doesn't match the final outcome?

As usual, I'll solve this problem in two ways. First I'll compute the result using some gratuitous math, and then I'll run a simulation of the random process to cross-check my analytical results. Since this problem isn't substantially harder if we assume that 20% of votes are submitted early, or that an arbitrary fraction of votes are submitted early, I'll solve the prompt in general and then report the solution for the prompt as a special case.
"""

# ╔═╡ 7c4713bb-7e75-4c0b-815c-e817b71e2f8b
md"""
## Analytical Solution

### Notation
First let's introduce a bit of notation. Throughout, we will use the subscripts $E$ and $R$ to indicate quantities computed respectively for the early voter and regular voter subpopulations. Similarly, we will use a subscript $T$ to refer to the total population. We will use $v$ to denote the number of votes in favor of candidate A in a population and $n$ will denote the number of voters in a population. E.g. $v_E$ is the number of voters who voted for candidate A in the early voter subpopulation, and $n_E$ is the number of voters. Finally, we will use $f$ to denote the fracction of voters who belong to a subpopulation, e.g. $f_E = n_E / n_T$.

### General Solution
Since each voter places their vote by independently flipping a fair coin, the total number of votes for candidate A is distributed as $v_\text{T} \sim \text{Binomial}(n_T,~0.5)$, and this remains true when we sub-divide into regular and early voter sub-populations. That is,

$$\begin{array}{ccl}
v_R &\sim& \text{Binomial}(n_R,~0.5). \\
v_E &\sim& \text{Binomial}(n_E,~0.5), \\
\end{array}$$

The prompt tells us that the population is "large," which is a hint that we should  think about using the Normal approximation to the Binomial. To that end, we recall that if $m \sim \text{Binomial}(n, p)$, then $m$ is approximately $\text{Normal}(\mu_m,\sigma_m)$, where $\mu_m = E[m] = pn$, and $\sigma_m = \sqrt{np(1-p)}$. Hence,

$$\begin{array}{ccl}
v_R &\approx& \text{Normal}(\mu_R,\sigma_R). \\
v_E &\approx& \text{Normal}(\mu_E,\sigma_E), \\
\end{array}$$

where $\mu_R = 0.5n_R$, $\sigma_R = 0.5 \sqrt{n_R}$ and an analogous expressions hold for $\mu_E$ and $\sigma_E$.

An election upset will occur if candidate A wins the regular vote but loses the total vote, or if candidate A loses the regular vote and wins the total vote. Since both events occur with equal probability (since candidate A and B are interchangeable), it follows that

$$P(\text{upset}) = 2 \cdot P(\text{A loses the total vote} \cap \text{A wins the regular vote}).$$

We can then re-write this as,

$$\begin{eqnarray}
P(\text{upset}) &=& 2 \cdot P(\text{A wins the regular vote} \cap \text{A loses the total vote}) \\
&=& 2 \cdot P(v_R > 0.5 \cdot n_R \text{ and }  v_T < 0.5 \cdot n_T)\\
&=& 2 \cdot P(v_R > 0.5 \cdot n_R \text{ and } v_E + v_R < 0.5 \cdot (n_E + n_R)) \\
&=& 2 \cdot P(v_R > \mu_R \text{ and } v_E + v_R < \mu_E+\mu_R) \\
&=& 2 \cdot P(v_R > \mu_R \text{ and } v_E < \mu_E + \mu_R - v_R). \\
\end{eqnarray}$$

Here we have used the fact that candidate A wins a round of votes if $v > 0.5\cdot n$, and used the expressions for $\mu_R, \mu_E$ from above.

Next, we note that $v_E$ and $v_R$ are independent so their joint density is just the product of their respective densities. Using this obsevation and the fact that both variables are normally distributed we obtain:

$$P(\text{upset}) = 2 \cdot \int_{\mu_R}^\infty \,dv_R \int_{-\infty}^{\mu_R + \mu_E - v_R}\,dv_E
\frac{e^{-\frac{1}{2}(\frac{v_R-\mu_R}{\sigma_R})^2}}{\sigma_R\sqrt{2\pi}}
\frac{e^{-\frac{1}{2}(\frac{v_E-\mu_E}{\sigma_E})^2}}{\sigma_E\sqrt{2\pi}}.$$

We will now manipulate this expression so that it matches an expression that I managed to find in a table of integrals 🙂. To that end, we make the substitution $w_R = \frac{v_R-\mu_R}{\sigma_R\sqrt{2}}$, $w_E = \frac{v_E -\mu_E}{\sigma_E\sqrt{2}}$, which (after some book-keeping) yields:

$$P(\text{upset}) = 2 \cdot \int_{0}^\infty \,dw_R \frac{e^{-w_R^2}}{\sqrt{\pi}} \int_{-\infty}^{-\frac{\sigma_R}{\sigma_E}w_R}\,dw_E \frac{e^{-w_E^2}}{\sqrt{\pi}}$$

Next, we recall that $$\text{erfc}(x) = \frac{2}{\sqrt{\pi}} \int_x^\infty e^{-t^2}\,dt$$, and that $$\text{erf}(x) = 1 - \text{erfc}(x)$$ so the above can be written:

$$\begin{eqnarray}
P(\text{upset}) &=&\int_{0}^\infty \,dw_R \frac{e^{-w_R^2}}{\sqrt{\pi}} \left(1 - \text{erf}(\frac{\sigma_R}{\sigma_E}w_R)\right)\\
&=&\int_{0}^\infty \,dw_R \frac{e^{-w_R^2}}{\sqrt{\pi}} - \frac{1}{\sqrt{\pi}} \cdot \int_{0}^\infty\,dw_R~e^{-w_R^2} \cdot \text{erf}(\frac{\sigma_R}{\sigma_E}w_R)
\end{eqnarray}$$

The first integral is equal to $\frac{1}{2}\text{erfc}(0) = \frac{1}{2}$. For the second integral, we apply formula 2 from section 4.3 of [this table of integrals involving error functions](https://nvlpubs.nist.gov/nistpubs/jres/73b/jresv73bn1p1_a1b.pdf) with $a = \sigma_R/\sigma_E$ and $b = 1$, to obtain:

$$\begin{eqnarray}
P(\text{upset})&=&\frac{1}{2} - \frac{1}{\sqrt{\pi}} \left[\frac{\sqrt{\pi}}{2} - \frac{1}{\sqrt{\pi}}\tan^{-1}\left(\frac{\sigma_E}{\sigma_R}\right)\right]\\
&=&\frac{1}{\pi}\tan^{-1}\left(\frac{\sigma_E}{\sigma_R}\right)
\end{eqnarray}$$

Finally, since $\sigma_E = \sqrt{\frac{1}{2}f_E\cdot n_\text{total}}$, $\sigma_R = \sqrt{\frac{1}{2}f_R\cdot n_\text{total}}$, and $f_E + f_R = 1$, we conclude that:

$$\boxed{P(\text{upset})=\frac{1}{\pi}\tan^{-1}\left(\sqrt{\frac{f_E}{1-f_E}}\right)}$$

### Solution for the Prompt
In particular, for the case outlined in prompt with $f_E = 0.2$, we find:

$$\boxed{P\left(\text{upset}_\text{prompt}\right) = \frac{1}{\pi} \tan^{-1}\left(\frac{1}{2}\right) \approx 0.14758}$$

"""

# ╔═╡ 70ef11be-81b9-4e4d-b83a-f7a225a9001b
function exact_prob_upset(frac_early)
	return (1/π) * atan(sqrt(frac_early / (1 - frac_early)))
end

# ╔═╡ 24e2efe2-f6f2-4648-8c28-bffedf98095b
exact_prob_upset(0.2)

# ╔═╡ 2383b70b-0693-4adf-99d4-9bcedec5d6cb
md"""
## Simulation Solution

As discussed above, the number of votes for candidate A in the early and regular vote subpopulations are both Binomial random variables. So, we  simulate the election tally by simulating appropriate Binomial random variables. From the simulated results, we next determine if A wins on election night (i.e. if more than 50% of the regular vote is for A) and determine the overall winner (i.e. if more than 50% of the total vote is for A). An election upset results if A wins in either the regular or total tally, but loses in the other tally. We simulate this process 100K times and tally up the fraction of times that an election upset occurred.

This process is repeated for 1001 evenly spaced early voter share fractions in [0,1], and the result is then compared (graphically) against the analytical solution.
"""

# ╔═╡ 562b2637-ec4f-4be2-a525-09e83c6458fa
"""
    simulate_election(frac_early, n_simulations, [n_country, p_A])

	Inputs:
	frac_early - percentage of early voters, 20% in the problem prompt
	n_simulations - number of simulations to run

	Optional:
	n_country - Has to be big-enough™. defaults to 100_001 votes
	p_A - probability that a voter selects candidate A. defaults to 0.5

	Output:
	estimated probability of an upset to the election night results
"""
function simulate_election(frac_early, n_simulations, n_country = 100_001, p_A=0.5)

	# number of early and regular voters
	n_early = floor(Int, frac_early * n_country)
	n_regular = n_country - n_early

	# distributions of votes for A for early and regular voters
	d_early = Binomial(n_early, p_A)
	d_regular = Binomial(n_regular, p_A)

	# simulate votes for A for early, regular, and total
	n_votes_A_early = rand(d_early, n_simulations)
	n_votes_A_regular = rand(d_regular, n_simulations)
	n_votes_A = n_votes_A_early + n_votes_A_regular

	# fraction of vote for A between regular and all votes
	frac_votes_A_regular = n_votes_A_regular / n_regular
	frac_votes_A = n_votes_A / n_country

	# determine if A wins on election night and if A wins overall
	a_wins_regular = frac_votes_A_regular .> 0.5
	a_wins = frac_votes_A .> 0.5

	# an upset occurs if
	#  1. A wins  regular, but loses overall (i.e. True, False)
	#  2. A loses regular, but wins overall (i.e. False, True)
	upset = a_wins_regular .!= a_wins
	prob_upset = sum(upset) / n_simulations

	return prob_upset
end

# ╔═╡ 5f82ec16-38a7-4c44-aa5a-f76d18d521dd
begin
	n_sim = 100_000
	n_pop = 10_00_001
	frac_earlys = LinRange(0, 1, 1_001)

	prob_upset_sim = map(fe->simulate_election(fe, n_sim, n_pop), frac_earlys)
	prob_upset_exact = map(exact_prob_upset, frac_earlys)
end

# ╔═╡ b2346518-43b5-4c4d-9536-244514a4d5a3
plot(
	frac_earlys,
	hcat(prob_upset_exact, prob_upset_sim),
	label=["Exact" "Simulated"],
	legend=:bottomright,
	linewidth=[5 2],
	xlabel="Fraction of Early Voters",
	ylabel="Probability of Election Upset",
	title="Probability of Election Upset \nvs Fraction of Early Voters",
	size=(800,500)
)

# ╔═╡ 06cdba4f-426d-47e5-a4c0-c35f9a41d547
md"""
The above figure shows great agreement between the exact solution discussed above and the simulation results -- so I'd wager that my exact solution is probably correct! 🙂
"""

# ╔═╡ 4a8020c4-b9f4-4c22-8919-eb0f135b1725
savefig("election_upset_probability.png")

# ╔═╡ e1bfd335-b818-4931-8e71-03b46c90c9ec
md"""
### Simulating the Prompt Solution
Finally, we'll simulate the prompt problem 10 million times to get a bit more accuracy in our simulation.
"""

# ╔═╡ e55c8934-7268-4c5f-af6c-2098a48cc094
begin
	Random.seed!(2021)
	simulate_election(0.2, 100_000_000, n_pop)
end

# ╔═╡ 537e2e5d-378b-4676-8ba4-3a50f54eb84c
md"""
Setting the seed for reproducibility (but not selecting it to cherry-pick!), the simulation result agrees with the theoretical result to around four decimal places, which is pretty good!
"""

# ╔═╡ Cell order:
# ╟─d79fcb85-28ca-4e32-bc4d-8c442b771cd6
# ╟─7c4713bb-7e75-4c0b-815c-e817b71e2f8b
# ╠═70ef11be-81b9-4e4d-b83a-f7a225a9001b
# ╠═24e2efe2-f6f2-4648-8c28-bffedf98095b
# ╟─2383b70b-0693-4adf-99d4-9bcedec5d6cb
# ╠═562b2637-ec4f-4be2-a525-09e83c6458fa
# ╠═5f82ec16-38a7-4c44-aa5a-f76d18d521dd
# ╠═b2346518-43b5-4c4d-9536-244514a4d5a3
# ╟─06cdba4f-426d-47e5-a4c0-c35f9a41d547
# ╟─4a8020c4-b9f4-4c22-8919-eb0f135b1725
# ╟─e1bfd335-b818-4931-8e71-03b46c90c9ec
# ╠═e55c8934-7268-4c5f-af6c-2098a48cc094
# ╟─537e2e5d-378b-4676-8ba4-3a50f54eb84c
# ╠═17ada06e-a49a-11eb-29f8-35891d4afe6c


export project_l1_Duchi!
function project_l1_Duchi!(v::Union{Vector{TF},Vector{Complex{TF}}}, b::TF) where {TF<:Real}
# % PROJECTONTOL1BALL Projects point onto L1 ball of specified radius.
# %
# % w = ProjectOntoL1Ball(v, b) returns the vector w which is the solution
# %   to the following constrained minimization problem:
# %
# %    min   ||w - v||_2
# %    s.t.  ||w||_1 <= b.
# %
# %   That is, performs Euclidean projection of v to the 1-norm ball of radius
# %   b.
# %
# % Author: John Duchi (jduchi@cs.berkeley.edu)
# Translated (with some modification) to Julia 1.1 by Bas Peters
if (b <= TF(0))
  error("Radius of L1 ball is negative");
end
if norm(v, 1) <= b
  return v
end
lv=length(v)
u = Vector{TF}(undef,lv)
sv= Vector{TF}(undef,lv)

#u = sort(abs(v),'descend');
#u = sort(abs.(v), rev=true) #faster than the line below
#u = sort(v, by=abs , rev=true)

#use RadixSort for Float32 (short keywords)
u=copy(v)
if TF==Float32
 u = sort!(abs.(u), rev=true,alg=RadixSort)
else
 u = sort!(abs.(u), rev=true,alg=QuickSort)
end

#sv = cumsum(u);
sv = cumsum(u)

#rho = find(u > (sv - b) ./ (1:length(u))', 1, 'last');
#tmp = (u .> ((sv.-b)./ LinSpace(1,lv,lv)))#::BitVector

rho = max(1,min(lv,findlast(u .> ((sv.-b)./ (1.0:1.0:lv)))))
#convert(TF,rho) why was this here...
#rho = findlast(tmp)::Int64

#theta = max(0, (sv(rho) - b) / rho);
theta = max.(TF(0) , (sv[rho] .- b) ./ rho)::TF

#w = sign(v) .* max(abs(v) - theta, 0);
v .= sign.(v) .* max.(abs.(v) .- theta, TF(0))

return v
end #end project_l1_Duchi
#
# function project_l1_Duchi!{TF<:Real}(v::Union{Array{TF,3},Array{Complex{TF},3}}, b::TF,mode::String)
#
# if (b < TF(0))
#   error("Radius of L1 ball is negative");
# end
# (n1,n2,n3)=size(v)
#
# ##Slice based projection
# if mode == "x_slice"
#   u = Vector{TF}(n2*n3)
#   sv= Vector{TF}(n2*n3)
#   v = reshape(v,n1,n2*n3)
#   Threads.@threads for i=1:n1
#     u = sort(abs.(v[i,:]), rev=true)
#     sv = cumsum(u)
#     rho = findlast(u .> ((sv.-b)./ (1.0:1.0:(n2*n3))))
#     theta = max.(TF(0) , (sv[rho] .- b) ./ rho)::TF
#     @inbounds v[i,:] .= sign.(v[i,:]) .* max.(abs.(v[:,i]).-theta, TF(0))
#   end
#   v = reshape(v,n1,n2,n3)
# elseif mode == "z_slice"
#   u = Vector{TF}(n1*n2)
#   sv= Vector{TF}(n1*n2)
#   v = reshape(v,n1*n2,n3)
#   Threads.@threads for i=1:n3
#     u = sort(abs.(v[:,i]), rev=true)
#     sv = cumsum(u)
#     rho = findlast(u .> ((sv.-b)./ (1.0:1.0:(n1*n2))))
#     theta = max.(TF(0) , (sv[rho] .- b) ./ rho)::TF
#     @inbounds v[:,i] .= sign.(v[:,i]) .* max.(abs.(v[:,i]).-theta, TF(0))
#   end
#   v = reshape(v,n1,n2,n3)
# elseif mode == "y_slice"
#   permutedims(v,[2;1;3]);
#   v = reshape(v,n2,n1*n3)
#   u = Vector{TF}(n1*n3)
#   sv= Vector{TF}(n1*n3)
#   Threads.@threads for i=1:size(v,1)
#     u = sort(abs.(v[i,:]), rev=true)
#     sv = cumsum(u)
#     rho = findlast(u .> ((sv.-b)./ (1.0:1.0:(n1*n3))))
#     theta = max.(TF(0) , (sv[rho] .- b) ./ rho)::TF
#     @inbounds v[i,:] .= sign.(v[i,:]) .* max.(abs.(v[:,i]).-theta, TF(0))
#   end
#   v = reshape(v,n2,n1,n3);
#   permutedims(v,[2;1;3]);
# else
#   error("provided incorred mode for l1_projection")
# end #end if
# v=vec(v)
#
# return v
# end #end project_l1_Duchi


include("../../common/numder/src/numder_module.jl")
# include("testfunctions.jl")

module opti
using ..numder
using LinearAlgebra

include("opti_1d.jl")
include("line_search.jl")
include("gradient_descent.jl")

mutable struct BFGS <: DescentMethod
    B
end
function BFGS(x::Vector{T}) where {T}
    BFGS( Matrix(one(T)*I, length(x), length(x)) )
end

function init!(m::BFGS, x)
    m.B = Matrix(one(eltype(x))*I, length(x), length(x))
end

function bfgs_matrix_update(B, g, g_prev, x, x_prev)
    dg = g - g_prev
    dx = x - x_prev

    dxdg = dot(dx, dg)

    if dxdg < 0
        println("No update happens, some accuracy errors")
        println(" dxdg = $(dxdg) ")
        return B
    end

    t1 = (dx ./ dxdg) * (dg' * B)
    t1 = t1 + t1'

    t2f = 1 + (dg' * (B * dg)) / dxdg
    t2 = (dx .* (t2f / dxdg)) * dx'

    # B = B - t1 + t2
    B += (t2 - t1)
    return B
end

function quasi_newton(
    fcn,
    x0;
    tol_options::ToleranceOptions       = ToleranceOptions(),
    search_options::LineSearchOptions   = LineSearchOptions(),
    gradient_fcn                        = numder.gradient_fw,
    line_search_fcn                     = line_search,
    log_path                            = false)

    m = BFGS(x0)

    x = x0
    f = fcn(x)
    g = gradient_fcn(fcn, x)

    step_direction  = -m.B * g
    step_size       = line_search_fcn(fcn, x, step_direction, f, search_options)
    dx              = step_size * step_direction
    g_new           = gradient_fcn(fcn, x + dx)
    dg              = g_new - g
    m.B             = m.B * dot(dg, dx) / dot(dg, dg)

    x_prev  = x
    x       = x_prev + dx
    f_prev  = f
    f       = fcn(x)
    g_prev  = g
    g       = g_new

    # out parameters
    x_sol = x
    y_sol = f
    stopping_crit = unknown::GradientDescentStoppingCrit

    iter = 0
    for outer iter = 1:tol_options.max_iter

        # update B
        # dg = g - g_prev
        # dx = x - x_prev

        # dxdg = dot(dx, dg)

        # t1 = (dx ./ dxdg) * (dg' * m.B)
        # t1 = t1 + t1'

        # t2f = 1 + (dg' * (m.B * dg)) / dxdg
        # t2 = (dx .* (t2f / dxdg)) * dx'

        # m.B = m.B - t1 + t2

        m.B = bfgs_matrix_update(m.B, g, g_prev, x, x_prev)

        # step dir
        step_direction = - m.B * g
        step_size      = line_search_fcn(fcn, x, step_direction, f, search_options)

        dx = step_direction * step_size

        # stopping criterion based on the step size
        abs_step_size = norm( dx )
        if abs_step_size <= tol_options.step_size_tol
            x_sol = x
            y_sol = f
            stopping_crit = small_step_size::GradientDescentStoppingCrit
            break
        end

        x_prev = x
        x      = x_prev + dx
        f_prev = f
        f      = fcn(x)
        g_prev = g

        # stopping criteria for function value
        if abs(f - f_prev) <= tol_options.f_abs_tol || abs(f - f_prev) / ( abs(f_prev) == 0 ? mach_eps : abs(f_prev)) < tol_options.f_rel_tol
            x_sol = x
            y_sol = f
            stopping_crit = small_function_value_delta::GradientDescentStoppingCrit
            break
        end

        # calculate the gradient at the new point
        g = gradient_fcn(fcn, x)

        # stopping criterion based on the gradient norm
        norm_g = norm(g)
        if norm_g <= tol_options.g_abs_tol
            x_sol = m.x
            y_sol = m.f
            stopping_crit = small_gradient::GradientDescentStoppingCrit
            break
        end

    end

    return (x_sol, y_sol, iter, stopping_crit)

end

function bfgs(
	fcn,
	x0;
	grad_fcn = numder.gradient_fw,
	options = ToleranceOptions() )

    n_dim = length(x0)
    H     = Matrix{Float64}(I, n_dim, n_dim)
    eye   = Matrix{Float64}(I, n_dim, n_dim)

    # first step is the steepest gradient descent
    g0        = grad_fcn(fcn, x0)
    step_size = line_search(fcn, x0, -g0)
    x1        = x0 - step_size * g0
    g1        = grad_fcn(fcn, x1)

    y            = g1 - g0
    s            = -step_size * g0
    scale_factor = dot(y, s) / dot(y, y)
    for ii = 1:n_dim
        H[ii, ii] *= scale_factor
    end

    # main loop
    iter = 0
    for outer iter = 1:options.max_iter

        # update of the matrix
        s       = x1 - x0
        y       = g1 - g0

		sdy     = dot(s, y)
		Hs 		= H * s
		sHs     = dot(s, Hs)
		theta   = 1.0
		if sdy < 0.2 * sHs
			theta = 0.8 * sHs / (sHs - sdy)
		end

		r = theta * y + (1 - theta) * Hs

        Hy      = H * y
		rho     = 1 / sdy
        rhos    = rho * s
        Hyst    = Hy * rhos'
        sytHyst = rhos * (y' * Hyst)
        # sst     = rhos * s'
        sst 	= r * r' / dot(s, r)

		Hnew    = H - (Hyst + Hyst') + sytHyst + sst
        # Hnew   = (eye - rho * s * y') * H * (eye - rho * y * s') + rho * s * s'

        # dH = norm(Htest - Hnew)
        # println(dH)

        x0        = x1
        p         = -(Hnew * g1)
        step_size = line_search(fcn, x0, p)
        dx        = p * step_size
        x1        = x0 + dx

        if norm(dx) <= options.step_size_tol
            break
        end

        # options.f_abs_tol
        # options.f_rel_tol

        g0        = g1
        g1        = grad_fcn(fcn, x1)

        if norm(g1) <= options.g_abs_tol
            break
        end

        H = Hnew
    end

    return (x1, iter)
end

end # opti

#=

function nonlinear_conjugate_gradient_descent_template(beta_method, fcn, x0, options, wolfe_condition_options)
	x_act = x0
	y_act = fcn(x_act)
	x_next = x_act
	x_sol = x_act

	# first step is a gradient descent step
	g_act = NumDiff.gradient_fw(fcn, x_act)
	direction_act = -g_act
	alpha = strong_wolfe_line_search(fcn, x_act, direction_act,
	 								wolfe_condition_options.alpha_0,
									wolfe_condition_options.beta,
									wolfe_condition_options.sigma)
	step = alpha * direction_act
	x_act = x_act + step
	y_act = fcn(x_act)
	g_prev = g_act
	direction_prev = direction_act

	# main loop
	println("x: $x_act")
	iter = 1
	for outer iter = 1:options.max_iter

		g_act = NumDiff.gradient_fw(fcn, x_act)

		## specific method is applied
		beta = beta_method(g_act, g_prev, direction_prev)

		direction_act = -g_act + beta .* direction_prev

		alpha = strong_wolfe_line_search(fcn, x_act, direction_act)
		step = alpha * direction_act
		x_next = x_act + step
		y_next = fcn(x_next)

		# convergence checks
		if norm(g_act) <= options.gradient_tolerance
			x_sol = x_next
			break
		end
		if norm(step) <= options.step_tolerance
			x_sol = x_next
			break
		end
		if abs(y_next - y_act) <= options.function_value_change_tolerance
			x_sol = x_next
			break
		end

		# updates
		y_act = y_next
		x_act = x_next
		direction_prev = direction_act
		println("x: $x_act")
	end

	if iter == options.max_iter
		x_sol = x_next
	end
	return x_sol
end

function beta_polak_ribiere(g_act, g_prev, direction_prev)
	y_hat = g_act - g_prev
	beta = dot(g_act, y_hat) / dot( g_prev, g_prev )
	beta = max(beta, 0)
	return beta
end

function beta_dai_yuan(g_act, g_prev, direction_prev)
	y_hat = g_act - g_prev
	beta = dot(g_act, g_act) / dot(y_hat, direction_prev)
	beta = max(beta, 0)
	return beta
end

function beta_hestenes_stiefel(g_act, g_prev, direction_prev)
	y_hat = g_act - g_prev
	beta = dot(y_hat, g_act) / dot(y_hat, direction_prev)
	beta = max(beta, 0)
	return beta
end

function beta_hager_zhang(g_act, g_prev, direction_prev)
	# probably implementation error
	y_hat = g_act - g_prev
	numer = dot(y_hat, direction_prev)
	t1 = y_hat - 2 .* direction_prev .* dot(y_hat, y_hat) ./ numer
	t2 = g_act ./ numer
	beta = dot(t1, t2)
	# beta = max(beta, 0)
	return beta
end

function nonlinear_conjugate_gradient_descent_polak_ribiere(
	fcn,
	x0,
	options::GradientDescentOptions = default_options(GradientDescentOptions),
	wolfe_condition_options = default_strong_wolfe_conditions_options()
	)
	x_sol = nonlinear_conjugate_gradient_descent_template(beta_polak_ribiere, fcn, x0, options, wolfe_condition_options)
end

function nonlinear_conjugate_gradient_descent_dai_yuan(
	fcn,
	x0,
	options::GradientDescentOptions = default_options(GradientDescentOptions),
	wolfe_condition_options = default_strong_wolfe_conditions_options()
	)
	x_sol = nonlinear_conjugate_gradient_descent_template(beta_dai_yuan, fcn, x0, options, wolfe_condition_options)
end

function nonlinear_conjugate_gradient_descent_hestenes_stiefel(
	fcn,
	x0,
	options::GradientDescentOptions = default_options(GradientDescentOptions),
	wolfe_condition_options = default_strong_wolfe_conditions_options()
	)
	x_sol = nonlinear_conjugate_gradient_descent_template(beta_hestenes_stiefel, fcn, x0, options, wolfe_condition_options)
end

function nonlinear_conjugate_gradient_descent_hager_zhang(
	fcn,
	x0,
	options::GradientDescentOptions = default_options(GradientDescentOptions),
	wolfe_condition_options = default_strong_wolfe_conditions_options()
	)
	x_sol = nonlinear_conjugate_gradient_descent_template(beta_hager_zhang, fcn, x0, options, wolfe_condition_options)
end

=#


using AdFem
using PyPlot
using SparseArrays

function f1func(x,y)
    18.8495559215388*pi^2*sin(pi*x)^2*sin(pi*y)*cos(pi*y) - 6.28318530717959*pi^2*sin(pi*y)*cos(pi*x)^2*cos(pi*y) + pi*sin(pi*y)*cos(pi*x)
end
function f2func(x,y)
    -18.8495559215388*pi^2*sin(pi*x)*sin(pi*y)^2*cos(pi*x) + 6.28318530717959*pi^2*sin(pi*x)*cos(pi*x)*cos(pi*y)^2 + pi*sin(pi*x)*cos(pi*y)
end

m = 50
n = 50
h = 1/n
ν = 0.5
K = ν*constant(compute_fem_laplace_matrix(m, n, h))
B = constant(compute_interaction_matrix(m, n, h))
Z = [K -B'
-B spzero(size(B,1))]

bd = bcnode("all", m, n, h)
# Due to the rank insufficiency of bilinear quadrilateral element,
# we add more pressure constraints
bd = [bd; bd .+ (m+1)*(n+1); ((1:m) .+ 2(m+1)*(n+1))]

F1 = eval_f_on_gauss_pts(f1func, m, n, h)
F2 = eval_f_on_gauss_pts(f2func, m, n, h)
F = compute_fem_source_term(F1, F2, m, n, h)
rhs = [F;zeros(m*n)]
Z, rhs = impose_Dirichlet_boundary_conditions(Z, rhs, bd, zeros(length(bd)))
sol = Z\rhs

sess = Session(); init(sess)
S = run(sess, sol)

xy = fem_nodes(m, n, h)
x, y = xy[:,1], xy[:,2]
U = @. 2*pi*sin(pi*x)*sin(pi*x)*cos(pi*y)*sin(pi*y)
figure(figsize=(12,5))
subplot(121)
visualize_scalar_on_fem_points(U, m, n, h)
title("Reference")
subplot(122)
visualize_scalar_on_fem_points(S[1:(m+1)*(n+1)], m, n, h)
title("Computed")
savefig("stokes1.png")

U = @. -2*pi*sin(pi*x)*sin(pi*y)*cos(pi*x)*sin(pi*y)
figure(figsize=(12,5))
subplot(121)
visualize_scalar_on_fem_points(U, m, n, h)
title("Reference")
subplot(122)
visualize_scalar_on_fem_points(S[(m+1)*(n+1)+1:2(m+1)*(n+1)], m, n, h)
title("Computed")
savefig("stokes2.png")

xy = fvm_nodes(m, n, h)
x, y = xy[:,1], xy[:,2]
p = @. sin(pi*x)*sin(pi*y)
figure(figsize=(12,5))
subplot(121)
visualize_scalar_on_fvm_points(p, m, n, h)
title("Reference")
subplot(122)
visualize_scalar_on_fvm_points(S[2(m+1)*(n+1)+1:end], m, n, h)
title("Computed")
savefig("stokes3.png")


module problem122

description = """
Efficient exponentiation

The most naive way of computing n15 requires fourteen multiplications:
n × n × ... × n = n15

But using a "binary" method you can compute it in six multiplications:

n × n = n^2
n^2 × n^2 = n^4
n^4 × n^4 = n^8
n^8 × n^4 = n^12
n^12 × n^2 = n^14
n^14 × n = n^15

However it is yet possible to compute it in only five multiplications:

n × n = n^2
n^2 × n = n^3
n^3 × n^3 = n^6
n^6 × n^6 = n^12
n^12 × n^3 = n^15

We shall define m(k) to be the minimum number of multiplications to compute nk; for example m(15) = 5.
For 1 ≤ k ≤ 200, find ∑ m(k).
"""

# Calculates the minimum number of multiplications to compute n^k
# given the known powers, and a maximum cutoff
function muls(k :: Int, known = [1], maxp = k-1)
  if (k % 2) == 0
    return muls(div(k,2)) + 1 # for even powers, solution is always to square the half power

  elseif known[end] == k
    return length(known)-1  # solution found

  elseif length(known) >= maxp # no solution for given known path
    return -1

  elseif (maxp - length(known)) < floor(log2(k / known[end])) # no possible solution for given knowns
    return -1
  end

  # search for solutions by summing items in the knowns path
  for i = length(known):-1:1
    for j = i:-1:1
      n = known[i] + known[j]
      if n <= k && n > known[end]
        push!(known, n)
        result = muls(k, known, maxp)
        pop!(known)
        if result == -1
          break
        elseif result == length(known)
          return result
        elseif result < maxp
          maxp = result
        end
      end
    end
  end

  return maxp
end

function solve(maxk = 200)
  total = 0
  for i = 1:maxk
    total += muls(i)
  end
  return total
end

end


using NLopt, Test

count = 0 # keep track of # function evaluations

function myfunc(x::Vector, grad::Vector)
    if length(grad) > 0
        grad[1] = 0
        grad[2] = 0.5/sqrt(x[2])
    end

    global count
    count::Int += 1
    println("f_$count($x)")

    sqrt(x[2])
end

function myconstraint(x::Vector, grad::Vector, a, b)
    if length(grad) > 0
        grad[1] = 3a * (a*x[1] + b)^2
        grad[2] = -1
    end
    (a*x[1] + b)^3 - x[2]
end

opt = Opt(:LD_MMA, 2)
opt.lower_bounds = [-Inf, 0.]
opt.xtol_rel = 1e-4
opt.min_objective = myfunc
opt.inequality_constraint = (x,g) -> myconstraint(x,g,2,0)
opt.inequality_constraint = (x,g) -> myconstraint(x,g,-1,1)

(minf,minx,ret) = optimize(opt, [1.234, 5.678])
println("got $minf at $minx after $count iterations (returned $ret)")

@test minx[1] ≈ 1/3 rtol=1e-5
@test minx[2] ≈ 8/27 rtol=1e-5
@test minf ≈ sqrt(8/27) rtol=1e-5
@test ret == :XTOL_REACHED
@test opt.numevals == count


function ABCDQR_kfilter(y,u,A,B,C,D,Q,R,x10,P10)
	#
	# Kalman filter for model
	#
	# x_{t+1} = A*x_{t} + B*u_{t} + w_{t}
	# y_{t}   = C*x_{t} + D*u_{t} + v_{t}
	#
	# cov(w_{t},v_{t}) = [Q 0;0 R]
	#
	# javier.cara@upm.es, 2015-10
	#

	(ny,nt) = size(y)
	nx = size(A,1)

	# allocation
	xtt = zeros(nx,nt)
	Ptt = zeros(nx,nx,nt)
	xtt1 = zeros(nx,nt+1)
	Ptt1 = zeros(nx,nx,nt+1)
	et = zeros(ny,nt)
	St = zeros(ny,ny,nt)
	Kt = zeros(nx,ny,nt)
	loglik = 0.0

	# Filter
	xtt1[:,1] = x10
	Ptt1[:,:,1] = P10
	for t in 1:nt
		#  innovations
		et[:,t] = y[:,t] - C*xtt1[:,t] - D*u[:,t]
		St[:,:,t] = C*Ptt1[:,:,t]*C' + R # et variance

		# Kalman gain
		Stinv = eye(ny,ny)/St[:,:,t] # numerically preferible to Stinv=inv(St)
		Kt[:,:,t] = Ptt1[:,:,t]*C'*Stinv # kalman gain

		# filtered values
		xtt[:,t] = xtt1[:,t] + Kt[:,:,t]*et[:,t]
		Ptt[:,:,t] = (eye(nx) - Kt[:,:,t]*C)*Ptt1[:,:,t]

		# one-step ahead prediction
		xtt1[:,t+1] = A*xtt[:,t] + B*u[:,t]
		Ptt1[:,:,t+1] = A*Ptt[:,:,t]*A' + Q

		# likelihood
		l0 = et[:,t]'*Stinv*et[:,t] # typeof(l0) = Array{Float64,1}
		loglik = loglik + log(det(St[:,:,t])) + l0[1]
	end

	loglik =  - ny*nt/2*log(2*pi) - 0.5*loglik

	return xtt,Ptt,xtt1,Ptt1,et,St,Kt,loglik

end


@doc raw"""
    PeriodicKernel([α=1 [,p=π]])

The periodic kernel is given by:

```math
\kappa(\mathbf{x},\mathbf{y}) =
\exp\left(-\alpha \sum_{i=1}^n \sin(p(x_i - y_i))^2\right)
\qquad p >0, \; \alpha > 0
```

where ``\mathbf{x}`` and ``\mathbf{y}`` are ``n`` dimensional vectors. The parameters ``p``
and ``\alpha`` are scaling parameters for the periodicity and the magnitude, respectively.
This kernel is useful when data has periodicity to it.
"""
struct PeriodicKernel{T<:AbstractFloat} <: MercerKernel{T}
    alpha::HyperParameter{T}
    PeriodicKernel{T}(α::Real) where {T<:AbstractFloat} = new{T}(
        HyperParameter(convert(T,α), interval(OpenBound(zero(T)), nothing))
    )
end
PeriodicKernel(α::T1 = 1.0) where {T1<:Real} = PeriodicKernel{floattype(T1)}(α)

@inline basefunction(::PeriodicKernel) = SineSquared()
@inline kappa(κ::PeriodicKernel{T}, z::T) where {T} = squaredexponentialkernel(z, getvalue(κ.alpha))


"""
Example modelling a linear beam element

Formulation:

K=-E*A*[dw_i!dx*du_j!dx]
"""

# local coordinates:
local_interp=InterpolationFunction(
    :(a0+a1*xi), # interpolation function expression
    [:xi], # local coordinate system degrees of freedom
    [:a0, :a1], # linear coefficients for regression
    [
        [0] # conditions to impose with regression: absolute value (order 0 derivative in the first axis)
    ];
    maximum_derivative=1
)

# absolute coordinates
abs_interp=InterpolationFunction(
    :(a0+a1*x), # interpolation function expression
    [:x], # absolute coordinate system degrees of freedom
    [:a0, :a1], # linear coefficients for regression
    [
        [0] # conditions to impose with regression: absolute value (order 0 derivative)
    ];
    maximum_derivative=1
)

# defining an integration domain:
dmn=Domain(
    local_interp, # local coordinate system
    [
        [0.0, 1.0] # boundaries for local coordinates: xi between 0 and 1
    ],
    [
        [0.0], # positions at which to impose values for interpolation:
        [1.0] # xi=0, xi=1
    ]
)

# defining a residual for within the beam:
domain_resd=Residual(
    [
        ( # variable w (weight), interpolated by InterpolationFunction abs_interp,
            :w, abs_interp, [
                (:dw!dx, [1]) # accompanied by its first order derivative
            ]
        )
        ( # variable u (displacement), interpolated by InterpolationFunction abs_interp,
            :u, abs_interp, [
                (:du!dx, [1]) # accompanied by its first order derivative
            ]
        )
    ],
    :(-E*(dw!dx*du!dx)),
    3; # Gauss-Legendre quadrature order
    extra_args=[:E, :A] # additional argument: elasticity
)

dmn_resd_func=get_volume_residual_function(domain_resd)

# defining a specific finite element:
fin=Finel(
    [
        [0.0],
        [1.0] # defined by corner points at x=0 and x=1 (corresponding to xi=0 and xi=1 in local coordinates: see Domain)
    ],
    dmn
)

# obtaining elasticity matrix:
K=zeros(Float64, 2, 2)

ws=zeros(Float64, 2)
us=zeros(Float64, 2)

E=1.0
A=1.0

for i=1:2
    ws[i]=1.0

    for j=1:2
        us[j]=1.0

        variable_vals=[ws, us]
        K[i, j]=dmn_resd_func(variable_vals, fin, E, A)

        us[j]=0.0
    end

    ws[i]=0.0
end

@assert isapprox(
    K,
    [
        -1.0 1.0;
        1.0 -1.0
    ]
)


using HyperGraphs
using LinearAlgebra: Diagonal

## matrices ##

# incidence matrix of undirected graph from https://en.wikipedia.org/wiki/Incidence_matrix
es = [HyperEdge([1, 2]), HyperEdge([1, 3]), HyperEdge([1, 4]), HyperEdge([3, 4])]
@test incidence_matrix(HyperGraph(es)) == [1 1 1 0; 1 0 0 0; 0 1 0 1; 0 0 1 1]

# incidence matrix and properties of undirected hypergraph from Fig. 1 of Konstantinova, E. V., & Skorobogatov, V. A. (2001). Application of hypergraph theory in chemistry. Discrete Mathematics, 235(1–3), 365–383.
x = HyperGraph([HyperEdge([1]), HyperEdge([2, 3]), HyperEdge([1, 2, 4])])
@test incidence_matrix(x) == [1 0 1; 0 1 1; 0 1 0; 0 0 1]
@test cardinalities(x) == [1, 2, 3]
@test degree(x, 2) == 2
@test degree(x, 4) == 1

# example from: Chen, G., Liu, V., Robinson, E., Rusnak, L. J., & Wang, K. (2018). A characterization of oriented hypergraphic Laplacian and adjacency matrix coefficients. Linear Algebra and Its Applications, 556, 323–341.
srcs = [[2], [1, 2], Int[]]; tgts = [[1], [2, 3], [3, 3]]
chen2018_es = [ChemicalHyperEdge(src, tgt) for (src, tgt) in zip(srcs, tgts)]
chen2018_x = ChemicalHyperGraph([1, 2, 3], chen2018_es)
ref_degree = [2; 3; 3]
ref_incidence = [1 -1 0; -1 0 0; 0 1 2]
ref_adjacency = [0 1 1; 1 2 0; 1 0 -2]
@test ref_incidence * transpose(ref_incidence) == Diagonal(ref_degree) - ref_adjacency # II^T = D - A
@test degrees(chen2018_x) == ref_degree
@test degree_matrix(chen2018_x) == Diagonal(ref_degree)
@test incidence_matrix(chen2018_x) == ref_incidence

# undirected graph from https://en.wikipedia.org/wiki/Laplacian_matrix
wikipedia_edges = HyperEdge.([[1, 2], [2, 3], [3, 4], [4, 5], [5, 1], [5, 2], [6, 4]])
wikipedia_graph = HyperGraph(wikipedia_edges)
@test degree_matrix(wikipedia_graph) == Diagonal([2; 3; 2; 3; 3; 1])

## loops ##

# NOTES
# loops should always have degree 2

# unoriented loop
unoriented_loop = HyperGraph(HyperEdge([1, 1]))
@test incidence_matrix(unoriented_loop) == 2*ones(Int, 1, 1)
@test degrees(unoriented_loop) == [2]

# oriented loops
positive_loop = ChemicalHyperGraph(ChemicalHyperEdge([1], [1]))
@test incidence_matrix(positive_loop) == zeros(Int, 1, 1)
@test degrees(positive_loop) == [2]
negative_loop = ChemicalHyperGraph(ChemicalHyperEdge(Int[], [1, 1]))
@test incidence_matrix(negative_loop) == 2*ones(Int, 1, 1)
@test degrees(negative_loop) == [2]


using LinearAlgebra
using SparseArrays
using Plots

"This code solves Laplace's equation with Dirichlet boundary conditions using a
finite difference method."

m = 100 # number of points
x = LinRange(-1,1,m+2)
xint = x[2:end-1]

f(x) = 5*(Float64((x > -.5) && (x <= 0.0)) - Float64((x < .5) && (x >= 0.0)))
α,β = 1.0,pi

h = x[2]-x[1]
A = (1/h^2) * spdiagm(0=>2*ones(m),-1=>-ones(m-1),1=>-ones(m-1))
b = f.(xint)
b[1] += α/h^2
b[m] += β/h^2

# solve Au = F
uu = A\b
plot(xint,uu,linewidth=2,label="Solution")
plot!(x,uexact.(x),linestyle=:dash)
# plot!(x,.1*f.(x),linecolor=:red,ls=:dash,label="f(x)")
# plot!(leg=:topleft)

# uu - uexact.(xint)


# functions for scaling features

"""
    standardization!(data::Matrix)

In-place standardization of the features in the matrix `data`.
"""
function standardization!(data::Matrix)

  # find dim. for data
  dim = minimum(size(data))

  # standardize data
  if size(data)[1] > size(data)[2]
    for i = 1:dim
        data[:,i] = (data[:,i] - mean(data[:,i]))/std(data[:,i])
    end
  else
    for i = 1:dim
        data[i,:] = (data[i,:] - mean(data[i,:]))/std(data[i,:])
    end
  end

end

"""
    standardization(data::Matrix, return_column_major_order::Bool=true)

Standardization of the features in the matrix `data`.
"""
function standardization(data::Matrix, return_column_major_order::Bool=true)

  # find dim. for data
  dim = minimum(size(data))

  # transform data to column-major order if necessary
  if size(data)[1] > size(data)[2]
      data = data'
  end
  data_standardized = copy(data)

  # standadize data
  for i = 1:dim
      data_standardized[i,:] = (data[i,:] - mean(data[i,:]))/std(data[i,:])
  end

  # return data in column major order
  if !return_column_major_order
      data_standardized = data_standardized'
  end

  return data_standardized

end

"""
    standardization!(data::Vector)

Standardization of the vector `data`.
"""
function standardization(data::Vector)
  data_standardized = (data - mean(data))/std(data)
  return data_standardized
end

"""
    standardization!(data::Vector)

In-place standardization of the vector `data`.
"""
function standardization!(data::Vector)
  data[:] = (data - mean(data))/std(data)
end


# Headcount.jl

###### headcount #####
"""
    headcount(v, z)

Compute the Headcount Ratio of a vector `v` at a specified poverty threshold `z`.

# Examples
```julia
julia> using Inequality
julia> headcount([8, 5, 1, 3, 5, 6, 7, 6, 3], 4)
0.3333333333333333
```
"""
headcount(v::AbstractVector{<:Real}, z::Real)::Float64 = length(v[v .< z]) / length(v)

###### weighted headcount #####
"""
    headcount(v, w, z)

Compute the Headcount Ratio of a vector `v` at a specified poverty threshold `z`,
using weights given by a weight vector `w`.

Weights must not be negative, missing or NaN. The weights and data vectors must have the same length.

# Examples
```julia
julia> using Inequality
julia> headcount([8, 5, 1, 3, 5, 6, 7, 6, 3], [0.1,0.5,0.3,0.8,0.1,0.5,0.3,0.8,0.2], 4)
0.36111111111111116
```
"""
function headcount(v::AbstractVector{<:Real}, w::AbstractVector{<:Real}, z::Real)::Float64

    checks_weights(v, w)

    return sum(w[v .< z]) / sum(w)
end

function headcount(v::AbstractVector{<:Real}, w::AbstractWeights, z::Real)::Float64

    checks_weights(v, w)

    return sum(w[v .< z]) / w.sum

end

wheadcount(v::AbstractVector{<:Real}, w::AbstractVector{<:Real}, z::Real)::Float64 = headcount(v, w, z)

wheadcount(v::AbstractVector{<:Real}, w::AbstractWeights, z::Real)::Float64 = headcount(v, w, z)


function vandermonde_1d(N, r)
    V = zeros(length(r), N+1)
    for j in 1:N+1
        @. V[:, j] = jacobi(r, 0, 0, j-1)
    end
    return V
end

" Initialize the gradient of the modal basis (i) at (r) at order N. "
function ∂vandermonde_1d(N, r)
    DVr = zeros(length(r), N+1)
    for i in 0:N
        @. DVr[:, i+1] = ∂jacobi(r, 0, 0, i)
    end
    return DVr
end

" Initialize the (r) differentiation matrices on the interval, evaluated at (r) at order N. "
function Dmatrix1D(N, r, V)
    Vr = ∂vandermonde_1d(N, r)
    Dr = Vr / V
    return Dr
end


"""
    TF(r, Z)

Approximate solution of the [Thomas-Fermi equation](https://en.wikipedia.org/wiki/Thomas-Fermi_model)
for neutral atom with nuclear charge `Z`. Returns density at the distance `r`.

See Moliere, G. (1947). Theorie der streuung schneller geladener teilchen i. einzelstreuung am abgeschirmten coulomb-feld.
Zeitschrift für Naturforschung A, 2(3), 133-145.

## Example
For a density function ``\\rho(r)`` of ``N`` particles the following equation holds.
```math
N = 4\\pi\\int dr \\rho(r)r^2
```
On the grid ``r_i = \\exp(x_i)`` it is
```math
N = 4\\pi\\int dx \\rho(r(x))r^3
```
```jldoctest
julia> x = -30:0.1:20
-30.0:0.1:20.0

julia> r, n, dx = exp.(x), length(x), step(x);

julia> isapprox(4π * dx * sum(TF.(r, 18) .* r .^3), 18, atol=1e-10)
true
```
"""
function TF(r, Z)
  b = 1/2*(3π/4)^(2/3)*Z^(-1/3)
  x = r/b

  B₁, B₂, B₃ = 0.1, 0.55, 0.35
  β₁, β₂, β₃ = 6.0, 1.20, 0.30

  Z/(4π*b^3*x)*(B₁*β₁^2*exp(-β₁*x) + B₂*β₂^2*exp(-β₂*x) + B₃*β₃^2*exp(-β₃*x))
end


#' ---
#' title: Sudoku
#' ---

#' **Originally Contributed by**: Iain Dunning

#'
#'
A partially solved Sudoku puzzle

#'
#' [Sudoku](http://en.wikipedia.org/wiki/Sudoku) is a popular number puzzle. The goal is to place the digits 1,...,9 on a nine-by-nine grid, with some of the digits already filled in. Your solution must satisfy the following rules:
#'
#' * The numbers 1 to 9 must appear in each 3x3 square
#' * The numbers 1 to 9 must appear in each row
#' * The numbers 1 to 9 must appear in each column
#'
#' This isn't an optimization problem, its actually a *feasibility* problem: we wish to find a feasible solution that satsifies these rules. You can think of it as an optimization problem with an objective of 0.
#'
#' We can model this problem using 0-1 integer programming: a problem where all the decision variables are binary. We'll use JuMP to create the model, and then we can solve it with any integer programming solver.

using JuMP
using GLPK

#' We will define a binary variable (a variable that is either 0 or 1) for each possible number in each possible cell. The meaning of each variable is as follows:
#'
#'     x[i,j,k] = 1  if and only if cell (i,j) has number k
#'
#' where `i` is the row and `j` is the column.

# Create a model
sudoku = Model(with_optimizer(GLPK.Optimizer))

# Create our variables
@variable(sudoku, x[i=1:9, j=1:9, k=1:9], Bin);

#' Now we can begin to add our constraints. We'll actually start with something obvious to us as humans, but what we need to enforce: that there can be only one number per cell.

for i = 1:9, j = 1:9  # Each row and each column
    # Sum across all the possible digits
    # One and only one of the digits can be in this cell,
    # so the sum must be equal to one
    @constraint(sudoku, sum(x[i,j,k] for k in 1:9) == 1)
end

#' Next we'll add the constraints for the rows and the columns. These constraints are all very similar, so much so that we can actually add them at the same time.

for ind = 1:9  # Each row, OR each column
    for k = 1:9  # Each digit
        # Sum across columns (j) - row constraint
        @constraint(sudoku, sum(x[ind,j,k] for j in 1:9) == 1)
        # Sum across rows (i) - column constraint
            @constraint(sudoku, sum(x[i,ind,k] for i in 1:9) == 1)
    end
end

#' Finally, we have the to enforce the constraint that each digit appears once in each of the nine 3x3 sub-grids. Our strategy will be to index over the top-left corners of each 3x3 square with `for` loops, then sum over the squares.

for i = 1:3:7, j = 1:3:7, k = 1:9
    # i is the top left row, j is the top left column
    # We'll sum from i to i+2, e.g. i=4, r=4, 5, 6
    @constraint(sudoku, sum(x[r,c,k] for  r in i:i+2, c in j:j+2) == 1)
end

#' The final step is to add the initial solution as a set of constraints. We'll solve the problem that is in the picture at the start of the notebook. We'll put a `0` if there is no digit in that location.

# The given digits
init_sol = [ 5 3 0 0 7 0 0 0 0;
             6 0 0 1 9 5 0 0 0;
             0 9 8 0 0 0 0 6 0;
             8 0 0 0 6 0 0 0 3;
             4 0 0 8 0 3 0 0 1;
             7 0 0 0 2 0 0 0 6;
             0 6 0 0 0 0 2 8 0;
             0 0 0 4 1 9 0 0 5;
             0 0 0 0 8 0 0 7 9]
for i = 1:9, j = 1:9
    # If the space isn't empty
    if init_sol[i,j] != 0
        # Then the corresponding variable for that digit
        # and location must be 1
        @constraint(sudoku, x[i,j,init_sol[i,j]] == 1)
    end
end

# solve problem
optimize!(sudoku)

# Extract the values of x
x_val = value.(x)
# Create a matrix to store the solution
sol = zeros(Int,9,9)  # 9x9 matrix of integers
for i in 1:9, j in 1:9, k in 1:9
    # Integer programs are solved as a series of linear programs
    # so the values might not be precisely 0 and 1. We can just
    # round them to the nearest integer to make it easier
    if round(Int,x_val[i,j,k]) == 1
        sol[i,j] = k
    end
end
# Display the solution
sol

#' Which is the correct solution:
#'


"""
    combinationrank(xs...)

Returns the index of the combination `xs` in the combinatorial number system. The elements
of `xs` all have to be unique, greater than zero and sorted in ascending order.
"""
function combinationrank(xs...)
    @assert length(unique(xs)) == length(xs)
    @assert sort([xs...]) == [xs...]
    @assert all(xs .> 0)
    k = length(xs)
    1 + sum(binomial.(xs .- 1, 1:k))
end

"""
    combinationunrank(k, i)

Returns the `i`-th `k`-element combination in the combinatorial number system.
"""
function combinationunrank(k, idx)
    @assert k > 0
    @assert idx > 0
    v = zeros(Int, k)
    for i = k:-1:1
        ck = _max_ck(i, idx - 1)
        v[i] = ck + 1
        idx -= binomial(ck, i)
    end
    tuple(v...)
end

function _max_ck(k, idx)
    ck = 0
    while true
        binomial(ck + 1, k) > idx && return ck
        ck += 1
    end
    error("Unreachable reached.")
end

"""
    findcombinationdiff(c1::NTuple{N,Int}, c2::NTuple{N,Int}) -> (idx1, idx2) or nothing

If the two ordered combinations of the same length have all elements except one in common,
return a pair of indices of those elements. Otherwise, return `nothing`.
"""
function findcombinationdiff(c1::NTuple{N,Int}, c2::NTuple{N,Int}) where N
    idx1::Union{Nothing,Int}, idx2::Union{Nothing,Int} = nothing, nothing
    i = 1
    while i < N
        i1 = isnothing(idx1) ? i : i + 1
        i2 = isnothing(idx2) ? i : i + 1
        if c1[i1] == c2[i2]
            i += 1
        elseif (c1[i1] < c2[i2]) && isnothing(idx1)
            idx1 = i1
        elseif (c1[i1] > c2[i2]) && isnothing(idx2)
            idx2 = i2
        else
            return nothing
        end
    end
    isnothing(idx1) && (idx1 = N)
    isnothing(idx2) && (idx2 = N)
    c1[idx1] == c2[idx2] ? nothing : (idx1, idx2)
end


export cycle_detection_floyd, cycle_detection_brent

# https://en.wikipedia.org/wiki/Cycle_detection#Algorithms

cycle_detection_floyd(f, x0) = begin
    # Main phase of algorithm: finding a repetition x_i = x_2i.
    # The hare moves twice as quickly as the tortoise and
    # the distance between them increases by 1 at each step.
    # Eventually they will both be inside the cycle and then,
    # at some point, the distance between them will be
    # divisible by the period λ.
    tortoise = f(x0) # f(x0) is the element/node next to x0.
    hare = (f ∘ f)(x0)
    while tortoise ≠ hare
        tortoise = f(tortoise)
        hare = (f ∘ f)(hare)
    end

    # At this point the tortoise position, ν, which is also equal
    # to the distance between hare and tortoise, is divisible by
    # the period λ. So hare moving in circle one step at a time,
    # and tortoise (reset to x0) moving towards the circle, will
    # intersect at the beginning of the circle. Because the
    # distance between them is constant at 2ν, a multiple of λ,
    # they will agree as soon as the tortoise reaches index μ.

    # Find the position μ of first repetition.
    μ = 0
    tortoise = x0
    while tortoise ≠ hare
        tortoise = f(tortoise)
        hare = f(hare)   # Hare and tortoise move at same speed
        μ += 1
    end

    # Find the length of the shortest cycle starting from x_μ
    # The hare moves one step at a time while tortoise is still.
    # λ is incremented until λ is found.
    λ = 1
    hare = f(tortoise)
    while tortoise ≠ hare
        hare = f(hare)
        λ += 1
    end

    λ, μ
end

cycle_detection_brent(f, x0) = begin
    # main phase: search successive powers of two
    power = λ = 1
    tortoise = x0
    hare = f(x0)  # f(x0) is the element/node next to x0.
    while tortoise ≠ hare
        if power == λ  # time to start a new power of two?
            tortoise = hare
            power *= 2
            λ = 0
        end
        hare = f(hare)
        λ += 1
    end

    # Find the position of the first repetition of length λ
    μ = 0
    tortoise = hare = x0
    for i in 0:λ-1
        hare = f(hare)
    end
    # The distance between the hare and tortoise is now λ.

    # Next, the hare and tortoise move at same speed until they agree
    while tortoise ≠ hare
        tortoise = f(tortoise)
        hare = f(hare)
        μ += 1
    end

    λ, μ
end


# ---
# title: 70. Climbing Stairs
# id: problem70
# author: zhwang
# date: 2022-01-16
# difficulty: Easy
# categories: Dynamic Programming
# link: <https://leetcode.com/problems/climbing-stairs/description/>
# hidden: true
# ---
#
# You are climbing a staircase. It takes `n` steps to reach the top.
#
# Each time you can either climb `1` or `2` steps. In how many distinct ways can
# you climb to the top?
#
#
#
# **Example 1:**
#
#
#
#     Input: n = 2
#     Output: 2
#     Explanation: There are two ways to climb to the top.
#     1. 1 step + 1 step
#     2. 2 steps
#
#
# **Example 2:**
#
#
#
#     Input: n = 3
#     Output: 3
#     Explanation: There are three ways to climb to the top.
#     1. 1 step + 1 step + 1 step
#     2. 1 step + 2 steps
#     3. 2 steps + 1 step
#
#
#
#
# **Constraints:**
#
#   * `1 <= n <= 45`
#
#
## @lc code=start
using LeetCode

function climbing_stairs(n::Int)::Int
    n == 1 && return 1
    a, b = 1, 2
    for _ in 3:n
        a, b = b, a + b
    end
    return b
end

## @lc code=end


# Simulate a Poisson point process on a disk
# Author: H. Paul Keeler, 2019.
# Website: hpaulkeeler.com
# Repository: github.com/hpaulkeeler/posts
# For more details, see the post:
# hpaulkeeler.com/testing-the-julia-language-with-point-process-simulations/

#Note: Need the .+ for adding a scalar to an array
#Also need . for sqrt, exp, cos, sin etc and assinging scalars to arrays
#Best to use vectors instead of 1-D matrices eg x=rand(n),  NOT x=rand(n,1).

using Distributions #for random simulations
using Plots #for plotting

#Simulation window parameters
r=1; #radius of disk
xx0=0; yy0=0; #centre of disk
areaTotal=pi*r^2; #area of disk

#Point process parameters
lambda=100; #intensity (ie mean density) of the Poisson process

#Simulate Poisson point process
numbPoints=rand(Poisson(areaTotal*lambda)); #Poisson number of points
theta=2*pi*(rand(numbPoints));#angular coordinates  of Poisson points
rho=r*sqrt.(rand(numbPoints));#radial coordinates of Poisson points

#Convert polar to Cartesian coordinates
xx=rho.*cos.(theta);
yy=rho.*sin.(theta);

#Shift centre of disk to (xx0,yy0)
xx=xx.+xx0;
yy=yy.+yy0;

plot1=scatter(xx,yy,xlabel ="x",ylabel ="y", leg=false);
display(plot1);


## Maximum entropy degree distribution of a network with S species and L links

function p_k(k::Int64, S::Int64, L::Int64)
    # k: degree of a species
    # S: number of species in the food web
    # L: number of links in the food web

    # return probability of degree k from MaxEnt distribution
    # p_k = c * r^k

    # constraint of mean
    mean_degree = (2 * L) / S

    # maximum entropy distribution
    c = 1 / (mean_degree - 1)
    r = (mean_degree - 1) / mean_degree

    return c * r ^ k
end

function maxent_dd(S::Int64, L::Int64)
    # S: number of species
    # L: number of links

    # return degree distribution with maximum entropy
    # i.e. probability for each k between 1 and S
    degree_dist = zeros(S)
    for k in 1:S
        degree_dist[k] = p_k(k, S, L)
    end

    return degree_dist
end


function naiveDFT(hk::Array{T}) where T
    N = length(hk);
    Wnk = Array{Complex{T}}(undef, N, N)

    prefactor = -2 * π * im / N

    for n = 0:(N - 1), k = 0:(N - 1)
        Wnk[k + 1, n + 1] = exp(prefactor * n * k)
    end

    return Wnk * hk
end

function naiveDFTIterative(hk::Array{T}) where T
    N = length(hk);
    Hk = zeros(Complex{T}, N)

    prefactor = -2 * π * im / N

    @inbounds @simd for k = 1:N
        s = zero(Complex{T})
        km1 = k - 1
        for j = 1:N
            s += exp(prefactor * (j - 1) * km1) * hk[j]
        end
        Hk[k] = s
    end

    return Hk
end


function log_oneplusexp(E::Float64)
    return (E < 50) ? log(1+exp(E)) : E
end

function xlogy(x::Float64, y::Float64)
    if x == 0.0
        return 0.0
    else
        return x*log(y)
    end
end

function log_binom(n::Float64, k::Float64)
    @assert n >= k
    if (n == 0) || (k == 0)
       return 0
    end
    return n*log(n)-k*log(k)-(n-k)*log(n-k)
end


# fibonacci seq - LaTeX
#JL Plots

Pkg.add("LaTeXStrings")
Pkg.add("Measures")
using LaTeXStrings, Measures
gr(bg = :white)

fib = zeros(12);
for i = 1:12
    fib[i] = (((1+sqrt(5))/2)^i - ((1-sqrt(5))/2)^i)/sqrt(5);
end

Plots.plot(fib,
    marker=:circle,
    linewidth= 2,
    xlabel=L"n",
    ylabel=L"F_n",
    annotation=(5, 100, L"F_n = \frac{1}{\sqrt{5}} \left[\left( \frac{1+\sqrt{5}}{2} \right)^n - \left( \frac{1-\sqrt{5}}{2} \right)^n \right]"))


### A Pluto.jl notebook ###
# v0.12.16

using Markdown
using InteractiveUtils

# ╔═╡ e077ab16-22e5-11eb-15ed-5f0654bf7328
using Pkg, DrWatson

# ╔═╡ e75a9a06-22e5-11eb-2e36-8d4f62b830ed
begin
	@quickactivate "StatisticsWithJuliaPlutoNotebooks"
	using Random, Distributions, Plots, Measures
end;

# ╔═╡ ca5fbdc8-22e5-11eb-0a60-cb0a127a3ca5
md"## Listing6.09"

# ╔═╡ 1aa41898-3afa-11eb-2f49-570853ce6435
begin
	N = 5*10^3
	alpha = 0.05
	confLevel = 1 - alpha
	z = quantile(Normal(),1-alpha/2)
end;

# ╔═╡ 7e997ce4-3b27-11eb-167a-f74a6dff6b1c
function randCI(n,p)
    sample = rand(n) .< p
    pHat = sum(sample)/n
    serr = sqrt(pHat*(1-pHat)/n)
    (pHat - z*serr, pHat + z*serr)
end

# ╔═╡ 7e99af3e-3b27-11eb-2b94-051fc56f27cf
cover(p,ci) = ci[1] <= p && p <= ci[2]

# ╔═╡ 7e9a284c-3b27-11eb-37b2-6fcc5cbb3adb
begin
	pGrid = 0.1:0.01:0.9
	nGrid = 5:1:50
	errs = zeros(length(nGrid),length(pGrid))

	for i in 1:length(nGrid)
		for j in 1:length(pGrid)
			Random.seed!(0)
			n, p = nGrid[i], pGrid[j]
			coverageRatio = sum([cover(p,randCI(n,p)) for _ in 1:N])/N
			errs[i,j] = confLevel - coverageRatio
		end
	end
end;

# ╔═╡ 7ea98df0-3b27-11eb-292f-4d1335c99239
begin
	default(xlabel = "p", ylabel = "n",
		xticks =([1:5:length(pGrid);], minimum(pGrid):0.05:maximum(pGrid)),
		yticks =([1:5:length(nGrid);], minimum(nGrid):5:maximum(nGrid)))

	p1 = heatmap(errs, c=cgrad([:white, :black]))
	p2 = heatmap(abs.(errs) .<= 0.04, legend = false, c=cgrad([:black, :white]))
	plot(p1,p2, size = (1000,400), margin = 5mm)
end

# ╔═╡ 475ff888-22e6-11eb-2354-09f8f40a8e12
md"## End of listing6.09"

# ╔═╡ Cell order:
# ╟─ca5fbdc8-22e5-11eb-0a60-cb0a127a3ca5
# ╠═e077ab16-22e5-11eb-15ed-5f0654bf7328
# ╠═e75a9a06-22e5-11eb-2e36-8d4f62b830ed
# ╠═1aa41898-3afa-11eb-2f49-570853ce6435
# ╠═7e997ce4-3b27-11eb-167a-f74a6dff6b1c
# ╠═7e99af3e-3b27-11eb-2b94-051fc56f27cf
# ╠═7e9a284c-3b27-11eb-37b2-6fcc5cbb3adb
# ╠═7ea98df0-3b27-11eb-292f-4d1335c99239
# ╟─475ff888-22e6-11eb-2354-09f8f40a8e12


## spherical.jl : implementation of spherical covariance function

## Spherical ##
struct Spherical{T} <: IsotropicCovarianceStructure{T}
    λ::T
    σ::T
    p::T

    function Spherical{T}(λ::T, σ::T, p::T) where T
        λ > 0 || throw(DomainError(λ, "correlation length λ of spherical covariance cannot be negative or zero"))
        σ > 0 || throw(DomainError(σ, "marginal standard deviation σ of spherical covariance cannot be negative or zero"))
        p >= 1 || throw(DomainError(p, "in p-norm, p must be greater than or equal to 1"))
        isinf(p) && throw(DomainError(p, "in p-norm, p cannot be infinity"))

        new{T}(λ, σ, p)
    end
end

"""
    Spherical(λ, [σ = 1], [p = 2])

Spherical covariance structure with correlation length `λ`, (optional) marginal standard deviation `σ` and (optional) `p`-norm, defined as

``C(x, y) = \\begin{cases} σ \\left(1 - \\displaystyle\\frac{3}{2}\\frac{ρ}{λ} + \\frac{1}{2}\\left(\\frac{ρ}{λ}\\right)^3\\right) & \\text{for }ρ≤λ\\\\0 & \\text{for }ρ>λ\\end{cases}``

with ``ρ = ||x - y||_p``.

# Examples
```jldoctest
julia> Spherical(0.1)
spherical (λ=0.1, σ=1.0, p=2.0)

julia> Spherical(1.0, σ=2)
spherical (λ=1.0, σ=2.0, p=2.0)

```
See also: [`Exponential`](@ref), [`Linear`](@ref), [`Whittle`](@ref), [`Gaussian`](@ref), [`SquaredExponential`](@ref), [`Matern`](@ref)
"""
Spherical(λ::Real; σ::Real=1.0, p::Real=2) = Spherical{promote_type(typeof(λ),typeof(σ),typeof(p))}(promote(λ, σ, p)...)

# evaluate spherical covariance
apply(s::Spherical, x::Real) = max(zero(x), 1 - 3/2 * x/s.λ + 1/2 * (x/s.λ)^3)

# short name
shortname(::Spherical) = "spherical"


"""
    VaR(states,prob,alpha::Float32)

implements the Value-at-Risk at level ``\\alpha`` defined by
```math
VaR_\\alpha (Y) = \\arg \\min_x \\left( x\\in \\mathbb{R} : F_Y(x) \\geq \\alpha \\right),
```
for the random variable ``Y`` defined by `states` and `prob`.
"""
function VaR(x::Vector{Float64}, f::Vector{Float64}, α::Float32)
    ind = sortperm(x[1:length(x)])
    x = x[ind]; f = f[ind];
    i = findfirst(p -> p≥α, cumsum(f))
    if i === nothing
        return x[end]
    else
        return x[i]
    end
end

"""
    CTE(states,prob,alpha::Float32)

implements the Conditional Value-at-Risk at level ``\\alpha`` defined by
```math
CTE_\\alpha (Y) = VaR_\\alpha(Y) + \\frac{1}{1-\\alpha} \\mathbb{E} \\left( Y- VaR_\\alpha (Y) \\right)_+ ,
```
for the random variable ``Y`` defined by `states` and `prob`.
"""
function CTE(x::Vector{Float64}, f::Vector{Float64}, α::Float32)
    x = -x; α = 1-α;
    x_α = VaR(x, f, α)
    if iszero(α)
        return -x_α
    else
        tail = x .≤ x_α
        result = (sum(x[tail] .* f[tail]) - (sum(f[tail]) - α) * x_α) / α
        return -result
    end
end

"""
    EVaR2(states,prob,beta)

Solves the optimization problem associated with the primal formulation of the Entropic Value-at-Risk:

```math
EVaR_\\alpha(Y) = \\min_{x >0} \\frac{1}{x} \\left( \\beta +  \\log\\mathbb{E} e^{xY} \\right),
```
where ``Y`` is the discrete random variable defined by `states` and `prob`.
Here the optimization is done via the goldenSearch optimization routine implemented as part of this package.
"""
function EVaR2(states::Vector{Float64}, prob::Vector{Float64},beta::Float32)
	if sum(prob) == 0
		prob = ones(length(states))./ length(states)
	end
	dualZ = zeros(length(states))
	esssup = maximum(states[vec(prob.>0.0)])
	varm = dot(prob, (states .- dot(prob,states)).^2)
	if beta == 0.0
		Fvalue = dot(prob, states)
		tOpt = 0.0
		dualZ = exp.(tOpt .*states)
	else
		if varm >0.0
			tOpt = sqrt.(2 *beta/varm)
			function objective1(t)
				t=max(t,0.0)
				w = states./t
				offset = maximum(w)
    			we = exp.(w .- offset)
    			s = dot(prob, we)
    			w = log(s) + offset
				tmp = t*beta + t*w
				#tmp = t*beta + t*log(dot(prob,exp.(states./t)))
				return(tmp)
			end
			out = goldenSearch(objective1,tOpt)
			tOpt = out[1]
			Fvalue = out[2]
		else
			tOpt = 0.0
			Fvalue = 0.0
			dualZ = esssup
		end
		#if Fvalue < esssup
			dualZ = exp.(tOpt .*states)
		#else
		#	Fvalue = esssup
		#	dualZ[states .>= esssup] .= 1
		#end
		dualZ = dualZ / dot(prob,dualZ)
	end
	return(Fvalue,tOpt,dualZ) # for numerical stability
end

"""
    EVaR(states,prob,beta::Float32)

Solves the optimization problem associated with the primal formulation of the Entropic Value-at-Risk:

```math
EVaR_\\alpha(Y) = \\min_{x >0} \\frac{1}{x} \\left( \\beta +  \\log\\mathbb{E} e^{xY} \\right),
```
where ``Y`` is the discrete random variable defined by `states` and `prob`. Here, the optimization is done using JuMP and Ipopt.
"""
function EVaR(states::Vector{Float64}, prob::Vector{Float64},beta::Float32)
	if sum(prob) == 0
		prob = ones(length(states))./ length(states)
	end
	dualZ = zeros(length(states))
	esssup = maximum(states[prob.>0.0]) #error
	var = dot(prob, (states .- dot(prob,states)).^2 )
	if beta == 0.0
		EVaR = dot(prob, states)
		tOpt = 0
		dualZ = exp.(tOpt .*states)
	else
		if var >0
			tOpt = sqrt.(2 *beta/var)
			EV = Model(optimizer_with_attributes(Ipopt.Optimizer, "print_level" =>0))
			function objective1(t)
				return( (beta.+ log(dot(prob,exp.(t.*states))))/t)
			end
			register(EV, :objective1, 1, objective1, autodiff=true)

	    	@variable(EV,t,start = tOpt)
			@constraint(EV, t .>=0)
	    	@NLobjective(EV, Min, objective1(t) )

			JuMP.optimize!(EV)
			EVaR = JuMP.objective_value(EV)
	    	tOpt = JuMP.value(t)
		else
			tOpt = 0
			EVaR = 0
			dualZ = esssup
		end
		if EVaR < esssup
			dualZ = exp.(tOpt .*states)
		else
			EVaR = esssup
			#dualZ[states==esssup] .= 1
		end
		#dualZ = dualZ / dot(prob,dualZ)
	end
	return(EVaR,tOpt,dualZ)
end

"""
    AVaR(states,prob,alpha::Float32)

Solves the optimization problem associated with the primal formulation of the Average Value-at-Risk:

```math
AVaR_\\alpha(Y) = \\min_{x\\in \\mathbb{R}} x + \\frac{1}{1-\\alpha} \\mathbb{E} \\left( Y - x \\right)_+,
```
where ``Y`` is the discrete random variable defined by `states` and `prob`.
"""
function AVaR(states::Vector{Float64},prob::Vector{Float64}, alpha::Float32)
    var2 = dot(prob, (states .- dot(prob,states)).^2);
     #First trivial case
    if alpha == 0.0
        return(dot(prob, states))
  #Now set up optimization problem using goldenSearch
    else
       tOpt = sqrt.(2 *alpha/var2)
       EV = Model(optimizer_with_attributes(Ipopt.Optimizer, "print_level" =>0))
		function objective1(t)
			return( t + 1/(1-alpha) * dot(prob, max.(states .- t,0)) )
		end
		register(EV, :objective1, 1, objective1, autodiff=true)
	    @variable(EV,t,start = tOpt)
		@NLobjective(EV, Min, objective1(t) )

		JuMP.optimize!(EV)
		AVaR = JuMP.objective_value(EV)
	    tOpt = JuMP.value(t)
    end
    return(AVaR, tOpt)
end


# The following reproduces the example introduced in Section 3.2.3 of Lütkepohl (2006)

using VectorAutoregressions
using CSV, DataFrames
using Dates

exdir = pkgdir(VectorAutoregressions) * "/examples";
datadf = DataFrame(CSV.File(exdir * "/lutkepohl_data.csv"));
first(datadf, 6)

# Take the subset from 1960q1 to 1978q4, and use log-difference for each series
datadf = datadf[datadf.date.<Date(1979, 3, 1), :];
Y = DataFrame(
    investment = diff(log.(datadf.invest)),
    income = diff(log.(datadf.income)),
    consumption = diff(log.(datadf.cons)),
);
first(Y, 6)

# ## Calculate lag selection criteria
ls = LagSelection(Y, 8)
# ## Estimate the VAR
v = VAR(Y, ls.selection["AIC"])
# ## Check stability of the VAR
sc = StabilityCheck(v)
# ## Check for autocorrelation in the residuals
pt = PortmanteauTest(v, 12)
bg1 = LMCorrTest(v, 4)
bg2 = LMCorrTest(v, 4, smallsample = true)


# This does simple Poisson estimation using the meps1996.data

# The MEPS data

# Define dep and expl vbls
#
# 	The dep. vbls, in corresponding column
# 	Office based doctor visits	1
# 	Outpatient doctor visits	2
# 	Emergency room visits		3
# 	Inpatient visits		4
# 	Dental visits			5
# 	Prescriptions			6
#
using Econometrics, DelimitedFiles
which_dep = 1
if (which_dep == 1) dep = "OBDV"   end
if (which_dep == 2) dep = "OPV"    end
if (which_dep == 3) dep = "IPV"    end
if (which_dep == 4) dep = "ERV"    end
if (which_dep == 5) dep = "DV"     end
if (which_dep == 6) dep = "PRESCR" end
data = readdlm("../Data/meps1996.data")
y = data[:,which_dep]
x = data[:,7:12]
n = size(x,1)
x = [ones(n,1) x]
# to check effects of poor scaling, try commenting the next line
x[:,end] = x[:,end]/1000.0 # put income in thousands
names = ["constant", "pub. ins.","priv. ins.", "sex", "age","edu","inc"]
title = "Poisson model, "*dep* ",  MEPS 1996 full data set"
model = theta -> poisson(theta, y, x)
theta = zeros(size(x,2)) # start values for estimation
# try adding the option vc=2 for "Hessian" or vc=3 for OPG
thetahat, objvalue, V, converged = mleresults(model, theta, title, names, vc=1)
nothing


export ExtensionalSet, IntensionalSet, settuple, orderedpair, cardinality

abstract type MathematicalSet <: AbstractExpression end
istree(s::MathematicalSet) = false # TODO: refactor structure conditionals to make more sense

struct ExtensionalSet{T} <: MathematicalSet
    elements::Set{T}
end
ExtensionalSet(elements::Vector{T}) where {T} = ExtensionalSet(Set{T}(elements))
ExtensionalSet(elements::Tuple{Vararg{T}}) where {T} = ExtensionalSet(Set{T}([elements...]))
elements(es::ExtensionalSet) = es.elements
Base.length(es::ExtensionalSet) = length(elements(es))
Base.isempty(es::ExtensionalSet) = Base.isempty(elements(es))
Base.hash(es::ExtensionalSet, h::UInt) = hash(elements(es), h)
Base.:(==)(es1::ExtensionalSet, es2::ExtensionalSet) = elements(es1) == elements(es2)
cardinality(es::ExtensionalSet) = length(es)

# expression methods
variables(es::ExtensionalSet) = length(es) > 0 ? reduce(∪, variables.(elements(es))) : Set{LogicalSymbol}()
operations(es::ExtensionalSet) = length(es) > 0 ? reduce(∪, operations.(elements(es))) : Set{LogicalOperation}()

function Base.show(io::IO, es::ExtensionalSet)
    if isempty(es)
        print(io, "∅")
        return
    end

    print(io, "{")
    i = 1
    truncated_elements = truncate([elements(es)...], 5)
    for el ∈ truncated_elements
        print(io, el)
        if i < cardinality(es)
            print(io, ", ")
        end
        i += 1
    end
    if length(truncated_elements) < cardinality(es)
        missed_elements = cardinality(es) - length(truncated_elements)
        print(io, "...$(missed_elements) more")
    end
    print(io, "}")
end

orderedpair(a, b) = ExtensionalSet([ExtensionalSet([a]), ExtensionalSet([a, b])])
function settuple(a...)
    a = [a...]
    if length(a) == 0
        return ∅
    end
    x = settuple(a[1:end-1]...)
    ExtensionalSet([ExtensionalSet([x]), ExtensionalSet([x, a[end]])])
end

struct IntensionalSet <: MathematicalSet
    transform::AbstractExpression
    rule::AbstractExpression
end
IntensionalSet(transform::AbstractExpression, rules::Vector{LogicalExpression}) = IntensionalSet(transform, reduce(∧, rules))
IntensionalSet(transform::AbstractExpression, rules::Set{LogicalExpression}) = IntensionalSet(transform, [rules...])
IntensionalSet(transform::AbstractExpression, rules::Tuple{Vararg{LogicalExpression}}) = IntensionalSet(transform, [rules...])
transform(is::IntensionalSet) = is.transform
rule(is::IntensionalSet) = is.rule

# expression methods
variables(is::IntensionalSet) = variables(rule(is))
operations(is::IntensionalSet) = operations(rule(is))

function Base.show(io::IO, is::IntensionalSet)
    print(io, "{")
    print(io, transform(is))
    print(io, " | ")
    print(io, rule(is))
    print(io, "}")
end


# This file includes investigates the 2D interpolation mean square error(MSE) with respect to inteprolation point number.

using FractalTools
using Makie

# Construct interpolation data
f(x, y) = x^2 + y^2 + 1
vtx = [
    BigFloat.([0.0, 0.0]),
    BigFloat.([1.0, 0.0]),
    BigFloat.([0.5, 1.0])
    ]
freevar = 0.001
npts    = 50 : 5 : 150
ntpts   = 2 * npts[end]

# Construct test data
tpts = getdata(vtx, ntpts)

# Compute errors
mse = map(npts) do npt
    @info npt
    # Construct interpolation data
    pts = getdata(f, vtx, npt)

    # Construct interpolant
    interp = interpolate(pts, Interp2D(freevar))

    # Compute error
    fvals = map(pnt -> f(pnt...), tpts)
    ivals = map(pnt -> interp(pnt...), tpts)
    sum((fvals - ivals).^2) / length(tpts)
end

# Plot mse
fig = Figure()
ax = fig[1, 1] = Axis(fig, xlabel="Number of Points", ylabel="MSE", title="2D Interpolation MSE")
stem!(ax, npts, mse, color=:black)
save(joinpath(@__DIR__, "interp2d_mse.png"), fig)
display(fig)


using JuMP, GLPK, LinearAlgebra, DataFrames

# Define some input data about the test system
# Maximum power output of generators
g_max = [1000, 1000];
# Minimum power output of generators
g_min = [0, 300];
# Incremental cost of generators
c_g = [50, 100];
# Fixed cost of generators
c_g0 = [1000, 0]
# Incremental cost of wind generators
c_w = 50;
# Total demand
d = 1500;
# Wind forecast
w_f = 200;

# In this cell we create function solve_ed, which solves the economic dispatch problem for a given set of input parameters.
function solve_ed(g_max, g_min, c_g, c_w, d, w_f)
    #Define the economic dispatch (ED) model
    ed = Model(GLPK.Optimizer)

    # Define decision variables
    @variable(ed, 0 <= g[i = 1:2] <= g_max[i]) # power output of generators
    @variable(ed, 0 <= w <= w_f) # wind power injection

    # Define the objective function
    @objective(ed, Min, dot(c_g, g) + c_w * w)

    # Define the constraint on the maximum and minimum power output of each generator
    @constraint(ed, [i = 1:2], g[i] <= g_max[i]) #maximum
    @constraint(ed, [i = 1:2], g[i] >= g_min[i]) #minimum

    # Define the constraint on the wind power injection
    @constraint(ed, w <= w_f)

    # Define the power balance constraint
    @constraint(ed, sum(g) + w == d)

    # Solve statement
    optimize!(ed)

    # return the optimal value of the objective function and its minimizers
    return value.(g), value(w), w_f - value(w), objective_value(ed)
end

# Solve the economic dispatch problem
(g_opt, w_opt, ws_opt, obj) = solve_ed(g_max, g_min, c_g, c_w, d, w_f);

println("\n")
println("Dispatch of Generators: ", g_opt, " MW")
println("Dispatch of Wind: ", w_opt, " MW")
println("Wind spillage: ", w_f - w_opt, " MW")
println("\n")
println("Total cost: ", obj, "\$")

c_g_scale_df = DataFrame(Symbol("Dispatch of Generator 1(MW)") => Float64[],
               Symbol("Dispatch of Generator 2(MW)") => Float64[],
               Symbol("Dispatch of Wind(MW)") => Float64[],
               Symbol("Spillage of Wind(MW)") => Float64[],
               Symbol("Total cost(\$)") => Float64[])
for c_g1_scale = 0.5:0.1:3.0
    c_g_scale = [c_g[1] * c_g1_scale, c_g[2]] # update the incremental cost of the first generator at every iteration
    g_opt, w_opt, ws_opt, obj = solve_ed(g_max, g_min, c_g_scale, c_w, d, w_f) # solve the ed problem with the updated incremental cost
    push!(c_g_scale_df, (g_opt[1], g_opt[2], w_opt, ws_opt, obj))
end

ENV["COLUMNS"]=250 # Helps us display the complete table
c_g_scale_df

function solve_ed_inplace(c_w_scale)
    start = time()
    obj_out = Float64[]
    w_out = Float64[]
    g1_out = Float64[]
    g2_out = Float64[]

    ed = Model(GLPK.Optimizer)

    # Define decision variables
    @variable(ed, 0 <= g[i = 1:2] <= g_max[i]) # power output of generators
    @variable(ed, 0 <= w <= w_f ) # wind power injection

    # Define the objective function
    @objective(ed, Min, dot(c_g, g) + c_w * w)

    # Define the constraint on the maximum and minimum power output of each generator
    @constraint(ed, [i = 1:2], g[i] <= g_max[i]) #maximum
    @constraint(ed, [i = 1:2], g[i] >= g_min[i]) #minimum

    # Define the constraint on the wind power injection
    @constraint(ed, w <= w_f)

    # Define the power balance constraint
    @constraint(ed, sum(g) + w == d)

    optimize!(ed)

    for c_g1_scale = 0.5:0.01:3.0
        @objective(ed, Min, c_g1_scale*c_g[1]*g[1] + c_g[2]*g[2] + c_w_scale*c_w*w)
        optimize!(ed)
        push!(obj_out, objective_value(ed))
        push!(w_out, value(w))
        push!(g1_out, value(g[1]))
        push!(g2_out, value(g[2]))
    end
    elapsed = time() - start
    print(string("elapsed time: ", elapsed, " seconds"))
    return obj_out, w_out, g1_out, g2_out
end

solve_ed_inplace(2.0);

demandscale_df = DataFrame(Symbol("Dispatch of Generators(MW)") => Float64[],
               Symbol("Dispatch of Generator 2(MW)") => Float64[],
               Symbol("Dispatch of Wind(MW)") => Float64[],
               Symbol("Spillage of Wind(MW)") => Float64[],
               Symbol("Total cost(\$)") => Float64[])

for demandscale = 0.2:0.1:1.5
    g_opt,w_opt,ws_opt,obj = solve_ed(g_max, g_min, c_g, c_w, demandscale*d, w_f)

    push!(demandscale_df, (g_opt[1], g_opt[2], w_opt, ws_opt, obj))
end

demandscale_df

# In this cell we introduce binary decision u to the economic dispatch problem (function solve_ed)
function solve_uc(g_max, g_min, c_g, c_w, d, w_f)
    #Define the unit commitment (UC) model
    uc = Model(GLPK.Optimizer)

    # Define decision variables
    @variable(uc, 0 <= g[i=1:2] <= g_max[i]) # power output of generators
    @variable(uc, u[i = 1:2], Bin) # Binary status of generators
    @variable(uc, 0 <= w <= w_f ) # wind power injection

    # Define the objective function
    @objective(uc, Min, dot(c_g, g) + c_w * w)

    # Define the constraint on the maximum and minimum power output of each generator
    @constraint(uc, [i = 1:2], g[i] <= g_max[i]) #maximum
    @constraint(uc, [i = 1:2], g[i] >= g_min[i]) #minimum

    # Define the constraint on the wind power injection
    @constraint(uc, w <= w_f)

    # Define the power balance constraint
        @constraint(uc, sum(g) + w == d)

    # Solve statement
    optimize!(uc)

    status = termination_status(uc)
    if status != MOI.OPTIMAL
        return status, zeros(length(g)), 0.0, 0.0, zeros(length(u)), Inf
    end
    return status, value.(g), value(w), w_f - value(w), value.(u), objective_value(uc)
end

# Solve the economic dispatch problem
status, g_opt, w_opt, ws_opt, u_opt, obj = solve_uc(g_max, g_min, c_g, c_w, d, w_f);

println("\n")
println("Dispatch of Generators: ", g_opt[:], " MW")
println("Commitments of Generators: ", u_opt[:])
println("Dispatch of Wind: ", w_opt, " MW")
println("Wind spillage: ", w_f - w_opt, " MW")
println("\n")
println("Total cost: ", obj, "\$")

uc_df = DataFrame(Symbol("Commitment of Generator 1(MW)") => Float64[],
               Symbol("Commitment of Generator 2(MW)") => Float64[],
               Symbol("Dispatch of Generator 1(MW)") => Float64[],
               Symbol("Dispatch of Generator 2(MW)") => Float64[],
               Symbol("Dispatch of Wind(MW)") => Float64[],
               Symbol("Spillage of Wind(MW)") => Float64[],
               Symbol("Total cost(\$)") => Float64[])

for demandscale = 0.2:0.1:1.5
    status, g_opt, w_opt, ws_opt, u_opt, obj = solve_uc(g_max, g_min, c_g, c_w, demandscale*d, w_f)

    if status == MOI.OPTIMAL
    push!(uc_df, (u_opt[1], u_opt[2], g_opt[1], g_opt[2], w_opt, ws_opt, obj))
    else
        println("Status: $status for demandscale = $demandscale \n")
    end
end

uc_df


using DifferentialEquations
using Plots
plotlyjs()

#constants
l = 1.0                             # length [m]
m = 1.0                             # mass[Kg]
g = 9.81                            # gravitational acceleration [m/s²]

#function to compute ODE
function pendulum!(du,u,p,t)
    du[1] = u[2]                    # θ'(t) = ω(t)
    du[2] = -(g\l)*sin(u[1])  # ω'(t) = -(g/l) sin θ(t)
end

#initial conditions
θ₀ = π/6                         # initial angular deflection [rad]
ω₀ = 0.0                            # initial angular velocity [rad/s]
u₀ = [θ₀, ω₀]                       # initial state vector
tspan = (0.0,100.0)                  # time interval

#defining the problem and solving
prob = ODEProblem(pendulum!,u₀,tspan)
sol = solve(prob,saveat=0.1)

#plotting the solution
display(plot(sol,linewidth=2,xaxis="t (s)",
    color=["green" "blue"],label=["θ [rad]" "ω [rad/s]"],layout=(2,1)))
savefig("./plots/simple pendulum-plot.png")

# trajectory
t=sol.t
U=sol[1:end,:]
(θ,ω)=[U[x,:] for x in 1:size(U,1)]
x=l.*sin.(π.-θ)
y=zeros(size(t))
z=l.*cos.(π.-θ)
display(plot(x,y,z,linewidth=2,label="trajectory",
    xaxis="x (m)",yaxis="y (m)",zaxis="z (m)",grid=(:on,:black)))
savefig("./plots/simple pendulum-trajectory.png")

Modeling the Dragon Module as a Pendulum/Scripts/pendulum.jl


using Printf

agm1step(x, y) = (x + y) / 2, sqrt(x * y)

function approxπstep(x, y, z, n::Integer)
    a, g = agm1step(x, y)
    k = n + 1
    s = z + 2 ^ (k + 1) * (a ^ 2 - g ^ 2)
    return a, g, s, k
end

approxπ(a, g, s) = 4a ^ 2 / (1 - s)

function testmakepi()
	setprecision(512)
	a, g, s, k = BigFloat(1.0), 1 / √BigFloat(2.0), BigFloat(0.0), 0
	oldπ = BigFloat(0.0)
	println("Approximating π using ", precision(BigFloat), "-bit floats.")
	println("   k     Error  Result")
	for i in 1:100
		a, g, s, k = approxπstep(a, g, s, k)
		estπ = approxπ(a, g, s)
		if abs(estπ - oldπ) < 2eps(estπ) break end
		oldπ = estπ
		err = abs(π - estπ)
		@printf("%4d%10.1e%68.60e\n", i, err, estπ)
	end
end

testmakepi()


module MyCalculus

export EulerMethod

"""
    EulerMethod(f, x₀, y₀, step= 0.5, n=100.0)

Function to approximate solution to an ordinary differential equation using the Euler's Method.
dy/dx = f(x, y) where y(x₀) = y₀ and x₀ = x₀.

# Arguments
    f: function to approximate solution to, of the form f(x, y).
        This is the right hand side of the differential equation.
    x₀: initial x value condition
    y₀: initial y value condition
    step: step size for the Euler's Method
    n: number of steps to take

# Returns
    x: array of x values
    y: array of y values

"""
function EulerMethod(f::Function, x₀::Real, y₀::Real, step::Real, n::Real=100.0)

    y =[y₀] #initialize y array with intiial condition
    x = [x for x in range(x₀, step=step, length=n)] # array of x values starting with initial condition
    m = [] # this array will hold slope (right hand side of differential equation)

    push!(m, f(x[1], y[1])) # push initial slope onto array

    for k in 2:length(x) # iterate through x values starting at 2nd element

        push!(y, y[k-1] + m[k-1]*step) # push new y value to array using Euler's Method
        push!(m, f(x[k], y[k])) # push slope to array
    end

    return x, y

end

end


using DelimitedFiles
using LinearAlgebra

function convolve_linear(signal::Array{T, 2}, filter::Array{T, 2},
                         output_size) where {T <: Number}

    # convolutional output
    out = Array{Float64,2}(undef, output_size)
    sum = 0

    for i = 1:output_size[1]
        for j = 1:output_size[2]
            for k = max(1, i-size(filter)[1]):i
                for l = max(1, j-size(filter)[2]):j
                    if k <= size(signal)[1] && i-k+1 <= size(filter)[1] &&
                       l <= size(signal)[2] && j-l+1 <= size(filter)[2]
                        sum += signal[k,l] * filter[i-k+1, j-l+1]
                    end
                end
            end

            out[i,j] = sum
            sum = 0
        end
    end

    return out
end

function create_gaussian_kernel(kernel_size)

    kernel = zeros(kernel_size, kernel_size)

    # The center must be offset by 0.5 to find the correct index
    center = kernel_size * 0.5 + 0.5

    sigma = sqrt(0.1*kernel_size)

    for i = 1:kernel_size
        for j = 1:kernel_size
            kernel[i,j] = exp(-((i-center)^2 + (j-center)^2) / (2*sigma^2))
        end
    end

    return normalize(kernel)

end

function create_sobel_operators()
    Sx = [1.0, 2.0, 1.0]*[-1.0 0.0 1.0] / 9
    Sy = [-1.0, 0.0, 1.0]*[1.0 2.0 1.0] / 9

    return Sx, Sy
end

function compute_sobel(signal)
    Sx, Sy = create_sobel_operators()

    Gx = convolve_linear(signal, Sx, size(signal) .+ size(Sx))
    Gy = convolve_linear(signal, Sy, size(signal) .+ size(Sy))

    return sqrt.(Gx.^2 .+ Gy.^2)
end

# Simple function to create a square grid with a circle embedded inside of it
function create_circle(image_resolution, grid_extents, radius)
    out = zeros(image_resolution, image_resolution)

    for i = 1:image_resolution
        x_position = ((i-1)*grid_extents/image_resolution)-0.5*grid_extents
        for j = 1:image_resolution
            y_position = ((j-1)*grid_extents/image_resolution)-0.5*grid_extents
            if x_position^2 + y_position^2 <= radius^2
                out[i,j] = 1.0
            end
        end
    end

    return out
end

function main()

    # Random distribution in x
    x = rand(100, 100)

    # Gaussian signals
    y = [exp(-(((i-50)/100)^2 + ((j-50)/100)^2)/.01) for i = 1:100, j=1:100]

    # Normalization is not strictly necessary, but good practice
    normalize!(x)
    normalize!(y)

    # full convolution, output will be the size of x + y
    full_linear_output = convolve_linear(x, y, size(x) .+ size(y))

    # simple boundaries
    simple_linear_output = convolve_linear(x, y, size(x))

    # outputting convolutions to different files for plotting in external code
    writedlm("full_linear.dat", full_linear_output)
    writedlm("simple_linear.dat", simple_linear_output)

    # creating simple circle and 2 different Gaussian kernels
    circle = create_circle(50,2,0.5)

    normalize!(circle)

    small_kernel = create_gaussian_kernel(3)
    large_kernel = create_gaussian_kernel(25)

    small_kernel_output = convolve_linear(circle, small_kernel,
                                          size(circle).+size(small_kernel))
    large_kernel_output = convolve_linear(circle, large_kernel,
                                          size(circle).+size(large_kernel))

    writedlm("small_kernel.dat", small_kernel_output)
    writedlm("large_kernel.dat", large_kernel_output)

    # Using the circle for Sobel operations as well
    sobel_output = compute_sobel(circle)

    writedlm("sobel_output.dat", sobel_output)

end


using Test, FinancialMonteCarlo;
@show "LogNormalMixture Model"
S0 = 100.0;
K = 100.0;
r = 0.02;
T = 1.0;
d = 0.01;
D = 90.0;

Nsim = 10000;
Nstep = 30;
sigma = [0.2, 0.2];
lam = Float64[0.999999999];
mc = MonteCarloConfiguration(Nsim, Nstep);
mc1 = MonteCarloConfiguration(Nsim, Nstep, FinancialMonteCarlo.AntitheticMC());
toll = 0.8

rfCurve = ZeroRate(r);

FwdData = Forward(T)
EUData = EuropeanOption(T, K)
AMData = AmericanOption(T, K)
BarrierData = BarrierOptionDownOut(T, K, D)
AsianFloatingStrikeData = AsianFloatingStrikeOption(T)
AsianFixedStrikeData = AsianFixedStrikeOption(T, K)
Model = LogNormalMixture(sigma, lam, Underlying(S0, d));
param_ = FinancialMonteCarlo.get_parameters(Model)

FinancialMonteCarlo.set_parameters!(Model, param_)
display(Model)

@test_throws(ErrorException, FinancialMonteCarlo.set_parameters!(Model, [1, 4]))

@show FwdPrice = pricer(Model, rfCurve, mc, FwdData);
@show EuPrice = pricer(Model, rfCurve, mc, EUData);
@show AmPrice = pricer(Model, rfCurve, mc, AMData);
@show BarrierPrice = pricer(Model, rfCurve, mc, BarrierData);
@show AsianPrice1 = pricer(Model, rfCurve, mc, AsianFloatingStrikeData);
@show AsianPrice2 = pricer(Model, rfCurve, mc, AsianFixedStrikeData);

@test abs(FwdPrice - 99.1078451563562) < toll
@test abs(EuPrice - 8.43005524824866) < toll
@test abs(AmPrice - 8.450489415187354) < toll
@test abs(BarrierPrice - 7.5008664470880735) < toll
@test abs(AsianPrice1 - 4.774451704549382) < toll

@show "Test LogNormalMixture Parameters"
eta = [0.2, 0.2]
lam11 = [0.9999]
@test_throws(ErrorException, LogNormalMixture(-0.2 * ones(3), 0.1 * ones(2), Underlying(S0, d)))
@test_throws(ErrorException, LogNormalMixture(0.2 * ones(3), -0.1 * ones(2), Underlying(S0, d)))
@test_throws(ErrorException, LogNormalMixture(0.2 * ones(3), ones(2), Underlying(S0, d)))
@test_throws(ErrorException, LogNormalMixture(0.2 * ones(3), 0.2 * ones(3), Underlying(S0, d)))


# N is the number of dimensions of the prediction
abstract Loss{N}
typealias UnivariateLoss Loss{0}
typealias MultivariateLoss Loss{1}

## Abs loss (for regression)
#
#   loss(p, y) = |p - y|
#
immutable AbsLoss <: UnivariateLoss
end

value{T<:BlasReal}(::AbsLoss, p::T, y::T) = abs(p - y)
deriv{T<:BlasReal}(::AbsLoss, p::T, y::T) = sign(p - y)
value_and_deriv{T<:BlasReal}(::AbsLoss, p::T, y::T) = (r = p - y; (abs(r), sign(r)))

## Squared loss (for regression)
#
#   loss(p, y) := (1/2) * (p - y)^2
#
immutable SqrLoss <: UnivariateLoss
end

value{T<:BlasReal}(::SqrLoss, p::T, y::T) = half(abs2(p - y))
deriv{T<:BlasReal}(::SqrLoss, p::T, y::T) = p - y
value_and_deriv{T<:BlasReal}(::SqrLoss, p::T, y::T) = (r = p - y; v = half(abs2(r)); (v, r))

## Quantile loss (for quantile regression, asymmetric version of Abs loss)
#
#   loss(p, y) := t * (p - y)         ... (p >= y)
#               = (1 - t) * (y - p)   ... (p < y)
#
immutable QuantileLoss <: UnivariateLoss
    t::Float64

    function QuantileLoss(t::Real)
        zero(t) < t < one(t) || error("t must be a real value in (0, 1).")
        new(convert(Float64, t))
    end
end

function value{T<:BlasReal}(loss::QuantileLoss, p::T, y::T)
    t = convert(T, loss.t)
    p >= y ? t * (p - y) : (one(T) - t) * (y - p)
end

function deriv{T<:BlasReal}(loss::QuantileLoss, p::T, y::T)
    t = convert(T, loss.t)
    p > y ? t :
    p < y ? t - one(T) : zero(T)
end

function value_and_deriv{T<:BlasReal}(loss::QuantileLoss, p::T, y::T)
    t = convert(T, loss.t)
    if p > y
        (t * (p - y), t)
    elseif p < y
        c = t - one(T)
        (c * (p - y), c)
    else
        (zero(T), zero(T))
    end
end

## Epsilon Insensitive loss (for support vector regression)
#
#   loss(p, y) := 0                   ... abs(y - p) <= eps
#               = abs(y - p) - eps    ... otherwise
#
immutable EpsilonInsLoss <: UnivariateLoss
    epsilon::Float64

    function EpsilonInsLoss(epsilon::Real)
        new(convert(Float64, epsilon))
    end
end

function value{T<:BlasReal}(loss::EpsilonInsLoss, p::T, y::T)
    eps = convert(T, loss.epsilon)
    a = abs(p - y)
    a > eps ? a - eps : zero(T)
end

function deriv{T<:BlasReal}(loss::EpsilonInsLoss, p::T, y::T)
    eps = convert(T, loss.epsilon)
    r = p - y
    abs(r) > eps ? sign(r) : zero(T)
end

function value_and_deriv{T<:BlasReal}(loss::EpsilonInsLoss, p::T, y::T)
    eps = convert(T, loss.epsilon)
    r = p - y
    a = abs(r)
    a > eps ? (a - eps, sign(r)) : (zero(T), zero(T))
end

## Huber loss (for regression, smoothed version of Abs loss)
#
#   loss(p, y) := (1/2) * (p - y)^2      ... (|p - y| <= h)
#                 h * |p - y| - h^2/2    ... otherwise
#
immutable HuberLoss <: UnivariateLoss
    h::Float64

    function HuberLoss(h::Real)
        h > zero(h) || error("h must be a positive value.")
        new(convert(Float64, h))
    end
end

function value{T<:BlasReal}(loss::HuberLoss, p::T, y::T)
    h = convert(T, loss.h)
    a = abs(p - y)
    a <= h ? half(a * a) : h * a - half(h * h)
end

function deriv{T<:BlasReal}(loss::HuberLoss, p::T, y::T)
    h = convert(T, loss.h)
    r = p - y
    r > h ? h : r < -h ? -h : r
end

function value_and_deriv{T<:BlasReal}(loss::HuberLoss, p::T, y::T)
    h = convert(T, loss.h)
    r = p - y
    r > h ? (h * r - half(h * h), h) :
    r < -h ? (-h * r - half(h * h), -h) :
    (half(r * r), r)
end

## Hinge loss (for L1-SVM)
#
#   loss(p, y) := max(1 - y * p, 0)
#
immutable HingeLoss <: UnivariateLoss
end

value{T<:BlasReal}(::HingeLoss, p::T, y::T) = nonneg(one(T) - y * p)
deriv{T<:BlasReal}(::HingeLoss, p::T, y::T) = y * p < one(T) ? -y : zero(T)

function value_and_deriv{T<:BlasReal}(::HingeLoss, p::T, y::T)
    yp = y * p
    yp >= one(T) ? (zero(T), zero(T)) : (one(T) - yp, -y)
end

## Squared Hinge loss (for L2-SVM)
#
#   loss(p, y) := max(1 - y * p, 0)^2
#
immutable SqrHingeLoss <: UnivariateLoss
end

function value{T<:BlasReal}(::SqrHingeLoss, p::T, y::T)
    yp = y * p
    yp >= one(T) ? zero(T) : abs2(nonneg(one(T) - yp))
end

deriv{T<:BlasReal}(::SqrHingeLoss, p::T, y::T) = y * p < one(T) ? 2(p - y) : zero(T)

function value_and_deriv{T<:BlasReal}(::SqrHingeLoss, p::T, y::T)
    yp = y * p
    yp >= one(T) ? (zero(T), zero(T)) : (abs2(one(T) - yp), 2(p - y))
end

## Smoothed HingeLoss
#
#   loss(p, y) := 0              ... y * p > 1 + h
#                 1 - y * p      ... y * p < 1 - h
#                 (1 + h - y * p)^2 / 4h   ... otherwise
#
#  Reference
#
#   O. Chapelle, "Training a Support Vector Machine in the Primal", Neural Computation.
#
immutable SmoothedHingeLoss <: UnivariateLoss
    h::Float64

    function SmoothedHingeLoss(h::Real)
        h > zero(h) || error("h must be a positive value.")
        new(convert(Float64, h))
    end
end

function value{T<:BlasReal}(loss::SmoothedHingeLoss, p::T, y::T)
    h = convert(T, loss.h)
    yp = y * p
    yp >= one(T) + h ? zero(T) :
    yp <= one(T) - h ? one(T) - yp :
    abs2(one(T) + h - yp) / 4h
end

function deriv{T<:BlasReal}(loss::SmoothedHingeLoss, p::T, y::T)
    h = convert(T, loss.h)
    yp = y * p
    yp >= one(T) + h ? zero(T) :
    yp <= one(T) - h ? -y :
    y * (yp - one(T) - h) / 2h
end

function value_and_deriv{T<:BlasReal}(loss::SmoothedHingeLoss, p::T, y::T)
    h = convert(T, loss.h)
    yp = y * p
    if yp >= one(T) + h
        (zero(T), zero(T))
    elseif yp <= one(T) - h
        (one(T) - yp, -y)
    else
        z = yp - one(T) - h
        (abs2(z) / 4h, y * z / 2h)
    end
end

## Logistic loss (for logistic regression)
#
#   loss(p, y) := log(1 + exp(-y * p))
#
immutable LogisticLoss <: UnivariateLoss
end

value{T<:BlasReal}(::LogisticLoss, p::T, y::T) =
    (yp = y * p; yp >= zero(T) ? log1p(exp(-yp)) : log1p(exp(yp)) - yp)

function deriv{T<:BlasReal}(::LogisticLoss, p::T, y::T)
    yp = y * p
    if yp >= zero(T)
        e = exp(-yp)
        -y * e / (one(T) + e)
    else
        -y / (one(T) + exp(yp))
    end
end

function value_and_deriv{T<:BlasReal}(::LogisticLoss, p::T, y::T)
    yp = y * p
    if yp >= zero(T)
        e = exp(-yp)
        (log1p(e), -y * e / (one(T) + e))
    else
        e = exp(yp)
        (log1p(e) - yp, -y / (one(T) + e))
    end
end

## general functions for multivariate loss

grad{T<:BlasReal}(loss::MultivariateLoss, p::StridedVector{T}, y) =
    grad!(loss, zeros(T, length(p)), p, y)

value_and_grad{T<:BlasReal}(loss::MultivariateLoss, p::StridedVector{T}, y) =
    value_and_grad!(loss, zeros(T, length(p)), p, y)

## SumLoss
#
#   loss(p, y) := sum_k loss.intern(p[k], y[k])
#
immutable SumLoss{L<:UnivariateLoss} <: MultivariateLoss
    intern::L
end

SumLoss{L<:UnivariateLoss}(loss::L) = SumLoss{L}(loss)

typealias SumSqrLoss SumLoss{SqrLoss}
SumSqrLoss() = SumLoss{SqrLoss}(SqrLoss())

function value{T<:BlasReal}(s::SumLoss, p::StridedVector{T}, y::StridedVector{T})
    loss = s.intern
    k = length(p)
    @_checkdims k == length(y)
    s = value(loss, p[1], y[1])
    @inbounds for i = 2:k
        s += value(loss, p[i], y[i])
    end
    s
end

function grad!{T<:BlasReal}(s::SumLoss, g::StridedVector{T}, p::StridedVector{T}, y::StridedVector{T})
    loss = s.intern
    k = length(p)
    @_checkdims k == length(g) == length(y)
    @inbounds for i = 1:k
        g[i] = deriv(loss, p[i], y[i])
    end
    g
end

function value_and_grad!{T<:BlasReal}(s::SumLoss, g::StridedVector{T}, p::StridedVector{T}, y::StridedVector{T})
    loss = s.intern
    k = length(p)
    @_checkdims k == length(g) == length(y)
    s, dv = value_and_deriv(loss, p[1], y[1])
    g[1] = dv
    @inbounds for i = 2:k
        v, dv = value_and_deriv(loss, p[i], y[i])
        s += v
        g[i] = dv
    end
    (s, g)
end

## Multinomial logistic loss (for Multinomial logistic regression)
#
#   loss(p, y) := log(sum_k exp(p[k])) - p[y]
#
immutable MultiLogisticLoss <: MultivariateLoss
end

function value{T<:BlasReal}(::MultiLogisticLoss, p::StridedVector{T}, y::Integer)
    k = length(p)
    pmax = maximum(p)
    s = zero(T)
    @inbounds for i = 1:k
        s += exp(p[i] - pmax)
    end
    pmax + log(s) - p[y]
end

function grad!{T<:BlasReal}(::MultiLogisticLoss, g::StridedVector{T}, p::StridedVector{T}, y::Integer)
    k = length(p)
    @_checkdims length(g) == k
    pmax = maximum(p)
    s = zero(T)
    @inbounds for i = 1:k
        pi = exp(p[i] - pmax)
        g[i] = pi
        s += pi
    end
    @inbounds for i = 1:k
        g[i] /= s
    end
    g[y] -= one(T)
    g
end

function value_and_grad!{T<:BlasReal}(::MultiLogisticLoss, g::StridedVector{T}, p::StridedVector{T}, y::Integer)
    k = length(p)
    @_checkdims length(g) == k

    pmax = maximum(p)
    p_y = p[y]  # g and p can be the same array, so have to cache p[y] before p is overwritten

    s = zero(T)
    @inbounds for i = 1:k
        pi = exp(p[i] - pmax)
        g[i] = pi
        s += pi
    end

    @inbounds for i = 1:k
        g[i] /= s
    end
    g[y] -= one(T)
    return (pmax - p_y + log(s), g)
end


using BenchmarkTools
using Random
using Test

@testset "algorithmic order" begin
    ##
    ## The goal of this problem is to get you thinking about algorithmic order.
    ##
    # There's no src/ code for this problem, write everything here.
    # Note that `@elapsed` is like time and is simple in that it just runs your operation once.
    # You may need to execute this file more than once in case compilation time complicates the interpretation.

    str = randstring(10^6)
    d = Dict{Char,Int}()
    t = @elapsed for c in ['a':'z'; 'A':'Z'; '0':'9']
        d[c] = count(==(c), str)
    end

    d2 = Dict{Char,Int}()
    t2 = @elapsed begin
        for ch in ['a':'z'; 'A':'Z'; '0':'9']
            d2[ch] = 0
        end
        for ch in str
            d2[ch] += 1
        end# ...
    end
    @test t2 < t
    # Since it's a discrete list of values, can you do even better with an array?
    # Hint: try `Int('z')` for ideas.
    """So comparison between integers are faster than comparisons between str or chars. """
    counter = zeros(Int, 26*2+10)
    t3 = @elapsed begin
        chint_list = Int.(collect(str))
        for (i, ch) in enumerate(['a':'z'; 'A':'Z'; '0':'9'])
            counter[i] = sum(chint_list==Int(ch))
        end
    end
    @test t3 < t2
    println("t1 $t;\t t2 $t2;\t t3 $t3")
end


using PyCall, PyPlot, Random
Random.seed!(5)
sp = pyimport("scipy.interpolate")
x = LinRange(0,1,50)
y = (0.9 .+ 0.1rand(length(x))).*sin.(2*pi*(x.-0.5))
t = collect(x[2:2:end-1])
s1 = sp.LSQUnivariateSpline(x, y, t)
x2 = LinRange(0, 1, 201) # new x-grid
y2 = s1(x2) # evaluate spline on that new grid
figure()
plot(x,y,label="original")
plot(x2,y2,label="interp", color="k")
knots = s1.get_knots()
c = s1.get_coeffs()
newknots(knots, k) = vcat(fill(knots[1],k),knots,fill(knots[end],k))
forscipyknots = newknots(knots, 3)
s2 = sp.BSpline(forscipyknots, c, 3)
y3 = s2(x2)
plot(x2,y3,"--r", label="reconstructed \nfrom knots and coeff")
legend()
##
knots = [0,.4,.4,.4,.4,.7,1]
c = [2,-5,5,2,-3,-1,2,1.5]
forscipyknots = newknots(knots, 3)
s2 = sp.BSpline(forscipyknots, c, 3)
figure()
plot(x2, s2(x2))
##
using Images
img = convert(Array{Float64, 2}, Gray.(load("func.png")))


function RouwenNormalize!(A)
    A[2:end-1, :] ./= 2
    nothing
end

function rouwenupdate!(Πold, Πnew, p, q)
    Nold = size(Πold, 1)
    Πnew[1:Nold, 1:Nold] = p*Πold
    Πnew[1:Nold, 2:(Nold+1)] .+= (1-p)*Πold
    Πnew[2:(Nold+1), 1:Nold] .+= (1-q)*Πold
    Πnew[2:(Nold+1), 2:(Nold+1)] .+= q*Πold
    nothing
end

function rouwenmat(N, ρ, p, q)
    initmat = [p 1-p;1-q q]
    if N==2
        ret = initmat
    else
        currmat = initmat
        for n = 3:N
            nextmat = fill(zero(ρ), (n, n))
            rouwenupdate!(currmat, nextmat, p, q)
            currmat = nextmat
            RouwenNormalize!(currmat)
        end
        ret = currmat
    end
    return(ret)
end

"""
    rouwenhorst(npts, ρ, σ)

Discretizes AR(1) process `y′ = ρ y +  σ ϵ`.

# Example

npts = 10
rho = 0.9
sig = 0.01
points, Π = rouwenhorst(npts, rho, sig)

"""
function rouwenhorst(npts, ρ, σ)
    ω = σ/sqrt(1-ρ^2) # long run std dev
    # println("long run variance: $ω")
    q = (1+ρ)/2

    points = range(-ω*sqrt(npts - 1),
                   ω*sqrt(npts-1),
                   length=npts)

    # points = GenGrid(uniform(npts, bds))
    Π = (npts > 1) ? rouwenmat(npts, ρ, q, q) : fill(one(ρ), (1,1))

    return points, Π
end

"Returns a MarkovChain with discretized AR(1) from applying `rouwenhorst`"
rouwenhorst_mc(npts, ρ, σ)=MarkovChain(rouwenhorst(npts, ρ, σ)...)


function check_symplecticity(tab::Tableau{T}; atol=16eps(T), rtol=16eps(T)) where {T}
    a, b = tab.a, tab.b
    [isapprox(b[i] * a[i,j] + b[j] * a[j,i], b[i] * b[j]; atol=atol, rtol=rtol) for i in axes(a,1), j in axes(a,2)]
end

function issymplectic(tab::Tableau; kwargs...)
    all(check_symplecticity(tab; kwargs...))
end

function check_symplecticity(tab::PartitionedTableau{T}; atol=16eps(T), rtol=16eps(T)) where {T}
    a, b = tab.q.a, tab.q.b
    ā, b̄ = tab.p.a, tab.p.b
    ([isapprox(b[i] * ā[i,j] + b̄[j] * a[j,i], b[i] * b̄[j]; atol=atol, rtol=rtol) for i in axes(a,1), j in axes(a,2)],
     [isapprox(b[i], b̄[i]; atol=atol, rtol=rtol) for i in eachindex(b,b̄)])
end

function issymplectic(tab::PartitionedTableau; kwargs...)
    all(all.(check_symplecticity(tab; kwargs...)))
end

function compute_symplecticity_error(tab::Tableau)
    a, b = tab.a, tab.b
    [b[i] * a[i,j] + b[j] * a[j,i] - b[i] * b[j] for i in axes(a,1), j in axes(a,2)]
end

function get_symplectic_conjugate_coefficients!(a̅::AbstractMatrix{T}, a::AbstractMatrix{T}, b::AbstractVector{T}) where {T}
    @assert size(a) == size(a̅)
    @assert length(b) == size(a,1) == size(a,2)

    for i in axes(a̅, 1)
        for j in axes(a̅, 2)
            a̅[i,j] = b[j] / b[i] * ( b[i] - a[j,i] )
        end
    end

    return a̅
end

get_symplectic_conjugate_coefficients(a, b) = get_symplectic_conjugate_coefficients!(zero(a), a, b)
get_symplectic_conjugate_coefficients(tab::Tableau) = Tableau(tab.name, tab.s, get_symplectic_conjugate_coefficients(tab.a, tab.b), tab.b, tab.c)

function symplecticize(tab::Tableau; name=nothing, T=Float64)
    a̅ = get_symplectic_conjugate_coefficients(tab.a, tab.b)
    Tableau{T}(name === nothing ? Symbol(tab.name, "S") : name, tab.o, (tab.a .+ a̅) ./ 2, tab.b, tab.c)
end


using NumericalMethodsforEngineers, IterativeSolvers

a = [16. 4. 8.; 4. 5. -4.; 8. -4. 22.]
b = [4., 2., 5.]

upper = chol(a)
lower = upper'
y = lower \ b
c = upper \ y

tol = 1.0e-10
iters = 20
x,ch = cg(a, b;tol=tol,maxiter=iters);

println("\nSolution Vector: $x")
println("Converged: $(ch.isconverged)")
println("Number of iters: $(ch.mvps)")

@test round(x, 9) == round(c, 9)