# ------------------------------------------------------------------------------------------
# ## Intro to neurons
#
# At this point, we know how to build models and have a computer automatically learn how to
# match the model to data. This is the core of how any machine learning method works.
#
# Now, let's narrow our focus and look at **neural networks**. Neural networks (or "neural
# nets", for short) are a specific choice of a **model**. It's a network made up of
# **neurons**; this, in turn, leads to the question, "what is a neuron?"
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# ### Models with multiple inputs
#
# So far, we have been using the sigmoid function as our model. One of the forms of the
# sigmoid function we've used is
#
# $$\sigma_{w, b}(x) = \frac{1}{1 + \exp(-wx + b)},$$
#
# where `x` and `w` are both single numbers. We have been using this function to model how
# the amount of the color green in an image (`x`) can be used to determine if an image shows
# an apple or a banana.
#
# But what if we had multiple data features we wanted to fit?
#
# We could then extend our model to include multiple features like
#
# $$\sigma_{\mathbf{w},b}(\mathbf{x}) = \frac{1}{1 + \exp(-w_1 x_1 - w_2 x_2 - \cdots - w_n
# x_n + b)}$$
#
# Note that now $\mathbf{x}$ and $\mathbf{w}$ are vectors with many components, rather than
# a single number.
#
# For example, $x_1$ might be the amount of the color green in an image, $x_2$ could be the
# height of the object in the picture, $x_3$ could be the width, and so forth. We can add
# information for as many features as we have! Our model now has more parameters, but the
# same idea of gradient descent ("rolling the ball downhill") will still work to train our
# model.
#
# This version of the sigmoid model that takes multiple inputs is an example of a
# **neuron**.
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# In the video, we see that one huge class of learning techniques is based around neurons,
# that is, *artificial neurons*. These are caricatures of real, biological neurons. Both
# *artificial* and *biological* neurons have several inputs $x_1, \ldots, x_n$, and a single
# output, $y$. Schematically they look like this:
# ------------------------------------------------------------------------------------------

include("draw_neural_net.jl")

number_inputs, number_neurons = 4, 1

draw_network([number_inputs, number_neurons])

# ------------------------------------------------------------------------------------------
# We should read this as showing how information flows from left to right:
# - 4 pieces of input information arrive (shown in green on the left);
#
# - the neuron (shown in red on the right) receives all the inputs, processes them, and
# outputs a single number to the right.
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# In other words, a neuron is just a type of function that takes multiple inputs and returns
# a single output.
#
# The simplest interesting case that we will look at in this notebook is when there are just
# two pieces of input data:
# ------------------------------------------------------------------------------------------

draw_network([2, 1])

# ------------------------------------------------------------------------------------------
# Each link between circles above represents a **weight** $w$ that can be modified to allow
# the neuron to learn, so in this case the neuron has two weights, $w_1$ and $w_2$.
#
# The neuron also has a single bias $b$, and an **activation function**, which we will take
# to be the $\sigma$ sigmoidal function that we have been using. (Note that other activation
# functions can be used!)
#
# Let's call our neuron $f_{w_1,w_2, b}(x_1, x_2)$, where
#
# $$f_{w_1,w_2, b}(x_1, x_2) := \sigma(w_1 x_1 + w_2 x_2 + b).$$
#
# Note that $f_{w_1,w_2, b}(x_1, x_2)$ has 3 parameters: two weights and a bias.
#
# To simplify the notation, and to prepare for more complicated scenarios later, we put the
# two weights into a vector, and the two data values into another vector:
#
# $$
# \mathbf{w} = \begin{pmatrix} w_1 \\ w_2 \end{pmatrix};
# \qquad
# \mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}.
# $$
#
# We thus have
#
# $$f_{\mathbf{w}, b}(\mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b),$$
#
# where the dot product $\mathbf{w} \cdot \mathbf{x}$ is an abbreviated syntax for $w_1 x_1
# + w_2 x_2$.
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# #### Exercise 1
#
# Declare the function `f(x, w, b)` in Julia. `f` should take vectors `x` and `w` as vectors
# and `b` as a scalar. Furthermore `f` should call
#
# ```julia
# σ(x) = 1 / (1 + exp(-x))
# ```
#
# What output do you get for
#
# ```julia
# f(3, 4, 5)
# ```
# ?
# ------------------------------------------------------------------------------------------

.nbexports/introductory-tutorials/broader-topics-and-ecosystem/intro-to-ml/11. ML - Intro to neurons.jl


using Turing
using Distributions
using Test

@model gdemo(x) = begin
  s ~ InverseGamma(2, 3)
  # println(s)
  m ~ Normal(0, sqrt(s))
  # println(m)
  x[1] ~ Normal(m, sqrt(s))
  x[2] ~ Normal(m, sqrt(s))
  # println(vi)
  return s, m
end

alg = HMC(3000, 0.15, 10)

res = sample(gdemo([1.5, 2.0]), alg)

println(mean(res[:s])," ≈ ", 49/24, "?")
println(mean(res[:m])," ≈ ", 7/6, "?")
@test mean(res[:s]) ≈ 49/24 atol=0.2
@test mean(res[:m]) ≈ 7/6 atol=0.2


using Plots;
using Printf;
gr();
anim = Animation();
f(x) = 1/2*cos(8pi*(x-1/2));

function forward(u_old::Array, i::Int64)
	return (3u_old[ir[i]] + 3u_old[i] - 7u_old[il[i]] + u_old[il2[i]]) / 8;
end

function backward(u_old::Array, i::Int64)
	return (3u_old[il[i]] + 3u_old[i] - 7u_old[ir[i]] + u_old[ir2[i]]) / 8;
end

function main()
#=********Numerical Set Up*********=#
	global c = 1.0;
	global nx = 101;
	global lx = 100;
	global x = range(0.0, stop=lx, length=nx);
	global dx = lx /(nx-1);
	global u  = zeros(Float64, nx);
	@inbounds for i = 1:nx
		if 40.0 < x[i] < 60.0;
			u[i] = f(x[i]);
		end
	end
	global t = 0.0;
	global dt = 2e-2;
	global tlims = 300;
	global C = dt * c / dx;
	global ir = zeros(Int64, nx);
	global ir2 = zeros(Int64, nx);
	global il = zeros(Int64, nx);
	global il2 = zeros(Int64, nx);
	@inbounds for i = 1:nx
		ir[i] = i + 1;
		ir2[i] = i + 2;
		il[i] = i - 1;
		il2[i] = i - 2;
	end
	ir[end] = 1;
	ir2[end - 1] = 1;
	ir2[end] = 2;
	il[begin] = nx;
	il2[begin + 1] = nx;
	il2[begin] = nx - 1;
#=******Finish Numerical Set Up*************=#

	while t < tlims
		u_old = copy(u);
		##First substep
		@inbounds for i = 1:nx-1
			u[i] = u_old[i] - C * forward(u_old, i);
		end
		u[end] = u[begin];

		##Second substep
		@inbounds for i = 1:nx
			u_old[i] = 0.5*(u_old[i] + u[i]) - 0.5*C/8.0*(forward(u, i));
		end
		u_old[end] = u_old[begin];

		u = copy(u_old);

		plt = plot(x, u, xlims=(0, lx), xticks=0:lx/5:lx, ylims=(-0.2, 1.2), yticks=-0.2:0.2:1.2,
		xlabel="x", ylabel="u", color="blue", title="QUICK method by using Heun", legend=false);
		frame(anim, plt);
		t += dt;
		@printf("t = %.4f\n", t);
	end
	gif(anim, "quick.gif", fps=5);
end

main();


module FakeSegmentations
using LinearAlgebra

const ZERO_UINT32 = UInt32(0)
const ONE_UINT32 = UInt32(1)
const RADIUS = Float32(10)
const HEIGHT = 100
const THICKNESS = 10

function cylinder(;radius::Float32=RADIUS, height::Integer=HEIGHT)
    width = Int( radius*2 + 1 )
    center = Int(radius + 1)
    seg = zeros(UInt32, (width,width,height))
    for x in Int(center-radius):Int(center+radius)
        for y in Int(center-radius):Int(center+radius)
            if norm([x,y].-[center, center]) < radius
                seg[x,y,:] .= ONE_UINT32
            end
        end
    end
    seg
end

function broken_cylinder(;radius::Float32=RADIUS, height::Integer=HEIGHT,
                         breakStartZ::Integer=div(height,2),
                         thickness::Integer = THICKNESS )
    seg = cylinder(radius=radius, height=height)
    # create a breaking part
    seg[ :,:, breakStartZ : breakStartZ+10 ] .= ZERO_UINT32
    seg
end

function ring(; centerLineRadius::Integer = 50, ringRadius::Integer=5)
    seg = zeros(UInt32, (2*centerLineRadius+2*ringRadius + 5,
                         2*centerLineRadius+2*ringRadius + 5, 2*ringRadius + 5) )
    center = [map(x->div(x,2), size(seg))...]
    for y in 1:size(seg, 2)
        for x in 1:size(seg, 1)
            d = norm([x,y] .- center[1:2])
            if abs(d-centerLineRadius) < 2
                # this point is on the center line of the ring
                for rz in center[3]-ringRadius : center[3]+ringRadius
                    for ry in y-ringRadius : y+ringRadius
                        for rx in x-ringRadius : x+ringRadius
                            d2 = norm([x,y,center[3]] .- [rx,ry,rz])
                            if d2 <= ringRadius
                                seg[rx,ry,rz] = ONE_UINT32
                            end
                        end
                    end
                end
            end
        end
    end
    @assert ndims(seg) == 3
    seg
end

function broken_ring(; centerLineRadius::Integer = 50, ringRadius::Integer=5)
    seg = ring(; centerLineRadius = centerLineRadius, ringRadius = ringRadius)
    center = map(x->div(x,2), size(seg))
    d = div(centerLineRadius, 10)
    seg[center[1]-d:center[1]+d, :, :] .= ZERO_UINT32
    seg[:, center[2]-d:center[2]+d, :] .= ZERO_UINT32
    seg
end

function Y_shape()
    error("unimplemented")
end

end # module


wdot(wx, x, wy, y) = dot(wx.*x, wy.*y)
wnorm(w, x) = norm(w.*x)

assess_convergence(f, ftol) = assess_convergence(NaN, NaN, f, NaN, ftol)
function assess_convergence(x,
                            x_previous,
                            f,
                            xtol,
                            ftol)
    x_converged, f_converged = false, false
    if norm(x-x_previous) <= xtol
        x_converged = true
    end
    if maximum(abs, f) <= ftol
        f_converged = true
    end

    return x_converged, f_converged
end

function check_isfinite(x::AbstractArray)
    if any(!isfinite, x)
        i = findall(!isfinite, x)
        throw(IsFiniteException(i))
    end
end

"""
    qrdelete!(Q, R, k)

Delete the left-most column of F = Q[:, 1:k] * R[1:k, 1:k] by updating Q and R.

Only Q[:, 1:(k-1)] and R[1:(k-1), 1:(k-1)] are valid on exit.
"""
function qrdelete!(Q::AbstractMatrix, R::AbstractMatrix, k::Int)
  n, m = size(Q)
  m == LinearAlgebra.checksquare(R) || throw(DimensionMismatch())
  1 ≤ k ≤ m || throw(ArgumentError())

  # apply Givens rotations
  for i in 2:k
      g = first(givens(R, i - 1, i, i))
      lmul!(g, R)
      rmul!(Q, g')
  end

  # move columns of R
  @inbounds for j in 1:(k-1)
    for i in 1:(k-1)
      R[i, j] = R[i, j + 1]
    end
  end

  Q, R
end

"""
    qradd!(Q, R, v, k)

Replace the right-most column of F = Q[:, 1:k] * R[1:k, 1:k] with v by updating Q and R.

This implementation modifies vector v as well. Only Q[:, 1:k] and R[1:k, 1:k] are valid on
exit.
"""
function qradd!(Q::AbstractMatrix, R::AbstractMatrix, v::AbstractVector, k::Int)
  n, m = size(Q)
  n == length(v) || throw(DimensionMismatch())
  m == LinearAlgebra.checksquare(R) || throw(DimensionMismatch())
  1 ≤ k ≤ m || throw(ArgumentError())

  @inbounds for i in 1:(k-1)
    q = view(Q, :, i)
    r = dot(q, v)

    R[i, k] = r
    axpy!(-r, q, v)
  end

  @inbounds begin
    d = norm(v)
    R[k, k] = d
    @. Q[:, k] = v / d
  end

  Q, R
end


scalaradd(a, b) = a + b

function sum(v)
  s = 0.0
  N = length(v)
  for i = 1:N
      s += v[i]
  end
  return s
end

# y = dot(a,b)
function dot(a, b)
  s = 0.0
  N = length(a)
  for i = 1:N
      s += a[i] * b[i]
  end
  return s
end

function tuplereturn(x)
  a = x * 1.2
  b = x * 1.3
  return a, b
end

function tuplecall(x)
  ab = tuplereturn(x)
  r = ab[1]
  #r = getfield(ab,1)
  return r
end

# y = W*X
mul(W, X) = mul_(zeros(size(W,1), size(X,2)), W, X)
function mul_(y, W, X)
    szW = size(W)
    szX = size(X)
    s = 0.0
    for n = 1:szX[2]
        for i = 1:szW[1]
            s = 0.0
            for j = 1:szW[2]
                s += W[i,j] * X[j,n]
            end
            y[i,n] = s
        end
    end
    return y
end

# y = W*X .+ b
muladd(W, X, b) = muladd_(zeros(size(W,1), size(X,2)), W, X, b)
function muladd_(y, W, X, b)
    szW = size(W)
    szX = size(X)
    s = 0.0
    for n = 1:szX[2]
        for i = 1:szW[1]
            s = 0.0
            for j = 1:szW[2]
                s += W[i,j] * X[j,n]
            end
            y[i,n] = s + b[i]
        end
    end
    return y
end

# y = a .+ b
add(a, b) = add_(zeros(size(a)), a, b)
function add_(r, a, b)
  N = length(r)
  for i = 1:N
      r[i] = a[i] + b[i]
  end
  return r
end

function exports()
  scalaradd(1.0, 2.0)

  a=randn(3,1)
  b=randn(3,1)
  dot(a,b)
  res = add(a,b)

  tuplereturn(1.2)
  tuplecall(2.1)

  M1=randn(4,2)
  M2=randn(2,5)
  r = mul(M1,M2)

  b = randn(4,1)
  r2 = muladd(M1,M2, b)
  sum(a)
end

exports()


"""
	OrderedPair

OrderedPair representation.
Ex:
```
julia> o = OrderedPair(1,3.2)
(1,3.2)
```
"""
mutable struct OrderedPair
	left::Real
	right::Real
	OrderedPair(left,right) = left > right ? error("OrderedPair must be ordered") : new(left,right)
end

"""
	string(p)

Returns a string representation of an `OrderedPair`.
"""
function string(p::OrderedPair)
	"($(p.left),$(p.right))"
end

"""
	show(io,p)

Prints the string representation of an `OrderedPair`.
"""
function show(io::IO, p::OrderedPair)
	println(string(p))
end

"""
	o1 == o2

Equality defined for `OrderedPair`.
"""
function ==(o1::OrderedPair, o2::OrderedPair)
	(o1.left == o2.left) && (o1.right == o2.right)
end

"""
	o1 < o2

Lexicographical order for `OrderedPair`.
"""
function isless(o1::OrderedPair, o2::OrderedPair)
	o1.left == o2.left ? o1.right < o2.right : o1.left < o2.left
end

"""
	Interval

`Interval` representation.
Attributes:
	- limits::OrderedPair : The left and right limits of the interval.
	- open_left::Bool : Set to true if the interval is open on the left.
	- open_right::Bool : Set to true if the interval is open on the right.
Ex:
```
julia> i = Interval(0,1)
[0,1]
julia> j = Interval(1,true,3.4)
]1,3.4]
julia> k = Interval(-1,true,3.12,true)
]-1,3.12[
```
"""
mutable struct Interval
    limits::OrderedPair
    open_left::Bool
    open_right::Bool
end

"""
	i1 == i2

Equality defined for `Interval`.
"""
function ==(i1::Interval, i2::Interval)
	(i1.limits == i2.limits) && (i1.open_left == i2.open_left) && (i1.open_right == i2.open_right)
end

"""
	i1 < i2

Lexicographical order for `Interval`.
Ex:
```
julia> Interval(0,2) < Interval(1,2,true)
true
julia> Interval(1,3) > Interval(-1,10)
true
julia> Interval(0,1) < Interval(0,1,true)
true
```
"""
function isless(i1::Interval, i2::Interval)
	if left(i1) < left(i2)
        return true
    elseif left(i1) == left(i2)
        if !i1.open_left & i2.open_left
        	return true
        elseif i1.open_left & !i2.open_left
        	return false
        else
        	if right(i1) < right(i2)
        		return true
        	elseif right(i1) == right(i2)
        		if !i1.open_right & i2.open_right
        			return true
        		else
        			return false
        		end
        	else
        		return false
        	end
        end
    else
        return false
    end
end

"""
	Interval(left, open_left, right, open_right)

Full constructor for `Interval`.
Requires:
	- left::Real Left limit of the interval.
	- open_left::Bool Set to true if the interval is open on the left.
	- right::Real Right limit of the interval.
	- open_right::Bool Set to true if the interval is open on the left.
Ex:
```
julia> i = Interval(0,false,1,true)
[0,1[
julia> j = Interval(1,false,2,false)
[1,2]
```
"""
function Interval(left::Real, open_left::Bool, right::Real, open_right::Bool)
	p = OrderedPair(left,right)
	Interval(p,open_left,open_right)
end

"""
	Interval(left, right)

Simplified constructor for closed `Interval`.
Requires:
	- left::Real Left limit of the interval.
	- right::Real Right limit of the interval.
Ex:
```
julia> i = Interval(0,2)
[0,2]
```
"""
function Interval(left::Real, right::Real)
	p = OrderedPair(left,right)
	Interval(p,false,false)
end

"""
	Interval(left, open_left, right)

Simplified constructor for `Interval` closed on the right.
Requires:
	- left::Real Left limit of the interval.
	- open_left::Bool Set to true if the interval is open on the left.
	- right::Real Right limit of the interval.
Ex:
```
julia> i = Interval(2,true,3)
]2,3]
```
"""
function Interval(left::Real, open_left::Bool, right::Real)
	p = OrderedPair(left,right)
	Interval(p,open_left,false)
end

"""
	Interval(left, right, open_right)

Simplified constructor for `Interval` closed on the left.
Requires:
	- left::Real Left limit of the interval.
	- right::Real Right limit of the interval.
	- open_right::Bool Set to true if the interval is open on the right.
Ex:
```
julia> i = Interval(-1,2,true)
[-1,2[
```
"""
function Interval(left::Real, right::Real, open_right::Bool)
	p = OrderedPair(left,right)
	Interval(p,false,open_right)
end

"""
	Interval()

Empty `Interval`.
Ex:
```
julia> i = Interval()
∅
```
"""
function Interval()
	Interval(0,true,0,true)
end

"""
	left(i)

Returns the left limit of the given `Interval`.
Ex:
```
julia> i = Interval(0,2)
[0,2]
julia> left(i)
0
```
"""
function left(i::Interval)
	i.limits.left
end

"""
	right(i)

Returns the right limit of the given `Interval`.
Ex:
```
julia> i = Interval(0,2)
[0,2]
julia> right(i)
2
```
"""
function right(i::Interval)
	i.limits.right
end

"""
	empty(i)

Returns true if the given `Interval` is empty.
Ex:
```
julia> i = Interval(0,1)
[0,1]
julia> empty(i)
false
julia> j = Interval()
∅
julia> empty(j)
true
```
"""
function empty(i::Interval)
	(left(i) == right(i)) && i.open_left && i.open_right
end

"""
	string(i)

Returns a string representation of an `Interval`.
Ex:
```
julia> i = Interval(0,1)
[0,1]
julia> string(i)
"[0,1]"
```
"""
function string(i::Interval)
	empty(i) && return "∅"
	if i.open_left
        i.open_right ? "]$(left(i)),$(right(i))[" : "]$(left(i)),$(right(i))]"
    else
        i.open_right ? "[$(left(i)),$(right(i))[" : "[$(left(i)),$(right(i))]"
    end
end

"""
	show(io,i)

Prints the string representation of an `Interval`.
"""
function show(io::IO, i::Interval)
	println(string(i))
end

"""
	x ∈ i

Returns true if x ∈ i and false if x ∉ i.
Ex:
```
julia> i = Interval(-1,true,2)
]-1,2]
julia> 0 ∈ i
true
julia> 2 ∈ i
true
julia> -1 ∈ i
false
julia> -0.9999999 ∈ i
true
```
"""
function ∈(x::Real, i::Interval)
    if i.open_left
        i.open_right ? left(i) < x < right(i) : left(i) < x <= right(i)
    else
        i.open_right ? left(i) <= x < right(i) : left(i) <= x <= right(i)
    end
end

"""
	i1 ⊆ i2

Returns true if the interval i1 is included in the interval i2.
Ex:
```
julia> i = Interval(0,3)
[0,3]
julia> j = Interval(1,2)
[1,2]
julia> j ⊆ i
true
julia> k = Interval(0,true,3,true)
]0,3[
julia> k ⊆ i
true
julia> i ⊈ k
true
```
"""
function ⊆(i1::Interval, i2::Interval)
    if i2.open_left
        if i2.open_right
        	if i1.open_left
        		c1 = left(i2) <= left(i1)
        	else
        		c1 = left(i2) < left(i1)
        	end
        	if i1.open_right
        		c2 = right(i1) <= right(i2)
        	else
        		c2 = right(i1) < right(i2)
        	end
        	return c1 && c2
        else
        	if i1.open_left
        		c1 = left(i2) <= left(i1)
        	else
        		c1 = left(i2) < left(i1)
        	end
        	return c1 && (right(i1) <= right(i2))
        end
    else
        if i2.open_right
        	if i1.open_right
        		c2 = right(i1) <= right(i2)
        	else
        		c2 = right(i1) < right(i2)
        	end
        	return (left(i2) <= left(i1)) && c2
        else
        	return (left(i2) <= left(i1)) && (right(i1) <= right(i2))
        end
    end
end

"""
	cardinal(i)

Returns the cardinal of the given `Interval` defined as:
```math
\\left| [a,b] \\right| = b - a
```
**Warning:** An `Interval` can be non empty and have a cardinal of zero (ex: [1,1]).
Ex:
```
julia> i = Interval(1,true,3)
]1,3]
julia> cardinal(i)
2
julia> cardinal(Interval(1,1))
0
```
"""
function cardinal(i::Interval)
	right(i) - left(i)
end

"""
	disjoint(i1,i2)

Returns true if interval i1 and interval i2 are disjoint, false otherwise.
Ex:
```
julia> i = Interval(0,1,true)
[0,1[
julia> j = Interval(1,true,2)
]1,2]
julia> disjoint(i,j)
true
julia> k = Interval(-1,0)
]-1,0]
julia> disjoint(i,k)
false
```
"""
function disjoint(i1::Interval, i2::Interval)
	i1 == i2 && return false
	(empty(i1) || empty(i2)) && return false
	if i1 < i2
		if right(i1) < left(i2)
			return true
		elseif right(i1) == left(i2)
			(i1.open_right || i2.open_left) ? (return true) : (return false)
		else
			return false
		end
	else
		disjoint(i2,i1)
	end
end

"""
	i1 ∩ i2

Returns the `Interval` corresponding to the intersection between `Interval` i1 and `Interval` i2.
If i1 and i2 are disjoint, this function returns an empty `Interval`.
Ex:
```
julia> i = Interval(0,1,true)
[0,1[
julia> j = Interval(0.8,true,1.2)
]0.8,1.2]
julia> i ∩ j
]0.8,1[
```
"""
function ∩(i1::Interval, i2::Interval)
    disjoint(i1,i2) && return Interval()
    (empty(i1) || empty(i2)) && return Interval()

    l::Real = left(i2)
    open_left::Bool = i2.open_left
    r::Real = right(i2)
    open_right::Bool = i2.open_right

    if left(i1) > left(i2)
        l = left(i1)
        open_left = i1.open_left
    elseif left(i1) == left(i2)
        open_left = i1.open_left | i2.open_left
    end

    if right(i1) < right(i2)
        r = right(i1)
        open_right = i1.open_right
    elseif right(i1) == right(i2)
        open_right = i1.open_right | i2.open_right
    end

    l == r ? Interval(OrderedPair(l,r),false,false) : Interval(OrderedPair(l,r),open_left,open_right)
end

"""
	i1 ∪ i2

Returns the `Interval` corresponding to the union between `Interval` i1 and `Interval` i2.
If i1 and i2 are disjoint, this function raises an error since the results wouldn't be an `Interval`.
Please use the `IntervalUnion` type to work with unions of disjoint intervals.
Ex:
```
julia> i = Interval(0,1,true)
[0,1[
julia> j = Interval(0.8,true,1.2)
]0.8,1.2]
julia> i ∪ j
[0,1.2]
```
"""
function ∪(i1::Interval,i2::Interval)
    if disjoint(i1,i2)
    	if right(i1)==left(i2) && (!i1.open_right || !i2.open_left)
    		return Interval(left(i1),i1.open_left,right(i2),i2.open_right)
    	elseif right(i2)==left(i1) && (!i2.open_right || !i1.open_left)
    		return Interval(left(i2),i2.open_left,right(i1),i1.open_right)
    	else
	        throw("Interval $(string(i1)) and $(string(i2)) are disjoints.")
	    end
    end
    l::Real = left(i2)
    open_left::Bool = i2.open_left
    r::Real = right(i2)
    open_right::Bool = i2.open_right

    if left(i1) < left(i2)
        l = left(i1)
        open_left = i1.open_left
    elseif left(i1) == left(i2)
        open_left = i1.open_left & i2.open_left
    end

    if right(i1) > right(i2)
        r = right(i1)
        open_right = i1.open_right
    elseif right(i1) == right(i2)
        open_right = i1.open_right & i2.open_right
    end

    l == r ? Interval(OrderedPair(l,r),false,false) : Interval(OrderedPair(l,r),open_left,open_right)
end

"""
	setdiff(i1,i2)

Setdiff is not implemented for `Interval`s.
Please use `IntervalUnion`s instead.
"""
function setdiff(i1::Interval, i2::Interval)
	throw("Not implemented.")
end

"""
	symdiff(i1,i2)

Symdiff is not implemented for `Interval`s.
Please use `IntervalUnion`s instead.
"""
function symdiff(i1::Interval, i2::Interval)
	throw("Not implemented.")
end

"""
	compact(i)

Returns the compact of the given `Interval`.
Ex:
```
julia> i = Interval(0,true,1)
]0,1]
julia> compact(i)
[0,1]
```
"""
function compact(i::Interval)
	empty(i) && return Interval()
	Interval(left(i),right(i))
end

"""
	compactness(i)

Retuns the compactness of a simple `Interval`, which is always 1 unless it is empty.
Ex:
```
julia> compactness(Interval(0,true,1))
1.0
julia> compactness(Interval())
0.0
```
"""
function compactness(i::Interval)
	empty(i) ? 0.0 : 1.0
end

"""
	superset(i)

Returns the superset of the given `Interval`, which is the `Interval` itself.
Ex:
```
julia> superset(Interval(0,true,1))
]0,1]
```
"""
function superset(i::Interval)
	return i
end

"""
	sample(i)

Returns a number drawn uniformly at random in the given `Interval`.
Ex:
```
julia> i = Interval(2,4,true)
[2,4[
julia> sample(i)
3.935796996224805
```
"""
function sample(i::Interval)
	if cardinal(i) == 0 && (i.open_left && i.open_right)
		throw("Cannot sample from empty Interval.")
	end
	s = rand() * (right(i)-left(i)) + left(i)
	if i.open_left && s == left(i)
		return sample(i)
	elseif i.open_right && s == right(i)
		return sample(i)
	else
		return s
	end
end

"""
	sample(i,n)

Returns an array of n random numbers drawn uniformly in the given `Interval`.
Ex:
```
julia> i = Interval(3,true,6)
]3,6]
julia> sample(i,4)
4-element Array{Float64,1}:
 5.543827369817867
 4.678054798740224
 3.1740822420010355
 3.6870186624440504
```
"""
function sample(i::Interval, n::Int64)
	[sample(i) for x in 1:n]
end

"""
	jaccard(i1,i2)

Returns the jaccard coefficient between two `Interval`s defined as:
```math
\\frac{\\left| A \\cap B \\right|}{\\left| A \\cup B \\right|}
```
Note: If |i1 ∪ i2|=0, this function returns 0.
Ex:
```
julia> i = Interval(3,true,6)
]3,6]
julia> j = Interval(4,7)
[4,7]
julia> jaccard(i,j)
0.5
```
"""
function jaccard(i1::Interval, i2::Interval)
	i1 == i2 && return 1.0
	u = i1 ∪ i2
	cardu = cardinal(u)
	cardu != 0 ? cardinal(i1 ∩ i2) / cardu : 0.0
end

"""
	overlap_coefficient(i1,i2)

Returns the overlap coefficient between two `Interval`s defined as:
```math
\\frac{\\left| A \\cap B \\right|}{min \\left( \\left|A \\right|, \\left|B \\right| \\right)}
```
This means that if setA is a subset of B or the converse then the overlap coefficient is equal to 1.
Note: If `|A|=0` and `|B|=0`, this function returns 0.
Ex:
```
julia> i = Interval(3,true,6)
]3,6]
julia> j = Interval(4,7)
[4,7]
julia> overlap_coefficient(i,j)
0.6666666666666666
```
"""
function overlap_coefficient(i1::Interval, i2::Interval)
	i1 == i2 && return 1.0
	denom = min(cardinal(i1),cardinal(i2))
	denom != 0 ? cardinal(i1 ∩ i2) / denom : 0
end

"""
	dice_coefficient(i1,i2)

Returns the Sørensen–Dice coefficient between two `Interval`s defined as:
```math
\\frac{2 \\left| A \\cap B \\right|}{\\left|A \\right| + \\left|B \\right|}
```
Note: If `|A|+|B|=0`, this function returns 0.
Ex:
```
julia> i = Interval(3,true,6)
]3,6]
julia> j = Interval(4,7)
[4,7]
julia> dice_coefficient(i,j)
0.6666666666666666
```
"""
function dice_coefficient(i1::Interval, i2::Interval)
	i1 == i2 && return 1.0
	denom = cardinal(i1) + cardinal(i2)
	denom != 0 ? 2 * cardinal(i1 ∩ i2) / denom : 0
end

"""
	IntervalUnion

`IntervalUnion` representation.
"""
mutable struct IntervalUnion
	components::Array{Interval,1}
	function IntervalUnion(intervals::Array{Interval,1})
		sorted_intervals = sort(intervals)
		cleaned_intervals = Interval[]
		for interval in sorted_intervals
			if !empty(interval)
				if length(cleaned_intervals) == 0
					push!(cleaned_intervals,interval)
				else
					if !disjoint(cleaned_intervals[end],interval) || ((right(cleaned_intervals[end]) == left(interval)) && (!cleaned_intervals[end].open_right || !interval.open_left))
						cleaned_intervals[end] = cleaned_intervals[end] ∪ interval
					else
						push!(cleaned_intervals,interval)
					end
				end
			end
		end
		new(cleaned_intervals)
	end
end

"""
	IntervalUnion()

Empty union of `Interval`s.
Ex:
```
julia> i = IntervalUnion()
∅
```
"""
function IntervalUnion()
	IntervalUnion(Interval[])
end

"""
	IntervalUnion(left,right)

Simplified constructor for `IntervalUnions` with only one `Interval` closed on both side.
Ex:
```
julia> i = IntervalUnion(2,3)
[2,3]
"""
function IntervalUnion(left::Real, right::Real)
	IntervalUnion([Interval(left,right)])
end

"""
	IntervalUnion(left,open_left,right)

Simplified constructor for `IntervalUnions` with only one `Interval` closed on the right side.
Ex:
```
julia> i = IntervalUnion(2,true,3)
]2,3]
julia> i = IntervalUnion(2,false,3)
[2,3]
"""
function IntervalUnion(left::Real, open_left::Bool, right::Real)
	IntervalUnion([Interval(left,open_left,right)])
end

"""
	IntervalUnion(left,right,open_right)

Simplified constructor for `IntervalUnions` with only one `Interval` closed on the left side.
Ex:
```
julia> i = IntervalUnion(2,3,true)
[2,3[
julia> i = IntervalUnion(2,3,false)
[2,3]
"""
function IntervalUnion(left::Real, right::Real, open_right::Bool)
	IntervalUnion([Interval(left,right,open_right)])
end

"""
	IntervalUnion(left,open_left,right,open_right)

Simplified constructor for `IntervalUnions` with only one `Interval`.
Ex:
```
julia> i = IntervalUnion(2,true,3,true)
]2,3[
julia> i = IntervalUnion(2,false,3,false)
[2,3]
"""
function IntervalUnion(left::Real, open_left::Bool, right::Real, open_right::Bool)
	IntervalUnion([Interval(left,open_left,right,open_right)])
end

"""
	empty(iu)

Returns true if the given `IntervalUnion` is empty.
Ex:
```
julia> empty(IntervalUnion())
true
julia> empty(IntervalUnion([Interval(0,1)]))
false
```
"""
function empty(iu::IntervalUnion)
	(number_of_components(iu) == 0) || all([empty(c) for c in iu.components])
end

"""
	string(iu)

Returns a string representation of an `IntervalUnion`.
Ex:
```
julia> string(IntervalUnion([Interval(0,1,true),Interval(1,true,2)]))
"[0,1[ ∪ ]1,2]"
```
"""
function string(iu::IntervalUnion)
	if empty(iu)
		return "∅"
	elseif number_of_components(iu) == 1
		return string(iu.components[1])
	else
		return reduce((x,y)->"$(string(x)) ∪ $(string(y))", iu.components)
	end
end

"""
	show(io,iu)

Prints the string representation of an `IntervalUnion`.
"""
function show(io::IO, iu::IntervalUnion)
	println(string(iu))
end

"""
	iu1 == iu2

Equality defined for `IntervalUnion`s.
"""
function ==(iu1::IntervalUnion, iu2::IntervalUnion)
	if number_of_components(iu1) != number_of_components(iu2)
		return false
	end
	for (i,j) in zip(iu1.components,iu2.components)
		if i != j
			return false
		end
	end
	true
end

"""
	left(iu)

Returns a vector of left limits of the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2)])
[0,1[ ∪ ]1,2]
julia> left(i)
2-element Array{Int64,1}:
 0
 1
```
"""
function left(iu::IntervalUnion)
	[left(i) for i in iu.components]
end

"""
	right(iu)

Returns a vector of right limits of the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2)])
[0,1[ ∪ ]1,2]
julia> left(i)
2-element Array{Int64,1}:
 1
 2
```
"""
function right(iu::IntervalUnion)
	[right(i) for i in iu.components]
end

"""
	limits(iu)

Returns a sorted array of limits of the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2)])
[0,1[ ∪ ]1,2]
julia> limits(i)
3-element Array{Real,1}:
 0
 1
 2
```
"""
function limits(iu::IntervalUnion)
	limit_set = Set{Real}()
	for i in iu.components
		push!(limit_set,left(i))
		push!(limit_set,right(i))
	end
	limit_array = Real[]
	for l in limit_set
		push!(limit_array,l)
	end
	sort(limit_array)
end

"""
	compact(iu)

Returns the compact of the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1),Interval(2,3)])
]0,1] ∪ [2,3]
julia> compact(i)
[0,3]
```
"""
function compact(iu::IntervalUnion)
	empty(iu) && return IntervalUnion()
	l = minimum(left(iu))
	r = maximum(right(iu))
	if l == -Inf || r == Inf
		throw("IntervalUnion $(string(iu)) has no compact.")
	end
	IntervalUnion([Interval(l,r)])
end

"""
	compactness(iu)

Returns the compactness of the given `IntervalUnion` defined as:
```math
\\frac{ \\left| iu \\right|}{\\left| compact(iu) \\right|}
```
Note: If `|compact(iu)|=0`, this function returns 0.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1),Interval(3,5)])
]0,1] ∪ [3,5]
julia> compactness(i)
0.6
```
"""
function compactness(iu::IntervalUnion)
	cciu = cardinal(compact(iu))
	cciu != 0 ? cardinal(iu) / cciu : 0
end

"""
	superset(iu)

Returns the superset of the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1),Interval(2,3)])
]0,1] ∪ [2,3]
julia> superset(i)
]0,3]
```
Note: `superset` and `compact` are strongly related but `compact` is always closed on both sides.
"""
function superset(iu::IntervalUnion)
	if empty(iu) || number_of_components(iu) < 2
		throw("IntervalUnion $(string(iu)) has not enough components.")
	end
	smallest = minimum(iu.components)
	largest  = maximum(iu.components)
	IntervalUnion([Interval(left(smallest), smallest.open_left, right(largest), largest.open_right)])
end

"""
	super_complement(iu)

Returns the complement of the superset of the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1),Interval(2,3)])
]0,1] ∪ [2,3]
julia> super_complement(i)
]-Inf,0] ∪ ]3,Inf[
```
"""
function super_complement(iu::IntervalUnion)
	complement(superset(iu))
end

"""
	restricted_complement(iu)

Returns the complement of iu intersected with its superset.
Ex:
```
julia> i = IntervalUnion([Interval(0,1),Interval(4,5)])
[0,1] ∪ [4,5]
julia> restricted_complement(i)
]1,4[
julia> j = IntervalUnion([Interval(0,true,1,true),Interval(2.5,true,3),Interval(3.5,6)])
]0,1[ ∪ ]2.5,3] ∪ [3.5,6]
julia> restricted_complement(j)
[1,2.5] ∪ ]3,3.5[
```
"""
function restricted_complement(iu::IntervalUnion)
	superset(iu) ∩ complement(iu)
end

"""
	complement_cardinal(iu)

Returns the cardinal of the restricted complement of iu.
Ex:
```
julia> IntervalUnion([Interval(0,1),Interval(4,5)])
[0,1] ∪ [4,5]
julia> complement_cardinal(i)
3
julia> j = IntervalUnion([Interval(0,true,1,true),Interval(2.5,true,3),Interval(3.5,6)])
]0,1[ ∪ ]2.5,3] ∪ [3.5,6]
julia> complement_cardinal(j)
2
```
"""
function complement_cardinal(iu::IntervalUnion)
	cardinal(restricted_complement(iu))
end

"""
	x ∈ iu

Returns true if x ∈ iu and false if x ∉ iu.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1,true),Interval(2.5,true,3),Interval(3.5,6)])
]0,1[ ∪ ]2.5,3] ∪ [3.5,6]
julia> 0.5 ∈ i
true
julia> 1 ∈ i
false
julia> 0.9999999 ∈ i
true
```
"""
function ∈(x::Real, iu::IntervalUnion)
	for i in iu.components
		x ∈ i && return true
	end
	false
end

"""
	cardinal(iu)

Returns the cardinal of the given `IntervalUnion`.
```math
\\left| [a,b] ∪ ]c,d] ∪ [e,f[ ∪ ]g,h[ \\right| = (b - a) + (d-c) + (f-e) + (h-g)
```
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1,true),Interval(2.5,true,3),Interval(3.5,6)])
]0,1[ ∪ ]2.5,3] ∪ [3.5,6]
julia> cardinal(i)
4.0
```
"""
function cardinal(iu::IntervalUnion)
	empty(iu) && return 0
	sum([cardinal(i) for i in iu.components])
end

"""
	number_of_components(iu)

Returns the number of components in the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1,true),Interval(2.5,true,3),Interval(3.5,6)])
]0,1[ ∪ ]2.5,3] ∪ [3.5,6]
julia> number_of_components(i)
3
```
"""
function number_of_components(iu::IntervalUnion)
	length(iu.components)
end

"""
	iu1 ⊆ iu2

Returns true if the `IntervalUnion` iu1 is included in the `IntervalUnion` iu2.
Ex:
```
julia> i = IntervalUnion([Interval(0,true,1,true),Interval(2.5,true,3),Interval(3.5,6)])
]0,1[ ∪ ]2.5,3] ∪ [3.5,6]
julia> j = IntervalUnion([Interval(2.6,3),Interval(3.5,4,true)])
[2.6,3] ∪ [3.5,4[
julia> j ⊆ i
true
```
"""
function ⊆(iu1::IntervalUnion, iu2::IntervalUnion)
    for i in iu1.components
    	!any([i ⊆ j for j in iu2.components]) && return false
    end
    true
end

"""
	iu1 ∪ iu2

Returns the `IntervalUnion` corresponding to the union between
`IntervalUnion` iu1 and `IntervalUnion` iu2.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> i ∪ j
[0,4[
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> i ∪ k
[-1,1[ ∪ ]1,2.3] ∪ [3,4]
julia> i ∪ IntervalUnion()
[0,1[ ∪ ]1,2.3]
```
"""
function ∪(iu1::IntervalUnion, iu2::IntervalUnion)
	IntervalUnion(vcat(iu1.components,iu2.components))
end

"""
	iu1 ∩ iu2

Returns the `IntervalUnion` corresponding to the intersection between
`IntervalUnion` iu1 and `IntervalUnion` iu2.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> i ∩ j
∅
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> j ∩ k
[3,4[
julia> i ∩ IntervalUnion()
∅
```
"""
function ∩(iu1::IntervalUnion, iu2::IntervalUnion)
	components = Interval[]
	for c1 in iu1.components
		for c2 in iu2.components
			if !disjoint(c1,c2)
				push!(components,c1 ∩ c2)
			end
		end
	end
	IntervalUnion(components)
end

"""
	complement(iu)

Returns the complement of the given `IntervalUnion`.

Ex:
```
julia> i = IntervalUnion([Interval(0,1)])
[0,1]
julia> complement(i)
]-Inf,0[ ∪ ]1,Inf[
julia> j = IntervalUnion([Interval(0,0)])
[0,0]
julia> complement(j)
]-Inf,0[ ∪ ]0,Inf[
julia> k = IntervalUnion([Interval(0,true,0,true)])
]0,0[
julia> complement(k)
]-Inf,Inf[
julia> complement(IntervalUnion())
]-Inf,Inf[
julia> l = IntervalUnion([Interval(0,1,true),Interval(2,true,3)])
[0,1[ ∪ ]2,3]
julia> complement(l)
]-Inf,0[ ∪ [1,2] ∪ ]3,Inf[
```
"""
function complement(iu::IntervalUnion)
	components = Interval[]
	if empty(iu)
		return IntervalUnion([Interval(-Inf,true,Inf,true)])
	elseif number_of_components(iu) == 1
		if left(iu.components[1]) == -Inf && right(iu.components[1]) == Inf
			return IntervalUnion()
		elseif left(iu.components[1]) == -Inf && right(iu.components[1]) != Inf
			if iu.components[1].open_right
				return IntervalUnion([Interval(right(iu.components[1]),Inf,true)])
			else
				return IntervalUnion([Interval(right(iu.components[1]),true,Inf,true)])
			end
		elseif left(iu.components[1]) != -Inf && right(iu.components[1]) == Inf
			if iu.components[1].open_left
				return IntervalUnion([Interval(-Inf,true,left(iu.components[1]))])
			else
				return IntervalUnion([Interval(-Inf,true,left(iu.components[1]),true)])
			end
		else
			if iu.components[1].open_left
				if iu.components[1].open_right
					return IntervalUnion([Interval(-Inf,true,left(iu.components[1])),Interval(right(iu.components[1]),Inf,true)])
				else
					return IntervalUnion([Interval(-Inf,true,left(iu.components[1])),Interval(right(iu.components[1]),true,Inf,true)])
				end
			else
				if iu.components[1].open_right
					return IntervalUnion([Interval(-Inf,true,left(iu.components[1]),true),Interval(right(iu.components[1]),Inf,true)])
				else
					return IntervalUnion([Interval(-Inf,true,left(iu.components[1]),true),Interval(right(iu.components[1]),true,Inf,true)])
				end
			end
		end
	else
		if left(iu.components[1]) != -Inf
			if iu.components[1].open_left
				push!(components,Interval(-Inf,true,left(iu.components[1])))
			else
				push!(components,Interval(-Inf,true,left(iu.components[1]),true))
			end
		end
		for (c1,c2) in zip(iu.components[1:(end-1)],iu.components[2:end])
			if c1.open_right && c2.open_left
				push!(components,Interval(right(c1),left(c2)))
			elseif !c1.open_right && c2.open_left
				push!(components,Interval(right(c1),true,left(c2)))
			elseif c1.open_right && !c2.open_left
				push!(components,Interval(right(c1),left(c2),true))
			else
				push!(components,Interval(right(c1),true,left(c2),true))
			end
		end
		if right(iu.components[end]) != Inf
			if iu.components[end].open_right
				push!(components,Interval(right(iu.components[end]),Inf,true))
			else
				push!(components,Interval(right(iu.components[end]),true,Inf,true))
			end
		end
	end
	IntervalUnion(components)
end

"""
	setdiff(iu1,iu2)

Returns the `IntervalUnion` with elements in iu1 but not in iu2.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> julia> setdiff(j,k)
[1,1] ∪ ]2.3,3[
```
"""
function setdiff(iu1::IntervalUnion, iu2::IntervalUnion)
	complement(iu2) ∩ iu1
end

"""
	symdiff(iu1,iu2)

Returns the `IntervalUnion` with elements in iu1 or iu2 but not in both.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> symdiff(i,k)
[-1,1[ ∪ ]1,2.3] ∪ [3,4]
julia> symdiff(j,k)
[-1,0[ ∪ [1,1] ∪ ]2.3,3[ ∪ [4,4]
```
"""
function symdiff(iu1::IntervalUnion, iu2::IntervalUnion)
	setdiff(iu1 ∪ iu2, iu1 ∩ iu2)
end

"""
	sample(iu)

Returns a number drawn uniformly at random in the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> sample(i)
0.22557298488974253
```
"""
function sample(iu::IntervalUnion)
	if empty(iu)
		throw("Cannot sample from empty union of Intervals.")
	end
	size_components = [cardinal(i) for i in iu.components]
	full_size = sum(size_components)
	if full_size == 0
		return left(iu.components[rand(1:number_of_components(iu))])
	end
	size_components_normalized = size_components ./ full_size
	probabilities = cumsum(size_components_normalized)
	r = rand()
	idx_component = findfirst(x->x>r,probabilities)
	sample(iu.components[idx_component])
end

"""
	sample(iu,n)

Returns an array of n random numbers drawn uniformly in the given `IntervalUnion`.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> sample(i,4)
4-element Array{Float64,1}:
 0.7974550091944592
 0.36823887828930935
 1.1796141777238074
 2.2731170537623138
```
"""
function sample(iu::IntervalUnion, n::Int64)
	[sample(iu) for x in 1:n]
end

"""
	jaccard(iu1,iu2)

Returns the jaccard similarity between two `IntervalUnion`s.
If |iu1 ∪ iu2|=0, this function returns 0.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> jaccard(i,j)
0.0
julia> jaccard(j,k)
0.37037037037037035
```
"""
function jaccard(iu1::IntervalUnion, iu2::IntervalUnion)
	u = iu1 ∪ iu2
	cardu = cardinal(u)
	cardu != 0 ? cardinal(iu1 ∩ iu2) / cardu : 0.0
end

"""
	overlap_coefficient(iu1,iu2)

Returns the overlap coefficient between two `IntervalUnion`s defined as:
```math
\\frac{\\left| A \\cap B \\right|}{min \\left( \\left|A \\right|, \\left|B \\right| \\right)}
```
This means that if set A is a subset of B or the converse then the overlap coefficient is equal to 1.
Note: If `|A|=0` and `|B|=0`, this function returns 0.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> overlap_coefficient(i,j)
0.0
julia> overlap_coefficient(j,k)
0.588235294117647
```
"""
function overlap_coefficient(iu1::IntervalUnion, iu2::IntervalUnion)
	denom = min(cardinal(iu1),cardinal(iu2))
	denom != 0 ? cardinal(iu1 ∩ iu2) / denom : 0
end

"""
	dice_coefficient(iu1,iu2)

Returns the Sørensen–Dice coefficient between two `IntervalUnion`s defined as:
```math
\\frac{2 \\left| A \\cap B \\right|}{\\left|A \\right| + \\left|B \\right|}
```
Note: If `|A|+|B|=0`, this function returns 0.
Ex:
```
julia> i = IntervalUnion([Interval(0,1,true),Interval(1,true,2.3)])
[0,1[ ∪ ]1,2.3]
julia> j = IntervalUnion([Interval(1,1),Interval(2.3,true,4,true)])
[1,1] ∪ ]2.3,4[
julia> k = IntervalUnion([Interval(-1,0,true),Interval(3,4)])
[-1,0[ ∪ [3,4]
julia> dice_coefficient(i,j)
0.0
julia> dice_coefficient(j,k)
0.5405405405405405
```
"""
function dice_coefficient(iu1::IntervalUnion, iu2::IntervalUnion)
	denom = cardinal(iu1) + cardinal(iu2)
	denom != 0 ? 2 * cardinal(iu1 ∩ iu2) / denom : 0
end


### A Pluto.jl notebook ###
# v0.12.17

using Markdown
using InteractiveUtils

# This Pluto notebook uses @bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of @bind gives bound variables a default value (instead of an error).
macro bind(def, element)
    quote
        local el = $(esc(element))
        global $(esc(def)) = Core.applicable(Base.get, el) ? Base.get(el) : missing
        el
    end
end

# ╔═╡ 85d5a830-4116-11eb-1479-630b53b50e94
begin
	using Pkg
    Pkg.activate(pwd())
end

# ╔═╡ 4862c2d0-4116-11eb-3f26-1bdefcb9ce3d
begin
	using UCIData
	using PlutoUI
	using DataFrames
	using Statistics
	using LinearAlgebra
	using Plots
	using Clustering
	using Distances
end

# ╔═╡ c73d5420-4126-11eb-19b3-bf6c04aef086
md"""
# Data Mining
## Assignment 2 - K-means Clustering
### Aadam & Obaidullah (CS1945 & CS1947)
"""

# ╔═╡ 360ab320-411d-11eb-2d11-c9fe7b3137d1
plotly()

# ╔═╡ 73936e9e-4116-11eb-0594-1d202bd073cd
function Kmeans(X, k; max_iters = 300, tol = 1e-5)
    # Reshape 2D array to a 1D array with length of all training examples
    X_array_list = collect(eachrow(X))

    # Save some info on the incoming data
    N = length(X_array_list)  # Length of all training examples
    n = length(X_array_list[1])  # Length of a single training example
    distances = zeros(N)  # Empty vector for all training examples.

    # Step 1: Random initialization
    reps_centroids = [zeros(n) for grp = 1:k]  # Initiate centroids for each
    labels = rand(1:k, N)  # Randomly assign labels to all training examples

    J_previous = Inf

    for iter = 1:max_iters

        # Step 2: Update the representative centroids for each group
        for j = 1:k
            # get group indices for each group
            group_idx = [i for i = 1:N if labels[i] == j]

            # use group indices to locate each group
            reps_centroids[j] = mean(X_array_list[group_idx]);
        end;

        # Step 3: Update the group labels
        for i = 1:N
            # compute the distance between each example and the updated centroid
            nearest_rep_distance =
			[norm(X_array_list[i] - reps_centroids[x]) for x = 1:k]

            # update distances and label arrays
            # findmin returns the min value & index location
            distances[i], labels[i] = findmin(nearest_rep_distance)
        end;

        # Step 4: Compute the clustering cost
        J = (norm(distances)^ 2) / N

        # Show progress and terminate if J stopped decreasing.
        println("Iteration ", iter, ": Jclust = ", J, ".")

        # Final Step 5: Check for convergence
        if iter > 1 && abs(J - J_previous) < (tol * J)
            # Terminate algorithm with the assumption that K-means has converged
            return labels, reps_centroids
        end

        J_previous = J
    end
	return labels, reps_centroids
end

# ╔═╡ aac40900-41cc-11eb-1103-97d9373bdd65
function calculate_average_distance(i, X, result)
	dist = 0
	center = result.centers[i]
	for instance in X[:, result.assignments .== i]
		dist += Euclidean()(instance, center)
	end
	return dist/result.counts[i]
end

# ╔═╡ 12615c80-41c7-11eb-2aed-c1eb96003ff1
function dbi(X, result)
	num_of_clusters = length(result.counts)

	#average distance of all points in a cluster to its center, for each cluster
    average_centroid_distances = [
		calculate_average_distance(i, X, result) for i in 1:num_of_clusters]

	sum_Dij = 0

	#calculate the sum Dij
    for i in 1:num_of_clusters
        max_Dij = 0

        #calculate the max Dij
        for j in 1:num_of_clusters
            if i != j

				#calculate di + dj
				numerator =
				average_centroid_distances[i] + average_centroid_distances[j]

				#calculate dij
				denominator =
				Euclidean()( result.centers[i], result.centers[j] )

				current_Dij = numerator/denominator

				if max_Dij < current_Dij
					max_Dij = current_Dij
				end
			end

		end
		sum_Dij += max_Dij
	end

    return sum_Dij / num_of_clusters
end

# ╔═╡ c7001000-42c6-11eb-23d1-314932816f0a
function big_delta(maskci, distances)
	maximum(distances[maskci, :][:, maskci])
end

# ╔═╡ c6ca5a00-42c6-11eb-386e-2f13a99d75b6
function delta(maskci, maskcj, distances)
	minimum(distances[maskci, :][:, maskcj])
end

# ╔═╡ c69d2f80-42c6-11eb-10cf-e70c07e40e97
function dunn(X, result)
	n = length(result.counts) # number of clusters

	distances = pairwise(Euclidean(), X, dims=2)

	deltas = ones(n, n) * Inf
	big_deltas = zeros(n)

	for k in 1:n
		for l in 1:n
			if k == l
				continue
			end
            deltas[k, l] = delta(
				result.assignments .== k,
				result.assignments .== l,
				distances)
		end

        big_deltas[k] = big_delta(result.assignments .== k, distances)

	end
	minimum(deltas)/maximum(big_deltas)
end

# ╔═╡ 9accd320-4117-11eb-0fc4-15e8e85040a8
datasets = ["iris" => "iris", "glass-identification" => "glass"];

# ╔═╡ 8b43a9a0-4118-11eb-127d-d57dcea189e1
md"""
Dataset: $(@bind selected_dataset Select(datasets))

"""

# ╔═╡ 73c57b20-4116-11eb-3dba-01c9f7b91fa7
data = UCIData.dataset(selected_dataset);

# ╔═╡ 5cddc6a0-411c-11eb-2380-1b9442938a9b
features = collect(Matrix(data[:, 2:end-1])');

# ╔═╡ bec93fa0-4119-11eb-026e-0be14d6a5779
md"""
Vlaue of **k**: $(@bind k Slider(2:15; show_value=true))
"""
# size(UCIData.dataset(selected_dataset))[1]

# ╔═╡ 737b2bb0-4116-11eb-124b-d981032a9638
result = kmeans(features, k);

# ╔═╡ 725d2e10-42c2-11eb-3ee7-4ba089ebb440
md"""
### Davies Bouldin Index: $(round(dbi(features, result), digits=5))
### Dunn Index : $(round(dunn(features, result), digits=5))
"""

# ╔═╡ ce4134a0-4124-11eb-2a7c-9761e81b2064
md"""
## Plot Clusters
Feature $(@bind feature1 Select(string.(collect(1:size(features)[1])))) vs.
Feature $(@bind feature2 Select(string.(collect(1:size(features)[1])));)
"""

# ╔═╡ 73454ea0-4116-11eb-0938-f9a06f3dbaf8
begin
	theme(:dark)
	scatter(
		features[parse(Int,feature1), :],
		features[parse(Int, feature2), :],
		marker_z = result.assignments,
		markershape = :diamond,
		markersize = 3,
		xlabel="Feature $(feature1)",
		ylabel="Feature $(feature2)",
		color = :seaborn_bright,
		legend = false)
	scatter!(
		result.centers[parse(Int,feature1), :],
		result.centers[parse(Int, feature2), :],
		markersize = 6,
		marker_z = result.centers',
		color = :temperaturemap
	)
end

# ╔═╡ 6cd3fe00-41c4-11eb-2e84-376a14af7398
md"""
### Cluster Centers
$(
df = DataFrame(hcat(result.centers', result.counts));
last_col_name = names(df)[end];
df[!,last_col_name] = convert.(Int,df[:,last_col_name]);
rename!(df, last_col_name => :NodeCount)
)
"""

# ╔═╡ Cell order:
# ╟─c73d5420-4126-11eb-19b3-bf6c04aef086
# ╟─85d5a830-4116-11eb-1479-630b53b50e94
# ╠═4862c2d0-4116-11eb-3f26-1bdefcb9ce3d
# ╟─360ab320-411d-11eb-2d11-c9fe7b3137d1
# ╠═73936e9e-4116-11eb-0594-1d202bd073cd
# ╠═aac40900-41cc-11eb-1103-97d9373bdd65
# ╠═12615c80-41c7-11eb-2aed-c1eb96003ff1
# ╠═c7001000-42c6-11eb-23d1-314932816f0a
# ╠═c6ca5a00-42c6-11eb-386e-2f13a99d75b6
# ╠═c69d2f80-42c6-11eb-10cf-e70c07e40e97
# ╟─9accd320-4117-11eb-0fc4-15e8e85040a8
# ╟─8b43a9a0-4118-11eb-127d-d57dcea189e1
# ╠═73c57b20-4116-11eb-3dba-01c9f7b91fa7
# ╠═5cddc6a0-411c-11eb-2380-1b9442938a9b
# ╟─bec93fa0-4119-11eb-026e-0be14d6a5779
# ╠═737b2bb0-4116-11eb-124b-d981032a9638
# ╠═725d2e10-42c2-11eb-3ee7-4ba089ebb440
# ╟─ce4134a0-4124-11eb-2a7c-9761e81b2064
# ╟─73454ea0-4116-11eb-0938-f9a06f3dbaf8
# ╟─6cd3fe00-41c4-11eb-2e84-376a14af7398


export lanczosBidiag

function lanczosBidiag(A::SparseMatrixCSC{T1,Int},b::Array{T2,1}, k::Int) where {T1,T2}
	T  = promote_type(T1,T2)
	x1 = zeros(T,size(A,1))
	x2 = zeros(T,size(A,2))

	Af(x,flag) = (flag=='F') ? mul!(x1,A,x,1.0,0.0) : mul!(x2,transpose(A),x,1.0,0.0)
	return lanczosBidiag(Af,b,k)
end

lanczosBidiag(A,b,k) = lanczosBidiag((x,flag) -> ((flag=='F') ? A*x : A'*x),b,k)

"""
U, B, V =  lanczosBidiag(A,p::Vector,k::Int)

Lanczos bidiagonalization of matrix A.

Input:

  A       - function computing A*x = A(x,'F') and A'*x = A(x,'T')
  p       - starting vector
  k       - dimension of subspace

Output:

  U,B,V   - Lanczos vectors
"""
function lanczosBidiag(A::Function,p::Vector,k::Int)

	m      = length(p)
	beta   = norm(p)
	u      = p/beta
	ATu    = A(u,'T')

	n      = length(ATu)
	v      = zeros(n)

	# alocate space for Lanczos vectors
	U = zeros(m,k+1)
	V = zeros(n,k)
	B = zeros(k,2)
	# Prepare for Lanczos iteration.
	U[:,1] = u

	for i=1:k # perform Lanczos bidiagonalization with reorthogonalization.
		if i>1;
			ATu = A(u,'T')
		end
		r = ATu - beta*v

		for j=1:i-1 # reorthogonalization of V
			r -= dot(V[:,j],r)*V[:,j]
		end

		alpha = norm(r)
		v = r/alpha
		B[i,2] = alpha
		V[:,i] = v

		Av = A(v,'F')
		p = Av - alpha*u

		for j=1:i; p = p - dot(U[:,j],p)*U[:,j]; end
		beta = norm(p)
		u    = p/beta

		B[i,1]   = beta
		U[:,i+1] = u
	end
	II,JJ,VV = SparseArrays.spdiagm_internal(-1=>B[:,1], 0=>B[:,2])
	B = sparse(II,JJ,VV,k+1,k)
	# Bt = spdiagm((B[:,1],B[:,2]),[-1,0],k+1,k)
	return U, B, V
end


#=

  Crypto problem (alphametic) in Julia ConstraintSolver.jl

  This is a standard alphametic problem in mathematical recreations,
  constraint programming etc.

  From GLPK:s model cryto.mod.

  """
  This problem comes from the newsgroup rec.puzzle.
  The numbers from 1 to 26 are assigned to the letters of the alphabet.
  The numbers beside each word are the total of the values assigned to
  the letters in the word (e.g. for LYRE: L, Y, R, E might be to equal
  5, 9, 20 and 13, or any other combination that add up to 47).
  Find the value of each letter under the equations:

  BALLET  45     GLEE  66     POLKA      59     SONG     61
  CELLO   43     JAZZ  58     QUARTET    50     SOPRANO  82
  CONCERT 74     LYRE  47     SAXOPHONE 134     THEME    72
  FLUTE   30     OBOE  53     SCALE      51     VIOLIN  100
  FUGUE   50     OPERA 65     SOLO       37     WALTZ    34

  Solution:
  A, B,C, D, E,F, G, H, I, J, K,L,M, N, O, P,Q, R, S,T,U, V,W, X, Y, Z
  5,13,9,16,20,4,24,21,25,17,23,2,8,12,10,19,7,11,15,3,1,26,6,22,14,18

  Reference:
  Koalog Constraint Solver <http://www.koalog.com/php/jcs.php>,
  Simple problems, the crypto-arithmetic puzzle ALPHACIPHER.
  """

  Model created by Hakan Kjellerstrand, hakank@gmail.com
  See also my Julia page: http://www.hakank.org/julia/

=#

using ConstraintSolver, JuMP
using Cbc, GLPK, Ipopt
const CS = ConstraintSolver
include("constraints_utils.jl")

function crypto(print_solutions=true,all_solutions=true)

    cbc_optimizer = optimizer_with_attributes(Cbc.Optimizer, "logLevel" => 0)
    glpk_optimizer = optimizer_with_attributes(GLPK.Optimizer)
    ipopt_optimizer = optimizer_with_attributes(Ipopt.Optimizer)

    model = Model(optimizer_with_attributes(CS.Optimizer,   "all_solutions"=> all_solutions,
                                                            # "all_optimal_solutions"=>all_solutions,
                                                            "logging"=>[],

                                                            "traverse_strategy"=>:BFS,
                                                            # "traverse_strategy"=>:DFS,
                                                            # "traverse_strategy"=>:DBFS,

                                                            # "branch_split"=>:Smallest,
                                                            # "branch_split"=>:Biggest,
                                                            "branch_split"=>:InHalf,

                                                            # https://wikunia.github.io/ConstraintSolver.jl/stable/options/#branch_strategy-(:Auto)
                                                            "branch_strategy" => :IMPS, # default
                                                            # "branch_strategy" => :ABS, # Activity Based Search
                                                            # "activity.decay" => 0.999, # default 0.999
                                                            # "activity.max_probes" => 10, # default, 10
                                                            # "activity.max_confidence_deviation" => 20, # default 20

                                                            # "simplify"=>false,
                                                            # "simplify"=>true, # default

                                                            "time_limit"=>6,

                                                            # "backtrack" => false, # default true
                                                            # "backtrack_sorting" => false, # default true

                                                            # "lp_optimizer" => cbc_optimizer,
                                                            # "lp_optimizer" => glpk_optimizer,
                                                            # "lp_optimizer" => ipopt_optimizer,
                                        ))

    BALLET     =  45
    CELLO      =  43
    CONCERT    =  74
    FLUTE      =  30
    FUGUE      =  50
    GLEE       =  66
    JAZZ       =  58
    LYRE       =  47
    OBOE       =  53
    OPERA      =  65
    POLKA      =  59
    QUARTET    =  50
    SAXOPHONE  = 134
    SCALE      =  51
    SOLO       =  37
    SONG       =  61
    SOPRANO    =  82
    THEME      =  72
    VIOLIN     = 100
    WALTZ      =  34

    n = 26
    @variable(model, 1 <= x[1:n] <= n, Int)
    A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z = x

    @constraint(model, x in CS.AllDifferent())

    @constraint(model,B + A + L + L + E + T == BALLET)
    @constraint(model,C + E + L + L + O == CELLO)
    @constraint(model,C + O + N + C + E + R + T == CONCERT)
    @constraint(model,F + L + U + T + E == FLUTE)
    @constraint(model,F + U + G + U + E == FUGUE)
    @constraint(model,G + L + E + E == GLEE)
    @constraint(model,J + A + Z + Z == JAZZ)
    @constraint(model,L + Y + R + E == LYRE)
    @constraint(model,O + B + O + E == OBOE)
    @constraint(model,O + P + E + R + A == OPERA)
    @constraint(model,P + O + L + K + A == POLKA)
    @constraint(model,Q + U + A + R + T + E + T == QUARTET)
    @constraint(model,S + A + X + O + P + H + O + N + E == SAXOPHONE)
    @constraint(model,S + C + A + L + E == SCALE)
    @constraint(model,S + O + L + O == SOLO)
    @constraint(model,S + O + N + G == SONG)
    @constraint(model,S + O + P + R + A + N + O == SOPRANO)
    @constraint(model,T + H + E + M + E == THEME)
    @constraint(model,V + I + O + L + I + N == VIOLIN)
    @constraint(model,W + A + L + T + Z == WALTZ)

    # Solve the problem
    optimize!(model)

    status = JuMP.termination_status(model)
    # println("status:$status")
    if status == MOI.OPTIMAL
        num_sols = MOI.get(model, MOI.ResultCount())
        println("num_sols:$num_sols\n")
        if print_solutions
            for sol in 1:num_sols
                println("solution #$sol")
                x_val = convert.(Integer,JuMP.value.(x; result=sol))
                println("x:$x_val")

            end
        end
    else
        println("status:$status")
    end

    return status
end

@time crypto()


using NewtonsMethod
using Test

@testset "NewtonsMethod.jl" begin
    atol = 1e-4
    f1(x) = 2x+1
    g1(x) = 2
    f2(x) = (x-1)^2
    g2(x) = 2(x-1)
    f3(x) = exp(x)-1
    g3(x) = exp(x)
    f4(x) = 2 + x^2
    g4(x) = 2x

    # Tests for known functions - analytical derivatives
    @test isapprox(newtonroot(f1, g1; x₀ = 10.0).value, -0.5, atol=atol)
    @test isapprox(newtonroot(f2, g2; x₀ = 10.0).value, 1.0, atol=atol)
    @test isapprox(newtonroot(f3, g3; x₀ = 10.0).value, 0.0, atol=atol)

    # Tests for known functions - automatic derivatives
    @test isapprox(newtonroot(f1; x₀ = 10.0).value, -0.5, atol=atol)
    @test isapprox(newtonroot(f2; x₀ = 10.0).value, 1.0, atol=atol)
    @test isapprox(newtonroot(f3; x₀ = 10.0).value, 0.0, atol=atol)

    # Tests using BigFloat
    @test isapprox(newtonroot(f1, g1; x₀ = BigFloat(10.0)).value, -0.5, atol=atol)
    @test isapprox(newtonroot(f2, g2; x₀ = BigFloat(10.0)).value, 1.0, atol=atol)
    @test isapprox(newtonroot(f3, g3; x₀ = BigFloat(10.0)).value, 0.0, atol=atol)
    @test isapprox(newtonroot(f1; x₀ = BigFloat(10.0)).value, -0.5, atol=atol)
    @test isapprox(newtonroot(f2; x₀ = BigFloat(10.0)).value, 1.0, atol=atol)
    @test isapprox(newtonroot(f3; x₀ = BigFloat(10.0)).value, 0.0, atol=atol)

    # Tests for non-convergence handling
    @test isnothing(newtonroot(f4, g4; x₀ = 10.0).value)
    @test isnothing(newtonroot(f4; x₀ = 10.0).value)

    # Tests that maxiter is working
    @test newtonroot(f4, g4; x₀ = 10.0, maxiter = 5).iter == 6
    @test newtonroot(f4; x₀ = 10.0, maxiter = 5).iter == 6

    # Tests that tol is working
    @test abs(newtonroot(f2, g2; x₀ = 10.0, tol = 1e-1).value - 1.0) >
          abs(newtonroot(f2, g2; x₀ = 10.0, tol = 1e-10).value - 1.0)
    @test abs(newtonroot(f2; x₀ = 10.0, tol = 1e-1).value - 1.0) >
          abs(newtonroot(f2; x₀ = 10.0, tol = 1e-10).value - 1.0)
    @test abs(newtonroot(f3, g3; x₀ = 10.0, tol = 1e-1).value) >
          abs(newtonroot(f3, g3; x₀ = 10.0, tol = 1e-10).value)
    @test abs(newtonroot(f3; x₀ = 10.0, tol = 1e-1).value) >
          abs(newtonroot(f3; x₀ = 10.0, tol = 1e-10).value)
end


using Plots
plotly()

x = 0:0.01:2pi
y_sin = sin.(x)
y_cos = cos.(x)

plot(x, [y_sin y_cos], labels=["sin" "cos"])

_round(number) = round(number, sigdigits=4)
_point_label(x, y, fun) = "$fun($(_round(x))) = $(_round(y))"
tooltips_sin = [_point_label(x, y, "sin") for (x, y) in zip(x, y_sin)]
tooltips_cos = [_point_label(x, y, "cos") for (x, y) in zip(x, y_cos)]

plot(x, [y_sin y_cos], labels=["sin" "cos"], hover=[tooltips_sin tooltips_cos])


"""
    AdaptiveExtrapolationD

This module is  implementing an
adaptive extrapolation of the explicit midpoint rule according to Deuflhard.
"""
module AdaptiveExtrapolationD

"""
    (Δ,Δx,statisitic) = (mySolver::solver)(f::Function, x₀::Vector{T}, t₀::S, tEnd::S; <options>)
        where {T<:Number,S<:AbstractFloat}

Computes the grid function `xΔ` on the grid `Δ` approximating the solution of the initial value problem

        x′ = f(t,x),  x(t₀) = x₀

on the interval `[t₀,tEnd]` (or `[tEnd,t₀]` if `tEnd < t₀`). The integrator is the extrapolated explicit midpoint rule.
The algorithm adaptively controls stepsize and extrapolation order in the line of thought of Deuflhard.
Note that `solver` is a callable instance of the `struct solver`.

***
## Initialize `solver`

    solver(N::Integer, sequence::String)

Initializes and returns the an instance of `solver`.
`N ≧ 1` is the maximal order of extrapolation. `sequence` specifies the subdividing sequence
used. The options are
* `Harmonic`, that is `1, 2, 3, 4, 5, 6,...`
* `Romberg`, that is `1, 2, 4, 8, 16, 32,...`
* `Bulirsch`, that is `1, 2, 3, 4, 6, 8,... `
The structure contains all quantities for the adaptive extrapolation of the the explicit
midpoint rule for all orders in `1:N` that can be tabulated before the actual computation .
These are the *subdividing sequence* and the *weights* for the extrapolation based on the first barycentric formula.

***
## Options

The following options can be passed as keyword arguments.
* `tol::S > 0` is tolerance used for accuracy check. Default is `1e-3`.
* `relativeScaling::Vector{S}` and `absoluteScaling::Vector{S}` are vectors of the same length as `x₀`
    containing no negative elements.
    Default (in every component) is `relativeScaling[i] = 1.0` and `absoluteScaling[i] = 1e-3`.
    The accuracy check is passed iff the estimated error satisfies in every component:

        |relativeError[i]| ≦ relativeScaling[i]⋅tol & |absoluteError[i]| ≦ absoluteScaling[i]⋅tol

* `firstGuessStepsize::S` is the length of the first step. Its modulus must be greater than
    `eps(S)` Default is `1e-3`.
* `maximalSteplength::S > 0` is the maximal absolute stepsize used for the integration. Default is `Inf`.
* `minimalStepsizeScaling::S` and `maximalStepsizeScaling::S` are safety factors for the stepsize
    selection. Default are `0.25` and `4.0`. The new stepsize satisfies:

         0 ≦ minimalStepsizeScaling ≦ newStepsize / oldStepsize ≦ maximalStepsizeScaling

* `safetyStepsizeSelection::S>0`. Use `safetyStepsizeSelection⋅tol` instead of `tol`. Default is `0.25`.
* `minimumOrder::Int64 ≧ 1` and `maximumOrder::Int64 ≦ N` are the minimal and maximal order of extrapolation for the integration (`N` is the number that has been passed to the constructor). Default are `1` and `N`.
* `firstGuessOrder::Int64` is the extrapolation order for the first step. It must satisfy
    `minimumOrder ≦ firstGuessOrder ≦ maximumOrder`. Default is `round(Int64,(maximumOrder-minimumOrder)/2)`.
* `maximumReduction::Int64` is the upper bound of the number of successive reductions allowed per step.
    Exceeding this boundary leads to ending the integration prematurely (`Δ[end] ≠ tEnd`). Default is `10`.
* `maximumSteps::Int64 ≧ 0` is the maximal length of the array `Δ`. Exceeding this boundary leads to
    ending the integration prematurely (`Δ[end] ≠ tEnd`). Default is `10 000`.
* `rescaleWeights::Bool` If `true` Lagrange interpolation is used otherwise the frist barycentric fromula. Default is `false`
* `detailedStatistics::Bool` flag for the content of `statisitic`. Default is `false`
    + `true`: `statisitic[:,i]` contains [extrapolation order; stepsize; number of `f`-evaluations;
        number of reductions] for the ith step.
    + `false`: `statisitic` contains [total number of evaluations of the right side `f`, total number of reductions].
"""
struct solver
    """
    The structure contains all quantities for the adaptive extrapolation of the the explicit
    midpoint rule that can be tabulated before the acutal computation.
    These are for all orders in `1:N` (`N` is provided the constructor):
    The subdividing sequence, the weights and scaling used for
    the extrapolation based on the first barycentric formula.
    """
   subdividingSequence::Array{BigInt,1}

   # weights and scaling factors for extrapolation operators
   ω::Array{Rational{BigInt},2}
   ρ::Array{Rational{BigInt},1}

   # weights and scaling factors for internal extrapolation operators (used for error estimate)
   ω2::Array{Rational{BigInt},2}
   ρ2::Array{Rational{BigInt},1}

   # constructor
   function solver(N::Integer, sequence::String)
       # check input and initialize subdividing sequence subdividingSequence for barycentric weights
       if N < 0
           error("Order of extrapolation must not be negative. But I got N = $N")
       end
       if sequence == "Harmonic"
           subdividingSequence = [BigInt(n+1) for n = 0:N]
       elseif sequence == "Romberg"
           subdividingSequence = [BigInt(2)^n for n = 0:N]
       elseif sequence == "Bulirsch"
           subdividingSequence = [n==0 ? BigInt(1) : (isodd(n) ? BigInt(2)^Int64(n/2+0.5) : 3*BigInt(2^Int64(n/2-1))) for n = 0:N]
       else
           error("Name of subdividing sequence must be ''Harmonic'', ''Romberg'' or ''Bulirsch''. But I got ''$sequence''")
       end

       # compute nodes corresponding to the subdividing sequence subdividingSequence
       nodes = BigInt(1).// subdividingSequence.^2

       # compute barycentric weights for internal extrapolation operators
       ω2 = zeros(Rational{BigInt},N,N)
       ω2[1,:] = ones(Rational{BigInt},1,N)
       for n = 2:N
           distance = nodes[2:n] .- nodes[n+1]
           ω2[1:(n-1),n] = ω2[1:n-1,n-1] .// distance
           ω2[n,n] = 1 // prod(-distance)
       end

       # compute barycentric weights for extrapolation operators
       ω = zeros(Rational{BigInt},N+1,N+1)
       for n = 1:N
           ω[n+1,(n+1):(N+1)] = ω2[n,n:N] // (nodes[n+1]-nodes[1])
           ω[1,n] = 1 // prod(nodes[1].-nodes[2:n])
       end
       ω[1,N+1] = 1 // prod(nodes[1].-nodes[2:N+1])

       #rescale barycentric weights to obtain weights of 1. barycentric formula
       for m = 1:(N+1)
           ω[1:m,m] = - ω[1:m,m] .// nodes[1:m]
           if 2 <= m
               ω2[1:m-1,m-1] = - ω2[1:m-1,m-1] .// nodes[2:m]
           end
       end

       # compute scaling factors for internal extrapolation operators
       ρ2 = ones(Rational{BigInt},N)
       ρ2[1] = -nodes[2]
       for n = 1:(N-1)
           ρ2[n+1] = -ρ2[n]*nodes[n+2]
       end

       # compute scaling factors for extrapolation operators
       ρ = -nodes[1]*[BigInt(1); ρ2]

       # initialize structure
       new(subdividingSequence,ω,ρ,ω2,ρ2)
   end
end

function (mySolver::solver)(
            f::Function, x₀::Vector{T}, t₀::S, tEnd::S;
            tol::S = parse(S,"1e-3"),
            relativeScaling::Vector{S} =ones(S,size(x₀)),
            absoluteScaling::Vector{S} = fill(parse(S,"1e-3"),size(x₀)),
            firstGuessStepsize::S = parse(S,"1e-3"),
            maximalSteplength::S = parse(S,"Inf"),
            minimalStepsizeScaling::S = parse(S,"0.02"),
            maximalStepsizeScaling::S = parse(S,"4"),
            safetyStepsizeSelection::S =  parse(S,"0.25"),
            minimumOrder::Int64 = 1,
            maximumOrder::Int64 = length(mySolver.subdividingSequence)-1,
            firstGuessOrder = round(Int64,(maximumOrder-minimumOrder)/2),
            maximumReduction::Int64 = 10,
            maximumSteps::Int64 = 10000,
            rescaleWeights::Bool = false,
            detailedStatistics::Bool = false,
            kwargs...
            )where {T<:Number,S<:AbstractFloat}
    # initialize
    # 1. constants
    d = length(x₀) # systems dimension
    if 1<= minimumOrder <= maximumOrder <= length(mySolver.ρ)-1
        n_ex, N_ex = minimumOrder, maximumOrder
    else
        error("Minimal and maximal order must satisfy: 1 ≦ minimumOrder ≦ maximumOrder ≦ $(length(mySolver.ρ)-1)!")
    end

    if length(relativeScaling) == length(absoluteScaling) == d
        if all(el->el>=zero(S),relativeScaling ) && all(el->el>=zero(S),absoluteScaling)
            σᵣ, σₐ = relativeScaling,  absoluteScaling # scaling factors for the  error estimate
        else
            error("All elements of relativeScaling and absoluteScaling must be non-negative!")
        end
    else
        error("relativeScaling and absoluteScaling must be vectors of length $(d)!")
    end

    if zero(S) <= minimalStepsizeScaling <= maximalStepsizeScaling
        p1, p2 = minimalStepsizeScaling, maximalStepsizeScaling
    else
        error("Minimal and maximal stepsize scaling must satisfy: 0.0 <= minimalStepsizeScaling <= maximalStepsizeScaling!")
    end

    # 2. arrays
    if maximumSteps >= 0
        Δ = zeros(S, maximumSteps + 1) # time grid
        xΔ = zeros(T, d, maximumSteps + 1) # solution
    else
        error("maximumSteps must not be negative!")
    end
    X = zeros(T, d, N_ex + 1) # storage for the internal discretisations obtained by the explicit midpoint rule
    λ = zeros(S, N_ex - n_ex +1) # storage for scaling factors of stepsize. λ[k] contains the scalar for extr. order (k + n_ex - 1)
    τ_opt =zeros(S, N_ex - n_ex +1) # storage for optimal stepsize. τ_opt[k] contains the stepsize for extr. order (k + n_ex - 1)
    s = [2*sum(Int64.(mySolver.subdividingSequence[1:n+1])) - n for n in n_ex:N_ex] # s[k] is the number of stages for  extrapolation order (k + n_ex - 1)
    statistics = detailedStatistics ? zeros(S,4,maximumSteps) : zeros(Int64,2)

    # 3. initialization for integration loop
    Δ[1], xΔ[:,1] = t₀, x₀ # store inital values
    tRest = tEnd - Δ[1] # remaining size of the interval
    if maximalSteplength > zero(S)
        maximalSteplength = min(abs(tRest),maximalSteplength)
    else
        error("maximalSteplength must be positive!")
    end

    if abs(firstGuessStepsize) > eps(S)
        τ = flipsign(min(abs(firstGuessStepsize),maximalSteplength),tRest) # stepsize for first step, accomodate backward integration
    else
        error("Absolute value of firstGuessStepsize must be greater than eps($S) = $(eps(S))!")
    end
    if n_ex <= firstGuessOrder <= N_ex
        m = firstGuessOrder # extrapolation order for first step
    else
        error("firstGuessOrder must be between $n_ex and $(N_ex)!")
    end

    counterStep = 1
    counterReduction = 0
    counterEvaluation = 0
    converged = false

    finished = maximumSteps == 0

    x_n = ones(T,d) # storage for the latest solution
    ϵ_n = zero(S) # storage for the latest error estimate
    if safetyStepsizeSelection > 0
        tol = safetyStepsizeSelection*tol # the tolerance for accuracy checks
    else
        error("safetyStepsizeSelection must be positive!")
    end

    # integration loop
    while !finished
        # the nth step: the approximation for the time t₀ + τ is saved in xΔ[n+1].

        # check if the computation must be aborted
        if counterStep > maximumSteps
            println("Not finished. Exceeded maximal number of steps ($maximumSteps)")
            return (Δ[1:counterStep], xΔ[:,1:counterStep], detailedStatistics ? statistics[:,1:counterStep-1] : statistics)
        end
        if counterReduction > maximumReduction
            println("Not finished. Too many reductions in step number $(counterStep)!")
            return (Δ[1:counterStep], xΔ[:,1:counterStep], detailedStatistics ? statistics[:,1:counterStep-1] : statistics)
        end
        if abs(τ) <= eps(S)
            println("Stepsize is becoming too small in step $(counterStep)!")
            return (Δ[1:counterStep], xΔ[:,1:counterStep], detailedStatistics ? statistics[:,1:counterStep-1] : statistics)
        end

        # preperations for current step:
        n_win, N_win = max(n_ex, m - 1), min(N_ex, m + 1) # order window
        n = n_win # start with smalles order in the order window

        # compute all necessary internal approximations:
        f₀ = f(t₀,x₀) # the information for the Euler step
        counterEvaluation = counterEvaluation + 1

        for i = 0:n
            X[:,i+1] = explicitMidpointRule(t₀,x₀,f₀,f,τ,2*Int64(mySolver.subdividingSequence[i+1]))
            counterEvaluation = counterEvaluation + 2*Int64(mySolver.subdividingSequence[i+1]) - 1
        end

        #  compute all information relating to an extrapolation order ≦ n_win:
        for i = n_ex : n
            if rescaleWeights
                # rescaling weights is equivalent to Lagrange interpolation
                x_i= X[:, 1:(i+1)]*T.(broadcast(*,mySolver.ω,mySolver.ρ')[1:(i+1),(i+1)]) # discretisation of order i
                xx_i = X[:, 2:(i+1)]*T.(broadcast(*,mySolver.ω2,mySolver.ρ2')[1:i, i]) # its internal counterpart
            else
                x_i = T.(mySolver.ρ[i+1])*(X[:, 1:(i+1)]*T.(mySolver.ω[1:(i+1), (i+1)]))
                xx_i = T.(mySolver.ρ2[i])*(X[:, 2:(i+1)]*T.(mySolver.ω2[1:i, i]))
            end
            σ = max.(σₐ, σᵣ.*abs.(x_i)) # scaling for error estimate
            ϵ_i = sqrt(sum((abs.(x_i-xx_i)./σ).^2)/d) # scaled mean square norm of the estimated error
            λ[i-n_ex+1] = min(p2, max(p1, (tol/ϵ_i)^(1/(2i+1)))) # factor for optimal stepsize, including safety scaling
            if i == n
                x_n, ϵ_n = x_i, ϵ_i # save solution and error for the current order n
            end
        end

        # check if soltution with for n in the order window can be accepted
        while n <= N_win
            if ϵ_n <= tol
                # accept order n
                converged = true
                break
            elseif ϵ_n <= tol^(s[n-n_ex+1]/s[N_win-n_ex+1])
                # reject order n but pass convergence monitor
                n = n + 1
                # compute x_n, λ_n and ϵ_n for new n:
                X[:,n+1] = explicitMidpointRule(t₀,x₀,f₀,f,τ,2*Int64.(mySolver.subdividingSequence[n+1]))
                counterEvaluation = counterEvaluation + 2Int64(mySolver.subdividingSequence[n+1]) - 1
                if rescaleWeights
                    # rescaling weights is equivalent to Lagrange interpolation
                    x_n= X[:, 1:(n+1)]*T.(broadcast(*,mySolver.ω,mySolver.ρ')[1:(n+1),(n+1)])
                    xx_n = X[:, 2:(n+1)]*T.(broadcast(*,mySolver.ω2,mySolver.ρ2')[1:n, n])
                else
                    x_n = T.(mySolver.ρ[n+1])*(X[:, 1:(n+1)]*T.(mySolver.ω[1:(n+1), (n+1)])) # approximation of extrapolation order n
                    xx_n = T.(mySolver.ρ2[n])*(X[:, 2:(n+1)]*T.(mySolver.ω2[1:n, n])) # and its internal counterpart
                end
                σ = max.(σₐ, σᵣ.*abs.(x_n)) # scaling for error estimate
                ϵ_n = sqrt(sum((abs.(x_n-xx_n)./σ).^2)/d) # scaled mean square norm of the estimated error
                λ[n-n_ex+1] = min(p2,max(p1,(tol/ϵ_n)^(1/(2n+1)))) # factor for optimal stepsize, including safety scaling

            else
                # reject order n and not pass convergence monitor
                break
            end
        end

        if converged
            #  step is accepted, save x_n for the time t₀ + τ and update statistics
            Δ[counterStep+1] = t₀ + τ
            xΔ[:, counterStep+1] = x_n
            tRest = tEnd - Δ[counterStep+1]
            maximalSteplength = min(abs(tRest),maximalSteplength)
            if detailedStatistics
                statistics[:,counterStep] = [n, τ, counterEvaluation, counterReduction]
            else
                statistics = statistics + [counterEvaluation,counterReduction]
            end

            #  check if tEnd is already reached
            if abs(tRest) < eps(S)
                finished = true
            else
                # preperations for the next step:
                t₀, x₀  = Δ[counterStep+1], xΔ[:, counterStep+1] # new inital data

                ## compute optimal order m and stepsize τ for the next step acc. Deuflhard
                # first compute the optimal extrapolation order
                temp = (n_ex:n) .- n_ex .+ 1 # index range of computed quantities
                τ_opt[temp] = min.(abs(τ)*λ[temp], maximalSteplength) # safety boundary for τ_opt
                ω = s[temp]./(τ_opt[temp]) # work per step

                m = argmin(ω) + n_ex - 1 # optimal order

                # check if we may increase m
                if m == n < N_win
                    # compute scaling for order m+1, including safety scaling
                    λ[n-n_ex+2] = min(p2, max(p1, (tol^(s[n-n_ex+1]/s[n-n_ex+2])/ϵ_n)^(1/(2n+1))))
                    # check if work dereases from order m to  m+1
                    if ω[end] > s[n-n_ex+2]/(τ_opt[n-n_ex+2] = min(abs(τ)*λ[n-n_ex+2], maximalSteplength))
                        m = m + 1
                    end
                end
                τ = flipsign(τ_opt[m-n_ex+1],τ) # accomodate backward integration

                counterStep = counterStep + 1
                converged = false
                counterEvaluation = 0
                counterReduction = 0
                fill!(λ,zero(S))
                fill!(τ_opt,zero(S))
            end
        else
            counterReduction = counterReduction + 1
            # compute reduced stepsize τ for order m. Use latest error estimate ϵ_n if estimate of order m is not available
            if n < m
                 λ[m-n_ex+1] = min(p2,max(p1,(tol^(s[n-n_ex+1]/s[m-n_ex+1])/ϵ_n)^(1/(2n+1))))
            end
            τ_opt[m-n_ex+1] = min(abs(τ)*λ[m-n_ex+1],maximalSteplength) # safety boundary for τ_opt
            τ = flipsign(τ_opt[m-n_ex+1],τ) # accomodate backward integration
        end
    end # integration loop
    return (Δ[1:counterStep+1], xΔ[:,1:counterStep+1], detailedStatistics ? statistics[:,1:counterStep] : statistics)
end

"""
    explicitMidpointRule(t₀,x₀,f₀,f,τ,N)

Computes the value ``x_\\Delta(t_0 + \\tau)`` recursively. Here ``x_\\Delta`` is the grid-function generated
by the explicit midpoint rule on the equidistant grid `[t₀ + n/N*τ for n = 0:N]`
with the initial data `(t₀,x₀)` and the right side `f`.
In the starting step `f₀` is used instead of `f(t₀,x₀)`.
"""
function explicitMidpointRule(t₀::S,x₀::Vector{T},f₀::Vector{T},f::Function,τ::S,N) where{S<:AbstractFloat, T<:Number}
    τ_n = τ/N
    x = x₀ + τ_n*f₀
    for i = 0:N-2
        temp = x
        x = x₀+2*τ_n*f(t₀+(i+1)*τ_n,x)
        x₀ = temp
    end
    return x
end

export solver
end # module AdaptiveExtrapolationD


function lgwt(N0::Integer,a::Real=-1,b::Real=1)
    # This script is for computing definite integrals using Legendre-Gauss
    # Quadrature. Computes the Legendre-Gauss nodes and weights on an interval
    # [a,b] with truncation order N
    #
    # Suppose you have a continuous function f(x) which is defined on [a,b]
    # which you can evaluate at any x in [a,b]. Simply evaluate it at all of
    # the values contained in the x vector to obtain a vector f. Then compute
    # the definite integral using sum(f.*w);
    #
    # Written by Greg von Winckel - 02/25/2004
    # First adapted to Julia by Tyler Ransom on 08-04-2015
    # Updated on 08-17-2020

    N  = N0-1
    N1 = N+1
    N2 = N+2

    xu = range(-1,stop=1,length=N1)

    # Initial guess
    y = cos.((2*(0:N) .+ 1)*pi/(2*N .+ 2))  .+  ( 0.27/N1 ) .* sin.( pi .* xu .* N/N2 )

    # Legendre-Gauss Vandermonde Matrix
    L  = zeros(N1,N2)

    # Derivative of LGVM
    Lp = zeros(N1,N2)

    # Compute the zeros of the N+1 Legendre Polynomial
    # using the recursion relation and the Newton-Raphson method

    y0 = 2

    vareps = 2e-52
    # Iterate until new points are uniformly within epsilon of old points

    i = 0
    tracker=0
    it_max=10
    while (norm(y.-y0,Inf)>vareps && tracker<=it_max)
        d=norm(y.-y0,Inf)

        L[:,1]  .= 1
        Lp[:,1] .= 0

        L[:,2] .= y

        for k=2:N1
            L[:,k+1] = ( (2*k-1)*y .* L[:,k] .- (k-1)*L[:,k-1] )/k
        end

        Lp = (N2)*( L[:,N1] .- y .* L[:,N2] )./(1 .- y.^2)
        y0 = y
        y  = y0 - L[:,N2]./Lp
        if norm(y.-y0,Inf)==d
            tracker+=1
        end
        i+=1

    end

    # Linear map from[-1,1] to [a,b]
    x = (a.*(1 .- y) .+ b .* (1 .+ y))./2

    # Compute the weights
    w=(b-a)./((1 .- y.^2).*Lp.^2)*(N2/N1)^2

    return x,w
end

#--------------------------------------------------

# Dr. Ransom Function for mlogit from PS3 solution
function mlogit_with_Z(theta, X, Z, y)

        alpha = theta[1:end-1]
        gamma = theta[end]
        K = size(X,2)
        J = length(unique(y))
        N = length(y)
        bigY = zeros(N,J)
        for j=1:J
            bigY[:,j] = y.==j
        end
        bigAlpha = [reshape(alpha,K,J-1) zeros(K)]

        T = promote_type(eltype(X),eltype(theta))
        num   = zeros(T,N,J)
        dem   = zeros(T,N)
        for j=1:J
            num[:,j] = exp.(X*bigAlpha[:,j] .+ (Z[:,j] .- Z[:,J])*gamma)
            dem .+= num[:,j]
        end

        P = num./repeat(dem,1,J)

        loglike = -sum( bigY.*log.(P) )

        return loglike
    end


### A Pluto.jl notebook ###
# v0.11.14

using Markdown
using InteractiveUtils

# ╔═╡ 4133e4a2-ed4d-11ea-11cd-0da48c09c2e8
using Pkg, DrWatson

# ╔═╡ 54bd3938-ed4d-11ea-29e4-c1c7020c1400
begin
	@quickactivate "SR2TuringPluto"
	using StatisticalRethinking
end

# ╔═╡ 0c5fe59e-ed4f-11ea-35a8-45c3aff83362
md"## Broadcasting.jl"

# ╔═╡ 54ccfc74-ed4d-11ea-366d-b738d54ff79a
p_grid = range(0, 1, length = 5)

# ╔═╡ 54d9bce8-ed4d-11ea-279e-3bc3bf465e28
prior = ones(5)

# ╔═╡ 54da91d4-ed4d-11ea-04f5-e9a4f8c659a3
likelihood = pdf.(Binomial.(9, p_grid), 6)

# ╔═╡ 54e45ff4-ed4d-11ea-0288-e3475c709007
posterior = likelihood .* prior

# ╔═╡ 54e8f906-ed4d-11ea-2fee-316c2df1d8f6
posterior ./= sum(posterior)

# ╔═╡ 54f1592c-ed4d-11ea-0210-0be7741d2604
md"##### Broadcasting is another important difference between Julia and R.

You don't have automatic
broadcasting (making functions work the same whether you run it on a single
number or on an array of numbers). However, fixing this is very easy,
you only have to annotate your function call with a dot:
  `f(single_number)` -> `f.(array_of_numbers)`.

That means the translation of the line `likelihood = pdf.(Binomial.(9, p_grid), 6)` goes something like:
1. First or every value in `p_grid` make a binomial distribution with that p value.
2. For every distribution then take the pdf of 6.

The same in the next line, this is elementwise multiplication.

For more infos read [this](https://julialang.org/blog/2017/01/moredots)."

# ╔═╡ e822d97e-ed4f-11ea-2c18-17ecfc81fdbd
md"## End of broadcasting.jl"

# ╔═╡ Cell order:
# ╟─0c5fe59e-ed4f-11ea-35a8-45c3aff83362
# ╠═4133e4a2-ed4d-11ea-11cd-0da48c09c2e8
# ╠═54bd3938-ed4d-11ea-29e4-c1c7020c1400
# ╠═54ccfc74-ed4d-11ea-366d-b738d54ff79a
# ╠═54d9bce8-ed4d-11ea-279e-3bc3bf465e28
# ╠═54da91d4-ed4d-11ea-04f5-e9a4f8c659a3
# ╠═54e45ff4-ed4d-11ea-0288-e3475c709007
# ╠═54e8f906-ed4d-11ea-2fee-316c2df1d8f6
# ╟─54f1592c-ed4d-11ea-0210-0be7741d2604
# ╟─e822d97e-ed4f-11ea-2c18-17ecfc81fdbd


"""
    advection_pde!(duhat,uhat,p,t)

RHS for the advection equation ``u_t = - u_x`` for numerical integration in Fourier space.

"""
function advection_pde!(duhat,uhat,p,t)
    N, k, dL, nu, alpha = p;
    uhat[Int(N/2)+1] = 0; # Set the most negative mode to zero to prevent an asymmetry
    duhat .= -alpha*im*k.*uhat;
end

"""
    advection_diffusion_pde!(duhat,uhat,p,t)

RHS for the advection-diffusion equation \$u_t = - u_x + ν u_{xx}\$ for numerical integration in Fourier space where ν is the viscosity.

"""
function advection_diffusion_pde!(duhat,uhat,p,t)
    N, k, dL, nu, alpha = p;
    uhat[Int(N/2)+1] = 0; # Set the most negative mode to zero to prevent an asymmetry
    duhat .= -alpha*im*k.*uhat + nu*(im*k).^2 .*uhat;
end

"""
    viscous_burgers_pde!(duhat,uhat,p,t)

RHS for the viscous Burgers equation \$u_t = - u u_x + ν u_{xx}\$ for numerical integration in Fourier space where ν is the viscosity.

"""
function viscous_burgers_pde!(duhat,uhat,p,t)
    N, k, dL, nu = p;
    alpha = 1.0;
    uhat[Int(N/2)+1] = 0.0; # Set the most negative mode to zero to prevent an asymmetry
    uhat_nonlinear = quadratic_nonlinear(uhat,N,dL,alpha);
    duhat .= -nu*k.^2 .*uhat .- uhat_nonlinear;
end

"""
    inviscid_burgers_pde!(duhat,uhat,p,t)

RHS for the inviscid Burgers equation \$u_t = - u u_x\$ for numerical integration in Fourier space.

"""
function inviscid_burgers_pde!(duhat,uhat,p,t)
    N, k, dL, nu = p;
    alpha = 1.0;
    uhat[Int(N/2)+1] = 0; # Set the most negative mode to zero to prevent an asymmetry
    uhat_nonlinear = quadratic_nonlinear(uhat,N,dL,alpha);
    duhat .= -uhat_nonlinear;
end

"""
    quadratic_nonlinear!(uhat,N,dL,alpha)

Compute the convolution sum \$\\frac{ik}{2}\\sum_{p+q=k} u_p u_q\$ resulting from the quadratic nonlinearity of Burgers equation \$u u_x\$ in Fourier space. Convolution sum is padded with the 3/2 rule for dealiasing.

"""
function quadratic_nonlinear(uhat,N,dL,alpha)
    M = Int((3/2)*N); # For dealiasing -> For MZ models must be 3*N to account for unresolved modes + dealiasing
    k_M = reduce(vcat,(2*π/dL)*[0:M/2-1 -M/2:-1]); # Wavenumbers for convolution sum with padding
    u_M = zeros(Complex{Float64},Int(M));
    u_M[1:Int(N/2)] = uhat[1:Int(N/2)];
    u_M[end-Int(N/2)+2:end] = uhat[Int(N/2)+2:end];
    u_M_sum = fft_norm(ifft_norm(u_M).*ifft_norm(u_M));
    u_M_convolution = alpha*im.*k_M/2.0 .*u_M_sum;
    uhat_nonlinear  = zeros(Complex{Float64},Int(N)); # Extract N modes from convolution
    uhat_nonlinear[1:Int(N/2)] = u_M_convolution[1:Int(N/2)];
    uhat_nonlinear[end-Int(N/2)+2:end] = u_M_convolution[end-Int(N/2)+2:end];
    return uhat_nonlinear
end

"""
    generate_fourier_solution(L1,L2,tspan,N,initial_condition,pde_function;dt=1e-3,nu=0.1,rtol=1e-10,atol=1e-14)

Generate the solution for a given `pde_function` and `initial_condition` on a periodic domain using a `N` mode Fourier expansion.

"""
function generate_fourier_solution(L1,L2,tspan,N,initial_condition,pde_function;dt=1e-3,nu=0.1,rtol=1e-10,atol=1e-14,alpha=1.0)
    # Transform random initial condition to Fourier domain
    uhat0 = fft_norm(initial_condition);
    dL = abs(L2-L1);

    # Generate Fourier Galerkin solution for N
    k = reduce(vcat,(2*π/dL)*[0:N/2-1 -N/2:-1]);
    p = [N,k,dL,nu,alpha]
    t_length = Int(round(tspan[2]/dt)+1);

    # Solve the system of ODEs in Fourier domain
    prob = ODEProblem(pde_function,uhat0,tspan,p);
    sol = solve(prob,DP5(),reltol=rtol,abstol=atol,saveat = dt)

    u_sol = zeros(t_length,N);
    for j in 1:size(sol.t,1) # Reshape output and plot
        u_sol[j,:] = real.(ifft_norm(sol.u[j]));
    end
    return u_sol, sol.u, k
end

"""
    central_difference(u_j,u_jpos,u_jneg,mu)

Compute the second order central difference for the viscous term of the viscous Burgers equation \$u_t = - u u_x + ν u_{xx}\$. `mu` is equal to \$ \\frac{\\nu}{\\Delta x^2}\$.

"""
function central_difference(u_j,u_jpos,u_jneg,mu)
    ux = zeros(size(u_j,2));
    ux = mu.*(u_jpos-2*u_j+u_jneg);
    return ux
end

"""
    minmod(x,y)

"""
function minmod(x,y)
    return sign(x)*max(0,min(abs(x),y*sign(x)));
end

"""
    ub(ulv,urv)

"""
function ub(ulv,urv)
    if ulv != urv
        return ((1/2)*urv^2 - (1/2)*ulv^2) / (urv - ulv); # ̄u_{j+/-1/2}
    else
        return ulv
    end
end

"""
    fl(ulv,urv,ubv)

"""
function fl(ulv,urv,ubv)
    return (1/2).*((1/2).*ulv.^2 .+ (1/2).*urv.^2 .- abs.(ubv).*(urv .- ulv)); # f_{j+/-1/2}
# fln(ulv,urv,ubv) = (1/2)*(flux.(urv) - flux.(ulv) - abs.(ubv).*(ulv - urv)); # f_{j+/-1/2}
end

"""
    ulpl(ujp,uj,ujn,kappa,omega)

"""
function ulpl(ujp,uj,ujn,kappa,omega)
    return uj .+ ((1-kappa)/4).*minmod.((uj .- ujn),omega.*(ujp .- uj)) .+ ((1+kappa)/4).*minmod.((ujp .- uj),omega.*(uj .- ujn)); # u_{j+1/2}^L
end

"""
    urpl(ujpp,ujp,uj,kappa,omega)

"""
function urpl(ujpp,ujp,uj,kappa,omega)
    return ujp  .- ((1+kappa)/4).*minmod.((ujp .- uj),omega.*(ujpp .- ujp)) .- ((1-kappa)/4).*minmod.((ujpp .- ujp),omega.*(ujp .- uj)); # u_{j+1/2}^R
end

"""
    ulnl(uj,ujn,ujnn,kappa,omega)

"""
function ulnl(uj,ujn,ujnn,kappa,omega)
    return ujn .+ ((1-kappa)/4).*minmod.((ujn .- ujnn),omega.*(uj .- ujn)) .+ ((1+kappa)/4).*minmod.((uj .- ujn),omega.*(ujn .- ujnn)); # u_{j-1/2}^L
end

"""
    urnl(ujp,uj,ujn,kappa,omega)

"""
function urnl(ujp,uj,ujn,kappa,omega)
    return uj .- ((1+kappa)/4).*minmod.((uj .- ujn),omega.*(ujp .- uj)) .- ((1-kappa)/4).*minmod.((ujp .- uj),omega.*(uj .- ujn)); # u_{j-1/2}^R
end

"""
    muscl_minmod_RHS!(du,u,p,t)

"""
function muscl_minmod_RHS!(du,u,p,t)
    dx, kappa, omega, nu = p;

    ulpvp = zeros(size(u,1));
    urpvp = zeros(size(u,1));
    ulnvp = zeros(size(u,1));
    urnvp = zeros(size(u,1));

    # ulp(ujp,uj,ujn,omega)
    ulpvp[2:end-1] .= ulpl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    ulpvp[1] = ulpl(u[2],u[1],u[end],kappa,omega);
    ulpvp[end] = ulpl(u[1],u[end],u[end-1],kappa,omega);

    # urp(ujpp,ujp,uj,omega)
    urpvp[1:end-2] .= urpl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    urpvp[end-1] = urpl(u[1],u[end],u[end-1],kappa,omega);
    urpvp[end] = urpl(u[2],u[1],u[end],kappa,omega);

    ubpp = ub.(ulpvp,urpvp);
    fpp = fl(ulpvp,urpvp,ubpp);

    # uln(uj,ujn,ujnn,omega)
    ulnvp[3:end] .= ulnl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    ulnvp[2] = ulnl(u[2],u[1],u[end],kappa,omega);
    ulnvp[1] = ulnl(u[1],u[end],u[end-1],kappa,omega);

    # urn(ujp,uj,ujn,omega)
    urnvp[2:end-1] .= urnl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    urnvp[1] = urnl(u[2],u[1],u[end],kappa,omega);
    urnvp[end] = urnl(u[1],u[end],u[end-1],kappa,omega);

    ubnp = ub.(ulnvp,urnvp);
    fnp = fl(ulnvp,urnvp,ubnp);

    du .= -(1/dx).*(fpp .- fnp);
end

"""
    muscl_minmod_viscous_RHS!(du,u,p,t)

"""
function muscl_minmod_viscous_RHS!(du,u,p,t)
    dx, kappa, omega, nu = p;

    mu = nu/(dx)^2;

    ulpvp = zeros(size(u,1));
    urpvp = zeros(size(u,1));
    ulnvp = zeros(size(u,1));
    urnvp = zeros(size(u,1));
    ux_viscous = zeros(size(u,1));

    # ulp(ujp,uj,ujn,omega)
    ulpvp[2:end-1] .= ulpl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    ulpvp[1] = ulpl(u[2],u[1],u[end],kappa,omega);
    ulpvp[end] = ulpl(u[1],u[end],u[end-1],kappa,omega);

    # urp(ujpp,ujp,uj,omega)
    urpvp[1:end-2] .= urpl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    urpvp[end-1] = urpl(u[1],u[end],u[end-1],kappa,omega);
    urpvp[end] = urpl(u[2],u[1],u[end],kappa,omega);

    ubpp = ub.(ulpvp,urpvp);
    fpp = fl(ulpvp,urpvp,ubpp);

    # uln(uj,ujn,ujnn,omega)
    ulnvp[3:end] .= ulnl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    ulnvp[2] = ulnl(u[2],u[1],u[end],kappa,omega);
    ulnvp[1] = ulnl(u[1],u[end],u[end-1],kappa,omega);

    # urn(ujp,uj,ujn,omega)
    urnvp[2:end-1] .= urnl(u[3:end],u[2:end-1],u[1:end-2],kappa,omega);
    urnvp[1] = urnl(u[2],u[1],u[end],kappa,omega);
    urnvp[end] = urnl(u[1],u[end],u[end-1],kappa,omega);

    ux_viscous[1] = central_difference(u[1],u[2],u[end],mu);
    ux_viscous[2:end-1] = central_difference(u[2:end-1],u[3:end],u[1:end-2],mu);
    ux_viscous[end] = central_difference(u[end],u[1],u[end-1],mu);

    ubnp = ub.(ulnvp,urnvp);
    fnp = fl(ulnvp,urnvp,ubnp);

    du .= -(1/dx).*(fpp .- fnp) .+ ux_viscous;
end

"""
    generate_muscl_minmod_solution(L1,L2,t_end,N,u0,pde_function_handle;dt=1e-4,kappa=-1)

"""
function generate_muscl_minmod_solution(L1,L2,t_end,N,u0,pde_function_handle;dt=1e-4,kappa=-1,nu=0.1)

    omega = ((3-kappa)/(1-kappa))

    dL = abs(L2-L1);
    # Set up periodic domain
    j = reduce(vcat,[0:1:N-1]);
    x = (dL.*j)./N;
    dx = x[2]-x[1];
    t = reduce(vcat,[0:dt:t_end]);

    p = [dx,kappa,omega,nu]
    t_span = (0,t_end);

    prob = ODEProblem(pde_function_handle,u0,t_span,p);
    sol = solve(prob,BS3(),reltol=1e-6,abstol=1e-8,saveat = dt);

    u_sol = zeros(size(sol.t,1),size(x,1));
    for i in 1:size(sol.t,1)
        u_sol[i,:] = sol.u[i];
    end

    return u_sol

end

"""
    generate_muscl_reduced(L1,L2,t_end,dt,M,N,ic_func,pde_function_handle;kappa=-1,nu=0.1)

"""
# function generate_muscl_reduced(L1,L2,t_end,dt,M,N,ic_func,pde_function_handle;kappa=-1,nu=0.1)
function generate_muscl_reduced(L1,L2,t_end,dt,M,N,ic,pde_function_handle;kappa=-1,nu=0.1)
    dL = abs(L2 - L1);
    j = reduce(vcat,[0:1:N-1]);
    x = (dL.*j)./N;
    j_full = reduce(vcat,[0:1:M-1]);
    x_full = (dL.*j_full)./M;
    # ic = ic_func.(x_full);
    u_full = generate_muscl_minmod_solution(L1,L2,t_end,M,ic,pde_function_handle;dt=dt,kappa=kappa,nu=nu)
    u_reduced = solution_spatial_sampling(x,x_full,u_full);
    return u_full, u_reduced
end

"""
    get_1D_energy_fft(u_solution)

Compute the energy in the Fourier domain using the scaling of \$ \\frac{1}{N} \$. Note: this does not include the 2π multiplier found in Parseval's identity for Fourier series and computes \$ \\frac{1}{2} \\sum \\vert \\hat{u}_k \\vert^2 \$.

"""
function get_1D_energy_fft(u_solution) # To Do: Add and extraction of specific mode step
    energy = zeros(size(u_solution,1))
    for i in 1:size(u_solution,1)
        u_hat = fft_norm(u_solution[i,:]);
        energy[i] = (1/2)*real((u_hat'*u_hat));
    end
    return energy
end

"""
    get_1D_energy_custom(basis,u_solution,L1,L2,weights;multiplier=1/(4*pi))

Compute the energy in the custom basis domain: \$ \\frac{1}{2} \\sum \\vert \\a_k \\vert^2 \$. Multiplier defaults to \$ \\frac{1}{4\\pi} \$ to match the Fourier calculation.

"""
function get_1D_energy_custom(basis,u_solution,nodes,weights;multiplier=1/(4*pi))
    energy = zeros(size(u_solution,1));
    for i in 1:size(u_solution,1)
        a_hat = expansion_coefficients(basis,u_solution[i,:],nodes,weights);
        energy[i] = get_1D_energy_custom_coefficients(a_hat,zeta=multiplier);
    end
    return energy
end

"""
    get_1D_energy_custom_coefficients(u_coefficients;zeta=1/(4*pi))

Compute the energy in the custom basis domain: \$ \\zeta \\sum \\vert \\a_k \\vert^2 \$. \$ \\zeta \$ defaults to \$ \\frac{1}{4\\pi} \$ to match the Fourier calculation.

"""
function get_1D_energy_custom_coefficients(u_coefficients;zeta=1/(4*pi))
        # energy[i] = (1/2)*(a_hat'*a_hat);
        energy = zeta*(u_coefficients'*u_coefficients);
    return energy
end

"""
    mode_extractor(uhat,N)

"""
function mode_extractor(uhat,N)
    uhat_N = zeros(typeof(uhat[1]),N);
    uhat_N[1:Int(N/2)] = uhat[1:Int(N/2)];
    uhat_N[end-Int(N/2)+2:end] = uhat[end-Int(N/2)+2:end];
    return uhat_N
end

"""
    get_1D_energy_upwind(u_solution_full,u_solution,N)

"""
function get_1D_energy_upwind(u_solution_full,u_solution,N)
    energy = zeros(size(u_solution,1))
    for i in 1:size(u_solution,1)
        u_upwind_fft = mode_extractor(fft_norm(u_solution_full[i,:]),N);
        energy[i] = (1/2)*real(u_upwind_fft'*u_upwind_fft);
    end
    return energy
end

"""
    spectral_approximation_fourier(x_locations,k,coefficients)

"""
function spectral_approximation_fourier(x_locations,k,coefficients)
    approximation = zeros(size(x_locations,1));
    for i = 1:length(k)
        approximation += coefficients[i]*exp.(im*k[i]*(x_locations));
    end
    return approximation
end


h₁(i::Int) = max(6 - abs(i-7), 0)
h₂(i::Int) = h₁(i - 8)
h₃(i::Int) = h₁(i - 4)

"""
  triangular_test_function(c1::Int, c2::Int, c3::Int, L::Int=32)

Generates a set of triangluar test functions with 3 classes.
"""
function triangular_test_functions(c1::Int, c2::Int, c3::Int; L::Int=32, shuffle::Bool=false)
  @assert c1 >= 0
  @assert c2 >= 0
  @assert c3 >= 0

  u = rand(Uniform(0,1),1)[1]
  ϵ = rand(Normal(0,1),(L,c1+c2+c3))

  y = string.(vcat(ones(c1), ones(c2) .+ 1, ones(c3) .+ 2))

  H₁ = Array{Float64,2}(undef,L,c1)
  H₂ = Array{Float64,2}(undef,L,c2)
  H₃ = Array{Float64,2}(undef,L,c3)
  for i in 1:L
    H₁[i,:] .= u * h₁(i) + (1 - u) * h₂(i)
    H₂[i,:] .= u * h₁(i) + (1 - u) * h₃(i)
    H₃[i,:] .= u * h₂(i) + (1 - u) * h₃(i)
  end

  H = hcat(H₁, H₂, H₃) + ϵ

  if shuffle
    idx = [1:(c1+c2+c3)...]
    shuffle!(idx)
    return H[:,idx], y[idx]
  end

  return H, y
end


using Flux
using Statistics
using CalibrationErrors
using ReliabilityDiagrams

# TODO: Work on GPU friendly implementations of these metrics

function get_mean_std_dev(x::AbstractArray; dims = ndims(x))
    μ = mean(x, dims = dims)
    σ = std(x, dims = dims, corrected = false)
    return μ, σ
end

function calculate_entropy(probs, eps = 1e-6; from_logits = true)
    if from_logits
        probs = softmax(probs, dims = 1)
    end
    return -sum(probs .* log.(probs .+ eps), dims = 1)
end

function mean_loglikelihood(preds, labels)
    return mean(log(p[s]) for (s, p) in zip(labels, preds))
end

function brier_score(preds, labels)
    return mean(
        sum(abs2(pi - (i == s)) for (i, pi) in enumerate(p)) for
        (s, p) in zip(labels, preds)
    )
end

function expected_calibration_error(preds, labels, num_bins = 10; from_logits = true)
    if from_logits
        preds = softmax(preds, dims = 1)
    end
    preds = cpu(preds)
    labels = cpu(labels)
    preds = [x for x in eachcol(preds)]
    ece_estimator = ECE(UniformBinning(num_bins), SqEuclidean())
    ece = ece_estimator(preds, labels)
    return ece
end

function prediction_metrics(preds, labels)
    metrics = Dict()
    preds = [x for x in eachcol(preds)]
    # Calulate mean loglikelihood
    metrics["mean_log_likelihood"] = mean_loglikelihood(preds, labels)
    # Brier Score
    metrics["brier_score"] = brier_score(preds, labels)
    return metrics
end