#' ---
#' title: Intro to Julia
#' ---

#' **Originally Contributed by**: Juan Pablo Vielma

#' ## Introduction

#' Since JuMP is embedded in Julia, knowing some basic Julia is important
#' for learning JuMP. This notebook is designed to provide a minimalist
#' crash course in the basics of Julia. You can find resources that provide
#' a more comprehensive introduction to Julia [here](https://julialang.org/learning/).

#' ### How to Print
#' In Julia, we usually use println() to print

println("Hello, World!")

#' ### Basic Data Types
#' Integers

typeof(1 + -2)

#' Floating point numbers

typeof(1.2 - 2.3)

#' There are also some cool things like an irrational representation of π. To make π
#' (and most other greek letters), type \pi and then press [TAB].

π

#+

typeof(π)

#' Julia has native support for complex numbers

typeof(2 + 3im)

#' Double quotes are used for strings

typeof("This is Julia")

#' Unicode is fine in strings

typeof("π is about 3.1415")

#'Julia symbols provide a way to make human readable unique identifiers.

:my_id
typeof(:my_id)

#' ### Arithmetic and Equality Testing

#' Julia is great for math

1 + 1

#'Even math involving complex numbers

(2 + 1im) * (1 - 2im)

#' We can also write things like the following using √ (\sqrt)

sin(2π/3) == √3/2

#' Wait. What???

sin(2π/3) - √3/2

#' Let's try again using ≈ (\approx).

sin(2π/3) ≈ √3/2

#' Note that this time we used ≈ instead of ==. That is because computers don't use
#' real numbers. They use a discrete representation called floating point. If you aren't
#' careful, this can throw up all manner of issues. For example:

1 + 1e-16 == 1

#' It even turns out that floating point numbers aren't associative!

(1 + 1e-16) - 1e-16 == 1 + (1e-16 - 1e-16)

#' ### Vectors, Matrices and Arrays
#' Similar to Matlab, Julia has native support for vectors, matrices and tensors; all of which are represented by arrays of different dimensions.
#' Vectors are constructed by comma-separated elements surrounded by square brackets:

b = [5, 6]

#' Matrices can by constructed with spaces separating the columns, and semicolons separating the rows:

A = [1 2; 3 4]

#' We can do linear algebra:

x = A \ b

#+

A * x

#+

A * x == b

#' Note that when multiplying vectors and matrices, dimensions matter. For example, you can't multiply a vector by a vector:
#+ tangle = false

b * b

#' But multiplying transposes works:

@show b' * b
@show b * b';

#' ### Tuples
#' Julia makes extensive use of a simple data structure called Tuples.  Tuples are immutable collections of values.
#' For example,

t = ("hello", 1.2, :foo)

#+

typeof(t)

#' Tuples can be accessed by index, similar to arrays,

t[2]

#' And can be "unpacked" like so,

a, b, c = t
b

#' The values can also be given names, which is a convenient way of making light-weight data structures.

t = (word="hello", num=1.2, sym=:foo)

#' Then values can be accessed using a dot syntax,

t.word

#' ### Dictionaries
#' Similar to Python, Julia has native support for dictionaries.  Dictionaries provide a very generic way of mapping keys to values.  For example, a map of integers to strings,

d1 = Dict(1 => "A", 2 => "B", 4 => "D")

#' Looking up a values uses the bracket syntax,

d1[2]

#' Dictionaries support non-integer keys and can mix data types,

Dict("A" => 1, "B" => 2.5, "D" => 2 - 3im)

#' Dictionaries can be nested

d2 = Dict("A" => 1, "B" => 2, "D" => Dict(:foo => 3, :bar => 4))

#+

d2["B"]

#+

d2["D"][:foo]

#' ### For-Each Loops
#' Julia has native support for for-each style loops with the syntax `for <value> in <collection> end`.

for i in 1:5
    println(i)
end

#+

for i in [1.2, 2.3, 3.4, 4.5, 5.6]
    println(i)
end

#' This for-each loop also works with dictionaries.

for (key, value) in Dict("A" => 1, "B" => 2.5, "D" => 2 - 3im)
    println("$key: $value")
end

#' Note that in contrast to vector languages like Matlab and R, loops do not result in a significant performance degradation in Julia.

#' ### Control Flow
#' Julia control flow is similar to Matlab, using the keywords `if-elseif-else-end`, and the logical operators `||` and `&&` for *or* and *and* respectively.

i = 10
for i in 0:3:15
    if i < 5
        println("$(i) is less than 5")
    elseif i < 10
        println("$(i) is less than 10")
    else
        if i == 10
            println("the value is 10")
        else
            println("$(i) is bigger than 10")
        end
    end
end

#' ### Comprehensions
#' Similar to languages like Haskell and Python, Julia supports the use of simple loops in the construction of arrays and dictionaries, called comprehenions.
#'
#' A list of increasing integers,

[i for i in 1:5]

#' Matrices can be built by including multiple indices,

[i*j for i in 1:5, j in 5:10]

#' Conditional statements can be used to filter out some values,

[i for i in 1:10 if i%2 == 1]

#' A similar syntax can be used for building dictionaries

Dict("$i" => i for i in 1:10 if i%2 == 1)

#' ### Functions
#' A simple function is defined as follows,

function print_hello()
    println("hello")
end
print_hello()

#' Arguments can be added to a function,

function print_it(x)
    println(x)
end
print_it("hello")
print_it(1.234)
print_it(:my_id)

#' Optional keyword arguments are also possible

function print_it(x; prefix="value:")
    println("$(prefix) $x")
end
print_it(1.234)
print_it(1.234, prefix="val:")

#' The keyword `return` is used to specify the return values of a function.

function mult(x; y=2.0)
    return x * y
end
mult(4.0)

#+

mult(4.0, y=5.0)

#' ### Other notes on types
#' Usually, specifing types is not required to use Julia.  However, it can be helpful to understand the basics of Julia types for debugging.
#' For example this list has a type of `Array{Int64,1}` indicating that it is a one dimensional array of integer values.

[1, 5, -2, 7]

#' In this example, the decimal values lead to a one dimensional array of floating point values, i.e. `Array{Float64,1}`.  Notice that the integer `7` is promoted to a `Float64`, because all elements in the array need share a common type.

[1.0, 5.2, -2.1, 7]

#' ### Mutable vs immutable objects
#' Some types in Julia are *mutable*, which means you can change the values inside them. A good example is an array. You can modify the contents of an array without having to make a new array.
#' In contrast, types like `Float64` are *immutable*. You can't modify the contents of a `Float64`.
#' This is something to be aware of when passing types into functions. For example:

function mutability_example(mutable_type::Vector{Int}, immutable_type::Int)
    mutable_type[1] += 1
    immutable_type += 1
    return
end

mutable_type = [1, 2, 3]
immutable_type = 1

mutability_example(mutable_type, immutable_type)

println("mutable_type: $(mutable_type)")
println("immutable_type: $(immutable_type)")

#' Because `Vector{Int}` is a mutable type, modifying the variable inside the function changed the value outside of the function. In constrast, the change to `immutable_type` didn't modify the value outside the function.
#' You can check mutability with the `isimmutable` function.

@show isimmutable([1, 2, 3])
@show isimmutable(1);

#' ### Using Packages and the Package Manager
#' No matter how wonderful Julia's base language is, at some point you will want to use an extension package.  Some of these are built-in, for example random number generation is available in the `Random` package in the standard library. These packages are loaded with the commands `using` and `import`.

using Random
Random.seed!(33);

[rand() for i in 1:10]

#' The Package Manager is used to install packages that are not part of Julia's standard library.
#' For example the following can be used to install JuMP,
#+ tangle = false

using Pkg
Pkg.add("JuMP")

#' For a complete list of registed Julia packages see the package listing at https://pkg.julialang.org/.
#' From time to you may wish to use a Julia package that is not registered.  In this case a git repository URL can be used to install the package.
#+ tangle = false

using Pkg
Pkg.add("https://github.com/user-name/MyPackage.jl.git")

#' Note that for clarity this example uses the package manager `Pkg`.  Julia includes an interactive package manager that can be accessed using `]`.  [This video](https://youtu.be/76KL8aSz0Sg) gives an overview of using the interactive package manager environment.
#' The state of installed packages can also be saved in two files: `Project.toml` and `Manifest.toml`. If these files are stored in the same directory than a notebook, the state of the packages can be recovered by running
#+ tangle = false
import Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()

#' ### HELP!
#' Julia includes a help mode that can be accessed using `?`.  Entering any object (e.g. function, type, struct, ...) into the help mode will show its documentation, if any is available.

#' ### Some Common Gotchas

#' #### MethodError
#' A common error in Julia is `MethodError`, which indicates that the function is not defined for the given value.  For example, by default the `ceil` function is not defined for complex numbers.  The "closest candidates" list suggest some Julia types that the function is defined for.
#+ tangle = false

ceil(1.2 + 2.3im)


# poissonreg.jl
#
# Code for poisson regression.

using Distributions;

# (A, b, x) = GeneratePoissonData(m::Int64 = 200, n::Int64 = 20)
#
# Generates an m-by-n matrix A of measurement vectors a_i = A[i, :], where each
# a_i is uniform on the unit sphere. Chooses x uniformly from the sphere of
# radius sqrt(n), and draws b_i as
#
#   b_i ~ Poisson(exp(a_i' * x))
function GeneratePoissonData(m::Int64 = 200, n::Int64 = 20;
                             generate_gaussian::Bool = false,
                             condition_number::Float64 = 1.0)
  A = randn(m, n);
  if (!generate_gaussian)
    A = A ./ sqrt.(repeat(sum(A.^2, dims=2), outer = (1, n)));
  end
  if (condition_number > 1.0)
    # Make A have condition number given
    condition_vec = Vector(LinRange(1.0, condition_number, n));
    condition_vec = condition_vec / norm(condition_vec);
    A = A .* repeat(condition_vec', outer = (m, 1));
  end
  x = randn(n);
  x = sqrt(n) * x / norm(x);
  b = zeros(Int64, m);
  for ii = 1:m
    R = Poisson(exp(dot(A[ii, :], x)));
    b[ii] = rand(R);
  end
  return (A, b, x);
end

# PoissonObj(A::Matrix{Float64}, b::Vector{Int64}, x::Vector{Float64})
#
# Computes the (average) log loss for the poisson regression problem,
# where we recall that the log-likelihood for b ~ Poisson(lambda) is
#
#  p(b | lambda) = exp(-lambda) * lambda^b / b!
#
# For a data vector a, the parameter lambda = exp(a' * x), so we have log loss
# (ignoring the factorial term)
#
# l(x; (a, b)) = -b * dot(a, x) + exp(dot(a, x)).
function PoissonObj(A::Matrix{Float64}, b::Vector{Int64}, x::Vector{Float64})
  mm = size(A, 1);
  objective = -sum(b .* (A * x)) + sum(exp.(A*x)); # no need to add b!
  return objective / mm;
end

# g = FullPoissonGradient(A, b, x)
#
# Sets g to be the gradient of the poisson objective and returns it.
function FullPoissonGradient(A::Matrix{Float64}, b::Vector{Int64},
                             x::Vector{Float64})
  mm = size(A, 1);
  grad = (-A' * b + A' * exp.(A * x)) / mm;
  return grad;
end

# x = ProximalUpdatePoisson(a::Vector{Float64}, b::Float64,
#                              x_init::Vector, stepsize::Float64; accuracy)
#
# Sets x to minimize
#
#  -b * a' * x + exp(x' * a) + norm(x - x_init)^2 / (2 * stepsize).
#
# Returns the minimizing x. The accuracy parameter governs the
# accuracy of the solution: we recommend a line search.
function ProximalUpdatePoisson(a::Vector{Float64}, b::Int64,
                               x_init::Vector{Float64}, stepsize::Float64;
                               eps_accuracy::Float64 = 1e-10)
  # Replace this code to return the correct update
  return x_init;
end

# x = TruncatedUpdatePoisson(a::Vector{Float64}, b::Int64,
#                            x_init::Vector{Float64}, stepsize::Float64)
#
# Let l(x; (a, b)) = -b * a' * x + exp(a' * x). Sets x to minimize the
# lower-bounded approximation to F at x_init, that is, for
#
#  l_approx(x) = max{ l(x_init) + l'(x_init) * (x - x_init)),
#                     inf_x l(x; (a, b)) }
#
# sets x to minimize
#
#  l_approx(x) + norm(x - x_init)^2 / (2 * stepsize).
#
# Returns the minimizing x.
function TruncatedUpdatePoisson(a::Vector{Float64}, b::Int64,
                                x_init::Vector{Float64}, stepsize::Float64)
  # Replace this code to return the correct update
  return x_init;
end

# x = SGUpdatePoisson(a::Vector{Float64}, b::Int64,
#                     x_init::Vector{Float64}, stepsize::Float64)
#
# Computes the stochastic gradient update for Poisson regression.
function SGUpdatePoisson(a::Vector{Float64}, b::Int64,
                         x_init::Vector{Float64}, stepsize::Float64)
  # Replace this code to return the correct update
  return x_init;
end

Machine Learning Summer School 2019 (London, UK)/tutorials/optimization/poissonreg.jl


#=
Plot the dividend process and the state process for the Markov asset pricing
lecture.

=#

using QuantEcon
using Plots
using LaTeXStrings

pyplot()

n = 25
mc = tauchen(n, 0.96, 0.25)
sim_length = 80

x_series = simulate(mc, sim_length; init=round(Int, n / 2))
lambda_series = exp(x_series)
d_series = cumprod(lambda_series) # assumes d_0 = 1

series = [x_series lambda_series d_series log(d_series)]
labels = [L"$X_t$" L"$g_t$" L"$d_t$" L"$log (d_t)$"]
plot(series, layout=4, labels=labels)


# граница xэмминга про упаковку шарами
function hammingND(n, d)
  t = div(d - 1, 2)
  binoms = [binomial(n, i) for i in 0:t]
  println("binoms = $binoms")
  z = sum(binoms)
  println("z = $z")
  M = 2^n / z
  return (M, log2(M))
end

function hammingNK(n, k)
  t = 0
  b = 2^(n - k)
  while sum([binomial(n, i) for i in 0:t]) ≤ b
    t += 1
  end
  return t - 1
end

function varshamovGuilbertND(n, d)
  k = 0
  binomSum = sum([binomial(n - 1, i) for i in 0:d-2])
  println("binomSum = $binomSum")

  while 2^(n - k) > binomSum
    k += 1
  end

  return k - 1
end

function varshamovGuilbertNK(n, k)
  d = 2
  b = 2^(n - k)
  while b > sum([binomial(n - 1, i) for i in 0:d-2])
    d += 1
  end
  return d - 1
end

function graismer(k, d)
  n = 0
  for i = 0:k-1
    n += ceil(Int64, d / 2^i)
  end
  return n
end

function graismerD(n, k)
  # n > sum(d/2^i) from 0 to k
  d = 0
  while true
    sums = sum([ceil(Int64, d / 2^i) for i in 0:k-1])
    if n > sums
      d += 1
    else
      break
    end
  end

  return d - 1
end

n, k, d = 21, 13, 14

# наxождение k по n и d
#
# M, klog = hammingND(n, d)
# println("граница Хэмминга: M ≤ $M, k ≈ $(klog)")

# println("==============================")

# mink = varshamovGuilbertND(n, d)
# println("граница Варшамова-Гилберта: k ≥ $mink")

# println("==============================")
# println(graismer(9, d))
# println("###################################")

# наxождение d по n и k
#
t = hammingNK(n, k)
println("граница Хэмминга: t ≤ $t")

println("==============================")

mind = varshamovGuilbertNK(n, k)
println("граница Варшамова-Гилберта: d ≥ $mind")

println("==============================")
println("граница Грайсмера: $(graismerD(n, k))")
println("###################################")


### ntt.jl
#
# Copyright (C) 2019 Jakub Wronowski.
#
# Maintainer: Jakub Wronowski <jakubwro@users.noreply.github.com>
# Keywords: number theoretic transform
#
# This file is a part of NumberTheoreticTransforms.jl.
#
# License is MIT.
#
### Commentary:
#
# This file contains implementation of general Number Theoretic Transform.
#
### Code:

export ntt, intt

"""
Constraints on NTT params to ensure that inverse can be computed
"""
function validate(N, g, q)
    @assert mod(q - 1, N) == 0
    @assert powermod(g, N, q) == 1
    @assert !(1 in powermod.(g, 2:N-1, q)) # this may be redundant
    @assert gcd(N, q) == 1
end

"""
    ntt(x::Array{T,1}, g, q) -> Array{T,1}

The [Number Theoretic Transform](https://en.wikipedia.org/wiki/Discrete_Fourier_transform_(general)#Number-theoretic_transform)
transforms data in a similar fashion to DFT, but instead complex roots of unity
it uses integer roots with all operation defined in a finite field (modulo an
integer number).

`ntt` function implements Number Theoretic Transform directly from the formula,
so it is flexible about choosing transformation params but lacks performance.

``\\bar{x}_k = \\sum_{n=1}^N{x_n g^{(n-1)(k-1)} } \\mod q``

There is also a few constraints on choosing parameters and input length to
ensure that inverse exists and equals to the original input:
-   ``g`` must ``N``-th root of one in modulo ``q`` arithmetic
-   ``q-1`` mod ``N`` must be equal zero
-   ``q`` must be grater than maximum element present in transformed array

To find parameter set you may use fint-ntt script.

The arguments of `ntt` function are

-   `x`: input data, its elements must be smaller than q
-   `g`: transform power base, must have inversion modulo q
-   `q`: defines modulo arithmetic
"""
function ntt(x::Array{T,1}, g::T, q::T) where {T<:Integer}
    N = length(x)
    #TODO: more validation of p,q, decompose it to struct
    #TODO: create transform object that validates input in the constructor

    validate(N, g, q)
    (lo, hi) = extrema(x)
    @assert lo >= 0
    @assert hi <= q-1

    t = [powermod(g, n * k, q) for n in 0:N-1, k in 0:N-1]
    #TODO: make result of ntt a struct that will hold infrmation about g
    return mod.(t * x, q)
end

# TODO: change implementation to handle any number of dimensions
# it can be done calling N-1 dimensional transform in a loop

function ntt(x::Array{T,2}, g::T, q::T) where {T<:Integer}
    N, M = size(x)
    @assert N == M #TODO: make it work for N != M (need different g for each dim)
    y = zeros(T, size(x))

    for n in 1:N
        y[n, :] = ntt(x[n, :], g, q)
    end

    for m in 1:M
        y[:, m] = ntt(y[:, m], g, q)
    end

    return y
end

"""
    intt(y::Array{T,1}, g, q) -> Array{T,1}

Inverse Number Theoretic Transform implementation directly from the formula.

``x_k = N^{-1} \\sum_{n=1}^N{\\bar{x}_n g^{-(n-1)(k-1)} } \\mod q``

The same input parameters constraints as for `ntt` function must be applied
"""
function intt(y::Array{T,1}, g::T, q::T) where {T<:Integer}
    N = length(y)

    validate(N, g, q)

    inv_g = invmod(g, q)
    inv_N = invmod(N, q)
    t = [powermod(inv_g, l * k, q) for k in 0:N-1, l in 0:N-1]
    return mod.(inv_N * t * y, q)
end

function intt(y::Array{T,2}, g::T, q::T) where {T<:Integer}
    N, M = size(y)
    x = zeros(T, size(y))

    for m in 1:M
        x[:, m] = intt(y[:, m], g, q)
    end

    for n in 1:N
        x[n, :] = intt(x[n, :], g, q)
    end

    return x
end

# those implementations are incorrect but transform results were interesing anyway
# TODO: check if those calculations are useful

# function ntt(g::T, q::T, x::Array{T,N}) where {T <: Integer,N}
#     return reshape(ntt(g, q, reshape(x, length(x))), size(x))
# end

# function intt(g::T, q::T, y::Array{T,N}) where {T <: Integer,N}
#     return reshape(intt(g, q, reshape(y, length(y))), size(y))
# end


export RDFT

"""
`RDFT([domainType=Float64::Type,] dim_in::Tuple [,dims=1])`

`RDFT(dim_in...)`

`RDFT(x::AbstractArray [,dims=1])`

Creates a `LinearOperator` which, when multiplied with a real array `x`, returns the DFT over the dimension `dims`, exploiting Hermitian symmetry.

```julia
julia> RDFT(Float64,(10,10))
ℱ  ℝ^(10, 10) -> ℂ^(6, 10)

julia> RDFT((10,10,10),2)
ℱ  ℝ^(10, 10, 10) -> ℂ^(10, 6, 10)

```

"""
struct RDFT{T <:Number,
	       N,
	       T1<:AbstractFFTs.Plan,
	       T2<:AbstractFFTs.Plan,
	       T3<:AbstractArray{Complex{T},N}
	      } <: LinearOperator
	dim_in::NTuple{N,Int}
	dim_out::NTuple{N,Int}
	A::T1
	At::T2
	b2::T3
	y2::T3
	Zp::ZeroPad{Complex{T},N}
end

# Constructors
#standard constructor

function RDFT(x::AbstractArray{T,N}, dims::Int=1) where {T<:Real,N}
	A = plan_rfft(x,dims)
	b2 = zeros(complex(T),size(x))
	y2 = zeros(complex(T),size(x))
	At = plan_bfft(y2,dims)
	dim_in = size(x)
	dim_out = ()
	for i = 1:N
		dim_out = i == dims ? (dim_out...,div(dim_in[i],2)+1) : (dim_out...,dim_in[i])
	end
	Z = ZeroPad(Complex{T},dim_out,size(b2))
	RDFT{T,N,typeof(A),typeof(At),typeof(b2)}(dim_in,dim_out,A,At,b2,y2,Z)
end

RDFT(T::Type,dim_in::NTuple{N,Int},dims::Int=1) where {N} = RDFT(zeros(T,dim_in),dims)
RDFT(dim_in::NTuple{N,Int},dims::Int=1) where {N} = RDFT(zeros(dim_in),dims)
RDFT(dim_in::Vararg{Int}) = RDFT(dim_in)
RDFT(T::Type,dim_in::Vararg{Int}) = RDFT(T,dim_in)

# Mappings

function mul!(y::T3,
              L::RDFT{T,N,T1,T2,T3},
              b::T4) where {N,T,T1,T2,T3,T4 <: AbstractArray{T,N}}
	mul!(y,L.A,b)
end

function mul!(y::T4,
              L::AdjointOperator{RDFT{T,N,T1,T2,T3}},
              b::T3) where {N,T,T1,T2,T3,T4 <: AbstractArray{T,N}}
    A = L.A
	mul!(A.b2,A.Zp,b)
	mul!(A.y2,A.At,A.b2)
	y .= real.(A.y2)
	return y

end

# Properties

size(L::RDFT) = (L.dim_out,L.dim_in)

fun_name(A::RDFT) = "ℱ"

domainType(L::RDFT{T,N}) where {T,N} = T
codomainType(L::RDFT{T,N}) where {T,N} = Complex{T}

is_AAc_diagonal(    L::RDFT) = false #TODO but might be true?
is_invertible(      L::RDFT) = true
is_full_row_rank(   L::RDFT) = true


"""
Area scaling factor for a function f with partial derivatives fx, fy
    * fx::Function
    * fy::Function

Returns:
    * [Function] : Area scaling factor at x.
"""
function af(fx, fy)
    return x -> norm(fx(x)×fy(x))
end

function af(f)
    # Find partial derivatives
    fx(x) = ForwardDiff.derivative(z -> f([z, x[2]]), x[1])
    fy(x) = ForwardDiff.derivative(z -> f([x[1], z]), x[2])

    # Compute area scaling factor
    return x -> norm(fx(x)×fy(x))
end

"""
Derivative of area scaling factor for a function f with derivatives
    * fx::Function
    * fy::Function
    * fxx::Function
    * fxy::Function
    * fyy::Function

Returns:
    * [Function] : Derivative of area scaling factor at x.
"""
function Daf(af, fx, fy, fxx, fxy, fyy)
    return x -> [
        (fxx(x) × fy(x) - fxy(x) × fx(x)) ⋅ (fx(x) × fy(x)) / af(x)
        (fxy(x) × fy(x) - fyy(x) × fx(x)) ⋅ (fx(x) × fy(x)) / af(x)
    ]
end

"""
Get q-map and derivative of a given function
    * f::Function

Returns:
    * q::Function, Dq::Function
"""
function Qmap(f)
    # Find partial derivatives
    fx(x) = ForwardDiff.derivative(z -> f([z, x[2]]), x[1])
    fy(x) = ForwardDiff.derivative(z -> f([x[1], z]), x[2])

    # Compute area scaling factor
    a = af(fx, fy)

    # Return q-map and derivative
    q(x) = √a(x) * f(x)
    Dq(x) = ForwardDiff.jacobian(q, x)
    return q, Dq
end

"""
Get q-map and derivative of a given function with derivatives
    * f::Function
    * fx::Function
    * fy::Function

Returns:
    * q::Function, Dq::Function
"""
function Qmap(f, fx, fy)
    a = af(fx, fy)

    q(x) = √a(x) * f(x)
    Dq(x) = ForwardDiff.jacobian(q, x)
    return q, Dq
end

"""
Get q-map and derivative of a given function with derivatives
    * f::Function
    * fx::Function
    * fy::Function
    * fxx::Function
    * fxy::Function
    * fyy::Function

Returns:
    * q::Function, Dq::Function
"""
function Qmap(f, fx, fy, fxx, fxy, fyy)
    a = af(fx, fy)
    Da = Daf(a, fx, fy, fxx, fxy, fyy)

    Df(x) = [fx(x) fy(x)]
    q(x) = √a(x) * f(x)
    Dq(x) = 1. / (2. * √a(x)) * f(x) * Da(x)' + √a(x) * Df(x)
    return q, Dq
end

"""
Compute the centroid of the parametric surface.
    * f::Function
    * af:: Function

Returns:
    * [Array{Float}] : Vector value of surface centre
"""
function surface_centroid(f, af)
    return hcubature(x -> f(x)*af(x), [0, 0], [1, 1])[1] / hcubature(af, [0, 0], [1, 1])[1]
end

"""
Translate function so that centroid coincides with the origin.
    * f::Function
    * af:: Function

Returns:
    * [Function] : Translated Surface
"""
function centered(f, af)
    c = surface_centroid(f, af)
    return x -> f(x) - c
end

"""
Compute the parametric surface area.
    * f::Function
    * af:: Function

Returns:
    * [Float] : Surface Area
"""
function surface_area(f, af)
    return hcubature(af, [0, 0], [1, 1])[1]
end

"""
Scale surface to unit area.
    * f::Function
    * af:: Function

Returns:
    * [Function] : Scaled Surface
"""
function scale_surface(f, af)
    s = surface_area(f, af)
    return x -> f(x) / s
end

"""
Translate and scale surface
    * f::Function

Returns:
    * [Function] : Scaled Surface
"""
function normalize_surface(f)
    # Find partial derivatives
    fx(x) = ForwardDiff.derivative(z -> f([z, x[2]]), x[1])
    fy(x) = ForwardDiff.derivative(z -> f([x[1], z]), x[2])

    # Compute area scaling factor
    a = af(fx, fy)

    c = surface_centroid(f, a)
    s = surface_area(f, a)
    return x -> (f(x) - c) / s
end

"""
Translate and scale surface
    * f::Function
    * af:: Function

Returns:
    * [Function] : Scaled Surface
"""
function normalize_surface(f, af)
    c = surface_centroid(f, af)
    s = surface_area(f, af)
    return x -> (f(x) - c) / s
end


using Plots; plotlyjs()
using FFTW

# Construct a signal
f = 1
fs = 5
ts = 1 / fs
l = 2^10 + 1
t = (-(l - 1) ÷ 2 : (l - 1) ÷ 2) * ts
x = sinc.(f * t)

# Construct the frequency spectrum
xf = fft(x)
ff = (0 : 1 / (l - 1) : 1) * fs

# Signal energy
Δt = ts
Δf = fs / (l - 1)
Et = sum(abs.(x).^2) * Δt
Ef = sum(abs.(xf).^2) * Δf / (fs^2)
@show Et, Ef

# Plots
plt = plot(layout=2)
plot!(t, x, subplot=1)
plot!(ff, abs.(xf), subplot=2)
display(plt)


# This file is pat of RLinearAlgebra.jl
# 1. Specifies type
# 2. Implements rsubsolve function
# 3. Exports Type

#using LinearAlgebra

"""
    LinSysVecRowProjFO <: LinSysVecRowProjection

A mutable structure that represents the standard row projection method
with full orthogonalization against all previous projections. Equivalently, this type
represents a solver for incrementally constructed matrix sketches.

See McCormick, S. F. "The methods of Kaczmarz and row orthogonalization for solving linear
    equations and least squares problems in Hilbert space." Indiana University Mathematics
    Journal 26.6 (1977): 1137-1150. https://www.jstor.org/stable/24891603

See Patel, Vivak, Mohammad Jahangoshahi, and Daniel Adrian Maldonado. "An Implicit
    Representation and Iterative Solution of Randomly Sketched Linear Systems."
    SIAM Journal on Matrix Analysis and Applications (2021) 42:2, 800-831.
    https://doi.org/10.1137/19M1259481

# Fields

- `S::Union{Matrix{Float64}, Nothing}` is a matrix used for orthogonalizing against
    all previous search directions.

Calling `LinSysVecRowProjFO()` defaults to `LinSysVecRowProjFO(nothing)`.
"""
mutable struct LinSysVecRowProjFO <: LinSysVecRowProjection
    S::Union{Matrix{Float64}, Nothing}
end
LinSysVecRowProjFO() = LinSysVecRowProjFO(nothing)

# Common rsubsolve interface for linear systems
function rsubsolve!(
    type::LinSysVecRowProjFO,
    x::AbstractVector,
    samp::Tuple{U,V} where {U<:AbstractVector,V<:Real},
    iter::Int64,
)
    # samp[1] is vector in the row space of the coefficient matrix
    # samp[2] is a scalar corresponding to samp[1]

    # Allocate space for orthogonalization matrix S
    if iter == 1
        d = length(x)
        type.S = diagm(ones(Float64, d))
    end

    #Compute orthogonal component of q using S (projection matrix)
    u = type.S * samp[1]

    if dot(u,u) < eps()^2 * length(x)
        return nothing
    end

    #Update Iterate
    res = samp[2] - dot(samp[1], x)
    γ = dot(u, samp[1])
    x .= x + u * ( res / γ )

    #Update Projection matrix
    type.S .= (I - (u/γ)*samp[1]')*type.S

    return nothing
end


function isValid(input::String)
    checkNine(n) = n > 9 ? n - 9 : n

    input = filter(x -> x != ' ' ,collect(input))

    length(input) <= 1 && return false

    parsedArr = try
        map(x -> parse(Int,x), input)
    catch
        return false
    end

    flag = true
    result = 0
    for i in reverse(parsedArr)
        if flag
            result += i
        else
            result += checkNine(2 * i)
        end
        flag = !flag
    end

    0 == (result % 10)
end


using Pkg
Pkg.activate(pwd())

# # Neural networks
# ## Prepare data

using RDatasets

iris = dataset("datasets", "iris")
X = Matrix(iris[:, 1:4])
y = iris.Species

#### Exercise:
#Write the `split` function, which randomly splits the dataset and the labels into training
#and testing sets. Its input should be the dataset `X` and the labels `y`. It should have
#four outputs. Include 80% of data in the training set and 20% of data in the testing set
# by default.
#
#**Hints:**
#- Use the `randperm` function from the `Random` package.
#- While `y` can be assumed to a vector, `X` is a matrix or a more-dimensional array. Then
#it is beneficial to use the `selectdim` function to select subindices along the correct
#dimension.
#
#---
#### Solution:

using Random

function split(X, y::AbstractVector; dims=1, ratio_train=0.8, kwargs...)
    n = length(y)
    size(X, dims) == n || throw(DimensionMismatch("..."))

    n_train = round(Int, ratio_train*n)
    i_rand = randperm(n)
    i_train = i_rand[1:n_train]
    i_test = i_rand[n_train+1:end]

    return selectdim(X, dims, i_train), y[i_train], selectdim(X, dims, i_test), y[i_test]
end

#+

X_train, y_train, X_test, y_test = split(X, y)

# ---
#
# ### Exercise:
# Write the `normalize` functions as described above. It should have two inputs and two
# outputs. The keyword argument `dims` should also be included.
#
# **Hint**: check the help for the `mean` function.
#
# ---
# ### Solution:

using Statistics

function normalize(X_train, X_test; dims=1, kwargs...)
    col_mean = mean(X_train; dims)
    col_std = std(X_train; dims)

    return (X_train .- col_mean) ./ col_std, (X_test .- col_mean) ./ col_std
end

#+

X_train, X_test = normalize(X_train, X_test)

# ---
#
# ### Exercise:
# Write the `onehot` function that converts the labels `y` into their one-hot
# representation. The samples should be along the second dimension. Write the `onecold`
# function that converts the one-hot representation into the one-cold (original)
# representation. Both these functions need to have two arguments; the second one is
# `classes`, which equals `unique(y)`.
#
# Write a check that both functions work correctly.
#
# **Hints:**
# - The one-hot representation for a label has the size equalling to the number of classes.
# All entries besides one are zeros.
# - Since the one-hot representation represents probabilities, the prediction is the class
# with the highest probability.
#
# ---
# ### Solution:

function onehot(y, classes)
    y_onehot = falses(length(classes), length(y))
    for (i, class) in enumerate(classes)
        y_onehot[i, y .== class] .= 1
    end
    return y_onehot
end

onecold(y, classes) = [classes[argmax(y_col)] for y_col in eachcol(y)]

#+

classes = unique(y)
isequal(onecold(onehot(y, classes), classes), y)

# ---

using LinearAlgebra

function prepare_data(X, y; do_normal=true, do_onehot=true, kwargs...)
    X_train, y_train, X_test, y_test = split(X, y; kwargs...)

    if do_normal
        X_train, X_test = normalize(X_train, X_test; kwargs...)
    end

    classes = unique(y)

    if do_onehot
        y_train = onehot(y_train, classes)
        y_test = onehot(y_test, classes)
    end

    return X_train, y_train, X_test, y_test, classes
end

#+

Random.seed!(666)

iris = dataset("datasets", "iris")
X = Matrix(iris[:, 1:4])
y = iris.Species

X_train, y_train, X_test, y_test, classes = prepare_data(X', y; dims=2)

#+

Random.seed!(666)
aux1 = prepare_data(X, y; dims=1)

#+

Random.seed!(666)
aux2 = prepare_data(X', y; dims=2)

#+

norm(aux1[1] - aux2[1]')

# ## Create the network

struct SimpleNet{T<:Real}
    W1::Matrix{T}
    b1::Vector{T}
    W2::Matrix{T}
    b2::Vector{T}
end

# ### Exercise:
# Write an outer constructor for `SimpleNet`. Its inputs should be three integers
# representing the input size of the three layers. All matrices should be initialized based
# on the normal distribution.
#
# **Hint**: think about the representation of the dense layer.
#
# ---
# ### Solution:

SimpleNet(n1, n2, n3) = SimpleNet(randn(n2, n1), randn(n2), randn(n3, n2), randn(n3))

# ---

Random.seed!(666)
m = SimpleNet(size(X_train,1), 5, size(y_train,1))

# ### Exercise:
# Write a functor `function (m::SimpleNet)(x)` which computes the prediction (forward pass)
# of the neural network `SimpleNet`.
#
# **Bonus**: try to make the functor work for both vectors (one sample) and matrices
# (multiple samples) `x`.
#
# ---
# ### Solution:

function (m::SimpleNet)(x)
    z1 = m.W1*x .+ m.b1
    a1 = max.(z1, 0)
    z2 = m.W2*a1 .+ m.b2
    return exp.(z2) ./ sum(exp.(z2), dims=1)
end

# ---

m(X_train[:,1:2])

# ## Train the network

function grad(m::SimpleNet, x::AbstractVector, y; ϵ=1e-10)
    z1 = m.W1*x .+ m.b1
    a1 = max.(z1, 0)
    z2 = m.W2*a1 .+ m.b2
    a2 = exp.(z2) ./ sum(exp.(z2), dims=1)
    l = -sum(y .* log.(a2 .+ ϵ))

    e_z2 = exp.(z2)
    l_part = (- e_z2 * e_z2' + Diagonal(e_z2 .* sum(e_z2))) / sum(e_z2)^2

    l_a2 = - y ./ (a2 .+ ϵ)
    l_z2 = l_part * l_a2
    l_a1 = m.W2' * l_z2
    l_z1 = l_a1 .* (a1 .> 0)
    l_x = m.W1' * l_z1

    l_W2 = l_z2 * a1'
    l_b2 = l_z2
    l_W1 = l_z1 * x'
    l_b1 = l_z1

    return l, l_W1, l_b1, l_W2, l_b2
end

#+

g_all = [grad(m, X_train[:,k], y_train[:,k]) for k in 1:size(X_train,2)]
typeof(g_all)

#+

function mean_tuple(d::AbstractArray{<:Tuple})
    Tuple([mean([d[k][i] for k in 1:length(d)]) for i in 1:length(d[1])])
end

#+

g_mean = mean_tuple(g_all)
typeof(g_mean)

# ### Exercise:
# Train the network with a gradient descent with stepsize ``\alpha=0.1`` for ``200``
# iterations. Save the objective value at each iteration and plot the results.
#
# ---
# ### Solution:

α = 1e-1
max_iter = 200
L = zeros(max_iter)

for iter in 1:max_iter
    grad_all = [grad(m, X_train[:,k], y_train[:,k]) for k in 1:size(X_train,2)]
    grad_mean = mean_tuple(grad_all)

    L[iter] = grad_mean[1]

    m.W1 .-= α*grad_mean[2]
    m.b1 .-= α*grad_mean[3]
    m.W2 .-= α*grad_mean[4]
    m.b2 .-= α*grad_mean[5]
end

# ---

using Plots

plot(L;
    xlabel="Iteration",
    ylabel="Loss function",
    label="",
    title="Loss function on the training set"
)

# ## Prediction
# ### Exercise:
#
# Write a function which predict the labels for samples. Show the accuracy on both training # and testing sets.
#
# ---
# ### Solution:

predict(X) = m(X)
accuracy(X, y) = mean(onecold(predict(X), classes) .== onecold(y, classes))

println("Train accuracy = ", accuracy(X_train, y_train))
println("Test accuracy = ", accuracy(X_test, y_test))


# NOTE: we now have an AbstractODEProblem
abstract type AbstractODEProblem end

struct ODEProblem{F,T<:Tuple{Number,Number},U<:AbstractVector,P<:AbstractVector} <: AbstractODEProblem
    f::F
    tspan::T
    u0::U
    θ::P
end

abstract type ODESolver end

struct Euler{T} <: ODESolver
    dt::T
end

function (solver::Euler)(prob::AbstractODEProblem, u, t)
    f, θ, dt  = prob.f, prob.θ, solver.dt
    (u + dt*f(u,θ), t+dt)
end
struct RK2{T} <: ODESolver
    dt::T
end
function (solver::RK2)(prob::AbstractODEProblem, u, t)
    f, θ, dt  = prob.f, prob.θ, solver.dt
    du = f(u,θ)
    uh = u + du*dt
    u + dt/2*(du + f(uh,θ)), t+dt
end

function solve(prob::AbstractODEProblem, solver::ODESolver)
    t = prob.tspan[1]; u = prob.u0
    us = [u]; ts = [t]
    while t < prob.tspan[2]
        (u,t) = solver(prob, u, t)
        push!(us,u)
        push!(ts,t)
    end
    # NOTE: we are not `reduce(hcat,us)`ing any more
    ts, us
end


"""
polyinterp(t,y)

Return a callable polynomial interpolant through the points in
vectors `t`,`y`. Uses the barycentric interpolation formula.
"""
function polyinterp(t,y)
    n = length(t)-1
    C = (t[end]-t[1]) / 4       # scaling factor to ensure stability
    tc = t/C

    # Adding one node at a time, compute inverses of the weights.
    omega = ones(n+1)
    for m = 1:n
        d = tc[1:m] .- tc[m+1]      # vector of node differences
        omega[1:m] = omega[1:m].*d  # update previous
        omega[m+1] = prod( -d )     # compute the new one
    end
    w = 1 ./ omega                  # go from inverses to weights

    p = function (x)
        # Compute interpolant.
        terms = @. w / (x - t)
        if any(isinf.(terms))       # divided by zero here
            # Apply L'Hopital's Rule exactly.
            idx = findfirst(x.==t)
            f = y[idx]
        else
            f = sum(y.*terms) / sum(terms)
        end
    end
    return p
end

"""
triginterp(t,y)

Return trigonometric interpolant for points defined by vectors `t`
and `y`.
"""
function triginterp(t,y)
    N = length(t)

    function trigcardinal(x)
        if isodd(N)      # odd
            tau = sin(N*pi*x/2) / (N*sin(pi*x/2))
        else             # even
            tau = sin(N*pi*x/2) / (N*tan(pi*x/2))
        end
        if isnan(tau)
            tau = 1
        end
        return tau
    end

    p = function (x)
        sum( y[k]*trigcardinal(x-t[k]) for k=1:N )
    end
    return p
end

"""
ccint(f,n)

Perform Clenshaw-Curtis integration for the function `f` on `n`+1
nodes in [-1,1]. Return integral and a vector of the nodes used.
Note: `n` must be even.
"""
function ccint(f,n)
    # Find Chebyshev extreme nodes.
    theta = [ i*pi/n for i=0:n ]
    x = -cos.(theta)

    # Compute the C-C weights.
    c = zeros(n+1)
    c[[1,n+1]] .= 1/(n^2-1)
    v = 1 .- 2*sum( cos.(2*k*theta[2:n])/(4*k^2-1) for k=1:n/2-1 )
    v -= cos.(n*theta[2:n])/(n^2-1)
    c[2:n] = 2*v/n

    # Evaluate integrand and integral.
    I = dot(c,f.(x))   # use vector inner product
    return I,x
end

"""
glint(f,n)

Perform Gauss-Legendre integration for the function `f` on `n` nodes
in (-1,1). Return integral and a vector of the nodes used.
"""
function glint(f,n)
    # Nodes and weights are found via a tridiagonal eigenvalue problem.
    beta = @. 0.5/sqrt(1-(2*(1:n-1))^(-2))
    T = diagm(-1=>beta,1=>beta)
    lambda,V = eigen(T)
    p = sortperm(lambda)
    x = lambda[p]           # nodes
    c = @. 2*V[1,p]^2       # weights

    # Evaluate the integrand and compute the integral.
    I = dot(c,f.(x))      # vector inner product
    return I,x
end

"""
intde(f,h,M)

Perform doubly-exponential integration of function `f` over
(-Inf,Inf), using discretization size `h` and truncation point `M`.
Return integral and a vector of the nodes used.
"""
function intde(f,h,M)
    # Find where to truncate the trapezoid sum.
    K = ceil( log(4/pi*log(2*M))/h )

    # Integrate by trapezoids in a transformed variable t.
    t = h*(-K:K)
    x = @. sinh(pi/2*sinh(t))
    dxdt = @. pi/2*cosh(t)*cosh(pi/2*sinh(t))

    I = h*dot(f.(x),dxdt)
    return I,x
end

"""
intsing(f,h,delta)

Integrate function `f` (possibly singular at 1 and -1) over
[-1+`delta`,1-`delta`] using discretization size `h`. Return
integral and a vector of the nodes used.
"""
function intsing(f,h,delta)
    # Find where to truncate the trapezoid sum.
    K = ceil(log(-2/pi*log(delta/2))/h)

    # Integrate over a transformed variable.
    t = h*(-K:K)
    x = @. tanh(pi/2*sinh(t))
    dxdt = @. pi/2*cosh(t) / (cosh(pi/2*sinh(t))^2)

    I = h*dot(f.(x),dxdt)
    return I,x
end


"""
**Nadam constructor (Nesterov Adam)**

```julia
     Nadam(;η::Real=0.001, β₁::Real=0.9, β₂::Real=0.999, ϵ::Real=10e-8)
```

Algorithm :

```math
\\begin{align*}
m_t =& \\beta_1 m_{t-1} + (1-\\beta_1)g_t\\\\
v_t =& \\beta_2 v_{t-1} + (1-\\beta_2)g_t^2\\\\
\\hat{m}_t =& \\frac{m_t}{1-\\beta_1^t}\\\\
\\hat{v}_t =& \\frac{v_t}{1-\\beta_2^t}\\\\
\\Delta x_t =&\\frac{\\eta}{\\sqrt{\\hat{v}_t+\\epsilon}}(\\beta_1\\hat{m}_t+\\frac{(1-\\beta_1)g_t}{1-\\beta_1^t})
\\end{align*}
```

[Algorithm Reference](http://cs229.stanford.edu/proj2015/054_report.pdf) and [other reference](http://www.cs.toronto.edu/~fritz/absps/momentum.pdf)
"""
mutable struct Nadam <: Optimizer
    opt_type::String
    t::Int64
    ϵ::Float64
    η::Float64
    β₁::Float64
    β₂::Float64
    m_t::AbstractArray
    v_t::AbstractArray
end

"Construct Nadam optimizer"
function Nadam(;η::Real=0.001, β₁::Real=0.9, β₂::Real=0.999, ϵ::Real=10e-8)
    @assert η > 0.0 "η must be greater than 0"
    @assert β₁ > 0.0 "β₁ must be greater than 0"
    @assert β₂ > 0.0 "β₂ must be greater than 0"
    @assert ϵ > 0.0 "ϵ must be greater than 0"

    Nadam("Nadam", 0, ϵ, η, β₁, β₂, [], [])
end

params(opt::Nadam) = "ϵ=$(opt.ϵ), η=$(opt.η), β₁=$(opt.β₁), β₂=$(opt.β₂)"

function update(opt::Nadam, g_t::AbstractArray{T}) where {T<:Real}
    # resize biased moment estimates if first iteration
    if opt.t == 0
        opt.m_t = zero(g_t)
        opt.v_t = zero(g_t)
    end

    # update timestep
    opt.t += 1

    # update biased first moment estimate
    opt.m_t = opt.β₁ * opt.m_t + (one(T) - opt.β₁) * g_t

    # update biased second raw moment estimate
    opt.v_t = opt.β₂ * opt.v_t + (one(T) - opt.β₂) * ((g_t) .^2)

    # compute bias corrected first moment estimate
    m̂_t = opt.m_t / (one(T) - opt.β₁^opt.t)

    # compute bias corrected second raw moment estimate
    v̂_t = opt.v_t / (one(T) - opt.β₂^opt.t)

    # apply update
    ρ = opt.η ./ (sqrt.(v̂_t .+ opt.ϵ))
    ρ .*= (opt.β₁ * m̂_t + (one(T) - opt.β₁) * g_t / (one(T) - opt.β₁^opt.t))

    return ρ
end


# catalan
@test catalannum(5) == 42
@test catalannum(30) == parse(BigInt,"3814986502092304")
@test_throws DomainError catalannum(-1)

# fibonacci
@test fibonaccinum(5) == 5
@test fibonaccinum(101) == parse(BigInt,"573147844013817084101")
@test_throws DomainError fibonaccinum(-1)

# lassalle
@test lassallenum(14) == parse(BigInt,"270316008395632253340")

# legendresymbol
@test legendresymbol(1001,9907) == jacobisymbol(1001,9907) == -1

# lucas
@test lucasnum(10) == 123
@test lucasnum(100) == parse(BigInt,"792070839848372253127")
@test_throws DomainError lucasnum(-1)

# stirlings1
@test_throws DomainError stirlings1(-1, 1)
@test typeof(stirlings1(6, 2)) == Int
@test stirlings1(0, 0) == 1
@test stirlings1(1, 1) == 1
@test stirlings1(2, 6) == 0
@test stirlings1(6, 0) == 0
@test stirlings1(6, 0, true) == 0
@test stirlings1(6, 1) == 120
@test stirlings1(6, 1, true) == -120
@test stirlings1(6, 2) == 274
@test stirlings1(6, 2, true) == 274
@test stirlings1(6, 3) == 225
@test stirlings1(6, 3, true) == -225
@test stirlings1(6, 4) == 85
@test stirlings1(6, 4, true) == 85
@test stirlings1(6, 5) == 15
@test stirlings1(6, 5, true) == -15
@test stirlings1(6, 6) == 1
@test stirlings1(6, 6, true) == 1
@test sum([abs(stirlings1(8, i, true)) for i = 0:8]) == factorial(8)

# stirlings2
@test_throws DomainError stirlings2(-1, 1)
@test typeof(stirlings2(6, 2)) == Int
@test stirlings2(0, 0) == 1
@test stirlings2(1, 1) == 1
@test stirlings2(2, 6) == 0
@test stirlings2(6, 0) == 0
@test stirlings2(6, 1) == 1
@test stirlings2(6, 2) == 31
@test stirlings2(6, 3) == 90
@test stirlings2(6, 4) == 65
@test stirlings2(6, 5) == 15
@test stirlings2(6, 6) == 1

# bell
@test bellnum(7) == 877
@test bellnum(42) == parse(BigInt,"35742549198872617291353508656626642567")
@test_throws DomainError bellnum(-1)


"""
    Hm... Power method should work well for symmatric tridiagonal matrix that is
    always changing its size... In fact it does, let's try inverse iterations
    instead.
"""

"""
    Left and right boundary can actually be infinite.
    Because of left and right bound, we have to restrict it to type Abstract Floats so that it's
    comparable, which is going to be used a lot for search the eigen values of the matrix.
"""
function InversePowerIterSearch(
        A::AbstractMatrix{T},
        left_bound::T,
        right_bound::T
    ) where {T<: AbstractFloat}
    # Get the mid point.
    @assert size(A, 1) == size(A, 2) "It has to be square matrix."
    @assert left_bound != right_bound "The interval must be non-degenerate."
    if left_bound > right_bound
        left_bound, right_bound = right_bound, left_bound
    end
    if left_bound == -Inf && right_bound == Inf
        error("Left and right bound cannot both be Inf, then it's unbounded. ")
    end
    v = randn(T, size(A, 1))    # previous iterations
    u = similar(v)                 # current iterations
    Au = similar(v)
    t = similar(v)              # temp
    MaxItr = 100
    while MaxItr > 0
        # Choose the right point.
        if left_bound == -Inf || right_bound == Inf
            θ = left_bound == -Inf ? 2*left_bound : 2*right_bound
            if θ == -Inf || θ == Inf
                error("Inverse Iteration Igen Search failed, bound limit reached for half opened interval.  ")
            end
        else
            θ = (left_bound + right_bound)/2
        end
        println("θ: $θ ∈ [$left_bound, $right_bound]")
        # Inverse Power iterations.
        λ = θ
        while λ == θ || norm(Au - λ*u, 1) > 1e-10
            println(λ)
            t .= u
            u .= (A - λ*I)\v
            u ./= norm(u)
            if norm(u) == NaN
                u .= v
                println("coverted to $λ")
                break
            else
                v .= t
                Au = A*u
                λ = dot(u, Au)
            end
        end
        # Interval shrink.
        if λ > left_bound && λ < right_bound

            return λ, u
        else
            if λ <= left_bound
                left_bound = θ

            else
                right_bound = θ
            end
        end
        MaxItr -= 1
    end
    if MaxItr == 0
       error("Inverse Power iterations method reached maximal iterations without convergence. ")
    end
end

# Basic Tests and shit
using LinearAlgebra
A = Diagonal(sort(randn(10)))
display(A)
λ, v = InversePowerIterSearch(A, A[1, 1], A[3, 3])
print("λ = $λ")
# λ, v = InversePowerIterSearch(A, -Inf, A[2, 2])
# print("λ = $λ")


"""
    fakedata_gaussian(pts_per_clust::Int = 10, nclust::Int = 2, xmin = 0.0, xmax = nclust * 7.0, nx = round(Int, (xmax - xmin) * 10), σ = 1.0, amin = 1.0, amax = 2.0)

DOCSTRING

# Arguments:
- `pts_per_clust`: DESCRIPTION
- `nclust`: DESCRIPTION
- `xmin`: DESCRIPTION
- `xmax`: DESCRIPTION
- `nx`: DESCRIPTION
- `σ`: DESCRIPTION
- `amin`: DESCRIPTION
- `amax`: DESCRIPTION
"""
function fakedata_gaussian(
    pts_per_clust::Int = 10,
    nclust::Int        = 2,
    xmin               = 0.0,
    xmax               = nclust * 7.0,
    nx                 = round(Int, (xmax - xmin) * 10),
    σ                  = 1.0,
    amin               = 1.0,
    amax               = 2.0,
)
    x = LinRange(xmin, xmax, nx)
    μ = LinRange(xmin, xmax, nclust + 2)[2:end-1]
    a = LinRange(xmin, xmax, pts_per_clust + 2)[2:end-1]

    npts = pts_per_clust * nclust
    data = zeros(nx, npts)
    labels = zeros(Int, npts)

    i = 1
    for c = 1:nclust
        for n = 1:pts_per_clust
            data[:, i] = gauss_func.(a[n], μ[c], σ, x)
            labels[i] = c
            i += 1
        end
    end

    return data, labels
end

gauss_func(a, μ, σ, x) = a * exp((-(x - μ) .^ 2) / (2 * σ^2))


# A version of the conjugate gradient method for linesearch.

"""A version of the conjugate gradient method for linesearch.
CG(A, b, ϵa, ϵr, itmax; quad) solves the symmetric linear system 'A * x = b'
If quad = true, the values of the quadratic model are computed
A can be positive definite or not.
"""
function CGlin(A, b, ϵa::Float64=1e-8, ϵr::Float64=1e-6, itmax::Int=0, ε::Float64=1e-6; quad::Bool=false)
    n = size(b, 1) # size of the problem
    (size(A, 1) == n & size(A, 2) == n) || error("Inconsistent problem size")
    @info(loggerCGlin, @sprintf("CGlin: system of %d equations in %d variables", n, n))

    x = zeros(n) # initial estimation x = 0
    xNorm = 0.0
    xNorms = [xNorm] # Values of ‖x‖
    r = -b # initial residual r = Ax-b = -b
    d = b # first descent direction
    rNorm = norm(r, 2)
    ϵ = ϵa + ϵr * rNorm

    iter = 0
    itmax == 0 && (itmax = 2 * n)

    if quad
        q = 0.0
        qvalues = [q] # values of the quadratic model
        @info(loggerCGlin, @sprintf("%5s %10s %10s\n", "Iter", "‖r‖", "q"))
        @info(loggerCGlin, @sprintf("    %d    %8.1e    %8.1e", iter, rNorm, q))
    end

    solved = rNorm ≤ ϵ
    tired = iter ≥ itmax

    while ! (solved || tired)
        iter += 1
        Ad = A * d
        dAd = dot(d, Ad)

        # if the model is not convexe, the algorithm stops
        if dAd ≤ ε * dot(d, d)
            @debug(loggerCGlin, @sprintf("non positive curvature dAd = %8.1e", dAd))
            iter == 1 && return b
            return x
        end

        α = rNorm^2 / dAd # step for x estimation
        x = x + α * d # new estimation
        xNorm = norm(x, 2)
        push!(xNorms, xNorm)
        roldNorm = rNorm
        r = r + α * Ad # new residual
        rNorm = norm(r, 2)

        if quad
            q = -dot(b, x) + 0.5 * dot(x, A * x)
            push!(qvalues, q)
            @info(loggerCGlin, @sprintf("    %d    %8.1e    %8.1e", iter, rNorm, q))
        end

        solved = rNorm ≤ ϵ
        tired = iter ≥ itmax

        (solved || tired) && continue
        β = rNorm^2 / roldNorm^2 # step for the next descent direction
        d = -r + β * d # new descent direction

        end
        return x
    end


@doc doc"""
Approximation de la solution du sous-problème ``q_k(s) = s^{t}g + (1/2)s^{t}Hs``
        avec ``s=-t g_k,t > 0,||s||< \delta_k ``

# Syntaxe
```julia
s1, e1 = Pas_De_Cauchy(gradient,Hessienne,delta)
```

# Entrées
 * **gradfk** : (Array{Float,1}) le gradient de la fonction f appliqué au point ``x_k``
 * **hessfk** : (Array{Float,2}) la Hessienne de la fonction f appliqué au point ``x_k``
 * **delta**  : (Float) le rayon de la région de confiance

# Sorties
 * **s** : (Array{Float,1}) une approximation de la  solution du sous-problème
 * **e** : (Integer) indice indiquant l'état de sortie:
        si g != 0
            si on ne sature pas la boule
              e <- 1
            sinon
              e <- -1
        sinon
            e <- 0

# Exemple d'appel
```julia
g1 = [0; 0]
H1 = [7 0 ; 0 2]
delta1 = 1
s1, e1 = Pas_De_Cauchy(g1,H1,delta1)
```
"""
function Pas_De_Cauchy(g,H,delta)

  a = (1/2)*g'*H*g
  b = norm(g)

  if (b > 0)
    t = (b^2)/(2*a)
    s = -t*g
    e = 1
    if norm(s)>delta
        t = delta/b
        s = -t*g
        e = -1
    end
  else
    t = 0
    s = -t*g
    e = 0
  end

  return s, e
end

2A/S7/UE Optimisation et RO/Optimisation/Optinum/src/Pas_De_Cauchy.jl


## Discretize given percentiles (unweighted)
function discretize_given_percentiles(inV :: Vector{Float64}, pctV :: Vector{Float64}, dbg :: Bool = false)
    edgeV = bin_edges_from_percentiles(inV, pctV, dbg);
    return discretize(inV, edgeV, dbg);
end

## Discretize given percentiles (weighted)
# function discretize_given_percentiles(inV :: Vector{Float64}, wtV :: AbstractWeights,
#    pctV :: Vector{Float64}, dbg :: Bool = false)

#    edgeV = bin_edges_from_percentiles(inV, wtV, pctV, dbg);
#    return discretize(inV, edgeV, dbg);
# end

## Discretize given bounds
# Equivalent to Discretizers LinearDiscretizer
function discretize(inV :: Vector{Float64}, edgeV :: Vector{Float64}, dbg :: Bool = false)
    n = length(edgeV);
    outV = zeros(Int64, size(inV));
    for i1 = n : -1 : 2
        outV[inV .<= edgeV[i1]] .= i1 - 1;
    end
    outV[inV .<= edgeV[1]] .= 0;
    return outV
end

"""
Bin edges from percentiles (unweighted)
Lowest bin includes minimum of inV
"""
function bin_edges_from_percentiles(inV :: Vector{Float64}, pctV :: Vector{Float64}, dbg :: Bool = false)
    edgeV = quantile(inV, [0.0; pctV]);
    # Ensure that lowest point is inside edges
    edgeV[1] -= 1e-8;
    if dbg
        @assert (minimum(inV) >= edgeV[1])
    end
    return edgeV
end

# ------------


 function j(n,k)
    p, i, seq=[0:n-1], 0, Int[]
    while !isempty(p)
        i=(i+k-1)%length(p)
        push!(seq,splice!(p,i+1))
    end
    @sprintf("Prisoner killing order: %s.\nSurvivor: %i",replace(chomp(string(seq[1:end-1])),"\n",", "),seq[end])
end


@doc raw"""
    rigid_body_velocity(vel_P, ω, r_PQ)

Calculate rigid solid velocity field.

Return velocity of a point Q of a rigid solid given the velocity of a point P (vel_P), the
rotational velocity of the solid (ω) and the relative position of Q with respect to P.

If the reference frame 1 is attached to the solid and the velocity is calculated with
respect to reference frame 0:

``v_{10}^{Q} = v_{10}^{P} + \omega_{10} \times r^{PQ}``

being:
- ``v_{10}^{Q}`` the velocity of point Q, fixed to 1, wrt 0
- ``\omega_{10}`` the angular velocity of the solid 1 wrt 0
- ``r^{PQ}`` the position of Q wrt P (``r^{Q}-r^{P}``)

Every vector needs to be expressed in the same coordinate system.

# References

1. Stevens, B. L., Lewis, F. L., (1992). Aircraft control and simulation: dynamics, controls
   design, and autonomous systems. John Wiley & Sons. (Section 1.3, page 26)
"""
function rigid_body_velocity(vel_P, ω, r_PQ)
    vel_Q = vel_P + ω × r_PQ
    return vel_Q
end

@doc raw"""
    rigid_body_acceleration(acc_P, ω, ω_dot, r_PQ)

Calcualte rigid body acceleration field.

Return the acceleration of a point Q of a rigid solid given the acceleration of a point P
(acc_P), the rotational velocity of the solid (ω), the rotational acceleration of the solid
(ω_dot) and the relative position of Q with respect to P.

``a_{10}^{Q} = a_{10}^{P} + \omega_{10} \times (\omega_{10} \times r^{PQ}) + \dot{\omega}_{10} \times r^{PQ}``

being:
- ``a_{10}^{Q}`` the acceleration of point Q, fixed to 1, wrt 0
- ``\omega_{10}`` the angular velocity of the solid 1 wrt 0
- ``\dot{\omega}_{10}`` the angular acceleration of the solid 1 wrt 0
- ``r^{PQ}`` the position of Q wrt P (``r^{Q}-r^{P}``)

# References

1. Stevens, B. L., Lewis, F. L., (1992). Aircraft control and simulation: dynamics, controls
   design, and autonomous systems. John Wiley & Sons. (Section 1.3, Formaula 1.3-14c, page 26)
"""
function rigid_body_acceleration(acc_P, ω, ω_dot, r_PQ)
    acc_Q = acc_P + ω × (ω × r_PQ) + ω_dot × r_PQ
    return acc_Q
end

"""
    pqr_2_ψθϕ_dot(p, q, r, ψ, θ, ϕ)

Transform body angular velocity (p, q, r) [rad/s] to Euler angles rates (ψ_dot, θ_dot,
ϕ_dot) [rad/s] given the euler angles (θ, ϕ) [rad] using kinematic angular equations.

# See also

[`ψθϕ_dot_2_pqr`](@ref), [`pqr_2_quat_dot`](@ref)

# References

1. Stevens, B. L., Lewis, F. L., & Johnson, E. N. (2015). Aircraft control and simulation:
   dynamics, controls design, and autonomous systems. John Wiley & Sons. Equation (1.4-4) (page 20)
"""
function pqr_2_ψθϕ_dot(p, q, r, θ, ϕ)

    sθ, cθ = sin(θ), cos(θ)
    sϕ, cϕ = sin(ϕ), cos(ϕ)

    ψ_dot = (q * sϕ + r * cϕ) / cθ
    θ_dot = q * cϕ - r * sϕ
    # ϕ_dot = p + (q * sϕ + r * cϕ) * tan(θ)
    ϕ_dot = p + ψ_dot * sθ

    return [ψ_dot, θ_dot, ϕ_dot]
end

"""
    ψθϕ_dot_2_pqr(ψ_dot, θ_dot, ϕ_dot, ψ, θ, ϕ)

Transform Euler angles rates (ψ_dot, θ_dot, ϕ_dot) [rad/s] to body angular velocity (p, q,
r) [rad/s] given the euler angles (θ, ϕ) [rad] using kinematic angular equations.

# See also

[`pqr_2_ψθϕ_dot`](@ref)

# References

1. Stevens, B. L., Lewis, F. L., & Johnson, E. N. (2015). Aircraft control and simulation:
   dynamics, controls design, and autonomous systems. John Wiley & Sons. Equation (1.4-3) (page 20)
"""
function ψθϕ_dot_2_pqr(ψ_dot, θ_dot, ϕ_dot, θ, ϕ)

    sθ, cθ = sin(θ), cos(θ)
    sϕ, cϕ = sin(ϕ), cos(ϕ)

    p = ϕ_dot - sθ * ψ_dot
    q = cϕ * θ_dot + sϕ*cθ * ψ_dot
    r = -sϕ * θ_dot + cϕ*cθ * ψ_dot

    return [p, q, r]
end

"""
    pqr_2_quat_dot(p, q, r, q0, q1, q2, q3)

Transform body angular velocity (p, q, r) [rad/s] to quaternion rates [1/s].

# See also

[`pqr_2_ψθϕ_dot`](@ref)

# References

1. Stevens, B. L., Lewis, F. L., & Johnson, E. N. (2015). Aircraft control and simulation:
   dynamics, controls design, and autonomous systems. John Wiley & Sons. Equation (1.8-15)
   (page 51).
"""
function pqr_2_quat_dot(p, q, r, q0, q1, q2, q3)

    Ω = [
        0 -p -q -r;
        p  0  r -q;
        q -r  0  p;
        r  q -p  0;
    ]
    q = [q0; q1; q2; q3]
    q_dot = 0.5 * Ω * q

    return q_dot
end

"""
    uvw_to_tasαβ(u, v, w)

Calculate true air speed (TAS), angle of attack (α) and angle of side slip (β) from velocity
expressed in body axis.

# Notes

This function assumes that u, v, w are the body components of the aerodynamic speed. This is
not true in genreal (wind speed different from zero), as u, v, w represent velocity with
respect to an inertial reference frame.

# References

1. Stevens, B. L., Lewis, F. L., & Johnson, E. N. (2015). Aircraft control and simulation:
   dynamics, controls design, and autonomous systems. John Wiley & Sons. Equation (2.3-6b)
   (page 78).
"""
function uvw_to_tasαβ(u, v, w)

    tas = sqrt(u*u + v*v + w*w)
    α = atan(w, u)
    β = asin(v / tas)
    return [tas, α, β]
end

"""
    uvw_dot_to_tasαβ_dot(u, v, w, u_dot, v_dot, w_dot)

Calculate time derivatives of velocity expressed as TAS, AOA, AOS.

# Notes
Note that tas here is not necessarily true air speed. Could also be inertial speed in the
direction of airspeed. It will concide whith TAS for no wind.

# See also

[`tasαβ_dot_to_uvw_dot`](@ref)

# References
1. Morelli, Eugene A., and Vladislav Klein. Aircraft system identification: Theory and
   practice. Williamsburg, VA: Sunflyte Enterprises, 2016. Equation 3.33 (page 44).
2. Stevens, B. L., Lewis, F. L., & Johnson, E. N. (2015). Aircraft control and simulation:
   dynamics, controls design, and autonomous systems. John Wiley & Sons. Equation (2.3-10)
   (page 81).
"""
function uvw_dot_to_tasαβ_dot(u, v, w, u_dot, v_dot, w_dot)

    # [1] 3.31
    tas = sqrt(u*u + v*v + w*w)
    # [1] 3.33a
    tas_dot = (u * u_dot + v * v_dot + w * w_dot) / tas
    # [1] 3.33b
    β_dot = ((u*u + w*w) * v_dot - v * (u * u_dot + w * w_dot)) / (tas^2 * sqrt(u*u + w*w))
    # [2] 2.3.10b
    # β_dot = (v_dot * tas - v * tas_dot) / (tas * sqrt(u*u + w*w))
    # [1] 3.33c
    α_dot = (w_dot * u - w * u_dot) / (u*u + w*w)

    return [tas_dot, α_dot, β_dot]
end

"""
    tasαβ_dot_to_uvw_dot(tas, α, β, tas_dot, α_dot, β_dot)

Obatain body velocity derivatives given velocity in wind axis and its derivatives.

# See also

[`uvw_dot_to_tasαβ_dot`](@ref)

# References

1. Morelli, Eugene A., and Vladislav Klein. Aircraft system identification: Theory and
   practice. Williamsburg, VA: Sunflyte Enterprises, 2016. Derived from equation 3.32
   (page 44).
"""
function tasαβ_dot_to_uvw_dot(tas, α, β, tas_dot, α_dot, β_dot)
    u_dot = tas_dot * cos(α) * cos(β) - tas * (α_dot * sin(α) * cos(β) + β_dot * cos(α) * sin(β))
    v_dot = tas_dot * sin(β) + tas * β_dot * cos(β)
    w_dot = tas_dot * sin(α) * cos(β) + tas * (α_dot * cos(α) * cos(β) - β_dot * sin(α) * sin(β))
    return [u_dot, v_dot, w_dot]
end

@doc raw"""
    rate_of_climb_constrain_no_wind(γ, α, β, ϕ)

Calculate pitch angle (θ rad) to obtain a flight path angle (γ rad) at certain angle of
attack (α rad), angle of sideslip (β rad) and roll (ϕ rad).

Inertial velocity and aerodynamic velocity are equivalent in the absence of wind:

`` v_{cg-e}^e = R_{eb} R_{bw} v_{cg-air}^w ``

# References

1. Stevens, B. L., Lewis, F. L., (1992). Aircraft control and simulation: dynamics, controls
   design, and autonomous systems. John Wiley & Sons. (Section 3.6, equation 3.6-3,
   page 187)
"""
function rate_of_climb_constrain_no_wind(γ, α, β, ϕ)
    a = cos(α) * cos(β)
    b = sin(ϕ) * sin(β) + cos(ϕ) * sin(α) * cos(β)
    sq = sqrt(a^2 - sin(γ)^2 + b^2)
    θ = (a * b + sin(γ) * sq) / (a^2 - sin(γ)^2)
    θ = atan(θ)
    return θ
end


using OrthoPoly
using Polynomials
using Test

function orthogonal_poly_test(F,Ps,num=[])
    if num == []
        num = ones(eltype(Ps[1]), length(Ps))
    end
    for n=0:length(Ps)-1
        f = F(n)
        p = Poly(Ps[n+1])/num[n+1]
        @test f == p
    end
end

println("Testing the first 11 Legendre polynomials")
orthogonal_poly_test(legendre, [[1],[0,1],[-1,0,3],[0,-3,0,5],[3,0,-30,0,35],[0,15,0,-70,0,63],[-5,0,105,0,-315,0,231],[0,-35,0,315,0,-693,0,429],[35,0,-1260,0,6930,0,-12012,0,6435],[0,315,0,-4620,0,18018,0,-25740,0,12155],[-63,0,3465,0,-30030,0,90090,0,-109395,0,46189]], [1 1 2 2 8 8 16 16 128 128 256])

println("Testing the first 16 Chebyshev polynomials of the first kind")
orthogonal_poly_test(chebyshevT, [[1],[0,1],[-1,0,2],[0,-3,0,4],[1,0,-8,0,8],[0,5,0,-20,0,16],[-1,0,18,0,-48,0,32],[0,-7,0,56,0,-112,0,64],[1,0,-32,0,160,0,-256,0,128],[0,9,0,-120,0,432,0,-576,0,256],[-1,0,50,0,-400,0,1120,0,-1280,0,512],[0,-11,0,220,0,-1232,0,2816,0,-2816,0,1024],[1,0,-72,0,840,0,-3584,0,6912,0,-6144,0,2048],[0,13,0,-364,0,2912,0,-9984,0,16640,0,-13312,0,4096],[-1,0,98,0,-1568,0,9408,0,-26880,0,39424,0,-28672,0,8192],[0,-15,0,560,0,-6048,0,28800,0,-70400,0,92160,0,-61440,0,16384]])

println("Testing the first 16 Chebyshev polynomials of the second kind")
orthogonal_poly_test(chebyshevU, [[1],[0,2],[-1,0,4],[0,-4,0,8],[1,0,-12,0,16],[0,6,0,-32,0,32],[-1,0,24,0,-80,0,64],[0,-8,0,80,0,-192,0,128],[1,0,-40,0,240,0,-448,0,256],[0,10,0,-160,0,672,0,-1024,0,512],[-1,0,60,0,-560,0,1792,0,-2304,0,1024],[0,-12,0,280,0,-1792,0,4608,0,-5120,0,2048],[1,0,-84,0,1120,0,-5376,0,11520,0,-11264,0,4096]])

println("Testing the first 11 Hermite polynomials")
orthogonal_poly_test(hermite_prob, [[1],[0,1],[-1,0,1],[0,-3,0,1],[3,0,-6,0,1],[0,15,0,-10,0,1],[-15,0,45,0,-15,0,1],[0,-105,0,105,0,-21,0,1],[105,0,-420,0,210,0,-28,0,1],[0,945,0,-1260,0,378,0,-36,0,1],[-945,0,4725,0,-3150,0,630,0,-45,0,1]])
orthogonal_poly_test(hermite, [[1],[0,2],[-2,0,4],[0,-12,0,8],[12,0,-48,0,16],[0,120,0,-160,0,32],[-120,0,720,0,-480,0,64],[0,-1680,0,3360,0,-1344,0,128],[1680,0,-13440,0,13440,0,-3584,0,256],[0,30240,0,-80640,0,48384,0,-9216,0,512],[-30240,0,302400,0,-403200,0,161280,0,-23040,0,1024]])

println("Testing the first 9 Laguerre polynomials")
orthogonal_poly_test(n -> laguerre(n, Poly([0,1//1])), [[1],[1,-1],[2,-4,1],[6,-18,9,-1],[24,-96,72,-16,1],[120,-600,600,-200,25,-1],[720,-4320,5400,-2400,450,-36,1],[5040,-35280,52920,-29400,7350,-882,49,-1],[40320,-322560,564480,-376320,117600,-18816,1568,-64,1]], [1 1 2 6 24 120 720 5040 40320]//1)

println("Testing the first 4 associated Laguerre polynomials")
for α=[1,1.0,2,2.0,2.1]
    orthogonal_poly_test(n -> laguerre_assoc(n,α,Poly([0,one(α)])),
                         [[1],[1+α,-1],[(α+2)*(α+1),-2(α+2),1],[(α+1)*(α+2)*(α+3),-3(α+2)*(α+3),3(α+3),-1]], [1.0 1 2 6])
end


using Roots
using Polynomials
using Base.Test
using Primes

## can use functions
f = x -> (x-1)*(x-2)^2*(x-3)^3
zs, mults = multroot(f)
@test mults == [1,2,3]

x = poly([0.0])

p = (x-1)*(x-2)^2*(x-3)^3
zs, mults = multroot(p)
@test mults == [1,2,3]

p = (x-1)^2*(x-2)^2*(x-3)^4
zs, mults = multroot(p)
@test mults == [2,2,4]

p = (x-1)^2
zs, mults = multroot(p^14)
@test mults == [28]

## test for roots of polynomial functions
roots(x -> x^5 - x + 1)

## test for real roots of polynomial functions
fzeros(x -> x^5 - 1.5x + 1)

## for polynomials in Z[x], Q[x] can use algorithm to be accurate for higher degree

@test fzeros(x -> x - 1)[1] == 1.0 # linear
f = x -> (x-1)*(x-2)*(x-3)^3*(x^2+1)
rts = fzeros(f)
rts = Float64[r for r in rts]
@test maximum(abs(sort(rts) - [1.0, 2.0, 3.0])) <= 1e-12
x = poly([big(0)])
p = prod([x - i for i in 1:20])
Roots.real_roots(p) ## can find this
f = x -> (x-20)^5 - (x-20) + 1
a = fzeros(f)[1]
@assert abs(f(a)) <= 1e-14

x = poly([0.0])
@test abs(fzeros((x-20)^5 - (x-20) + 1)[1] - (20 + fzeros(x^5 - x + 1)[1])) <= 1/2

fzeros(x -> x^5 - 2x^4 + x^3)

## factor
factor(x -> (x-2)^4*(x-3)^9)
factor(x -> (x-1)^3 * (x-2)^3 * (x^5 - x + 1))
factor(x -> x*(x-1)*(x-2)*(x^2 + x + 1))

factor(x -> (x-1)^2 * (x-.99)^2 * (x-1.01)^2) ## can have issue with nearby roots (or high powers)

factor(x -> (x-1//1)^2 * (x-99//100)^2 * (x-101//100)^2) ## conversion is to Float, not Rational{Int}
delta = 1//10
VERSION >= v"0.5.0-" && Primes.factor(convert(Poly{Rational{Int}}, x -> (x-1//1)^2 * (x-1 - delta)^2 * (x-1 + delta)^2))

## Test conversion of polynomials to Int
f = x -> 3x^3 - 2x
convert(Polynomials.Poly{Int}, f)
f = x -> (3x^3 - 2x)/3
convert(Polynomials.Poly{Int}, f)
f = x ->  x^2 / x # rational functions fails
@test_throws MethodError convert(Polynomials.Poly{Int64}, f)
f = x ->  (x^2 + x)^2
convert(Polynomials.Poly{Int}, f)
f = x ->  (x^2 + x)^(1/2)
@test_throws MethodError convert(Polynomials.Poly{Int64}, f)

## polynomial conversions from functions in practice
fzeros(x -> 265 - 0.65x)
fzeros(x -> -16x^2 + 200x)


using LinearAlgebra
function MSE(y, t; mean_f = true) # this is Mean Square Error
    if size(y) != size(t)
        y = y'
    end
    mse = sum((y - t) .^ 2)
    if mean_f
        mse /= length(y) #mse is Mean Square error
    end
    return mse
end
function CEE(p, t; mean_f = true, t_f = false) #this is Cross entropy Error
    function safe_log(x, miniv = 0.00000000001)
        return map(log, clamp.(x, miniv, Inf))
    end
    if t_f
        loss = -sum(t .* safe_log(p) + (1 .- t) .* safe_log(1 .- p))
    else
        loss = -sum(t .* safe_log(p))
    end
    if mean_f
        loss = loss / length(t)
    end
    return loss
end


using DelimitedFiles
using Test

"""
    get_seat_ids(data)

Get all seat ids. Map to binary strings, parse as Int-base2, add.
"""
function get_seat_ids(data)
    rowbin = Dict('F' => '0', 'B' => '1')
    colbin = Dict('L' => '0', 'R' => '1')
    rows = [parse(Int, join(rowbin[c] for c in row[1:7]), base=2) for row in data]
    cols = [parse(Int, join(colbin[c] for c in row[8:10]), base=2) for row in data]
    return [8*row + col for (row, col) in zip(rows, cols)]
    end

test = readdlm("./data/day05_test.txt")
@test all(get_seat_ids(test) .== [567, 119, 820])

data = readdlm("./data/day05.txt")
seats = get_seat_ids(data)
println("Part 1 max id: $(maximum(seats))")

"""
    get_my-seat(data)

Find missing id for which id-1 and id+1 are present.
Just going to brute force this, there aren't that many.
"""
function get_my_seat(seats)
    for s in 1:maximum(seats)
        if s ∉ seats && s-1 ∈ seats && s+1 ∈ seats
            return s
            end
        end
    end

println("Part 2 my seat: $(get_my_seat(seats))")


#=
  From https://math.stackexchange.com/questions/3633307/bayes-rule-broken
  """
  Bayes' Rule broken?!?!

  This question has been driving me CRAZY for 4 days now. The question comes from
  the textbook 'One Thousand Exercises in Probability', specifically Exercise 3 in
  section 1.4. The solution does not make sense! The question goes as follows:

     'A man possesses five coins, two double-headed, two normal and one double-tailed.
     The man shuts his eyes, picks a coin at random, and tosses the coin. He opens
     his eyes, sees a head: what is the probability the lower face is also a head?.

  The book gives an answer that is 2/3
  ....
  """

  Distributions of variable coin (num:0)
  1.00000 =>    6585  (0.658500)
  2.00000 =>    3415  (0.341500)

  Distributions of variable flip_is_head (num:0)
  1.00000 =>    6585  (0.658500)
  0.00000 =>    3415  (0.341500)

   Cf ~/blog/five_coins.blog
      ~/webppl/family_out_problem.wppl
=#

using Turing, StatsPlots, DataFrames
include("jl_utils.jl")

@model function five_coins()
    doubleHead = 1
    normalCoin = 2
    doubleTail = 3

    head = 1
    tail = 2

    coin ~ Categorical([2/5,2/5,1/5]) # [doubleHead,normalCoin,doubleTail]

    #  Toss the coin
    toss ~
        (coin == doubleHead) ? Dirac(head) :
        (coin == doubleTail) ? Dirac(tail) :
        Categorical([1/2,1/2]) # [head,tail]

    #  Flip the coin (i.e. turn it around)
    flip ~ Dirac((coin == doubleHead) ? head :
                 (coin == doubleTail) ? tail :
                 (toss == head) ? tail : head
                 )

    #  We observe that the toss give a head.
    true ~ Dirac(toss == head);

    #  What is the probability that the other side (flip side)
    #  is head?
    flip_is_head ~ Dirac(flip == head)
end

model = five_coins()

num_chains = 4

# chns = sample(model, Prior(), 10_000)
# chns = sample(model, MH(), 10_000)
chns = sample(model, PG(5), 10_000)
# chns = sample(model, SMC(), 10_000)
# chns = sample(model, IS(), 10_000)

display(chns)
# display(plot(chns))

show_var_dist_pct(chns,:coin)
show_var_dist_pct(chns,:flip_is_head)


using TuringModels

Turing.setadbackend(:reverse_diff);
Turing.turnprogress(false);

# In Rethinking the model actually has priors that are Uniform[-Inf, Inf], or as the Stan manual (2.17.0, pp. 127) tells us:

# "A parameter declared without constraints is thus given a uniform prior
# on (−∞, ∞) ..."

# Setting -Inf and Inf in Turing doesn't work. So the below works of
# course better since we've restrained it to [-1,1]

@model m8_2(y) = begin
    σ ~ Uniform(0, Inf)
    α ~ Uniform(-Inf, Inf)

    for i ∈ 1:length(y)
        y[i] ~ Normal(α, σ)
    end
end

y = [-1,1];

# Sample

chns = sample(m8_2(y), NUTS(0.65), 1000)

# Results rethinking

m82rethinking = "
       mean   sd  5.5% 94.5% n_eff Rhat
a      9.22 0.15  9.00  9.46   205    1
bR    -0.20 0.08 -0.34 -0.07   192    1
bA    -1.95 0.24 -2.36 -1.59   203    1
bAR    0.40 0.14  0.19  0.63   186    1
sigma  0.95 0.05  0.88  1.04   361    1
";

# Describe the posterior samples

describe(chns)

# End of `08/m8.2t.jl`


"""
    fixedStateCV(
        states :: Vector{<:AbstractMatrix},
        kraus_ops :: Vector{<:AbstractMatrix}
    ) :: Tuple{Float64, Vector{Matrix{ComplexF64}}}

For a fixed ensemble of `states` and quantum channel described by `kraus_ops`,
the communication value (CV) is evaluated by maximizing over all POVM measurements.
This optimization is expressed in primal form as the semidefinite program:

```math
\\begin{matrix}
    \\text{fixedStateCV}(\\mathcal{N})&= \\max_{\\{\\Pi_x\\}_{x}} \\sum_x \\text{Tr}\\left[\\Pi_x\\mathcal{N}(\\rho_x)\\right] \\\\
    & \\\\
    & \\text{s.t.} \\quad \\sum_x\\Pi_x = \\mathbb{I} \\;\\; \\text{and} \\;\\; \\Pi_x \\geq 0
\\end{matrix}
```

where each state ``\\rho_x`` satisfies ``\\text{Tr}[\\rho_x] = 1`` and ``\\rho_x \\geq 0``.
The channel ``\\mathcal{N}`` is applied to each state as
``\\mathcal{N}(\\rho_x) = \\sum_j k \\rho_x k^\\dagger``.

# Returns

A `Tuple`, `(cv, opt_povm)` where `cv` is the evaluated communication value and
`opt_povm` is the optimal POVM measurement.
"""
function fixedStateCV(
    states :: Vector{<:AbstractMatrix},
    kraus_ops :: Vector{<:AbstractMatrix}
) :: Tuple{Float64, Vector{Matrix{ComplexF64}}}
    d = size(kraus_ops[1],1)
    n = length(states)

    # add povm variables and constraints
    povm_vars = map(i -> HermitianSemidefinite(d), 1:n)
    constraints = sum(map(Πy -> real(Πy), povm_vars)) == Matrix{Float64}(I, d, d)
    constraints += sum(map(Πy -> imag(Πy), povm_vars)) == zeros(Float64, d, d)

    # apply channel to states
    evolved_states = map(ρ ->  sum(k -> k * ρ * k' , kraus_ops), states)

    # maximize CV over POVM measurements
    objective = maximize(real(sum(tr.(povm_vars .* evolved_states))), constraints)
    qsolve!(objective)

    cv = objective.optval
    opt_povm = map(Πy -> Πy.value, povm_vars)

    return cv, opt_povm
end

"""
    fixedMeasurementCV(
        povm :: Vector{<:AbstractMatrix},
        kraus_ops :: Vector{<:AbstractMatrix}
    ) :: Tuple{Float64, Vector{Matrix{ComplexF64}}}

For a fixed `povm` measurement and quantum channel described by `kraus_ops`, the
communication value (CV) and optimal state encodings are computed.
The fixed measurement CV is evaluated as

```math
\\text{fixedMeasurementCV}(\\mathcal{N}) = \\sum_y ||\\mathcal{N}^{\\dagger}(\\Pi_y)||_{\\infty}
```

where ``||\\mathcal{N}^{\\dagger}(\\Pi_y)||_{\\infty}`` is the largest eigenvalue
of the POVM element ``\\Pi_y`` evolved by the adjoint channel,
``\\mathcal{N}^{\\dagger}(\\Pi_y) = \\sum_j k^{\\dagger}_j \\Pi_y k_j``.
The states which maximize the CV are simply the eigenvectors corresponding to the
largest eigenvalue of each respective POVM element.

# Returns

A `Tuple`, `(cv, opt_states)` where `cv` is the communication value and
`opt_states` is the set of optimal states.
"""
function fixedMeasurementCV(
    povm :: Vector{<:AbstractMatrix},
    kraus_ops :: Vector{<:AbstractMatrix}
) :: Tuple{Float64, Vector{Matrix{ComplexF64}}}
    d = size(kraus_ops[1],2)
    n = length(povm)

    # apply adjoint channel to POVM
    evolved_povm = map(Π ->  sum(k -> k' * Π * k , kraus_ops), povm)

    opt_states = Vector{Matrix{ComplexF64}}(undef, n)
    cv = 0
    for i in 1:n
        povm_el = evolved_povm[i]
        vals, vecs = eigen(povm_el)

        opt_states[i] = vecs[:,end] * vecs[:,end]'
        cv += real(vals[end])
    end

    return cv, opt_states
end

"""
    seesawCV(
        init_states :: Vector{<:AbstractMatrix},
        kraus_ops :: Vector{<:AbstractMatrix},
        num_steps :: Int64;
        verbose :: Bool = false
    )

Performs the see-saw optimization technique to maximize the communication
value (CV) of the channel described by `kraus_ops` over all states and measurements.
This iterative and biconvex optimization technique combines coordinate ascent
maximization with semidefinite programming.
The number of iterations is determined by `num_steps` where each iteration
consists of a two-step procedure:

1. The POVM measurement is optimized with respect to a fixed state ensemble
   using the [`fixedStateCV`](@ref) function.
2. The state ensemble is optimized with respect to a fixed povm state using
   the [`fixedMeasurementCV`](@ref) function.

This procedure is initialized with `init_states` and after many iterations, a
local maximum of the CV is found.
The `verbose` keyword argument can be used to print out the CV evaluated in each
step.

The see-saw method has shown success in similar encoding/decoding
optimization problems in quantum information, *e.g.*,
[https://arxiv.org/abs/quant-ph/0307138v2](https://arxiv.org/abs/quant-ph/0307138v2)
and [https://arxiv.org/abs/quant-ph/0606078v1](https://arxiv.org/abs/quant-ph/0606078v1).
We note that our implementation is quite distinct from previous works, however,
the core iterative approach remains the same.

# Returns

A `Tuple` containing the following data in order:

1. `max_cv_tuple :: Tuple`, `(max_cv, opt_states, opt_povm)` A 3-tuple containing the maximal
   communication value and the optimal states/POVM that achieve this value.
2. `cvs :: Vector{Float64}`, A list of each evaluated CV. Since states and measurements
   are optimized in each iteration, we have `length(cvs) == 2 * num_steps`.
3. `opt_ensembles :: Vector{Vector{Matrix{ComplexF64}}}`, A list of state ensembles
   optimized in each step, where `length(opt_ensembles) == num_steps`.
4. `opt_povms :: Vector{Vector{Matrix{ComplexF64}}}`, A list of POVM measurements
   optimized in each step, where `length(opt_povms) == num_steps`.

!!! warning "Optimum Not Guaranteed"
    This function is not guaranteed to find a global or local optima. However,
    `seesawCV` will always provide a lower bound on the communication value.
"""
function seesawCV(
    init_states :: Vector{<:AbstractMatrix},
    kraus_ops :: Vector{<:AbstractMatrix},
    num_steps :: Int64;
    verbose :: Bool = false
) :: Tuple
    opt_ensembles = Vector{Vector{Matrix{ComplexF64}}}(undef, num_steps)
    opt_povms = Vector{Vector{Matrix{ComplexF64}}}(undef, num_steps)
    cvs = zeros(Float64, 2*num_steps)
    cv_id = 1

    max_cv_tuple = (1, [], [])

    for i in 1:num_steps
        # maximizing CV over POVMs
        opt_states = (i == 1) ? init_states : opt_ensembles[i-1]

        fixed_state_cv, opt_povm = fixedStateCV(opt_states, kraus_ops)

        cvs[cv_id] = fixed_state_cv
        cv_id += 1
        opt_povms[i] = opt_povm

        if fixed_state_cv > max_cv_tuple[1]
            max_cv_tuple = (fixed_state_cv, opt_states, opt_povm)
        end

        # maximizing CV over states
        fixed_povm_cv, opt_states = fixedMeasurementCV(opt_povm, kraus_ops)

        cvs[cv_id] = fixed_povm_cv
        cv_id += 1
        opt_ensembles[i] = opt_states

        if fixed_povm_cv > max_cv_tuple[1]
            max_cv_tuple = (fixed_povm_cv, opt_states, opt_povm)
        end

        if verbose
            println("i = ", i)
            println("fixed_state_cv = ", fixed_state_cv)
            println("fixed_povm_cv = ", fixed_povm_cv)
            println("max_cv = ", max_cv_tuple[1])
        end
    end

    return max_cv_tuple, cvs, opt_ensembles, opt_povms
end


#=
Section Allocation

   Suppose you have $n$ students in a class who need to be assigned to
      $m$ discussion sections. Each student needs to be assigned to exactly
      one section. Each discussion section should have between 6 and 10
      students. Suppose an $n \times m$ preference matrix $P$ is given,
      where $P_{ij}$ gives student $i$'s ranking for section $j$
      (1 would mean it is the student's top choice, 10,000 or a large number
      would mean the student can not attend that section).

   The goal will be to get an allocation matrix $X$, where $X_{ij} = 1$
      if student $i$ is assigned to section $j$ and $0$ otherwise.

  source: https://github.com/JuliaOpt/Convex.jl/blob/master/examples/
                             section_allocation/section_allocation.ipynb
=#

using Convex, GLPKMathProgInterface

include("data.jl"); # data.jl has our preference matrix, P

X = Variable(size(P), :Bin)

#=
   We want every student to be assigned to exactly one section.
   So, every row must have exactly one non-zero entry.
   In other words, the sum of all the columns for every row is 1.
   We also want each section to have between 6 and 10 students,
       so the sum of all the rows for every column should be between
       these constraints = [sum(X, 2) == 1, sum(X, 1) <= 10, sum(X, 1) >= 6]

   Our objective is simple sum(X .* P), which can be more efficiently
       represented as vec(X)' * vec(P).
   Since each entry of X is either 0 or 1, this is basically summing up
       the rankings of students that were assigned to them.
   If all students got their first choice, this value will be the number
       of students since the ranking of the first choice is 1.
=#

p = minimize(vec(X)' * vec(P), constraints)

solve!(p, GLPKSolverMIP())


####
#### Parametric gates
####

"""
    Rphi(ϕ)

The phase shift gate ``R_ϕ``.
This is equivalent to qiskit's `u1` gate.
"""
function Rphi(ϕ)
    return [1 0; 0 cis(ϕ)]
end

# Could use complex(reverse(sincospi(z))...)
"""
    _exp_ipi(z)

Return `exp(im * pi * z)`. This uses the accurate `cospi` and `sinpi` functions.
"""
_exp_ipi(z) = cispi(z)

"""
    Rphipi(z)

Return the ``R_ϕ`` gate for ``ϕ = zπ``. This is more accurate
than `Rphi(pi * z)`.
"""
function Rphipi(z)
    return [1 0; 0 _exp_ipi(z)]
end

"""
    U2(ϕ, λ)

u2 gate. Need to put math definitions in here.
"""
function U2(ϕ, λ)
    return [1 -cis(λ); cis(ϕ) cis(ϕ + λ)] / sqrt(2)
end

# This does not appar any faster than computing each separately
# function RXalt(θ)
#     (s, c) = sincos(θ/2)
#     si = - im * s
#     return [c si; si c]
# end

function RX(θ)
    c = cos(θ/2)
    s = - im * sin(θ/2)
    return [c s; s c]
end

function RXpi(z)
    c = cospi(z/2)
    s = -im * sinpi(z/2)
    return [c s; s c]
end

function RY(θ)
    c = cos(θ/2)
    s = sin(θ/2)
    return [c -s; s c]
end

function RYpi(z)
    c = cospi(z/2)
    s = sinpi(z/2)
    return [c -s; s c]
end

function RZ(θ)
    return [cis(-θ/2) 0; 0 cis(θ/2)]
end

function RZpi(z)
    return [_exp_ipi(-z/2) 0; 0 _exp_ipi(z/2)]
end

"""
    Ualt(θ, ϕ, λ)

Matrix from SU(2).
Alternative parameterization.
"""
function Ualt(θ, ϕ, λ)
    c = cos(θ/2)
    s = sin(θ/2)
    fpl =  (ϕ + λ) / 2
    cfpl = cos(fpl)
    sfpl = sin(fpl)
    fml =  (ϕ - λ) / 2
    cfml = cos(fml)
    sfml = sin(fml)
    f00 = complex(cfpl, -sfpl) * c
    f01 = complex(-cfml, sfml) * s
    f10 = complex(cfml, sfml) * s
    f11 = complex(cfpl, sfpl) * c
    return [f00 f01; f10 f11]
end

"""
    U(θ, ϕ, λ)

Matrix from SU(2). This is
the same as qiskit's U or U3.
"""
function U(θ, ϕ, λ)
    c = cos(θ/2)
    s = sin(θ/2)
    fpl =  (ϕ + λ)
    cfpl = cos(fpl)
    sfpl = sin(fpl)
    f00 = c
    f01 = -s * complex(cos(λ), sin(λ))
    f10 = s * complex(cos(ϕ), sin(ϕ))
    f11 = c * complex(cfpl, sfpl)
    return [f00 f01; f10 f11]
end

"""
    Ualtpi(θ, ϕ, λ)

Matrix from SU(2), with `θ`, `ϕ`, and `λ` given as multiples of `π`.
This is more accurate than `Ualt`.
"""
function Ualtpi(θ, ϕ, λ)
    c = cospi(θ/2)
    s = sinpi(θ/2)
    fpl =  (ϕ + λ) / 2
    fml =  (ϕ - λ) / 2
    cfpl = cospi(fpl)
    sfpl = sinpi(fpl)
    cfml = cospi(fml)
    sfml = sinpi(fml)
    f00 = complex(cfpl, -sfpl) * c
    f01 = complex(-cfml, sfml) * s
    f10 = complex(cfml, sfml) * s
    f11 = complex(cfpl, sfpl) * c
    return [f00 f01; f10 f11]
end

@doc raw"""
    R(θ, ϕ)

The R gate.
```math
R(\theta, \phi) = e^{-i \frac{\theta}{2} (\cos{\phi} x + \sin{\phi} y)}
```
"""
function R(θ, ϕ)
    c = cos(θ/2)
    s = sin(θ/2)
    return [c -im*exp(-im*ϕ)*s;
            -im*exp(im*ϕ)*s c]
end

@doc raw"""
    Rpi(θ, ϕ)

The R gate with θ and ϕ reduced by π. This is may be more accurate,
for example with integral and half-integral multiples of π.
```math
Rpi(\theta, \phi) = e^{-i \frac{\pi\theta}{2} (\cos{\pi\phi} x + \sin{\pi\phi} y)}
```
"""
function Rpi(θ, ϕ)
    c = cospi(θ/2)
    s = sinpi(θ/2)
    return [c -im*cispi(-ϕ)*s;
            -im*cispi(ϕ)*s c]
end

"""
    RXXYY(θ)

The R_XXYY gate, `exp(-im θ / 2 (X⊗X + Y⊗Y))`.
"""
function RXXYY(θ)
    _RXXYY(θ, cos, sin)
end

"""
    RXXYYpi(θ)

The R_XXYY gate with `θ` scaled by `π`. `exp(-im * pi * θ / 2 * (X⊗X + Y⊗Y))`
"""
function RXXYYpi(θ)
    _RXXYY(θ, cospi, sinpi)
end

function _RXXYY(θ, cfunc, sfunc)
    II = I2 ⊗ I2 / 2
    XX = X ⊗ X / 2
    YY = Y ⊗ Y / 2
    ZZ = Z ⊗ Z / 2
    return II + ZZ + cfunc(θ) * (II - ZZ) -im * sfunc(θ) * (XX + YY)
end


# Copyright (c) 2022 Code Komali
#
# This software is released under the MIT License.
# https://opensource.org/licenses/MIT

using Test, Printf, Memoize

# A global variable just for simple benchmarking
called = 0

# Using the Memoize package
@memoize Dict function knapsack(values, weights, MaxWieght)
    global called += 1
    if MaxWieght < 0
        return -Inf
    elseif isempty(values)
        return 0
    else
        max(values[end] + knapsack(values[1:end-1], weights[1:end-1], MaxWieght - weights[end]),
            knapsack(values[1:end-1], weights[1:end-1], MaxWieght))
    end
end

# Non memoized bruteforce approach
function knapsack_bruteforce(values, weights, MaxWieght)
    global called += 1
    if MaxWieght < 0
        return -Inf
    elseif isempty(values)
        return 0
    else
        max(values[end] + knapsack_bruteforce(values[1:end-1], weights[1:end-1], MaxWieght - weights[end]),
            knapsack_bruteforce(values[1:end-1], weights[1:end-1], MaxWieght))
    end
end

function compute_knapsack(values, weights, MaxWieght; memoized = true)
    empty!(memoize_cache(knapsack))
    global called = 0
    result = memoized ? knapsack(values, weights, MaxWieght) : knapsack_bruteforce(values, weights, MaxWieght)
    #uncomment the below code for call count
    #@printf "\n Function called: %d \n" called
    return result
end

@testset "0-1 knapsack problem tests" begin
    @test compute_knapsack([20, 5, 10, 40, 15, 25], [1, 2, 3, 8, 7, 4], 10, memoized = false) == 60
    @test compute_knapsack([60, 100, 120], [10, 20, 30], 50, memoized = false) == 220
    @test compute_knapsack([20, 5, 10, 40, 15, 25], [1, 2, 3, 8, 7, 4], 10) == 60
    @test compute_knapsack([60, 100, 120], [10, 20, 30], 50) == 220
    @test compute_knapsack([92, 57, 49, 68, 60, 43, 67, 84, 87, 72],
        [23, 31, 29, 44, 53, 38, 63, 85, 89, 82], 165) == 309
end

@time compute_knapsack([92, 57, 49, 68, 60, 43, 67, 84, 87, 72],
    [23, 31, 29, 44, 53, 38, 63, 85, 89, 82], 165)
@time compute_knapsack([92, 57, 49, 68, 60, 43, 67, 84, 87, 72],
    [23, 31, 29, 44, 53, 38, 63, 85, 89, 82], 165, memoized = false)

#Note:
# The memoized version is being called less number of times as expected.
# But, it is taking more time to compute. Need to investigate more.


# # [Basic Epidemiology Models](@id epidemiology_example)
#
#md # [![](https://img.shields.io/badge/show-nbviewer-579ACA.svg)](@__NBVIEWER_ROOT_URL__/examples/epidemiology.ipynb)

using Petri
using LabelledArrays
using Plots
using DiffEqJump
using StochasticDiffEq
using OrdinaryDiffEq

# ### SIR Model
#
# The SIR model represents the epidemiological dynamics of an infectious disease
# that causes immunity in its victims. There are three *states:* `Suceptible
# ,Infected, Recovered`. These states interact through two *transitions*.
# Infection has the form `S+I -> 2I` where a susceptible person meets an
# infected person and results in two infected people. The second transition is
# recovery `I -> R` where an infected person recovers spontaneously.

S  = [:S,:I,:R]
Δ  = LVector(
       inf=(LVector(S=1, I=1), LVector(I=2)),
       rec=(LVector(I=1),      LVector(R=1)),
     )
sir = Petri.Model(S, Δ)

Graph(sir)

# Once a model is defined, we can define out initial parameters `u0`, a time
# span `tspan`, and the transition rates of the interactions `β`

u0 = LVector(S=990.0, I=10.0, R=0.0)
tspan = (0.0,40.0)
# add a dynamic transition rate for infection
# where the rate of infection decreases over time
# as is dependent on the current state of the system
β = LVector(inf=((u,t)->((3/sum(u))/(t+1))), rec=0.25);

# each transition rates can one of three options:
#
# - constant: `β = [.25]`
#   - where the rate is specified by a value of type `Number`
# - time dependent: `β = [t->((3/1000)/(t+1))]`
#   - where `t` is the current time step
# - state and time dependent: `β = [(u,t)->((3/sum(u))/(t+1))]`
#   - where `u` is the current state of `u0` and `t` is the current time step

# Petri.jl provides interfaces to StochasticDiffEq.jl, DiffEqJump.jl, and
# OrdinaryDiffEq.jl Here, we call the `JumpProblem` function that returns an
# DiffEqJump problem object that can be passed to the DiffEqJump solver which
# can then be plotted and visualized

prob = JumpProblem(sir, u0, tspan, β)
sol = DiffEqJump.solve(prob,SSAStepper())

plot(sol)

# Similarly, we can generated `SDEProblem` statements that can be used with
# StochasticDiffEq solvers

prob, cb = SDEProblem(sir, u0, tspan, β)
sol = StochasticDiffEq.solve(prob,LambaEM(),callback=cb)

plot(sol)

# Lastly, we can generated `ODEProblem` statements that can be used with
# OrdinOrdinaryDiffEq solvers

prob = ODEProblem(sir, u0, tspan, β)
sol = OrdinaryDiffEq.solve(prob,Tsit5(),reltol=1e-8,abstol=1e-8)

plot(sol)

# ### SEIR Model

S = [:S,:E,:I,:R]
Δ = LVector(
      exp=(LVector(S=1, I=1), LVector(I=1, E=1)),
      inf=(LVector(E=1),      LVector(I=1)),
      rec=(LVector(I=1),      LVector(R=1)),
    )
seir = Petri.Model(S, Δ)

Graph(seir)
#-
u0 = LVector(S=990.0, E=10.0, I=0.0, R=0.0)
tspan = (0.0,40.0)
β = LVector(exp=0.7/sum(u0), inf=0.5, rec=0.25)

prob, cb = SDEProblem(seir, u0, tspan, β)
sol = StochasticDiffEq.solve(prob,LambaEM(),callback=cb)

plot(sol)

# ### SEIRD Model

S = [:S,:E,:I,:R, :D]
Δ = LVector(
      exp=(LVector(S=1, I=1), LVector(I=1, E=1)),
      inf=(LVector(E=1),      LVector(I=1)),
      rec=(LVector(I=1),      LVector(R=1)),
      die=(LVector(I=1),      LVector(D=1)),
    )
seird = Petri.Model(S, Δ)

Graph(seird)
#-
u0 = LVector(S=990.0, E=10.0, I=0.0, R=0.0, D=0.0)
tspan = (0.0,40.0)
β = LVector(exp=0.9/sum(u0), inf=0.9, rec=0.25, die=0.03)

prob, cb = SDEProblem(seird, u0, tspan, β)
sol = StochasticDiffEq.solve(prob,LambaEM(),callback=cb)

plot(sol)


grid_size = 25
gamma_vals = linspace(0.05, 0.95, grid_size)
w_bar_vals = similar(gamma_vals)

mcm = McCallModel()

for (i, gamma) in enumerate(gamma_vals)
    mcm.gamma = gamma
    w_bar = compute_reservation_wage(mcm)
    w_bar_vals[i] = w_bar
end

plot(gamma_vals,
    w_bar_vals,
    lw=2,
    alpha=0.7,
    xlabel="job offer rate",
    ylabel="reservation wage",
    label=L"$\bar w$ as a function of $\gamma$")


module p3

# Return boundary conditions values and indices, ϕb and β, for problem 2,
# with N subdivisions
function boundaryconditions3(N::Integer)
    ϕb = zeros(N+1,N+1)                 # boundary conditions
    β  = zeros(Int,N+1,N+1)             # boundary condition locations

    # Conditions on the perimeter
    ϕb[:,1] = ϕb[:,N+1] = ϕb[1,:] = ϕb[N+1,:] = 0.0
    β[:,1]  = β[:,N+1]  = β[1,:]  = β[N+1,:]  = 1

    # Conditions on the interior
    # xmin < x < xmax  →  N*xmin + 1 < i < N*xmax + 1
    xmin = 0.5
    xmax = 0.75
    ymin = 0.625
    ymax = 0.875

    xs = int([ceil(N*xmin+1):floor(N*xmax+1)])
    ys = int([ceil(N*ymin+1):floor(N*ymax+1)])

    ϕb[xs,ys] = 0.0
    β[xs,ys]  = 1

    return ϕb, β
end

# Calculate the potential
function V(x::Real, y::Real, N::Integer, q::Real)

    a = b = 1:(N+1)
    a .-= (N*x + 0.5)           # avoid singularity
    b .-= (N*y + 0.5)
    a .*= a
    b .*= b
    return -N*q ./ √(a .+ b')

end

V(N::Integer, q::Real) = V(0.5, 0.25, N, q)

function minor(A::Matrix, β::Vector)

    size(A)[1] == size(A)[2] || error("Matrix must be square")

    N = sum(1 .- β)             # dimensions of new matrix

    B = Array(typeof(A[1]),N,N)

    ii = 1
    jj = 1

    for j in 1:size(A)[1]
        β[j] == 1 && continue
        for i in 1:size(A)[2]
            β[i] == 1 && continue
            B[ii,jj] = A[i,j]
            ii += 1
        end
        jj +=1
        ii = 1
    end

    return B
end

# repopulate a vector v with zeros for fixed parameters
function addwater(v::Vector, β::Vector)

    vf = Array(typeof(v[1]),length(β))
    ii = 1

    length(v) == sum(1-β) || error("v and β have different num free params")

    for i in 1:length(β)
        if β[i] == 1
            vf[i] = 0               # put zero for a fixed value
        elseif β[i] == 0
            vf[i] = v[ii]           # fill in the free value
            ii += 1                 # prep the next free value
        else
            error("β can only have 0 or 1 values")
        end
    end
    return vf

end

end


using LinearAlgebra

A = [1 2 3;
     1 2 4;
     0 2 1;]

println("rank=", rank(A))
println("nullity=", length(nullspace(A)))
println("dim(V)=", size(A)[2])


using Flux
using Statistics
using Plots
using DifferentialEquations

# Solve the ode given by $$ u' = \cos 2\pi t $$ by approximation of $u$ with a Neural Network
NNODE = Chain(x -> [x], # Take in a scalar and transform it into an array
              Dense(1,32,tanh),
              Dense(32,1),
              first) # Take first value, i.e. return a scalar
NNODE(1.0)

# construct function $g(t)$ to enforce initial condition $$ g(t) = t NNODE(t) + u(0)
u₀ = 1.0
g(t) = t*NNODE(t) + u₀

#Construct loss function to numericaly compute derivative and check against R.H.S.
ϵ = sqrt(eps(Float32))
loss() = mean(abs2(((g(t+ϵ)-g(t))/ϵ) - cos(2π*t)) for t in 0:1f-2:1f0)

opt = Flux.Descent(0.01)
data = Iterators.repeated((), 5000)
iter = 0
cb = function () #callback function to observe training
    global iter += 1
    if iter % 500 == 0
        display(loss())
    end
end
display(loss())
Flux.train!(loss, Flux.params(NNODE), data, opt; cb=cb)

# compare against true solution
t = 0:1e-3:1.0

plot(t, g.(t), label="NN")
plot!(t, u₀ .+ sin.(2π.*t)/2π, label="True Solution")


const AbstractVec = SVector{3, T} where T <: AbstractFloat
const Vec = AbstractVec{Float64}

function normalize(v::AbstractVec)::AbstractVec
    v / norm(v)
end

@inline function x(v::AbstractVec)::AbstractFloat
    v[1]
end

@inline function y(v::AbstractVec)::AbstractFloat
    v[2]
end

@inline function z(v::AbstractVec)::AbstractFloat
    v[3]
end

function random_on_unit_disk()::AbstractVec
    v::AbstractVec = Vec(randn(), randn(), 0.0)
    normalize(v)
end

function random_in_unit_disk()::AbstractVec
    v::AbstractVec = random_on_unit_disk()
    r::AbstractFloat = rand()
    sqrt(r) * v
end

function random_on_unit_sphere()::AbstractVec
    v::AbstractVec = Vec(randn(), randn(), randn())
    normalize(v)
end

function random_in_unit_sphere()::AbstractVec
    v::AbstractVec = random_on_unit_disk()
    r::AbstractFloat = rand()
    cbrt(r) * v
end

function reflect(v::AbstractVec, n::AbstractVec)::AbstractVec
    v::AbstractVec = normalize(v)
    v - 2dot(v, n)n
end

function refract(v::AbstractVec, n::AbstractVec, index::AbstractFloat)::Tuple{Bool, Union{Nothing, AbstractVec}}
    v::AbstractVec = normalize(v)
    dt::AbstractFloat = dot(v, n)
    discriminant::AbstractFloat = 1.0 - index^2 * (1.0 - dt^2)
    if discriminant > 0
        refracted::AbstractVec = index * (v - n * dt) - n * sqrt(discriminant)
        true, refracted
    end
    false, nothing
end


function isdivisiblebyall(n, tab)
	for v in tab
		if n % v != 0
			return false
		end
	end
	return true
end

import Base.factorize
function factorize(n,F)
	f = zeros(eltype(n), F)
	f[1] = 1
	i = n
	while i > 1
		if isprime(i) && n % i == 0
			f[i] += 1
			n = div(n,i)
			i = n
		else
			i -= 1
		end
	end
	return f
end

function main()
	N = 20
	f = zeros(eltype(N), N, N)
	for i = N:-1:1
		f[:,i] = factorize(i, N)
	end
	println(f)
	resf = maximum(f, 2)
	println(resf')
	n = 1
	for i = 1:N
		for j = 1:resf[i]
			n *= i
		end
	end
	println(n)
	assert(isdivisiblebyall(n, [1:N]))
end

main()


function nominal_diameter(method,P,Hn,Vdotn)
        if method == "Bier" # default
            D = 0.176*(P/Hn)^0.466
        elseif method == "Sarkaria"
            D = 0.71*P^0.43/Hn^0.65
        elseif method == "Warnick"
            D = 0.72*P^0.43/Hn^0.63
        elseif method == "Moffat"
            D = 0.52*P^0.43/Hn^0.6
        # methods not depending on efficiency
        elseif method == "USBR"
            D = 1.517*Vdotn^0.5/Hn^0.25
        elseif method == "Fahlbusch"
            D = 1.12*Vdotn^0.45/Hn^0.12
        end
    return D*1.5
end
# for darcys friction factor Sami and Jain
function f_Darcy(NRe, pipe_roughness,D)
    f = 1/(1.8*log10(6.9/NRe+(pipe_roughness/D/3.7)^1.1))^2
    return f
end
# for Reynolds number
function renoldsnum(rho,v,D,mu)
    return rho*v*D/mu
end
#


using OptimalTransport
using Distances

import LinearAlgebra
import Random
using Test

Random.seed!(1234)

@testset "Initial map" begin
    # example from "Computational Optimal Transport"
    P = zeros(3, 3)
    OptimalTransport.initialmap!(P, [0.2, 0.5, 0.3], [0.5, 0.1, 0.4])
    @test P == [0.2 0 0; 0.3 0.1 0.1; 0 0 0.3]
end

# example taken from https://www.me.utexas.edu/~jensen/models/network/net8.html
@testset "Simple problem" begin
    C = [3 1 1000; 4 2 4; 1000 3 3]
    P, f, g = earthmover([5, 7, 3], [7, 3, 5], C)

    @test vec(sum(P; dims=1)) == [7, 3, 5]
    @test vec(sum(P; dims=2)) == [5, 7, 3]
    @test LinearAlgebra.dot(C, P) == 46
end

@testset "Total variation" begin
    histogram1 = LinearAlgebra.normalize!(rand(20), 1)
    histogram2 = LinearAlgebra.normalize!(rand(20), 1)

    C = (!==).(1:20, (1:20)')
    P, f, g = earthmover(histogram1, histogram2, C)

    @test vec(sum(P; dims=1)) ≈ histogram2
    @test vec(sum(P; dims=2)) ≈ histogram1
    @test LinearAlgebra.dot(C, P) ≈ totalvariation(histogram1, histogram2)
end