### A Pluto.jl notebook ###
# v0.17.3

using Markdown
using InteractiveUtils

# This Pluto notebook uses @bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of @bind gives bound variables a default value (instead of an error).
macro bind(def, element)
    quote
        local iv = try Base.loaded_modules[Base.PkgId(Base.UUID("6e696c72-6542-2067-7265-42206c756150"), "AbstractPlutoDingetjes")].Bonds.initial_value catch; b -> missing; end
        local el = $(esc(element))
        global $(esc(def)) = Core.applicable(Base.get, el) ? Base.get(el) : iv(el)
        el
    end
end

# ╔═╡ 6c3082f0-6196-11ec-1d58-03f55d178603
begin
	using Pkg; Pkg.activate(@__DIR__); Pkg.instantiate()
	Pkg.precompile()

	using PlutoUI
	using LinearAlgebra
	using SparseArrays
	using Images
	using MAT
end;

# ╔═╡ 6ff4cfec-ba3e-4937-bc3a-bf2d22f1a8f6
PlutoUI.TableOfContents(aside=true, indent=true, depth=3)

# ╔═╡ 63f2a85b-a426-4c37-a1a0-049d00cfed23
html"""

"""

# ╔═╡ a3ead5b3-0251-4054-b6f7-63bf51b5ff71
md"""
# 📐 Linear Algebra

A lot of the Data Science methods we will see in this tutorial require some understanding of Linear Algebra, and, in this notebook, we will focus on how Julia handles matrices, the types that exist, and how to call basic Linear Algebra tasks.

[YouTube link](https://www.youtube.com/watch?v=bndXPsRHPg0)

**Note:** matrices in Julia are stored column-wise. It's not actually a matrix (2D structure). In fact, it's an 1D structure.

```math
\begin{bmatrix}
	1   & 2   & 3   \\
	11  & 12  & 13  \\
	110 & 120 & 130 \\
\end{bmatrix}
```

is stored in memory as:

```math
\begin{bmatrix}
	1   \\
	11  \\
    110 \\
    2   \\
	12  \\
    120 \\
    3   \\
	13  \\
    130 \\
\end{bmatrix}
```

"""

# ╔═╡ 1fe44825-0933-4fb8-9b36-511f0e7ffeea
md"""
## Getting started

We will get started with creating a random (2D) matrix...
"""

# ╔═╡ cd961076-6baf-4626-9089-d33b9dbc94ce
A = rand(10, 10)

# ╔═╡ eaaffef2-903c-49cf-80b4-bb5adf7d8e92
md"""
We can transpose the matrix with the `'` operator...
"""

# ╔═╡ 92c1a395-bd96-477a-95dc-294a2ac99784
Aᵗ = A'

# ╔═╡ 5c852935-3055-4178-8ac3-3a6f1d6e8de9
md"""
The `transpose` function creates a matrix of type `Adjoint`. In Julia, it is a lazy adjoint. Often, we can easily perform Linear Algebra operations such as `A * A'` without actually transposing the matrix. This type is intended for Linear Algebra usage.
"""

# ╔═╡ de94c2dc-b4b0-42a2-aff0-e54543df82e1
typeof(Aᵗ)

# ╔═╡ 4770ca47-71ea-4fe1-a5ad-52c4e31f3bdd
"Adjoint = $(sizeof(Aᵗ)) bytes"

# ╔═╡ eccced3e-5898-4d93-8950-58aa77fc4c0b
"Matrix = $(sizeof(copy(Aᵗ))) bytes"

# ╔═╡ 02cbd31b-bd62-48f4-be5e-242279e215d9
md"""
**Note:** we can convert an `Adjoint` object into a `Matrix` by copying it with the `copy` function.

We can check if $A$ is positive definite with the `isposdef` function.
"""

# ╔═╡ 89fb6b4c-911b-4a39-bdef-b70dbe17f14f
isposdef(A)

# ╔═╡ 2082cb17-c373-459d-abc1-abe41cfa023a
md"""
We can do matrix multiplication with the `*` operator.
"""

# ╔═╡ efc88903-189b-48f4-8cde-17bb2c407010
A * Aᵗ

# ╔═╡ e6c8e66b-f9a3-4b93-b5a4-97c030d381d4
md"""
We basically can access matrix elements in two different ways. We can think about a matrix as an 1D column-wise structure and use just one index...
"""

# ╔═╡ bb678a13-b45d-410b-b8d5-6346eb839c56
A[11]

# ╔═╡ 41e22cf0-2448-4643-89ea-7315fb8b0b32
md"""
Or we can intuitively access matrix elements with a 2D indexing syntax...
"""

# ╔═╡ e759d9a1-29cb-4e15-b629-a913ce721b6f
A[1,2]

# ╔═╡ f662e8dd-d998-4e8c-8924-aee9fe633129
md"""
Both methods are equivalent, but the first one is faster...
"""

# ╔═╡ 39797209-2206-495f-b517-2ed219ff47ca
A[13] == A[3,2]

# ╔═╡ f9e97b10-4908-4811-a255-c76c0eaed01c
md"""
We can use the sliders below to access matrix `A` elements using the 2nd method...

Row $i$ $(@bind i Slider(1:10, show_value=true))

Column $j$ $(@bind j Slider(1:10, show_value=true))
"""

# ╔═╡ 157971c4-04d9-41de-b378-4a9e998725e9
A[i,j]

# ╔═╡ d82db838-5ba6-456d-bdeb-cb75a20e081c
md"""
Now, we can create a (1D) vector of random values...
"""

# ╔═╡ 5dc61a15-48e9-46a2-b893-9fb97ecc8b55
b = rand(10)

# ╔═╡ 0ec44c81-40d5-4b3f-bdec-d3e0cab7de74
md"""
Suppose that we have a linear system $Ax = b$ and we want the solution $x$. We can easily solve this system, using the `\` operator.

The `\` operator gives the multiplication of `b` by the inverse of `A` on the left. It gives floating-point results for integer arguments. It allows you to solve a system of linear equations, and often uses a suitable matrix factorization to solve the problem. We will cover factorizations next.

**Note:** `\` is always the recommended way to solve a linear system. You almost never want to call the `inv` function.
"""

# ╔═╡ 10474148-5a43-4cc1-8e9f-de57bd3019bc
x = A \ b

# ╔═╡ 859cb7ee-d9d8-4bd7-b7c9-5777867297b7
md"""
where `x` is the solution vector.

When there isn't any solution `x` for the system $Ax = b$, the `\` operator will return a vector whose the norm of $Ax-b$ is the minimum least square solution.

**Note:** `A` is a `Matrix` type, and `b` is a `Vector` type. A `Matrix` is a 2D `Array`, and a `Vector` is an 1D `Array`.
"""

# ╔═╡ 9f56f781-5abe-407c-b449-0d917dfa2767
Matrix{Float64} == Array{Float64, 2}

# ╔═╡ 32b59524-9b72-40a2-828f-96f20b0dd924
Vector{Float64} == Array{Float64, 1}

# ╔═╡ 4732c64c-c957-4bdb-9332-7ccb8f40f655
md"""
We can also calculate the norm of a structure with the `norm` function. Let's calculate the norm of $Ax-b$, which should be a number close to zero, since $Ax=b$...
"""

# ╔═╡ b4b29ad0-8573-40ea-a2d8-c25183825149
norm(A*x - b)

# ╔═╡ 8c19fd6c-2f71-480d-8fd0-2241b6f3debe
md"""
## Factorization

A common tool used in Linear Algebra is matrix factorizations. These factorizations are often used to solve linear systems like $Ax=b$, and as we will see later in this tutorial... $Ax=b$ comes up in a lot of Data Science problems.
"""

# ╔═╡ 11bc3a82-19d2-453f-b40b-1eb488ebb323
md"""
### LU factorization

The **Lower–Upper (LU) factorization** (decomposition) factors a matrix as the product of a _lower triangular matrix_ $L$ and an _upper triangular matrix_ $U$. The product sometimes includes a _permutation matrix_ $P$ as well.

In LU factorization (with partial pivoting), we know that $LU=PA$, where $L$ and $U$ are lower and upper matrices and $P$ and $A$ are permutation and original matrices.

In Julia, we can use the `lu` function to perform LU decomposition.
"""

# ╔═╡ 983aeaea-a556-49ba-aedf-898c1b71407d
# LU factorization
LUₐ = lu(A)

# ╔═╡ c8d51d21-ec99-4451-a87e-b51d8f118e24
md"""
We can access lower, upper and permutation matrices...
"""

# ╔═╡ b2b6c7d0-8abb-415e-b668-1fe089b42b04
# lower triangular matrix
Lₐ = LUₐ.L

# ╔═╡ 790d0305-216f-4335-ab37-acff2de0f18f
# upper triangular matrix
Uₐ = LUₐ.U

# ╔═╡ 8bb9722c-9cb6-4316-831b-26cb64f93619
# permutation matrix
Pₐ = LUₐ.P

# ╔═╡ e3571e13-1fa6-44d6-8a31-f810c11cd1f4
# check if LU = PA
Lₐ*Uₐ ≈ Pₐ*A

# ╔═╡ 311b85d4-6509-4945-a451-41020cac465e
# norm should be near zero
norm(Lₐ*Uₐ - Pₐ*A)

# ╔═╡ ea01ba90-0b30-4998-b1a5-e7fa17e53679
md"""
### QR factorization

The **QR factorization**, also known as a QR factorization or QU factorization, is a decomposition of a matrix $A$ into a product $A = QR$ of an _orthogonal matrix_ $Q$ and an _upper triangular matrix_ $R$.

QR decomposition is often used to solve the linear least squares problem and is the basis for a particular eigenvalue algorithm, the QR algorithm.

In Julia, we can use the `qr` function to perform QR decomposition.
"""

# ╔═╡ ada5e3a2-b801-4d2b-8ed8-0b9d8e0cf132
# QR factorization
QRₐ = qr(A)

# ╔═╡ 930c8f5d-a05b-4623-84c3-9b58fe59a721
md"""
We can access both orthogonal and upper triangular matrices...
"""

# ╔═╡ c03c19b1-ec14-4215-9b54-f6cc6b5a2e5c
# orthogonal matrix
Qₐ = QRₐ.Q

# ╔═╡ 7371a76e-ae89-4138-a614-fa41bfb2dfb0
# upper triangular matrix
Rₐ = QRₐ.R

# ╔═╡ da750f88-e883-4bec-8e2e-79a91da23f9b
# check if A = QR
A ≈ Qₐ*Rₐ

# ╔═╡ 2669897d-cd5a-45c6-afe0-7a91153e3686
# norm should be near zero
norm(A - Qₐ*Rₐ)

# ╔═╡ 737762ba-b142-4fd9-9862-af69f7415030
md"""
### Cholesky factorization

The **Cholesky factorization**,  or Cholesky decomposition, is a decomposition of a Hermitian, positive-definite matrix into the product of a _lower triangular matrix_ $L$ and its _conjugate transpose_ $L^t$. In other words, $A = LL^t$ which is useful for efficient numerical solutions.

Cholesky factorization is a type of LU factorization, where $L$ and $U$ matrices are the transposed matrices of each other. That's why we call them $L$ and $L^t$.

**Note:** The input matrix needs to be _symmetric positive definite_.
"""

# ╔═╡ b1ebbd9d-01d5-4eed-85c3-732491322582
# make a positive definite matrix
Aᵖ = A * Aᵗ

# ╔═╡ 6cea7788-57a1-426b-8ede-0a424970e2e3
# check if Aˢ is positive definite
isposdef(Aᵖ)

# ╔═╡ f7a4fffa-a633-45c9-98a5-b25a047a8a3f
# Cholesky factorization
choₐ = cholesky(Aᵖ)

# ╔═╡ c64cf987-9329-4139-b7e3-0fc21e6f0d92
md"""
We can access both lower triangular matrix and its conjugate transpose...
"""

# ╔═╡ aa195042-2f7e-4881-9d01-834085dccc11
# lower triangular matrix
L = choₐ.L

# ╔═╡ 503192fc-8384-4116-b389-38f04015953c
# lower triangular conjugate transpose
Lᵗ = choₐ.U

# ╔═╡ 61ea5d3c-63aa-480d-8252-0762f586232f
# check A = LLᵗ
Aᵖ ≈ L*Lᵗ

# ╔═╡ 49803952-b650-4e5b-9b65-a0d84b21134b
# norm should be near zero
norm(Aᵖ - L*Lᵗ)

# ╔═╡ 3f0a1c9f-8b8a-4886-97bd-3810dfa0263b
md"""
### The `factorize` function

The `factorize` function compute a _convenient factorization of_ $A$, based upon the type of the input matrix. factorize checks $A$ to see if it is symmetric/triangular/etc. If $A$ is passed as a generic matrix, `factorize` checks every element of $A$ to verify/rule out each property.

|     Properties of `A`    |	    Type of factorization       |
|:------------------------:|:----------------------------------:|
Positive-definite          | Cholesky (see `cholesky`)          |
Dense Symmetric/Hermitian  | Bunch-Kaufman (see `bunchkaufman`) |
Sparse Symmetric/Hermitian | LDLt (see `ldlt`)                  |
Triangular                 | Triangular                         |
Diagonal                   | Diagonal                           |
Bidiagonal                 | Bidiagonal                         |
Tridiagonal	               | LU (see `lu`)                      |
Symmetric real tridiagonal | LDLt (see `ldlt`)                  |
General square	           | LU (see `lu`)                      |
General non-square         | QR (see `qr`)                      |

**Example:** if `factorize` is called on a Hermitian positive-definite matrix, for instance, then `factorize` will return a Cholesky factorization.
"""

# ╔═╡ 433c016e-334a-43fa-b315-608b4bbdef51
factorize(A)

# ╔═╡ 1d9e27be-c21b-4b45-be25-dd1f5b954cb2
md"""
In this case, `factorize` performs the LU factorization, since `A` is a square matrix (no need to perform QR factorization), although it is not a positive definite matrix (cannot perform Cholesky factorization).
"""

# ╔═╡ d806f387-587a-438c-bb22-f6f14b4d4509
md"""
## Some LinearAlgebra.jl functions

With `LinearAlgebra.jl`, we can specify what kind of matrix we want to build. For example, we can use the `diagm` function to create a diagonal matrix whose main diagonal is a vector...
"""

# ╔═╡ 1755c27b-b170-4b5f-acf9-87e722fd6fad
D = diagm([1,2,3,4,5])

# ╔═╡ 10f4304f-06da-4f0a-bfa9-a148169b531f
typeof(D)

# ╔═╡ 926cb115-c47f-44e3-bb40-53533d0a1354
md"""
Although it is a diagonal matrix, it's a `Matrix`. To convert it into a `Diagonal`, we can do...
"""

# ╔═╡ d86cf3cc-0490-4144-b732-41e40f380813
Diagonal(D)

# ╔═╡ 7a7002b3-2523-4b33-8dde-511728dfaf0e
md"""
We can also build a identity matrix with the `I` function by just passing the number of elements...

**Note:** the matrix below is `Diagonal` type.
"""

# ╔═╡ 9980569d-ca11-432e-afb5-33001dea3746
I(3)

# ╔═╡ a31e0b0d-159f-43f3-aed8-a3b7a33bb73b
typeof(I(3))

# ╔═╡ 05bd36f7-b0db-4a58-a591-4cde55139ade
md"""
We can perform $A + I$ and Julia will figure it out the convenient size of $I$.
"""

# ╔═╡ fee37f40-b39d-4cfa-8f25-882a723d5874
A + I

# ╔═╡ 83ba1696-9345-49d8-993a-78517d4ffecb
A + 2I

# ╔═╡ 1d8970d9-7fae-4fef-b4e0-a92aa3e911a7
md"""
## Sparse Linear Algebra

Sparse matrices are stored in Compressed Sparse Column (CSC) form. This form is useful when we are dealing with matrices in which there are very few non-zero elements.

Here, we will use the `SparseArrays.jl` package.

We can create a sparse random matrix with the `sprand` function. The arguments are: number os rows, number of columns and the probability of non-zero values.
"""

# ╔═╡ 512da58a-f8d5-4e0c-8c12-de12cc2a4ba5
S = sprand(5, 8, 0.2)

# ╔═╡ 0293d061-05bc-484f-abcb-76dd4b54fd30
md"""
We can convert it into a regular `Matrix`...
"""

# ╔═╡ f051a1cc-cd3d-465d-9ec0-cb711ad691d7
M = Matrix(S)

# ╔═╡ ab47cafa-3d00-4e7a-8af9-103241fc68c1
md"""
Let's compare the memory allocation of both `SparseMatrixCSC` and `Matrix` types...
"""

# ╔═╡ 8ebe9e1d-981b-4f96-9598-91c8893b8818
"Regular matrix = $(sizeof(M)) bytes."

# ╔═╡ 223cca30-9c72-4ae3-aa51-85c54a362cf4
"Sparse matrix = $(sizeof(S)) bytes."

# ╔═╡ 640da366-e78b-4a62-aedc-2668ae69ae95
md"""
We can access the non-zero elements of the matrix, using the `nzval` attribute. This attribute returns a `Vector` with all non-zero elements.
"""

# ╔═╡ 3a5c505a-0620-4cf4-a546-0a01c191b512
S.nzval

# ╔═╡ 6b140d76-a5c3-4b86-9cbd-116a724851af
md"""
We can access the number of rows with the `m` attribute...
"""

# ╔═╡ 12bdbf11-c352-4243-b080-b28c34527e3b
"There are $(S.m) rows in the matrix."

# ╔═╡ 6d53f3e5-66d1-433f-ad4e-0800b0ecb70f
md"""
And we can also access the number of columns with the `n` attribute...
"""

# ╔═╡ 17764c04-a202-4563-98a4-eddb9f432e6d
"There are $(S.n) columns in the matrix."

# ╔═╡ a441d0b6-9a18-46e4-899d-f3ea2642c7b7
md"""
## Images as matrices

Let's get to the more _"data science-y"_ side. We will do so by working with images (which can be viewed as matrices), and we will use the `SVD` decomposition.

First let's load an image. I chose this image as it has a lot of details.
"""

# ╔═╡ 0f82de90-1a24-49a6-9673-3abf01283817
X₁ = load("data/khiam-small.jpg")

# ╔═╡ 4eb22f0f-b287-4be2-b013-824371a44c26
md"""
An image may be treated as a `Matrix` of `RGB` pixels as we can see below...
"""

# ╔═╡ 3dc31e0a-194c-4ed2-ae6f-c258170aba1a
typeof(X₁)

# ╔═╡ 83c6b744-750c-4c74-8b79-a3d78e5b2c12
md"""
Thus, we can access pixels by slicing an image...
"""

# ╔═╡ b61a2c57-6443-4e8a-b48a-28e2ad1fca4f
# the first pixel
X₁[1,1]

# ╔═╡ 7f003ee7-6958-48e1-a3d8-11876cfa1dbd
# a random chosen pixel
X₁[100,20]

# ╔═╡ c2d97352-fb67-4d66-a999-a8362e41ee84
# first row of pixels
X₁[1,:]

# ╔═╡ 45f5c9ac-d450-49d6-af9d-28704161f360
# first column of pixels
X₁[:,1]

# ╔═╡ 946bf2e5-51e4-4b8b-a316-27e29f15c2e3
md"""
We can easily convert the image to gray scale. This is useful when working with images, since we can deal with a single value instead of a RGB tuple.
"""

# ╔═╡ 69c75582-f4a1-443e-bda2-385681712821
Xgray = Gray.(X₁)

# ╔═╡ 42a1ee17-86ab-41d8-8b89-5064700e2e64
md"""
We can easily extract the RGB layers from the image. We will make use of the `reshape` function below to reshape a vector to a matrix...
"""

# ╔═╡ ff77b413-a905-44d6-973d-f7d1193dbb25
begin
	# red layer
	r_arr = map(i -> X₁[i].r, 1:length(X₁))
	r_mtx = Float64.(reshape(r_arr, size(X₁)))

	# red.(X₁)
end

# ╔═╡ f98fdf05-e6e8-4d0e-b6a6-d3398c2d1ab5
begin
	# green layer
	g_arr = map(i -> X₁[i].g, 1:length(X₁))
	g_mtx = Float64.(reshape(g_arr, size(X₁)))

	# green.(X₁)
end

# ╔═╡ 34b5d940-7d9a-40fb-a38f-16b267130026
begin
	b_arr = map(i -> X₁[i].b, 1:length(X₁))
	b_mtx = Float64.(reshape(b_arr, size(X₁)))

	# blue.(X₁)
end

# ╔═╡ 3b2724d7-f429-447d-948c-e11a8fc20572
md"""
We can now create a matrix of all zeros of equal size as the image and then plot each layer separately...
"""

# ╔═╡ c071dc2a-e53e-402b-b800-6601025b62c7
Z_mtx = zeros(size(r_mtx)...)

# ╔═╡ 5e54ac09-2276-4760-adc9-a05082a5c17f
# red layer
RGB.(r_mtx, Z_mtx, Z_mtx)

# ╔═╡ 33b3de91-010a-4bd2-863d-199bfbfb8ed8
# green layer
RGB.(Z_mtx, g_mtx, Z_mtx)

# ╔═╡ ac394a4c-f983-487c-9d90-1cada3b9e398
# blue layer
RGB.(Z_mtx, Z_mtx, b_mtx)

# ╔═╡ 59daefac-6e47-4ecc-9f4a-033faa55fc59
md"""
We can easily obtain the `Float64` values of the grayscale image.
"""

# ╔═╡ 9d1fa923-e20a-44cc-b0b6-0656f85178fc
Xgrayvalues = Float64.(Xgray)

# ╔═╡ a551a1cb-0d7d-4c82-8f7b-9d92274516f3
md"""
### SVD factorization

The **Singular Value Decomposition (SVD)** is a factorization of a real or complex matrix. It generalizes the eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any $n \times m$ matrix.

The SVD of an $n \times m$ complex matrix $A$ is a factorization of the form $A = U \Sigma V^t$, where $U$ is an $n \times m$ _complex unitary matrix_, $\Sigma$ is an $n \times m$ _rectangular diagonal matrix with singular values_ (non-negative real numbers), and $V$ is an $n \times n$ _complex unitary matrix_.

The SVD is an indicator of how much redundant information you have within your data. For example, if we have just a few non-zero values (i.e. low rank matrix), we may assume that we could store this data in much less memory.

**Note:** _low rank_ means that the matrix has very few linearly independent rows or columns.

We will downsample the grayscale values using the SVD. First, let's obtain the factorization using the function `svd`.
"""

# ╔═╡ d66ae839-2d03-40f6-a82e-1e8a063e27f0
SVD = svd(Xgrayvalues)

# ╔═╡ 54caef76-2f0d-4e9f-a3a1-725ab3d64648
md"""
We may access $U$, $\Sigma$ and $V$...
"""

# ╔═╡ a28364a5-5c03-4b42-b354-2cdb1880ca1d
# rectangular unitary matrix
U = SVD.U

# ╔═╡ 0a30e77d-8770-4b75-9c7d-e45f6d301022
# diagonal matrix with singular values
Σ = diagm(SVD.S)

# ╔═╡ ac8e9a63-708c-408f-bc91-a57356dae648
# transposed square unitary matrix
Vᵗ = SVD.V'

# ╔═╡ 96766fac-6950-4554-b89a-32143f120f04
# check if A = UΣVᵗ
Xgrayvalues ≈ U*Σ*Vᵗ

# ╔═╡ 1de26f9d-89eb-4397-adba-ed566cd054d6
# norm should be near zero
norm(Xgrayvalues - U*Σ*Vᵗ)

# ╔═╡ 90922832-4ed1-4fa6-bb61-1242c17e1cd8
md"""
According to the Eckart–Young theorem, we can get the closest rank estimation to the matrix $A$ by extracting the top $k$ singular values.

Let's try to extract the top 4 singular vectors/values to form a new matrix/image.
"""

# ╔═╡ 8fc6ee4b-5abc-45e5-a642-6570af32a492
begin
	# range of singular values
	r₄ = 1:4
	# first 4 columns of U
	u₄ = U[:,r₄]
	# first 4 columns of V
	v₄ = SVD.V[:,r₄]
	# Σ with the top 4 singular values
	σ₄ = spdiagm(Σ[r₄])

	# downsampled image
	img₄ = u₄*σ₄*v₄'

	# grayscale final image
	Gray.(img₄)
end

# ╔═╡ ddf9e602-ae0a-43c3-a4b9-c052c513ffc5
md"""
As you can see, it's still far away from the original image. Let's try using 50 singular vectors/values.
"""

# ╔═╡ 91575c63-16b5-4b30-8f50-392c058218c9
begin
	# range of singular values
	r = 1:50
	# first 50 columns of U
	u = U[:,r]
	# first 50 columns of V
	v = SVD.V[:,r]
	# Σ with the top 50 singular values
	σ = spdiagm(0 => SVD.S[r])

	# downsampled image
	img = u*σ*v'

	# grayscale final image
	Gray.(img)
end

# ╔═╡ 4113a8aa-af35-40fa-836a-f9e347431b5b
md"""
This looks better, even though it's not identical to the original image. We can see this from the norm difference below.
"""

# ╔═╡ eb0f7a49-c6b9-4b94-9fb1-f9cdcf6c8ee6
norm(Xgrayvalues - img)

# ╔═╡ bb25ecf5-b4ed-42a7-8ccd-b67d6d8b8bf0
md"""
### Face recognition

Our next problem will still be related to images, but this time we will solve a simple form of the face recognition problem. Let's get the data first.

**Note:** we can access the data (`Matrix` type) using the notation `data["V2"]`.
"""

# ╔═╡ 716fec98-ef10-465a-a216-beceb4057a91
data = matread("data/face_recog_qr.mat")

# ╔═╡ a9dd7d9e-c91e-4a35-ba77-b765161b58b5
md"""
Each one of the 490 vectors in `data["V2"]` is a fase image. Let's reshape the first one and take a look...
"""

# ╔═╡ 194a229c-0a2f-46a4-bdbe-678df9dae106
begin
	fig₁ = reshape(data["V2"][:,1], 192, 168)
	Gray.(fig₁)
end

# ╔═╡ 049cf246-8dbf-4af8-a60d-01e162097272
md"""
Now we will go back to the vectorized version of this image, and try to select the images that are most similar to it from the "dictionary" matrix.

Let's use `b = q[:]` to be the query image.

**Note:** the notation `[:]` vectorizes a matrix column wise.
"""

# ╔═╡ 70338fd5-570d-4c71-becd-b30a768554cc
b¹ = fig₁[:]

# ╔═╡ f1704b39-f48d-49ce-9ec2-77b1fa2af0f8
md"""
We will now remove the first image from the dictionary. The goal is to find the solution of the linear system $Ax=b$ where $A$ is the dictionary of all images.

In face recognition problem, we really want to minimize the norm differece $norm(Ax-b)$ but the `\` operator actually solves a least squares problem even when the matrix at hand is not invertible.
"""

# ╔═╡ dfbaf736-ccb1-4e7c-a2eb-b432e1498008
begin
	A¹ = data["V2"][:,2:end]
	x¹ = A¹\b¹

	# display image
	Gray.(reshape(A¹*x¹, 192, 168))
end

# ╔═╡ 383a0be3-ce45-43e6-9fe9-d925a40ad9f5
md"""
Now, let's check the norm difference...
"""

# ╔═╡ dd60c14e-d157-48a7-955f-845656636beb
norm(A¹*x¹-b¹)

# ╔═╡ 3f20fcb5-6e94-48e0-b2c3-9ff828269e23
md"""
This was an easy problem. Let's try to make the picture harder to recover. We will add some random error...
"""

# ╔═╡ 50c8a7f4-b620-441f-b941-ece127899ce0
begin
	fig_temp = fig₁ + rand(size(fig₁,1), size(fig₁,2)) * 0.5
	fig₂ = fig_temp ./ maximum(fig_temp)

	# display noisy image
	Gray.(fig₂)
end

# ╔═╡ 6aee282e-e19d-4e8f-bc45-bf20474d41b9
md"""
Let's define $b$ by vectorizing the matrix `fig₂` and then get $A$...
"""

# ╔═╡ 29e803bd-402e-484e-acb7-9053fc8f6a70
b² = fig₂[:]

# ╔═╡ 5511fbe1-901b-4c0f-b354-21795f3316dc
A² = A¹

# ╔═╡ bec4214d-6cbe-43d1-a02a-82d4245bdd27
md"""
Now, let's find the solution $x$ and check the norm difference...
"""

# ╔═╡ 798f5b17-48b4-4349-b850-e3e2e62418de
begin
	x² = A²\b²
	norm(A²*x² - b²)
end

# ╔═╡ ea16d6a3-34ac-4c36-b916-47439fbcd0fa
md"""
The error is so much bigger this time. Finally, let's check the resulting image...
"""

# ╔═╡ 64f6dd6a-0cbf-4f05-b780-c4b2c74ce839
Gray.(reshape(A²*x², 192, 168))

# ╔═╡ Cell order:
# ╟─6c3082f0-6196-11ec-1d58-03f55d178603
# ╟─6ff4cfec-ba3e-4937-bc3a-bf2d22f1a8f6
# ╟─63f2a85b-a426-4c37-a1a0-049d00cfed23
# ╟─a3ead5b3-0251-4054-b6f7-63bf51b5ff71
# ╟─1fe44825-0933-4fb8-9b36-511f0e7ffeea
# ╠═cd961076-6baf-4626-9089-d33b9dbc94ce
# ╟─eaaffef2-903c-49cf-80b4-bb5adf7d8e92
# ╠═92c1a395-bd96-477a-95dc-294a2ac99784
# ╟─5c852935-3055-4178-8ac3-3a6f1d6e8de9
# ╠═de94c2dc-b4b0-42a2-aff0-e54543df82e1
# ╠═4770ca47-71ea-4fe1-a5ad-52c4e31f3bdd
# ╠═eccced3e-5898-4d93-8950-58aa77fc4c0b
# ╟─02cbd31b-bd62-48f4-be5e-242279e215d9
# ╠═89fb6b4c-911b-4a39-bdef-b70dbe17f14f
# ╟─2082cb17-c373-459d-abc1-abe41cfa023a
# ╠═efc88903-189b-48f4-8cde-17bb2c407010
# ╟─e6c8e66b-f9a3-4b93-b5a4-97c030d381d4
# ╠═bb678a13-b45d-410b-b8d5-6346eb839c56
# ╟─41e22cf0-2448-4643-89ea-7315fb8b0b32
# ╠═e759d9a1-29cb-4e15-b629-a913ce721b6f
# ╟─f662e8dd-d998-4e8c-8924-aee9fe633129
# ╠═39797209-2206-495f-b517-2ed219ff47ca
# ╟─f9e97b10-4908-4811-a255-c76c0eaed01c
# ╠═157971c4-04d9-41de-b378-4a9e998725e9
# ╟─d82db838-5ba6-456d-bdeb-cb75a20e081c
# ╠═5dc61a15-48e9-46a2-b893-9fb97ecc8b55
# ╟─0ec44c81-40d5-4b3f-bdec-d3e0cab7de74
# ╠═10474148-5a43-4cc1-8e9f-de57bd3019bc
# ╟─859cb7ee-d9d8-4bd7-b7c9-5777867297b7
# ╠═9f56f781-5abe-407c-b449-0d917dfa2767
# ╠═32b59524-9b72-40a2-828f-96f20b0dd924
# ╟─4732c64c-c957-4bdb-9332-7ccb8f40f655
# ╠═b4b29ad0-8573-40ea-a2d8-c25183825149
# ╟─8c19fd6c-2f71-480d-8fd0-2241b6f3debe
# ╟─11bc3a82-19d2-453f-b40b-1eb488ebb323
# ╠═983aeaea-a556-49ba-aedf-898c1b71407d
# ╟─c8d51d21-ec99-4451-a87e-b51d8f118e24
# ╠═b2b6c7d0-8abb-415e-b668-1fe089b42b04
# ╠═790d0305-216f-4335-ab37-acff2de0f18f
# ╠═8bb9722c-9cb6-4316-831b-26cb64f93619
# ╠═e3571e13-1fa6-44d6-8a31-f810c11cd1f4
# ╠═311b85d4-6509-4945-a451-41020cac465e
# ╟─ea01ba90-0b30-4998-b1a5-e7fa17e53679
# ╠═ada5e3a2-b801-4d2b-8ed8-0b9d8e0cf132
# ╟─930c8f5d-a05b-4623-84c3-9b58fe59a721
# ╠═c03c19b1-ec14-4215-9b54-f6cc6b5a2e5c
# ╠═7371a76e-ae89-4138-a614-fa41bfb2dfb0
# ╠═da750f88-e883-4bec-8e2e-79a91da23f9b
# ╠═2669897d-cd5a-45c6-afe0-7a91153e3686
# ╟─737762ba-b142-4fd9-9862-af69f7415030
# ╠═b1ebbd9d-01d5-4eed-85c3-732491322582
# ╠═6cea7788-57a1-426b-8ede-0a424970e2e3
# ╠═f7a4fffa-a633-45c9-98a5-b25a047a8a3f
# ╟─c64cf987-9329-4139-b7e3-0fc21e6f0d92
# ╠═aa195042-2f7e-4881-9d01-834085dccc11
# ╠═503192fc-8384-4116-b389-38f04015953c
# ╠═61ea5d3c-63aa-480d-8252-0762f586232f
# ╠═49803952-b650-4e5b-9b65-a0d84b21134b
# ╟─3f0a1c9f-8b8a-4886-97bd-3810dfa0263b
# ╠═433c016e-334a-43fa-b315-608b4bbdef51
# ╟─1d9e27be-c21b-4b45-be25-dd1f5b954cb2
# ╟─d806f387-587a-438c-bb22-f6f14b4d4509
# ╠═1755c27b-b170-4b5f-acf9-87e722fd6fad
# ╠═10f4304f-06da-4f0a-bfa9-a148169b531f
# ╟─926cb115-c47f-44e3-bb40-53533d0a1354
# ╠═d86cf3cc-0490-4144-b732-41e40f380813
# ╟─7a7002b3-2523-4b33-8dde-511728dfaf0e
# ╠═9980569d-ca11-432e-afb5-33001dea3746
# ╠═a31e0b0d-159f-43f3-aed8-a3b7a33bb73b
# ╟─05bd36f7-b0db-4a58-a591-4cde55139ade
# ╠═fee37f40-b39d-4cfa-8f25-882a723d5874
# ╠═83ba1696-9345-49d8-993a-78517d4ffecb
# ╟─1d8970d9-7fae-4fef-b4e0-a92aa3e911a7
# ╠═512da58a-f8d5-4e0c-8c12-de12cc2a4ba5
# ╟─0293d061-05bc-484f-abcb-76dd4b54fd30
# ╠═f051a1cc-cd3d-465d-9ec0-cb711ad691d7
# ╟─ab47cafa-3d00-4e7a-8af9-103241fc68c1
# ╠═8ebe9e1d-981b-4f96-9598-91c8893b8818
# ╠═223cca30-9c72-4ae3-aa51-85c54a362cf4
# ╟─640da366-e78b-4a62-aedc-2668ae69ae95
# ╠═3a5c505a-0620-4cf4-a546-0a01c191b512
# ╟─6b140d76-a5c3-4b86-9cbd-116a724851af
# ╠═12bdbf11-c352-4243-b080-b28c34527e3b
# ╟─6d53f3e5-66d1-433f-ad4e-0800b0ecb70f
# ╠═17764c04-a202-4563-98a4-eddb9f432e6d
# ╟─a441d0b6-9a18-46e4-899d-f3ea2642c7b7
# ╠═0f82de90-1a24-49a6-9673-3abf01283817
# ╟─4eb22f0f-b287-4be2-b013-824371a44c26
# ╠═3dc31e0a-194c-4ed2-ae6f-c258170aba1a
# ╟─83c6b744-750c-4c74-8b79-a3d78e5b2c12
# ╠═b61a2c57-6443-4e8a-b48a-28e2ad1fca4f
# ╠═7f003ee7-6958-48e1-a3d8-11876cfa1dbd
# ╠═c2d97352-fb67-4d66-a999-a8362e41ee84
# ╠═45f5c9ac-d450-49d6-af9d-28704161f360
# ╟─946bf2e5-51e4-4b8b-a316-27e29f15c2e3
# ╠═69c75582-f4a1-443e-bda2-385681712821
# ╟─42a1ee17-86ab-41d8-8b89-5064700e2e64
# ╠═ff77b413-a905-44d6-973d-f7d1193dbb25
# ╠═f98fdf05-e6e8-4d0e-b6a6-d3398c2d1ab5
# ╠═34b5d940-7d9a-40fb-a38f-16b267130026
# ╟─3b2724d7-f429-447d-948c-e11a8fc20572
# ╠═c071dc2a-e53e-402b-b800-6601025b62c7
# ╠═5e54ac09-2276-4760-adc9-a05082a5c17f
# ╠═33b3de91-010a-4bd2-863d-199bfbfb8ed8
# ╠═ac394a4c-f983-487c-9d90-1cada3b9e398
# ╟─59daefac-6e47-4ecc-9f4a-033faa55fc59
# ╠═9d1fa923-e20a-44cc-b0b6-0656f85178fc
# ╟─a551a1cb-0d7d-4c82-8f7b-9d92274516f3
# ╠═d66ae839-2d03-40f6-a82e-1e8a063e27f0
# ╟─54caef76-2f0d-4e9f-a3a1-725ab3d64648
# ╠═a28364a5-5c03-4b42-b354-2cdb1880ca1d
# ╠═0a30e77d-8770-4b75-9c7d-e45f6d301022
# ╠═ac8e9a63-708c-408f-bc91-a57356dae648
# ╠═96766fac-6950-4554-b89a-32143f120f04
# ╠═1de26f9d-89eb-4397-adba-ed566cd054d6
# ╟─90922832-4ed1-4fa6-bb61-1242c17e1cd8
# ╠═8fc6ee4b-5abc-45e5-a642-6570af32a492
# ╟─ddf9e602-ae0a-43c3-a4b9-c052c513ffc5
# ╠═91575c63-16b5-4b30-8f50-392c058218c9
# ╟─4113a8aa-af35-40fa-836a-f9e347431b5b
# ╠═eb0f7a49-c6b9-4b94-9fb1-f9cdcf6c8ee6
# ╟─bb25ecf5-b4ed-42a7-8ccd-b67d6d8b8bf0
# ╠═716fec98-ef10-465a-a216-beceb4057a91
# ╟─a9dd7d9e-c91e-4a35-ba77-b765161b58b5
# ╠═194a229c-0a2f-46a4-bdbe-678df9dae106
# ╟─049cf246-8dbf-4af8-a60d-01e162097272
# ╠═70338fd5-570d-4c71-becd-b30a768554cc
# ╟─f1704b39-f48d-49ce-9ec2-77b1fa2af0f8
# ╠═dfbaf736-ccb1-4e7c-a2eb-b432e1498008
# ╟─383a0be3-ce45-43e6-9fe9-d925a40ad9f5
# ╠═dd60c14e-d157-48a7-955f-845656636beb
# ╟─3f20fcb5-6e94-48e0-b2c3-9ff828269e23
# ╠═50c8a7f4-b620-441f-b941-ece127899ce0
# ╟─6aee282e-e19d-4e8f-bc45-bf20474d41b9
# ╠═29e803bd-402e-484e-acb7-9053fc8f6a70
# ╠═5511fbe1-901b-4c0f-b354-21795f3316dc
# ╟─bec4214d-6cbe-43d1-a02a-82d4245bdd27
# ╠═798f5b17-48b4-4349-b850-e3e2e62418de
# ╟─ea16d6a3-34ac-4c36-b916-47439fbcd0fa
# ╠═64f6dd6a-0cbf-4f05-b780-c4b2c74ce839


# helpful bits for working with n \geq 2

"""
   uvec(x)

Helper to find a unit vector.
"""
function uvec(x)
    nm = norm(x)
    nm == 0 && return x
    return x/nm
end

"""
    `unzip(vs)`
    `unzip(v1, v2, ...)`
    `unzip(r::Function, a, b)`

Take a vector of points described by vectors (as returned by, say
`r(t)=[sin(t),cos(t)], r.([1,2,3])`, and return a tuple of collected x
values, y values, and optionally z values.

If the argument is specified as a comma separated collection of vectors, then these are combined and passed along.

If the argument is a function and two end point, then the function is
evaluated at 100 points between `a` and `b`.

This is useful for plotting when the data is more conveniently
represented in terms of vectors, but the plotting interface requires the x and y values collected.

Examples:
```
using Plots
r(t) = [sin(t), cos(t)]
rp(t) = [cos(t), -sin(t)]
plot(unzip(r, 0, 2pi)...)  # calls plot(xs, ys)

t0, t1 = pi/6, pi/4

p, v = r(t0), rp(t0)
plot!(unzip(p, p+v)...)  # connect p to p+v with line

p, v = r(t1), rp(t1)
quiver!(unzip([p])..., quiver=unzip([v]))
```

Based on `unzip` from the `Plots` package.
"""
unzip(vs::Vector) = Tuple([[vs[i][j] for i in eachindex(vs)] for j in eachindex(vs[1])])
function unzip(ws::Array; recursive=false)
    if recursive
        unzip([unzip(ws[:,j]) for j in 1:size(ws)[end]])
    else
        Tuple(eltype(first(ws))[xyz[j] for xyz in ws] for j in eachindex(first(ws)))
    end
end
#unzip(vs) = (A=hcat(vs...); Tuple([A[i,:] for i in eachindex(vs[1])]))
unzip(v,vs...) = unzip([v, vs...])
unzip(r::Function, a, b, n=100) = unzip(r.(range(a, stop=b, length=n)))

## -----------------------------------

## The gradient in SymPy.
## We avoid name collision with  ForwardDiff.gradient we use `grad`
import ForwardDiff: gradient
gradient(ex::SymPy.Sym, vars::AbstractArray=free_symbols(ex)) = diff.(ex, [vars...])
gradient(f::Function) = (x, xs...) -> ForwardDiff.gradient(f, vcat(x, xs...))

"""
    curl(F)

Find curl of a 2 or 3-D vector field.
"""
function curl(J::Matrix)
    if size(J) == (2,2)
        Mx, Nx, My, Ny = J
        return Nx - My # a scalar
    elseif size(J) == (3,3)
        Mx, Nx, Px, My, Ny, Py, Mz, Nz, Pz = J
        return [Py-Nz, Mz-Px, Nx-My] # ∇×VF
    else
        throw(ArgumentError("Wrong size jacobian matrix for a curl"))
    end
end
curl(F::Vector{Sym}, vars=free_symbols(F)) = curl(F.jacobian(vars))
curl(F::Tuple) = curl(F[1], F[2])
curl(F::Function, pt) = curl(ForwardDiff.jacobian(F, float.(pt)))
curl(F::Function) = (pt, pts...) -> curl(F, vcat(pt, pts...))

"""
    divergence(F)

Find divergence of a 3-D vector vield.
"""
divergence(F::Vector{Sym}, vars=free_symbols(F)) = sum(diff.(F, vars))
divergence(F::Tuple) = divergence(F[1], F[2])
divergence(F::Function, pt) = sum(diag(ForwardDiff.jacobian(F, float.(pt))))
divergence(F::Function) = (pt, pts...) -> divergence(F, vcat(pt, pts...))

## Is this a bad idea?
## syntax is a bit heavy with parentheses

struct DelOperator end
const ∇ = DelOperator()

## For symbolic objects, these use `free_symbols(ex)` to find variables
## this *may* not be right if sorting isn't as desired, or variables don't
## appear in the formula
(::DelOperator)(f) = return gradient(f)
(::DelOperator)(f::Tuple) = gradient(f[1],f[2])
LinearAlgebra.dot(::DelOperator, F) = divergence(F)
LinearAlgebra.dot(::DelOperator, F, vars) = divergence(F, vars)

LinearAlgebra.cross(::DelOperator, F) = curl(F)
LinearAlgebra.cross(::DelOperator, F, vars) = curl(F, vars)

##


# Canonical Correlation Analysis

#### CCA Type
"""
Canonical Correlation Analysis Model
"""
struct CCA{T<:Real} <: RegressionModel
    xmean::Vector{T}  # sample mean of X: of length dx (can be empty)
    ymean::Vector{T}  # sample mean of Y: of length dy (can be empty)
    xproj::Matrix{T}  # projection matrix for X, of size (dx, p)
    yproj::Matrix{T}  # projection matrix for Y, of size (dy, p)
    corrs::Vector{T}  # correlations, of length p

    function CCA(xm::Vector{T},
                 ym::Vector{T},
                 xp::Matrix{T},
                 yp::Matrix{T},
                 crs::Vector{T}) where T<:Real

        dx, px = size(xp)
        dy, py = size(yp)

        isempty(xm) || length(xm) == dx ||
            throw(DimensionMismatch("Incorrect length of xmean."))

        isempty(ym) || length(ym) == dy ||
            throw(DimensionMismatch("Incorrect length of ymean."))

        px == py ||
            throw(DimensionMismatch("xproj and yproj should have the same number of columns."))

        length(crs) == px ||
            throw(DimensionMismatch("Incorrect length of corrs."))

        new{T}(xm, ym, xp, yp, crs)
    end
end

## properties

"""
    size(M:CCA)

Return a tuple with the dimension of `X`, `Y`, and the output dimension.
"""
size(M::CCA) = (size(M.xproj, 1), size(M.yproj, 1), size(M.xproj, 2))

"""
    mean(M::CCA, c::Symbol)

Get the mean vector for the component `c` of the model `M`.
The component parameter can be `:x` or `:y`.
"""
function mean(M::CCA, c::Symbol)
    xi, yi, o = size(M)
    if c == :x
        fullmean(xi, M.xmean)
    elseif c == :y
        fullmean(yi, M.ymean)
    else
        throw(ArgumentError("Unknown component $c"))
    end
end

"""
    projection(M::CCA, c::Symbol)

Get the projection matrix for the component `c` of the model `M`.
The component parameter can be `:x` or `:y`.
"""
function projection(M::CCA, c::Symbol)
    if c == :x
        M.xproj
    elseif c == :y
        M.yproj
    else
        throw(ArgumentError("Unknown component $c"))
    end
end

"""
    cor(M::CCA)

The correlations of the projected componnents (a vector of length `p`).
"""
cor(M::CCA) = M.corrs

## use
"""
    predict(M::CCA, Z::AbstractVecOrMat{<:Real}, c::Symbol)

Given a [`CCA`](@ref) model, one can transform observations into both spaces into a common space, as

```math
\\mathbf{z}_x = \\mathbf{P}_x^T (\\mathbf{x} - \\boldsymbol{\\mu}_x) \\\\
\\mathbf{z}_y = \\mathbf{P}_y^T (\\mathbf{y} - \\boldsymbol{\\mu}_y)
```

Here, ``\\mathbf{P}_x`` and ``\\mathbf{P}_y`` are projection matrices for ``X`` and ``Y``;
``\\boldsymbol{\\mu}_x`` and ``\\boldsymbol{\\mu}_y`` are mean vectors.

Parameter `Z` can be either a vector of length `dx`, `dy`, or a matrix where each column is an observation. The component parameter `c` can be `:x` or `:y`.
"""
function predict(M::CCA, Z::AbstractVecOrMat{<:Real}, c::Symbol)
    if c == :x
        transpose(M.xproj) * centralize(Z, M.xmean)
    elseif c == :y
        transpose(M.yproj) * centralize(Z, M.ymean)
    else
        throw(ArgumentError("Unknown component $c"))
    end
end

## show

function show(io::IO, M::CCA)
    xi, yi, o = size(M)
    print(io, "CCA (xindim = $xi, yindim = $yi, outdim = $o)")
end

#### Perform CCA on data

"""
    ccacov(Cxx, Cyy, Cxy, xmean, ymean, p)

Compute CCA based on analysis of the given covariance matrices, using generalized
eigenvalue decomposition, and return [`CCA`](@ref) model.

Parameters:
- `Cxx`: The covariance matrix of `X`.
- `Cyy`: The covariance matrix of `Y`.
- `Cxy`: The covariance matrix between `X` and `Y`.
- `xmean`: The mean vector of the **original** samples of `X`, which can be
a vector of length `dx`, or an empty vector indicating a zero mean.
- `ymean`: The mean vector of the **original** samples of `Y`, which can be
a vector of length `dy`, or an empty vector indicating a zero mean.
- `p`: The output dimension, *i.e* the dimension of the common space.
"""
function ccacov(Cxx::DenseMatrix{T},
                Cyy::DenseMatrix{T},
                Cxy::DenseMatrix{T},
                xmean::Vector{T},
                ymean::Vector{T},
                p::Int) where T<:Real

    # argument checking
    dx, dx2 = size(Cxx)
    dy, dy2 = size(Cyy)
    dx == dx2 || error("Cxx must be a square matrix.")
    dy == dy2 || error("Cyy must be a square matrix.")
    size(Cxy) == (dx, dy) ||
        throw(DimensionMismatch("size(Cxy) should be equal to (dx, dy)"))

    isempty(xmean) || length(xmean) == dx ||
        throw(DimensionMismatch("Incorrect length of xmean."))

    isempty(ymean) || length(ymean) == dy ||
        throw(DimensionMismatch("Incorrect length of ymean."))

    1 <= p <= min(dx, dy) ||
        throw(DimensionMismatch(""))

    _ccacov(Cxx, Cyy, Cxy, xmean, ymean, p)
end

function _ccacov(Cxx, Cyy, Cxy, xmean, ymean, p::Int)
    dx = size(Cxx, 1)
    dy = size(Cyy, 1)

    # solve Px and Py

    if dx <= dy
        # solve Px: (Cxy * inv(Cyy) * Cyx) Px = λ Cxx * Px
        # compute Py: inv(Cyy) * Cyx * Px

        G = cholesky(Cyy) \ Cxy'
        Ex = eigen(Symmetric(Cxy * G), Symmetric(Cxx))
        ord = sortperm(Ex.values; rev=true)
        vx, Px = extract_kv(Ex, ord, p)
        Py = qnormalize!(G * Px, Cyy)
    else
        # solve Py: (Cyx * inv(Cxx) * Cxy) Py = λ Cyy Py
        # compute Px: inv(Cx) * Cxy * Py

        H = cholesky(Cxx) \ Cxy
        Ey = eigen(Symmetric(Cxy'H), Symmetric(Cyy))
        ord = sortperm(Ey.values; rev=true)
        vy, Py = extract_kv(Ey, ord, p)
        Px = qnormalize!(H * Py, Cxx)
    end

    # compute correlations
    # Note: Px' * Cxx * Px == I
    #       Py' * Cyy * Py == I
    crs = coldot(Px, Cxy * Py)

    # construct CCA model
    CCA(xmean, ymean, Px, Py, crs)
end

"""
    ccasvd(Zx, Zy, xmean, ymean, p)

Compute CCA based on singular value decomposition of centralized sample matrices `Zx` and `Zy`, and return [`CCA`](@ref) model[^1].

Parameters:
- `Zx`: The centralized sample matrix for `X`.
- `Zy`: The centralized sample matrix for `Y`.
- `xmean`: The mean vector of the **original** samples of `X`, which can be
a vector of length `dx`, or an empty vector indicating a zero mean.
- `ymean`: The mean vector of the **original** samples of `Y`, which can be
a vector of length `dy`, or an empty vector indicating a zero mean.
- `p`: The output dimension, *i.e* the dimension of the common space.
"""
function ccasvd(Zx::DenseMatrix{T},
                Zy::DenseMatrix{T},
                xmean::Vector{T},
                ymean::Vector{T},
                p::Int) where T<:Real

    dx, n = size(Zx)
    dy, n2 = size(Zy)
    n == n2 ||
        throw(DimensionMismatch("Zx and Zy must have the same number of columns."))

    isempty(xmean) || length(xmean) == dx ||
        throw(DimensionMismatch("Incorrect length of xmean."))

    isempty(ymean) || length(ymean) == dy ||
        throw(DimensionMismatch("Incorrect length of ymean."))

    1 <= p <= min(dx, dy) ||
        throw(DimensionMismatch(""))

    _ccasvd(Zx, Zy, xmean, ymean, p)
end

# The implementation is partly based on:
#
#   David Weenink.
#   Canonical Correlation Analysis.
#   Institute of Phonetic Sciences, Univ. of Amsterdam,
#   Proceedings 25 (2003), 81-99.
#
#   Note: in this paper, each row is considered as an observation.
#   The algorithm is adpated to the column-major format here.
#
function _ccasvd(Zx::DenseMatrix{T}, Zy::DenseMatrix{T}, xmean::Vector{T}, ymean::Vector{T}, p::Int) where T<:Real
    # svd factorization of Z

    n = size(Zx, 2)

    # svd decomposition
    Sx = svd(Zx)
    Sy = svd(Zy)
    S = svd!(Sx.Vt * transpose(Sy.Vt)) # svd of Vx * Vy'

    # compute Px and Py
    ord = sortperm(S.S; rev=true)
    si = ord[1:p]
    Px = rmul!(Sx.U, Diagonal(1.0 ./ Sx.S)) * S.U[:, si]
    Py = rmul!(Sy.U, Diagonal(1.0 ./ Sy.S)) * S.V[:, si]

    # scale so that Px' * Cxx * Py == I
    #           and Py' * Cyy * Py == I,
    #
    # with Cxx = Zx * Zx' / (n - 1)
    #      Cyy = Zy * Zy' / (n - 1)
    #
    rmul!(Px, sqrt(n-1))
    rmul!(Py, sqrt(n-1))

    # compute correlations
    crs = rmul!(coldot(Zx'Px, Zy'Py), one(T)/(n-1))

    # construct CCA model
    CCA(xmean, ymean, Px, Py, crs)
end

## interface functions

"""
    fit(CCA, X, Y; ...)

Perform CCA over the data given in matrices `X` and `Y`.
Each column of `X` and `Y` is an observation.

`X` and `Y` should have the same number of columns (denoted by `n` below).

This method returns an instance of [`CCA`](@ref).

**Keyword arguments:**
- `method`: The choice of methods:
    - `:cov`: based on covariance matrices
    - `:svd`: based on SVD of the input data (*default*)
- `outdim`: The output dimension, *i.e* dimension of the common space (*default*: `min(dx, dy, n)`)
- `mean`: The mean vector, which can be either of:
    - `0`: the input data has already been centralized
    - `nothing`: this function will compute the mean (*default*)
    - a pre-computed mean vector

**Notes:** This function calls [`ccacov`](@ref) or [`ccasvd`](@ref) internally, depending on the choice of method.
"""
function fit(::Type{CCA}, X::AbstractMatrix{T}, Y::AbstractMatrix{T};
             outdim::Int=min(min(size(X)...), min(size(Y)...)),
             method::Symbol=:svd,
             xmean=nothing,
             ymean=nothing) where T<:Real

    dx, n = size(X)
    dy, n2 = size(Y)

    n2 == n ||
        throw(DimensionMismatch("X and Y should have the same number of columns."))

    (n >= dx && n >= dy) ||
        @warn("CCA would be numerically instable when n < dx or n < dy.")

    xmv = preprocess_mean(X, xmean)
    ymv = preprocess_mean(Y, ymean)

    Zx = centralize(X, xmv)
    Zy = centralize(Y, ymv)

    if method == :cov
        Cxx = rmul!(Zx*transpose(Zx), inv(n - 1))
        Cyy = rmul!(Zy*transpose(Zy), inv(n - 1))
        Cxy = rmul!(Zx*transpose(Zy), inv(n - 1))
        M = ccacov(Cxx, Cyy, Cxy, xmv, ymv, outdim)
    elseif method == :svd
        M = ccasvd(Zx, Zy, xmv, ymv, outdim)
    else
        error("Invalid method name $(method)")
    end

    return M::CCA
end


using Distributions, NLsolve, Statistics

function effectsize(xs::AbstractVector, ys::AbstractVector)
    if length(xs) != length(ys)
        throw(ArgumentError("samples must have the same number of observations"))
    end
    n, m = length(xs), length(ys)
    x̄, ȳ = mean(xs), mean(ys)
    σx², σy² = var(xs), var(ys)
    σ = sqrt(0.5*(σx² + σy²))
    abs(x̄ - ȳ)/σ
end

function power(xs::AbstractVector, ys::AbstractVector, n::Int=length(xs), p::Float64=0.05; unequalvar=true)
    dof = if unequalvar
        varx, vary = var(xs), var(ys)
        nx, ny = length(xs), length(ys)
        (varx / nx + vary / ny)^2 / ((varx / nx)^2 / (nx - 1) + (vary / ny)^2 / (ny - 1))
    else
        length(xs) + length(ys) - 2
    end
    power(effectsize(xs, ys), n, dof, p)
end

function power(d::Float64, n::Int=length(xs), dof=2n-2, α::Float64=0.05)
    δ = d*sqrt(n/2)
    tc = quantile(TDist(dof), 1 - α/2)
    ccdf(NoncentralT(dof, δ), tc) + cdf(NoncentralT(dof, δ), -tc)
end

function power(d::Float64, n::Float64=length(xs), dof=2n-2, α::Float64=0.05)
    δ = d*sqrt(n/2)
    tc = quantile(TDist(dof), 1 - α/2)
    ccdf(NoncentralT(dof, δ), tc) + cdf(NoncentralT(dof, δ), -tc)
end

function solvepower(d::Float64, dof::Float64, β::Float64=0.8, α::Float64=0.05)
    first(nlsolve((F,x) -> F[1] = power(d, x[1], dof, α) - β, [2.0]).zero)
end

function solvepower(xs::AbstractVector, ys::AbstractVector, β::Float64=0.8, α::Float64=0.05; unequalvar=true)
    dof = if unequalvar
        varx, vary = var(xs), var(ys)
        nx, ny = length(xs), length(ys)
        (varx / nx + vary / ny)^2 / ((varx / nx)^2 / (nx - 1) + (vary / ny)^2 / (ny - 1))
    else
        length(xs) + length(ys) - 2
    end
    solvepower(effectsize(xs, ys), dof, β, α)
end


#=

Log linear optimal growth model, with log utility, CD production and
multiplicative lognormal shock, so that

    y = f(k, z) = z k^alpha

with z ~ LN(mu, s).

=#

alpha = 0.4
beta = 0.96
mu = 0
s = 0.1

ab = alpha * beta
c1 = log(1 - ab) / (1 - beta)
c2 = (mu + alpha * log(ab)) / (1 - alpha)
c3 = 1 / (1 - beta)
c4 = 1 / (1 - ab)

# Utility
u(c) = log(c)

u_prime(c) = 1 / c

# Deterministic part of production function
f(k) = k^alpha

f_prime(k) = alpha * k^(alpha - 1)

# True optimal policy
c_star(y) = (1 - alpha * beta) * y

# True value function
v_star(y) = c1 + c2 * (c3 - c4) + c4 * log(y)


module CentroidLinearClassifierImpl

export train_model

using LinearAlgebra
using ..Clusters, ..LinearClassifiers

"""
    train_model(positive_data, negative_data)

Trains a binary classification model.

The returned model has a field `classify`, a function that classifies a point
and returns a scalar value, where positive and negative results indicate that
the point is positively or negatively classified, respectively.
"""
function train_model(positive_data::AbstractMatrix{<:Number},
                     negative_data::AbstractMatrix{<:Number})::CentroidLinearClassifier
    @assert size(positive_data, 2) == size(negative_data, 2)

    positive_data_centroid = compute_centroid(positive_data)
    negative_data_centroid = compute_centroid(negative_data)

    normal_vector = positive_data_centroid .- negative_data_centroid
    midpoint = (positive_data_centroid .+ negative_data_centroid) ./ 2

    offset = -1 * dot(normal_vector, midpoint)

    decision_function =
        function (x)
            return dot(normal_vector, x) + offset
        end

    return CentroidLinearClassifier(decision_function, normal_vector, offset)
end

end # module


#=

Plots reservation wage against the discount factor

=#

using Plots, LaTeXStrings
pyplot()

include("mccall_bellman_iteration.jl")
include("compute_reservation_wage.jl")

grid_size = 25
beta_vals = linspace(0.8, 0.99, grid_size)
w_bar_vals = similar(beta_vals)

mcm = McCallModel()

for (i, beta) in enumerate(beta_vals)
    mcm.beta = beta
    w_bar = compute_reservation_wage(mcm)
    w_bar_vals[i] = w_bar
end

plot(beta_vals,
    w_bar_vals,
    lw=2,
    alpha=0.7,
    xlabel="discount rate",
    ylabel="reservation wage",
    label=L"$\bar w$ as a function of $\beta$")


# # SVM with L^1 regularization
## Generate data for SVM classifier with L1 regularization.
using Random
Random.seed!(3);
n = 20;
m = 1000;
TEST = m;
DENSITY = 0.2;
beta_true = randn(n, 1);
idxs = randperm(n)[1:round(Int, (1 - DENSITY) * n)];
beta_true[idxs] .= 0
offset = 0;
sigma = 45;
X = 5 * randn(m, n);
Y = sign.(X * beta_true .+ offset .+ sigma * randn(m, 1));
X_test = 5 * randn(TEST, n);

#-

## Form SVM with L1 regularization problem.
using Convex, SCS, ECOS

beta = Variable(n);
v = Variable();
loss = sum(pos(1 - Y .* (X * beta - v)));
reg = norm(beta, 1);

## Compute a trade-off curve and record train and test error.
TRIALS = 100
train_error = zeros(TRIALS);
test_error = zeros(TRIALS);
lambda_vals = exp10.(range(-2, stop = 0, length = TRIALS);)
beta_vals = zeros(length(beta), TRIALS);
for i in 1:TRIALS
    lambda = lambda_vals[i]
    problem = minimize(loss / m + lambda * reg)
    solve!(problem, SCS.Optimizer; silent_solver = true)
    train_error[i] =
        sum(
            float(
                sign.(X * beta_true .+ offset) .!=
                sign.(evaluate(X * beta - v)),
            ),
        ) / m
    test_error[i] =
        sum(
            float(
                sign.(X_test * beta_true .+ offset) .!=
                sign.(evaluate(X_test * beta - v)),
            ),
        ) / TEST
    beta_vals[:, i] = evaluate(beta)
end

#-

# Plot the train and test error over the trade-off curve.
using Plots
plot(lambda_vals, train_error, label = "Train error");
plot!(lambda_vals, test_error, label = "Test error");
plot!(xscale = :log, yscale = :log, ylabel = "errors", xlabel = "lambda")

#-

# Plot the regularization path for beta.

plot()
for i in 1:n
    plot!(lambda_vals, vec(beta_vals[i, :]), label = "beta$i")
end
plot!(xscale = :log, ylabel = "betas", xlabel = "lambda")


using LinearAlgebraicRepresentation
Lar = LinearAlgebraicRepresentation
using ViewerGL
GL = ViewerGL

V,(VV,EV,FV) = Lar.simplex(2, true)
triangle = (V,EV,FV)
model = Lar.Struct([ triangle, Lar.t(.15,.15), Lar.s(.5,.5), triangle ])
V,EV,FV = Lar.struct2lar(model)
VV = [[k] for k=1:size(V,2)]
GL.VIEW(GL.numbering(.4)((V,[VV, EV, FV]),GL.COLORS[1],0.2));

triangles = Lar.triangulate2d(V, EV)
GL.VIEW([ GL.GLGrid(V,triangles) ])

# generate edges
ev = map(sort,cat([[[u,v], [v,w], [w,u]] for (u,v,w) in triangles]))
# remove duplicated edges from triangulation
nodupev = collect(Set(ev))
GL.VIEW(GL.numbering(.4)((V,[VV, nodupev])));


using Printf

include("gen_rmesh_exp.jl")
include("gen_drmesh_exp.jl")
include("integrate.jl")

function my_func(r)
    return exp(-0.2*r)
    #return exp(-0.2*r*r)
end

function main()
    r_min = 0.0
    r_max = 200.0
    a = 1e9
    N = 500

    rmesh = gen_rmesh_exp(r_min, r_max, a, N)
    drmesh = gen_drmesh_exp(r_min, r_max, a, N)

    f = my_func.(rmesh)
    s = integrate_trapz_7(f, drmesh)

    println("s = ", s)

end

main()


"""
    random_unit_vector(N)

Generates N random unit vectors, i.e. random points on a unit sphere.
"""
function random_unit_vector(N::Int64)
    phis = 2 * pi * rand(Float64, N)
    cts = 2 * rand(Float64, N) .- 1
    sts = sqrt.(1. .- cts .* cts) # sin(acos(cts)) # max(0., )

    [SVector{3, Float64}(
        sts[i] .* cos.(phis[i]),
        sts[i] .* sin.(phis[i]),
        cts[i]
    ) for i in 1:N]
end


@testset "Kronecker sums" begin

    A = rand(4,4); IA = oneunit(A)
    B = rand(3,3); IB = oneunit(B)

    KS = A ⊕ B
    kronsum = kron(A, IB) + kron(IA, B)

    @test collect(KS) ≈ kronsum
    @test collect(KS) isa SparseMatrixCSC

    @test issquare(KS)

    C = rand(5,5); IC = oneunit(C)
    KS3 = A ⊕ B ⊕ C
    kronsum3 = kron(A,IB,IC) + kron(IA,B,IC) + kron(IA,IB,C)

    @test collect(KS3) ≈ kronsum3
    @test collect(KS3) isa SparseMatrixCSC

    @test order(KS) == 2
    @test order(KS3) == 3

    @test getmatrices(KS) == (A,B)

    @test getindex(KS,2,3) == kronsum[2,3]
    @test getindex(KS3,2,3) == kronsum3[2,3]

    D = rand(ComplexF64,6,6); ID = oneunit(D)

    @testset "Structure of sums" begin
        @test size(A ⊕ D) == size(A) .* size(D,1)
        @test size(B ⊕ C ⊕ D) == size(B) .* size(C) .* size(D)
        @test eltype(A ⊕ B) == Float64
        @test eltype(C ⊕ D) == ComplexF64
    end

    @testset "Basic linear algebra for sums" begin
        @test tr(KS) ≈ tr(kronsum)
        @test KS' == kronsum'
        @test transpose(KS) == transpose(kronsum)
        @test conj(KS) == conj(kronsum)
    end

    A = rand(10,10); B = rand(10,10); V = Diagonal(rand(10))
    @testset "Vec trick for sums" begin
        @test (A ⊕ B) * vec(V) == vec(B*V + V*transpose(A))
    end

    A = rand(3, 3); IA = oneunit(A)
    B = rand(4, 4); IB = oneunit(B)
    @testset "exp for Kronecker sum" begin
        EKS = exp(A ⊕ B)
        @test EKS isa AbstractKroneckerProduct
        @test EKS ≈ exp(kron(A, IB) + kron(IA, B))
    end

    @testset "sum over Kronecker sum" begin
        @test sum(KS) ≈ sum(kronsum)
        @test sum(KS3) ≈ sum(kronsum3)

        for dims in 1:2
            @test sum(KS, dims=dims) ≈ sum(kronsum, dims=dims)
            @test sum(KS3, dims=dims) ≈ sum(kronsum3, dims=dims)
        end

        @test_throws ArgumentError sum(KS, dims=-1)
        @test_throws ArgumentError sum(KS3, dims=3)
    end

end


# # Worst case risk analysis
# Generate data for worst-case risk analysis.
using Random

Random.seed!(2);
n = 5;
r = abs.(randn(n, 1)) / 15;
Sigma = 0.9 * rand(n, n) .- 0.15;
Sigma_nom = Sigma' * Sigma;
Sigma_nom .-= (maximum(Sigma_nom) - 0.9)

#-

# Form and solve portfolio optimization problem.
# Here we minimize risk while requiring a 0.1 return.
using Convex, SCS
w = Variable(n);
ret = dot(r, w);
risk = sum(quadform(w, Sigma_nom));
problem = minimize(risk, [sum(w) == 1, ret >= 0.1, norm(w, 1) <= 2])
solve!(problem, SCS.Optimizer; silent_solver = true)
wval = vec(evaluate(w))

#-

# Form and solve worst-case risk analysis problem.
Sigma = Semidefinite(n);
Delta = Variable(n, n);
risk = sum(quadform(wval, Sigma));
problem = maximize(
    risk,
    [
        Sigma == Sigma_nom + Delta,
        diag(Delta) == 0,
        abs(Delta) <= 0.2,
        Delta == Delta',
    ],
);
solve!(problem, SCS.Optimizer; silent_solver = true)
println(
    "standard deviation = ",
    round(sqrt(wval' * Sigma_nom * wval), sigdigits = 2),
);
println(
    "worst-case standard deviation = ",
    round(sqrt(evaluate(risk)), sigdigits = 2),
);
println("worst-case Delta = ");
println(round.(evaluate(Delta), sigdigits = 2));


# TSP & Simulated Annealing

# Import Packages and Files

using Plots
using Distances

include("sumLen.jl")
include("plotPath.jl")

# Constants

n = 20                              # Number of Cities
temperature = Float64(100 * n)      # Starting Temperature
iter = 100                          # Iteration Times

# Randomizing Coordinates

city = rand(n, 2) * 100 .+ 1

# Interation Notes

l = 1                               # Interation Times
len = []                            # Sum of Distance

plot()
plotPath(city)                      # Plot the Path

# Iteration

while true

    global n, temperature, iter, city, cityNew, l, len

    for ii = 1: iter

        len1 = sumLen(city)         # Sum of Distance before Exchange

        cityNew = copy(city)        # Exchange Coordinates

        while true

            p =  Int64.(floor.(n * rand(2, 1) .+ 1))

            if p[1] != p[2]

                cityNew[p[1], :] = city[p[2], :]
                cityNew[p[2], :] = city[p[1], :]

                break

            end

        end

        len2 = sumLen(cityNew)      # Sum of Distance after Exchange

        deltaE = len2 - len1        # Energy difference

        if (deltaE < 0) || (exp(- deltaE / temperature) > rand())

            city = cityNew          # Accept

        end

    end

    len = [len; sumLen(city)]       # New Length
    l += 1
    temperature *= 0.99             # Lower Temperature

    if temperature < 0.001

        break                       # Stop

    end

end

# Output

plotPath(city)                      # Plot Result
println("min = $(len[end])")


## Configuration
n = 5
adj = Float64[0 0 1 1 0;
              0 0 1 0 0;
              0 1 0 0 0;
              1 0 1 0 1;
              0 1 1 0 0]
adj ./= sum(adj,2)

## Graph algorithm
pageRank = ones(Float64,n)
N = 30
for t = 1:N
  temp = zeros(pageRank)
  for j = 1:n
    temp[j] = 0.15 + 0.85 * vecdot(pageRank,adj[:,j])
  end
  pageRank = temp
end
pageRank

## Markov Chain algorithm
comAdj = [0.85*adj 0.15*ones(n); ones(n)'/n 0.0]
comPageRank = eigvecs(comAdj')[:,1]
comPageRank = comPageRank / sum(comPageRank) * 1.15n


function test_mod_multiplicativeorder()
    @test multiplicativeorder(1, 11) == 1
    @test multiplicativeorder(2, 7) == 3
    @test multiplicativeorder(3, 7) == 6
    @test multiplicativeorder(7, 5) == 4
    @test multiplicativeorder(37, 1000) == 100

    @test_throws ErrorException multiplicativeorder(3, -6)
end

function test_mod_linearmod()
    @test linearmod(3, 5, 6) == []
    @test linearmod(3, 4, 5) == [3]
    @test linearmod(3, 6, 9) == [2, 5, 8]
    @test linearmod(-14, 30, 100) == [5, 55]

    @test_throws ErrorException linearmod(3, 4, 1)
end

function test_mod_primitiveroot()
    @test primitiveroot(2) == 1
    @test primitiveroot(17) == 3
    @test primitiveroot(71) == 7
    @test primitiveroot(101) == 2

    @test_throws ErrorException primitiveroot(4)
end

function test_mod_all()
    print(rpad("Math.NumberTheory.Mod...", 50, ' '))

    test_mod_multiplicativeorder()
    test_mod_linearmod()
    test_mod_primitiveroot()

    println("PASS")
end


# ------------------------------------------------------------------
# Licensed under the MIT License. See LICENCE in the project root.
# ------------------------------------------------------------------

# -------------
# COMPOSITIONS
# -------------

"""
    alr(c)

Additive log-ratio transformation of composition `c`.
"""
function alr(c::Composition{D}) where {D}
  w = components(c) .+ eps()
  SVector(ntuple(i -> log(w[i] / w[D]), D-1))
end

"""
    alrinv(x)

Inverse alr transformation of coordinates `x`.
"""
alrinv(x::SVector{D,T}) where {D,T<:Real} =
  Composition(𝒞([exp.(x); SVector(one(T))]))

alrinv(x::AbstractVector) = alrinv(SVector{length(x)}(x))

# -------
# TABLES
# -------

"""
    ALR([refvar])

Additive log-ratio transform following the
[TableTransforms.jl](https://github.com/JuliaML/TableTransforms.jl)
interface.

Optionally, specify the reference variable `refvar` for the ratios.
Default to the last column of the input table.
"""
struct ALR <: LogRatio
  refvar::Union{Symbol,Nothing}
end

ALR() = ALR(nothing)

refvar(transform::ALR, vars) =
  isnothing(transform.refvar) ? last(vars) : transform.refvar

newvars(::ALR, n) = collect(n)[begin:end-1]

oldvars(::ALR, vars, rvar) = [collect(vars); rvar]

function applymatrix(::ALR, X)
  L = log.(X .+ eps())
  L[:,begin:end-1] .- L[:,end]
end

function revertmatrix(::ALR, Y)
  E = [exp.(Y) ones(size(Y,1))]
  mapslices(𝒞, E, dims=2)
end


#=
HackerRank - Binomial distribution
https://www.hackerrank.com/challenges/s10-binomial-distribution-1/problem
https://www.hackerrank.com/challenges/s10-binomial-distribution-2/problem
=#
using Printf

function comb(n::Int, r::Int)
    r == 0 ? 1.0 : reduce(*, n-r+1:n, init=1) / reduce(*, 1:r, init=1)
end

function binomial(p::Float64, n::Int64, r::Int64)
    comb(n, r) * p ^ r * (1 - p) ^ (n - r)
end

# Binomial Problem 1
# What is the probability of having at least three boys out of six kids.
let ratio = 1.09    # Ratio of a boy or an event
    local p = ratio / (1.0 + ratio)
    # at least 3 out of 6
    local n = 6
    local c = 3
    local result = mapreduce(c -> binomial(p, n, c), +, c:n)
    @printf("%.3f", result)
end

## Binomial Problem 2
## What are the probabilities of:
##  No more than two rejected, and
##  At least two rejected.
let defect_ratio = 0.12, n = 10
    # No more than two rejected
    local result1 = mapreduce(x -> binomial(defect_ratio, n, x), +, 0:2)
    @printf("%.3f\n", result1)
    local result2 = 1.0 - mapreduce(x -> binomial(defect_ratio, n, x), +, 0:1)
    @printf("%.3f\n", result2)
end


import FractionalDiffEq.solve, FractionalDiffEq.FractionalDiffEqAlgorithm

using LinearAlgebra, InvertedIndices

"""
Using [triangular strip matrices](https://en.wikipedia.org/wiki/Triangular_matrix) to discrete fractional differential equations to simple algebra system and solve the system.

```tex
@inproceedings{Podlubny2000MATRIXAT,
  title={MATRIX APPROACH TO DISCRETE FRACTIONAL CALCULUS},
  author={Igor Podlubny},
  year={2000}
}
```
"""
struct MatrixDiscrete <: FractionalDiffEqAlgorithm end

"""
@inproceedings{Podlubny1998FractionalDE,
  title={Fractional differential equations},
  author={Igor Podlubny},
  year={1998}
}
"""

"""
@inproceedings{Podlubny2000MATRIXAT,
  title={MATRIX APPROACH TO DISCRETE FRACTIONAL CALCULUS},
  author={Igor Podlubny},
  year={2000}
}
"""

"""
    solve(p1, α, p2, c, h, T, MatrixDiscrete())

Using the **Matrix Discretization algorithm** proposed by [Prof Igor Podlubny](http://people.tuke.sk/igor.podlubny/index.html) to approximate the numerical solution.
"""
function solve(p1, α, p2, c, h, T, ::MatrixDiscrete)
    n=Int64(floor(α))
    rows=collect(1:n)
    N=Int64(T/h+1)
    equation = p1*B(N, α, h)+p2*(zeros(N, N)+I)
    equation = eliminator(N, rows)*equation*eliminator(N, rows)'
    righthand = c*eliminator(N, rows)*ones(N)
    result = equation\righthand
    return vcat(zeros(n), result)
end

function solve(equation, right, h, T)
    N=Int64(T/h+1)
    equation = eliminator(N, rows)*equation*eliminator(N, rows)'
    right = right*eliminator(N, rows)*ones(N)
    result = equation\right
    return vcat(zeros(n), result)
end

"""
    eliminator(n, row)

Compute the eliminator matrix Sₖ by omiting n-th row
"""
function eliminator(n, row)
    temp = zeros(n, n)+I
    return temp[Not(row), :]
end

"""
Generating elements in Matrix.
"""
function omega(n, p)
    omega = zeros(n+1)

    omega[1]=1
    for i in range(1, n, step=1)
        omega[i+1]=(1-(p+1)/i)*omega[i]
    end

    return omega

end

function B(N, p, h)
    result=zeros(N, N)
    temp=omega(N, p)

    for i in range(1, N, step=1)
        result[i, 1:i]=reverse(temp[1:i])
    end

    return h^(-p)*result
end

function F(N, p, h)
    result=zeros(N, N)
    temp = omega(N, p)

    for i in range(1, N, step=1)
        result[i, 1:i]=reverse(temp[1:i])
    end

    result=reverse(reverse(result, dims=1), dims=2)

    return (-1)^(ceil(p))*h^(-p)*result
end

"""
    bagleytorvik(p1, p2, p3, T, h)

By specifying the parameters of Bagley Torvik Equation, we can use **bagleytorvik** to directly obtain the numerical approximation.

!!! info p2 ≠ 0
    Please note that the parameter of fractional derivative item must not be 0
"""
function bagleytorvik(p1, p2, p3, T, h)
    N=Int64(T/h+1)
    equation = p1*B(N, 2, h)+p2*B(N, 1.5, h)+p3*(zeros(N, N)+I)
    equation = eliminator(N, [1,2])*equation*eliminator(N, [1,2])'
    right = eliminator(N, [1,2])*ones(N)
    result = equation\right

    return vcat(zeros(2), result)
end


module MadelungConstant

import Plots

function Madelung(m)
	madelung = 0
	for i = -m:m
		for j = -m:m
			for k = -m:m
				if i == 0 && j == 0 && k == 0
					continue
				end
				madelung += ( (i + j + k) % 2 == 0 ? -1 : 1) / √(i^2 + j^2 + k^2)
			end
		end
	end
	madelung
end

function Madelung2(m)
	madelung = 0
	for i = -m:m
		for j = -m:m
			for k = -m:m
				if i == 0 && j == 0 && k == 0
					continue
				end
				r = √(i^2 + j^2 + k^2)
				if r > m
					continue
				end
				madelung += ((i + j + k) % 2 == 0 ? -1 : 1) / r
			end
		end
	end
	madelung
end

function plot(range=1:50)
    Plots.plot([Madelung, Madelung2], range, show=true)
end

end # module


"""
    Hermite( nodes::NodesType, epsilon:Real, gamma:Real )

Hermite interpolation

"""
mutable struct Hermite <: InterpolationType

    nnodes     :: Int64
    epsilon    :: Float64
    gamma      :: Float64
    nodes      :: Array{Float64,1}
    colloc_mat :: Array{Float64,2}

    function Hermite( nodes_t::NodesType, epsilon::Float64, gamma::Float64 )

        nodes  = nodes_t.xk
        nnodes = nodes_t.nx
        colloc_mat = evaluate_hermite(nodes, nnodes, epsilon, gamma)
        new( nnodes, epsilon, gamma, nodes, colloc_mat )

    end

end

"""
    Hermite( xe )

This function returns the matrix He of the values of the HermiteGF basis
functions. The computation is done via three term recurrence for Hermite
functions with an argument gamma*x and then appropriate exponential scaling.

More details can be found in Section 5.1 of the paper

*STABLE EVALUATION OF GAUSSIAN RADIAL BASIS
 FUNCTIONS USING HERMITE POLYNOMIALS*
 by Anna Yurova and Katharina Kormann.

"""
function (interp::Hermite)( x::Array{Float64,1} )

   nx      = interp.nnodes
   epsilon = interp.epsilon
   gamma   = interp.gamma

   evaluate_hermite(x, nx, epsilon, gamma) / interp.colloc_mat

end

"""

    evaluate_hermite(xk, n, epsilon, gamma)

This function returns the matrix He of the values of the HermiteGF basis
functions. The computation is done via three term recurrence for Hermite
functions with an argument gamma*x and then appropriate exponential scaling.

More details can be found in Section 5.1 of the paper

*STABLE EVALUATION OF GAUSSIAN RADIAL BASIS
 FUNCTIONS USING HERMITE POLYNOMIALS*
 by Anna Yurova and Katharina Kormann.

This function should be used both for computation of the
collocation and evaluation matrices.

"""
function evaluate_hermite(xk, n, epsilon, gamma)

  # Initialize the result matrix
  result = zeros(Int(length(xk)), n)

  # Write the values of the first two Hermite functions to initialize the
  # three-term recurrence
  result[:,1] = exp.(-0.5.*((gamma*xk).^2))
  result[:,2] = gamma*xk.*sqrt(2).*exp.(-0.5.*((gamma*xk).^2))

  # Three term recurrence for the Hermite functions with argument gamma*x
  for i = 3:n
    result[:,i] = sqrt(2.0/(i-1)) .* (gamma*xk) .* result[:,i-1] - sqrt((i-2)/(i-1)) .* result[:,i-2]
  end

  # Scaling the Hermite functions with the exponential factor
  # exp(-epsilon^2 x^2 + (gamma x)^2/2)

  for i=1:n
      result[:,i] = (pi^(1/4)) * result[:,i] .* exp.((xk.^2) .* (gamma*gamma*0.5 - epsilon^2))
  end

  result

end


"""
    PODsvd(X)

Uses the SVD based decomposition technique to calculate the POD basis of X.
"""
function PODsvd(X; subtractmean::Bool = false)

    Xcop = deepcopy(X)
    PODsvd!(Xcop, subtractmean = subtractmean)

end

"""
    PODsvd(X,W)

Same as `PODsvd(X)` but uses weights for each data point. The weights are equal to the
cell volume for a volume mesh.
"""
function PODsvd(X,W::AbstractVector; subtractmean::Bool = false)

    Xcop = deepcopy(X)
    PODsvd!(Xcop, W, subtractmean = subtractmean)

end

"""
    PODsvd!(X)

Same as `PODsvd(X)` but overwrites memory.
"""
function PODsvd!(X; subtractmean::Bool = false)

    if subtractmean
        X .-= mean(X,dims=2)
    end

    # Economy sized SVD
    F = svd!(X)

    # Mode coefficients
    a = Diagonal(F.S)*F.Vt

    POD = PODBasis(a, F.U)

    return POD, F.S

end

"""
    PODsvd!(X,W)

Same as `PODsvd!(X)` but uses weights for each data point. The weights are equal to the
cell volume for a volume mesh.
"""
function PODsvd!(X,W::AbstractVector; subtractmean::Bool = false)

    if subtractmean
        X .-= mean(X,dims=2)
    end

    # Take into account the weights
    X = sqrt.(W).*X

    # Economy sized SVD
    F = svd!(X)

    # Mode coefficients
    a = Diagonal(F.S)*F.Vt
    Φ = 1 ./sqrt.(W).*F.U

    POD = PODBasis(a, Φ)

    return POD, F.S

end


using LabelledArrays

export ar1_bidir

"""
    eom_ar1_bidir(x, p, n) → SVector{2}

Equations of motion for a system consisting of two mutually
coupled first order autoregressive processes.

## Equations of motion

```math
\\begin{aligned}
x(t+1) &= a_{1}x + c_{yx}y + \\epsilon_{x} \\\\
y(t+1) &= b_{1}y + c_{xy}x + \\epsilon_{y}
\\end{aligned}
```

where ``\\epsilon_{x}`` and ``\\epsilon_{y}`` are drawn independently
at each time step from normal distributions with zero mean and standard
deviations `σx` and `σy`.
"""
function eom_ar1_bidir(x, p, n)
    a₁, b₁, c_xy, c_yx, σx, σy = (p...,)
    x, y = (x...,)

    ϵx = rand(Normal(0, σx))
    ϵy = rand(Normal(0, σy))

    dx = a₁*x + c_yx*y + ϵx
    dy = b₁*y + c_xy*x + ϵy
    return SVector{2}(dx, dy)
end

function ar1_bidir(u₀,a₁, b₁, c_xy, c_yx, σx, σy)
    p = @LArray [a₁, b₁, c_xy, c_yx, σx, σy] (:a₁, :b₁, :c_xy, :c_yx, :σx, :σy)
    logistic_system = DiscreteDynamicalSystem(eom_ar1_bidir, u₀, p)

    return logistic_system
end

"""
    ar1_bidir(;u₀ = rand(2), a₁ = 0.5, b₁ = 0.7, c_xy = 0, c_yx = 0.2,
        σx = 0.3, σy = 0.3) → DiscreteDynamicalSystem

A system consisting of two mutually coupled first order autoregressive processes.

## Equations of motion

```math
\\begin{aligned}
x(t+1) &= a_{1}x + c_{yx}y + \\epsilon_{x} \\\\
y(t+1) &= b_{1}y + c_{xy}x + \\epsilon_{y}
\\end{aligned}
```

where ``\\epsilon_{x}`` and ``\\epsilon_{y}`` are drawn independently
at each time step from normal distributions with zero mean and standard
deviations `σx` and `σy`.
"""
ar1_bidir(;a₁ = 0.5, b₁ = 0.7, u₀ = rand(2), c_xy = 0, c_yx = 0.2, σx = 0.3, σy = 0.3) =
    ar1_bidir(u₀, a₁, b₁, c_xy, c_yx, σx, σy)


# Unit 02 - Nonlinear Classification, Linear regression, Collaborative Filtering

# Lecture 5

# Hinge loss computation

x = [1 0 1; 1 1 1; 1 1 -1; -1 1 1]
y = [2,2.7,-0.7, 2]
θ = [0,1,2]
z = y - (θ' * x')'

hinge(z) = (z >= (1 - eps()) ) ? 0 : (1 - z)
l = mean(map(z -> hinge(z), z))

squared_error(z) = z^2 / 2
l = mean(map(z -> squared_error(z), z))

# Lecture 6

# radial basis kernel
using LinearAlgebra
radial_kernel(x,xᵖ) = exp(-1/2 * norm(x-xᵖ)^2)

x = [1,0,0]

xᵖ = [0,1,0]

k = radial_kernel(x,xᵖ)

Unit 02 - Nonlinear Classification, Linear regression, Collaborative Filtering/Unit 02 - Nonlinear Classification, Linear regression, Collaborative Filtering.jl


module LossFunctions

export mse, cross_entropy

using Statistics

"""
    mse(x, y)

Computes the mean-squared error between `x` and `y`, returning a scalar.

For the multi-sample case, takes the error along columns (dimensions), and
averages along rows (count).

- `x`: Either (1) a vector containing a single prediction, or (2) a matrix where
  rows index predictions in a batch and columns indicate dimensions of the
  prediction.
- `y`: Either (1) a vector containing a single label, or (2) a matrix where rows
  index labels in a batch and columns indicate dimensions of the label.

`x` and `y` should have the same dimensionality.
"""
mse(x, y) = mean(sum((x .- y) .^2, dims=2), dims=1)[1]

"""
    cross_entropy(x, y)

Computes the cross-entropy error between `x` and `y`, returning a scalar.

For the multi-sample case, takes the error along columns (dimensions), and
averages along rows (count).

- `x`: Either (1) a vector containing a single prediction, or (2) a matrix where
  rows index predictions in a batch and columns indicate dimensions of the
  prediction.
- `y`: Either (1) a vector containing a single label, or (2) a matrix where rows
  index labels in a batch and columns indicate dimensions of the label.

`x` and `y` should have the same dimensionality.
"""
cross_entropy(x, y) = mean((-1) * sum(y .* log.(x), dims=2), dims=1)[1]

end # module


@testset "Test linear solvers" begin
    As = @SMatrix [4..6 -1..1 -1..1 -1..1;-1..1 -6.. -4 -1..1 -1..1;-1..1 -1..1 9..11 -1..1;-1..1 -1..1 -1..1 -11.. -9]
    bs = @SVector [-2..4, 1..8, -4..10, 2..12]

    Am = Matrix(As)
    bm = Vector(bs)
    jac = Jacobi()
    gs = GaussSeidel()
    hbr = HansenBliekRohn()
    kra = LinearKrawczyk()

    for (A, b) in zip([As, Am], [bs, bm])
        xgs = solve(A, b, gs)
        xjac = solve(A, b, jac)
        xhbr = solve(A, b, hbr)
        xkra = solve(A, b, kra)

        @test all(interval_isapprox.(xgs, [-2.6..3.1, -3.9..1.65, -1.48..2.15, -2.35..0.79]; atol=0.01))
        @test all(interval_isapprox.(xjac, [-2.6..3.1, -3.9..1.65, -1.48..2.15, -2.35..0.79]; atol=0.01))

        @test all(interval_isapprox.(xhbr, [-2.5..3.1, -3.9..1.2, -1.4..2.15, -2.35..0.6]; atol=0.01))

        @test all(interval_isapprox.(xkra, [-8..8, -8..8, -8..8, -8..8]; atol=0.01))
    end

    ge = GaussianElimination()
    xge = solve(Am, bm, ge)
    @test all(interval_isapprox.(xge, [-2.6..3.1, -3.9..1.5, -1.43..2.15, -2.35..0.6]; atol=0.01))

    xdef = solve(Am, bm)
    @test all(interval_isapprox.(xdef, [-2.6..3.1, -3.9..1.5, -1.43..2.15, -2.35..0.6]; atol=0.01))

    A = [2..4 -2..1; -1..2 2..4]
    b = [-2..2, -2..2]

    x1 = solve(A, b)
    @test all(interval_isapprox.(x1, [-14..14, -14..14]))
    x2 = solve(A, b, HansenBliekRohn())
    @test all(interval_isapprox.(x2, [-14..14, -14..14]))

    # test exceptions
    @test_throws DimensionMismatch solve(Am, bm[1:end-1])
    @test_throws DimensionMismatch solve(Am[:, 1:end-1], bm, hbr)
    @test_throws DimensionMismatch solve(Am, bm, gs, NoPrecondition(), [1..2, 3..4])
end

@testset "Reduced Row Echelon Form" begin
    A1 = [1..2 1..2;2..2 3..3]
    @test rref(A1) == [2..2 3..3; 0..0 -2..0.5]

    A2 = fill(0..0, 2, 2)
    @test_throws ArgumentError rref(A2)
end


# NEED:
# A Julia JIT Compiler
# USAGE:
# type "python pi.py [iterations]" into terminal/prompt

# needed variables to approximate pi
iterations = 10000
total = 0.0
in_circle = 0.0
radius = 1.0

# get user input, otherwise use default value
if length(ARGS) > 0
    iterations = parse(ARGS[1])
else
    println("No iteration limit specified --> used default: ", iterations)
end

# initialize random 2 dimensional array and square each value for later calculation quadratic
points = (rand!(zeros(iterations,2)) .* radius) .^ 2

# is point in circle/ out of circle
for i = 1:iterations
    total += 1
    distance = sqrt(points[i,1] + points[i,2])
    if distance < radius
        in_circle += 1
    end
end

# pi approximation
pi = 4 * (in_circle / total)

# user output
println("Iterations: ", iterations)
println("PI-Approximation: ", pi)


function fibonacci_sum(limit)
    sequence = [1, 2]; total = 0
    while sequence[end] <= limit
        if iseven(sequence[end])
            total += sequence[end]
        end
        push!(sequence, sequence[end] + sequence[end-1])
    end
    println(total)
end

fibonacci_sum(4000000)


@doc raw"""
```
DiscreteShape(iterations::Int = 2000, smoothness::Int = 1000, albedo::Real = Inf, illumination_direction::Vector{T} where T <: Real = [Inf, Inf, Inf])
```
Contins `DiscreteShape` algorithm which ttempts to produce a heightmap from a
grayscale image by minimization of a set of Euler-Lagrange equations. This is done
discretely at each point in the image utilizing the second derivates of the
surface normals and their Fourier Transforms.
# Output
Returns a `DiscreteShape` functor which can be run on an image to attempt reconstruction
of the surface contianed in the image.
# Details
The algorithm attempts to minimise the brightness deviation ``ϵ`` through the
brightness constraint ``\epsilon_1`` and smoothness constraint ``\epsilon_2``
as defined bellow:
```math
\epsilon=\iint((\epsilon_1+\lambda\epsilon_2))dxdy=\iint((E(x,y)-R(p,q))^2+
\lambda(p_x^2+q_x^2+p_y^2+q_y^2))
```
This is minimised using the Euler-Lagrange equations defined as;
```math
\dfrac{\delta\epsilon}{\delta p}-\dfrac{\delta}{\delta x}\dfrac{\delta\epsilon}{
\delta p_x}-\dfrac{\delta}{\delta y}\dfrac{\delta\epsilon}{\delta p_y}=0
```
and
```math
\dfrac{\delta
\epsilon}{\delta q}-\dfrac{\delta}{\delta x}\dfrac{\delta\epsilon}{\delta q_x}-
\dfrac{\delta}{\delta y}\dfrac{\delta\epsilon}{\delta q_y}=0
```
which become:
```math
\begin{gathered}
-2(E-R)\dfrac{\delta R}{\delta p}-2\lambda p_{xx}-2\lambda p_{yy}=0\\
-2(E-R)\dfrac{\delta R}{\delta p}-2\lambda q_{xx}-2\lambda q_{yy}=0
\end{gathered}
```
which can be further simplified to give:
```math
\begin{gathered}
\nabla^2p=\dfrac{1}{\lambda}(R-E)\dfrac{\delta R}{\delta p}\\
\nabla^2q=\dfrac{1}{\lambda}(R-E)\dfrac{\delta R}{\delta q}
\end{gathered}
```
where ``\nabla^2p=p_{xx}+p_{yy}`` and ``\nabla^2q=q_{xx}+q_{yy}`` are Laplacians
of p and q.

However, for computation we are dealing with a discreate case of these equations
which can be defined as below:
```math
\begin{gathered}
p_{i,j}=\bar{p}_{i,j}+\dfrac{1}{4\lambda}(E-R)\dfrac{\delta R}{\delta p}\\q_{i,j}=
\bar{q}_{i,j}+\dfrac{1}{4\lambda}(E-R)\dfrac{\delta R}{\delta q}\\
\end{gathered}
```
where
```math
\bar{p}_{i,j}=\dfrac{p_{i+1,j}+p_{i-1,j}+p_{i,j+1}+p_{i,j-1}}{4}
```
and
```math
\bar{q}_{i,j}=\dfrac{q_{i+1,j}+q_{i-1,j}+q_{i,j+1}+q_{i,j-1}}{4}
```
Finally, the algorithm needs to enforce integrability on p and q and retrieve
the surface Z. This can be done by taking the Fast Fourier Transform of p and q
()``c_p(\omega_x,\omega_y)`` and ``c_q(\omega_x,\omega_y)``) and then using the
Inverse Fast Fourier Transform to recover Z and update p and q as per bellow:
```math
\begin{gathered}
p=\sum c_p(\omega_x,\omega_y)e^{j(\omega_xx+\omega_yy)}\\
q=\sum c_q(\omega_x,\omega_y)e^{j(\omega_xx+\omega_yy)}\\Z=\sum c(\omega_x,\omega_y)
e^{j(\omega_xx+\omega_yy)}\\
\end{gathered}
```
where
```math
c(\omega_x,\omega_y)=\dfrac{-j(\omega_xc_p(\omega_x,\omega_y)+\omega_yc_q(\omega_x
,\omega_y))}{\omega_x^2+\omega_y^2}
```
# Arguments
The arguments are described in more detail below.
## `albedo`
A `Real` that specifies the albedo (amount of light reflected) of the image. If
`albedo` is specified to must the `illumination_direction`. Defults to `Inf` which
will trigger the algorithm to run `estimate_img_properties`.
## `illumination_direction`
A `Vector{T} where T <: Real` that specifies the tilt value to be used by the
algorithm. The `illumination_direction` should be a vector of the form [x,y,z]
where x,y,z are int he range [0,1]. If `illumination_direction` is specified
to must the `albedo`. Defults to `[Inf, Inf, Inf]` which
will trigger the algorithm to run `estimate_img_properties`.
## `iterations`
An `Int` that specifies the number of iterations the algorithm is to perform. If
left unspecified a default value of 2000 is used.
## `smoothness`
An `Int` that specifies the strength of the smoothness constraint in the minimised
function. Defults to `1000`.
!!! note
    If `albedo` and `illumination_direction` are not defined (i.e. have defulted
    to `Inf`) they will be calculated at runtime using `estimate_img_properties`.
# Example
Compute the heightmap for a synthetic image generated by `generate_surface`.
```julia
using Images, Makie, ShapeFromShading

#generate synthetic image
img = generate_surface(SynthSphere(50), 1.0, [0.2,0,0.9])

#calculate the heightmap (using 500 iterations)
discreteShape = DiscreteShape(iterations = 500, albedo = 1.0, illumination_direction = [0.2,0,0.9])
Z,p,q = discreteShape(img)

#normalize to maximum of 1 (not necessary but makes displaying easier)
Z = Z./maximum(Z)

#display using Makie (Note: Makie can often take several minutes first time)
r = 0.0:0.1:2
surface(r, r, Z)
```
# Reference
1. S. Elhabian, "Hands on Shape from Shading", Computer Vision and Image Processing, 2008.
"""
function (algorithm::DiscreteShape)(img::AbstractArray)
    ρ = algorithm.albedo
    I = algorithm.illumination_direction
    smoothness = algorithm.smoothness
    iterations = algorithm.iterations

    if ρ == Inf || I == [Inf, Inf, Inf]
        ρ,I,σ,τ = estimate_img_properties(img)
    end

    E = Array{Float64}(img)

    #initialize variables
    p = zeros(Complex{Float64},axes(E))
    q = zeros(Complex{Float64},axes(E))
    R = zeros(Complex{Float64},axes(E))
    Z = zeros(axes(E))
    return solve_EulerLagrange(ρ, I, iterations, p, q, R, smoothness, E, Z)
end


# ------------------------------------------------------------------
# Licensed under the MIT License. See LICENSE in the project root.
# ------------------------------------------------------------------

"""
    ZScore()

Applies the z-score transform (a.k.a. normal score) to all columns of the table.
The z-score transform of the column `x`, with mean `μ` and standard deviation `σ`,
is defined by `(x .- μ) ./ σ`.
"""
struct ZScore <: Colwise end

assertions(::Type{ZScore}) = [assert_continuous]

isrevertible(::Type{ZScore}) = true

function colcache(::ZScore, x)
  μ = mean(x)
  σ = std(x, mean=μ)
  (μ=μ, σ=σ)
end

colapply(::ZScore, x, c)  = @. (x - c.μ) / c.σ

colrevert(::ZScore, y, c) = @. c.σ * y + c.μ


# Rational Example

struct RationalNumber
    n::Integer
    d::Integer
    function RationalNumber(n, d)
        (n, d) = _normalize(n, d)
        return new(n, d)
    end
end

function _normalize(n::Integer, d::Integer)
    g = gcd(n, d)
    m = d < 0 ? -1 : 1
    (m * n ÷ g, m * d ÷ g)
end

Base.:+(r1::RationalNumber, r2::RationalNumber) = RationalNumber(r1.n+r2.n, r1.d+r2.d)
Base.:-(r1::RationalNumber, r2::RationalNumber) = RationalNumber(r1.n-r2.n, r1.d-r2.d)
Base.:*(r1::RationalNumber, r2::RationalNumber) = RationalNumber(r1.n*r2.n, r1.d*r2.d)
Base.:/(r1::RationalNumber, r2::RationalNumber) = RationalNumber(r1.n*r2.d, r1.d*r2.n)

str(r::RationalNumber) = "$(r.n)/$(r.d)"

# Demonstration

x = half = RationalNumber(5, 10)
println("One half can be expressed as $(half.n)/$(half.d)")

y = two_thirds = RationalNumber(2,3)

ystr = str(y)
println("two thirds is $(ystr)")
println("two thirds is $(str(y))")

println("x+y = $(str(x+y))")
println("x-y = $(str(x-y))")
println("x*y = $(str(x*y))")
println("x/y = $(str(x/y))")


function lglnodes(N,a,b)

# %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
# %
# % lglnodes.m
# %
# % Computes the Legendre-Gauss-Lobatto nodes, weights and the LGL Vandermonde
# % matrix. The LGL nodes are the zeros of (1-x^2)*P'_N(x). Useful for numerical
# % integration and spectral methods.
# %
# % Reference on LGL nodes and weights:
# %   C. Canuto, M. Y. Hussaini, A. Quarteroni, T. A. Tang, "Spectral Methods
# %   in Fluid Dynamics," Section 2.3. Springer-Verlag 1987
# %
# % Written by Greg von Winckel - 04/17/2004
# % Contact: gregvw@chtm.unm.edu
# %
# %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#
# % Truncation + 1
	N1=N+1;

	#% Use the Chebyshev-Gauss-Lobatto nodes as the first guess
	x=cos(pi*[0:N]/N);

	# The Legendre Vandermonde Matrix
	P=zeros(N1,N1);

	# 	% Compute P_(N) using the recursion relation
	# 	% Compute its first and second derivatives and
	# 	% update x using the Newton-Raphson method.

	xold=2;

	while maximum(abs(x-xold))>eps()

	    xold=x;

	    P[:,1]=1;    P[:,2]=x;

	    for k=2:N
		P[:,k+1]=( (2*k-1)*x.*P[:,k]-(k-1)*P[:,k-1] )/k;
	    end

	    x=xold-( x.*P[:,N1]-P[:,N] )./( N1*P[:,N1] );

	end

	w=2./(N*N1*P[:,N1].^2)/2.*(b-a);
	x = (x+1)*(b-a)/2.+a
	return x,w,P
end


"""
        critic(decisionMat, fns)

Apply CRITIC (Combined Compromise Solution) method for a given matrix and criteria types.

# Arguments:
 - `decisionMat::DataFrame`: n × m matrix of objective values for n alternatives and m criteria
 - `fns::Array{Function, 1}`: m-vector of functions to be applied on the columns.

# Description
critic() applies the CRITIC method to rank n alterntives subject to m criteria which are supposed to be
either maximized or minimized.

# Output
- `::CRITICResult`: CRITICResult object that holds multiple outputs including weighting and best index.

# Examples
```julia-repl

julia> decmat
3×4 Array{Float64,2}:
 12.9918  0.7264  -1.1009  1.59814
  4.1201  5.8824   3.4483  1.02156
  4.1039  0.0     -0.5076  0.984469

julia> df = makeDecisionMatrix(decmat)

3×4 DataFrame
 Row │ Crt1     Crt2     Crt3     Crt4
     │ Float64  Float64  Float64  Float64
─────┼─────────────────────────────────────
   1 │ 12.9918   0.7264  -1.1009  1.59814
   2 │  4.1201   5.8824   3.4483  1.02156
   3 │  4.1039   0.0     -0.5076  0.984469

julia> fns = [maximum, maximum, minimum, maximum];

julia> result = critic(df, fns);

julia> result.w
4-element Array{Float64,1}:
 0.16883905506169491
 0.41844653698732126
 0.24912338769165807
 0.16359102025932576

julia> result.bestIndex
2
```
# References

Diakoulaki, D., Mavrotas, G., & Papayannakis, L. (1995). Determining objective weights in multiple criteria problems: The critic method. Computers & Operations Research, 22(7), 763–770. doi:10.1016/0305-0548(94)00059-h
Akçakanat, Ö., Aksoy, E., Teker, T. (2018). CRITIC ve MDL Temelli EDAS Yöntemi ile TR-61 Bölgesi Bankalarının Performans Değerlendirmesi. Süleyman Demirel Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 1 (32), 1-24.

"""
function critic(decisionMat::DataFrame, fns::Array{Function,1})::CRITICResult

    row, col = size(decisionMat)
    colMax = colmaxs(decisionMat)
    colMin = colmins(decisionMat)

    A = similar(decisionMat)

    for i in 1:row
        for j in 1:col
            if fns[j] == maximum
                @inbounds A[i, j] = (decisionMat[i, j] - colMin[j]) / (colMax[j] - colMin[j])
            elseif fns[j] == minimum
                @inbounds A[i, j] = (colMax[j] - decisionMat[i, j]) / (colMax[j] - colMin[j])
            end
        end
    end

    # normalizedMat = convert(Matrix, A)
    normalizedMat = Matrix(A)

    corMat = 1 .- cor(normalizedMat)

    scores = zeros(Float64, col)
    for i in 1:col
        scores[i] = sum(corMat[:, i]) .* std(normalizedMat[:, i])
    end

    w = zeros(Float64, col)

    for i in 1:col
        w[i] = scores[i] ./ sum(scores)
    end

    rankings = sortperm(w)

    bestIndex = rankings |> last

    result = CRITICResult(
        decisionMat,
        w,
        rankings,
        bestIndex
    )

    return result
end

"""
        critic(setting)

Apply CRITIC (Combined Compromise Solution) method for a given matrix and criteria types.

# Arguments:
 - `setting::MCDMSetting`: MCDMSetting object.

# Description
critic() applies the CRITIC method to rank n alterntives subject to m criteria which are supposed to be
either maximized or minimized.

# Output
- `::CRITICResult`: CRITICResult object that holds multiple outputs including weighting and best index.
"""
function critic(setting::MCDMSetting)::CRITICResult
    critic(
        setting.df,
        setting.fns
    )
end


"""
    smoothstep(x, x_step, dx, a=zero(x), b=one(x))

smooth step function:
- = a for x <= x_step - dx
- = b for x >= x_step + dx
- = (a-b)/2 for x == x_step
and smooth in between.
"""
function smoothstep(x, x_step, dx, a=zero(x), b=one(x))
    @assert dx > 0
    edge0 = x_step - dx
    x <= edge0 && return(a)
    edge1 = x_step + dx
    x >= edge1 && return(b)
    y0 = _smoothstep(x,edge0,edge1)
    y0 * (b-a) + a
end

function _smoothstep(x, edge0, edge1)
    #https://en.wikipedia.org/wiki/Smoothstep
    # Scale, bias and saturate x to 0..1 range
    # assume that
    @assert edge0 < x < edge1
    #x = clamp((x - edge0) / (edge1 - edge0), 0.0, 1.0);
    xs = (x - edge0) / (edge1 - edge0)
    # Evaluate polynomial
    xs * xs * (3 - 2 * xs)
  end


function volume(ϵ::Ellipsoid)
    n = size(ϵ.H, 1)
    return (n^(n / 2) * _unit_ball_volume(n)) / sqrt(det(ϵ.H))
end

function _unit_ball_volume(n::Integer)
    return π^(n / 2) / gamma(n / 2 + 1)
end


using ForwardDiff, BenchmarkTools, StaticArrays

# Using StaticArrays #
#--------------------#

@inline eom(u) = @SVector [10.0(u[2]-u[1]), u[1]*(28.0-u[3]) - u[2], u[1]*u[2] - (8/3)*u[3]]

@inline function manual_jacobian(u)
    i = one(eltype(u))
    return @SMatrix [       -10i    10i     0i;
                     (28i - u[3])    -i  -u[1];
                             u[2]  u[1]  -8i/3]
end

@inline forward_jacobian(u) = ForwardDiff.jacobian(eom, u)

u = SVector{3}(rand(3))

# sanity check
@assert manual_jacobian(u) == forward_jacobian(u)

println("---- eom(::SVector) benchmark results ----")
display(@benchmark eom($u))

println("---- manual_jacobian(::SVector) benchmark results ----")
display(@benchmark manual_jacobian($u))

println("---- forward_jacobian(::SVector) benchmark results ----")
display(@benchmark forward_jacobian($u))

# Using Base Arrays #
#-------------------#

@inline function eom!(du, u)
    du[1] = 10.0(u[2]-u[1])
    du[2] = u[1]*(28.0-u[3]) - u[2]
    du[3] = u[1]*u[2] - (8/3)*u[3]
end

@inline function manual_jacobian!(J, u)
    i = one(eltype(J))
    J[1,1] = -10i
    J[1,2] = 10i
    J[1,3] = 0i
    J[2,1] = 28i - u[3]
    J[2,2] = -i
    J[2,3] = -u[1]
    J[3,1] = u[2]
    J[3,2] = u[1]
    J[3,3] = -8i/3
    return J
end

@inline forward_jacobian!(J, du, u, cfg) = ForwardDiff.jacobian!(J, eom!, du, u, cfg)

u = rand(3)
du = zeros(3)
J = zeros(3, 3)
cfg = ForwardDiff.JacobianConfig(eom!, du, u)

# sanity check
@assert manual_jacobian!(J, u) == forward_jacobian!(J, du, u, cfg)

println("---- eom!(::Vector, ::Vector) benchmark results ----")
display(@benchmark eom!($du, $u))

println("---- manual_jacobian!(::Matrix, ::Vector) benchmark results ----")
display(@benchmark manual_jacobian!($J, $u))

println("---- forward_jacobian!(::Matrix, ::Vector, ::Vector, ::JacobianConfig) benchmark results ----")
display(@benchmark forward_jacobian!($J, $du, $u, $cfg))


using NLopt
using Base.Test

count = 0 # keep track of # function evaluations

function myfunc(x::Vector, grad::Vector)
    if length(grad) > 0
        grad[1] = 0
        grad[2] = 0.5/sqrt(x[2])
    end

    global count
    count::Int += 1
    println("f_$count($x)")

    sqrt(x[2])
end

function myconstraint(x::Vector, grad::Vector, a, b)
    if length(grad) > 0
        grad[1] = 3a * (a*x[1] + b)^2
        grad[2] = -1
    end
    (a*x[1] + b)^3 - x[2]
end

opt = Opt(:LD_MMA, 2)
lower_bounds!(opt, [-Inf, 0.])
xtol_rel!(opt,1e-4)

min_objective!(opt, myfunc)
inequality_constraint!(opt, (x,g) -> myconstraint(x,g,2,0), 1e-8)
inequality_constraint!(opt, (x,g) -> myconstraint(x,g,-1,1), 1e-8)

(minf,minx,ret) = optimize(opt, [1.234, 5.678])
println("got $minf at $minx after $count iterations (returned $ret)")

@test_approx_eq_eps minx[1] 1/3 1e-5
@test_approx_eq_eps minx[2] 8/27 1e-5
@test_approx_eq_eps minf sqrt(8/27) 1e-5
@test ret == :XTOL_REACHED


"""
    struct CartesianSpace <: EuclideanSpace{ℝ}

A real euclidean space `ℝ^d`, which is therefore self-dual. `CartesianSpace` has no
additonal structure and is completely characterised by its dimension `d`. This is the
vector space that is implicitly assumed in most of matrix algebra.
"""
struct CartesianSpace <: EuclideanSpace{ℝ}
    d::Int
end
CartesianSpace(V::CartesianSpace) = V

Base.getindex(::RealNumbers) = CartesianSpace
Base.getindex(::RealNumbers, d::Int) = CartesianSpace(d)
Base.:^(::RealNumbers, d::Int) = CartesianSpace(d)

# Corresponding methods:
#------------------------
dim(V::CartesianSpace) = V.d
Base.axes(V::CartesianSpace) = Base.OneTo(dim(V))

Base.oneunit(::Type{CartesianSpace}) = CartesianSpace(1)
⊕(V1::CartesianSpace, V2::CartesianSpace) = CartesianSpace(V1.d+V2.d)
fuse(V1::CartesianSpace, V2::CartesianSpace) = CartesianSpace(V1.d*V2.d)
flip(V::CartesianSpace) = V

Base.min(V1::CartesianSpace, V2::CartesianSpace) = CartesianSpace(min(V1.d, V2.d))
Base.max(V1::CartesianSpace, V2::CartesianSpace) = CartesianSpace(max(V1.d, V2.d))

Base.show(io::IO, V::CartesianSpace) = print(io, "ℝ^$(V.d)")
Base.show(io::IO, ::Type{CartesianSpace}) = print(io, "CartesianSpace")


"""
    barycentric_weights{T<:Real}(x::AbstractVector{T})

Compute the barycentric weights corresponding to the nodes `x`.
[Kopriva, Implementing Spectral Methods for PDEs, Algorithm 30].
"""
function barycentric_weights(x::AbstractVector)
    w = fill!(similar(x), one(eltype(x)))
    @inbounds barycentric_weights!(w, x)
    w
end

function barycentric_weights!(w, x)
    @boundscheck begin
        length(w) == length(x)
    end
    fill!(w, 1)
    @inbounds for j in 2:length(w), k in 1:j-1
        diff = x[k] - x[j]
        w[k] *= diff
        w[j] *= -diff
    end
    @inbounds for j in 1:length(w)
        w[j] = 1 / w[j]
    end
    nothing
end

"""
    interpolate(x::Real, nodes, values, baryweights)

Interpolate the function represented by `values` on the `nodes` using the
corresponding barycentric weights `baryweights`.
[Kopriva, Implementing Spectral Methods for PDEs, Algorithm 31].
"""
function interpolate(x::Real, values, nodes, baryweights)
    @boundscheck begin
        @assert size(values) == size(nodes) == size(baryweights)
    end
    num = zero(first(baryweights)*first(values)/first(nodes)*one(x))
    den = zero(first(baryweights)/first(nodes)*one(x))
    @inbounds for idx in eachindex(nodes)
        xval = nodes[idx]
        if xval ≈ x
            return values[idx]
        end
        t = baryweights[idx] / (x - xval)
        num += t*values[idx]
        den += t
    end
    num / den
end

function interpolate(x::AbstractVector, values, nodes, baryweights)
    ret = similar(x)
    interpolate!(ret, x, values, nodes, baryweights)
    ret
end

function interpolate!(ret::AbstractVector, x::AbstractVector, values, nodes, baryweights)
    @boundscheck begin
        @assert size(ret) == size(x)
        @assert size(values) == size(nodes) == size(baryweights)
    end
    @inbounds for idx in eachindex(ret)
        ret[idx] = interpolate(x[idx], values, nodes, baryweights)
    end
    nothing
end

function interpolate(x, values, basis::NodalBasis)
    @unpack nodes, baryweights = basis
    interpolate(x, values, nodes, baryweights)
end

function interpolate!(ret, x, values, basis::NodalBasis)
    @unpack nodes, baryweights = basis
    interpolate!(ret, x, values, nodes, baryweights)
end

"""
    interpolation_matrix(destination, source, baryweights)

Compute the matrix performing interpolation from `src` to `dest`, where
`baryweights` are the barycentric weights corresponding to `src`.
[Kopriva, Implementing Spectral Methods for PDEs, Algorithm 32].
"""
function interpolation_matrix(dest, src, baryweights)
    @boundscheck begin
        @assert length(src) == length(baryweights)
    end
    T = promote_type(eltype(dest), eltype(src), eltype(baryweights))
    mat = Array{T}(undef, length(dest), length(src))
    @inbounds interpolation_matrix!(mat, dest, src, baryweights)
    mat
end

function interpolation_matrix!(mat, dest, src, baryweights)
    @boundscheck begin
        @assert length(src) == length(baryweights)
        @assert size(mat,1) == length(dest)
        @assert size(mat,2) == length(src)
    end
    fill!(mat, zero(eltype(mat)))
    @inbounds for k in 1:size(mat,1)
        row_has_match = false
        for j in 1:size(mat,2)
            if dest[k] ≈ src[j]
                row_has_match = true
                mat[k,j] = 1
            end
        end
        if row_has_match == false
            s = zero(eltype(mat))
            for j in 1:size(mat,2)
                t = baryweights[j] / (dest[k] - src[j])
                mat[k,j] = t
                s = s + t
            end
            for j in 1:size(mat,2)
                mat[k,j] /= s
            end
        end
    end
    nothing
end

# special methods of interpolation_matrix for SymPy and SymEngine are in __init__

function interpolation_matrix(dest, basis::NodalBasis)
    @unpack nodes, baryweights = basis
    interpolation_matrix(dest, nodes, baryweights)
end

function interpolation_matrix!(mat, dest, basis::NodalBasis)
    @unpack nodes, baryweights = basis
    interpolation_matrix!(mat, dest, nodes, baryweights)
end


# assert file to test polynomial implementation
using Base.Test
using Polynomial

pNULL = Poly(Float32[])
p0 = Poly([0])
p1 = Poly([0,0,0,0,0,0,0,0,0,0,0,0,0,1])
p2 = Poly([0,0,1,1])
p3 = Poly([0,0,0,0,1,2,1])
p4 = Poly([0,1,3,3,1])
p5 = Poly([0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,6,4,1])
pN = Poly([0,24,15,87,3,276])
pR = Poly([1//1, -2//1, 3//4])
p1000 = Poly(randn(1000))

@test length(pNULL) == 0
@test length(p1000) == 1000
sprint(show, p1000)
sprint(show, pNULL)

@test p3 == Poly([1,2,1])
@test pN*10 == Poly([240, 150, 870, 30, 2760])
@test pN/10 == Poly([2.4, 1.5, 8.7, 0.3, 27.6])
@test 10*pNULL + pN == pN
@test 10*p0 + pN == pN
@test p5 + 2*p1 == Poly([1,4,6,4,3])
@test 10*pNULL - pN == -pN
@test p0 - pN == -pN
@test p5 - 2*p1 == Poly([1,4,6,4,-1])
@test p2*p2*p2 == p4
@test p2^4 == p5
@test pNULL^3 == pNULL
@test pNULL*pNULL == pNULL

@test polyval(pN, -.125) == 276.9609375
@test polyval(pNULL, 10) == 0
@test polyval(p0, -10) == 0
@test polyval(poly([1//2, 3//2]), 1//2) == 0//1
@test polyder(polyint(pN)) == pN
@test polyder(pR) == Poly([2//1, -2//1])
@test polyint(pNULL,1) == p1
@test polyint(Poly(Rational[3, 2, 1])) == Poly(Rational[1, 1, 1, 0])
@test polyder(p3) == Poly([2,2])
@test polyder(p1) == polyder(p0) == polyder(pNULL) == pNULL

@test poly([-1,-1]) == p3
@test roots(p0)==roots(p1)==roots(pNULL)==[]
@test roots(p2) == [-1]
a_roots = copy(pN.a)
@test all(abs(sort(roots(poly(a_roots))) - sort(a_roots)) .< 1e6)
@test length(roots(p5)) == 4
@test roots(pNULL) == []
@test roots(pR) == [1//2, 3//2]

@test pNULL + 2 == p0 + 2 == 2 + p0 == Poly([2])
@test p2 - 2 == -2 + p2 == Poly([1,-1])
@test 2 - p2 == Poly([-1,1])

p0 = Poly([0])
p1 = Poly([1])
p2 = Poly([4, 2, -3, 6, 5])
p3 = Poly([6, 2, -3, 7])
p4 = p2 * p3
@test divrem(p4, p2) == (p3, zero(p3))
@test p3%p2 == p3
@test all((abs((p2/p3 - Poly([2/3,1/9])).a)) .< eps())
@test divrem(p0,p1) == (p0,p0)
@test divrem(p1,p1) == (p1,p0)
@test divrem(p2,p2) == (p1,p0)
@test divrem(pR, pR) == (one(pR), zero(pR))
@test_throws p1/p0
@test_throws divrem(p0,p0)

@test_throws gcd(p1, p2)
p1 = Poly([0,0,1.0,1.0])
p2 = Poly([0,0,0,1.0,2.0,1.0])
@test gcd(p1, p2) == p1
@test gcd(p2, p1) == p1

#Tests for multivariable support
pX = Poly([1, 2, 3, 4, 5])
pS1 = Poly([1, 2, 3, 4, 5], "s")
pS2 = Poly([1, 2, 3, 4, 5], 's')
pS3 = Poly([1, 2, 3, 4, 5], :s)
@test pX != pS1
@test pS1 == pS2
@test pS1 == pS3
@test_throws pS1 + pX
@test_throws pS1 - pX
@test_throws pS1 * pX
@test_throws pS1 / pX
@test_throws pS1 % pX