### A Pluto.jl notebook ###
# v0.17.4

using Markdown
using InteractiveUtils

# ╔═╡ 0810c98f-2526-4ab4-9cd5-a662434e332d
begin
	using Statistics
	using StatsBase
	using Plots
end

# ╔═╡ a4283cac-c253-11ec-1d76-495287bbab72
md"""
# Bootstrap Method Notes

I'm working through Lorna Yen's article [An Introduction to the Bootstrap Method](https://towardsdatascience.com/an-introduction-to-the-bootstrap-method-58bcb51b4d60). The Bootstrap Method was introduced by Bradley Efron in 1979.

Date: 2022 April 22

## The Bootstrap Sampling Method

We need to make an inference about an estimate (e.g. sample mean) for a population parameter $$\theta$$ (e.g. population mean).

Basic Steps in a bootstrap method:

1. Start with a sample of size $$n$$.
2. Create $$B$$ Bootstrap Samples of size $$n$$ by drawing from the original sample with replacement.
3. Evaluate the statistic $$\theta$$ on each sample, giving us $$B$$ estimates.
4. Constrtuct a sampling distribution with these bootstrap statistics and use it to make further statistical inference, such as estimating the standard error for $$\theta$$ or obtaining a confidence interval for $$\theta$$.
"""

# ╔═╡ c33fde47-ac17-4194-973b-cdb277907b60
md"""
## Motivation - The Esimator's Standard Error

How can we evaluate the accuracy of an estimator?

Example: We want to esimate the number of times your 100 student pick up their phones in a day. 30 students installed an app which tells you how many times they picked up their phones. We will calculate the mean of these 30 students and get an estimate for pickups.
"""

# ╔═╡ e42b8dfb-1a75-448c-acc1-3255c87a0edf
pickups = rand(0:500, 100)

# ╔═╡ 0e14ff27-5d05-4367-947e-d97e96ddfa8e
md"""

The population mean is **$(mean(pickups))** pickups. Now, let's get our sample of 30 students. Note that this is performed without replacement.

"""

# ╔═╡ fbbc6413-0afb-4531-8113-bb85afd2710d
responses = sample(pickups, 30)

# ╔═╡ 0ed657f2-8f2b-4e8f-a375-9c8986448340
md"""

Our sample mean is **$(round(mean(responses); digits=2))** with standard deviation **$(round(std(responses); digits=2))**.

This is called a point estimate, but what we want is the true number of pickups in the whole lab.

An **estimator** is a rule for calculating an estimate. In this case, we are using sample mean, denoted $$\bar{X}$$. A **population parameter** is a numeric summary about a population. In this case, this is the average time of phone pickups per day, denoted $$\mu$$.

But how accurate is the estimate result? (How far is $$\bar{X}$$ from $$\mu$$?) The common measure is the standard error.
"""

# ╔═╡ c30c49a1-2821-4ada-ab10-5cac70a99e87
md"""
## Standard Error

**The standard error of an estimator is it's standard deviation.** Often times we do not know the parameters, and must use an estimated standard error.

### Standard Error of Sample Mean

$$\sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}}$$

where $$\sigma$$ is the standard deviation of the population and $$n$$ is the size of the sample. However, we do not know the standard devation of the population. Thus, we use estimated standard error, where we use the sample standard deviation $$S$$ in place of $$\sigma$$.

$$\sigma_{\bar{x}} \approx \frac{S}{\sqrt{n}}$$

where

$$S = \sum_{i=1}^n\frac{(x_i-\bar{x})^2}{n-1}$$

This means that for us, **$$S=$$ $(round(std(responses);digits=2))** and **$$\sigma_{\bar{x}}\approx$$ $(round(std(responses) / sqrt(length(responses));digits=2))**.

With the assumption that the estimator as a normal distribution, then we expect our estimate to be within $$\bar{x}\pm\sigma_{\bar{x}}$$ 68% of the time and within $$\bar{x}\pm2\sigma_{\bar{x}}$$ 95% of the time.
"""

# ╔═╡ 520524ab-53ef-4edb-bc58-f6a86620a126
md"""
### The Real World

Assumptions we have used so far:

 - We can easily estimate the standard error of our statistic of interest.
 - We assume the estimator follows a normal distribution.

But in the real world:

 - It is hard to know information about the population or it's distribution.
 - The standard error is hard to evaluate standard error in general. For instance, the standard error of the median is hard to know.

The bootstrap method helps us in these cases.
"""

# ╔═╡ fda6b303-a082-4f44-8341-49446b100609
md"""
## The Bootstrap Method Explained

### Scenario

Assume want to estimate the standard error of a statistic to make an inference about a population parameter, such as for construction the corresponding confidence interval. Assume further that:

 - We don't know anything about the population.
 - There is no precise formula for estimating the standard error of the statistic.

Let $$X_1, X_2, \dots, X_n$$ be a random sample from a population $$P$$ with distribution function $$F$$ Let $$M=g(X_1, X_2, \dots, X_n)$$ be our statistic for the parameter of interest. We want to know the variance of $$M$$,denoted $$Var(M)$$.

Challenges:

 - We can't determine the variance of the population since we don't know it. Thus, we need to estimate the variance and estimate the standard error.
 - In the real world, we don't always have a simple formula for evaluting the estimated variance or than sample means.

Before we get to esimating the variance, we'll introduce a common practical way, through simulation, assuming we know $$P$$.
"""

# ╔═╡ 4453da4c-12fc-427d-9f9c-89476bf65541
md"""
### Simulation

If we know $$P$$, we can generate $$B$$ random samples of size $$n$$, and calculate the statistic on each. Using these $$B$$ measurements, we can get a mean and a variance. The law of large numbers tells us the sample mean and variance of these $$B$$ statistics converges to the true variance as more samples are taken.

Below, we will use $$B=100000$$ for the phone problem above. First we'll look at  the distribution of the sample means, and then we will look at what the average mean converges to.
"""

# ╔═╡ f9153c34-d26b-4e54-847e-c3bae8cdb4df
sims = [mean(sample(pickups, 30; replace=false)) for x in 1:100000]

# ╔═╡ d273d1b8-e66c-4461-ab55-b2a4fe578d0c
histogram(sims, fillcolor=:black, plot_title="Histogram of Sample Means", legend = false)

# ╔═╡ 14b9e9b0-d67b-4376-80a5-cb0b76602174
convergence = [mean(sims[1:i]) for i in 1:100000]

# ╔═╡ ceb8a078-4c55-40e7-a1bf-41bc357eaf86
plot(convergence,
	linecolor=:black,
	plottitle="Convergence of Sample Means",
	xlabel="Number of Simulations",
	ylabel="Mean of first N Simulations",
	legend=false)

# ╔═╡ 3c1f21fb-be76-446d-b27f-54119ee4835a
md"""
In the distribution chart, we see that the means are normally distributed, and centered just below 250.

In the bottom chart, after running 1000000 experiments, we see that the mean quickly approaches a stable value. The final value is $(round(convergence[100000]; digits=2)) which is very close to the true value of $(round(mean(pickups);digits=2)).
"""

# ╔═╡ 1e05bcbc-cf0a-47ba-a1be-cb2e9bafff78
md"""
### The Empirical Distribution Function and Plug-in Principle

But we can't use simulation because we don't know $$P$$.

We will use the empirical distribution function to approximate the distribution function of the population and will apply the plug-in principle to get an estimate for Var(M), which is the plug-in estimator.

#### The Empirical Distribution Function

We'll build a CDF from data. The EDF is a discrete distribution that weights each point equally. It forms a step function that increases by $$1/n$$ at each data point. (The data points are placed on the x-axis and the y-value incrases by $$1/n$$ at each.)

#### The Statistical Functional

The bootstram method uses the EDF as an estimator for the CDF of the population. However, to apply the EDF as an estimator for our statistic $$M$$, we need to formulate M as a function of a CDF. Functions of CDFs are known as **statistical functionals**.

Examples:
If a random variable $$X$$ has CDF $$F$$ and PDF $$p(x)$$,

$$E(x) = \int x dF(x) = \int xp(x)dx,$$

noting that $$dF(x) = p(x)dx$$.

When $$X$$ is discrete, we instead have

$$E(X) = \int xdF(x) = \sum_x P(x)$$

How about Variance?

$$\sigma^2 = \int x^2dF(x) - (\int xdF(x))^2$$

What we're doing is rewriting statistics as functions of their CDFs. Thus we can rewrite our statistic as $$M=g(F)$$ for population with CDF $$F$$.

#### The Plug-In Principle

Combining the EDF and the Statistical Functional gives us the Plug-In Principle.

Simply plug in the EDF for the true CDF $$F$$, enabling us to evaluate $$M=g(F).$$

Formally, we say that for a parameter of interest $$\theta=g(F)$$, if $$\hat{F}$$ is an estimator for $$F$$, the plug-in estimator for $$\theta$$ is defined to be $$\hat{\theta} = g(\hat{F})$$.
"""

# ╔═╡ 56116619-fd06-4af1-9e6c-d78b8f166542
md"""
#### Plugging in the EDF for an unknown distribution
Say we have some statistic defined as follows

$$g(F) = \int r(x)dF(x)$$.

The EDF for our data is

$$\hat{F} = \frac{1}{n}\sum_{i=1}^n1(x_i\leq x)$$

Its PMF is

$$f(x) = \frac{1}{n}, x\in {\{x_1, \dots, x_n\}}$$

Thus, the esimator for $$M$$ is

$$\hat{M} = g(\hat{F}) = \int r(x)d\hat{F}(x).$$

Using our knowledge of the PDF (I guess), we can translate this into:

$$\hat{M} = \frac{1}{n}\sum_{i=1}^n r(x_i)$$.

Note that if we let $$r(x)=x$$, we get the plug-in estimator for the population mean, which is just the sample mean.

---
Ok, so we're almost there, but we still can't use this method to compute $$Var(\hat{M})$$, I assume because we can't make the jump to discrete like we did above. So we're returning to simulation.
"""

# ╔═╡ 06fef3c2-21cf-41d4-b645-6d3dbb75450c
md"""

#### Bootstrap Variance Estimation

Let's use simulation to approximate the variance of our estimator. The original procedure has us draw a sample from $$F$$, $$B$$ times. We don't have $$F$$, we'll use the EDF instead.

Steps for simulation:

1. Draw a sample from the EDF.
2. Obtain a plug-in statistic $$\hat{M} = g(\hat{F})$$
3. Replicate the two procedures $$B$$ times.
4. Get the variance in these $$B$$ statistics and use it to **approximate the true variance of the plug-in statistic**.
"""

# ╔═╡ a2a3d276-f33e-40e3-bf66-bfeeb0c99f95
md"""

**Drawing from the EDF**

By defintion of the EDF, each point is equally weighted. Thus, sampling from the EDF is just drawing n observations with replacement, as introduced at the beginning.
"""

# ╔═╡ 58ac0581-bde8-4940-9020-a738f695a2fd
md"""

## Assumptions, Applicability

Questions:

1. Will the variance result obtained from this process, $$S^2$$, be a good estimator for $$Var(\hat{M})$$?
2. Will $$Var(\hat{M})$$ be a good approximation of $$Var(M)$$?

$$S^2 = \frac{1}{B}\sum_{j=1}^B(\hat{M_j})^2 - (\frac{1}{B}\sum_{j=1}^B\hat{M_j})^2 \approx Var(\hat{M};\hat{F}) \approx Var(M;F)$$.

These are two distinct types of error. The difference between $$S^2$$ and $$Var(\hat{M})$$ is **simulation error** (the first approximate equality sign above). The difference between $$Var(\hat{M})$$ and $$Var(M)$$ is **approximation error** (the second approximate equality sign above).

Simulation error is improved with larger $$B$$.

Approximation error will be small if the following conditions are met:
1. The EDF must use enough data to approximate $$F$$ well.
2. There are some smoothness conditions on the statistic g.

1 is easy but 2 is concerning. The important thing is that variance does in fact meet these smoothness conditions so the bootstrap method can be used to estimate it.

## Implementation

Let's actually do the thing on the original problem.
"""

# ╔═╡ dd9e01ed-64c6-43cf-bde2-add77d470770
bootmeans = [mean(sample(responses, 30; replace=true)) for x in 1:100000]

# ╔═╡ 38c6b0b4-08a9-49ed-8268-1cdfd3861d2a
bootmean = mean(bootmeans)

# ╔═╡ fddd7c16-90b7-45dc-bcd0-8fbae1b64bb4
truemean = mean(pickups)

# ╔═╡ cb88575d-719b-44d6-aebe-e071e5a90069
bootmeanstd = sqrt(var(bootmeans))

# ╔═╡ 57c0deb6-feb6-4e79-bc1d-a57a6e678eda
truestderror = std(pickups) / (30^(.5))

# ╔═╡ 5b9bc0f6-c43d-4ea5-9ce6-2aa41d0ec74a
md"""
And that's it, we're done. We see that we have calculated an estimate for the standard deviation. The bootstrap simulation gave us **$(round(bootmeanstd;digits=2))** and the true standerd error for a sample size of 30 was **$(round(truestderror;digits=2))**.

Before, we needed to assume that the standard error of the sample was a good estimate for the standard deviation of the population, but we had to assume our estimator followed a normal distribution and that it was easy to calculate the standard error for our statistic. The bootstrap method let us sidestep that, and we only needed the following assumtions:
1. We need a smooth measure of error, formulated as a statistical functional (we used variance)
2. We needed a good estimate of $$F$$

**How can we reduce the error in our error estimate?**

With higher a higher sample size, (we used 30) we will have a better EDF and less approximation error. With more simulations, we will have less simulation error. This will give us a better estimate of the error.
"""

# ╔═╡ d9166184-8705-49c4-a7e9-0bba61adcaae
md"""

### Oh and one more thing...

Did we get a better mean? We had only one sample, which had a mean of $(mean(responses)). The bootstrap method provided us with $(round(bootmean;digits=2)), which is nearly identical. So we didn't use this to get a better mean, merely to to an estimate of the error in that measurement. Perhaps however, this would not be true with a different statistic. I don't know, this was just my own observation.
"""

# ╔═╡ e41995a5-1591-4404-b585-3c4e9aac49bf
md"""

## Thoughts

- How do we know if we have a good enough estimate of $$F$$?
- How do we apply this to statistical simulation of computer architecture?
  - Can we use this to get estimates on the error for our Markov models?
- How do we use this in an online fashion?

"""

# ╔═╡ 00000000-0000-0000-0000-000000000001
PLUTO_PROJECT_TOML_CONTENTS = """
[deps]
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"

[compat]
Plots = "~1.27.6"
StatsBase = "~0.33.16"
"""

# ╔═╡ 00000000-0000-0000-0000-000000000002
PLUTO_MANIFEST_TOML_CONTENTS = """
# This file is machine-generated - editing it directly is not advised

julia_version = "1.7.1"
manifest_format = "2.0"

[[deps.Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "af92965fb30777147966f58acb05da51c5616b5f"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.3"

[[deps.ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[deps.Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[deps.Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[deps.Bzip2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "19a35467a82e236ff51bc17a3a44b69ef35185a2"
uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0"
version = "1.0.8+0"

[[deps.Cairo_jll]]
deps = ["Artifacts", "Bzip2_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"]
git-tree-sha1 = "4b859a208b2397a7a623a03449e4636bdb17bcf2"
uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a"
version = "1.16.1+1"

[[deps.ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "9950387274246d08af38f6eef8cb5480862a435f"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.14.0"

[[deps.ChangesOfVariables]]
deps = ["ChainRulesCore", "LinearAlgebra", "Test"]
git-tree-sha1 = "bf98fa45a0a4cee295de98d4c1462be26345b9a1"
uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0"
version = "0.1.2"

[[deps.ColorSchemes]]
deps = ["ColorTypes", "Colors", "FixedPointNumbers", "Random"]
git-tree-sha1 = "12fc73e5e0af68ad3137b886e3f7c1eacfca2640"
uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4"
version = "3.17.1"

[[deps.ColorTypes]]
deps = ["FixedPointNumbers", "Random"]
git-tree-sha1 = "024fe24d83e4a5bf5fc80501a314ce0d1aa35597"
uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
version = "0.11.0"

[[deps.Colors]]
deps = ["ColorTypes", "FixedPointNumbers", "Reexport"]
git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40"
uuid = "5ae59095-9a9b-59fe-a467-6f913c188581"
version = "0.12.8"

[[deps.Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "b153278a25dd42c65abbf4e62344f9d22e59191b"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.43.0"

[[deps.CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[deps.Contour]]
deps = ["StaticArrays"]
git-tree-sha1 = "9f02045d934dc030edad45944ea80dbd1f0ebea7"
uuid = "d38c429a-6771-53c6-b99e-75d170b6e991"
version = "0.5.7"

[[deps.DataAPI]]
git-tree-sha1 = "cc70b17275652eb47bc9e5f81635981f13cea5c8"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.9.0"

[[deps.DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "3daef5523dd2e769dad2365274f760ff5f282c7d"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.11"

[[deps.DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[deps.Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.6"

[[deps.Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[deps.EarCut_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "3f3a2501fa7236e9b911e0f7a588c657e822bb6d"
uuid = "5ae413db-bbd1-5e63-b57d-d24a61df00f5"
version = "2.2.3+0"

[[deps.Expat_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "bad72f730e9e91c08d9427d5e8db95478a3c323d"
uuid = "2e619515-83b5-522b-bb60-26c02a35a201"
version = "2.4.8+0"

[[deps.FFMPEG]]
deps = ["FFMPEG_jll"]
git-tree-sha1 = "b57e3acbe22f8484b4b5ff66a7499717fe1a9cc8"
uuid = "c87230d0-a227-11e9-1b43-d7ebe4e7570a"
version = "0.4.1"

[[deps.FFMPEG_jll]]
deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "Pkg", "Zlib_jll", "libass_jll", "libfdk_aac_jll", "libvorbis_jll", "x264_jll", "x265_jll"]
git-tree-sha1 = "d8a578692e3077ac998b50c0217dfd67f21d1e5f"
uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5"
version = "4.4.0+0"

[[deps.FixedPointNumbers]]
deps = ["Statistics"]
git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc"
uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
version = "0.8.4"

[[deps.Fontconfig_jll]]
deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "21efd19106a55620a188615da6d3d06cd7f6ee03"
uuid = "a3f928ae-7b40-5064-980b-68af3947d34b"
version = "2.13.93+0"

[[deps.Formatting]]
deps = ["Printf"]
git-tree-sha1 = "8339d61043228fdd3eb658d86c926cb282ae72a8"
uuid = "59287772-0a20-5a39-b81b-1366585eb4c0"
version = "0.4.2"

[[deps.FreeType2_jll]]
deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "87eb71354d8ec1a96d4a7636bd57a7347dde3ef9"
uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7"
version = "2.10.4+0"

[[deps.FriBidi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "aa31987c2ba8704e23c6c8ba8a4f769d5d7e4f91"
uuid = "559328eb-81f9-559d-9380-de523a88c83c"
version = "1.0.10+0"

[[deps.GLFW_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libglvnd_jll", "Pkg", "Xorg_libXcursor_jll", "Xorg_libXi_jll", "Xorg_libXinerama_jll", "Xorg_libXrandr_jll"]
git-tree-sha1 = "51d2dfe8e590fbd74e7a842cf6d13d8a2f45dc01"
uuid = "0656b61e-2033-5cc2-a64a-77c0f6c09b89"
version = "3.3.6+0"

[[deps.GR]]
deps = ["Base64", "DelimitedFiles", "GR_jll", "HTTP", "JSON", "Libdl", "LinearAlgebra", "Pkg", "Printf", "Random", "RelocatableFolders", "Serialization", "Sockets", "Test", "UUIDs"]
git-tree-sha1 = "af237c08bda486b74318c8070adb96efa6952530"
uuid = "28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71"
version = "0.64.2"

[[deps.GR_jll]]
deps = ["Artifacts", "Bzip2_jll", "Cairo_jll", "FFMPEG_jll", "Fontconfig_jll", "GLFW_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pixman_jll", "Pkg", "Qt5Base_jll", "Zlib_jll", "libpng_jll"]
git-tree-sha1 = "cd6efcf9dc746b06709df14e462f0a3fe0786b1e"
uuid = "d2c73de3-f751-5644-a686-071e5b155ba9"
version = "0.64.2+0"

[[deps.GeometryBasics]]
deps = ["EarCut_jll", "IterTools", "LinearAlgebra", "StaticArrays", "StructArrays", "Tables"]
git-tree-sha1 = "83ea630384a13fc4f002b77690bc0afeb4255ac9"
uuid = "5c1252a2-5f33-56bf-86c9-59e7332b4326"
version = "0.4.2"

[[deps.Gettext_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"]
git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046"
uuid = "78b55507-aeef-58d4-861c-77aaff3498b1"
version = "0.21.0+0"

[[deps.Glib_jll]]
deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "a32d672ac2c967f3deb8a81d828afc739c838a06"
uuid = "7746bdde-850d-59dc-9ae8-88ece973131d"
version = "2.68.3+2"

[[deps.Graphite2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011"
uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472"
version = "1.3.14+0"

[[deps.Grisu]]
git-tree-sha1 = "53bb909d1151e57e2484c3d1b53e19552b887fb2"
uuid = "42e2da0e-8278-4e71-bc24-59509adca0fe"
version = "1.0.2"

[[deps.HTTP]]
deps = ["Base64", "Dates", "IniFile", "Logging", "MbedTLS", "NetworkOptions", "Sockets", "URIs"]
git-tree-sha1 = "0fa77022fe4b511826b39c894c90daf5fce3334a"
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
version = "0.9.17"

[[deps.HarfBuzz_jll]]
deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"]
git-tree-sha1 = "129acf094d168394e80ee1dc4bc06ec835e510a3"
uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566"
version = "2.8.1+1"

[[deps.IniFile]]
git-tree-sha1 = "f550e6e32074c939295eb5ea6de31849ac2c9625"
uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f"
version = "0.5.1"

[[deps.InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[deps.InverseFunctions]]
deps = ["Test"]
git-tree-sha1 = "91b5dcf362c5add98049e6c29ee756910b03051d"
uuid = "3587e190-3f89-42d0-90ee-14403ec27112"
version = "0.1.3"

[[deps.IrrationalConstants]]
git-tree-sha1 = "7fd44fd4ff43fc60815f8e764c0f352b83c49151"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.1.1"

[[deps.IterTools]]
git-tree-sha1 = "fa6287a4469f5e048d763df38279ee729fbd44e5"
uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e"
version = "1.4.0"

[[deps.IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"

[[deps.JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "abc9885a7ca2052a736a600f7fa66209f96506e1"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.4.1"

[[deps.JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "3c837543ddb02250ef42f4738347454f95079d4e"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.3"

[[deps.JpegTurbo_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b53380851c6e6664204efb2e62cd24fa5c47e4ba"
uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8"
version = "2.1.2+0"

[[deps.LAME_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "f6250b16881adf048549549fba48b1161acdac8c"
uuid = "c1c5ebd0-6772-5130-a774-d5fcae4a789d"
version = "3.100.1+0"

[[deps.LERC_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "bf36f528eec6634efc60d7ec062008f171071434"
uuid = "88015f11-f218-50d7-93a8-a6af411a945d"
version = "3.0.0+1"

[[deps.LZO_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "e5b909bcf985c5e2605737d2ce278ed791b89be6"
uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac"
version = "2.10.1+0"

[[deps.LaTeXStrings]]
git-tree-sha1 = "f2355693d6778a178ade15952b7ac47a4ff97996"
uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
version = "1.3.0"

[[deps.Latexify]]
deps = ["Formatting", "InteractiveUtils", "LaTeXStrings", "MacroTools", "Markdown", "Printf", "Requires"]
git-tree-sha1 = "6f14549f7760d84b2db7a9b10b88cd3cc3025730"
uuid = "23fbe1c1-3f47-55db-b15f-69d7ec21a316"
version = "0.15.14"

[[deps.LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[deps.LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[deps.LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[deps.LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[deps.Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[deps.Libffi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "0b4a5d71f3e5200a7dff793393e09dfc2d874290"
uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490"
version = "3.2.2+1"

[[deps.Libgcrypt_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll", "Pkg"]
git-tree-sha1 = "64613c82a59c120435c067c2b809fc61cf5166ae"
uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4"
version = "1.8.7+0"

[[deps.Libglvnd_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll", "Xorg_libXext_jll"]
git-tree-sha1 = "7739f837d6447403596a75d19ed01fd08d6f56bf"
uuid = "7e76a0d4-f3c7-5321-8279-8d96eeed0f29"
version = "1.3.0+3"

[[deps.Libgpg_error_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "c333716e46366857753e273ce6a69ee0945a6db9"
uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8"
version = "1.42.0+0"

[[deps.Libiconv_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "42b62845d70a619f063a7da093d995ec8e15e778"
uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531"
version = "1.16.1+1"

[[deps.Libmount_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "9c30530bf0effd46e15e0fdcf2b8636e78cbbd73"
uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9"
version = "2.35.0+0"

[[deps.Libtiff_jll]]
deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"]
git-tree-sha1 = "c9551dd26e31ab17b86cbd00c2ede019c08758eb"
uuid = "89763e89-9b03-5906-acba-b20f662cd828"
version = "4.3.0+1"

[[deps.Libuuid_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "7f3efec06033682db852f8b3bc3c1d2b0a0ab066"
uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700"
version = "2.36.0+0"

[[deps.LinearAlgebra]]
deps = ["Libdl", "libblastrampoline_jll"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[deps.LogExpFunctions]]
deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "a970d55c2ad8084ca317a4658ba6ce99b7523571"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.12"

[[deps.Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[deps.MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "3d3e902b31198a27340d0bf00d6ac452866021cf"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.9"

[[deps.Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[deps.MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.3"

[[deps.MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[deps.Measures]]
git-tree-sha1 = "e498ddeee6f9fdb4551ce855a46f54dbd900245f"
uuid = "442fdcdd-2543-5da2-b0f3-8c86c306513e"
version = "0.3.1"

[[deps.Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "bf210ce90b6c9eed32d25dbcae1ebc565df2687f"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.0.2"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[deps.NaNMath]]
git-tree-sha1 = "737a5957f387b17e74d4ad2f440eb330b39a62c5"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "1.0.0"

[[deps.NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[deps.Ogg_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "887579a3eb005446d514ab7aeac5d1d027658b8f"
uuid = "e7412a2a-1a6e-54c0-be00-318e2571c051"
version = "1.3.5+1"

[[deps.OpenBLAS_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"]
uuid = "4536629a-c528-5b80-bd46-f80d51c5b363"

[[deps.OpenSSL_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "ab05aa4cc89736e95915b01e7279e61b1bfe33b8"
uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95"
version = "1.1.14+0"

[[deps.Opus_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "51a08fb14ec28da2ec7a927c4337e4332c2a4720"
uuid = "91d4177d-7536-5919-b921-800302f37372"
version = "1.3.2+0"

[[deps.OrderedCollections]]
git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.1"

[[deps.PCRE_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b2a7af664e098055a7529ad1a900ded962bca488"
uuid = "2f80f16e-611a-54ab-bc61-aa92de5b98fc"
version = "8.44.0+0"

[[deps.Parsers]]
deps = ["Dates"]
git-tree-sha1 = "3b429f37de37f1fc603cc1de4a799dc7fbe4c0b6"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.3.0"

[[deps.Pixman_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b4f5d02549a10e20780a24fce72bea96b6329e29"
uuid = "30392449-352a-5448-841d-b1acce4e97dc"
version = "0.40.1+0"

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[deps.PlotThemes]]
deps = ["PlotUtils", "Statistics"]
git-tree-sha1 = "8162b2f8547bc23876edd0c5181b27702ae58dce"
uuid = "ccf2f8ad-2431-5c83-bf29-c5338b663b6a"
version = "3.0.0"

[[deps.PlotUtils]]
deps = ["ColorSchemes", "Colors", "Dates", "Printf", "Random", "Reexport", "Statistics"]
git-tree-sha1 = "bb16469fd5224100e422f0b027d26c5a25de1200"
uuid = "995b91a9-d308-5afd-9ec6-746e21dbc043"
version = "1.2.0"

[[deps.Plots]]
deps = ["Base64", "Contour", "Dates", "Downloads", "FFMPEG", "FixedPointNumbers", "GR", "GeometryBasics", "JSON", "Latexify", "LinearAlgebra", "Measures", "NaNMath", "Pkg", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "RecipesPipeline", "Reexport", "Requires", "Scratch", "Showoff", "SparseArrays", "Statistics", "StatsBase", "UUIDs", "UnicodeFun", "Unzip"]
git-tree-sha1 = "6f2dd1cf7a4bbf4f305a0d8750e351cb46dfbe80"
uuid = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
version = "1.27.6"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "47e5f437cc0e7ef2ce8406ce1e7e24d44915f88d"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.3.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.Qt5Base_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "Libdl", "Libglvnd_jll", "OpenSSL_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libxcb_jll", "Xorg_xcb_util_image_jll", "Xorg_xcb_util_keysyms_jll", "Xorg_xcb_util_renderutil_jll", "Xorg_xcb_util_wm_jll", "Zlib_jll", "xkbcommon_jll"]
git-tree-sha1 = "c6c0f690d0cc7caddb74cef7aa847b824a16b256"
uuid = "ea2cea3b-5b76-57ae-a6ef-0a8af62496e1"
version = "5.15.3+1"

[[deps.REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.RecipesBase]]
git-tree-sha1 = "6bf3f380ff52ce0832ddd3a2a7b9538ed1bcca7d"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.2.1"

[[deps.RecipesPipeline]]
deps = ["Dates", "NaNMath", "PlotUtils", "RecipesBase"]
git-tree-sha1 = "dc1e451e15d90347a7decc4221842a022b011714"
uuid = "01d81517-befc-4cb6-b9ec-a95719d0359c"
version = "0.5.2"

[[deps.Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[deps.RelocatableFolders]]
deps = ["SHA", "Scratch"]
git-tree-sha1 = "cdbd3b1338c72ce29d9584fdbe9e9b70eeb5adca"
uuid = "05181044-ff0b-4ac5-8273-598c1e38db00"
version = "0.1.3"

[[deps.Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.3.0"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[deps.Scratch]]
deps = ["Dates"]
git-tree-sha1 = "0b4b7f1393cff97c33891da2a0bf69c6ed241fda"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.1.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[deps.Showoff]]
deps = ["Dates", "Grisu"]
git-tree-sha1 = "91eddf657aca81df9ae6ceb20b959ae5653ad1de"
uuid = "992d4aef-0814-514b-bc4d-f2e9a6c4116f"
version = "1.0.3"

[[deps.Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[deps.SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.0.1"

[[deps.SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[deps.StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "cd56bf18ed715e8b09f06ef8c6b781e6cdc49911"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.4.4"

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[deps.StatsAPI]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "c82aaa13b44ea00134f8c9c89819477bd3986ecd"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.3.0"

[[deps.StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "8977b17906b0a1cc74ab2e3a05faa16cf08a8291"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.16"

[[deps.StructArrays]]
deps = ["Adapt", "DataAPI", "StaticArrays", "Tables"]
git-tree-sha1 = "57617b34fa34f91d536eb265df67c2d4519b8b98"
uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a"
version = "0.6.5"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[deps.TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.1"

[[deps.Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "OrderedCollections", "TableTraits", "Test"]
git-tree-sha1 = "5ce79ce186cc678bbb5c5681ca3379d1ddae11a1"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.7.0"

[[deps.Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[deps.Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[deps.URIs]]
git-tree-sha1 = "97bbe755a53fe859669cd907f2d96aee8d2c1355"
uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
version = "1.3.0"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[deps.UnicodeFun]]
deps = ["REPL"]
git-tree-sha1 = "53915e50200959667e78a92a418594b428dffddf"
uuid = "1cfade01-22cf-5700-b092-accc4b62d6e1"
version = "0.4.1"

[[deps.Unzip]]
git-tree-sha1 = "34db80951901073501137bdbc3d5a8e7bbd06670"
uuid = "41fe7b60-77ed-43a1-b4f0-825fd5a5650d"
version = "0.1.2"

[[deps.Wayland_jll]]
deps = ["Artifacts", "Expat_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg", "XML2_jll"]
git-tree-sha1 = "3e61f0b86f90dacb0bc0e73a0c5a83f6a8636e23"
uuid = "a2964d1f-97da-50d4-b82a-358c7fce9d89"
version = "1.19.0+0"

[[deps.Wayland_protocols_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4528479aa01ee1b3b4cd0e6faef0e04cf16466da"
uuid = "2381bf8a-dfd0-557d-9999-79630e7b1b91"
version = "1.25.0+0"

[[deps.XML2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "1acf5bdf07aa0907e0a37d3718bb88d4b687b74a"
uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a"
version = "2.9.12+0"

[[deps.XSLT_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"]
git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a"
uuid = "aed1982a-8fda-507f-9586-7b0439959a61"
version = "1.1.34+0"

[[deps.Xorg_libX11_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll", "Xorg_xtrans_jll"]
git-tree-sha1 = "5be649d550f3f4b95308bf0183b82e2582876527"
uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc"
version = "1.6.9+4"

[[deps.Xorg_libXau_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4e490d5c960c314f33885790ed410ff3a94ce67e"
uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec"
version = "1.0.9+4"

[[deps.Xorg_libXcursor_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXfixes_jll", "Xorg_libXrender_jll"]
git-tree-sha1 = "12e0eb3bc634fa2080c1c37fccf56f7c22989afd"
uuid = "935fb764-8cf2-53bf-bb30-45bb1f8bf724"
version = "1.2.0+4"

[[deps.Xorg_libXdmcp_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4fe47bd2247248125c428978740e18a681372dd4"
uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05"
version = "1.1.3+4"

[[deps.Xorg_libXext_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "b7c0aa8c376b31e4852b360222848637f481f8c3"
uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3"
version = "1.3.4+4"

[[deps.Xorg_libXfixes_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "0e0dc7431e7a0587559f9294aeec269471c991a4"
uuid = "d091e8ba-531a-589c-9de9-94069b037ed8"
version = "5.0.3+4"

[[deps.Xorg_libXi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll", "Xorg_libXfixes_jll"]
git-tree-sha1 = "89b52bc2160aadc84d707093930ef0bffa641246"
uuid = "a51aa0fd-4e3c-5386-b890-e753decda492"
version = "1.7.10+4"

[[deps.Xorg_libXinerama_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll"]
git-tree-sha1 = "26be8b1c342929259317d8b9f7b53bf2bb73b123"
uuid = "d1454406-59df-5ea1-beac-c340f2130bc3"
version = "1.1.4+4"

[[deps.Xorg_libXrandr_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll"]
git-tree-sha1 = "34cea83cb726fb58f325887bf0612c6b3fb17631"
uuid = "ec84b674-ba8e-5d96-8ba1-2a689ba10484"
version = "1.5.2+4"

[[deps.Xorg_libXrender_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "19560f30fd49f4d4efbe7002a1037f8c43d43b96"
uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa"
version = "0.9.10+4"

[[deps.Xorg_libpthread_stubs_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "6783737e45d3c59a4a4c4091f5f88cdcf0908cbb"
uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74"
version = "0.1.0+3"

[[deps.Xorg_libxcb_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"]
git-tree-sha1 = "daf17f441228e7a3833846cd048892861cff16d6"
uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b"
version = "1.13.0+3"

[[deps.Xorg_libxkbfile_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "926af861744212db0eb001d9e40b5d16292080b2"
uuid = "cc61e674-0454-545c-8b26-ed2c68acab7a"
version = "1.1.0+4"

[[deps.Xorg_xcb_util_image_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "0fab0a40349ba1cba2c1da699243396ff8e94b97"
uuid = "12413925-8142-5f55-bb0e-6d7ca50bb09b"
version = "0.4.0+1"

[[deps.Xorg_xcb_util_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll"]
git-tree-sha1 = "e7fd7b2881fa2eaa72717420894d3938177862d1"
uuid = "2def613f-5ad1-5310-b15b-b15d46f528f5"
version = "0.4.0+1"

[[deps.Xorg_xcb_util_keysyms_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "d1151e2c45a544f32441a567d1690e701ec89b00"
uuid = "975044d2-76e6-5fbe-bf08-97ce7c6574c7"
version = "0.4.0+1"

[[deps.Xorg_xcb_util_renderutil_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "dfd7a8f38d4613b6a575253b3174dd991ca6183e"
uuid = "0d47668e-0667-5a69-a72c-f761630bfb7e"
version = "0.3.9+1"

[[deps.Xorg_xcb_util_wm_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "e78d10aab01a4a154142c5006ed44fd9e8e31b67"
uuid = "c22f9ab0-d5fe-5066-847c-f4bb1cd4e361"
version = "0.4.1+1"

[[deps.Xorg_xkbcomp_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxkbfile_jll"]
git-tree-sha1 = "4bcbf660f6c2e714f87e960a171b119d06ee163b"
uuid = "35661453-b289-5fab-8a00-3d9160c6a3a4"
version = "1.4.2+4"

[[deps.Xorg_xkeyboard_config_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xkbcomp_jll"]
git-tree-sha1 = "5c8424f8a67c3f2209646d4425f3d415fee5931d"
uuid = "33bec58e-1273-512f-9401-5d533626f822"
version = "2.27.0+4"

[[deps.Xorg_xtrans_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "79c31e7844f6ecf779705fbc12146eb190b7d845"
uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10"
version = "1.4.0+3"

[[deps.Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[deps.Zstd_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "e45044cd873ded54b6a5bac0eb5c971392cf1927"
uuid = "3161d3a3-bdf6-5164-811a-617609db77b4"
version = "1.5.2+0"

[[deps.libass_jll]]
deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "HarfBuzz_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "5982a94fcba20f02f42ace44b9894ee2b140fe47"
uuid = "0ac62f75-1d6f-5e53-bd7c-93b484bb37c0"
version = "0.15.1+0"

[[deps.libblastrampoline_jll]]
deps = ["Artifacts", "Libdl", "OpenBLAS_jll"]
uuid = "8e850b90-86db-534c-a0d3-1478176c7d93"

[[deps.libfdk_aac_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "daacc84a041563f965be61859a36e17c4e4fcd55"
uuid = "f638f0a6-7fb0-5443-88ba-1cc74229b280"
version = "2.0.2+0"

[[deps.libpng_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "94d180a6d2b5e55e447e2d27a29ed04fe79eb30c"
uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f"
version = "1.6.38+0"

[[deps.libvorbis_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Ogg_jll", "Pkg"]
git-tree-sha1 = "b910cb81ef3fe6e78bf6acee440bda86fd6ae00c"
uuid = "f27f6e37-5d2b-51aa-960f-b287f2bc3b7a"
version = "1.3.7+1"

[[deps.nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[deps.p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

[[deps.x264_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4fea590b89e6ec504593146bf8b988b2c00922b2"
uuid = "1270edf5-f2f9-52d2-97e9-ab00b5d0237a"
version = "2021.5.5+0"

[[deps.x265_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "ee567a171cce03570d77ad3a43e90218e38937a9"
uuid = "dfaa095f-4041-5dcd-9319-2fabd8486b76"
version = "3.5.0+0"

[[deps.xkbcommon_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Wayland_jll", "Wayland_protocols_jll", "Xorg_libxcb_jll", "Xorg_xkeyboard_config_jll"]
git-tree-sha1 = "ece2350174195bb31de1a63bea3a41ae1aa593b6"
uuid = "d8fb68d0-12a3-5cfd-a85a-d49703b185fd"
version = "0.9.1+5"
"""

# ╔═╡ Cell order:
# ╟─0810c98f-2526-4ab4-9cd5-a662434e332d
# ╟─a4283cac-c253-11ec-1d76-495287bbab72
# ╟─c33fde47-ac17-4194-973b-cdb277907b60
# ╟─e42b8dfb-1a75-448c-acc1-3255c87a0edf
# ╟─0e14ff27-5d05-4367-947e-d97e96ddfa8e
# ╠═fbbc6413-0afb-4531-8113-bb85afd2710d
# ╟─0ed657f2-8f2b-4e8f-a375-9c8986448340
# ╟─c30c49a1-2821-4ada-ab10-5cac70a99e87
# ╟─520524ab-53ef-4edb-bc58-f6a86620a126
# ╟─fda6b303-a082-4f44-8341-49446b100609
# ╟─4453da4c-12fc-427d-9f9c-89476bf65541
# ╟─f9153c34-d26b-4e54-847e-c3bae8cdb4df
# ╟─d273d1b8-e66c-4461-ab55-b2a4fe578d0c
# ╟─14b9e9b0-d67b-4376-80a5-cb0b76602174
# ╟─ceb8a078-4c55-40e7-a1bf-41bc357eaf86
# ╟─3c1f21fb-be76-446d-b27f-54119ee4835a
# ╟─1e05bcbc-cf0a-47ba-a1be-cb2e9bafff78
# ╟─56116619-fd06-4af1-9e6c-d78b8f166542
# ╟─06fef3c2-21cf-41d4-b645-6d3dbb75450c
# ╟─a2a3d276-f33e-40e3-bf66-bfeeb0c99f95
# ╟─58ac0581-bde8-4940-9020-a738f695a2fd
# ╠═dd9e01ed-64c6-43cf-bde2-add77d470770
# ╠═38c6b0b4-08a9-49ed-8268-1cdfd3861d2a
# ╠═fddd7c16-90b7-45dc-bcd0-8fbae1b64bb4
# ╠═cb88575d-719b-44d6-aebe-e071e5a90069
# ╠═57c0deb6-feb6-4e79-bc1d-a57a6e678eda
# ╟─5b9bc0f6-c43d-4ea5-9ce6-2aa41d0ec74a
# ╟─d9166184-8705-49c4-a7e9-0bba61adcaae
# ╟─e41995a5-1591-4404-b585-3c4e9aac49bf
# ╟─00000000-0000-0000-0000-000000000001
# ╟─00000000-0000-0000-0000-000000000002


function linsolve_pcg( Lmat::SparseMatrixCSC{Float64,Int64},
                       prec::SparseMatrixCSC{Float64,Int64},
                       b::Array{Float64,1};
                       x0 = nothing,
                       NiterMax = 1000, TOL=5.e-10,
                       convmsg=false, showprogress=false )
    #
    Npoints = size(b)[1]
    if x0 == nothing
        x = zeros( Float64, Npoints )
    else
        x = copy(x0)
    end
    #
    r = zeros( Float64, Npoints )
    p = zeros( Float64, Npoints )
    z = zeros( Float64, Npoints )
    #
    nabla2_x = Lmat*x
    r = b - nabla2_x
    z = apply_prec_ilu0( prec, r)
    p = copy(z)

    rsold = dot( r, z )

    for iter = 1 : NiterMax
        #
        nabla2_x = Lmat*p
        #
        alpha = rsold/dot( p, nabla2_x )
        #
        x = x + alpha * p
        r = r - alpha * nabla2_x
        z = apply_prec_ilu0( prec, r)
        #
        rsnew = dot( z, r )
        # deltars = rsold - rsnew
        if showprogress
            @printf("%8d %18.10e\n", iter, sqrt(abs(rsnew)))
        end
        #
        if sqrt(abs(rsnew)) < TOL
            if convmsg
                @printf("#Convergence achieved in linsolve_pcg: %8d iterations.\n", iter)
            end
            break
        end
        #
        p = z + (rsnew/rsold) * p
        #
        rsold = rsnew
    end
    #
    return x
    #
end # of function


function trinary_to_decimal(str)
    dec = 0
    for (i, ch) in enumerate(reverse(str))
        d = Int(ch) - 48
        if !(0 <= d <= 2)
            return 0
        end
        dec += d * 3^(i - 1)
    end
    return dec
end


import Base.randperm

"The path graph on n vertices"
function pathGraph(n::Int64)
  x = append!(collect(1:(n-1)), collect(2:n))
  y = append!(collect(2:n), collect(1:(n-1)))
  w = ones(2 * (n - 1))
  return sparse(x, y, w)
end # pathGraph

"The complete graph"
function completeGraph(n::Int64)
  return sparse(ones(n,n) - eye(n))
end # completeGraph

"""The simple ring on n vertices"""
function ringGraph(n::Int64)
    a = spdiagm(ones(n-1),1,n,n)
    a[1,n] = 1
    a = a + a'
end

"""A generalization of a ring graph.
The vertices are integers modulo n.
Two are connected if their difference is in gens.
For example,

```
generalizedRing(17, [1 5])
```
"""
function generalizedRing(n::Int64, gens)
    k = length(gens)
    m = 2*n*k
    ai = zeros(Int64,m)
    aj = zeros(Int64,m)
    ind = 1
    for i in 0:(n-1)
        for j in 1:k
            ai[ind] = i
            aj[ind] = mod(i+gens[j],n)
            ind = ind + 1
            ai[ind] = i
            aj[ind] = mod(i-gens[j],n)
            ind = ind + 1
        end
    end
    return sparse(1+ai,1+aj,ones(m),n,n)
    #return ai, aj
end

"""A random generalized ring graph of degree k.
Gens always contains 1, and the other k-1 edge types
are chosen from an exponential distribution"""
function randGenRing(n::Int64, k::Integer)
    # if any of n, 2n, 3n etc. is in gens we will have self loops
    gens = [0]
    while 0 in (gens % n)
        gens = [1; 1 + ceil(Integer,exp(rand(k-1)*log(n-1)))]
    end

    return generalizedRing(n, gens)
end

"""
~~~
generalizedNecklace{Tv, Ti}(A::SparseMatrixCSC{Tv, Ti}, H::SparseMatrixCSC, k::Int64)
~~~

 Constructs a generalized necklace graph starting with two graphs A and H. The
resulting new graph will be constructed by expanding each vertex in H to an
instance of A. k random edges will be generated between components. Thus, the
resulting graph may have weighted edges.
"""

function generalizedNecklace{Tv, Ti}(A::SparseMatrixCSC{Tv, Ti}, H::SparseMatrixCSC, k::Int64)
  a = findnz(A)
  h = findnz(H)

  # these are square matrices
  n = A.n
  m = H.n

  newI = Ti[]
  newJ = Ti[]
  newW = Tv[]

  # duplicate the vertices in A so that each vertex in H corresponds to a copy of A
  for i in 1:m
    newI = append!(newI, a[1] + n * (i - 1))
    newJ = append!(newJ, a[2] + n * (i - 1))
    newW = append!(newW, a[3])
  end

  # for each edge in H, add k random edges between two corresponding components
  # multiedges will be concatenated to a single edge with higher cost
  for i in 1:length(h[1])
    u = h[1][i]
    v = h[2][i]

    if (u < v)
      #component x is from 1 + (x - 1) * n to n + (x - 1) * n
      for edgeToAdd in 1:k
        newU = rand(1:n) + n * (u - 1)
        newV = rand(1:n) + n * (v - 1)
        append!(newI, [newU, newV])
        append!(newJ, [newV, newU])
        append!(newW, [1, 1])
      end
    end
  end

  return sparse(newI, newJ, newW)
end # generalizedNecklace

"""The d dimensional hypercube.  Has 2^d vertices"""
function hyperCube(d::Int64)
  a = sparse([0 1; 1 0])

  for i = 1:(d-1)
    k = 2^i
    D = speye(k)
    a = [a D; D a]
  end

  return a
end # hyperCube

"""The complete binary tree on n vertices"""
function completeBinaryTree(n::Int64)

  k = div(n-1,2)
  a = sparse(collect(1:k),2*collect(1:k),1,n,n) + sparse(collect(1:k),2*collect(1:k)+1,1,n,n)

  if 2*k+1 < n
    a[n-1,n] = 1
  end

  a = a + a'

  return a
end # completeBinaryTree

""" An n by n grid with random weights. User can specify the weighting scheme. """
function wGrid2(n::Int64; weightGen::Function=rand)
    gr2 = sparse(grid2(n));

    gr2.nzval = Float64[weightGen() for i in 1:nnz(gr2)]

    # symmetrize
    gr2 = tril(gr2) + tril(gr2)'

    return gr2
end

""" An n^3 grid with random weights. User can specify the weighting scheme. """
function wGrid3(n::Int64; weightGen::Function=rand)
    gr2 = grid2(n);

    a = kron(speye(n), gr2);
    b = kron(gr2, speye(n));

    gr3 = sparse(a + b);
    gr3.nzval = Float64[weightGen() for i in 1:nnz(gr3)]

    # symmetrize
    gr3 = tril(gr3) + tril(gr3)'

    return gr3
end

"""An n-by-m grid graph.  iostropy is the weighting on edges in one direction."""
function grid2(n::Int64, m::Int64; isotropy=1)
  a = kron(speye(n),spdiagm(ones(m-1),1,m,m))
  a = a + isotropy*kron(spdiagm(ones(n-1),1,n,n), speye(m))
  a = a + a'
  return a
end # grid2

grid2(n::Int64) = grid2(n,n)

"""Coordinates for plotting the vertices of the n-by-m grid graph"""
function grid2coords(n::Int64, m::Int64)
  x = kron(collect(1:n),ones(m))
  y = kron(ones(n),collect(1:m))
  return x, y
end # grid2coords

grid2coords(n) = grid2coords(n, n)

"""A random matching on n vertices"""
function randMatching(n::Int64)

  p = randperm(n)
  n1 = convert(Int64,floor(n/2))
  n2 = 2*n1
  a = sparse(p[1:n1],p[(n1+1):n2],ones(n1),n,n)

  a = a + a'

  return a

end # randMatching

"""A sum of k random matchings on n vertices"""
function randRegular(n::Int64, k::Int64)
  a = randMatching(n)
  for i in 2:k
    a = a + randMatching(n)
  end

  return a
end # randRegular

"""Create a graph on n vertices.
For each vertex, give it k edges to randomly chosen prior
vertices.
This is a variety of a preferential attachment graph.
"""
function grownGraph(n::Int64, k::Int64)
  a = spzeros(n,n)

  for i = 1:k
    a = a + sparse(2:n,ceil(Integer,collect(1:n-1).*rand(n-1)),1,n,n)
  end

  a = a + a'
end # grownGraph

# used in grownGraphD
function randSet(n::Integer,k::Integer)
    if n == k
        return collect(1:n)
    elseif n < k
        error("n must be at least k")
    else

        s = sort(ceil(Integer,n*rand(k)))
        good = (minimum(s[2:end]-s[1:(end-1)]) > 0)
        while good == false
            s = sort(ceil(Integer,n*rand(k)))
            good = (minimum(s[2:end]-s[1:(end-1)]) > 0)
        end

        return s

    end
end

"""Like a grownGraph, but it forces the edges to all be distinct.
It starts out with a k+1 clique on the first k vertices"""
function grownGraphD(n::Int64, k::Int64)
    a = spzeros(n,n)

    u = zeros(Int64, k*(n-k-1))
    v = zeros(Int64, k*(n-k-1))

    for i in (k+2):n
        nb = randSet(i-1,k)
        u[(i-k-2)*k + collect(1:k)] = i
        v[(i-k-2)*k + collect(1:k)] = nb
    end

    a = sparse(u,v,1,n,n)

    (ai,aj) = findnz(triu(ones(k+1,k+1),1))
    a = a + sparse(ai,aj,1,n,n)
    a = a + a'

end # grownGraphD

"""A preferential attachment graph in which each vertex has k edges to those
that come before.  These are chosen with probability p to be from a random vertex,
and with probability 1-p to come from the endpoint of a random edge.
It begins with a k-clique on the first k+1 vertices."""
function prefAttach(n::Int64, k::Int64, p::Float64)
    if n == (k+1)
        return sparse(ones(Float64,n,n) - eye(Float64,n))
    elseif n <= k
        error("n must be more than k")
    else

        u = zeros(Int64,n*k)
        v = zeros(Int64,n*k)

        # fill in the initial clique
        # this will accidentally double every edge in the clique
        # we clean it up at the end
        ind = 1
        for i in 1:(k+1)
            for j in 1:(k+1)
                if i != j
                    u[ind] = i
                    v[ind] = j
                    ind += 1
                end
            end
        end

        s = zeros(Int64,k)
        for i in (k+2):n
            distinct = false
            while distinct == false
                for j in 1:k
                    if rand(Float64) < p
                        s[j] = rand(1:(i-1))
                    else
                        s[j] = v[rand(1:(k*(i-1)))]
                    end
                end
                s = sort(s)
                distinct = true
                for ii in 1:(k-1)
                    if s[ii] == s[ii+1]
                        distinct = false
                    end
                end
                # distinct = (minimum(s[2:end]-s[1:(end-1)]) > 0)

            end

            for j in 1:k
                u[ind] = i
                v[ind] = s[j]
                ind += 1
            end

        end # for i

        w = ones(Float64,n*k)

        w[1:(k*(k+1))] = 1/2

        a = sparse(u,v,w,n,n)
        a = a + a'
        return a
    end
end

"""Randomly permutes the vertex indices"""
function randperm(mat::AbstractMatrix)
    perm = randperm(mat.n)
    return mat[perm,perm]
end

randperm(f::Expr) = randperm(eval(f))

"""Generate a random graph on n vertices with m edges.
The actual number of edges will probably be smaller, as we sample
with replacement"""
function ErdosRenyi(n::Integer, m::Integer)
    ai = rand(1:n, m)
    aj = rand(1:n, m)
    ind = (ai .!= aj)
    mat = sparse(ai[ind],aj[ind],1,n,n)
    uniformWeight!(mat)
    return mat
end

"""Generate an ER graph with average degree k,
and then return the largest component.
Will probably have fewer than n vertices.
If you want to add a tree to bring it back to n,
try ErdosRenyiClusterFix."""
function ErdosRenyiCluster(n::Integer, k::Integer)
    m = ceil(Integer,n*k/2)
    ai = rand(1:n, m)
    aj = rand(1:n, m)
    ind = (ai .!= aj)
    mat = sparse(ai[ind],aj[ind],1,n,n)
    mat = mat + mat'

    return biggestComp(mat)
end

"""Like an Erdos-Renyi cluster, but add back a tree so
it has n vertices"""
function ErdosRenyiClusterFix(n::Integer, k::Integer)
    m1 = ErdosRenyiCluster(n, k)
    n2 = n - size(m1)[1]
    if (n2 > 0)
        m2 = completeBinaryTree(n2)
        return joinGraphs(m1,m2,1)
    else
        return m1
    end
end

"""Generate a random graph with n vertices from one of our natural distributions"""
function pureRandomGraph(n::Integer)

    gr = []
    wt = []

    push!(gr,:(pathGraph($n)))
    push!(wt,1)

    push!(gr,:(ringGraph($n)))
    push!(wt,3)

    push!(gr,:(completeBinaryTree($n)))
    push!(wt,3)

    push!(gr,:(grownGraph($n,2)))
    push!(wt,6)

    push!(gr,:(grid2(ceil(Integer,sqrt($n)))[1:$n,1:$n]))
    push!(wt,6)

    push!(gr,:(randRegular($n,3)))
    push!(wt,6)

    push!(gr,:(ErdosRenyiClusterFix($n,2)))
    push!(wt,6)

    if n >= 4
        push!(gr,:(randGenRing($n,4)))
        push!(wt,6)
    end

    i = sampleByWeight(wt)

    # make sure get a connected graph
    its = 0
    mat = eval(gr[i])

    while (~isConnected(mat)) && (its < 100)
        i = sampleByWeight(wt)
        mat = eval(gr[i])
        its += 1
    end
    if its == 100
        error("Getting a disconnected graph from $(gr[i])")
    end

    if (sum(diag(mat)) > 0)
        error("nonzero diag from $(gr[i])")
    end

    return floatGraph(mat)

end

"""sample an index with probability proportional to its weight given here"""
function sampleByWeight(wt)
    r = rand(1)*sum(wt)
    find(cumsum(wt) .> r)[1]
end

"""A Chimera graph with some weights.  The weights just appear when graphs are combined.
For more interesting weights, use `wtedChimera`"""
function semiWtedChimera(n::Integer)

    if (n < 2)
        gr = sparse([0.0])

        return randperm(gr)
    end

    r = rand()^2

    if (n < 30) || (rand() < .2)

        gr = pureRandomGraph(n)

        return randperm(gr)
    end

    if (n < 200)
        # just join disjoint copies of graphs

        n1 = 10 + floor(Integer,(n-20)*rand())
        n2 = n - n1
        k = ceil(Integer,exp(rand()*log(min(n1,n2)/2)))

        gr = joinGraphs(r*chimera(n1),chimera(n2),k)

        return randperm(gr)
    end

    # split with probability .7

    if (rand() < .7)
        n1 = ceil(Integer,10*exp(rand()*log(n/20)))

        n2 = n - n1
        k = floor(Integer,1+exp(rand()*log(min(n1,n2)/2)))

        gr = joinGraphs(r*chimera(n1),chimera(n2),k)

        return randperm(gr)

    else
        n1 = floor(Integer,10*exp(rand()*log(n/100)))

        n2 = floor(Integer, n / n1)

        if (rand() < .5)

            gr = productGraph(r*chimera(n1),chimera(n2))

        else

            k = floor(Integer,1+exp(rand()*log(min(n1,n2)/10)))
            gr = generalizedNecklace(r*chimera(n1),chimera(n2),k)

        end

        n3 = n - size(gr)[1]
        if (n3 > 0)
            gr = joinGraphs(gr,chimera(n3),2)

        end

        return randperm(gr)

    end
end

"""Builds a chimeric graph on n vertices.
The components come from pureRandomGraph,
connected by joinGraphs, productGraph and generalizedNecklace"""
function chimera(n::Integer)

    gr = semiWtedChimera(n)
    unweight!(gr)

    return gr

end

"""Builds the kth chimeric graph on n vertices.
It does this by resetting the random number generator seed.
It should captute the state of the generator before that and then
return it, but it does not yet."""
function chimera(n::Integer, k::Integer)
    srand(100*n+k)
    g = chimera(n)
    return g
end

"""Applies one of a number of random weighting schemes to the edges of the graph"""
function randWeight(a)

    if (rand() < .2)
        return a
    end

    n = a.n
    (ai,aj) = findnz(a)
    m = length(ai)

    # potentials or edge-based

    if (rand() < .3)
        w = rand(m)

    else
        v = randn(a.n)

        # mult by matrix ?
        if (rand() < .5)

            invdeg = spdiagm(1./(a*ones(size(a)[1])))
            if (rand() < .5)
                for i in 1:10
                    v = a * (invdeg * v)
                    v = v - mean(v)
                end
            else
                for i in 1:10
                    v = v - a * (invdeg * v)
                    v = v - mean(v)
                end
            end
        end

        w = abs(v[ai]-v[aj])

    end

    # reciprocate or not?

    w[w.==0] = 1
    w[isnan(w)] = 1

    if (rand() < .5)
        w = 1./w
    end

    w = w / mean(w)

    ar = sparse(ai,aj,w,n,n)
    ar = ar + ar';
    return ar
end

"""Builds the kth wted chimeric graph on n vertices.
It does this by resetting the random number generator seed.
It should captute the state of the generator before that and then
return it, but it does not yet."""
function wtedChimera(n::Integer, k::Integer)
    srand(100*n+k)
    g = wtedChimera(n)
    return g
end

function semiWtedChimera(n::Integer, k::Integer)
    srand(100*n+k)
    g = semiWtedChimera(n)
    return g
end

"""Generate a chimera, and then apply a random weighting scheme"""
function wtedChimera(n::Integer)
    return randWeight(semiWtedChimera(n))
end


using ..Galerkin:
    galerkin_matrix,
    galerkin_projection!,
    galerkin_projection  # for Documenter only...

using LinearAlgebra:
    cholesky!,
    Cholesky,
    ldiv!

@doc raw"""
    MinimiseL2Error <: AbstractApproxMethod

Approximate a given function ``f(x)`` by minimisation of the ``L^2`` distance
between ``f`` and its spline approximation ``g(x)``.

# Extended help

Minimises the ``L^2`` distance between the two functions:

```math
{\left\lVert f - g \right\rVert}^2 = \left< f - g, f - g \right>,
```

where

```math
\left< u, v \right> = ∫_a^b u(x) \, v(x) \, \mathrm{d}x
```

is the inner product between two functions, and ``a`` and ``b`` are the
boundaries of the prescribed B-spline basis.
Here, ``g`` is the spline ``g(x) = ∑_{i = 1}^N c_i \, b_i(x)``, and
``\{ b_i \}_{i = 1}^N`` is a prescribed B-spline basis.

One can show that the optimal coefficients ``c_i`` minimising the ``L^2`` error
are the solution to the linear system ``\bm{M} \bm{c} = \bm{φ}``,
where ``M_{ij} = \left< b_i, b_j \right>`` and ``φ_i = \left< b_i, f \right>``.
These two terms are respectively computed by [`galerkin_matrix`](@ref) and
[`galerkin_projection`](@ref).

The integrals associated to ``\bm{M}`` and ``\bm{φ}`` are computed via
Gauss--Legendre quadrature.
The number of quadrature nodes is chosen as a function of the order ``k`` of the
prescribed B-spline basis, ensuring that ``\bm{M}`` is computed exactly (see
also [`galerkin_matrix`](@ref)).
In the particular case where ``f`` is a polynomial of degree ``k - 1``, this
also results in an exact computation of ``\bm{φ}``.
In more general cases, as long as ``f`` is smooth enough, this is still expected
to yield a very good approximation of the integral, and thus of the optimal coefficients ``c_i``.

"""
struct MinimiseL2Error <: AbstractApproxMethod end

function approximate(f, B::AbstractBSplineBasis, m::MinimiseL2Error)
    T = typeof(f(first(knots(B))))
    S = Spline{T}(undef, B)
    M = galerkin_matrix(B)  # by default it's a BandedMatrix

    # We annotate the return type to avoid inference issue in ArrayLayouts...
    # https://github.com/JuliaMatrices/ArrayLayouts.jl/issues/66
    Mfact = cholesky!(M) :: Cholesky{eltype(M), typeof(parent(M))}

    data = (; M = Mfact)
    A = SplineApproximation(m, S, data)
    approximate!(f, A)
end

function _approximate!(f, A, m::MinimiseL2Error)
    @assert method(A) === m
    S = spline(A)
    cs = coefficients(S)
    galerkin_projection!(f, cs, basis(S))  # computes rhs onto cs
    ldiv!(data(A).M, cs)  # now cs = M \ rhs
    A
end


module TSNE

using LinearAlgebra, ProgressBars
using Statistics: mean, mean!
using Printf: @printf, @sprintf

export tsne

"""
    Hbeta!(P::AbstractVector, D::AbstractVector, beta::Number)

Compute the point perplexities `P` given its squared distances to the other
points `D` and the precsion of Gaussian distribution `beta`.
"""
function Hbeta!(P::AbstractVector, D::AbstractVector, beta::Number)
    @inbounds P .= exp.(D .* -beta)
    sumP = sum(P)
    H = log(sumP) + beta * dot(D, P) / sumP
    @inbounds P ./= sumP
    return H
end

"""
    x2p(D::AbstractMatrix{T}, tol::Number = 1e-5, perplexity::Number = 30.0)

Convert `n×n` squared distances matrix `D` into `n×n` perplexities matrix `P`.
Performs a binary search to get P-values in such a way that each conditional
Gaussian has the same perplexity.
"""
function x2p(X::AbstractMatrix{T}, tol::Number = 1e-5,
     perplexity::Number = 30.0; max_iter::Integer = 50,
	 progress = true) where T <: Number
    # Initializing some variables
    n, d = size(X)
    sum_X = sum(x->x^2, X, dims = 2)
	D = fill(zero(T), (n, n))
	BLAS.gemm!('N', 'T', -2.0, X, X, 0.0, D)
    D .+= sum_X .+ sum_X'
    P = fill(zero(T), n, n)
    beta = fill(one(T), n)
    logU = log(perplexity)
    Di = fill(zero(T), n)
    thisP = similar(Di)
    # Loop over all datapoints
	if progress
		pb = ProgressBar(1:n)
		set_description(pb, "Computing P-values...")
	else
		pb = 1:n
	end
    for i in pb
        # Compute the Gaussian kernel and entropy for the current precision
        betamin = 0.0
        betamax = Inf
        betai = 1.0
        copyto!(thisP, view(P, i, :))
        copyto!(Di, view(D, :, i))
        Di[i] = prevfloat(Inf) 		 # exclude D[i,i] from minimum(), yet make it finite and exp(-D[i,i])==0.0
        minD = minimum(Di)     		 # distance of i-th point to its closest neighbour
        @inbounds Di .-= minD  		 # entropy is invariant to offsetting Di, which helps to avoid overflow
        H = Hbeta!(thisP, Di, betai)

        Hdiff = H - logU
        tries = 0

        while abs(Hdiff) > tol && tries < max_iter
            # If not, increase or decrease precision
            if Hdiff > 0.0
                betamin = betai
                betai = isfinite(betamax) ? (betai + betamax) / 2 : betai * 2
            else
                betamax = betai
                betai = (betai + betamin) / 2
            end
            # Recompute the values
            H = Hbeta!(thisP, Di, betai)
            Hdiff = H - logU
            tries += 1
        end
        # Set the final row of P
        @inbounds P[i, :] .= thisP
        beta[i] = betai
    end
    # Return final P matrix
    @printf("Mean σ: %f\n", mean(sqrt.(1.0 ./ beta)))
    return P
end

"""
    pca(X::AbstractMatrix, ndims::Integer = 50)

Run PCA on `X` to reduce the number of its dimensions to `ndims`.
"""
function pca(X::AbstractMatrix, ndims::Integer = 50)
    (n, d) = size(X)
    (d <= ndims) && return X
    Y = X .- mean(X, dims = 1)
    C = Symmetric((Y' * Y) ./ (n - 1))
    Ceig = eigen(C, (d - ndims + 1):d) # take eigvects for top ndims largest eigvals
    return Y * reverse(Ceig.vectors, dims = 2)
end

"""
    tsne(X::AbstractMatrix{T}, no_dims=2, initial_dims=50,
     max_iter::Integer=1000, perplexity=30.0) where T<:Number

Apply t-SNE (t-Distributed Stochastic Neighbor Embedding) to `X`,
i.e. embed its points into `ndims` dimensions preserving close neighbours.

Returns the `point×ndims` matrix of calculated embedded coordinates.

Different from original implementation: the default is not to use PCA for initialization.

### Arguments
	* `no_dims` determines the number of dimensions in the final t-SNE embedding
	* `initial_dims` the number of dimensions of the dataset after apllying
	PCA to initialize the solution
	* `max_iter` how many iterations of t-SNE to perform
	* `perplexity` the number of "effective neighbours" of a datapoint,
	usually increases with the amount of points in the dataset. Typical
	values are between 5 and 50, the default is 30
	* `min_gain`, `eta`, `cheat_scale`, `initial_momentum`, `final_momentum`,
	`stop_cheat_iter`, `momentum_switch_iter` low level parameters of t-SNE optimization
"""
function tsne(X::AbstractMatrix{T}, no_dims = 2, initial_dims::Integer = 50,
     max_iter::Integer = 1000, perplexity::Number = 30.0;
	 initial_momentum::Number = 0.5, final_momentum = 0.8, eta::Integer = 500,
	 min_gain::Number = 0.01, cheat_scale::Number = 4.0, progress = true,
	 stop_cheat_iter::Integer = 100, momentum_switch_iter::Integer = 20) where T <: Number

	X = pca(X, initial_dims)
    n, d = size(X)
    Y = randn(n, no_dims)			  # embedded final matrix
    dY = fill(zero(T), n, no_dims)    # gradient vector
    iY = fill(zero(T), n, no_dims)    # momentum vector
    gains = fill(one(T), n, no_dims)  # how much momentum is affected by gradient
    P = x2p(X, 1e-5, perplexity)
    P .+= P'						  # symmetriztion
    P .*= cheat_scale / sum(P)		  # early exaggeration + normalization
    P .= max.(P, 1e-12)

    # Pre-allocating some matrixes
	L = fill!(similar(P), zero(T))	  # temp matrix for Student-t and gradient steps
    Q = fill!(similar(P), zero(T))	  # temp matrix with low dimensional probabilities
	sum_Y = fill(zero(T), n)
	Y_mean = fill(zero(T), 1, no_dims)
	error = similar(P)
	last_error = NaN

    # Run iterations
	pb = progress ? ProgressBar(1:max_iter) : 1:max_iter
    for iter in pb
        # Compute pairwise affinities
        sum!(x->x^2, sum_Y, Y)
		# L = 2YY'
		BLAS.gemm!('N', 'T', -2.0, Y, Y, 0.0, L)
		# Student-t Distribution
		# L = 1/(1 + sum_Y + sum_Y' + L)
		@inbounds for j in 1:size(L, 2)
			@inbounds for i in 1:size(L, 1)
				L[i, j] = 1 / (1 + sum_Y[j] + sum_Y[i] + L[i, j])
			end
		end
		fill!(sum_Y, 0.0)
        inv_sum_Q = 1.0 / sum(L)
		@inbounds for j = 1:size(Q, 2)
			Pj = view(P, :, j)
			Qj = view(Q, :, j)
			Lj = view(L, :, j)
			# Diagonal should be zero
			Lj[j] = 0.0
			@inbounds for i = 1:size(Lj, 1)
				Qj[i] = ifelse(Lj[i] > 1e-12, Lj[i] * inv_sum_Q, 1e-12)
				# Reusing L for gradient step
				Lj[i] *= (Pj[i] - Qj[i])
				# Reusing sum_Y for column sums
				sum_Y[i] += Lj[i]
			end
		end
		# Compute gradient
		@inbounds for (i, ldiag) in enumerate(sum_Y)
			L[i, i] -= ldiag
		end
		# dY = -4LY
		BLAS.gemm!('N', 'N', -4.0, L, Y, 0.0, dY)
        # Perform the update
        momentum = iter <= momentum_switch_iter ? initial_momentum : final_momentum
        @inbounds for i in eachindex(gains)
            gains[i] = max(ifelse(((dY[i] > 0.) == (iY[i] > 0.)),
                                gains[i] * 0.8,
                                gains[i] + 0.2),
                                min_gain)
            iY[i] = momentum * iY[i] - eta * (gains[i] * dY[i])
            Y[i] += iY[i]
        end
        @inbounds Y .-= mean!(Y_mean, Y)
        # Compute current value of cost function
        if iter % 50 == 0
			map!((x, y)->x * log(x / y), error, P, Q)
			last_error = sum(error)
        end

		progress && set_description(pb, string(@sprintf("Error: %.4f", last_error)))

        # Stop lying about P-values
        if iter == stop_cheat_iter
            P ./= 4.0
        end
    end
    # Return solution
    return Y
end

end  # module TSNE


###################################
## Example code
## for Bayesian Gaussin Mixture Model

using PyPlot, PyCall
push!(LOAD_PATH,".")
import GaussianMixtureModel

"""
Visualize data & estimation in 2D space.
"""
function visualize_2D(X::Matrix{Float64}, S::Matrix{Float64}, S_est::Matrix{Float64}, text)
    cmp = get_cmap("jet")

    K1 = size(S, 1)
    K2 = size(S_est, 1)
    col1 = [pycall(cmp.o, PyAny, Int(round(val)))[1:3] for val in linspace(0,255,K1)]
    col2 = [pycall(cmp.o, PyAny, Int(round(val)))[1:3] for val in linspace(0,255,K2)]

    f, (ax1, ax2) = subplots(1,2,num=text)
    f[:clf]()
    f, (ax1, ax2) = subplots(1,2,num=text)

    for k in 1 : K1
        ax1[:scatter](X[1, S[k,:].==1], X[2, S[k,:].==1], color=col1[k])
    end
    ax1[:set_title]("truth")

    for k in 1 : K2
        ax2[:scatter](X[1, S_est[k,:].==1], X[2, S_est[k,:].==1], color=col2[k])
    end

    ax2[:set_title]("estimation")
end

"""
Run a test script for 2D data clustering.
"""
function test_2D()
    ## set model
    D = 2 # data dimension
    K = 4 #  number of mixture components
    alpha = 100.0 * ones(K)
    beta = 0.1
    m = zeros(D)
    nu = D + 1.0
    W = eye(D)
    cmp = [GaussianMixtureModel.GW(beta, m, nu, W) for _ in 1 : K]
    bgmm = GaussianMixtureModel.BGMM(D, K, alpha, cmp)

    ## generate data
    N = 300
    gmm = GaussianMixtureModel.sample_GMM(bgmm)
    X, S = GaussianMixtureModel.sample_data(gmm, N)

    ## inference
    max_iter = 100
    tic()
    S_est, post_bgmm, VB = GaussianMixtureModel.learn_VI(X, bgmm, max_iter)
    #S_est, post_bgmm, VB = GaussianMixtureModel.learn_GS(X, bgmm, max_iter)
    #S_est, post_bgmm, VB = GaussianMixtureModel.learn_CGS(X, bgmm, max_iter)
    toc()

    ## plot
    visualize_2D(X, S, GaussianMixtureModel.winner_takes_all(S_est), "2D plot")

    # VB check
    figure("ELBO")
    clf()
    plot(VB)
    ylabel("ELBO")
    xlabel("iterations")
    show()
end

test_2D()


# transfer_operators.jl : tranform quantities from fine to coarse grid and back

# TransferKind
abstract type TransferKind end

struct Injection <: TransferKind end

struct FullWeighting <: TransferKind end

struct HalfWeighting <: TransferKind end

struct Cubic <: TransferKind end

Base.broadcastable(kind::TransferKind) = Ref(kind)

# TransferOperator
abstract type TransferOperator end

struct Restriction <: TransferOperator end

struct Interpolation <: TransferOperator end

# GridTransferOperator
struct GridTransferOperator{K<:TransferKind,O<:TransferOperator}
    kind::K
    op::O
end

# stencil definitions
stencil(op::GridTransferOperator{Injection,Restriction}) = @SVector [0,1.,0]

stencil(op::GridTransferOperator{FullWeighting,Restriction}) = @SVector [1/4,1/2,1/4]

stencil(op::GridTransferOperator{FullWeighting,Interpolation}) = @SVector [1/2,1,1/2]

stencil(op::GridTransferOperator{Cubic,Interpolation}) = @SVector [-1/16,0,9/16,1,9/16,0,-1/16]

# TODO: half weighting needs a 2d stencil...

# restriction
R₁(op::TransferKind,n::Int) = expand(GridTransferOperator(op,Restriction()),n) # 1d restriction operator
R(op::TransferKind,n::Int) = R₁(op,n)
R(op::TransferKind,n::Int...) = kron(R₁.(op,n)...)
restrict(u::Vector{T},op::TransferKind,n::Int...) where {T<:AbstractFloat} = R(op,n...)*u

# interpolation
P₁(op::TransferKind,n::Int) = expand(GridTransferOperator(op,Interpolation()),2n) # 1d interpolation operator
P(op::TransferKind,n::Int) = P₁(op,n)
P(op::TransferKind,n::Int...) = kron(P₁.(op,n)...)
interpolate(u::Vector{T},op::TransferKind,n::Int...) where {T<:AbstractFloat} = P(op,n...)*u

# prolongation = interpolation
prolongate(u,op) = interpolate(u,op)

# expand
function expand(op::GridTransferOperator{K,O} where {K,O},n)
    st = stencil(op)
    R = CartesianIndices((n-1,))
    I1, Iend = extrema(R)
    Is = Int[]
    Js = Int[]
    Vs = Float64[]
    @inbounds for I in R
        if iseven(I.I[1])
            is,js,vs = _stencil2mat(st,I,I1,Iend)
            push!(Is,is.>>1...)
            push!(Js,js...)
            push!(Vs,vs...)
        end
    end
    operator2matrix(op,Is,Js,Vs)
end

operator2matrix(op::GridTransferOperator{K,Interpolation} where {K},Is,Js,Vs) = sparse(Js,Is,Vs)
operator2matrix(op::GridTransferOperator{K,Restriction} where {K},Is,Js,Vs) = sparse(Is,Js,Vs)


"Implements Zeckendorf's theorem and outputs a Zeckendorf sequence {cᵪ | c ∈ {0, 1}}"
function zeckendorf{T <: Integer}(n::T)
    @assert n >= 0 "n is required to be larger or equal than zero"

    x, z = 2, 2
    fib_cache = [1, 1]

    # find fib(z-1) <= n <= fib(z)
    while z <= n
        x += 1
        z = fib_cache[x-1] + fib_cache[x-2]
        push!(fib_cache, z)
    end

    c = falses(x-1)

    for i in (x-1):-1:1
        if fib_cache[i] <= n
            n -= fib_cache[i]
            c[i] = true
        end
    end

    return c
end

"Generates a binary string from a boolean array in reverse order"
binary_string{T <: AbstractArray{Bool}}(A::T) = join(map(Int, reverse(A)))


# Poisson

"""
poipmf!(lambda, prob; left = 0, right = length(prob)-1+left)
poipmf(lambda, right; left = 0)

Compute the p.m.f. of Poisson distribution with mean lambda.
`left` and `right` are the domain of Poisson distribution.
They should be choosen so that the total probability in the domain becomes 1.
In `poipmf!``, the p.m.f. is saved to the vector `prob`.
The right can be obtained from `rightbound`.
The retuen value is the normalizing constant so that the total sum of `prob` is 1, called the weight.
`poipmf` returns a tuple of (weight, prob).
"""

@origin (prob => left) function poipmf!(lambda::Tv, prob::Vector{Tv};
    left::Ti = 0, right::Ti = length(prob)-1+left) where {Tv, Ti}
    @inbounds begin
        log2piOver2::Tv = log(2*pi) / 2
        mode::Ti = floor(Ti, lambda)
        if mode >= 1
            prob[mode] = exp(-lambda + mode * log(lambda)
                - log2piOver2 - (mode + 1/2) * log(mode) + mode)
        else
            prob[mode] = exp(-lambda)
        end
        # down
        for j = mode:-1:left+1
            prob[j-1] = j / lambda * prob[j]
        end
        # up
        for j = mode:right-1
            prob[j+1] = lambda / (j+1) * prob[j]
        end
        # compute W
        weight::Tv = 0
        s::Ti = left
        t::Ti = right
        while s < t
            if prob[s] <= prob[t]
                weight += prob[s]
                s += 1
            else
                weight += prob[t]
                t -= 1
            end
        end
        weight += prob[s]
    end
end

function poipmf(lambda::Tv, right::Ti; left::Ti = 0) where {Tv, Ti}
    prob = Vector{Tv}(undef, right-left+1)
    weight = poipmf!(lambda, prob, left=left, right=right)
    (weight, prob)
end

"""
cpoipmf!(lambda, prob, cprob; left = 0, right = length(prob)-1+left)
cpoipmf(lambda, right; left = 0)

Compute the p.m.f. and complementary c.d.f. of Poisson distribution with mean lambda.
`left` and `right` are the domain of Poisson distribution.
They should be choosen so that the total probability in the domain becomes 1.
In `cpoipmf!``, the p.m.f. and c.d.f. are stored to `prob` and `cprob`, respectively.
The right can be obtained from `rightbound`.
The retuen value is the normalizing constant so that the total sum of `prob` is 1, called the weight.
`cpoipmf` returns a tuple of (weight, prob, cprob).
"""

@origin (prob => left, cprob => left) function cpoipmf!(lambda::Tv, prob::Vector{Tv}, cprob::Vector{Tv}; left::Ti = 0, right::Ti = length(prob)-1+left) where {Tv, Ti}
    weight::Tv = poipmf!(lambda, prob, left=left, right=right)
    @inbounds begin
        cprob[right] = 0
        for k = right-1:-1:left
            cprob[k] = cprob[k+1] + prob[k+1]
        end
        weight
    end
end

function cpoipmf(lambda::Tv, right::Ti; left::Ti = 0) where {Tv, Ti}
    prob = Vector{Tv}(undef, right-left+1)
    cprob = Vector{Tv}(undef, right-left+1)
    weight = cpoipmf!(lambda, prob, cprob, left=left, right=right)
    (weight, prob, cprob)
end

"""
rightbound(::Type{Ti} = Int, lambda::Tv, q::Tv = Tv(1.0e-8))

Compute the rightbound of Poisson distribution with mean lambda.
The rightbound finds a quantile so that the complementary c.d.f. becomes `q`.
"""

function rightbound(lambda::Tv, q::Tv = Tv(1.0e-8))::Int where {Tv}
    rightbound(Int, lambda, q)
end

function rightbound(::Type{Ti}, lambda::Tv, q::Tv = Tv(1.0e-8))::Ti where {Tv, Ti}
    z = cquantile(Normal(), q)
    if lambda < 3.0
        ll = exp(-lambda)
        total = ll
        right::Ti = 0
        while true
            right += 1
            ll *= lambda / right
            total += ll
            if total + q >= 1.0
                break
            end
        end
        right
    else
        right = floor(Ti, (z + sqrt(4.0 * lambda - 1.0))^2 / 4.0 + 1.0)
    end
end


"""
    GaussianLikelihood(σ²)

Gaussian likelihood with `σ²` variance. This is to be used if we assume that the
uncertainity associated with the data follows a Gaussian distribution.

```math
    p(y|f) = Normal(y | f, σ²)
```
On calling, this would return a normal distribution with mean `f` and variance σ².
"""
struct GaussianLikelihood{T<:Real}
    σ²::T
end

GaussianLikelihood() = GaussianLikelihood(1e-6)

@functor GaussianLikelihood

(l::GaussianLikelihood)(f::Real) = Normal(f, sqrt(l.σ²))

(l::GaussianLikelihood)(fs::AbstractVector{<:Real}) = MvNormal(fs, sqrt(l.σ²))

"""
    HeteroscedasticGaussianLikelihood(σ²)

Heteroscedastic Gaussian likelihood.
This is a Gaussian likelihood whose mean and the log of whose variance are functions of the
latent process.

```math
    p(y|[f, g]) = Normal(y | f, exp(g))
```
On calling, this would return a normal distribution with mean `f` and variance `exp(g)`.
"""
struct HeteroscedasticGaussianLikelihood end

(::HeteroscedasticGaussianLikelihood)(f::AbstractVector{<:Real}) = Normal(f[1], exp(f[2]))

(::HeteroscedasticGaussianLikelihood)(fs::AbstractVector) = MvNormal(first.(fs), exp.(last.(fs)))


function getPost(testResult)

    # Create a Dictionary containing the possible combinations:
    P = Dict()

    # Idea: solve by recursive calls of function,
    # constantly updating the prior --> not implemented

    deseaseFrequency = 0.001
    truePositives    = 0.99
    falsePositives   = 0.05

    ### Priors:
    P[":("] = deseaseFrequency
    P[":)"] = 1.0 - P[":("]

    ### Likelihoods
    # Positive test restuls, given status:
    P["+|:("] = truePositives
    P["+|:)"] = falsePositives
    # Negative test restuls, given status:
    P["-|:("] = 1.0 - truePositives
    P["-|:)"] = 1.0 - falsePositives

    # Update for each new measurement:
    for i in 1:length(testResult)
        # Marginal Likelihoods:
        P["+"] = P["+|:("] * P[":("] + P["+|:)"] * P[":)"]
        P["-"] = P["-|:("] * P[":("] + P["-|:)"] * P[":)"]

        ### Posteriors:
        P[":(|+"] = P["+|:("] * P[":("] / P["+"]
        P[":(|-"] = P["-|:("] * P[":("] / P["-"]
        P[":)|+"] = 1.0 - P[":(|+"]
        P[":)|-"] = 1.0 - P[":(|-"]

        # Update priors to the posteriors:
        P[":("] = P[string(":(|",testResult[i])]
        P[":)"] = 1.0 - P[":("]
    end

    # Desired Posteriors are the new Priors:
    # If no test result, then this also works, because the posterior = prior
    posteriors = [P[":("], P[":)"]]

    return posteriors
end

# Enter the received results:
myResult = "+-"

# Show the posteriors for each step when receiving test results,
# including "no result" at the start:
for i in 1:length(myResult)+1
    print("\nAccumulated Results: ")
    println(myResult[1:i-1])
    println("[P(:(|T), P(:)|T)] = ",getPost(myResult[1:i-1]))
    print("Sanity check: Sum(P) = ")
    println(sum(getPost(myResult[1:i-1])))
end


## --- Hafnium isotopes

    # Calculate the initial Hf ratio and epsilon Hf at time t Ma
    function eHf(Hf176_Hf177, Lu176_Hf177, t; eHfOnly::Bool=true)

        # Lutetium decay constant (Soderlund et al., 2004
        lambda = 1.867E-11

        # Present-day CHUR composition (Bouvier et al., 2008)
        CHUR_Hf176_Hf177 = 0.282785
        CHUR_Lu176_Hf177 = 0.0336

        # Calculate initial Hf ratio at time t
        Hf176_Hf177_t = Hf176_Hf177 .- Lu176_Hf177.*(exp.(t .* 10^6*lambda) .- 1)

        # Calculate CHUR Hf ratio at time t
        CHUR_Hf176_Hf177_t = CHUR_Hf176_Hf177 .- CHUR_Lu176_Hf177.*(exp.(t .* 10^6*lambda) .- 1)

        # Calculate corresponding epsilon Hf
        eHf=(Hf176_Hf177_t ./ CHUR_Hf176_Hf177_t .- 1) .* 10^4

        if eHfOnly
            return eHf
        else
            return (eHf, Hf176_Hf177_t)
        end
    end
    export eHf

    function bin_bsr_eHf(x,Hf176_Hf177,Lu176_Hf177,age,min,max,nbins,x_sigma,Hf176_Hf177_sigma,Lu176_Hf177_sigma,age_sigma,nresamples)
        data = hcat(x,Hf176_Hf177,Lu176_Hf177,age)
        sigma = hcat(x_sigma,Hf176_Hf177_sigma,Lu176_Hf177_sigma,age_sigma)

        means = Array{Float64}(undef,nbins,nresamples)
        c = Array{Float64}(undef,nbins)
        for i=1:nresamples
            dbs = bsresample(data,sigma,length(age))
            eHf_resampled = eHf(dbs[:,2], dbs[:,3], dbs[:,4])
            (c,m,s) = binmeans(dbs[:,1], eHf_resampled, min, max, nbins)
            means[:,i] = m
        end

        m = nanmean(means,dim=2)
        el = m - nanpctile(means,2.5,dim=2)
        eu = nanpctile(means,97.5,dim=2) - m

        return (c, m, el, eu)
    end
    export bin_bsr_eHf

## ---


# This file is a part of AstroLib.jl. License is MIT "Expat".
# Copyright (C) 2016 Mosè Giordano.

function _rhotheta{T<:AbstractFloat}(period::T, periastron::T, eccentricity::T,
                                     semimajor_axis::T, inclination::T,
                                     omega::T, omega2::T, epoch::T)
    rho = theta = -one(period)
    # See chapter 55.
    n = 360.0/period
    M = deg2rad(n*(epoch - periastron))
    E = kepler_solver(M, eccentricity)
    r  = semimajor_axis*(1.0 - eccentricity*cos(E))
    nu = trueanom(E, eccentricity)
    # Convert variables in radians.
    omega2      = deg2rad(omega2)
    inclination = deg2rad(inclination)
    omega       = deg2rad(omega)
    theta = omega + atan2(sin(nu + omega2)*cos(inclination), cos(nu + omega2))
    rho   = r*cos(nu + omega2)/cos(theta - omega)
    # Convert theta to degrees and for it to be in [0, 360) range.
    theta = cirrange(rad2deg(theta))
    return rho, theta
end

"""
    rhotheta(period, periastron, eccentricity, semimajor_axis, inclination, omega, omega2, epoch) -> rho, theta

### Purpose ###

Calculate the separation and position angle of a binary star.

### Explanation ###

This function will return the separation \$\\rho\$ and position angle
\$\\theta\$ of a visual binary star derived from its orbital elements.  The
algorithms described in the following book will be used: Meeus J., 1992,
Astronomische Algorithmen, Barth.  Compared to the examples given at page 400
and no discrepancy found.

### Arguments ###

* `period`: period [year]
* `periastro`: time of periastron passage [year]
* `eccentricity`: eccentricity of the orbit
* `semimajor_axis`: semi-major axis [arc second]
* `inclination`: inclination angle [degree]
* `omega`: node [degree]
* `omega2`: longitude of periastron [degree]
* `epoch`: epoch of observation [year]

All input parameters have to be scalars.

### Output ###

The 2-tuple \$(\\rho, \\theta)\$, where

* \$\\rho\$: separation [arc second]
* \$\\theta\$: position angle [degree]

### Example ###

Find the position of Eta Coronae Borealis at the epoch 2016

``` julia
ρ, θ = rhotheta(41.623, 1934.008, 0.2763, 0.907, 59.025, 23.717, 219.907, 2016)
# => (0.6351167848228113,214.42513388052114)
```

### Notes ###

Code of this function is based on IDL Astronomy User's Library.
"""
rhotheta(period::Real, periastron::Real, eccentricity::Real,
         semimajor_axis::Real, inclination::Real,
         omega::Real, omega2::Real, epoch::Real) =
             _rhotheta(promote(float(period), float(periastron),
                               float(eccentricity), float(semimajor_axis),
                               float(inclination), float(omega),
                               float(omega2), float(epoch))...)


#############################################################################
#############################################################################
#
# This file implements factorization
#
#############################################################################
#############################################################################

"""
Factors a polynomial over the field Z_p.

Returns a vector of tuples of (irreducible polynomials (mod p), multiplicity) such that their product of the list (mod p) is f. Irreducibles are fixed points on the function factor.
"""
function factor(f::Polynomial, prime::Int)::Vector{Tuple{Polynomial,Int}}
    #Cantor Zassenhaus factorization

    f_modp = mod(f, prime)
    degree(f_modp) ≤ 1 && return [(f_modp,1)]

    # make f primitive
    ff = prim_part(f_modp)(prime)
    # @show "after prim:", ff

     # make f square-free
    squares_poly = gcd(f, derivative(ff), prime)
    ff = (ff ÷ squares_poly)(prime)
    # @show "after square free:", ff

    # make f monic
    old_coeff = leading(ff).coeff
    ff = (ff ÷ old_coeff)(prime)
    # @show "after monic:", ff

    dds = dd_factor(ff, prime)

    ret_val = Tuple{Polynomial,Int}[]

    for (k,dd) in enumerate(dds)
        sp = dd_split(dd, k, prime)
        sp = map((p)->(p ÷ leading(p).coeff)(prime),sp) #makes the polynomials inside the list sp, monic
        for mp in sp
            push!(ret_val, (mp, multiplicity(f_modp,mp,prime)) )
        end
    end

    #Append the leading coefficient as well
    push!(ret_val, (leading(f_modp).coeff* one(Polynomial), 1) )

    return ret_val
end

"""
Expand a factorization.
"""
function expand_factorization(factorization::Vector{Tuple{Polynomial,Int}})::Polynomial
    length(factorization) == 1 && return first(factorization[1])^last(factorization[1])
    return *([first(tt)^last(tt) for tt in factorization]...)
end

"""
Compute the number of times g divides f
"""
function multiplicity(f::Polynomial, g::Polynomial, prime::Int)::Int
    degree(gcd(f, g, prime)) == 0 && return 0
    return 1 + multiplicity((f ÷ g)(prime), g, prime)
end

"""
Distinct degree factorization.

Given a square free polynomial `f` returns a list, `g` such that `g[k]` is a product of irreducible polynomials of degree `k` for `k` in 1,...,degree(f) ÷ 2, such that the product of the list (mod `prime`) is equal to `f` (mod `prime`).
"""
function dd_factor(f::Polynomial, prime::Int)::Array{Polynomial}
    x = x_poly()
    w = deepcopy(x)
    g = Array{Polynomial}(undef,degree(f)) #Array of polynomials indexed by degree

    #Looping over degrees
    for k in 1:degree(f)
        w = rem(pow_mod(w,prime,prime), f)(prime)
        g[k] = gcd(w - x, f, prime)
        f = (f ÷ g[k])(prime)
    end

    #edge case for final factor
    f != one(Polynomial) && push!(g,f)

    return g
end

"""
Distinct degree split.

Returns a list of irreducible polynomials of degree `d` so that the product of that list (mod prime) is the polynomial `f`.
"""
function dd_split(f::Polynomial, d::Int, prime::Int)::Vector{Polynomial}
    f = mod(f,prime)
    degree(f) == d && return [f]
    degree(f) == 0 && return []
    w = rand(Polynomial, degree = d, monic = true)
    w = mod(w,prime)
    n_power = (prime^d-1) ÷ 2
    g = gcd(pow_mod(w,n_power,prime) - one(Polynomial), f, prime)
    ḡ = (f ÷ g)(prime) # g\bar + [TAB]
    return vcat(dd_split(g, d, prime), dd_split(ḡ, d, prime) )
end


function set_up_operator(p::Param)

    # Variables
    rmin        = p.rmin
    rmax        = p.rmax
    rnodes      = p.rnodes

    thnodes     = p.thnodes

    #Create position vectors
    dr = (rmax - rmin)/(rnodes-1)
    r = [rmin + (i-1)*dr for i in 1:rnodes]

    dth = 2*pi/(thnodes)
    theta = [ (i-1)*dth for i in 1:thnodes]

    #Create Differential Operators
    Dr  =   Diff_Operator_2D(1 ,1, [dr,dth] , [rnodes,thnodes])
    Drr =   Diff_Operator_2D(2 ,1, [dr,dth] , [rnodes,thnodes])

    Dp  =  Diff_Operator_2D_Periodic(1 ,2, [dr,dth] , [rnodes,thnodes])
    Dpp =  Diff_Operator_2D_Periodic(2 ,2, [dr,dth] , [rnodes,thnodes])

    one_over_r2 = spzeros(rnodes*thnodes,rnodes*thnodes)

    for i in 1:rnodes*thnodes
        one_over_r2[i,i] = (1/r[1 + (i-1)%rnodes])^2
    end

    RHS_Opp = Drr + one_over_r2*Dpp + (0.25)*one_over_r2
    #sparse(Drr + (0.25)*one_over_r2*I  + one_over_r2*Dpp);

    av1 = zeros(rnodes*thnodes)
    av2 = zeros(rnodes*thnodes)

    #Operator structure
    return Opps(r, theta , av1 , av2  , Dr, Dp, RHS_Opp)

end


using Plots, LaTeXStrings; pyplot()

xGrid = 0:0.01:10
uGrid = 0:0.01:1
busy = 0.8

F(t)= t<=0 ? 0 : 1 - busy*exp(-(1-busy)t)

infimum(B) = isempty(B) ? Inf : minimum(B)
invF(u) = infimum(filter((x) -> (F(x) >= u),xGrid))

p1 = plot(xGrid,F.(xGrid), c=:blue, xlims=(-0.1,10), ylims=(0,1),
	xlabel=L"x", ylabel=L"F(x)")

p2 = plot(uGrid,invF.(uGrid), c=:blue, xlims=(0,0.95), ylims=(0,maximum(xGrid)),
	xlabel=L"u", ylabel=L"F^{-1}(u)")

plot(p1, p2, legend=:none, size=(800, 400))