\documentclass[12pt,a4paper,onecolumn]{article}

\usepackage[top=0.6in, left=0.6in,right=0.6in,bottom=0.6in]{geometry}
\usepackage{hyperref}
\usepackage{natbib}
\usepackage{listings}
\usepackage{color}
\usepackage{graphicx}
\usepackage{mathtools}

\author{Dionisio Perez-Mavrogenis}
\title{ELEC6032 Assignment}

\newcommand*\justify{%
  \fontdimen2\font=0.4em% interword space
  \fontdimen3\font=0.2em% interword stretch
  \fontdimen4\font=0.1em% interword shrink
  \fontdimen7\font=0.1em% extra space
  \hyphenchar\font=`\-% allowing hyphenation
}

\begin{document}
\maketitle

\section{Part 1}

The assigned elliptic curve E curve over the finite field $Z_{17}$ is

$$ y^2 = x^3 + 4x + 3$$

together with the point at infinity, $O$.

\subsection{Find the order of your unique group}

\begin{table}[h]
\center
\caption{Field $E(Z_{17})$}
	\begin{tabular}{c c c c c c}
$(1,\pm5)$ & $(1,\pm12)$ & $(2,\pm6)$ & $(2,\pm11)$ & $(3,\pm5)$ & $(3,\pm12)$ \\
$(4,\pm7)$ & $(4,\pm10)$ & $(7,0)$    & $(11,\pm1)$ & $(11,\pm16)$ & $(13,\pm5)$ \\
$(13,\pm12)$ & $(14,\pm7)$ & $(14,\pm10)$ & $(15,\pm2)$ & $(15,\pm15)$ & $(16,\pm7)$ \\
$(16,\pm10)$ & $O$\\
	\end{tabular}
\label{table:points}
\end{table}

By the script given in Listing \ref{script_ecc} the cardinality of $E(Z_{17})$ is 38, the elements given in Table \ref{table:points}. Although one would expect that points should occur in pairs due to the symmetrical properties of elliptic curves, e.g. (15,2) and (15,-2), points with negative coordinates are not in $Z_{17}$, but there exist equivalent coordinates satisfying the equation, e.g. (15,15) instead of (15,-2).

\subsection{Is your group cyclic? Explain your answer}
Every elliptic curve over a field is an Abelian group under addition, with $O$ acting as the identity element. Every elliptic curve is a cyclic group or the product of two cyclic groups (i.e. has at most two generators), unless it is or prime order in which case it is cyclic \citep{website:cloudflare_primer} \citep{notes:ecc_notes}.

\subsection{Explain how elliptic curves can be used to construct secure ciphers}
The following follows closely \citep{book:ent}.

Two parties publicly agree on a prime number $p$, an elliptic curve $E$ over $Z/pZ$ (which is the field $Z_p$ of characteristic $p$) and a point $P\in E(Z_p)$. Then sender $A$ chooses a secret element $a$ and sends $aP$ to  $B$. $B$ also computes his secret element $b$ and sends $bP$ to $A$. They can both compute $abP$, which is their secret key. For an adversary to compute $n$ from $nP$ (or $m$) it would require solving the discrete logarithm problem for elliptic curves over a field.

The motivation to use elliptic curves is that the discrete logarithm problem is considered (mathematically) harder to solve for an elliptic curve over a field than it is for a finite field. Furthermore they are able to provide the same level of security as traditional PKI systems, with smaller key lengths (making them more attractive for using on smaller devices such as phones).\\

\subsection{Encrypt the following message using an appropriate elliptic curve cipher}
"A little knowledge is a dangerous thing"\\

\section{Part 2}
I believe this message is enciphered with a Vigenère cipher and the original text is:\\

\texttt{\justify the message starts here recently hardware trojans have attracted the attention of governments and researchers one of the main concerns is that integrated circuits in military or critical infrastructure applications could be maliciously manipulated during the manufacturing process which often takes place abroad however since there have been no reported hardware trojans in practice yet little is known about how such a trojan would look like and how hard to implement one in practice one example is dopant trojan this can be used to compromise the security of a meaningful real world target while avoiding detection by functional testing as well as trojan detection mechanisms such trojans can be used to establish a hidden side channel in an otherwise side channel resistant design this trojan does not change the logic value of any gate but instead changes only the power profile of two gates an evaluator who is not aware of the trojan cannot attack the trojan design using common side channel attacks the owner of the trojan however can use his knowledge of the trojan power model to establish a hidden side channel that reliably leaks out secret keys this is the end of the message}\\

This message looks like structured text and hence it might be encrypted with some mono-alphabetic substitution method. First I run it through a Caesar cracking tool with no meaningful results. Then I run it through \citep{tool:vigenere_cracker}, which yielded a meaningful message for key \texttt{xayxayxayxayxay}. However, by observing the key that \citep{tool:vigenere_cracker} derived one can see that it repeats itself and hence ,due to the nature of the cipher, the key could simply be \texttt{xay}. Furthermore Script \ref{code2} reports that the highest index of coincidence is found for a key length of 18, which is a multiple of 3.

In order to crack a Vigenère cipher, and due to this text's length which allows for meaningful statistical analysis, one can either employ the Kasisky test or the index of coincidence text to determine they key length. Once the key length has been determined one can treat the parts that correspond to each key-letter as simple mono-alphabetic substitution cipher and employ statistical methods.

\paragraph{Kasinsky Test} The Kasinski test is founded on the notion that texts which appear similar on the cipher-text will be the same plain-text that coincidentally was encrypted with the same key. The cryptanalyst looks for identical ciphertext bits that are at least three characters long and records the distance between those($\mathbf{\Delta} = {\Delta _1, \Delta _2, \dots , \Delta _n}$), as the distance is likely to be a multiple of the key-length (i.e. $\text{Length} = \text{gcd}(\mathbf{\Delta})$).

\paragraph{Index of Coincidence} The IoC method exploits the fact that letters in a language are not uniformly distributed in words and hence some letters will be more frequent than others. If the letter frequency was equal, then the probability of selecting the same random letter twice (in English) would be $\sum_{i=a}^{i=z}(1/26)^2 = 0.038$. Given the letter frequencies in English this probability is $\sum_{i=a}^{i=z}f_i ^2 = 0.067$, $f_i$ is the frequency of letter $i$. One can use this fact to test whether the plain-text was enciphered with a \textit{single-alphabet} cipher (e.g. Caesar) or not ,as the cipher-text's IoC would approximate that of the plain-text, i.e. 0.067/0.038 = 1.73. Once key-lengths that approach plain-text IoC are found they can be factored and a more systematic search can reveal likely key lengths.

\section{Part 3}
Here we knew that the second stage is an XOR cipher and the first two characters of the output, Hg. We get the key for the first two bytes by doing \texttt{Q" XOR Hg = 0x1945}(or 25 69 in decimal). A naive assumption worth investigating is to try this key for every two bytes of text, done by the script shown in Listing \ref{code1}. This produces the intermediate text shown bellow : \\

\texttt{\justify {Hgt ltsbuba pm dumt uy hgt lpyh wqpmpxbr sbr tdxyuct lvyhtqv pm hgtl sdd, xbobpib hp tctb hgt aqtshtyh lubry. Yxqtdv sbvpbt igp htddy vpx hgtv gsct hgt sbyitq uy npouba, lsr pq yulwdv luyhsotb. Hgtqt sqt lsbv hgubay hgsh lsot dumt ipqhg gpdruba pb hp sbr yscpxquba. Exh dumt uy xbwqtrukhsedt sbr it sqt pmhtb lvyhtquty tctb hp pxqytdcty. It hgubo yxkktyy, gswwubtyy, gtdwuba phgtqy, pq yxqwsyyuba pxqytdcty iudd lsot dumt ipqhg ducuba, exh it ksb sdisvy et iqpba pq mqxyhqshtr ev tctbhy. Hguy uy s qsbrpl wgqsyt. Wgudpypwgtqy gsct s dph hp ysv sepxh hgt csdxt pm sdd hgtyt hgubay, sbr s duhhdt dtyy hp ysv sepxh pbt pm hgt lpyh csdxsedt hgubay pm sdd: dpct. Yp it ksb et kdtsq tbpxag sepxh igsh uh ltsby mpq dumt hp gsct ltsbuba sbr csdxt, exh igtb it wxh rpib pxq wgudpypwgv eppoy sbr skhxsddv ath pb iuhg ducuba, ltsbuba sbr csdxt ksb et tdxyuct. Ducuba itdd uy lpqt sqh hgsb ykutbkt pq wgudpypwgv. Hgtqtmpqt,  hgt pbdv ytbyt it ksb lsot pm hgt urts hgsh dumt gsy ltsbuba uy hgsh hgtqt sqt yplt qtsypby hp duct qshgtq hgsb hp rut, sbr hgpyt qtsypby sqt hp et mpxbr ub hgt ducuba pm dumt uhytdm.}}\\

This text is sufficiently large to meaningful provide statistical information about its contents. More importantly, it looks like structured text, something which suggests that the first encryption stage might be a mono-alphabetic substitution. By using \citep{tool:design215} to get frequency overviews and perform substitutions, we get : \\

\texttt{\justify{the meaning of life is the most profound and elusive mystery of them all, unknown to even the greatest minds. surely anyone who tells you they have the answer is joking, mad or simply mistaken. there are many things that make life worth holding on to and savouring. but life is unpredicTable~and we are often mysteries even to ourselves. we think success, happiness, helping others, or surpassing ourselves will make life worth living, but we can always be wrong or frustrated by events. this is a random phrase. philosophers have a lot to say about the value of all these things, and a little less to say about one of the most valuable things of all: love. so we can be clear enough about what it means for life to have meaning and value, but when we put down our philosophy books and actually get on with living, meaning and value can be elusive. living well is more art than science or philosophy. therefore, the only sense we can make of the idea that life has meaning is that there are some reasons to live rather than to die, and those reasons are to be found in the living of life itself.}}\\

In order to understand the substitutions I looked for 1-letter words, which is most likely the article \texttt{a}. Having done that 2 and 3-letter words start appearing and analysing those is easier, as well as observing effects on longer words. Furthermore, the three-grams \texttt{SRB} , \texttt{SDD} and \texttt{NGT} are frequently found, and if we assume that \texttt{S} is really \texttt{a}, then \texttt{SRB} is \texttt{and} and \texttt{SDD} is \texttt{all}(that is the most likely scenario for these two words). Making these substitutions helped reveal more common words.

The next step that helped a lot was letter frequency, exploiTable~due to the texts length. The frequencies produced by \citep{tool:design215} are give in Table~\ref{freqs_q3}. Given the text's length it would be a fair to assume that some of the first letters are vowels, and hence substituting \texttt{T} for \texttt{e} gave further insight on the text. Now words with odd repetitions could be exploited, like \texttt{TCTB(eCeB)}, which contains an averagely common letter with another rare one. Furthermore, it would be safe to assume that \texttt{C} and \texttt{B} are both consonants. By trying words like \texttt{ever, even, eyes} revealed more information and the rest of the text was deciphered in a similar fashion.

\begin{table*}[t]
	\caption{\footnotesize Statistical information about the second-stage decrypted text. The top row statistics are for the English language, while the bottom refer to the text.}
	\label{freqs_q3}
	\resizebox{18cm}{!} {
	\begin{tabular}{c c c c c c c c c c c c c c c c c c c c c c c c c c}
E & T & A & O & I & N & S & H & R & D & L & C & U & M & W & F & G & Y & P & B & V & K & J & X & Q & Z\\
12.7 & 9.1 & 8.2 & 7.5 & 7.0 & 6.7 & 3.3 & 6.1 & 6.0 & 4.3 & 4.0 & 2.8 & 2.8 & 2.4 & 2.4 & 2.2 & 2.0 & 2.0 & 1.9 & 1.5 & 1.0 & 0.8 & 0.15 & 0.15 & 0.1 & 0.07\\
\hline
T & S & H & P & B & Y & U & D & G & Q & X & C & L & M & A & I & R & V & W & E & K & O & N & F & J & Z\\
113 & 73 & 72 & 68 & 67 & 65 & 56 & 52 & 49 & 41 & 28 & 21 & 21 & 21 & 20 & 19 & 19 & 17 & 15 & 14 & 11 & 8 & 1 & 0 & 0 & 0 \\
12.9 & 8.38 & 8.26 & 7.80 & 7.69 & 7.46 & 6.42 & 5.97 & 5.62 & 4.70 & 3.21 & 2.41 & 2.41 & 2.41 & 2.29 & 2.18 & 2.18 & 1.95 & 1.72 & 1.60 & 1.26 & 0.91 & 0.11 & 0 & 0 & 0\\
\hline
e & a & t & o & n & s & i & l & h & r & u & v & m & f & g & w & d & y & p & b & c & k & j \\

	\end{tabular}
}
\end{table*}

\section{Part 4}
In order to solve this part I used some online tools, as well as scripts that I wrote myself. In particular, I used \citep{tool:freq_analyzer} to perform frequency analysis and interactive letter substitutions, \citep{website:puzzle_info} to learn how the puzzles work and \citep{tool:design215} to search for words matching a given pattern.\\

Appendix \ref{headline_images} gives snapshots from the tool with the deciphered headlines and the letter frequencies.
\subsection{Headline 1}
\paragraph{Solution} I believe this headline is the text: \texttt{ntsb urges new ways to combat rising runway incidents}.

One thing to note on this headline is that the first and last words ,\texttt{YNTS} and \texttt{CYOCMBYNT}, contain the same trigraph, \texttt{YNT}. Furthermore \texttt{CYOCMBYNT} contains two repeated letters and its ending must make sense in another word. The latter reasoning regarding word endings turned out to be misleading, as NTSB is an acronym for National Transportation Safety Board.

After going through a couple of words produced with the pattern \texttt{---------/abcadebfg} by \citep{tool:design215}, the word \texttt{incident} seemed to make more sense in the context of a headline and how the substitutions affected other words. Substituting \texttt{inciMBnts} to the sentence, \texttt{HCTCYA} turned into \texttt{HisinA} and hence \texttt{A} is most likely \texttt{g}. After performing that substitution and \texttt{B} for \texttt{e} it appeared that \texttt{K} was most likely \texttt{w}. Now words like \texttt{QHges new wJVs} are appearing, suggesting text along the lines of \texttt{urges new ways} (\texttt{new} was a big give-away) and helping conclude that \texttt{R} is \texttt{o} after performing the substitutions so far.

After these previously mentioned steps, the rest of the text is fairly simple to deduce, contains no mistakes and makes sense in the context of a newspaper headline.

\subsection{Headline 2}
\paragraph{Result} I believe that this headline is the text: \texttt{dutch authorities closing in on human smuggling ring}.\\

To decipher this headline I firstly attempted to investigate the repetition of the letter \texttt{B} in \texttt{OAXBBJNHB} and the 9-letter words matching this pattern are shown in Table~\ref{words1}. Furthermore \texttt{OAXBBJNHB} ends in \texttt{HB}, as does \texttt{WNHB} and \texttt{YJVONHB}.

Additionally the two two-letter words \texttt{NH}
and \texttt{VH} are in the set shown in Table~\ref{words2}, both end with the letter \texttt{H} and should make sense syntactically and meaning-wise. The letter \texttt{H} in these words eliminates all words with \texttt{g, l} in their $8^{th}$ position from Table~\ref{words1}. Furthermore the word list in Table~\ref{words2} is further filtered by word likeliness, words whose second letter is not in any other word and by the possible $8^{th}$ position letters of Table~\ref{words1} words. This leaves a reduced list, shown in Table~\ref{words2}.

After using words ending in \texttt{s} and after using \texttt{crossties}, it seemed unlikely that \texttt{B} was \texttt{s} or that the word started with a \texttt{c}. Substituting \texttt{OAXBBJNHB} for the word \texttt{smuggling} some patterns in the words appeared, starting with the word \texttt{authorities} and from there on it was a case of the text making sense.

\begin{table}
	\center
	\caption{9-letter words with repeated 4th and 5th letters, ending on that letter as well.}
	\label{words1}

	\begin{tabular}{c c c c c c c}
	\hline
	blessings & blottiest & brattiest & brittlest & brummagem & chasseurs &  	chattiest\\
	coassumes & crossings & crosslets & crossties & crossways & draggling &  	dressings\\
	glassines & glissades & glossinas & gnattiest & grottiest & knottiest & plottiest\\
	plussages & pressings & reassigns & scheelite & smuggling & threesome &  trussings\\
	unseeable  \\
	\end{tabular}
\end{table}

\begin{table}
\caption{2-letter words and reduced list.}
\label{words2}

	\begin{tabular}{|c c c c c c c c c c c|}
	\hline
	ad & am & an & as & at & be & by & de & do & el & en \\
	go & he & id & if & in & is & it & ma & me & my & no \\
	nu & of & oh & on & or & pi & so & to & up & us & we \\
	\hline
	\end{tabular}
	\quad
	\begin{tabular}{|c c c c c c c|}
	\hline
	an & on & in & as & is & us & at\\
	it & be & he & me & we & by & my\\
	do & go & so & to & no & if & of\\
	\hline
	\end{tabular}
\end{table}

\subsection{Headline 3}
I believe this headline is the text : \texttt{dollar,euro outpace yen in choppy trading}.\\
How this headline was found follows later.
\subsection{Headline 4}
I believe this headline is the text : \texttt{ripken looking for quick return from disabled list}.\\
How this headline was found follows later.
\subsection{Headline 5}
I believe this headline is the text : \texttt{chrldless employees see discrimination in family friendly policies}.\\

The starting point for this headline was the ending of the word \texttt{TBDSPKTTY}, the word \texttt{YTT} and the ending of the word \texttt{HQUSESTYY}. The word \texttt{YTT} contains two very frequent letters, and one is a repetition. The letter \texttt{T} could be a vowel. The word \texttt{see} seemed like a very good candidate. Substituting \texttt{see} for \texttt{YTT} made the word \texttt{TBDSPKTTY} seem like \texttt{employees}, and after substitution, the first words started appearing.

The next stage of decryption was the word \texttt{JCBRSKJURTWESK}. The tool \citep{tool:design215} produces a list of words matching this pattern and after trial and error the most likely word is \texttt{discrimination}.

After substitution of \texttt{discrimination}, the next word that seemed promising is \texttt{DPSRHRTY}, which most likely seems the be the word \texttt{policies}.

After performing all these substitutions, it is a matter of substituting the remaining letters and deciphering the message. The resulting message contains an error (one would expect ch\textit{i}ldless rather than ch\textit{r}ldless), but one would expect more errors to be in the sentence if the mapping was somehow forced to make the resulting plain-text.

\subsection{Discussion}
According to \citep{website:puzzle_info}, the first step to solving the puzzle is to build the mixed alphabet. Following the process described on the website (weaving of substitution chains of headlines 2 and 5 here, as I was unable to weave 2 and 3), the mixed alphabet produced was
\texttt{girumbnwxachqlsykfjopdetvz}. To verify that this is correct, the mapping generated by sliding the alphabet against itself should decipher the solved headlines and produce meaningful text for the unsolved ones, which happens. The shift in the alphabet that solves each headline is $9, 5, 21, 7, 1$ and by stacking the resulting shifted-putative alphabets we can find the setting word, which I found to be \texttt{toady}. One can then reorder the alphabet so that \texttt{toady} is in the first column of the stacked alphabets and hence the shifted alphabet becomes  \texttt{sykfjopdetvzgirumbnwxachql}. Inputing that into the tool \citep{website:key_finder} yields the key \texttt{sycophant} and keyword square presented in Table~\ref{tab:ks}, indicating that the hat is of length 10.

\begin{table}
\center
\caption{The keyword square.}
\label{tab:ks}
	\begin{tabular}{c}
		\texttt{SYCOPHANTB}\\
		\texttt{DEFGIJKLMQ}\\
		\texttt{RUVWXZ}\\
	\end{tabular}
\end{table}

\bibliographystyle{plain}
\bibliography{crypto_report}

\pagebreak
\appendix
\section{Source code}

\textbf{Note} : The scripts in this section were produced by me in order to help me understand some concepts or aid my investigation. \\

\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{bggray}{rgb}{0.9,0.9,0.9}
\definecolor{mygreen}{rgb}{0,0.5,0}

\lstdefinestyle{mypy}{
	backgroundcolor=\color{bggray},
	language=Python,
	basicstyle=\ttfamily,
  	captionpos=b,
	numbers=left,
	keywordstyle=\color{blue},
	showspaces=false,
	commentstyle=\color{mygreen},
	numberstyle=\footnotesize\color{gray},
}

\lstinputlisting[caption=Code that performed the XOR operation with the key for question 3.,
				label=code1,
				style=mypy]
					{cw_files/question_3/question_3.py}
\lstinputlisting[caption=Code that perform shifting of the putative alphabet.,
				label=script_shift,
				style=mypy]
					{cw_files/question_4/headline.py}

\lstinputlisting[style=mypy,
				caption=Code that iterates though the field $Z_{17}$ to check for solutions to elliptic curve E with equation $ y^2 \text{=} x^3 + 4x + 3 $.,
				label=script_ecc]
					{cw_files/question_1/ecc.py}

\pagebreak
\lstinputlisting[style=mypy,
				caption=Code that performs Index of Coincidence analysis.,
				label=code2]
					{cw_files/question_2/ioc.py}
\pagebreak

\section{Headline Puzzle Substitutions}
\label{headline_images}

	\begin{figure*}[h]
		\center
		\includegraphics[scale=0.7]{cw_files/question_4/q1/q1_sol.png}
		\caption{Letter frequency analysis and mapping for headline 1.}
	\end{figure*}

	\begin{figure*}[h]
		\center
		\includegraphics[scale=0.7]{cw_files/question_4/q2/Q2_sol.png}
		\caption{Letter frequency analysis and mapping for headline 2.}
	\end{figure*}

	\begin{figure*}[h]
		\center
		\includegraphics[scale=0.7]{cw_files/question_4/q5/puzzle_5.png}
		\caption{Letter frequency analysis and mapping for headline 5.}
	\end{figure*}

\end{document}


\section{RNG02: Random Number Generators -- 02}
[Warren D. Smith provided the analysis and the generators.]

The goal is to combine several random number generators to get at least the
randomness of the strongest of them, but to do it in such a way as
to get nearly the speed of the fastest of them.

NOTE: I use a lot of global variables here to save statistice,
necessary in modula3 since C's "statisticic" feature does not exist.
Of course these variables are local to the current module.
For that reason, please keep this a seperate module from others,
e.g. the derived routines for NormalDeviate, etc. should not be in
same module as these underlying routines for uniform deviates.

Some component generators:
Following G.Marsaglia: A current view of RNGs, pages 3-10 in
Computer Science and Statistics, the interface, Elsevier 1985,
a fair number of my generators are of the FIBO(a,b,\%) type, where
$a>b>0$ are integers such that $x^a+x^b+1$ is a primitive trinomial
mod 2 and \% is a binary operation.

Suitable values of (b,a) are tabulated in D.Knuth: Seminumerical methods,
Addison-Wesley 1981, page 28, which is extracted from larger tables in
N.Zierler \& J.Brillhart: Information and Control 13 (1968) 541-554;
14 (1969) 566-569; 15 (1969) 67-69. Some pairs with "a" prime which
go beyond the Knuth table, which you can use if you *really*
want randomness, include:
\begin{verbatim}
  (7,127), (15,127), (30,127), (63,127),    (32,521), (158,521),
  (105,607), (147,607), (273,607),     (216,1279), (418,1279)
\end{verbatim}

The binary operation \% could be: [- mod 1] for floating point values,
[- mod M] or [* mod M] for some integer M,
or [this NOT recommended; it is same as - only mod 2] XOR.

Then the meta-procedure Fibo(a,b,\%) is as follows (it
requires an auxiliary arr : ARRAY [0..a-1], initially filled with
random seed values, and initially ia=a and ja=b:

\begin{verbatim}
MetaRandGen() : Type =
  BEGIN
    DEC(i);
    DEC(j);
    IF i<0 THEN
      i := a-1; (* wraparound *)
    ELSIF j<0 THEN
      j := a-1; (* wraparound *)
    END;
    arr[i] := arr[i] \% arr[j];
    RETURN arr[i];
  END MetaRandGen;
\end{verbatim}

Period [Marsaglia \& Tsay:  Lin. Alg. and its
   Appl. 67, 147-155, 1985]: If \% is $(- \mbox{mod} 2^{\mbox{wordsize}})$,
then if at least one of the arr[] is odd
the period will be maximal $= (2^a - 1) * 2^{(\mbox{wordsize}-1)}$ if
the characteristic axa matrix T of the recurrence has full order
$J=(2^a - 1)$ in mod 2 arithmetic (this assured by the trinomial condition)
and 2*J in mod 4 and 4*J in mod 8 arithmetic. That is, if you square
T a times, you get back T in mod 2, but do NOT get back T in mod 4,
arithmetic, and if you square T a+1 times, you do not get back T
in mod 8 arithmetic. In fact, this criterion will work for any
matrix T, not necessarily the fibo($-$) one...

If \% is $[* \mbox{mod} 2^{\mbox{wordsize}}]$, which may be implemented in m3 by Word.Times(),
then arr[] must be all odd integers, and by considering discrete logs
we see that the period is maximal $= (2^a - 1) * 2^{(\mbox{wordsize}-3)}$ if
the $F(a,b,+ \mbox{mod} 2^{\mbox{wordsize}})$ has maximal period
$(2^a - 1) * 2^{(\mbox{wordsize}-1)}$.

Marsaglia's test results for Fibo(a,b,\%) generators with
(a,b)=(17,5), (31,13) and (55,24):
\begin{description}
\item[XOR] With \%=XOR the fibo fail seven of the tests in Marsaglia's battery.
As Marsaglia says in his conclusion "never use XOR." Despite this,
these generators keep resurfacing, for example the "r250" Fibo(250,147,XOR)
generator of S. Kirkpatrick and E.  Stoll, Journal of Computational
Physics, 40, p. 517 (1981) and W. L. Maier, "A Fast Pseudo Random
Number Generator", Dr. Dobb's Journal \#176.

\item[Subtraction] With \%=-, among the tests in Marsaglia's DIEHARD battery
only the "birthday spacings test" (of frequencies of spacings in sorted sets
of deviates) is failed, a symptom both of the linear structure
and also of the specific subtractive structure of these generators.
Note that the Fibo(55,24,-) generator (which fails this test) is to be found
in Knuth's book and also is distributed with the DEC SRC implementation of
modula-3. Also note that postprocessing Fibo(55,24,-) sequence with a
Bays-Durham shuffler, recommended in Press et al. "Numerical Recipes" to
fix suspicious generators, will NOT work to make it now pass this test, since
the birthday spacings test does not depend upon the ordering of the
deviates only their values.

\item[Multiplication] With \%$=* \bmod 2^{32}$, these
FIBO gens passed all the tests in
Marsaglia's battery.
\end{description}

Note that the -,+ and XOR gens are "linear" and hence theoretically
bad and will always fail "empty slab tests" see below.

Note that shift register, Fibonacci() generators with +,- or XOR,
and linear congruential generators x <-- a*x+b mod M, AND linear
combinations of such possibly to different moduli, are all
"linear" and thus generate d-tuples of random numbers lying on
AT MOST $M^{(1/d)}$ hyperplanes in d-space. Consequently,
the nonrandomness of a linear RNG with period M is in principle
detectable by the "largest empty slab" test in .4*logM dimensions
after only N,
\begin{verbatim}
  N > e^(5/2) * (.4*logM)^2.5 * loglogM,
\end{verbatim}
random numbers have been generated. Hence:
Please do not rely on a linear generator.
If you are going, foolishly, to use a linear congruential generator, though,
you want a "good multiplier" a mod M. (Bad multipliers
will result in even fewer hyperplanes, for example IBM's
infamous RANDU generator generated points lying on only 15 planes in 3-space.
Even with fairly good multipliers we still get test failures, e.g. Marsaglia's
spectrally good multiplier $69069 \bmod 2^{32}$ fails his OPSO test, as does
the Berkeley PASCAL $62605 \bmod 2^{29})$. It will suffice, for
comparatively good behavior in d-space, that
\begin{verbatim}
   (a^d mod M)/M
\end{verbatim}
have only small partial quotients in its continued fraction expansion.
This should be tested for $d=1,2,..,8$ at least.

Thus for example, consider the prime modulus $M=2^{35} - 849$.
$(M-1)/2$ is also prime. I found the generator $g=145683$ by computer
search. The continued fraction expansions
\begin{verbatim}
CF(g/p)=[235852,1,3,6,1,1,5,1,1,2,1,1,1,1,6,2]
CF(g^2/p)=[1,1,1,1,1,1,1,1,21,2,7,1,8,1,2,1,1,2,5,1,1,1,2,1,2,3,12,2]
\end{verbatim}
are rather nice, and the $CF( (g^x \bmod p)/p )$ for $x=3,4,5,6,7,8,9$ are
also not bad (no partial quotient larger than 23):
\begin{verbatim}
 [2,1,1,1,1,1,1,1,1,16,1,3,5,1,2,1,1,3,1,1,1,21,1,8,10,5]
 [8,4,3,1,1,3,4,2,7,1,22,22,9,1,12,1,2,2]
 [1,1,1,20,1,23,1,4,2,2,1,1,1,8,1,1,2,1,2,2,1,22,1,2,1,3]
 [1,1,15,1,1,10,2,2,2,1,4,5,1,2,1,1,5,7,7,2,2,4,2]
 [1,1,5,2,2,1,1,1,7,11,3,4,1,9,1,6,3,1,8,1,3,1,1,1,2]
 [1,13,1,7,8,1,1,2,16,2,2,1,1,6,4,1,3,1,1,3,15,2]
 [2,1,2,1,15,2,1,5,1,8,1,1,2,1,4,3,1,3,1,1,1,1,2,1,3,1,3,5]
\end{verbatim}
so we conclude that using g as a multiplier should exhibit comparatively
good behavior in dimensions 2-9. This particular (g,M) pair
has the virtue that IEEE doubles can represent integers up to and including
$2^{53} - 1 $exactly, so that the modula-3 statisticement
\begin{verbatim} x := (g*x) MOD M; \end{verbatim}
will evaluate it exactly.

As another example, the Marsaglia multiplier
$69069 \bmod 2^{32}$, while spectrally good in
dimensions 2-5, is bad in dimension 6, as is revealed by the spectral
test directly but also simply by noticing that the CF expansion of
$69069^6 / 2^{32}$ is [1, 75, 1, 2, 2, 1, 20, 10, 3, 10, 2, 2, 1, 12, 9]
which contains the large number 75 early on.

You also probably want
full period M, which happens if [thm page 16 Knuth]
\begin{verbatim}
  (1) GCD(b,M)=1;
  (2) a-1 is a multiple of p for every prime p dividing M;
  (3) a-1 is a multiple of 4 if 4 divides M.
\end{verbatim}
If M is prime and b=0, you get maximal period M-1 if a primitive mod M.
If M>=16 is power of 2 and b=0, get maximal period M/4 if a=3 or 5 mod 8.

\subsection*{Other Fibo generators}
Instead of just using one lag term and binary operation \%, you could combine
with TWO lag terms via some TERNARY operation, or THREE lag terms
via a QUATERNARY operation, etc. I suggest the new generator
\begin{verbatim}
  QuaternaryFibo(a,b,c,d, x0 - (x2 XOR (x1 - x3) mod M) mod M ).
\end{verbatim}

If M is $2^{\mbox{wordsize}}$ and $x^a+x^b+x^c+x^d+1$, $a>b>c>d>0$,
 is a primitive
polynomial mod 2, then this generator's period will be at least $2^a - 1$
simply by considering the LS bit, which will follow a DeBruijn sequence
and thus exhibits good randomness in $\le a$ dimensions.
My idea is that by using
both XOR and +, we hope to avoid the weak behavior of either operation
alone, e.g. with respect to the birthday spacings test. This may also
make the generator "nonlinear" (i.e. not outputting a lattice), a point
I am unsure about. Ternary generators of this type do not seem to exist since
4-term primitive polynomials $x^a+x^b+x^c+1 \bmod 2$ do not exist (you
need an odd number of terms, as can easily be shown).
For tables of examples of primitive polynomials mod 2, see E.J.Watson: Math. of
Comput. 16 (1962) 368-9 but the ones given are the lexically first examples,
bad for our purposes.

Hence I constructed my own small table below of examples of primitive
polynomials $P=x^a+x^b+x^c+x^d+1 \bmod 2$, where a is prime and
$N=2^a - 1$
is a Mersenne prime. For such a, P is primitive if $P^N$
divides $x^N-1 \bmod 2$,
which you can test by the algorithm
\begin{verbatim}
  Q := x;
  FOR i:=1 TO a-1 do
    Q := Q*Q*x mod P;
  END;
  IF Q=1 THEN "P is primitive."
Table:
x^19 + x^9 + x^4 + x^3 + 1
x^19 + x^6 + x^2 + x^1 + 1
x^31 + x^22 + x^20 + x^8 + 1
x^31 + x^16 + x^11 + x^6 + 1
x^61 + x^5 + x^2 + x^1 + 1
x^61 + x^57 + x^2 + x^1 + 1
x^61 + x^19 + x^16 + x^9 + 1
x^89 + x^36 + x^2 + x^1 + 1
x^89 + x^57 + x^14 + x^5 + 1
x^89 + x^28 + x^26 + x^9 + 1
x^89 + x^37 + x^5 + x^1 + 1
x^107 + x^53 + x^45 + x^12 + 1
x^107 + x^93 + x^68 + x^2 + 1
x^127 + x^29 + x^17 + x^7 +
x^127 + x^67 + x^65 + x^17 + 1
x^127 + x^60 + x^22 + x^9 + 1
x^521 + x^46 + x^38 + x^37 + 1
x^521 + x^249 + x^92 + x^87 + 1
x^521 + x^353 + x^258 + x^6 + 1
x^607 + x^66 + x^60 + x^30 + 1
x^607 + x^247 + x^83 + x^71 + 1
x^607 + x^483 + x^449 + x^298 + 1
x^607 + x^442 + x^113 + x^23 + 1
\end{verbatim}

It is a disgusting fact that in modula3 as well as most high level languages,
the machine language multiply instruction that computes a*b to double
precision is NOT ACCESSIBLE. Ditto "add with carry". This makes an
efficient implementation of high precision multiplication (\& modular
version) obnoxiously difficult and inefficient. And you really
do need multiple precision to get a decent period with a lincong
or iterated squaring generator.
[Note the SQUARE ROOT of the period must be unreachably large for
good randomness, NOT the period, a common error.]
Here is a MetaAlgorithm which works
if numbers 0..modulus*2-2 are representable...:
\begin{verbatim}
  MetaModularMultiply(x,y, modulus: NumType) : NumType =
  VAR
    q : NumType := x;
  BEGIN
    FOR b = bits of y in MS-->LS order starting
        at the bit AFTER the first 1 bit DO
      q := q+q;
      IF q>=modulus THEN q := q-modulus; END;
      IF b=1 THEN
        q := q+x;
        IF q>=modulus THEN q := q-modulus; END;
      END;
    END;
    RETURN q;
  END MetaModularMultiply;
\end{verbatim}

One way to improve the randomness of a generator is
C.Bays and S.D. Durham's shuffling algorithm, ACMTOMS 2 (1976)
59-64, also described in D.Knuth: Seminumerical algorithms.
Knuth says this will output a "considerably more random" sequence.
However, the set of values will still be the same (although ordered
differently) hence, e.g., the subtractive and XOR fibo gens will
STILL fail Marsaglia's birthday spacings test even after improvement
by shuffling. However, linear congruential generators are probably
substantially improved by shuffling since the lattice structure is
destroyed.

L.Blum, M.Blum, M. Shub: A simple unpredictable psuedo-random
number generator, SIAM J. Comput 15,2 (1986) 364-
following
A.Yao: Theory and applications of trapdoor fns, 23rd SFOCS (1982) 80-91
and
W.Alexi, B.Chor, O.Goldreich, C.P.Schnorr: 25th SFOCS (1984) 449-457.
point out that iterated squaring generator
\begin{verbatim}
  x <--- x*x mod M
\end{verbatim}
where M=p*q, p, q primes that are 3 mod 4, will have these properties
(assuming it is computationally infeasible to compute the factorization of M)
\begin{enumerate}
\item infeasible to predict the PREVIOUS x.
\item infeasible to predict the least signif bit of the
    previous x with correctness prob $>$ 1/2 + 1/polynomial.
\item infeasible to predict the boolean "x<(M-1)/2" for the
    previous x with correctness prob $>$ 1/2 + 1/polynomial.
\end{enumerate}
Consequence of (2) is: no polynomial time statistical test
can invalidate the randomness of the sequence of LS bits of
the iterated squaring generator.

The full iterated squaring gen, not just its LS bits,
is a lot faster and (conjecturally!) just as random.
Its period is (p-1)*(q-1)/4.

Example: Here is a product of two primes both 3 mod 4:
\begin{verbatim}
 94906247 * 94906219 = 9007193062250093 = 2^53 - 6192490899.
\end{verbatim}
Here is a smaller example:
\begin{verbatim}
    M = 9739 * 9719 = 94653341.
\end{verbatim}
This modulus has the virtue that iterated squaring mod M
may be accomplished exactly in IEEE double arithmetic using
\begin{verbatim}
 x := x*x MOD M, since M^4 < 2^53 - 1.
\end{verbatim}
An even smaller modulus suitable for 32-bit unsigned arithmetic is
\begin{verbatim}
    M = 239*251; (* = 59989; 239 and 251 are each primes and 3 mod 4 *).
\end{verbatim}

More generally you could use iterated cubing (subject to the conjecture
it is infeasible to break any bit of the RSA cryptosystem) or in fact
any fixed exponent you want (under same conjecture) and in fact almost
any fixed integer linear operation on the discrete logarithms of a vector
will by the same reasoning be immune to any polynomialtime statistical test,
subject to RSA-type conjectures. In particular, we conjecture that the
Fibo[a,b,*] generator ought to be immune to polynomialtime statistical tests
if the modulus is the product of two suitable large primes... which
explains the results of the Marsaglia test battery above.


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 1.4 Experiments with sorting}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z#}::Indices(position=independent).

   \partial{#}::PartialDerivative.

   expr := C^{f}
           w^{e}
           B^{d}
           v^{c}
           A^{b}
           u^{a}.                        # cdb (ex-0104.100,expr)

   sort_product (expr)                   # cdb (ex-0104.101,expr)

   expr := \Omega_{f}
           \gamma_{e}
           \Pi_{d}
           \beta_{c}
           \Gamma_{b}
           \alpha_{a}.                   # cdb (ex-0104.200,expr)

   sort_product (expr)                   # cdb (ex-0104.201,expr)

   expr := C^{f}
           w^{e}
           B^{d}
           v^{c}
           A^{b}
           u^{a}
           \Omega_{f}
           \gamma_{e}
           \Pi_{d}
           \beta_{c}
           \Gamma_{b}
           \alpha_{a}.                   # cdb (ex-0104.300,expr)

   sort_product (expr)                   # cdb (ex-0104.301,expr)

   expr := \partial_{f}{C^{f}}
           w^{l}
           \partial_{d}{B^{d}}
           v^{k}
           \partial_{b}{A^{b}}
           u^{j}
           \Omega_{i}
           \partial^{e}{ \gamma_{e}}
           \Pi_{h}
           \partial^{c}{\beta_{c}}
           \Gamma_{g}
           \partial^{a}{\alpha_{a}}.     # cdb (ex-0104.400,expr)

   sort_product (expr)                   # cdb (ex-0104.401,expr)

   expr := \partial{C}
           w
           \partial{B}
           v
           \partial{A}
           u
           \Omega
           \partial{ \gamma}
           \Pi
           \partial{\beta}
           \Gamma
           \partial{\alpha}.             # cdb (ex-0104.500,expr)

   sort_product (expr)                   # cdb (ex-0104.501,expr)

   expr := A_{b}
           A_{a}
           A_{c d e}
           A_{f g}.                      # cdb (ex-0104.600,expr)

   sort_product (expr)                   # cdb (ex-0104.601,expr)

   expr :=  A_{a} A^{a}
          + A^{a} A_{a}.                 # cdb (ex-0104.700,expr)

   sort_product (expr)                   # cdb (ex-0104.701,expr)

\end{cadabra}

% \clearpage

\begin{dgroup*}[spread=2pt]
   \Dmath*{\cdb*{ex-0104.100}}\Dmath*{\cdb*{ex-0104.101}\V{10pt}}
   \Dmath*{\cdb*{ex-0104.200}}\Dmath*{\cdb*{ex-0104.201}\V{10pt}}
   \Dmath*{\cdb*{ex-0104.300}}\Dmath*{\cdb*{ex-0104.301}\V{10pt}}
   \Dmath*{\cdb*{ex-0104.400}}\Dmath*{\cdb*{ex-0104.401}\V{10pt}}
   \Dmath*{\cdb*{ex-0104.500}}\Dmath*{\cdb*{ex-0104.501}\V{10pt}}
   \Dmath*{\cdb*{ex-0104.600}}\Dmath*{\cdb*{ex-0104.601}\V{10pt}}
   \Dmath*{\cdb*{ex-0104.700}}\Dmath*{\cdb*{ex-0104.701}}
\end{dgroup*}

\end{document}


\documentclass{amsart}

\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{alltt}
\usepackage[dvipsnames]{xcolor}
\usepackage{tkz-graph}

\title{Probability: Final Exam}
\author{Mark Ditsworth}

\begin{document}
	\maketitle
	\section{Problem 1}
	\subsection{Part I}
	Alice and Bob each choose a number independently and uniformly random from the interval $[0,2]$. Consider the following events:\\
	\textbf{A}. the absolute difference between the two numbers is greater than 1/4.\\
	\textbf{B}. Alice's number is greater than 1/4.\\
	Find $\mathbf{P}(A\cap B)$.\\
	\\
	This is the fraction of the area of a $2\times 2$ square that satisfies both conditions if the $x$ and $y$ axes are Alice's and Bob's numbers, respectively. The inclusive area is made up of two isosceles right triangels with side lengths of 1.75 and 1.5, respectively. The combined area is
	\[
	\frac{1}{2}(1.75)^2 + \frac{1}{2}(1.5)^2 = 2.65625
	\]
	Thus, $\mathbf{P}(A\cap B) = \frac{2.65625}{4} = \frac{85}{128}$.
	\\
	\subsection{Part II}
	There are $m$ red balls and $n$ white balls in an urn. We draw two balls at random without replacement. What is the probability that the balls are a different color?\\
	\\
	This is the sum of two probabilities: drawing a red then white, and drawing a white then red.
	\[
	\frac{m}{m+n}\frac{n}{m+n-1} + \frac{n}{m+n}\frac{n}{m+n-1} = \frac{2mn}{(m+n)(m+n-1)}
	\]
	\\
	\subsection{Part III}
	20 black pebbles are arranged in 4 rows of 5 pebbles each. We choose 4 at random and color them red. What is the probability that each red pebble lies in a different row?\\
	\\
	There are $5^4$ possible orderings of 4 pebbles in different rows (5 places for the first, 5 places for the second, etc.). There are $\binom{20}{4}$ total permutations of the pebble sequences. Thus, the probability is
	\[
	\frac{5^4}{\binom{20}{4}} = \frac{5^4 4!16!}{20!}
	\]
	\\
	\subsection{Part IV}
	We have two light bulbs, A and B. Bulb A has an exponentially distributed lifetime with mean of 4 days. Bulb B has an exponentially distributed lifetime with mean 6 days. We select one bulb at random (each is equally likely). Given that the bulb has been working for 12 hours, what is the probability that we chose bulb A?\\
	\\
	By Bayes theorem, we have
	\[
	P(\text{selecting A}|l.t.>\text{0.5 days})=
	\frac{P(\text{selecting A})P(l.t.>\text{0.5 days}|\text{selecting A})}{P(l.t.>\text{0.5 days})}
	\]
	For an exponential random variable, $P(l.t.> t)=
	e^{\frac{1}{\mu}t}$.
	Thus, we have
	\[
	\frac{(0.5)e^{\frac{1}{4}\frac{1}{2}}}{0.5(e^{\frac{1}{4}\frac{1}{2}}+e^{\frac{1}{6}\frac{1}{2}})} = \frac{1}{1+e^{1/12 - 1/8}} = \frac{1}{1+e^{1/24}}
	\]
	\\
	\subsection{Part V}
	A test for a rare disease is correct 95\% of the time. A false positive has a probability 0.95 and a false negative has a probability 0.95. A random person drawn fro the population has a probability of 0.001 of having the disease. Given that a person just tested positive, what is the probability of having the disease?\\
	\\
	By Bayes theorem,\\
	$P(\text{sick}|\text{positive})=
	\frac{P(\text{sick})P(\text{positive}|\text{sick})}{P(\text{positive})} = \frac{(0.001)(0.95)}{(0.001)(0.95)+(0.999)(0.05)}=0.187$
	\\
	\subsection{Part VI}
	Let $X$ be a random variable with
	\[
	f_X(x) =
	\begin{cases}
		 2e^{-2x} & x\geq 0\\
		 0 & \text{otherwise}
	\end{cases}
	\]
	Let $Y=2X-4$. Find $f_Y(0)$.\\
	\\
	\[
	F_Y(y) = P(Y\leq y) = P\left(X \leq \frac{y}{2}+2\right) = F_X\left(\frac{y}{2}+2\right)
	\]
	\[
	\frac{d}{dy}F_Y(y) = \frac{d}{dy}F_X\left(\frac{y}{2}+2\right)
	\]
	\[
	f_Y(y) = \frac{1}{2}f_X\left(\frac{y}{2}+2\right)
	\]
	\[
	f_Y(0) = \frac{1}{2}f_X(2)= e^{-4}
	\]
	\\
	\subsection{Part VII}
	$X$ and $Y$ are independent random variables. $X$ is exponentially distributed with mean $1/3$. $Y$ is Bernoulli with $p=1/2$. Let $Z=X+Y$. What is the transform of $Z$, $M_Z(s)$ evaluated at $s=1$?\\
	\\
	\[
	M_X(s) = \frac{3}{3-s} \qquad, \qquad M_Y(s) = \frac{1}{2}+\frac{1}{2}e^s
	\]
	Since the PDF of $Z$ is the convolution of $X$ and $Y$, in the $s$ domain it is the multiplication of $M_X(s)$ and $M_Y(s)$.
	\[
	M_Z(s) = M_X(s)~M_Y(s)
	\]
	\[
	M_Z(1) = \frac{3}{2}\left(\frac{1}{2}+\frac{1}{2}e\right)=
	\frac{3}{4}(1+e)
	\]
	\\
	\subsection{Part VIII}
	A number $p$ is chosen at random from a uniform distribution $[0,1]$. A sequence of $k$ Bernoulli trials is then performed each with a success probability of $p$. What is the variance of the number of successes in the $k$ trials?\\
	\\
	By the law of total variance,
	\[
	\text{Var}(X) = \textbf{E}[\text{Var}(X|P)] + \text{Var}(\mathbf{E}[X|P])
	\]
	\[
	= \textbf{E}[kP(1-P)] + \text{Var}(kP)
	\]
	\[
	=k\textbf{E}[P] - k\textbf{E}[P^2] + k^2\text{Var}(P)
	\]
	\[
	=k\textbf{E}[P] -k\textbf{E}[P^2] + k\textbf{E}[P]^2 - k\textbf{E}[P]^2 + k^2\text{Var}(P)
	\]
	\[
	=k\textbf{E}[P] - k\text{Var}(P) -k\textbf{E}[P]^2 + k^2\text{Var}(P)
	\]
	\[
	=\frac{k}{2} - \frac{k}{12} - \frac{k}{4} + \frac{k^2}{12} = \frac{k}{6}+\frac{k^2}{12}
	\]
	\[
	=\frac{k(2+k)}{12}
	\]
	\\
	\subsection{Part IX}
	A police radar always over-estimates the speed of a car by an amount that is uniformly distributed between 0 and 5 mph. Assume that car speeds are uniformly distributed between 60 and 75 mph and are independent of the radar over-estimate. If the radar measures a speed of 76 mph, what is the least-squares estimate of the actual car speed?\\
	\\
	The least squares estimate of $Y$ is \textbf{E}[$Y|X$]. Given the radar speed of 76 mph. The possible values of the actual speed are uniformly distributed between 71 and 75 (since 75 is the maximum speed). Thus, the least squares estimate is $\frac{75+71}{2} = 73$ mph.
	\\
	\subsection{Part X}
	A mosquito lands on your back and bites according to a Bernoulli process. The expected time of the first bite is 10 seconds. What is the probability that the second bite occurs at exactly 10 seconds?\\
	\\
	The time to first arrival is a geometric distribution, with the average time equal to $\frac{1}{p}$. Thus, $p=0.1$.

	The $k$th arrival time of a Bernoulli process is modeled by the Pascal distribution, where $P_k(t)
	= \binom{t-1}{k-1}p^k (1-p)^{t-k}$.
	\[
	P_2(10) = \binom{10-1}{2-1}\left(\frac{1}{10}\right)^2\left(\frac{9}{10}\right)^{10-2} = \frac{9!}{8!}\frac{9^8}{10^2 10^8} = \frac{9^9}{10^{10}} \approx 0.0387
	\]
	\subsection{Part XI}
	In order to estimate $p$, the fraction of people who will vote for candidate A in the next election, you conduct a poll of $n$ people drawn randomly and independently from the population Your estimator $M_n$ is obtained by dividing $S_n$, the number of people voting for candidate A in the sample, by $n$. Find the smallest value of $n$ needed to guarantee that
	\[
	\mathbf{P}(|M_n - p|\geq 0.01) \leq 0.01
	\]
	Applying the WLLN to the Chebyshev inequality, we have
	\[
	\mathbf{P}(|X-\mu|\geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2}
	\]
	Since each sampled person's vote is a Bernoulli distribution, we have $sigma^2 = p(1-p)$ and $p(1-p)\leq \frac{1}{4}$ for all $p\in [0,1]$. Thus,
	\[
	\mathbf{P}(|M_n-p|\geq 0.01) \leq \frac{\sigma^2}{n(0.01)^2}\leq \frac{1}{n(4)(0.01)^2} \leq 0.01
	\]
	\[
	n \geq \frac{1}{4(0.01)^3} = 250,000
	\]
	\subsection{Part XII}
	It rains each day with a probability 0.1, independent of all other days. Use the CLT to approximate the probability that out of 365 days in the year, it will rain on at least 100 days. How good is this approximation?\\
	\\
	The precipitation state on a given day $X_i$ is Bernoulli with $p=0.1$. Thus the average value for a day $\mu=0.1$, and the variance $\sigma^2=p(p-1)=0.09$.
	\\
	Let $S_n = X_1 + X_2 + \dots +X_n$.
	\[
	\mathbf{P}(S_{n=365}\leq 100) = \Phi\left(
	\frac{100 - 365\mu}{\sigma\sqrt{365}}
	\right)=\Phi\left(\frac{635}{3\sqrt{365}}
	\right)
	\]
	So, $\mathbf{P}(S_n\geq 100) = 1 - \Phi\left(\frac{635}{3\sqrt{(365)}}\right) \approx 0$\\
	\\
	The true distribution is Binomial with $p(k) = \binom{365}{k}p^k(1-p)^{365-k}$\\\\
	\[
	\mathbf{P}(k\geq 100) = 1 -\mathbf{P}(k \leq 99) = 1-\sum_{i=0}^{99}\binom{365}{i}p^i(1-p)^{365-i} = 1.11\times 10^{-16} \approx 0
	\]
	The difference between this answer and the approximation with the CLT is within computational rounding difference.
	\pagebreak
	\section{Problem 2}
	Deer come to a river according to a Poisson process with arrival rate $\lambda_d=8$ per hour; elephants come to the river by an independent Poisson process with arrival rate $\lambda_e=2$ per hour. Assume that elephants and deer are the only animals that go to the river.
	\\
	\subsection{Part A}
	Let $N$ be the number of animals you see during the first 3 hours. Find $\mathbf{E}[N]$ and Var$(N)$.\\
	\\
	Let $D$ and $E$ be the number of deer and elephants seen in the first 3 hours, respectively.
	\[
	p_D(k) = \frac{(3\lambda_d)^k e^{-3\lambda_d}}{k!}
	\rightarrow M_D(s) = e^{e\lambda_d(e^s-1)}
	\]
	\[
	p_E(k) = \frac{(3\lambda_e)^k e^{-3\lambda_e}}{k!}
	\rightarrow M_E(s) = e^{e\lambda_e(e^s-1)}
	\]
	\[
	N = D + E \rightarrow M_N(s) = M_D(s)~M_E(s) =
	e^{3(\lambda_d+\lambda_e)(e^s-1)}
	\]
	$N$ is therefore a Poisson process with intensity $3(\lambda_d+\lambda_e) = 30$. So,
	\[
	\mathbf{E}[N] = 30 \qquad , \qquad \text{Var}(N)=30
	\]
	\subsection{Part B}
	What is the probability of seeing your 3rd elephant before your 9th deer?\\
	\\
	The $k$th arrival time in a Poisson process follows an Erlang distribution,
	\[
	f_{Y_k}(y) = \frac{\lambda^k y^{k-1}e^{-\lambda y}}{(k-1)!}
	\]
	We can determine this probability through Monte Carlo simulations of the difference in time between the third elephant's arrival and the ninth deer's arrival. The fraction of (a large number of) simulations for which this difference is negative will approach the probability that the third elephant comes first.

	\begin{alltt}
	\textcolor{ForestGreen}{In [1]:} \textcolor{OliveGreen}{import} \textcolor{blue}{numpy} \textcolor{OliveGreen}{as} \textcolor{blue}{np}
	\textcolor{ForestGreen}{In [2]:} \textcolor{OliveGreen}{from} \textcolor{blue}{scipy.stats} \textcolor{OliveGreen}{import} erlang
	\textcolor{ForestGreen}{In [3]:} lambda_e = 2.
	\textcolor{ForestGreen}{In [4]:} lambda_d = 8.
	\textcolor{ForestGreen}{In [5]:} third_ele_times = erlang.rvs(3,scale=1/lambda_e,size=5000000)
	\textcolor{ForestGreen}{In [6]:} ninth_deer_times = erlang.rvs(9,scale=1/lambda_d,size=5000000)
	\textcolor{ForestGreen}{In [7]:} diff = third_ele_times - ninth_deer_times
	\textcolor{ForestGreen}{In [8]:} len(np.where(diff<0)[0])/5000000.
	\textcolor{red}{Out[8]:} 0.382903
	\end{alltt}

	Therefore, we about a 38\% probability of seeing our third elephant before our ninth deer.

	\subsection{Part C}
	At the end of 3 hours, you have seen 24 deer but no elephants. What is the probability that you will see an elephant in the next hour?\\
	\\
	Since the deer and elephant arrival processes are independent, we can ignore the 24 deer information. Further, since Poisson process is memoryless, the fact that no elephants have been seen in the first three nours has no bearing on the probability of occurrence in the future.

	The number of elephants seen in a one hour time block is a Poisson random variable with PMF $\frac{\lambda_e^k e^{-\lambda_e}}{k!}$. The probability of seeing \textit{no} elephants in the next hour is $e^{-\lambda_e} = e^{-2}$. Thus, the probability of seeing at least one elephant within the hour is $1-e^{-2}$.
	\\
	\subsection{Part D}
	It has been 6 hours since you have started watching. You have seen 53 deer and no elephants. How many more deer do you expect to see before you see your first elephant?\\
	\\
	Again, since Poisson processes are memoryless, the 6 hours and 53 deer are irrelevant information. At the present moment, the expected arrival time of the first elephant $\mathbf{E}[T] = \frac{1}{\lambda_e}=0.5$ hours. We set this time as $tau$. The expected number of deer seen in $\tau$ units of time is $\lambda_d \tau = 8(0.5)=4$
	\\
	\subsection{Part E}
	You plan on staying at the river until you have both a picture of a deer and an elephant. How long do you expect to stay?\\
	\\
	The arrival rate for a general animal is the sum of the arrival rates for elephants and deer, $\lambda_e + \lambda_d = 10$. Thus, the expected wait time for the first animal of any kind is $\frac{1}{10}$ hour.

	There are then two possibilities, a deer was the first to show up (with probability $\frac{\lambda_d}{\lambda_d+\lambda_e}$), and we wait for an elephant (average time of $\frac{1}{\lambda_e}$); or the first animal was an elephant (probability $\frac{\lambda_e}{\lambda_d+\lambda_e}$), and we wait for a deer (average time of $\frac{1}{\lambda_d}$)

	In total, our expected wait time is
	\[
	\frac{1}{\lambda_d+\lambda_e} + \frac{\lambda_d}{\lambda_d+\lambda_e}\frac{1}{\lambda_e}
	+\frac{\lambda_e}{\lambda_d + \lambda_e}\frac{1}{\lambda_d} = \frac{1}{10}+\frac{4}{(5)(2)}+\frac{1}{(5)(8)}=\frac{21}{40}
	\]
	\\
	\subsection{Part F}
	Each time you take a picture, there is a probability of 0.1 that the picture turns our poor. You take a picture each time an animal arrives to the river. let $X$ be the time (in hours) until you get your third picture of an elephant. Find the PDF of $X$.\\
	\\
	This is a Bernoulli process ($p=0.9$) operating on the Poisson process ($\lambda = \lambda_e=2$). This results in a Poisson process with intensity $p\lambda_e$. Therefore, for the arrival time of the third successful picture, we have
	\[
	f_X(x) =
	\begin{cases}
		\frac{1.8^3 x^2 e^{-1.8x}}{2} & x \geq 0\\
		0 & \text{otherwise}
	\end{cases}
	\]
	\pagebreak
	\section{Problem 3}
	If a football team has won the past two games, it has a 0.7 probability of winning the next game. If it lost the last game, but won the game before, it has a 0.4 probability of winning. If it won the last game but lost the one before, it has a 0.6 probability of winning. If it lost the last two games, it has a 0.3 probability of winning the next game. No game can end in a draw. Consider the starting time when the team has won the past two games.
	\subsection{Part A}
	Graph the minimum state Markov chain.
	\begin{table}[h!]
		\centering
		\caption{State Definitions}
		\begin{tabular}{c|c}
			State & Description\\
			\hline
			0 & lost both previous games\\
			1 & lost last game, won the one before that\\
			2 & won last game, lost he one before that\\
			3 & won both previous games
		\end{tabular}
	\end{table}
	\begin{figure}[h!]
		\centering
	\begin{tikzpicture}
		%\node[draw,circle] at (1,1){$0$};
		\SetGraphUnit{3}
		\tikzset{LabelStyle/.style = {rectangle,rounded corners, draw, fill=white!50},
		VertexStyle/.append style={inner sep=8pt}}
		\Vertex{0}
		\EA(0){1}
		\SO(0){3}
		\SO(1){2}
		\tikzset{EdgeStyle/.append style = {->, bend right}}
		\Edge[label=0.6](1)(0)
		\Edge[label=0.3](0)(2)
		\tikzset{EdgeStyle/.append style = {->, bend left}}
		\Edge[label=0.3](3)(1)
		\Edge[label=0.6](2)(3)
		\tikzset{EdgeStyle/.append style = {<->,bend right}}
		\Edge[label=0.4](2)(1)
		\Loop[dist=2cm, dir=NO, label=0.7](0.west)
		\Loop[dist=2cm, dir=NO, label=0.7](3.west)
	\end{tikzpicture}
	\caption{Markov chain model}
	\end{figure}

	\subsection{Part B}
	Find the probability that the first future loss will be followed by another loss.\\
	\\
	Since we are starting in state 3, the first future loss will put us in state 1. The probability that another loss will follow is the probability of moving from state 1 to state 0, which is 0.6.
	\\
	\subsection{Part C}
	Let $X$ be the number of games played up to, but not including, the first loss. Find the PMF of $X$.\\
	\\
	This is the same as the number of games in the initial winning streak. Since we start in state 3, the minimum value of $X$ is 2, with a probability of 0.3 since losing the next game will result in 2 wins. For each game won beyond, we multiply by a probability of 0.7. Thus, the PMF is
	\[
	p_X(x) =
	\begin{cases}
		(0.3)(0.7)^{x-2} & x \geq 2\\
		0 & \text{otherwise}
	\end{cases}
	\]
	\subsection{Part D}
	Either evaluate the steady state probabilities, or explain why they do not exist.\\
	\\
	The calculation of $[\pi_0~\pi_1 ~\pi_2~\pi_3]$ evaluates to finding the eigenvector of $P^T$ corresponding to the eigenvalue of 1. $P$ is the probability matrix of the Markov Chain, where $p_{ij}$ is the probability of moving from state $i$ to $j$. The eigenvector must then be normalized by the sum of the elements.

	This results in $[\pi_0~\pi_1 ~\pi_2~\pi_3] = [\frac{1}{3}~\frac{1}{6}~\frac{1}{6}~\frac{1}{3}]$
	\\
	\subsection{Part E}
	Find a good approximation that the team will win it's $1000^{th}$ game, given that the outcomes of games 1000 and 1001 are the same.\\
	\\
	Assuming that 1000 games is enough to reach the steady state probabilities, the probability of both game 1000 and game 1001 being won is the same as the probability of being in state 3. The probability of both games being the same is the sum of the probabilities of being in state 3 or state 0. Thus, the probability is
	\[
	\frac{\pi_3}{\pi_0 + \pi_3} = \frac{1/3}{1/3~+~1/3} = \frac{1}{2}
	\]
	\subsection{Part F}
	Let $T$ be the number of games up to and including the team's 2nd consecutive loss. What is $\mathbf{E}[T]$?\\
	\\
	In this situation, we must modify the Markov Chain to make state 0 an absorbing state, as shown below.
	\begin{figure}[h!]
		\centering
		\begin{tikzpicture}
		%\node[draw,circle] at (1,1){$0$};
		\SetGraphUnit{3}
		\tikzset{LabelStyle/.style = {rectangle,rounded corners, draw, fill=white!50},
			VertexStyle/.append style={inner sep=8pt}}
		\Vertex{0}
		\EA(0){1}
		\SO(0){3}
		\SO(1){2}
		\tikzset{EdgeStyle/.append style = {->, bend right}}
		\Edge[label=0.6](1)(0)
		%\Edge[label=0.3](0)(2)
		\tikzset{EdgeStyle/.append style = {->, bend left}}
		\Edge[label=0.3](3)(1)
		\Edge[label=0.6](2)(3)
		\tikzset{EdgeStyle/.append style = {<->,bend right}}
		\Edge[label=0.4](2)(1)
		\Loop[dist=2cm, dir=NO, label=1.0](0.west)
		\Loop[dist=2cm, dir=NO, label=0.7](3.west)
		\end{tikzpicture}
		\caption{Markov chain model with state 0 as an absorbing state.}
	\end{figure}

	States 1-3 are transient states since they each reach state 0, but are not reachable by state 0. State 0 itself is a recurrent state. Thus the expected time to reach absorption starting at state 0, $\mu_0$ is 0.

	For the $i$th state of $m$ total states, the expected time to reach absorption is $\mu_i = 1+ \sum_{j=0}^{m}p_{ij}\mu_j$. Therefore we get the following system of equations:
	\[
		\begin{cases}
		\mu_0 = 0\\
		\mu_1 = 1 + 0.4\mu_2\\
		\mu_2 = 1 + 0.4\mu_1 + 0.6\mu_3\\
		\mu_3 = 1 + 0.3\mu_1 + 0.7\mu_3
		\end{cases}
	\]
	which can be represented in matrix form as
	\[
	\begin{bmatrix}
	1 & -0.4 & 0\\
	-0.4 & 1 & -0.6\\
	-0.3 & 0 & 0.3
	\end{bmatrix}
	\begin{bmatrix}
	\mu_1\\\mu_2\\\mu_3
	\end{bmatrix}
	=
	\begin{bmatrix}
	1\\1\\1
	\end{bmatrix}
	\]
	Since we start in state 3, $\mathbf{E}[T] = \mu_3$, which evaluates to 7 games.
	\\
	\subsection{Part G}
	The team is removed from competition after 3 losses in a row. Let $N$ be the number of games that the team will play in competition. What is $\mathbf{E}[N]$?\\
	\\
	We must add another state to the original Markov chain that signifies losing after being in state 0, shown below.
	\begin{figure}[h!]
		\centering
		\begin{tikzpicture}
		%\node[draw,circle] at (1,1){$0$};
		\SetGraphUnit{3}
		\tikzset{LabelStyle/.style = {rectangle,rounded corners, draw, fill=white!50},
			VertexStyle/.append style={inner sep=8pt}}
		\Vertex{0}
		\EA(0){1}
		\SO(0){3}
		\SO(1){2}
		\WE(0){4}
		\tikzset{EdgeStyle/.append style = {->, bend right}}
		\Edge[label=0.6](1)(0)
		\Edge[label=0.3](0)(2)
		\tikzset{EdgeStyle/.append style = {->, bend left}}
		\Edge[label=0.3](3)(1)
		\Edge[label=0.6](2)(3)
		\Edge[label=0.7](0)(4)
		\tikzset{EdgeStyle/.append style = {<->,bend right}}
		\Edge[label=0.4](2)(1)
		\Loop[dist=2cm, dir=NO, label=1.0](4.west)
		\Loop[dist=2cm, dir=NO, label=0.7](3.west)
		\end{tikzpicture}
	\caption{Markov chain model with added absorbing state 4 for three losses in a row.}
	\end{figure}

	This new Markov chain results in the following systems of equations for average absorption times.
	\[
	\begin{cases}
	\mu_0 = 1 + 0.3\mu_2\\
	\mu_1 = 1 + 0.6\mu_0 + 0.4\mu_2\\
	\mu_2 = 1 + 0.4\mu_1 + 0.6\mu_3\\
	\mu_3 = 1 + 0.3\mu_1 + 0.7\mu_3\\
	\mu_4 = 0
	\end{cases}
	\]
	which is expressed as matrices by
	\[
	\begin{bmatrix}
	1    & 0    & -0.3 & 0\\
	-0.6 & 1    & -0.4 & 0\\
	0    & -0.4 & 1    & -0.6\\
	0    & -0.3 & 0    & 0.3\\
	\end{bmatrix}
	\begin{bmatrix}
	\mu_0\\\mu_1\\\mu_2\\\mu_3
	\end{bmatrix}
	=
	\begin{bmatrix}
	1\\1\\1\\1
	\end{bmatrix}
	\]
	which evaluates to $\mathbf{E}[N] = \mu_3 = 11.28$ games.
\end{document}


\chapter{Finite fields}
In this short chapter, we classify all fields with finitely many elements
and compute the Galois groups.
Nothing in here is very hard, and so most of the proofs are just sketches;
if you like, you should check the details yourself.

The whole point of this chapter is to prove:
\begin{itemize}
	\ii A finite field $F$ must have order $p^n$, with $p$ prime and $n$ an integer.
	\ii In this case, $F$ has characteristic $p$.
	\ii All such fields are isomorphic,
	so it's customary to use the notation $\FF_{p^n}$
	for ``the'' finite field of order $p^n$ if we only care up to isomorphism.
	\ii The extension $F/\FF_p$ is Galois, and $\Gal(F/\FF_p)$ is a cyclic group of order $n$.
	The generator is the automorphism \[ \sigma : F \to F \quad\text{by}\quad x \mapsto x^p. \]
\end{itemize}
If you're in a hurry you can just remember these results and skip to the next chapter.

\section{Example of a finite field}
Before diving in, we give some examples.

Recall that the \emph{characteristic} of a field $F$
is the smallest positive integer $p$ such that
\[ \underbrace{1_F + \dots + 1_F}_{\text{$p$ times}} = 0 \]
or $0$ if no such integer $p$ exists.

\begin{example}[Base field]
	Let $\FF_p$ denote the field of integers modulo $p$.
	This is a field with $p$ elements, with characteristic $p$.
\end{example}

\begin{example}[The finite field of nine elements]
	Let
	\[ F \cong \FF_3[X]/(X^2+1) \cong \ZZ[i] / (3). \]
	We can think of its elements as \[ \left\{ a + bi \mid 0 \le a,b \le 2 \right\}. \]
	Since $(3)$ is prime in $\ZZ[i]$, the ring of integers of $\QQ(i)$,
	we see $F$ is a field with $3^2 = 9$ elements inside it.
	Note that, although this field has $9$ elements, every element $x$ has the property that
	\[ 3x = \underbrace{x + \dots + x}_{\text{$3$ times}} = 0. \]
	In particular, $F$ has characteristic $3$.
\end{example}

\section{Finite fields have prime power order}
\begin{lemma}
	If the characteristic of a field $F$ isn't zero,
	it must be a prime number.
\end{lemma}
\begin{proof}
	Assume not, so $n = ab$ for $a,b < n$.
	Then let
	\[ A = \underbrace{1_F + \dots + 1_F}_{\text{$a$ times}} \neq 0 \]
	and
	\[ B = \underbrace{1_F + \dots + 1_F}_{\text{$b$ times}} \neq 0. \]
	Then $AB = 0$, contradicting the fact that $F$ is a field.
\end{proof}

We like fields of characteristic zero, but unfortunately for finite fields
we are doomed to have nonzero characteristic.

\begin{lemma}
	[Finite fields have prime power orders]
	Let $F$ be a finite field.
	Then
	\begin{enumerate}[(a)]
		\ii Its characteristic is nonzero, and hence some prime $p$.
		\ii The field $F$ is a finite extension of $\FF_p$,
		and in particular it is an $\FF_p$-vector space.
		\ii We have $\left\lvert F \right\rvert = p^n$ for some prime $p$, integer $n$.
	\end{enumerate}
\end{lemma}
\begin{proof}
	Very briefly, since this is easy:
	\begin{enumerate}[(a)]
		\ii Apply Lagrange's theorem (or pigeonhole principle!)
		to $(F, +)$ to get the characteristic isn't zero.
		\ii The additive subgroup of $(F,+)$ generated
		by $1_F$ is an isomorphic copy of $\FF_p$.
		\ii Since it's a field extension,
		$F$ is a finite-dimensional vector space over $\FF_p$,
		with some basis $e_1, \dots, e_n$.
		It follows that there are $p^n$ elements of $F$. \qedhere
	\end{enumerate}
\end{proof}
\begin{remark}
	An amusing alternate proof of (c) by contradiction:
	if a prime $q \neq p$ divides $\left\lvert F \right\rvert$, then
	by Cauchy's theorem (\Cref{thm:cauchy_group}) on $(F, +)$
	there's a (nonzero) element $x$ of order $q$.
	Evidently \[ x \cdot ( \underbrace{1_F + \dots + 1_F}_{\text{$q$ times}} ) = 0 \]
	then, but $x \neq 0$, and hence the characteristic of $F$ also divides $q$,
	which is impossible.
\end{remark}

An important point in the above proof is that
\begin{lemma}[Finite fields are field extensions of $\FF_p$]
	If $\left\lvert F \right\rvert = p^n$ is a finite field,
	then there is an isomorphic copy of $\FF_p$ sitting inside $F$.
	Thus $F$ is a field extension of $\FF_p$.
\end{lemma}

We want to refer a lot to this copy of $\FF_p$, so in what follows:
\begin{abuse}
	Every integer $n$ can be identified as an element of $F$, namely
	\[ n \defeq \underbrace{1_F + \dots + 1_F}_{\text{$n$ times}}. \]
	Note that (as expected) this depends only on $n \pmod p$.
\end{abuse}

This notation makes it easier to think about statements like the following.
\begin{theorem}
	[Freshman's dream]
	For any $a,b \in F$ we have
	\[ (a+b)^p = a^p + b^p. \]
\end{theorem}
\begin{proof}
	Use the Binomial theorem, and the fact that $\binom pi$ is divisible by $p$ for $0 < i < p$.
\end{proof}
\begin{exercise}
	Convince yourself that this proof works.
\end{exercise}

\section{All finite fields are isomorphic}
We next proceed to prove ``Fermat's little theorem'':
\begin{theorem}
	[Fermat's little theorem in finite fields]
	Let $F$ be a finite field of order $p^n$.
	Then every element $x \in F$ satisfies
	\[ x^{p^n} - x = 0. \]
\end{theorem}
\begin{proof}
	If $x = 0$ it's true; otherwise, use Lagrange's theorem
	on the abelian group $(F, \times)$ to get $x^{p^n-1} = 1_F$.
\end{proof}

We can now prove the following result,
which is the ``main surprise'' about finite fields:
that there is a unique one up to isomorphism for each size.
\begin{theorem}[Complete classification of finite fields]
	A field $F$ is a finite field with $p^n$ elements if and only if
	it is a splitting field of $x^{p^n}-x$ over $\FF_p$.
\end{theorem}
\begin{proof}
	By ``Fermat's little theorem'', all the elements of $F$ satisfy this polynomial.
	So we just have to show that the roots of this polynomial are distinct
	(i.e.\ that it is separable).

	To do this, we use the derivative trick again: the derivative of this polynomial is
	\[ p^n \cdot x^{p^n-1} - 1  = -1 \]
	which has no roots at all, so the polynomial cannot have any double roots. \qedhere
\end{proof}

\begin{definition}
	For this reason, it's customary to denote \emph{the}
	field with $p^n$ elements by $\FF_{p^n}$.
\end{definition}

Note that the polynomial $x^{p^n}-x \pmod p$ is far from irreducible, but
the computation above shows that it's separable.
\begin{example}[The finite field of order nine again]
	The polynomial $x^9-x$ is separable modulo $3$ and has factorization
	\[ x(x+1)(x+2)(x^2+1)(x^2+x+2)(x^2+2x+2) \pmod 3. \]

	So if $F$ has order $9$, then we intuitively expect it to be the field
	generated by adjoining all the roots: $0$, $1$, $2$, as well as
	$\pm i$, $1 \pm i$, $2 \pm i$.
	Indeed, that's the example we had at the beginning of this chapter.

	(Here $i$ denotes \emph{an} element of $\FF_9$ satisfying $i^2=-1$.
	The notation is deliberately similar to the usual imaginary unit.)
\end{example}

\section{The Galois theory of finite fields}
Retain the notation $\FF_{p^n}$ now (instead of $F$ like before).
By the above theorem, it's the splitting field of a separable polynomial,
hence we know that $\FF_{p^n} /\FF_p$ is a Galois extension.
We would like to find the Galois group.

In fact, we are very lucky: it is cyclic.
First, we exhibit one such element $\sigma_p \in \Gal(\FF_{p^n} /\FF_p)$:

\begin{theorem}[The $p$th power automorphism]
	The map $\sigma_p : \FF_{p^n} \to \FF_{p^n}$ defined by
	\[ \sigma_p(x) = x^p \]
	is an automorphism, and moreover fixes $\FF_p$.
\end{theorem}
\begin{proof}
	It's a homomorphism since it fixes $1$,
	respects multiplication,
	and respects addition.
	\begin{ques}
		Why does it respect addition?
	\end{ques}
	Next, we claim that it is injective. To see this, note that
	\[ x^p = y^p
		\iff x^p - y^p = 0
		\iff (x-y)^p = 0
		\iff x=y.
	\]
	Here we have again used the Freshman's Dream.
	Since $\FF_{p^n}$ is finite, this injective map is automatically bijective.
	The fact that it fixes $\FF_p$ is Fermat's little theorem.
\end{proof}

Now we're done:
\begin{theorem}
	[Galois group of the extension $\FF_{p^n}/\FF_p$]
	We have $\Gal(\FF_{p^n}/\FF_p) \cong \Zc n$ with generator $\sigma_p$.
\end{theorem}
\begin{proof}
	Since $[\FF_{p^n}:\FF_p] = n$, the Galois group $G$ has order $n$.
	So we just need to show $\sigma_p \in G$ has order $n$.

	Note that $\sigma_p$ applied $k$ times gives $x \mapsto x^{p^k}$.
	Hence, $\sigma_p$ applied $n$ times is the identity,
	as all elements of $\FF_{p^n}$ satisfy $x^{p^n}=x$.
	But if $k < n$, then $\sigma_p$ applied $k$ times
	cannot be the identity or $x^{p^k}-x$ would have too many roots.
\end{proof}

We can see an example of this again with the finite field of order $9$.
\begin{example}
	[Galois group of finite field of order $9$]
	Let $\FF_9$ be the finite field of order $9$,
	and represent it concretely by $\FF_9 = \ZZ[i]/(3)$.
	Let $\sigma_3 : \FF_9 \to \FF_9$ be $x \mapsto x^3$.
	We can witness the fate of all nine elements:
	\begin{diagram}
		0 & 1 & 2 & i & 1+i & 2+i \\
		&&& \dIsom^\sigma & \dIsom^\sigma & \dIsom^\sigma \\
		&&& -i & 1-i & 2-i \\
	\end{diagram}
	(As claimed, $0$, $1$, $2$ are the fixed points, so I haven't drawn arrows for them.)
	As predicted, the Galois group has order two:
	\[ \Gal(\FF_9/\FF_3) = \left\{ \id, \sigma_3 \right\} \cong \Zc 2. \]
\end{example}

This concludes the proof of all results stated at the beginning of this chapter.

\section\problemhead
\begin{dproblem}[HMMT 2017]
	\gim
	What is the period of the Fibonacci sequence modulo $127$?
	\begin{hint}
	The Fibonacci sequence is given by
	$F_n = \frac{\alpha^n - \beta^n}{\alpha - \beta}$
	where $\alpha = \frac{1 + \sqrt5}{2}$ and $\beta = \frac{1 - \sqrt5}{2}$
	are the two roots of $P(X) \overset{\text{def}}{=} X^2-X-1$.
	Show the polynomial $P(X)$ is irreducible modulo $127$;
	then work in the splitting field of $P$, namely $\FF_{p^2}$.

	Show that $\FF_p=-1$, $\FF_{p+1}=0$, $\FF_{2p+1}=1$, $\FF_{2p+2}=0$.
	(Look at the action of $\Gal(\FF_{p^2}/\FF_p)$
	on the roots of $P$.)
	\end{hint}
	\begin{sol}
	Recall that the Fibonacci sequence is given by
	\[ F_n = \frac{\alpha^n - \beta^n}{\alpha - \beta} \]
	where $\alpha = \frac{1 + \sqrt5}{2}$ and $\beta = \frac{1 - \sqrt5}{2}$
	are the two roots of $P(X) \defeq X^2-X-1$.

	Let $p = 127$ and work modulo $p$.
	As \[ \left( \frac5p \right) = \left( \frac p5 \right) = \left( \frac 25
		\right) = -1 \]
	we see $5$ is not a quadratic residue mod $127$.
	Thus the polynomial $P(X)$,
	viewed as a polynomial in $\FF_p[X]$, is irreducible
	(intuitively, $\alpha$ and $\beta$ are not elements of $\FF_p$).
	Accordingly we will work in the finite field $\FF_{p^2}$,
	which is the $\FF_p$-splitting field of $P(X)$.
	In other words we interpret $\alpha$ and $\beta$ as elements
	of $\FF_{p^2}$ which do not lie in $\FF_p$.

	Let $\sigma \colon \FF_{p^2} \to \FF_{p^2}$ by $t \mapsto t^p$
	be the nontrivial element of
	$\opname{Gal}\left( \FF_{p^2} / \FF_p \right)$;
	in other words, $\sigma$ is the non-identity automorphism of $\FF_{p^2}$.
	Since the fixed points of $\sigma$ are the elements of $\FF_p$,
	this means $\sigma$ does not fix either root of $P$; thus we must have
	\begin{align*}
		\alpha^p = \sigma(\alpha) &= \beta \\
		\beta^p = \sigma(\beta) &= \alpha.
	\end{align*}
	Now, compute
	\begin{align*}
		F_{p} &= \frac{\alpha^{p} - \beta^{p}}{\alpha-\beta} =
		\frac{\beta-\alpha}{\alpha-\beta} = -1. \\
		F_{p+1} &= \frac{\alpha^{p+1} - \beta^{p+1}}{\alpha-\beta} =
		\frac{\alpha\beta-\beta\alpha}{\alpha-\beta} = 0. \\
		F_{2p+1} &= \frac{\alpha^{2p+1} - \beta^{2p+1}}{\alpha-\beta}
			= \frac{\beta^2\alpha-\alpha^2\beta}{\alpha-\beta}
			= - \alpha \beta = 1.  \\
		F_{2p+2} &= \frac{\alpha^{2p+2} - \beta^{2p+2}}{\alpha-\beta}
			= \frac{\beta^2\alpha^2-\alpha^2\beta^2}{\alpha-\beta} = 0.
	\end{align*}
	Consequently, the period must divide $2p+2$ but not $p+1$.

	We now use for the first time the exact numerical value $p=127$
	to see the period divides $2p+2 = 256 = 2^8$,
	but not $p+1 = 128 = 2^7$.
	(Previously we only used the fact that $(5/p)=-1$.)
	Thus the period must be exactly $256$.
	\end{sol}
\end{dproblem}


\documentclass{article}
\usepackage{amsmath}
\newcommand{\vG}{\mathbf{G}}
\newcommand{\vr}{\mathbf{r}}
\newcommand{\vI}{\mathbf{I}}
\title{Notes on Reciprocal-Space Jastrow Factors}
\date{\today}
\author{Ken Esler}
\begin{document}
\maketitle
\section{Two-body Jastrow}
\begin{equation}
J_2 = \sum_{\vG\neq \mathbf{0}}\sum_{i\neq j} a_\vG e^{i\vG\cdot(\vr_i-\vr_j)}
\end{equation}
This may be rewritten as
\begin{eqnarray}
J_2 & = & \sum_{\vG\neq \mathbf{0}}\sum_{i\neq j} a_\vG e^{i\vG\cdot\vr_i}e^{-i\vG\cdot\vr_j} \\
& = & \sum_{\vG\neq \mathbf{0}} a_\vG \left\{
\underbrace{\left[\sum_i e^{i\vG\cdot\vr_i} \right]}_{\rho_\vG}
\underbrace{\left[\sum_j e^{-i\vG\cdot\vr_j} \right]}_{\rho_{-\vG}}  -1 \right\}
\end{eqnarray}
The $-1$ is just a constant term and may be subsumed into the $a_\vG$
coefficient by a simple redefintion.  This leaves a simple, but
general, form:
\begin{equation}
J_2 = \sum_{\vG\neq\mathbf{0}} a_\vG \rho_\vG \rho_{-\vG}
\end{equation}
We may now further constrain this on physical grounds.  First, we
recognize that $J_2$ should be real.  Since $\rho_{-\vG} =
\rho_\vG^*$, it follows that $\rho_{\vG}\rho_{-\vG} = |\rho_\vG|^2$ is
real, so that $a_\vG$ must be real.  Furthermore, we group the $\vG$'s
into $(+\vG, -\vG)$ pairs, and sum over only the positive vectors to
save time.

\section{One-body Jastrow}
The one-body Jastrow has a similar form, but depends on the
displacment from the electrons to the ions in the system.
\begin{equation}
J_1 = \sum_{\vG\neq\mathbf{0}} \sum_{\alpha}
\sum_{i\in\vI^\alpha}\sum_{j\in\text{elec.}} b^{\alpha}_\vG
  e^{i\vG\cdot(\vI^{\alpha}_i - \vr_j)},
\end{equation}
where $\alpha$ denotes the different ionic species.
We may rewrite this in terms of $\rho^{\alpha}_\vG$,
\begin{equation}
J_1 = \sum_{\vG\neq\mathbf{0}} \left[\sum_\alpha b^\alpha_\vG
  \rho_\vG^\alpha\right] \rho_{-\vG},
\end{equation}
where
\begin{equation}
\rho^\alpha_\vG = \sum_{i\in\vI^\alpha} e^{i\vG\cdot\vI^\alpha_i}.
\end{equation}
We note that in the above equation, for a single configuration of the
ions, the sum in brackets can be rewritten as a single constant.  This
implies that the per-species one-body coefficents, $b^\alpha_\vG$, are
underdetermined for single configuration of the ions.  In general, if
we have $N$ species, we need $N$ linearly independent ion
configurations to uniquely determine $b^{\alpha}_\vG$.  For this
reason, we will drop the $\alpha$ superscript of $b_\vG$ for now.

If we do desire to find a reciprocal space one-body Jastrow that is
transferable to systems with different ion positions and $N$
ionic species, we must perform compute $b_\vG$ for $N$ different ion
configurations.  We may then construct $N$ equations at each value of
$\vG$ to solve for the $N$ unknown values, $b^\alpha_\vG$.

In the two-body case, $a_\vG$ was constrained to be real by the fact
that $\rho_\vG \rho_{-\vG}$ was real.  However, in the one-body case,
there is no such guarantee about $\rho^\alpha_\vG \rho_\vG$.
Therefore, in general, $b_\vG$ may be complex.

\section{Symmetry considerations}
For a crystal, many of the $\vG$-vectors will be equivalent by
symmetry.  It is useful then, to divide the $\vG$-vectors into
symmetry-related groups and then to require that they share a common
coefficient.  Two vectors, $\vG$ and $\vG'$, may be considered
symmetry related if, for all $\alpha$ and $\beta$,
\begin{equation}
\rho^\alpha_\vG \rho^\beta_{-\vG} = \rho^\alpha_{\vG'} \rho^\beta_{-\vG'}.
\end{equation}
For the one-body term, we may also omit from our list of $\vG$-vectors
those for which all species structure factors are zero.  This is
equivalent to saying that, if we are tiling a primitive cell, we
should include only the $\vG$-vectors of the primitive cell, and not
the supercell.  Note that this is not the case for the two-body term,
since the exchange-correlation hole should not have the periodicity of
the primitive cell.

\section{Gradients and Laplacians}
\begin{eqnarray}
\nabla_{\vr_i} J_2 & = & \sum_{\vG \neq 0} a_\vG \left[\left(\nabla_{\vr_i}\rho_\vG\right) \rho_{-\vG} + \text{c.c.}\right] \\
& = & \sum_{\vG\neq \mathbf{0}} 2\vG a_\vG \mathbf{Re}\left(i e^{i\vG\cdot\vr_i} \rho_{-\vG} \right) \\
& = & \sum_{\vG\neq \mathbf{0}} -2\vG a_\vG\mathbf{Im}\left(e^{i\vG\cdot\vr_i} \rho_{-\vG} \right)
\end{eqnarray}
The Laplacian is then given by
\begin{eqnarray}
  \nabla^2 J_2 & = & \sum_{\vG\neq\mathbf{0}} a_\vG \left[\left(\nabla^2 \rho_\vG\right) \rho_{-\vG} + \text{c.c.}
  + 2\left(\nabla \rho_\vG)\cdot(\nabla \rho_{-\vG}\right)\right] \\
& = & \sum_{\vG\neq\mathbf{0}} a_\vG \left[ -2G^2\mathbf{Re}(e^{i\vG\cdot\vr_i}\rho_{-\vG}) +
    2\left(i\vG e^{i\vG\cdot\vr_i}\right) \cdot \left(-i\vG e^{-i\vG\cdot\vr_i}\right)
\right] \\
& = & 2 \sum_{\vG\neq\mathbf{0}} G^2 a_\vG  \left[-\mathbf{Re}\left(e^{i\vG\cdot\vr_i}\rho_{-\vG}\right) + 1\right]
%  \nabla^2_{\vr_i} J_2 & = & \nabla_{\vr_i} \cdot \nabla_{\vr_i} J_2 \\
%  & = & -2\sum_{\vG \neq \mathbf{0}} a_\vG \vG \cdot \nabla_{\vr_i} \mathbf{Im}\left(e^{i\vG\cdot\vr_i} \rho_{-\vG}\right)
%  & = & -2\sum_{\vG \neq \mathbf{0}} a_\vG \vG \cdot \mathbf{Im}\left(i\vG e^{i\vG\cdot\vr_i}\rho_{-\vG} -i\vG \right)
\end{eqnarray}

\end{document}


\section{Evaluation}
\label{sec:evaluation}

To be able to compare the results of different experiments, we have fixed the number of iterations of Stochastic Gradient Descent to a constant value of $20000$.

\subsection{Train Error}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/error}
	\caption{Train and test error of the network for $P = 1000$. The training is done with a fixed learning rate $\eta = 0.05$.}
	\label{fig:training_error}
\end{figure}

\cref{fig:training_error} shows the train and test error for the network trained using Stochastic Gradient Descent on the first $1000$ examples.
Both the train error and test error drop very quickly during approximately the first $1000$ iterations, then remain almost constant for the rest of the training.
The test error is slightly bigger than the train error:
at the end of the training, the train error is around $0.10$ and the test error around $0.12$.
Since the test error never increases, the model does not seem to overfit the training data.

\subsection{Learned Weights}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/weights_p_1000}
    \caption{Weights of the hidden units after the training for $P = 1000$.}
	\label{fig:weights}
\end{figure}

\cref{fig:weights} shows the weights learned from the hidden units of the networks after the training for $P = 1000$.
As expected, the units learn different weights thanks to the random initialization of the weights.

\subsection{Train Dataset}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/error_ps}
	\caption{Train and test error of the network for different numbers of training examples $P$. The training is done with a fixed learning rate $\eta = 0.05$.}
	\label{fig:ps}
\end{figure}

\cref{fig:ps} shows the train and test error for different values of $P$, i.e. for a network trained on training dataset of different dimensions.
For very small training datasets ($P = 20$, $P = 50$), the train error drops very quickly and stabilizes close to $0$;
the test error is high and even increases during the training for $P = 50$.
It seems like the training set is too small for the network to generalize properly to new data, so it tends to overfit the train set.

For $P = 200$, the train error is higher, but the test error gets significantly smaller.
Still, the test error increases during the training, which indicates overfitting.

For $P = 500$, the train and test error get closer and remain constant during most of the training.
The results for higher values of $P$ are very similar and are thus not shown here.

\subsection{Train Policy}
The learning rate influences the number of iterations the training algorithm needs to converge to the optimal weights.
A high learning rate causes the process to be unstable, while a low one takes too long time to converge.
For this reason, we implement different schedulers for the learning rate able to change its value over time as a function of the current iteration: \textit{fixed}, \textit{step}, \textit{exponential} and \textit{cycle}.
\cref{fig:learning_rates_policies} shows the learning rate change over time for the different strategies.
From the application of these \ac{LRPs} we expect a change in the behavior of the error function.
In most \ac{LRPs} the learning rate decreases over time, so the error should get more stable.

\begin{figure}
	\centering
	\includegraphics[width=\columnwidth]{figures/learning_rates}
	\caption{Learning rate evolution over iterations for different learning strategies.}
	\label{fig:learning_rates_policies}
\end{figure}

\cref{fig:lrp_training_error} shows that different \ac{LRPs} have different effects on the error trend.
Overall, all implemented \ac{LRPs} perform better than the \textit{fixed}($0.05$).
The next sections discuss them in more detail.

\begin{figure}
	\centering
	\includegraphics[width=\columnwidth]{figures/error_strategies}
	\caption{Train and test error for different learning strategies.}
	\label{fig:lrp_training_error}
\end{figure}

\subsubsection{Fixed \ac{LRP}}
As the name suggests this \ac{LRP} leaves the learning rate fixed during iterations.
The error to drops down after around $500$ iterations, then decreases very slowly.
On our dataset, this strategy yields the best performances for a learning rate $\eta = 0.02$.

\subsubsection{Step \ac{LRP}}
This strategy consists in decreasing the learning rate of a defined quantity (\textit{drop}) after a certain number of iterations (\textit{step\_size}).
As shown in \cref{fig:learning_rates_policies}, we chose \textit{step\_size} = $5000$ and \textit{drop} = $0.01$.
\cref{fig:lrp_training_error} shows a drop in both train and test error close to the step in the learning rate.

\subsubsection{Exponential \ac{LRP}}
This strategy consist in decreasing the learning rate in an exponential way, i.e. in dividing it by a constant value at each iteration.
This \ac{LRP} lets the network learn quickly at the beginning, when the currently weights are likely to be far from optimal one, and then slowly, when they are getting closer to it.
\cref{fig:lrp_training_error} shows a smoother decrease of both the train and test errors, which reaches a value smaller than $0.1$ after just $3000$ iterations.

\subsubsection{Cycle}
This strategy consists in increasing and decreasing the learning rate linearly over the number of iterations, creating a sawteeth plot, as shown in \cref{fig:learning_rates_policies}.
The aim of this \ac{LRP} is to avoid local minima letting the value of the learning rate to grow again when it reaches its minimum value.
\cref{fig:lrp_training_error} shows that this strategy performs well as far as the value of the learning rate is smaller than $0.02$, then the errors increase following the increase of the learning rate.