\documentclass{article}
\usepackage{hyperref}
\usepackage{amsmath,amssymb}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage[section]{placeins}
\renewcommand{\thesubsection}{\thesection.\alph{subsection}}
\usepackage{listings}

\title{\bf{CSE397: Assignment \#1}}
\author{Nicholas Malaya \\ Institute for Computational Engineering and Sciences \\ University of Texas at Austin} \date{}

\begin{document}
\maketitle

\newpage
\section{Problem 1}

\begin{figure}[!htb]
  \includegraphics[scale=.5]{plots/data.pdf}
  \label{fig:data}
  \caption{The data (after applying the filter) with and without
 normally distributed noise. }
\end{figure}

After applying the new filter $k(x)$ and adding gaussian noise, the data
used as input for the inverse problem is plotted in figure 1.

Notice that this filter is significant: the raw data generated after
passing through the filter (even without statistical noise) has lost
several features of the underlying ``true'' signal.

\subsection{$T_{\text{SVD}}$}

\begin{figure}[!htb]
        \centering
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/tsvd0001.pdf}
                \caption{$\alpha=0.0001$}
        \end{subfigure}%
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/tsvd001.pdf}
                \caption{$\alpha=0.001$}
        \end{subfigure}
        \centering
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/tsvd01.pdf}
                \caption{$\alpha=0.1$}
        \end{subfigure}%
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/tsvd1.pdf}
                \caption{$\alpha=1.0$}
        \end{subfigure}
        \caption{$T_{\text{SVD}}$ at varying values of $\alpha$.}
        \label{fig:svd}
\end{figure}

Figure \ref{fig:svd} plots the solution to the inverse problem using
Truncated SVD at several different values of $\alpha$, the
regularization parameter. We can see that actually, the filter nicely
regularizing the inverse problem even for small values of $\alpha$. It
is only for the largest value of $\alpha$ that the parameter is obviously
too large, at which point almost all the frequencies are filtered out,
and the solution to the inverse problem is constant and zero.

As expected, none of the solutions capture any of the very fine (high
frequency) features of the original data.

\subsection{Tikhanov Filter}

\begin{figure}[!htb]
        \centering
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/reconstruct0001.pdf}
                \caption{$\alpha=0.0001$}
        \end{subfigure}%
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/reconstruct001.pdf}
                \caption{$\alpha=0.001$}
        \end{subfigure}
        \centering
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/reconstruct01.pdf}
                \caption{$\alpha=0.1$}
        \end{subfigure}%
        \begin{subfigure}[bh]{0.45\textwidth}
                \includegraphics[width=\textwidth]{plots/reconstruct1.pdf}
                \caption{$\alpha=1.0$}
        \end{subfigure}
        \caption{Solutions to the inverse problem using Tikhanov
 regularization at varying values of $\alpha$.}
 \label{fig:tik}
\end{figure}

Figure \ref{fig:tik} plots the solution to the inverse problem using
Tikhanov regularization at several different values of $\alpha$, the
regularization parameter.

The Tikhanov filter is more sensitive, with (a) and (b) (low values of
$\alpha]$) completely being dominated by noise. None of the underlying
signal is regained, as all the high frequency content blown up and
dominating.

For higher levels of the regularization parameter, we regain a solution
similar to in the TSVD cases, where we capture the underlying trend of
the data, but none of the sharp features. Also, the highest value of the
filter does not overwhelm the signal completely, as in the TSVD case,
which is interesting.

\subsection{L-Curve}

\begin{figure}[!htb]
  \includegraphics[scale=.6]{plots/L-curve.pdf}
  \caption{The L-curve}
 \label{fig:lcurve}
\end{figure}

Figure \ref{fig:lcurve} plots the results of the L-curve criterion. The
red dot plotted is at 0.01. This is approximately the ``optimal'' value
of regularization, which is generally consistent with the observations
of the previous plots.

\subsection{Morozov's Discrepancy Criterion}

\begin{figure}[!htb]
  \includegraphics[scale=.5]{plots/morozov.pdf}
  \caption{Morozov discrepancy Criterion}
 \label{fig:moro}
\end{figure}

Figure \ref{fig:moro} plots the results of Morozov discrepancy
criterion. I could not get the plots to draw a horizontal line, so that
red dot instead dictates the limit of $\delta$, the norm of the
error. This occurs at $\approx 4e-2$. This is smaller than the value
obtained using the L-curve, but not so much smaller that the value
appears suspect.

\subsection{Actual Error}

\begin{figure}[!htb]
  \includegraphics[scale=.5]{plots/true1d.pdf}
  \caption{The actual error in the reconstruction as a function of
 alpha. }
 \label{fig:actual}
\end{figure}

Figure \ref{fig:actual} plots actual error in the reconstruction,
e.g. the norm of the difference between the reconstruction and the
actual (true) image, as a function of alpha. You can see that this
plot forms a rotated ``S'' curve, with the minimum occuring
approximately at the red dot. The value plotted is 0.01,
which actually agrees with the L-curve estimate.

In any case, the values arrived at using the L-curve, Morozov's
discrepancy criterion and the actual error all agree within at least an
order of magnitude. So we are in the right ``ballpark''. As an aside, I
was actually quite pleased with the result, I would not have expected
these criterion to so closely arrive at values that are reasonable
guesses, even without knowledge of the true underlying signal. While
this is a toy problem, it does show that these criterion can be
effective means of approximating $\alpha$. Given the similarities
between the TSVD and Tikhanov, I would not be surprised to find that
these heuristics can be more generally useful for other regularizations as well.

\newpage
\section{Problem 2}

% \begin{figure}[!htb]
%   \includegraphics[scale=.5]{plots/longhorn.png}
%   \label{fig:true}
%   \caption{The true image.}
% \end{figure}

% Figure \ref{fig:true} plots the true image, that we will be attempting
% to reconstruct through inversion.

\subsection{Memory Requirement}

How much memory would be required to form the entire matrix, $K$?
We have $N^2$ model parameters. K lives in $\mathbb{R}^{N_2 * N_2}$.
With $N=128$, then
\begin{equation}
\frac{N^2 * N^2 * 8 \text{Bytes}}{(1024)^3} = 2 \text{GB}
\end{equation}
So while this is not inaccessible on modern HPC systems, it is certainly
not efficient.

\subsection{L-Curve}

\begin{figure}[!htb]
  \includegraphics[scale=.5]{plots/L-curve2d.pdf}
  \caption{The L-curve for the 2d problem.}
  \label{fig:l2d}
\end{figure}

Figure \ref{fig:l2d} plots the L-curve. The red dot is approximately at
the location of the optimal value of $\alpha$, which is found to be
$\approx 0.05 - 0.001$.

\subsection{Actual Error}

\begin{figure}[!htb]
  \includegraphics[scale=.5]{plots/2d-true.pdf}
  \caption{The actual error in the reconstruction as a function of
 alpha, for the 2d problem.}
  \label{fig:2dt}
\end{figure}

Figure \ref{fig:2dt} plots the true error. Surprisingly, the error curve
appears to flatten out at lower values of alpha, implying a constant
error. Clearly any values less than $1e-3$ appear to be prefered, as the
error increases at higher values of $\alpha$.

To calculate the error, I only needed to solve the conjugate gradient
(as before) and then calculate the norm of the misfit against $I$, the
true image:
\begin{lstlisting}
 I_alpha = pcg(@(in)apply(in,K1,K2,N1,N2,alpha),K_Ibn(:),1e-6,1500);
 misfit(k) = norm((K1 * (K2 * reshape(I_alpha,N2,N1))')' - I);
\end{lstlisting}
\begin{figure}[!htb]
        \centering
        \begin{subfigure}[bh]{0.65\textwidth}
                \includegraphics[width=\textwidth]{plots/tikrecon0001.pdf}
                \caption{$\alpha=0.0001$}
        \end{subfigure}%
        \begin{subfigure}[bh]{0.65\textwidth}
                \includegraphics[width=\textwidth]{plots/tikrecon001.pdf}
                \caption{$\alpha=0.001$}
        \end{subfigure}
 \caption{The reconstructed image.}
\label{fig:tik}
\end{figure}

The ``true'' image would imply then that the L-curve is too conservative
an estimate, and smaller values of $\alpha$ are acceptable. However, I
don't believe that. Figure 9 plots the Tikhanov
reconstruction for $\alpha = 0.001$ and $\alpha = 0.0001$. Clearly the
true image is recognizable in the right hand side, and not at all in the
left. So I believe that I either have a bug in my calculation of the
actual error, or the L-curve is, somehow, a more reliable estimate of
``good'' values of $\alpha$, here.

\end{document}


% !TEX root = ../../../proposal.tex

\section{TLS}\label{sec:subgroup_tls}

TLS (Transport Layer Security) is a transport layer protocol designed to provide confidentiality,
integrity and (most commonly) one-side authentication for application sessions.
It is widely used to protect HTTP and mail protocols.

A TLS client initiates a TLS handshake with the \texttt{Client\-Hello} message.
This message includes a list of supported cipher suites, and a client random
nonce $r_c$. The server responds with a \texttt{Server\-Hello} message containing
the chosen cipher suite and server random nonce $r_s$, and a
\texttt{Certificate} message that includes the server's X.509 certificate. If
the server selects a cipher suite using ephemeral Diffie-Hellman key exchange,
the server additionally sends a \texttt{Server\-Key\-Exchange} message containing
the server's choice of Diffie-Hellman parameters $p$ and $g$, the server's
Diffie-Hellman public value $y_s = g^{x_s} \bmod p$, a signature by the
server's private key over both the client and server nonces ($r_c$ and $r_s$),
and the server's Diffie-Hellman parameters ($p$, $g$, and $y_s$). The client
then verifies the signature using the public key from the server's certificate,
and responds with a \texttt{Client\-Key\-Exchange} message containing the client's
Diffie-Hellman public value $y_c = g^{x_c} \bmod p$. The Diffie-Hellman shared
secret $Y = g^{x_s x_c} \bmod p$ is used to derive encryption and MAC keys. The
client then sends \texttt{Change\-Cipher\-Spec} and \texttt{Finished} messages. The
\texttt{Finished} message contains a hash of the handshake transcript, and is
encrypted and authenticated using the derived encryption and MAC keys. Upon
decrypting and authenticating this message, the server verifies that the hash
of the transcript matches the expected hash.  Provided the hash matches, the
server then sends its own \texttt{Change\-Cipher\-Spec} and \texttt{Finished}
messages, which the client then verifies. If either side fails to decrypt or
authenticate the \texttt{Finished} messages, or if the transcript hashes do not
match, the connection fails immediately~\cite{rfc5246}.

TLS also specifies a mode of using Diffie-Hellman with fixed parameters from
the server's certificate~\cite{rfc3279}. This mode is not forward secret, was
never widely adopted, and has been removed from all modern browsers due to
dangerous protocol flaws~\cite{kci-tls-2015}. The only widely used form of
Diffie-Hellman in TLS today is ephemeral Diffie-Hellman, described above.

\subsection{Small Subgroup Attacks in TLS}
\label{sec:tls-subgroup-attack}

\paragraph{Small subgroup confinement attacks}
A malicious TLS server can perform a variant of the small subgroup attack
against a client by selecting group parameters $g$ and $p$ such that $g$
generates an insecure group order. TLS versions prior to 1.3 give the server
complete liberty to choose the group, and they do not include any method for
the server to specify the desired group order $q$ to the client.  This means a
client has no feasible way to validate that the group sent by the server has the
desired level of security or that a server's key exchange value is in
the correct group for a non-safe prime.
\looseness=-1

Similarly, a man in the middle with knowledge of the server's long-term private
signing key can use a small subgroup confinement attack to more easily
compromise perfect forward secrecy, without having to rewrite an entire
connection. The attack is similar to the those described by Bhargavan and
Delignat-Lavaud~\cite{bhargavan-channel-bindings-2015}. The attacker modifies the
server key exchange message, leaving the prime unchanged, but substituting a
generator $g_i$ of a subgroup of small order $q_i$ for the group generator and
$g_i$ for the server's key exchange value $y_s$. The attacker then forges a correct signature for the modified server key exchange message and passes it to the client.  The client then
responds with a client key exchange message $y_c = g_i^{x_c} \bmod p$, which
the man-in-the-middle leaves unchanged. The server's view of the shared secret
is then $g_i^{x_c x_s} \bmod p$, and the client's view of the shared secret is
$g_i^{x_c} \bmod p$. These views are identical when ${x_s} \equiv 1 \bmod q_i$,
so this connection will succeed with probability $1/q_i$. For small enough
$q_i$, this enables a man in the middle to use a compromised server signing key to
decrypt traffic from forward-secret ciphersuites with a reasonable probability
of success, while only requiring tampering with a single handshake message,
rather than having to actively rewrite the entire connection for the duration of the session.

Furthermore, if the server uses a static Diffie-Hellman key exchange value,
then the attacker can perform a small subgroup key-recovery attack as the
client in order to learn the server's static exponent $x_s \bmod q_i$ for the
small subgroup. This enables the attacker to calculate a custom generator such
that the client and server views of the shared secret are always identical,
raising the above attack to a 100\% probability of success.

\paragraph{Small subgroup key recovery attacks} In TLS, the client must
authenticate the handshake before the server, by providing a valid
\texttt{Finished} message. This forces a small subgroup key recovery attack
against TLS to be primarily online. To perform a Lim-Lee small subgroup key
recovery attack against a server static exponent, a malicious client initiates
a TLS handshake and sends a generator $g_i$ of a small subgroup of order $q_i$
as its client key exchange message $y_c$.  The server will calculate $Y_s =
g_i^{x_s} \bmod p$ as the shared secret. The server's view of the shared secret
is confined to the subgroup of order $q_i$.  However, since $g_i$ and $g$
generate separate subgroups, the server's public value $y_s = g^x_s$ gives the
attacker no information about the value of the shared secret $Y_s$. Instead,
the attacker must guess a value for $x_s \bmod q_i$, and send the corresponding
client \texttt{Finished} message. If the server continues the handshake, the
attacker learns that the guess is correct.  Therefore, assuming the server is
reusing a static value for $x_s$, the attacker needs to perform at most $q_i$
queries to learn the  server's secret $x_s \bmod q_i$~\cite{lim-1997}. This
attack is feasible if $q_i$ is small enough and the server reuses
Diffie-Hellman exponents for sufficiently many requests.

The attacker repeats this process for many different primes $q_i$, and uses the
Chinese remainder theorem to combine them modulo the product of the primes
$q_i$.  The attacker can also use the Pollard lambda algorithm to reconstruct
any remaining bits of the exponent~\cite{lim-1997}.

We note that the TLS False Start extension allows the
server to send application data before receiving the client's authentication~\cite{rfc7918}.
The specification only allows this behavior for abbreviated handshakes, which
do not include a full key exchange.  If a full key exchange were allowed, the
fact that the server authenticates first would allow a malicious client to
mount a mostly offline key recovery attack.
%\todo{Add discussion of the old unused TLS extension that makes this offline. \ref{rfc7918}}

\subsection{OpenSSL}

Prior to early 2015, OpenSSL defaulted to using static-ephemeral Diffie-Hellman
values. Server applications generate a fresh Diffie-Hellman secret exponent on
startup, and reuse this exponent until they are restarted.  A server would be
vulnerable to small subgroup attacks if it chose a DSA prime, explicitly
configured the \texttt{dh->length} parameter to generate a short exponent, and
failed to set \texttt{SSL\_OP\_SINGLE\_DH\_USE} to prevent repeated exponents.
OpenSSL provides some test code for key generation which configures DSA group
parameters, sets an exponent length to the group order, and correctly sets the
\texttt{SSL\_OP\_SINGLE\_DH\_USE} to generate new exponents on every
connection.  We found this test code widely used across many applications.  We
discovered that Unbound, a DNS resolver, used the same parameters as the tests,
but without setting \texttt{SSL\_OP\_SINGLE\_DH\_USE}, rendering them
vulnerable to a key recovery attack.  A number of other applications including
Lighttpd used the same or similar code with non-safe primes, but correctly set
\texttt{SSL\_OP\_SINGLE\_DH\_USE}.

In spring 2015, OpenSSL added explicit support for RFC~5114
groups~\cite{openssl-changelog-102}, including the ability for servers to
specify a subgroup order in a set of Diffie-Hellman group parameters. When the
subgroup order is specified, the exponent length is automatically adjusted to
match the subgroup size.  However, the update did not contain code to validate
subgroup order for key exchange values, leaving OpenSSL users vulnerable to
precisely the key recovery attack outlined in
Section~\ref{sec:tls-subgroup-attack}.

We disclosed this vulnerability to OpenSSL in January 2016. The vulnerability
was patched by including code to validate subgroup order when a subgroup was
specified in a set of Diffie-Hellman parameters and setting
\texttt{SSL\_OP\_SINGLE\_DH\_USE} by default~\cite{openssl-secadv-subgroup}.
Prior to this patch, any code using OpenSSL for DSA-style Diffie-Hellman
parameters was vulnerable to small subgroup attacks by default.

\ApplicationsTable

Exim~\cite{exim}, a popular mail server that uses OpenSSL, provides a clear
example of the fragile situation created by this update. By default, Exim uses
the RFC~5114 Group 23 parameters with OpenSSL, does not set an exponent length,
and does not set \texttt{SSL\_OP\_SINGLE\-\_DH\_USE}. In a blog post, an Exim
developer explains that because of ``numerous issues with automatic generation
of DH parameters'', they added support for fixed groups specified in RFCs and picked
Group~23 as the default~\cite{exim-blog}.  Exim narrowly avoided being fully
vulnerable to a key recovery attack by not including the size of the subgroup
generated by $q$ in the Diffie-Hellman parameters that it passes to OpenSSL.
Had this been included, OpenSSL would have automatically shortened the exponent
length, leaving the server fully vulnerable to a key recovery attack.  For this
group, an attacker can recover 130 bits of information about the secret
exponent using $2^{33}$ online queries, but this does not allow the attacker to
recover the server's 2048-bit exponent modulo the correct 224-bit group order
$q$ as the small subgroup orders $q_i$ are all relatively prime to $q$.

We looked at several other applications as well, but did not find them to be
vulnerable to key recovery attacks (Table~\ref{tab:common-applications}).

% comprising 53.62\% of all such servers~\cite{esoft2016}.

% Exim can be compiled either with GNU TLS or with OpenSSL. When compiled with OpenSSL,
% it supports single use DH keys with an option that is off by default
% (as of version 4.87, the latest at time of writing). In OpenSSL mode, it
% includes checks for unsafe primes and incorrect generators, however it
% ignores the generator check when the generator is not 2 or 5, due to a known
% bug in OpenSSL. More significantly, it utilizes the DSA primes with small
% subgroups. Their code comments note ``These have been thoroughly reviewed as
% meeting certain eligibility criteria, which is more than can be said for primes
% generated quickly''. However, they fail to note the difference in criteria for
% the DSA primes as opposed to the others, resulting in the selection of vulnerable
% groups. The subgroup size $q$ was not included in the Exim's default DSA parameter, saving
% a default Exim installation from being vulnerable to the attack described
% in Section~\ref{sec:tls-subgroup-attack} due the fact that OpenSSL did not have the possibility
% to match the exponent length to subgroup size. Said that, if the DSA parameter would have been generated using the
% OpenSSL's genpkey utility~\cite{genpkey} all the conditions to perform the Small subgroup key recovery attack
% as per Section ~\ref{sec:tls-subgroup-attack} would have been met.
% In comparison Postfix an MX  utilized for 32.80\% of mail servers~\cite{esoft2016}
% suggests generating a group at install time but neglects to set the single use OpenSSL option
% and includes two hardcoded groups by default.

\subsection{Other Implementations} \label{subsec:nss}

We examined the source code of multiple TLS implementations
(Table~\ref{tab:tls-implementations}). Prior to January 2016, no TLS
implementations that we examined validated group order, even for the well-known
DSA primes from RFC~5114, leaving them vulnerable to small subgroup confinement attacks.

Most of the implementations we examined attempt to match exponent length to the
perceived strength of the prime. For example, Mozilla Network Security Services
(NSS), the TLS library used in the Firefox browser and some versions of
Chrome~\cite{nss-overview,chrome-to-openssl}, uses NIST's ``comparable key strength'' recommendations
on key management~\cite{sp800} to determine secret exponent lengths from the length of the prime.~\cite{nss-line-of-code}  Thus NSS uses 160-bit exponents with a 1024-bit prime, and 224-bit exponents with a 2048-bit prime.
 In fall 2015, NSS added an additional check to ensure that the
shared secret $g^{x_ax_b} \not \equiv 1 \bmod p$~\cite{nss-code-secret-not-one}.

Several implementations go to elaborate lengths to match exponent length to
perceived prime strength.  The Cryptlib library fits a quadratic curve to the
small exponent attack cost table in the original van~Oorschot
paper~\cite{van1996diffie} and uses the fitted curve to determine safe key
lengths~\cite{cryptlib-fitted-curve}. The Crypto++ library uses an explicit
``work factor'' calculation, evaluating the function $2.4 n^{1/3} (\log
n)^{2/3}$~\cite{cryptoplusplus-work-factor}. Subgroup order and exponent
lengths are set to twice the calculated work factor. The work factor
calculation is taken from a 1995 paper by Odlyzko on integer
factorization~\cite{odlyzko-1995}. Botan, a C++ cryptography and TLS library,
uses a similar work factor calculation, derived from RFC~3766~\cite{rfc3766},
which describes best practices as of 2004 for selecting public key strengths
when exchanging symmetric keys. RFC~3766 uses a similar work factor algorithm
to Odlyzko, intended to model the running time of the number-field
sieve. Botan then doubles the length of the work factor to obtain subgroup and
exponent lengths~\cite{botan-double}.

% Apple Safari on OS X and iOS performed no validation of
% Diffie-Hellman prime lengths until July 2015, and supported connections where
% the server offered a trivially-broken group using a 16-bit prime~\cite{weakdh-ccs15}.

\subsection{Measurements}
\label{sec:tls-measurements}

We used ZMap~\cite{zmap-2013} to probe the public IPv4 address space for hosts
serving three TLS-based protocols: HTTPS, SMTP+STARTTLS, and POP3S\@.  To
determine which primes servers were using, we sent a \texttt{ClientHello}
message containing only ephemeral Diffie-Hellman cipher suites.  We combined
this data with scans from Censys~\cite{censys} to determine the overall
population.  The results are summarized in Table~\ref{tab:scandata}.

\ScanTable

\TLSHostValidationTable

In August 2016, we conducted additional scans of a random 1\% sample
of HTTPS hosts on the Internet.  First, we checked for nontrivial
small subgroup attack vulnerability. For servers that sent us a prime
$p$ such that $p-1$ was divisible by 7, we attempted a handshake using
a client key exchange value of $g_7\bmod p$, where $g_7$ is a
generator of a subgroup of order $7$.  (7 is the smallest prime factor
of $p-1$ for Group 22.) When we send $g_7$, we expect to correctly
guess the \texttt{PreMasterSecret} and complete the handshake with one
seventh of hosts that do not validate subgroup order. In our scan, we
were able to successfully complete a handshake with $1477$ of $10714$
hosts that offered a prime such that $p-1$ was divisible by $7$,
implying that approximately 96\% of these hosts fail to validate
subgroup order six months after OpenSSL pushed a patch adding group
order validation for correctly configured groups.

Second, we measured how many hosts performed even the most basic
validation of key exchange values. We attempted to connect to HTTPS hosts with
the client key exchange values of $y_c = 0 \bmod p, 1 \bmod p, -1 \bmod p$. As
Table~\ref{tab:tlsvalidation} shows, we found that over 5\% of hosts that
accepted DHE ciphersuites accepted the key exchange value of $-1 \bmod p$ and
derived the \texttt{PreMasterSecret} from it. These implementations are
vulnerable to a trivial version of the small subgroup confinement attacks
described in Section~\ref{sec:tls-subgroup-attack}, for \emph{any} prime
modulus $p$. By examining the default web pages of many of these hosts,
we identified products from several notable companies including Microsoft,
Cisco, and VMWare. When we disclosed these findings, VMWare notified us that
they had already applied the fix in the latest version of their products;
Microsoft acknowledged the missing checks but chose not to include them since
they only use safe primes, and adding the checks may break functionality
for some clients that were sending unusual key exchange values; and Cisco
informed us that they would investigate the issue.

Of 40.6\,M total HTTPS hosts found in our scans, 10.8\,M~(27\%) supported
ephemeral Diffie-Hellman, of which 1.6\,M~(4\%) used a non-safe prime, and
309\,K~(0.8\%) used a non-safe prime and reused exponents across multiple
connections, making them likely candidates for a small subgroup key recovery
attack.  We note that the numbers for hosts reusing exponents are an
underestimate, since we only mark hosts as such if we found them using the same
public Diffie-Hellman value across multiple connections, and some load
balancers that cycle among multiple values might have evaded detection.

While 77\%~of POP3S hosts and 39\%~of SMTP servers used a non-safe prime, a
much smaller number used a non-safe prime and reused exponents (<0.01\% in both
protocols), suggesting that the popular implementations (Postfix and
Dovecot~\cite{mail-2015}) that use these primes follow recommendations to use
ephemeral Diffie-Hellman values with DSA primes.

Table~\ref{tab:primes} shows nine groups that accounted for the majority of
non-safe primes used by hosts in the wild. Over 1.17\,M hosts across all of our
HTTPS scans negotiated Group~22 in a key exchange. To get a better picture of
which implementations provide support for this group, we examined the default
web pages of these hosts to identify companies and products, which we show in
Table~\ref{tab:tls-group22-support}.

\TLSGroupSupport

Of the the 307\,K HTTPS hosts that both use non-safe primes and reuse
exponents, 277\,K~(90\%) belong to hosts behind Amazon's Elastic Load
Balancer~\cite{amazon-elb}. These hosts use a 1024-bit prime with a 160-bit
subgroup. We set up our own load balancer instance and found that the
implementation failed to validate subgroup order. We were able to use a
small-subgroup key recovery attack to compute 17 bits of our load balancer's
private Diffie-Hellman exponent $x_s$ in only 3813 queries.  We responsibly
disclosed this vulnerability to Amazon. Amazon informed us that they have
removed Diffie-Hellman from their recommended ELB security policy, and are
encouraging customers to use the latest policy.  In May 2016, we performed
additional scans and found that 88\% of hosts using this prime no longer
repeated exponents.   We give a partial factorization for $p-1$ in
Table~\ref{tab:group-order-factorization}; the next largest subgroups have 61
and 89 bits and an offline attack against the remaining bits of a 160-bit
exponent would take $2^{71}$ time.  For more details on the computation, see
Section~\ref{sec:ecm}.

SSLeay~\cite{ssleay}, a predecessor for OpenSSL, includes several default
Diffie-Hellman primes, including a 512-bit prime. We found that 717 SMTP
servers used a version of the OpenSSL 512-bit prime with a single character
difference in the hexadecimal representation.  The resulting modulus that these
servers use for their Diffie-Hellman key exchange is no longer prime. We
include the factorization of this modulus along with the factors of the
resulting group order in Table~\ref{tab:group-order-factorization}. The use of
a composite modulus further decreases the work required to perform a small
subgroup attack.

Although TLS also includes static Diffie-Hellman cipher suites that require a
DSS certificate, we did not include them in our study; no browser supports
static Diffie-Hellman~\cite{kci-tls-2015}, and Censys shows no hosts with DSS
certificates, with only 652 total hosts with non-RSA or ECDSA certificates.

%In February 2016, we conducted an additional scan of a random 1\% sample of the
%Internet to check how many hosts performed even the most basic validation of
%key exchange values. We attempted to connect to HTTPS hosts with a client
%key exchange message of $y_c = 1$. We found that 322 of 589,241 hosts accepted
%this key exchange value and derived the \texttt{PreMasterSecret} from it. 248
%of these hosts were Cisco devices, another 28 were VMware Horizon View servers,
%and the remaining 46 devices did not fit into any clear category. These
%implementations are vulnerable to a trivial version of the small subgroup
%confinement attacks described in Section~\ref{sec:tls-subgroup-attack}.\todo{did we disclose this? we need to. also what about validation of values other than 0?}

%\TODO{classify implementations again and disclose to Cisco, etc}

\begin{landscape}
\TLSLibraryTable
\PrimesAllTLS
\end{landscape}


%%% Copyright 2017 International Business Machines Corporation
\documentclass[11pt]{article} % use larger type; default would be 10pt

\usepackage[utf8]{inputenc} % set input encoding (not needed with XeLaTeX)
\usepackage[T1]{fontenc}

%%% PAGE DIMENSIONS
\usepackage[top=1.5in,bottom=1in,right=1in,left=1in,headheight=90pt,headsep=1cm]{geometry} % to change the page dimensions
\geometry{a4paper} % or letterpaper (US) or a5paper or....
 \geometry{margin=0.7in} % for example, change the margins to 2 inches all round

\usepackage{graphicx} % support the \includegraphics command and options

 \usepackage[parfill]{parskip} % Activate to begin paragraphs with an empty line rather than an indent

%%% PACKAGES
\usepackage{amsmath}
\usepackage{cases}
\usepackage{ulem}
\usepackage{listings}

\usepackage{fancyhdr} % This should be set AFTER setting up the page geometry
\pagestyle{fancy} % options: empty , plain , fancy
\renewcommand{\headrulewidth}{0.4pt} % customise the layout...
\lhead{}\chead{\footnotesize \textit{IBM}}\rhead{}
\lfoot{}\cfoot{\thepage}\rfoot{}

%%% SECTION TITLE APPEARANCE
\usepackage{sectsty}

\newenvironment{aside}
  {\begin{mdframed}[style=0,%
      leftline=false,rightline=false,leftmargin=2em,rightmargin=2em,%
          innerleftmargin=0pt,innerrightmargin=0pt,linewidth=0.75pt,%
      skipabove=7pt,skipbelow=7pt]\small}
  {\end{mdframed}}

%COLOURING
\usepackage{color}
\newcommand{\new}{\textcolor{red}}
%\input{rgb}

%%% ToC (table of contents) APPEARANCE
\usepackage[nottoc,notlof,notlot]{tocbibind} % Put the bibliography in the ToC

\title{The Iterative Reweighted Least Squares Method\\ for $\ell_1$ Norm Minimisation}
\author{Cecilia Aas}
\date{October, 2015}

\def\layersep{2.5cm}

\begin{document}
\lstset{language=C++,
           commentstyle=\textcolor[rgb]{0.00,0.66,0.33},
           keywordstyle=\textcolor[rgb]{0.00,0.00,1.00},
           basicstyle=\footnotesize\ttfamily,
           frame=lines,
           framexleftmargin=2mm,
           numbers=left,
           numberstyle=\footnotesize,
           stepnumber=1,
           numbersep=1pt}
\maketitle
\section{Overview}
The algorithm aims to solve the problem
\begin{equation}
\label{eq:l1min}
\min\left|\left| \uline{x} \right| \right|_1 \text{ subject to } \uuline{A}\uline{x} = \uline{y}
\end{equation}
by finding the minimisers $\left\{x_0, x_1, \cdots \right\}$ of the functional
\begin{equation}
J_{\lambda}(x) = \frac{1}{2} \left| \left|\uuline{A} \uline{x} - \uline{y} \right| \right|_2^2 + \lambda \left| \left| \uline{x} \right| \right|_1 \; .
\end{equation}
for decreasing $\lambda$.  The algorithm does this by tracing out the \textit{homotopy path}, approaching the limit
\begin{equation}
\lim_{\lambda \rightarrow 0} J_{\lambda}(x) = \frac{1}{2} \left| \left|\uuline{A} \uline{x} - \uline{y} \right| \right|_2^2 \; .
\end{equation}

\section{Variables}
We have the following variables,
\begin{itemize}
\item $m \times n$ sensing matrix, $\uuline{A}$
\item signal vector of $m$ elements, $\uline{y}$
\item solution vector of $n$ elements, $\uline{x}$
\item weights vector of $n$ elements, $\uline{w}$
\item the error, $\epsilon$
\item iteration index $j$
\end{itemize}

\section{Algorithm}

The iterative reweighted least square algorithms requires initialisation of
\begin{itemize}
\item the value of $\epsilon_0 = 0$
\item the weights vector to $\uline{w} = \uline{1}$ (i.e., a vector with all elements equal to one)
\end{itemize}
In each iterative step $j$, the iterative reweighted least square algorithms updates the solution approximation, $\uline{x}^{(j)}$, and the weights, $\uline{w}^{(j)}$, as follows.
\begin{itemize}
\item the approximate solution is updated
\begin{equation}
\uline{x}^{(j+1)} = \uuline{D}_j^n \uuline{A}^T\left(\uuline{A} \uuline{D}^{-1} \uuline{A}^T\right) \uline{y}
\end{equation}
where the matrix $\uuline{D}$ is a diagonal matrix containing the weights,
\begin{equation}
D_{ii} = w_i^{(j)}
\end{equation}
\item the value of epsilon is updated
\begin{equation}
\epsilon_{j+1} =  \min\left(\epsilon_n, \frac{r\left(x^{(j+1)}\right)_{K+1}}{N}\right)
\end{equation}
where the vector $\uline{r}\left(\uline{x}\right)$ is defined to contain the non-increasing arrangement of the absolute values of the elements in $\uline{x}$; $r\left(\uline{x}\right)_i$ is thus the $i^{\text{th}}$ largest element of $\uline{x}$; a vector $\uline{x}$ is $k$-sparse if and only if $r\left(\uline{x}\right)_{k+1} = 0$
\item the weights are updated
\begin{equation}
w_j^{(j+1)} = \frac{1}{\sqrt{\left(x_j^{(j+1)}\right)^2 + \epsilon_{j+1}^2}}
\end{equation}
\end{itemize}

The loop breaks either
\begin{itemize}
\item when reaching the maximum number of iterations, $N_{iter}$, or
\item when the infinity-norm of the residual vector (i.e., the lambda parameter in Eq.~(\ref{eq:l1min})) becomes smaller than the pre-set tolerance,
\end{itemize}
whichever happens first.
\end{document}


\chapter{Reinforcement Learning}
\label{ch:reinforcement_learning}

The standard way to solve Markov decision processes is through dynamic programming, which simply consists in solving the Bellman fixed-point equations discussed in the previous chapter. Following this approach, the problem of finding the optimal policy is transformed into the problem of finding the optimal value function. However, apart from the simplest cases where the MDP has a limited number of states and actions, dynamic programming becomes computationally infeasible. Moreover, this approach requires complete knowledge of the Markov transition kernel and of the reward function, which in many real-world applications might be unknown or too complex to use. \glsfirst{RL} is a subfield of Machine Learning which aims to turn the infeasible dynamic programming methods into practical algorithms that can be applied to large-scale problems. \gls{RL} algorithms are based on two key ideas: the first is to use samples to compactly represent the unknown dynamics of the controlled system. The second idea is to use powerful function approximation methods to compactly estimate value functions and policies in high-dimensional state and action spaces. In the following sections, we present the \gls{RL} problem in more depth and discuss how it relates to the standard discrete-time stochastic optimal control theory and to the other subfields of machine learning, such as supervised learning. In particular, we will see how RL extends ideas from optimal control theory and stochastic approximation to address the broader goal of artificial intelligence. This quick overview of RL is propaedeutic to the following chapters, where we will present in more detail a particular class of algorithms, called policy gradient methods, which are well suited for continuous action spaces. For a more thorough presentation, the reader may consult \cite{sutton1998introduction}, \cite{szepesvari2010algorithms} or \cite{wiering2012reinforcement}.

\section{The Reinforcement Learning Problem}
Reinforcement Learning (RL) is a general class of algorithms in the field of machine learning that allows an agent to learn how to behave in a stochastic and possibly unknown environment, where the only feedback consists of a scalar reward signal. In order to maximize the long-run reward, the agent must learn which actions are the most profitable by trial-and-error. Therefore, RL algorithms can be seen as computational methods to solve Markov decision processes by directly interacting with the environment, for which a model may or may not be available. Trial-and-error search and a delayed reward signal can be seen as the most characteristic features of reinforcement learning.\\
Compared to supervised learning, one of the main branches of machine learning, the feedback the learner receives is much less. In supervised learning, the agent is provided with examples of the correct or expected behavior by a knowledgeable external supervisor and the agent's goal is to learn how to replicate these examples as well as possible and possibly generalize this knowledge to new examples. In reinforcement learning, the agent only receives a numerical reward that only gives a partial feedback of the goodness of actions taken. Therefore, this feedback system is evaluative rather than instructive and it is much more difficult for the agent to learn how to behave in uncharted territory without any external guidance.\\
This particular framework generates some challenges that are not present in other kinds of learning. The first one is the trade-off between exploration and exploitation. In order to maximize his reward, an agent would greedily select actions that have already been tried in the past and found to be effective in producing rewards. However, to find these actions, the agent must also test actions that have not been chosen before in order to evaluate their potential. Clearly, this might result in worse performance in the short-term because the actions might be suboptimal. However, without trying them, the agent might not be able to find possible improvements. Thus, an agent must exploit what is known to obtain rewards but also needs to explore to select better actions in the future. A second challenge is the credit assignment problem. Since rewards might be delayed in time, it will be difficult for the agent to understand which actions are mostly responsible for the outcome.

\section{Model-Free RL Methods}
In Section \ref{sec:policy_evaluation}, we discussed the policy iteration method for computing an optimal policy for an \gls{MDP} in a finite state and action spaces. This algorithm belongs to the class of \emph{model-based} methods, since it requires perfect knowledge of the Markov transition kernel and reward function, which consists of a model of the MDP. RL is primarily concerned with how to obtain an optimal policy when such a model is not available. In this section, we discuss some classes of \emph{model-free} methods which do not rely on the MDP model. The lack of a model generates the need to sample the MDP to gather statistical knowledge about this unknown model. In the control setting, the goal is to approximate the optimal policy, which depends on the optimal value, which in turn depends on the model of the MDP, as shown in Figure \ref{fig:control_dependences}. Indeed, we have already seen that a policy which is greedy with respect to the optimal action-value funtion $Q_*$, namely for which holds
\begin{equation}
	\int_{\A} \pi_*(s,a) Q_*(s,a) da = \sup_a Q_*(s,a)
\end{equation}
is optimal. Therefore, we can derive the following three methodologies that differ in which part of the solution process is approximated
\begin{enumerate}[label={\roman*)}]
	\item \emph{Model-approximation} algorithms approximate the MDP model and compute an estimate of the optimal policy by dynamic programming.
	\item \emph{Value-approximation} algorithms use samples to directly approximate $V_*$ or $Q_*$, from which an estimate of $\pi_*$ can be derived by acting greedily.
	\item \emph{Policy-approximation} algorithms directly try to estimate the optimal policy.
\end{enumerate}
It should be noticed that these approaches are not mutually exclusive and can be combined to derive hybrid algorithms. In the following sections we discuss these three classes of algorithm in more detail, following closely \cite{wiering2012reinforcement}.

\begin{figure}[t]
	\centering
	\begin{tikzpicture}[node distance = 6em, auto, thick]
		\node [block] (Model) {Model\\$\calP$, $\calR$};
		\node [block, below of=Model] (Value) {Value functions\\$V_*$, $Q_*$};
		\node [block, below of=Value] (Policy) {Policy\\$\pi_*$};
		\path [line] (Model) edge (Value);
		\path [line] (Value) edge (Policy);
	\end{tikzpicture}
	\caption{Solution process for the control problem.}
	\label{fig:control_dependences}
\end{figure}

\subsection{Model Approximation}
Model-approximation algorithms approximate the MDP model and compute an optimal policy by dynamic programming, using the techniques discussed in the previous chapter. Since $\S$, $\A$ and $\gamma$ are assumed to be known, these methods are based on learning an approximation of the Markov transition kernel $\calP$ and the reward function $\calR$. Thanks to the Markov property, these quantities only depend on the current state and action, so that their approximation corresponds to a density estimation problem and a regression problem respectively, which are fairly standard supervised learning problems. Learning the model may not be trivial, but it is in general easier than learning the value of a policy or optimizing the policy directly. The major drawback of model-based algorithms in continuous-state MDPs is that, even if the model is available, it is in general infeasible to compute the value functions by dynamic programming and to extract an optimal policy for all states by acting greedily. Alternatively, a transition model estimate may be used to generate sample trajectories from the MDP, which can then be used to estimate the value function or directly improve the policy. However, the value function and the policy estimated using these samples can only be as accurate as the learned model, so that in many cases it may be easier to directly approximate the value function or the policy using the methods described below.

\subsection{Value Approximation}
Value-approximation algorithms use samples from the MDP to approximate $V_*$ or $Q_*$ directly and then derive an estimate of the optimal policy by acting greedily with respect to $Q_*$. Typically, when the state and action spaces are large, the value functions are estimated using a parametric function approximator, whose parameters are iteratively updated given the observed samples. Many reinforcement learning algorithms fall into this category and they can be distinguishes based on whether they are on-policy or off-policy and whether they update the value function estimates online or offline. \emph{On-policy} algorithms approximate the state-value function $V_\pi$ or the action-value function $Q_\pi$ from samples of the MDP obtained by following the same policy $\pi$ to be evaluated. Although the optimal policy $\pi_*$ is initially unknown, such algorithms can eventually approximate the optimal value functions $V_*$ or $Q_*$ from which an approximation of the optimal policy can be derived. On the other hand, \emph{off-policy} algorithms can learn the value of a policy different from the one use for obtaining the MDP samples. \emph{Online} algorithms adapt their value approximation after each observed sample while \emph{Offline} algorithms operate on batches of samples. Online algorithms typically require less computation per sample but their convergence is slower.
%Online on-policy algorithms include temporal-difference (TD) algorithms, such as TD-learning, Sarsa. Offline on-policy algorithms include least-squares approaches, such least-squares temporal difference (LSTD), least-squares policy evaluation (LSPE) and least-squares policy iteration (LSPI). The most known model-free online off-policy algorithm is Q-learning.

\subsection{Policy Approximation}
Policy-approximation algorithms only store a policy and update this policy to maximize a given performance measure and eventually converge to the optimal policy. Since these algorithms only use an estimate of the optimal policy and do not rely on a value function approximation, they are also referred to as \emph{direct policy-search} or \emph{actor-only} algorithms. Algorithms that store both a policy and a value function are commonly known as \emph{actor-critic} algorithms \cite{sutton1999policy}, \cite{konda1999actor}. Policy gradient algorithms, the most studied policy-approximation methods, will be discussed in much more detail in the next chapter.


% !TeX root = ../main.tex
\begin{landscape}
\section{Semantics Derivation Tree Example}
\label{sec:semanticsTree}
This is an example of how to create a derivation tree for a program written in HCL.
It is based on the following code snippet.

\begin{lstlisting}[language=HCL,label=lis:derivationCode,firstnumber=1]
num x
x = 5
\end{lstlisting}

Using the abstract syntax described in section \ref{sec:semantics}, the code can be written as the following big step semantic:

\begin{center}
	$env_V, env_F \vdash \langle num\ x;\ x = 5, env_V, env_F, sto \rangle \rightarrow_{stm} (sto', env_V', env_F')$
\end{center}
Using the Compositional Big Step Semantic rule, the following derivation is produced:

\begin{center}
	\begin{math}
		\cfrac
		{env_V, env_F \vdash \langle num\ x, env_V, env_F, sto \rangle \rightarrow_{stm} (sto'', env_V'', env_F'')\quad env_V'', env_F'' \vdash \langle x = 5, env_V'', env_F'', sto'' \rangle \rightarrow_{stm} (sto', env_V', env_F')}
		{env_V, env_F \vdash \langle num\ x;\ x = 5, env_V, env_F, sto \rangle \rightarrow_{stm} (sto', env_V', env_F')}
	\end{math}
\end{center}

Table \ref{tbl:progstates} shows how the program state changes after each statement in the code is run.

\begin{table}[H]
	\centering
	\caption{Table of program states after each statement.}
	\label{tbl:progstates}
	\setlength\extrarowheight{5pt}
	\begin{tabular}{|c|c|c|}
		\hline
		& Abstract program state      & Concrete program state                          \\ \hline
		Initial state          & $(sto, env_V, env_F)$       & $(sto, env_V, env_F)$                           \\ \hline
		After first statement  & $(sto'', env_V'', env_F'')$ & $(sto[l \mapsto 0], env_V[x \mapsto l], env_F)$ \\ \hline
		After second statement & $(sto', env_V', env_F')$    & $(sto[l \mapsto 5], env_V[x \mapsto l], env_F)$  \\ \hline
	\end{tabular}
\end{table}

\end{landscape}