\section{Comparing Storkey and Hebbian learning}

When training a Hopfield network, we have employed two types of learning: the
first one is the Hebbian learning, while the second one is Storkey learning.
The advantage of the latter is that it increases the capacity of the network
and the basin of attraction of the clusters. We will now explain some
experiments we did with both types of learning in order to see how the
learning type affects the basin of attraction for clusters.

We will now show how the learning determines the basin of attraction of a
Gaussian distributed cluster. The generation was done using the T2 method
described in section ~\ref{sec:fexp}.

\begin{table}[h]
\centering
  \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    \tmtextbf{Learning} & N & Cluster size & $\mu$ & $\sigma$ & Average size
    of basin of attraction\\
    \hline
    Hebbian & 50 & 2 & 25 & 5 & 10.0\\
    \hline
    Storkey & 50 & 2 & 25 & 5 & 18.0\\
    \hline
    Hebbian & 50 & 4 & 25 & 5 & 1.5\\
    \hline
    Storkey & 50 & 4 & 25 & 5 & 4\\
    \hline
    Hebbian & 50 & 4 & 25 & 10 & 5.75\\
    \hline
    Storkey & 50 & 4 & 25 & 10 & 8.0\\
    \hline
    Hebbian & 50 & 5 & 25 & 5 & 4.4\\
    \hline
    Storkey & 50 & 5 & 25 & 5 & 4.8\\
    \hline
    Hebbian & 50 & 6 & 25 & 5 & 4.2\\
    \hline
    Storkey & 50 & 6 & 25 & 5 & 4.4\\
    \hline
    Hebbian & 50 & 6 & 25 & 10 & 4.45\\
    \hline
    Storkey & 50 & 6 & 25 & 10 & 5.61\\
    \hline
  \end{tabular}
  \caption{Results comparing Storkey and Hebbian learning for various parameters}
\end{table}

The results confirm what the mathematical theory showed us: Storkey learning
increases the average basin of attraction. We must note that this does not
come without a price: training the network using Storkey learning slowed down
our experiments, as it is more expensive. Depending to the application, this
is an acceptable trade off. All our functions and experiments can be performed
with both types of learning, just by changing a parameter (the learning type),
which enables any user of our libraries to make the choice depending on the
use case.

The following results show the difference between Storkey and Hebbian learning in the experiments we talked about before.
The aim of repeating the experiments with Storkey learning was to see what kind of impact the learning type has in respect to the average basin size.
Both curves (given by Hebbian and Storkey learning) follow the same shape, which also gives confidence in the initial results obtained with Hebbian learning: an initial fast drop in
average basin size, with a steadily increase afterwards.
\\*
The best way to express out results is by showing the two different trends, obtained with the 2 methods we have employed: T1 and T2.

\subsection{T1 Method}

For the T1 method, one can easily notice that Storkey learning gives bigger basin sizes. While this was expected when experimenting with one cluster
 (\ref{fig:plot-storkey-T1-onecluster}), the surprise comes from the second chart (\ref{fig:plot-storkey-T1-twoclusters}): the  increased basin size in the second cluster. Even though the network is trained with 2 clusters
and they affect each others basin sizes, both sizes increase with Storkey learning, without it giving a different ration between the average.

%T1

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T1-onecluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to a cluster generated using T1.}
\label{fig:plot-storkey-T1-onecluster}
\end{figure}

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T1-twocluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to two clusters generated using T1. The first cluster has fixed $p = 0.2$}
\label{fig:plot-storkey-T1-twoclusters}
\end{figure}

%T2
\clearpage
\subsection{T2 Method}

The results are different when looking at T2 type experiments:
the difference between the two learning methods is less straight forward.
When training a Hopfield network with one cluster (\ref{fig:plot-storkey-T2-onecluster})  there is a critical point from which the Storkey learning clearly gives a bigger basin size
(at a standard deviation approximately 5.0), but before that it seems like the two methods give very similar results.
\\*
Even more interesting results come from training the network with two clusters (as seen in \ref{fig:plot-storkey-T2-twoclusters}): surprisingly, when the standard deviation of the second cluster
is less than 9, the Hebbian learning provides a bigger basin size. After that point, Storkey learning acts as expected, by giving a
bigger basin size when compared to Hebbian. Note that the threshold point of this crossover is very close to the standard deviation of the first
cluster, which was fixed to 10.
\\* These results enables us to speculate that Storkey learning is more sensitive to non independent patterns. Both the results from T1 experiments with
2 clusters and the T2 experiments show this. However, we believe that more investigation needs to be done in order to have confidence in these results.

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T2-onecluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to a cluster generated using T2.}
\label{fig:plot-storkey-T2-onecluster}
\end{figure}

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T2-twocluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to two clusters generated using T2. The first cluster has fixed $\sigma = 10$}
\label{fig:plot-storkey-T2-twoclusters}
\end{figure}