\chapter{Simple Data Types}
PEARL defines the data types FIXED, FLOAT, BIT, CHARACTER, REF CHARACTER,
DURATION and CLOCK.

Specific for PEARL is the signal mechanism. It is expected that
each operation induces a signal (in C++ exception) if the operator
does no finish successfully. C++ does not provide this feature.

For each data type a separate class provides the required operations.
FIXED, FLOAT, BIT and CHARACTER may be defined in PEARL with a length
attribute. This is mapped to C++-templates.

For details about ctor parameters and member functions please check the
doxygen documentation, which can be created for a specific plattform
by \verb|make doc| in the corresponding directory.

The classes for the simple data types are named as:

\begin{tabular}{|l|l|l|}
\hline\\
PEARL name & class name & remarks\\
\hline
FIXED & Fixed & templated with the length\\
FLOAT & Float & templated with the length\\
BIT & BitString & templated with the length\\
CHAR & Character & templated with the length\\
REF CHAR & RefCharacter & \\
DURATION & Duration & \\
CLOCK & Clock & \\
\hline
\end{tabular}

\if0
\section{Fixed}
The class {\em Fixed63} provides save operations on fixed values
with 63 bits plus sign. This development of this class was influenced by
the header SafeInt.hxx, which is provided with the MSPL license.
This class is used internally for the implementation of fixed values
with individual lengths.

The application specific versions of  the type fixed are realized with the
C++ template mechanism in {\em Fixed.h},
 requiring the number of mantissa bits as template parameter.
All lengths from 1 to 63 are suported.
The operations are mapped to the usual C++-operators.
For PEARL-specific operations member functions are supplied.

All operations, which produce numerical {\em overflow}, {\em underflow}
 or {\em divide by zero}  throw an exception.

\begin{classSummary}
 Class & \verb|Fixed<int S>| \\
 Specification & Fixed.h \\
 Implementation & - \\
 \verb|size(Fixed<7>)| &   1 byte \\
 \verb|size(Fixed<15>)| & 2 bytes \\
 \verb|size(Fixed<31>)| & 4 bytes \\
 \verb|size(Fixed<63>)| & 8 byte  \\
\end{classSummary}

\begin{methodMapping}
  \verb|+| (monadic)      &        & to be treated by the compiler \\
  \verb|-| (monadic)    & \verb|operator-()| &  \\
 \verb|ABS|            & \verb|absVal()|  &  \\
 \verb|SIGN|           & \verb|sign()|  & \\
 \verb|TOBIT|          & \verb|BitString| &
                    realized as Ctor in BitString
                     with the component '.x' of the FIXED variable \\
 \verb|TOCHAR|         & \verb|toChar()|  & realized in Character.h \\
  \verb|+| (dyadic)     & \verb|operator+(rhs)| & \\
  \verb|-| (dyadic)     & \verb|operator-(rhs)| & \\
  \verb|*| (dyadic)     & \verb|operator*(rhs)| & \\
  \verb|//| (dyadic)   & \verb|operator/(rhs)| & \\
  \verb|REM| (dyadic)   & \verb|operator%(rhs)| & \\
  \verb|**| (dyadic)    & \verb|pow(rhs)|    & rhs is the exponent \\
  \verb|<|          & \verb|operator<(rhs)|  & \\
  \verb|>|           & \verb|operator>(rhs)|  & \\
  \verb|<=|         & \verb|operator<=(rhs)| & \\
  \verb|>=|         & \verb|operator>=(rhs)| & \\
  \verb|==|         & \verb|operator==(rhs)| &   \\
  \verb|/=|        & \verb|operator!=(rhs)|  &   \\
  FIT               & \verb|fit(rhs)|    & rhs defines target size \\
\end{methodMapping}

\paragraph{Remarks:}
\begin{itemize}
\item If no preset value is specified, the variable is initialized with 0.
\item There is no default length in the runtime system.
\item C++ literals correspond usually to 32 bit integers.
If we need longer constants the postfix \verb|LL| must be added by
the compiler to enforce 64-bit integers.
\end{itemize}

\begin{PEARLCode}
DCL x FIXED(31) INIT(15);
DCL y FIXED(33) INIT(16);
DCL z FIXED INIT(17);
DCL z1 FIXED;
DCL longFixed Fixed(53) INIT(123456789012);
z := x + y;
x := z FIT x;
x := z ** y;
\end{PEARLCode}

\begin{CppCode}
Fixed<31> x(15);
Fixed<33> y(16);
Fixed<31> z(17);
Fixed<31> z1;
Fixed<53> longFixed(123456789012LL);
z = x + y;
x = z.fit(x);
x = z.pow(y);
\end{CppCode}

\section{Float}

The type FLOAT is provided by templates with the lengths 24 and 53 bit
mantissa precission, both with the hidden one presentation.
These lengths are defined by the IEEE754
standard types single and double.

New float variables are initialized with {\em NaN}, which is defined
as \verb|NAN| in the {\em math.h} GNU-library if IEEE754 is
available on the current system. This preset avoids problems with
uninitialized float values in calculations.

Each operation (except assignment and comparison) contain checks
for unexpected results. A result of {\em NaN} or {\em INF} causes the signals
{\em FloatIsNaNSignal} or {\em FloatIsINFSignal} to be emitted.

The operations are mapped to the usual C++-operators.
For PEARL-specific operations member functions are supplied.

\begin{classSummary}
 Class & \verb|Float<24>| and \verb|Float<53>|\\
 Specification & Float.h \\
 Implementation & - \\
 \verb|size(Float<24>)| &   4 byte \\
 \verb|size(Float<53>)| & 8 bytes \\
\end{classSummary}

\begin{methodMapping}
  \verb|+| (monadic)      &        & to be treated by the compiler \\
  \verb|-| (monadic)    & \verb|operator-()| &  \\
 \verb|ABS|            & \verb|abs()|  &   \\
 \verb|SIGN|           & \verb|sign()|   &  \\
 \verb|ENTIER|          & \verb|entier()| & \\
 \verb|ROUND|          & \verb|round()| & \\
  \verb|+| (dyadic)     & \verb|operator+(rhs)| & rhs may be Fixed or Float\\
  \verb|-| (dyadic)     & \verb|operator-(rhs)| & rhs may be Fixed or Float\\
  \verb|*| (dyadic)     & \verb|operator*(rhs)| & rhs may be Fixed or Float\\
  \verb|/| (dyadic)   & \verb|operator/(rhs)| & rhs may be Fixed or Float\\
  \verb|**| (dyadic)    & \verb|pow(rhs)|    &
                rhs is the exponent with type \verb|Fixed<S>| \\
  \verb|<|          & \verb|operator<(rhs)|  & rhs may be Fixed or Float\\
  \verb|>|           & \verb|operator>(rhs)|  & rhs may be Fixed or Float \\
  \verb|<=|         & \verb|operator<=(rhs)| & rhs may be Fixed or Float \\
  \verb|>=|         & \verb|operator>=(rhs)| & rhs may be Fixed or Float \\
  \verb|==|         & \verb|operator==(rhs)| & rhs may be Fixed or Float \\
  \verb|/=|        & \verb|operator!=(rhs)|  & rhs may be Fixed or Float   \\
  FIT               & \verb|fit(rhs)|    & rhs defines target size \\
SIN			& \verb|sin()|	& rhs must be Float \\
COS			& \verb|cos()|	& rhs must be Float \\
TAN			& \verb|tan()|	& rhs must be Float \\
ATAN			& \verb|atan()|	& rhs must be Float \\
TANH			& \verb|tanh()|	& rhs must be Float \\
EXP			& \verb|exp()|	& rhs must be Float \\
LN			& \verb|ln()|	& rhs must be Float \\
SQRT			& \verb|sqrt()|	& rhs must be Float \\
\end{methodMapping}

\paragraph{Remarks:}
\begin{itemize}
\item There is no default length in the runtime system.
\item C++ literals are passed as {\em double} to ctor.
\item mixed operations with Fixed and Float are treated b non member methods
\item sin, cos, ... with fixed is not supported yet; this must be
   treated by the compiler with the semantic analysis by automatic
   type conversion towards Float.
\item entier and round behavior must be compared with existing implementations;
    this implementation behaves like:

    \begin{tabular}{c||c|c|c|c|c|c}
     x &         10.5 & 10.4 & 10.0 & -10.0 & -10.4 & -10.5 \\
     \hline
     entier(x) & 10 & 10& 10& -10& -11 & -11 \\
     \hline
     round(x) & 11& 10& 10& -10& -10& -11 \\
   \end{tabular}
\end{itemize}

\begin{PEARLCode}
DCL x FLOAT(24) INIT(1.5);
DCL y FLOAT(53) INIT(1.6);
DCL z FLOAT INIT(1.7);
DCL z1 FIXED(31);
y := x + y;
x := z FIT x;
x := z ** y;
z := SIN(y);
z := SIN(x+y);
\end{PEARLCode}

\begin{CppCode}
Float<24> x(1.5);
Float<53> y(1.6);
Float<53> z(1.7);
Float<31> z1;
y = x + y;
x = z.fit(x);
x = z.pow(y);
z = y.sin();
z = (x+y).sin();
\end{CppCode}

\section{Bit}
The type BIT is supported by the templated class {\em BitString}.
The name was choosen to make clear that there may be  more than one bit
in variable of this type.
The internal data type is used according the length of the BitString.
The smallest possible native data size is used (1,2,4 or 8 bytes).

The supported sizes are 1 to 64.

The alignment of the bits inside the BitString is left adjusted.
This means that the most significant bits are used. This was choosen due to
the definition of operations of BitStrings with differnt lengths.

Member functions are supplied for the PEARL operations.

Bit constants define their length intrinsic. All given digits are
considered to be part of the bit constant. E.g.

\begin{tabular}{|r|c|}
\hline
BIT constant & Type \\
\hline
'1'B & BIT(1) \\
'1'B2 & BIT(2) \\
'1'B3 & BIT(3) \\
'1'B4 & BIT(4) \\
'01'B4 & BIT(8) \\
\hline
\end{tabular}

\begin{classSummary}
 Class & \verb|BitString<int len>| \\
 Specification & BitString.h \\
 Implementation & - \\
 \verb|size(BitString<8>|) & 1 byte \\
 \verb|size(BitString<16>|) & 2 bytes \\
 \verb|size(BitString<32>)| & 4 bytes \\
 \verb|size(BitString<64>)| & 8 byte  \\
\end{classSummary}

\begin{methodMapping}
  \verb|NOT|       & \verb|bitNot()| &  \\
  \verb|TOFIXED|   & \verb|toFixed()| &  \\
  \verb|==|        & \verb|operator==(rhs)|   &  \\
  \verb|/=|        & \verb|operator!=(rhs)|   & \\
  \verb|AND|       & \verb|bitAnd(rhs)|   & \\
  \verb|OR|       & \verb|bitOr(rhs)|   & \\
  \verb|EXOR|     & \verb|bitXor(rhs)|    & \\
  \verb|><| (CAT) & \verb|bitCat(rhs)|    & \\
  \verb|<>| (CSHIFT) & \verb|bitCshift(rhs)| & \\
  \verb|SHIFT|       & \verb|bitShift(rhs)|  & \\
  \verb|.BIT(x)|     & \verb|getBit(...)|  &
				{\em on right hand side of the expression}\\
  \verb|.BIT(x)|    & \verb|setBit(...)|  &
				{\em on left hand side of the expression}\\
  \verb|.BIT(x,y)|  & \verb|getSlice(...)| &
				{\em on right hand side of the expression}\\
  \verb|.BIT(x,y)|   & \verb|setSlice(...)| &
				{\em on left hand side of the expression}\\
\end{methodMapping}

The slice operations \verb|.BIT(...)| may be used in PEARL
on the left hand side and the right hand side of the expression.

\paragraph{Remarks:}
\begin{itemize}
\item The preset value is passed to the ctor as integer type. The
   radix evaluation must be done by the compiler.
\item The passed integer is shifted internally by the number of not used
   bits in the used native data type to the left side. This results
   in the proper interal alignment.
\item If no preset value is specified, the variable is initialized with 0.
\item There is no default length in the runtime system.
\item the method names like {\em bitAnd} were choosen since {\em and}
is a reserved keyword in C for computer systems without \& character
\item C++ literals correspond usually to 32 bit integers.
If we need longer constants the postfix \verb|LL| must be added by
the compiler to enforce 64-bit integers.
\item The initialising constant is of type int and right adjusted!
\item {\em getSlice} and {\em setSlice} take the start index and a
    {\em BitString} defining the length and/or value as parameter.
\end{itemize}

Sample Code:
\begin{PEARLCode}
DCL x BIT(4) INIT ('1'B4);
DCL y BIT(4);
DCL z BIT(3);
DCL c BIT(8);
DCL f FIXED(3) INIT(2);
y := '01'B1;  // implicit conversion BIT(2) -> BIT(4)
y := x CSHIFT 1;
z := y AND x;
c := x CAT y;
y := TOBIT f;
f := TOFIXED y;
z.BIT(2,2)  := c.BIT(1,2);
\end{PEARLCode}

\begin{CppCode}
BitString<4> x4(1);
BitString<4> y;
Fixed<3> f(2);
y = (pearlrt::BitString<2>)(1);
y = x.bitCshift(1);
z = y.bitAnd(x);
c = x.bitCat(y);
y = BitString<4>(f.x);
f = y.toFixed();
z.setSlice(2,c.getSlice(1,BitString<2>dummy));
\end{CppCode}

\section{CHARACTER}
The data type CHARACTER in PEARL has a definite and fixed length.
The templated class {\em Character} provides the storage allocation
and necessary methods for this data type.
All operations check for range overflows and throw exceptions in case of
range violation.

The language report states at one location that the length is of type FIXED(15).
Therefore the supported lengths are 1 to 32787.

The preset of a character variable may be set as C-string constant.

\begin{classSummary}
 Class & \verb|Character<size_t len>| \\
 Specification & Character.h  \\
 Implementation  &  Character.cc \\
 size   &  1 byte per Character \\
 max length & 32767  \\
\end{classSummary}

\begin{methodMapping}
  \verb|UPB|      & \verb|upb()|    &  {\em returns size\_t}\\
  \verb|TOFIXED| & \verb|toFixed()| &     \\
  \verb|<|       & \verb|operator<()| &  \\
  \verb|>|       & \verb|operator>()| &  \\
  \verb|<=|      & \verb|operator<=()| & \\
  \verb|>=|      & \verb|operator>=()| & \\
  \verb|==|      & \verb|operator==()| & \\
  \verb|/=|      & \verb|operator!=()| & \\
  \verb|><| (CAT) & \verb|catChar()| & {\em see note below} \\
  \verb|.CHAR(x)| & \verb|setCharAt(..)|&
				on left hand side in expression
				{\em see note below}\\
  \verb|.CHAR(x)| & \verb|getCharAt(..)| &
				on right hand side in expression
				{\em see note below}\\
  \verb|.CHAR(x,y)| & \verb|setCharSlice()| &
				on left hand side in expression
				{\em see note below}\\
  \verb|.CHAR(x,y)| & \verb|getCharSlice()| &
				on right hand side in expression
				{\em see note below}\\
\end{methodMapping}
The slice operations \verb|.CHAR(...)| may be used in PEARL
on the left hand side and the right hand side of the expression.

Remarks:
\begin{itemize}
\item The slice operations are defined with a different interface than
   BitString. This was choosen to avoid character copies.
\item the setSliceChar and getSliceChar methods take as $4^{th}$ element
a Character value defining the retrieved/new data
 from/for the container Char-value
\end{itemize}

\begin{PEARLCode}
DCL x CHARACTER(5) INIT ('PEARL');
DCL y CHARACTER(4);
y := x.CHAR(2,5);
x.CHAR(2,5) := y;
\end{PEARLCode}
\begin{CppCode}
Character<5> x("PEARL");
Character<4> y;
getSliceChar(x, 2, 5, y);
setSliceChar(x, 2, 5, y);
\end{CppCode}

\section{REF CHARACTER}
The type REF CHARACTER allows character string with variable length.
A REF CHARACTER needs a normal CHARACTER variable as working storage.
It contains an additional value to maintain the current used length.

The class {\em RefCharacter} provides the necessary operations.

\begin{classSummary}
Class & RefCharacter  \\
 Specification & RefChar.h  \\
 Implementation & RefChar.cc \\
 size   &  12 byte \\
\end{classSummary}

\begin{methodMapping}
  \verb|:=|     & \verb|setWork()|    & REF CHAR := CHAR(x) \\
  \verb|CONT|   & \verb|\verb|store()|    & CHAR := CONT REF CHAR(x) \\
\end{methodMapping}

The content based operations like comparison are done upon the
CHARACTER-type.

\begin{PEARLCode}
DCL wrk CHARACTER(100);
DCL result CHARACTER(50);
DCL rc REF CHAR();
rc := wrk;
CONT rc := 'Hallo';
CONT rc := CONT rc CAT ' Welt.';
result := CONT rc;
\end{PEARLCode}

\begin{CppCode}
Character<100> wrk;
Character<50> result;
RefCharacter rc;
rc.setWork(wrk);
rc.clear();
Character<5> hello((char*)"Hallo");
rc.add(hello);
Character<6> welt((char*)" Welt.");
rc.add(welt);
rc.store(result);
\end{CppCode}

\section{DURATION}
The values of type DURATION are managed by the class {\em Duration}.
The values are stored in a 64 bit integer ({\em Fixed63}).
The base unit is $1 \mu s$.
The range of 64 bit allows durations up to $\approx 100 days$.

All calculations on durations are done within the full possible range.
Thus values larger than 24 hours may occur.
Even negativ values are possible.

The mapping of duration values to the C++ types is done via the type
\verb|double| giving the duration in seconds.

Operations are defined as required by the language to mix DURATION, CLOCK,
   FIXED and FLOAT.

\paragraph{Notes:}
\begin{itemize}
\item The division Duration / Duration delivers a \verb|Float<24>|,
   which may be assigned to larger Float type
\item The division Duration/Float is realized as template for
   all Float-lengths
\end{itemize}

\begin{classSummary}
 Class & Duration \\
 Specification & Duration.h \\
 Implementation & Duration.cc \\
 size   &  8 bytes \\
\end{classSummary}

\begin{methodMapping}
  \verb|+| (monadic)  &        & to be treated by the compiler \\
  \verb|-| (monadic)  & \verb|operator-()|  & \\
  \verb|ABS|          & \verb|abs()|  & \\
  \verb|SIGN|         & \verb|sign()|  & \\
  \verb|+| (dyadic)   & \verb|operator+(rhs)| & duration+duration \\
  \verb|-| (dyadic)   & \verb|operator-(rhs)| & duration-duration \\
  \verb|*| (dyadic)   & \verb|operator*(rhs)| & duration*\verb|Float<S>| \\
  \verb|/| (dyadic)   & \verb|operator/(rhs)| & duration/duration returns
        \verb|Float<24>| \\
  \verb|/| (dyadic)   & \verb|operator/(rhs)| & duration/\verb|Float<S>| \\
  \verb|<|            & \verb|operator<(rhs)|& \\
  \verb|>|            & \verb|operator>(rhs)| &  \\
  \verb|<=|           & \verb|operator<=(rhs)|& \\
  \verb|>=|           & \verb|operator>=(rhs)| & \\
  \verb|==|           & \verb|operator==(rhs)| & \\
  \verb|/=|           & \verb|operator!=(rhs)| & \\
\end{methodMapping}

\begin{PEARLCode}
DCL minute DURATION INIT(1 MIN);
DCL d1 DURATION INIT(1 HRS 3 MIN 3.1415 SEC);
DCL d2 DURATION INIT(0.001415 SEC);
d2 := d1 + minute;
d2 := d1 + minute / 60.0;
\end{PEARLCode}
\begin{CppCode}
Duration minute(60.0);
Duration d1(3783.1415);
Duration d2(0.001415);
d2 = d1 + minute;
d2 = d1 + minute / (Float<24>)(60);
\end{CppCode}

\section{CLOCK}
The values of type CLOCK are managed by the class {\em Clock}.
The values are stored in a 64 bit integer ({\em Fixed63}).
The base unit is $1 \mu s$.

All calculations on clock values are done within the
period of 1 day.

The mapping of duration values to the C++ types is done via the type
\verb|double| giving the duration in seconds.

Operations are defined as required by the language to mix DURATION and CLOCK.

The specification requires a procedure \verb|NOW| which delivers the current
time as \verb|CLOCK|-value. This routine is plattform specific.

\begin{classSummary}
 Class & Clock  \\
 Specification & Clock.h  \\
 Implementation &  Clock.cc \\
 size   &  8 byte \\
\end{classSummary}

\begin{methodMapping}
  \verb|+| (dyadic) & \verb|operator+(rhs)| & Clock + Duration \\
  \verb|-| (dyadic) & \verb|operator-(rhs)| & Clock-Duration and Clock-Clock \\
  \verb|<|          & \verb|operator<(rhs)| &
                    {\em returns bool } \\
  \verb|>|          & \verb|operator>(rhs)| &
                    {\em returns bool } \\
  \verb|<=|         & \verb|operator<=(rhs)| &
                    {\em returns bool } \\
  \verb|>=|         & \verb|operator>=(rhs)| &
                    {\em returns bool } \\
  \verb|==|         & \verb|operator==(rhs)| &
                    {\em returns bool } \\
  \verb|/=|         & \verb|operator!=(rhs)| &
                    {\em returns bool } \\
  \verb|NOW|        & \verb|Clock::now()| & this method is plattform specific\\
\end{methodMapping}

Remarks:
\begin{itemize}
\item the arithmetic operations are defined for also for
   mixed Duration and Clock parameters
\end{itemize}

\begin{PEARLCode}
DCL start CLOCK(11:30);
DCL current CLOCK;
DCL diff DURATION;
current := NOW;
diff := current - start;
\end{PEARLCode}

\begin{CppCode}
Clock start(41400.0);
Clock current;
Duration diff;
current = now();
diff = current - start;
\end{CppCode}
\fi


\acresetall
\chapter{Evaluation and Results}\label{ch:evaluation}
In this chapter, a benchmark of the analyzed algorithms is presented, based on dataset size and complexity. For the first part, the algorithms used for supervised learning are presented, for which an analysis of the different complexities used is made. From the benchmark, the trained models with the best performance are saved and then used in the live implementation presented in chapter~\ref{ch:live}. The process of saving trained models for later use is called \emph{model persistence}.

\section{Performance metrics}\label{ch:performance}

The impact of the dataset size on the algorithm performance is of special interest. Given that the data recorded using the procedure described in section~\ref{ch:measure} has a fixed size, this analysis was performed by slicing the resulting feature extraction files to emulate different dataset sizes. It is important to notice that only "slicing", in order to emulate smaller datasets, and not "repeating samples", for emulation of bigger datasets, has a significant impact on the performance, as noted in section~\ref{ch:size}. This is due to the fact that repeated samples present no new information to the model, and there is nothing new to learn from them.

The dataset is, then, sliced into four different sizes:

\begin{itemize}
    \item Small dataset (S): is one fourth (1/4) of the original dataset
    \item Medium dataset (M): is one third (1/3) of the original dataset
    \item Large dataset (L): is half (1/2) of the original dataset
    \item Extra Large dataset (XL): the whole set of features extracted as recorded from the signals over the air.
\end{itemize}

Furthermore, a benchmark of the learning algorithms is made based on variations on their complexities. Each algorithm has different ways to represent its own complexity, and this has a significant impact on its overall performance. The \ac{ML} algorithms used in this analysis are K-nearest neighbors, \ac{SVM} and binary tree classification. Their complexities are set as follows:

\begin{itemize}
    \item K-nearest neighbors: the complexity of this algorithm is set by the amount of neighbors that are considered in order to make a classification decision. For the benchmark, different models were trained using from 1 neighbor up to 200 neighbors
    \item \ac{SVM}: this algorithm allows the designer to select the type of kernel that is used during the learning process. The function of the kernel is to generate internally non-linear transformations over the input data while still behaving as a linear classificator. For this work, the \ac{RBF} is used as a kernel because of its known good performance on multiclass classification problems at the cost of longer training times. The complexity of this model is then set by providing different values to the '\(\gamma\)' parameter of this \ac{RBF} kernel, whose kernel function is:
        \begin{equation}
            \text{exp}(-\gamma || x - x'||^2
        \end{equation}
        For this work, values of \(\gamma=2^{-15}, 2^{-13}, 2^{-11}, 2^{-9}, 2^{-7}, 2^{-5}, 2^{-3}, 2^{-1}, 2^{1} \text{ and }2^{3} \) are used.

    \item Decision Trees: in this model, the complexity is ruled by the depth of the decision branch. Here, depths from 1 up to 500 are used.
\end{itemize}

Moreover, it is also interesting to determine how the models behave when the features have been scaled. A first glance at the impact of a scaler is presented by using the StandardScaler() from SKlearn, which simply removes the mean of the features and scales them to unit variance. A side-by-side comparison of model performance when trained with unscaled features vs.scaled features is shown as a result of the benchmark. The results are depicted in Figs~\ref{fig:knn}, \ref{fig:dtc} and \ref{fig:svc}.

\begin{figure}[!htb]
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={\# of Neighbors},
                ylabel={Accuracy},
                legend style={at={(1.2,1.20)},
                    anchor=north,legend columns=-1},
                  ]
                \addplot[color=blue] table [x=N, y=S, col sep=comma, mark=none] {data/knn_accs.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/knn_accs.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/knn_accs.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/knn_accs.csv};
                \legend{Dataset S, Dataset M, Dataset L, Dataset XL}
        \end{axis}
        \end{tikzpicture}
        \caption{Accuracy with unscaled features}
        \label{fig:knn_accs_scaled}
    \end{subfigure}
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={\# of Neighbors},
                ylabel={Accuracy},
                  ]
                \addplot[color=blue] table [x=N, y=S, col sep=comma, mark=none] {data/knn_accs_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/knn_accs_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/knn_accs_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/knn_accs_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Accuracy with scaled features}
        \label{fig:knn_accs_scaled}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={\# of Neighbors},
                ylabel={Training time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/knn_fit_times.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/knn_fit_times.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/knn_fit_times.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/knn_fit_times.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Training times with unscaled features}
        \label{fig:knn_fit_times}
    \end{subfigure}
    \centering
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={\# of Neighbors},
                ylabel={Training time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/knn_fit_times_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/knn_fit_times_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/knn_fit_times_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/knn_fit_times_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Training times with scaled features}
        \label{fig:knn_fit_times_scaled}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={\# of Neighbors},
                ylabel={Prediction time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/knn_pred_times.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/knn_pred_times.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/knn_pred_times.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/knn_pred_times.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Prediction times with unscaled features}
        \label{fig:knn_pred_times}
    \end{subfigure}
    \centering
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={\# of Neighbors},
                ylabel={Prediction time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/knn_pred_times_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/knn_pred_times_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/knn_pred_times_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/knn_pred_times_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Prediction times with scaled features}
        \label{fig:knn_pred_times}
    \end{subfigure}
    \caption{Metrics for K-Nearest Neighbors}
    \label{fig:knn}
\end{figure}

\begin{figure}[!htb]
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Tree depth},
                ylabel={Accuracy},
                legend style={at={(1.2,1.20)},
                    anchor=north,legend columns=-1},
                  ]
                \addplot[color=blue] table [x=N, y=S, col sep=comma, mark=none] {data/dtc_accs.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/dtc_accs.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/dtc_accs.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/dtc_accs.csv};
                \legend{Dataset S, Dataset M, Dataset L, Dataset XL}
        \end{axis}
        \end{tikzpicture}
        \caption{Accuracy with unscaled features}
        \label{fig:dtc_accs_scaled}
    \end{subfigure}
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Tree depth},
                ylabel={Accuracy},
                  ]
                \addplot[color=blue] table [x=N, y=S, col sep=comma, mark=none] {data/dtc_accs_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/dtc_accs_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/dtc_accs_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/dtc_accs_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Accuracy with scaled features}
        \label{fig:dtc_accs_scaled}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Tree depth},
                ylabel={Training time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/dtc_fit_times.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/dtc_fit_times.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/dtc_fit_times.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/dtc_fit_times.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Training times with unscaled features}
        \label{fig:dtc_fit_times}
    \end{subfigure}
    \centering
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Tree depth},
                ylabel={Training time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/dtc_fit_times_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/dtc_fit_times_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/dtc_fit_times_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/dtc_fit_times_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Training times with scaled features}
        \label{fig:dtc_fit_times_scaled}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Tree depth},
                ylabel={Prediction time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/dtc_pred_times.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/dtc_pred_times.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/dtc_pred_times.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/dtc_pred_times.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Prediction times with unscaled features}
        \label{fig:dtc_pred_times}
    \end{subfigure}
    \centering
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Tree depth},
                ylabel={Prediction time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/dtc_pred_times_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/dtc_pred_times_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/dtc_pred_times_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/dtc_pred_times_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Prediction times with scaled features}
        \label{fig:dtc_pred_times}
    \end{subfigure}
    \caption{Metrics for the decision tree classifier}
    \label{fig:dtc}
\end{figure}

\begin{figure}[!htb]
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                    xlabel={\(\gamma\) for \ac{RBF} kernel - \ac{SVM}},
                ylabel={Accuracy},
                legend style={at={(1.2,1.20)},
                    anchor=north,legend columns=-1},
                  ]
                \addplot[color=blue] table [x=N, y=S, col sep=comma, mark=none] {data/svc_accs.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/svc_accs.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/svc_accs.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/svc_accs.csv};
                \legend{Dataset S, Dataset M, Dataset L, Dataset XL}
        \end{axis}
        \end{tikzpicture}
        \caption{Accuracy with unscaled features}
        \label{fig:svc_accs_scaled}
    \end{subfigure}
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                    xlabel={\(\gamma\) for \ac{RBF} kernel - \ac{SVM}},
                ylabel={Accuracy},
                  ]
                \addplot[color=blue] table [x=N, y=S, col sep=comma, mark=none] {data/svc_accs_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/svc_accs_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/svc_accs_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/svc_accs_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Accuracy with scaled features}
        \label{fig:svc_accs_scaled}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                    xlabel={\(\gamma\) for \ac{RBF} kernel - \ac{SVM}},
                ylabel={Training time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/svc_fit_times.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/svc_fit_times.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/svc_fit_times.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/svc_fit_times.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Training times with unscaled features}
        \label{fig:svc_fit_times}
    \end{subfigure}
    \centering
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                    xlabel={\(\gamma\) for \ac{RBF} kernel - \ac{SVM}},
                ylabel={Training time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/svc_fit_times_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/svc_fit_times_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/svc_fit_times_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/svc_fit_times_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Training times with scaled features}
        \label{fig:svc_fit_times_scaled}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                    xlabel={\(\gamma\) for \ac{RBF} kernel - \ac{SVM}},
                ylabel={Prediction time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/svc_pred_times.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/svc_pred_times.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/svc_pred_times.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/svc_pred_times.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Prediction times with unscaled features}
        \label{fig:svc_pred_times}
    \end{subfigure}
    \centering
        \hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                    xlabel={\(\gamma\) for \ac{RBF} kernel - \ac{SVM}},
                ylabel={Prediction time (s)},
                  ]
                \addplot[color=blue]table [x=N, y=S, col sep=comma, mark=none] {data/svc_pred_times_scaled.csv};
                \addplot[color=red] table [x=N, y=M, col sep=comma, mark=none] {data/svc_pred_times_scaled.csv};
                \addplot[color=green] table [x=N, y=L, col sep=comma, mark=none] {data/svc_pred_times_scaled.csv};
                \addplot[color=orange] table [x=N, y=XL, col sep=comma, mark=none] {data/svc_pred_times_scaled.csv};
        \end{axis}
        \end{tikzpicture}
        \caption{Prediction times with scaled features}
        \label{fig:svc_pred_times}
    \end{subfigure}
    \caption{Metrics for \ac{SVM}}
    \label{fig:svc}
\end{figure}

%\begin{figure}[!htb]
    %\centering
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/knn_accs}
        %\label{fig:knn_accs}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/knn_accs_scaled}
        %\caption{Accuracy with scaled features}
        %\label{fig:knn_accs_scaled}
    %\end{subfigure}\\
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/knn_fit_times}
        %\caption{Training times with unscaled features}
        %\label{fig:knn_fit_times}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/knn_fit_times_scaled}
        %\caption{Training times with scaled features}
        %\label{fig:knn_fit_times_scaled}
    %\end{subfigure}\\
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/knn_pred_times}
        %\caption{Prediction times with unscaled features}
        %\label{fig:knn_pred_times}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/knn_pred_times_scaled}
        %\caption{Prediction times with scaled features}
        %\label{fig:knn_pred_times_scaled}
    %\end{subfigure}
    %\caption{K-nearest Neighbors}
    %\label{fig:knn}
%\end{figure}

%\begin{figure}[!htb]
    %\centering
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/dtc_accs}
        %\caption{Accuracy with unscaled features}
        %\label{fig:dtc_accs}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/dtc_accs_scaled}
        %\caption{Accuracy with scaled features}
        %\label{fig:dtc_accs_scaled}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/dtc_fit_times}
        %\caption{Training times with unscaled features}
        %\label{fig:dtc_fit_times}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/dtc_fit_times_scaled}
        %\caption{Training times with scaled features}
        %\label{fig:dtc_fit_times_scaled}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/dtc_pred_times}
        %\caption{Prediction times with unscaled features}
        %\label{fig:dtc_pred_times}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/dtc_pred_times_scaled}
        %\caption{Prediction times with scaled features}
        %\label{fig:dtc_pred_times_scaled}
    %\end{subfigure}
    %\caption{Decision Trees}
    %\label{fig:dtc}
%\end{figure}

%\begin{figure}
    %\centering
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/svc_accs}
        %\caption{Accuracy with unscaled features}
        %\label{fig:svc_accs}
    %\end{subfigure}
    %\begin{subfigure}[htb]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/svc_accs_scaled}
        %\caption{Accuracy with scaled features}
        %\label{fig:svc_accs_scaled}
    %\end{subfigure}
    %\begin{subfigure}[b]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/svc_fit_times}
        %\caption{Training times with unscaled features}
        %\label{fig:svc_fit_times}
    %\end{subfigure}
    %\begin{subfigure}[b]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/svc_fit_times_scaled}
        %\caption{Training times with scaled features}
        %\label{fig:svc_fit_times_scaled}
    %\end{subfigure}
    %\begin{subfigure}[b]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/svc_pred_times}
        %\caption{Prediction times with unscaled features}
        %\label{fig:svc_pred_times}
    %\end{subfigure}
    %\begin{subfigure}[b]{0.49\textwidth}
        %\centering
        %\includestandalone[width=\linewidth]{figures/svc_pred_times_scaled}
        %\caption{Prediction times with scaled features}
        %\label{fig:dtc_pred_times_scaled}
    %\end{subfigure}
    %\caption{Support Vector Machines}
    %\label{fig:svc}
%\end{figure}

From Figs~\ref{fig:knn}, \ref{fig:dtc} and \ref{fig:svc} a couple of conclusions can be extracted. First, it is seen that the accuracy of the model increases as more data is taken into consideration, which is well according to the theory and serves as a catalyst for applying the same implementation with more data as proposed in chapter~\ref{ch:conclusions}. Furthermore, three different behaviors are appreciated:

\begin{itemize}
    \item As complexity increases, the K-nearest neighbors algorithm shows a decrease on its overall accuracy, which is an indication that the model is over-generalizing, and suggests that with such a complexity (above few neighbors), the model is starting to loosely classify scenarios, allowing more samples to be misclassified, and is set to perform badly if unknown samples are feed to it, as its capabilities for generalization are in decay. Moreover, the prediction times for the highest level of complexity are already about two times greater than the second highest, making high K-nearest neighbors complexity not suitable for live implementations. Furthermore, this model shows both a reduction in accuracy and an increase on prediction time when the features have been scaled, which simply states that the model performs well without further data preprocessing right after the feature extraction.
    \item The decision tree model shows a steady increase on its accuracy as complexity increases, without reaching a clear overfitting state. Complexity has a clear effect on the training times, but it is still in the range of the milliseconds - being a process done only once, this does not affect dramatically the model selection and does not close the possibility of increasing the complexity for this specific use case. This can also be backed by the fact that the prediction times seem to stay constant with this increase. Interestingly, not only the training times are reduced after scaling, but also the prediction times are drastically improved, reaching about 10x faster predictions using the standard scaler. With comparable accuracy, scaling the features seems like a prospect procedure when this model is used in time demanding or live implementations.
        The low accuracy of this model when a depth=2 does not come as a surprise. As the classification is done based on \emph{if-then} steps, having such a short depth can only provide four possible predicted values, as it can be seen in the extracted tree shown in Fig~\ref{fig:dt_2}. This model is generated as an experimental exercise to show this behavior, and to make the statement that the depth of the tree has to be able to generate at least as many answers as classes of the problem, which in this specific use case is achieved with a depth=\(2^4 > 10\). Graphical representations of complex decision trees are easily generated, but they become overwhelming for them to be shown in this print. The procedure to generate these trees is explained in the development notebook of this thesis \cite{repo:cognitive_radio_ml}.

\begin{figure}[!htb]
    \centering
      \includegraphics[width=\textwidth]{figures/dt_2}
      \caption{Extracted representation of a trained decision tree with depth=2}
      \label{fig:dt_2}
\end{figure}

    \item the \ac{SVM} show a clear tendency to improve as complexity is increased only if the data has been scaled, which indicates the sensitivity of this model to unscaled data.  However, the best accuracy of this model is still below the accuracies of the other models regarded in this benchmark. Additionally, the training and prediction times exceed dramatically the times achieved with the other models, being about 8000 times longer for training and around 500 times longer for prediction. The fact that this model is that slow will definitely affect considerably the performance of real-time implementations, an aspect that is crucial in systems such as cognitive radio, and therefore would not be recommended for that type of implementations.
\end{itemize}

\section{Scenario Classification}
Besides of the general performance that the learning models have in regard to the whole testing set, it is paramount to determine how good they perform by classifying each of the specific scenarios of Table~\ref{table:scenarios}. To assess this, confusion matrices are used to determine the number of correctly classified scenarios, along with the false positives and false negatives. On this analysis, it is only of interest the number of correctly classified scenarios, and any misclassification affects the implementation equally, regardless of its type. Moreover, in section~\ref{ch:performance} can be seen that each algorithm behaves differently in terms not only of accuracy, but also in training time and prediction time. As the models are trained only once, the "training time" does not play a role in the model selection for this work, as it does not have any repercussion on the performance of the model when new values are applied to it. Therefore, "prediction time" and "accuracy" are metrics of quality that are considered for these models on its selection to be applied on real-time scenarios.\\

Confusion matrices show how each of the samples from the data set are classified for a given model. As a matter of illustration, a side by side comparison for the worst-performing vs. the best-performing model, with respect to accuracy, is given in Fig~\ref{fig:confusionknn}, \ref{fig:confusiondtc} and \ref{fig:confusionsvc}. The complete set of confusion matrices, corresponding to every trained model, can be found in the development notebook for this work \cite{repo:cognitive_radio_ml}.

\begin{figure}[!htb]
    \captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/knn_scaled_S_50}
        \caption{KNN-50 neighbors, StandardScaled, trained with small (S) dataset}
        \label{fig:knn_2}
    \end{subfigure}
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/knn_unscaled_XL_4}
        \caption{KNN-4 neighbors, unscaled, trained with whole (XL) dataset}
        \label{fig:knn_4}
    \end{subfigure}
    \caption{Confusion Matrixes for K-nearest Neighbor Models}
    \label{fig:confusionknn}
\end{figure}

\begin{figure}[!htb]
    \captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/dtc_unscaled_L_2}
        \caption{Decision Tree, unscaled, depth=2, trained with large (L) dataset}
        \label{fig:knn_2}
    \end{subfigure}
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/dtc_unscaled_XL_50}
        \caption{Decision Tree, unscaled, depth=50, trained with whole (XL) dataset}
        \label{fig:knn_4}
    \end{subfigure}
    \caption{Confusion Matrixes for Decision Tree Models}
    \label{fig:confusiondtc}
\end{figure}
\begin{figure}[!htb]
    \captionsetup[subfigure]{justification=centering}
    \centering
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/svc_scaled_S_1}
        \caption{\ac{SVM}, StandardScaled, \(\gamma=2^{-9}\), trained with small (S) dataset}
        \label{fig:knn_2}
    \end{subfigure}
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{figures/svc_scaled_S_1e6}
        \caption{\ac{SVM}, StandardScaled, \(\gamma=2^3\), trained with small (S) dataset}
        \label{fig:knn_4}
    \end{subfigure}
    \caption{Confusion Matrixes for \ac{SVM} Models}
    \label{fig:confusionsvc}
\end{figure}
It is also important to notice the clear difference on the number of samples present for each scenario. This is due to the fact that the measurements performed in section~\ref{ch:measure} were based on runtime and not on number of generated samples, and the feature extraction algorithm described in section~\ref{ch:features} generates a different number of samples for each scenario, because of its dependence on the \emph{frame events} generation consequently of the channel occupation and interframe time of arrival itself. A different approach for feature extraction based on a number of features generated is proposed in chapter~\ref{ch:conclusions}.

From these matrices it can be clearly seen, at a first glance, that the K-Nearest algorithm and the decision trees perform quite well regardless of its configuration, having a more populated diagonal in its confusion matrix in comparison with the \ac{SVM} models. Additionally, just a small improvement in the classification accuracy is seen as the complexity and dataset size increases for this model. In general, it is safe to assert that the classification performs generally well for most of the scenarios except scenario 5, where a higher number of false positives, as well as false negatives, is seen for all models, being misclassified as scenario 9. This indicates that the extracted features for these specific scenarios have a noticeable correlation, which is not utterly surprising given that these scenarios share the channel occupation and interframe delay, being different only in the distribution with which the interframe delay is set: deterministic for scenario 5 and stochastic, with Poisson distribution, for scenario 9. This serves as an invitation to determine a feature that confidently sets a difference between them.
\subsection{Image Classification}\label{ch:image_classification}
For the image classification part of this work, a sequential model based on the work of \cite{Paisana2017} was implemented. The basic structure of this model is shown in Fig~\ref{fig:cnn}. As it can be seen, the model is composed of the connection of different layers, each of them with an specific functionality:

\begin{figure}[!htb]
    \centering
      \includegraphics[width=\textwidth]{figures/cnn}
      \caption{Schema of the implemented CNN}
      \label{fig:cnn}
\end{figure}

\begin{itemize}
    \item Convolution2D: this comprises a convolutional kernel. It convolves the input to produce tensors at its output. This is the backbone of the \ac{ConvNet}, as it is the responsible for learning. Here, 2-Dimensional convolutional blocks of kernel size 2x2 are applied, as it is convolving over the area of the input images.
    \item MaxPool: this layer serves its purpose for dimension reduction, which is a form of downsampling. Briefly, it divides the input tensor into 2x2 non-overlapping squares and keeps the largest value present in each of these cells, effectively reducing the size of the representation, which in consequence reduces the number of parameters transitting the network. This has the effect of reducing the number of vector operations as well, making each layer of the network less processing-demandant. Additionally, as parameters are being dropped, this helps to avoid overfitting by disregarding eventual characteristics that are being "memorized". The principle of operation of such layers is shown in Fig~\ref{fig:maxpool}.
        \begin{figure}[!htb]
            \centering
              \includegraphics[width=0.4\textwidth]{figures/maxpool}
              \caption{2-D MaxPool}
              \label{fig:maxpool}
        \end{figure}
    \item \ac{ReLu}: this is an activation layer, i.e. a layer that applies a function that defines the type of output depending on its input. In the \ac{ReLu} case, the applied function is \(f(x) =\max(0,x) \), where x is the input of the neuron. Being a non-linear function, this activation increases the nonlinearity of the network without affecting the visual field, as there is no reduction of dimension.
        \begin{figure}[!htb]
            \centering
                \begin{tikzpicture}[scale=0.9]
                    \begin{axis}[
                        xlabel={$x$},
                        ylabel={$f(x)$},
                    ]
                        \addplot[domain=0.001:3,samples=201,purple,thick,smooth]{x};
                        \addplot[domain=-3:0.001,samples=201,purple,thick,smooth]{0};
                    \end{axis}
                \end{tikzpicture}
              \caption{ReLu activation function}
              \label{fig:maxpool}
        \end{figure}
    \item Fully connected layer: as the name states, this layer is connected to each one of the activations of the previous layers. These type of layers are the principle of operation of traditional neural networks, and is in charge of learning the non-linearities that have been propagated throughout the network.
    \item Softmax: this is also an activation layer. It takes vector q of dimension K and compress it together with values that add up to one. The function is given by
        \begin{equation}
            P(q)_i = \frac{e^{q_i}}{\sum_{k=1}^{K} \exp{q_j}}
        \end{equation}

        It can be easily understood that the dimension of the output from this softmax function is the number of labels to classify and that the value of each is the probability of the input sample to belong to that class. Consequently, the classification result is the \( \argmax \) of the function output.
\end{itemize}

The chosen layers and its disposition are strictly related to the input. In this case, the input picture has a 64x64 size, which describes then the height and width of the first layer. The depth is the number of filters that are used for each layer, and it is, to some extent, represented in the extension of the scale of Fig~\ref{fig:cnn}. 5 convolutional layers, with kernel dimensions 2x2, were used in this sequential model, with depths of 48, 128, 192, 192 and 128 filters correspondingly. At the end of the sequential model, 3 fully connected layers with depth 1024, 1024 and 10 were used. The succession of dimension changes of the sequential model and the number of parameters considered in each of the stages of the classification problem is summarized in Table~\ref{table:convnet_summary}. The number of parameters is the number of trainable weights that each layer contains. For a convolutional layer, this is calculated as \( \text{\# parameters} = \text{(depth of input tensor)}\cdot\text{(depth output tensor)}\cdot\text{height}_{kernel}\cdot \text{width}_{kernel} + \text{(depth output tensor)} \). As an example, the calculation for the first layer gives:
\begin{equation*}
    1 \cdot 48 \cdot 2 \cdot 2 + 48 = 240
\end{equation*}

\begin{table}[h!]
    \centering
    \begin{tabular}{|c|c|c|}
        \hline
        \textbf{Layer} & \textbf{Output Shape} & \textbf{\# Parameters} \\
        \hline\hline
        Convolution \# 1 & (63, 63, 48) & 240 \\\hline
        ReLu \# 1 & (63, 63, 48) & 0 \\\hline
        MaxPooling \# 1 & (31, 31, 48) & 0 \\\hline
        Convolution \# 2 & (30, 30, 128) & 24704 \\\hline
        ReLu \# 2 & (30, 30, 128) & 0 \\\hline
        MaxPooling \# 2 & (15, 15, 128) & 0 \\\hline
        Convolution \# 3 & (14, 14, 192) & 98496 \\\hline
        ReLu \# 3 & (14, 14, 192) & 0 \\\hline
        MaxPooling \# 3 & (7, 7, 192) & 0 \\\hline
        Convolution \# 4 & (6, 6, 192) & 147648 \\\hline
        ReLu \# 4 & (6,6, 192) & 0 \\\hline
        MaxPooling \# 4 & (3, 3, 192) & 0 \\\hline
        Convolution \# 5 & (2, 2, 128) & 98432 \\\hline
        ReLu \# 5 & (2,2, 128) & 0 \\\hline
        MaxPooling \# 5 & (1, 1, 128) & 0 \\\hline
        flatten & 128 & 0 \\\hline
        Fully Connected \#1 & 1024 & 132096 \\\hline
        Fully Connected \#2 & 1024 & 1049600 \\\hline
        Fully Connected \#3 & 10 & 10250 \\\hline
        Softmax & 10 & 0 \\\hline
        \hline
    \end{tabular}
    \caption{ConvNet dimensions and Number of Parameters}
    \label{table:convnet_summary}
\end{table}

The sequential model performs an optimization over its error function, for which the optimizer used can be selected. Keras provides a variety of these optimizers, using the approach of the steepest descent optimization over an objective function, \(J(\theta)\) with \(\theta \in \mathbb{R}^{d}\), being \(d\) the dimension of the objective function, by updating the parameters in the opposite direction of the gradient of \(J\), i.e.

\begin{equation}
    \nabla_\theta J(\theta) \text{  w.r.t } \theta, x, y
\end{equation}

A tunable parameter is the learning rate \(\eta\), that determines the size of the steps taken to reach the minimum. For this work, three different optimizers were used and analyzed based on the resulting accuracy of the classification over the test data, as well as its value for error during validation.

\begin{itemize}
    \item Stochastic gradient descent: this algorithm performs a parameter update after each training sample
        \begin{equation}
            \theta = \theta - \eta \cdot \nabla_\theta J(\theta;x^i;y^i)
        \end{equation}
        This optimizer was used in \cite{Paisana2017} and was used as a baseline. The learning rate \(\eta\) was decreased every one-fourth of the total iterations, which in this specific use case was set to 3000. Decreasing the learning rate increases the chance that this optimizer reaches a local minimum.
    \item Adadelta \cite{Zeiler2012a}: this optimizer is based on the Adagrad optimizer \cite{Duchi2011}, which adapts intrinsically its learning rate to the parameters, therefore doing bigger updates for infrequent parameters, and smaller updates for frequent ones. It is proven that this optimizer is more robust than the stochastic gradient descent \cite{Duchi2011} by storing all previous squared gradients and updating accordingly. Adadelta extends this robustness by restricting the amount of accumulated squared gradients to a fixed window, and using this accumulation for its updates.
    \item Adamax: based on the Adaptive Moment Estimation - Adam \cite{Kingma2014}, this optimizer also sets an adaptive learning rate based on the parameters of the objective function. Additionally, to set a fixed window of past squared decays as Adadelta, Adam also keeps an exponentially decaying average of past gradients, which is similar to a momentum \cite{Dahl2013}. Adamax extends Adam functionality by maximizing the norm of the update vector, which leads to a stable behavior.
\end{itemize}

The validation of the models was performed based on the accuracy that these optimizers have during training and testing, the loss factor during the same stages, which is a measure of the error provided by the optimization process. The results are shown in Figs~\ref{fig:sgd}, \ref{fig:adamax} and \ref{fig:adadelta}.

\begin{figure}[!htb]
    \centering
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Iterations},
                ylabel={Loss Factor},
                  ]
                \addplot table [x=epoch, y=loss, col sep=comma, mark=none] {data/sgd.csv};
                \addplot table [x=epoch, y=val_loss, col sep=comma, mark=none] {data/sgd.csv};
                \legend{Training loss, Validation loss}
        \end{axis}
        \end{tikzpicture}
    \end{subfigure}
        \hfill
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[legend pos=south east,
                xlabel={Iterations},
                ylabel={Accuracy},
                  ]
                \addplot table [x=epoch, y=acc, col sep=comma, mark=none] {data/sgd.csv};
                \addplot table [x=epoch, y=val_acc, col sep=comma, mark=none] {data/sgd.csv};
                \legend{Accuracy, Validation Accuracy}
        \end{axis}
        \end{tikzpicture}
    \end{subfigure}
    \caption{Metrics for Steepest Gradient descent Optimizer}
    \label{fig:sgd}
\end{figure}

\begin{figure}[!htb]
    \centering
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[
                xlabel={Iterations},
                ylabel={Loss Factor},
                  ]
                \addplot table [x=epoch, y=loss, col sep=comma, mark=none] {data/adamax.csv};
                \addplot table [x=epoch, y=val_loss, col sep=comma, mark=none] {data/adamax.csv};
                \legend{Training loss, Validation loss}
        \end{axis}
        \end{tikzpicture}
    \end{subfigure}
        \hfill
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.8]
            \begin{axis}[legend pos=south east,
                xlabel={Iterations},
                ylabel={Accuracy},
                  ]
                \addplot table [x=epoch, y=acc, col sep=comma, mark=none] {data/adamax.csv};
                \addplot table [x=epoch, y=val_acc, col sep=comma, mark=none] {data/adamax.csv};
                \legend{Accuracy, Validation Accuracy}
        \end{axis}
        \end{tikzpicture}
    \end{subfigure}
    \caption{Metrics for Adamax Optimizer}
    \label{fig:adamax}
\end{figure}

\begin{figure}[!htb]
    \centering
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.9]
            \begin{axis}[
                xlabel={Iterations},
                ylabel={Loss Factor},
                  ]
                \addplot table [x=epoch, y=loss, col sep=comma, mark=none] {data/adadelta.csv};
                \addplot table [x=epoch, y=val_loss, col sep=comma, mark=none] {data/adadelta.csv};
                \legend{Training loss, Validation loss}
        \end{axis}
        \end{tikzpicture}
    \end{subfigure}
    \hfill
    \begin{subfigure}[htb]{0.49\textwidth}
        \centering
        \begin{tikzpicture}[scale=0.9]
            \begin{axis}[legend pos=south east,
                xlabel={Iterations},
                ylabel={Accuracy},
                  ]
                \addplot table [x=epoch, y=acc, col sep=comma, mark=none] {data/adadelta.csv};
                \addplot table [x=epoch, y=val_acc, col sep=comma, mark=none] {data/adadelta.csv};
                \legend{Accuracy, Validation Accuracy}
        \end{axis}
        \end{tikzpicture}
    %}
    \end{subfigure}
    \caption{Metrics for Adadelta Optimizer}
    \label{fig:adadelta}
\end{figure}

From these plots, it can clearly be seen that SGD takes a greater number of iterations to reach its maximum accuracy as well as its minimum loss factor in comparison with Adamax and Adadelta. With respect to their comparative accuracies, it is not conclusive that one of the optimizers provide a better result. The maximum and final values of the accuracies of these models are shown in Table~\ref{table:convnet_accs}, whilst the minimum and final values for the loss factor is shown in Table~\ref{table:convnet_loss}.

\begin{table}[!ht]
  \centering
    \begin{tabular}{| >{\centering\arraybackslash}m{5em}| >{\centering\arraybackslash}m{7em}| >{\centering\arraybackslash}m{7em}| >{\centering\arraybackslash}m{7em}| >{\centering\arraybackslash}m{7em}|}
    \cline{2-5}
      \multicolumn{1}{c|}{} & Max Training Accuracy (Iter) & Final Training Accuracy & Max Validation Accuracy (iter) & Final Validation accuracy \\ \hline
      SGD & 99\% (2411)   & 98\%   & 90\% (981) & 81\%  \\ \hline
      Adamax & 92\% (2997)   & 88\%   & 87\% (2679) & 77\%  \\ \hline
      Adadelta & 100\%(2012)   & 99.6\%   & 89\% (489) & 85\%  \\ \hline
  \end{tabular}
    \caption{Maximum and Final Accuracies for ConvNets with various optimizers}
    \label{table:convnet_accs}
\end{table}

\begin{table}[!ht]
  \centering
    \begin{tabular}{| >{\centering\arraybackslash}m{5em}| >{\centering\arraybackslash}m{7em}| >{\centering\arraybackslash}m{7em}| >{\centering\arraybackslash}m{7em}| >{\centering\arraybackslash}m{7em}|}
    \cline{2-5}
      \multicolumn{1}{c|}{} & Min Training Loss (Iter) & Final Training Loss & Min Validation Loss (iter) & Final Validation Loss \\ \hline
  SGD & 0.625 (2866)   & 0.64   & 0.863 (861) & 1.134  \\ \hline
      Adamax & 0.24 (2997)   & 0.37   & 0.47 (1150) & 0.93  \\ \hline
      Adadelta & 0.03 (2998)   & 0.04   & 0.4 (297) & 0.74  \\ \hline
  \end{tabular}
    \caption{Maximum and Final Loss Factor for ConvNets with various optimizers}
    \label{table:convnet_loss}
\end{table}

The low values for losses as well as high values for accuracies in the training process do not come as a surprise, but it is the iteration where they are reached what is significant to be analyzed, as they state the rate of convergence of each of the optimizers. For the training phase, it is also expected that these values are constantly increasing along the whole process, as the model keeps learning from the data and fits to it. Therefore, it is of special interest to focus the attention to the validation part. Regarding the accuracy value, it is not conclusive to determine an algorithm that outperforms the others. Additionally, an accuracy greater than 87\% for all the optimizers is satisfactory, keeping in mind that these models are trained with spectrograms with SNR as low as -5dB, where the \ac{PU} signal is well below the noise level. It is clear that the rate of convergence of Adadelta surpasses the performance of the other optimizers, for both the maximum validation accuracy and the minimum validation loss, which states that using early stop techniques allows having a satisfactory model with short training time. More insights about the early stop techniques are given in the following chapter, where it will be clear why the maximum validation accuracy and minimum validation loss are important values to be considered.

Looking carefully at the loss factor plots for validation, a clear tendency of the increase of this parameter at the end of the iterative process can be seen for the Adamax and Adadelta optimizers, which is an indication of overfitting. Although not clear from the plots, SGD also shows overfitting behavior to some extent, as it can be seen in the noticeable difference on the maximum accuracy vs the final accuracy in table~\ref{table:convnet_accs}.