         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
         % Information sheet for the Matlab lab - Maths 6111 %
         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[10pt]{article}
\input ma_no_html_header

\usepackage{color}
\usepackage{hyperref}
\hypersetup{breaklinks=true,colorlinks=true}
%\input ma_header
\setlength{\parindent}{0pt}
\pagestyle{myheadings}
% \markright{
% \protect {\protect \epsfxsize=0.2 true cm \protect \epsffile {dolph.line.eps}}
% \it Maths 3018/6111 - Numerical methods \hfill}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\thispagestyle{empty}
\begin{center}
\textbf{\Large Maths 3018/6111 - Numerical Methods \\*[8mm]
Worksheet 6 - Solutions}\\*[.8cm]
\end{center}

\section*{Theory}

\begin{enumerate}
\item Explain the shooting method for BVPs.
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  The boundary value problem for $y(x)$ with boundary data at $x=a, b$
  is converted to an \emph{initial} value problem for $y(x)$ by, at
  first, guessing the additional (initial) boundary data $z$ at $x=a$
  that isrequired for a properly posed (i.e., completely specified)
  IVP. The IVP can then be solved using any appropriate solver to get
  some solution $y(x;z)$ that depends on the guessed initial data
  $z$. By comparing against the required boundary data at $y(b)$ we
  can check if we have the correct solution of the original BVP. To be
  precise, we can write
  \begin{equation*}
    f(z) = y(x;z)|_{x=b} - y(b),
  \end{equation*}
  a nonlinear equation for $z$. At the root where $f(z) = 0$ we have
  the appropriate initial data $z$ such that the solution of the IVP
  is also a solution of the original BVP. The root of this nonlinear
  equation can be found using any standard method such as bisection or
  the secant method.
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\item Give a \emph{complete} algorithm for solving the BVP
  \begin{equation*}
    y'' - 3 y' + 2 y = 0, \quad y(0) = 0, \quad y(1) = 1
  \end{equation*}
  using the finite difference method. Include the description of the
  grid, the grid spacing, the treatment of the boundary conditions,
  the finite difference operators and a description of the linear
  system to be solved. You do not need to say which method would be
  used to solve the linear system, but should mention any special
  properties of the system that might make it easier to solve.
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  We first choose the grid. We will use $N+2$ point to cover the
  domain $x \in [0, 1]$; this implies that we have a grid spacing $h =
  1 / (N+1)$ and we can explicitly write the coordinates of the grid
  points as
  \begin{equation*}
    x_i = h i, \qquad i = 0, 1, \dots, N+1.
  \end{equation*}
  We denote the value of the (approximate finite difference) solution
  at the grid points as $y_i (\approx y(x_i))$. We will impose the
  boundary conditions using
  \begin{align*}
    y_0 & = y(0) & y_{N+1} & = y(1) \\
    & = 0 & & = 1.
  \end{align*}
  We will use central differencing which gives
  \begin{align*}
    y'(x)|_{x = x_i} & \approx \frac{y_{i+1} - y_{i-1}}{2 h}, \\
    y''(x)|_{x = x_i} & \approx \frac{y_{i+1} + y_{i-1} - 2
      y_i}{h^2}.
  \end{align*}

  We can then substitute all of these definitions into the original
  definition to find the finite difference equation that holds for the
  interior points $i = 1, \dots, N$:
  \begin{equation*}
    y_{i+1} \left( 1 - \frac{3}{2} h \right) + y_i \left( -2 + 2 h^2 \right)
    + y_{i-1} \left( 1 + \frac{3}{2} h \right) = 0.
  \end{equation*}
  This defines a linear system for the unknowns $y_i, i = 1, \dots, N$
  of the form
  \begin{equation*}
    T {\bf y} = {\bf f}.
  \end{equation*}
  We can see that the matrix $T$ is tridiagonal and has the form
  \begin{equation*}
    T =
    \begin{pmatrix}
      -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & 0 & 0 & \dots & 0 \\
      1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2} h^2 & 0 & 0 & \dots
      & 0 \\
      0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & \dots
      & 0 \\
      0 & 0 & \ddots & \ddots & \ddots & \dots & 0 \\
      0 & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2} h
      & 0 \\
      0 & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2}
      h \\
      0 & \dots & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2
    \end{pmatrix}
  \end{equation*}
  The right hand side vector results from the boundary data and is
  \begin{align*}
    {\bf f} & =
    \begin{pmatrix}
      - \left( 1 + \tfrac{3}{2} h \right) y_0 \\
      0 \\
      \vdots \\
      0 \\
      - \left( 1 - \tfrac{3}{2} h \right) y_{N+1}
    \end{pmatrix} \\
    & =
    \begin{pmatrix}
      0 \\
      0 \\
      \vdots \\
      0 \\
      - \left( 1 - \tfrac{3}{2} h \right)
    \end{pmatrix}.
  \end{align*}
  As the system is given by a tridiagonal matrix it is simple and
  cheap to solve using, e.g., the Thomas algorithm.
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\item Explain how your algorithm would have to be modified to solve
  the BVP where the boundary condition at $x=1$ becomes the Neumann
  condition
  \begin{equation*}
    y'(1) = 1 + \frac{e}{e-1}.
  \end{equation*}
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  First a finite difference representation of the boundary condition
  is required. A first order representation would be to use backward
  differencing
  \begin{equation*}
    \frac{y_{N+1} - y_N}{h} = 1 + \frac{e}{e-1}.
  \end{equation*}
  This can be rearranged to give
  \begin{equation*}
    y_{N+1} = y_N + h \left( 1 + \frac{e}{e-1} \right).
  \end{equation*}
  So now whenever we replaced $y(1)$ as represented by $y_{N+1}$ by
  the boundary value in the previous algorithm we must instead replace
  it with the above equation which uses the known boundary data and
  unknown interior values.

  Explicitly, this modifies the matrix $T$ to
  \begin{equation*}
    T =
    \begin{pmatrix}
      -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & 0 & 0 & \dots & 0 \\
      1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & 0 & \dots
      & 0 \\
      0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & \dots
      & 0 \\
      0 & 0 & \ddots & \ddots & \ddots & \dots & 0 \\
      0 & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2} h
      & 0 \\
      0 & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \tfrac{3}{2}
      h \\
      0 & \dots & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 +
      \left(1 - \tfrac{3}{2} h \right)
    \end{pmatrix}
  \end{equation*}
  and the right hand side vector ${\bf f}$ to
  \begin{equation*}
    {\bf f}  =
    \begin{pmatrix}
      0 \\
      0 \\
      \vdots \\
      0 \\
      - \left( 1 - \tfrac{3}{2} h \right) h \left( 1 + \frac{e}{e-1}
        \right)
    \end{pmatrix}.
  \end{equation*}
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\end{enumerate}

\end{document}


\documentclass[12pt, a4paper]{article}

\usepackage[utf8]{inputenc} % encoding
\usepackage[english]{babel} % use special characters and also translates some elements within the document.

%%%%% math packages

\usepackage{amsmath} % a bunch of things
\usepackage{amsthm} % \newtheorem, \proof, etc
\usepackage{amssymb} % extended collection

%%%%%%%%%%%%%%%%%%%%%%

\usepackage{hyperref} % Hyperlinks \url{url} or \href{url}{name}
\usepackage{parskip} % Par starts on left (not idented)
\usepackage{abstract} % Abstract

%%%%%%%%%%%%%%%%%%%%%%

\renewcommand{\absnamepos}{flushleft}
\setlength{\absleftindent}{0pt}
\setlength{\absrightindent}{0pt}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\rhead{Arnau Abella}
\lhead{ADM}
\rfoot{Page \thepage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{A Purely Functional Naive Bayes Classifier}
\author{Arnau Abella}
\date{\today}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\usepackage{xcolor} % Colours

\definecolor{codegray}{rgb}{0.9, 0.9, 0.9}

\usepackage{minted} % Code highlight

%\begin{listing}[H]
  %\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
  %\end{minted}
  %\caption{Example of a listing.}
  %\label{lst:example} %  \ref{lst:example}
%\end{listing}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\maketitle

\begin{abstract}
\noindent
  This project provides a \textit{simple} implementation for the naive Bayes classifier in the purely functional programming language Haskell.
  The project provides a library for modeling and trainining a naive Bayes classifier that can be used to predict, with a certain level of confidence, the label of a given arbitrary sample.
\end{abstract}

\section{Introduction} \label{introduction}

\subsection{Probability Model} \label{probability-model}

The naive Bayes is a conditional probability model: given a problem instance to be classifier, represented by a vector $x = (x_1, \dots , x_n)$ representing some $n$ features (independent variables), it assigns to this instance probabilities

\[
  p(C_k | x_1, \dots , x_n)
\]

for each of $k$ possible outcomes or \textit{classes} $C_k$.

The problem with the above formulation is that if the number of features $n$ is large or if a feature can take on a large number of values, then basing such a model on probability tables is \textbf{infeasible}. We therefore reformulate the model to make it more tractable. Using \textbf{Bayes' theorem}, the conditional probability can be decomposed as

\[
  p(C_k | x) = \frac{p(C_k)p(x|C_k)}{p(x)}
\]

In practice, there is interest only in the numerator, because the denominator does not depend on $C$ and the values of the features $x_i$, so that denominator is effectively constant.

We can rewrite the above expression, using the \textbf{chain rule}, as follows

\begin{align*}
  p(C_k, x_1, \dots , x_n) &= p(x_1, \dots, x_n) \\
                           &= p(x_1 | x_2, \dots , x_n, C_k)p(x_2, \dots , x_n, C_k) \\
                           &= \dots \\
                           &= p(x_1 | x_2, \dots , x_n, C_k) \ \dots \ p(x_n | C_k) p(C_k)
\end{align*}

Now, we apply the \textbf{naive} assumption that all features in $x$ are \textbf{mutually independent}, conditional on the category $C_k$

\[
  p(x_i | x_{i+1}, \dots , x_n , C_k) = p(x_i | C_k)
\]

Hence, the joint model can be expressed as

\begin{align*}
  p(C_k | x_1, \dots , x_n) &\propto  p (C_k, x_1, \dots , x_n) \\
                            &= p(C_k)p(x_1|C_k)p(x_2|C)p(x_3|C_k) \dots \\
                            &= p(C_k)\prod_{i=1}^{n}p(x_i|C_k)
\end{align*}

where $\propto$ denotes proportionality.

\subsection{The classifier} \label{the-classifier}

Given the previous naive Bayes probability model, we can construct a classifier combining it with a \textbf{decision rule}.

A common decision rule is \textit{MAP} decision rule (picking the hypoteshis that is most probable).

The corresponding classifier, a Bayes classifier, is the function that assigns a class label $y = C_k$ for some $k$ as follows

\[
  y = \underset{k \in \{ 1, \dots, K\}}{argmax} \ p(C_k) \prod_{i=1}^{n} p(x_i|C_k)
\]

\subsection{Parameter estimation and event model}  \label{parameter-estimation}

A class's prior $p(C_k)$ can be computed by calculation an estimate for the class probability from the training set.

To estimate the parameters for a feature's distribution, one must assume a distribution from the training set.

The assumptions on distributions of features are called the \textit{event model} of the Naive Bayes classifier.

For discrete features, \textit{Multinomial} or \textit{Bernoulli} distributions are popular. For continuous features, it is typical to assume a \textit{Gaussian} distribution.

\section{Implementation in Haskell} \label{implementation-in-haskell}

Haskell is a general-purpose, statically typed, purely functional programming language with type inference and lazy evaluation.

The idea behind this project was to build a minimal executable to run naive Bayes over an input dataset to show that building such a predictor
in Haskell was feasible. But, at the end, it finished being a library that allow a final user to parse a dataset, train a model with the given dataset using
a validation technique such as cross validation, and predict unlabeled samples using the trained model.

The only drawback was that the Haskell ecosystem for Data Science is still immature \cite{stateofhaskell}. You can verified it by searching at
hackage \cite{hackage} for the keyword "naive bayes". The only mature option for Bayesian probabilistic was \textit{monad-bayes} \cite{monad-bayes}.

\subsection{The project} \label{project-building}

The project was organized as a Haskell \textbf{library} rather than an executable so the code can be extended and use by other people for free.

The main modules are the following:

\begin{itemize} \label{modules-list}
  \item Parser
  \begin{itemize}
    \item[$\ast$] Parser.Csv
  \end{itemize}
  \item Distrib
  \item Validation
  \item Classifier
  \begin{itemize}
    \item[$\ast$] Classifier.Dummy
    \item[$\ast$] Classifier.NaiveBayes
  \end{itemize}
\end{itemize}

Additionally, there is the Lib module that reexports all the previous modules so you don't need to export one by one.

Apart from the library, the project also includes an \textbf{executable} to run a couple of test data sets to experiment a bit with the library.

In the following sections, we are going to visit each module and its implementation.

If you are not interested in the implementation, jump to the next section \ref{running-the-project}

\subsubsection{Parser} \label{parser}

The module \textit{Parser} contains a collection of parser for common storage format such as Comma-separated values (CSV).

For now, the only implemented parser is for CSV. This parser uses the library cassava \cite{cassava} to levarage the work of parsing a csv file.

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
parseCsvFile
    :: Csv.FromRecord a
    => FilePath -> HasHeader -> ExceptT String IO [a]
parseCsvFile fp hasHeader = do
  liftEither (checkExtension fp)
  lbs <- liftIO (LBSC.readFile fp)
  liftEither $ fmap V.toList (Csv.decode hasHeader lbs)
\end{minted}

\subsubsection{Distrib} \label{distrib}

In the module \textit{Distrib} you will find common distributions such as Uniform, Gaussian, Gamma, Beta, Categorical, Geometric, Poisson, etc.

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
data Distrib where
  Distrib :: ContDistr d => d -> Distrib
\end{minted}

For now, you will only find the \textbf{Normal distribution} for numerical variables and the probability density function of it.

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
data Distrib where
-- | We don't know much about our numeric variable
-- so we will assume normallity to simplify the implementation.
compNormalDistr :: [Double] -> Distrib
compNormalDistr xs =
  Distrib (normalDistr mean' stdDev')
    where
      v = VU.fromList xs
      mean' = Statistics.mean v
      stdDev'  = max 0.1 (Statistics.stdDev v)

-- | Probability Density Function
pdf :: Distrib -> Double -> Probability
pdf (Distrib d) = Probability . density d
\end{minted}

\subsubsection{Validation} \label{validation}

The module \textit{Validation} implements model validation techniques such as simple sample validation, cross validation, bootstrapping validation, etc.

The current implementation only includes cross validation.

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
-- | X-Validation testing algorithm
--
-- Notice that every subset of <training, test> is different
-- and generates a new classifier, so the score is not for a
-- single classifier, but for many.
crossValidation
    :: forall c. Classifier c
    => [(Sample, Class)] -- ^ All Samples
    -> Proxy c
    -> Score
crossValidation allSamples _ =
  let groups = leaveOneOutN (groupBy' 10 allSamples)
  in foldMap test groups
    where
      test (trainingSet, testSet) =
        let classifier = (train trainingSet) :: c
         in foldMap (predictAndGetScore classifier) testSet

      predictAndGetScore classifier (sample, clazz) =
        let predictions = predict sample classifier
            cmp = (\(_, p1) (_, p2) ->
                    (_probability p1) `compare` (_probability p2))
            (pClazz, _) = maximumBy cmp predictions
         in
            if pClazz == clazz then hit' else miss'
\end{minted}

\subsubsection{Classifier} \label{classifier}

In the module \textit{Classifier} the user will find the basic interface of all classifiers

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
class Classifier c where
  -- | Given a training set returns a trained classifier
  train :: [(Sample, Class)] -> c
  -- | Given a random sample, predict a class
  predict :: Sample -> c -> [(Class, Probability)]
\end{minted}

The following classifier are currently implemented

\begin{itemize}
  \item A random classifier on \textit{Classifier.Dummy}
  \item A random classifier on \textit{Classifier.NaiveBayes}
\end{itemize}

The Naive Bayes classifier has the following structure

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
data NaiveBayes = NaiveBayes
  { labels  :: [Class] -- ^ All labels
  , priors  :: Map Class Probability
  , distrib :: Map Class (Map Index Distrib)
  }

instance Classifier NaiveBayes where
  train :: [(Sample, Class)] -> NaiveBayes
  train = trainNaiveBayes
  {-# INLINE train #-}

  predict :: Sample -> NaiveBayes -> [(Class, Probability)]
  predict = predictNaiveBayes
  {-# INLINE predict #-}
\end{minted}

The source code can be found on github \cite{naive-bayes-classifier}.

\subsection{Project testing} \label{project-testing}

\subsubsection{Compiling the project} \label{compiling}

Before running the project, we need to compile it.

I made the effort to simplify compilation and dependency management using nix \cite{nix}.
Nix is a purely functional package manager that, among other things, allow us to compile Haskell projects in a pure deterministic way.

In order to compile the project you need to follow the following steps:

\begin{enumerate}
  \item Install nix in your local machine
    \begin{minted}{bash}
      $ curl https://nixos.org/nix/install | sh
    \end{minted}
  \item Compile the project using nix
    \begin{minted}{bash}
      $ nix-build
    \end{minted}
  \item Run the executable on the input datasets
    \begin{minted}{bash}
      $ ./result/bin/naive-bayes-classifier
    \end{minted}
\end{enumerate}

The compilation steps are only for UNIX systems.

\subsubsection{Running the classifier} \label{compiling}

To test the implementation I added two data sets randomly selected from Kaggel \cite{kaggel}:

\begin{itemize}
  \item Wine quality \cite{wine-quality}
  \item Wine origin \cite{wine-origin}
\end{itemize}

The results are the following:

\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{bash}
---------Dataset: wine-quality
#Classes: 6
#Total of samples: 1599
Score {hit = 878, miss = 721, total = 1599}
Accuracy: 54.97 %

---------Dataset: wine-origin
#Classes: 3
#Total of samples: 178
Score {hit = 170, miss = 8, total = 178}
Accuracy: 95.50 %
\end{minted}

The accuracy for the first dataset is low, $54.97\%$.

The accuracy for the second dataset is very high, $95.50\%$.

I did not spent time analyzing why Naive Bayes performs poor on the first dataset but, in contrast, performs extremly well on the second one as it was not the objective
of this delivery but it would something nice to do as a future project.

\section{Conclusion} \label{conclusion}

This library is a proof of concept that Naive Bayes classifier can be easily coded in Haskell in about 700 lines of code with the benefits of static correctness, friendly and robust refactoring, high level abstractions, high performance, ... Haskell is a suitable language to build such a probabilistic models.

\section{Future work} \label{future-work}

Regarding to this project, there is a lot of margin for improvement. The current implementation of the library is missing a well defined structure which leads
to non composable code that is ready hard to extend. Also, the library is missing a lot of features such as other parsers, distributions, validation and classifiers.

Regarding to the Haskell ecosystem, there is a lot of work to be on the Data Science domain to be at the same level as other languages such as R or Python. This project is
a very little step towards a more mature haskell ecosystem. Early adopter companies such as \href{https://www.tweag.io/}{tweag.io} are working hard to bring more libraries to the ecosystem and make Haskell a mature enough place to start working on Data Science in it.

\begin{thebibliography}{9}
\bibitem{stateofhaskell} Gabriel Gonzalez. 2020. State of the Haskell ecosystem. \url{https://github.com/Gabriel439/post-rfc/blob/master/sotu.md}
\bibitem{hackage} hackage: The Haskell Package Repository. \url{https://docs.haskellstack.org/en/stable/README/}
\bibitem{monad-bayes} monad-bayes: A library for probabilistic programming. \url{https://hackage.haskell.org/package/monad-bayes}
\bibitem{cassava} cassava: A CSV parsing and encoding library. \url{https://hackage.haskell.org/package/cassava}
\bibitem{naive-bayes-classifier} naive-bayes-classifier \url{https://github.com/monadplus/naive-bayes-classifier}
\bibitem{kaggel} Kaggel datasets \url{https://www.kaggle.com/datasets}
\bibitem{wine-quality} Wine quality \url{https://www.kaggle.com/uciml/red-wine-quality-cortez-et-al-2009#winequality-red.csv}
\bibitem{wine-origin} Wine origin \url{https://archive.ics.uci.edu/ml/datasets/Wine}
\bibitem{kaggel} Kaggel datasets \url{https://www.kaggle.com/datasets}
\bibitem{nix} Nix: a purely functional package manager \url{https://nixos.org/nix/}
\end{thebibliography}

\end{document}


\subsection{Functions}\label{subsec:functions}

\begin{remark}\label{rem:function_definition}
  It is not straightforward to formalize the notion of correspondence between two values. We will synonymously use the terms \term{mapping}, \term{function}, \term{transformation} and \term{operator}. We will define functions as special binary relations in \fullref{def:function}. Despite this being a standard practice, this has several drawbacks:

  \begin{itemize}
    \item There are some more mappings than the functions defined in \fullref{def:function}. For example, assigning to a set \( A \) its \hyperref[def:basic_set_operations/power_set]{power set} \( P(A) \) can be cannot be regarded as a function because its domain and range should both be the set of all sets whose existence is inconsistent with \hyperref[def:zfc]{\logic{ZFC}} by \fullref{thm:simple_foundation_theorems/member_of_itself}.

    \item We often work with spaces that have some additional structure in addition to being sets. In this case, we are often only interested in maps that preserve this structure. This is the case with \hyperref[def:group/homomorphism]{group homomorphisms}, for example.

    In terms of \hyperref[def:first_order_structure]{first-order structures}, not every function is a \hyperref[def:first_order_homomorphism]{homomorphism}.

    This is a motivating example for the benefits of \hyperref[sec:category_theory]{category theory}, where the notion of \hyperref[def:category/morphisms]{morphism} is able to capture this additional structure.

    It is often important to consider functions that are not homomorphisms, however. For example, function spaces over \( \BbbR \) contain some very complicated functions that are not field homomorphisms, order homomorphisms nor continuous functions and thus do not aim to preserve structural properties of \( \BbbR \).

    \item Several generalizations of the standard notion of a function are often used. These include \hyperref[def:multi_valued_function]{multi-valued} and \hyperref[def:partial_function]{partial functions}. Both are not functions, strictly speaking.

    For simplicity of exposition, we take multi-valued functions as primitive notions and define single-valued functions as special cases. This is actually done in \cite[def. 2.31]{OpenLogicFull} and \cite[8]{Kelley1955} except that the corresponding authors conflate multi-valued functions and relations.

    \item Set-theoretic functions are often used in contexts where they do not refer to the intuitive notion of a mapping. Examples include \hyperref[def:cartesian_product]{Cartesian products} and \hyperref[def:cartesian_product/indexed_family]{indexed families}.
  \end{itemize}
\end{remark}

\begin{definition}\label{def:multi_valued_function}
  A \term{multi-valued function} from \( A \) to \( B \) is simply a \hyperref[def:binary_relation]{binary relation} \( (F, A, B) \). For us the difference between a multi-valued function and a relation is merely in how we treat them. Multi-valued function are also called set-valued.

  As discussed in \fullref{def:relation/graph}, the common practice of identifying a multi-valued function \( (F, A, B) \) with its \hyperref[def:relation/graph]{graph} \( F \) with no regard to its \hyperref[def:relation/signature]{signature} \( (A, B) \) has serious drawbacks that we wish to avoid.

  We use the more established notation \( F: A \multto B \) rather than \( (F, A, B) \) and call the string of symbols \( A \multto B \) the signature of \( F \) rather than the pair \( (A, B) \).

  \begin{thmenum}[series=def:multi_valued_function]
    \thmitem{def:multi_valued_function/value} The \term{value} of \( F \) at \( x \) is
    \begin{equation*}
      F(x) \coloneqq \set{ y \in B \given (x, y) \in F }.
    \end{equation*}

    In case \( x \) is not a concrete value, then \( F(x) \) stands for the function \( F \) itself. In other words, \( F(x) \) refers to a member of \( B \) if \( x \) is a \hyperref[def:first_order_syntax/formula_bound_variables]{bound variable} and \( F(x) \) refers to the function \( F \) if \( x \) is a \hyperref[def:first_order_syntax/formula_free_variables]{free variable}.

    \thmitem{def:multi_valued_function/set_value} We also define the value of \( F \) at a subset \( X \) of \( A \) as
    \begin{equation*}
      F[X] \coloneqq \bigcup \set{ F(x) \given x \in X }.
    \end{equation*}

    This is also called the \term{action} of \( F \) on the set \( X \) or the \term{image} of \( X \) under \( F \). We also refer to \( F(x) \) as the image of \( x \) under \( F \) because \( F(x) = F[\set{x}] \).

    The notation \( f[X] \) is mostly used withing set theory, for example in \cite[def. 2.31]{OpenLogicFull} and \cite[44]{Enderton1977Sets}, however the following notation is usually used preferred of set theory:
    \begin{equation*}
      f(X) = \bigcup \set{ F(x) \given x \in X }
    \end{equation*}

    We use \( f[X] \) almost exclusively for \fullref{sec:set_theory} because of the following nasty ambiguity: If \( A \) is a \hyperref[def:transitive_set]{transitive set}, then \( x \in A \) implies that \( x \subseteq A \) yet \( f(x) \neq f[x] \). For example, if \( A = \set{ \varnothing, \set{ \varnothing } } \), then
    \begin{equation*}
      f[\set{ \varnothing }] = \set{ f(\varnothing) } \neq f(\set{ \varnothing }).
    \end{equation*}

    This may be very confusing when dealing with \hyperref[def:ordinal]{ordinals}.

    \thmitem{def:multi_valued_function/arguments} As mentioned in \fullref{def:multi_valued_function/value}, given a function \( f: A \to B \), we sometimes use the notation \( f(x) \) where \( x \) is a free variable in the sense of \fullref{def:first_order_syntax/formula_free_variables}.

    If \( A = A_1 \times \cdots \times A_n \) is a \hyperref[def:cartesian_product/product]{finite Cartesian product}, we instead use the notation \( f(x_1, \ldots, x_n) \) and regard \( x_1, \ldots, x_n \) as free variables that have no assigned value.

    The variables are called \term{arguments} or sometimes \term{parameters}, although the latter term is a bit overloaded. This notion is somewhat informal and depends on the context since \( A \) can usually be represented as a Cartesian product in different ways and with different arities. For example, if \( A = B \times C \), we can write both \( f(a) \) and \( f(b, c) \) and the function has a different number of parameters in each case. In practice the number of arguments is usually clear from the context. We sometimes use \( \vect{a} \) when we regard \( a \) as a tuple.

    For example, in \hyperref[def:first_order_semantics]{classical first-order semantics}, to each \( n \)-ary functional symbol there corresponds an \( n \)-ary function with the unambiguous signature \( X^n \to X \).

    When working over a vector space like \( \BbbR^2 \), on the other side, depending on the context we regard \hyperref[rem:functional]{functionals} as either unary or binary functions.

    We sometimes refer to \( f \) as a \term{dependent variable} since it depends on its arguments. In this later case, we call the arguments \term{independent variables}.

    Common terms for functions with low arity:
    \begin{itemize}
      \item \term{Nullary} for \( n = 0 \).
      \item \term{Unary} or \term{univariate} for \( n = 1 \).
      \item \term{Binary} or \term{bivariate} for \( n = 2 \).
      \item \term{Ternary} for \( n = 3 \).
      \item \term{Multivariate} for \( n > 1 \).
    \end{itemize}
  \end{thmenum}

  The following terminology is consistent with relations:
  \begin{thmenum}[resume=def:multi_valued_function]
    \thmitem{def:multi_valued_function/graph} The \term{graph} \( \gph(F) \) of \( F \) is the graph of the relation \( F \). This is consistent with \fullref{def:relation/graph}.

    \thmitem{def:multi_valued_function/restriction} The \term{restriction} of \( F: A \multto B \) to \( X \subseteq A \) is the multi-valued function \( F\restr_{X}: X \rightarrow B \). We say that \( F \) is an \term{extension} of \( F\restr_{X} \). This is consistent with \fullref{def:binary_relation/restriction}.

    \mcite{nLab:corestriction} Dually, if we restrict \( B \) to its subset \( Y \), we say that the obtained function is a \term{corestriction}.

    \thmitem{def:multi_valued_function/composition} The \term{composition} \( G \bincirc F \) of two multi-valued functions \( F: A \multto B \) and \( G: B \multto C \) is the function
    \begin{equation*}
      [G \bincirc F](x) \coloneqq G(F(x)).
    \end{equation*}

    The square brackets around \( G \bincirc F \) are not a special notation, but rather another pair of delimiters that looks different from parentheses for the sake of reducing visual clutter.

    This definition is consistent with \fullref{def:binary_relation/composition}.

    \thmitem{def:multi_valued_function/empty} A function \( F: A \multto B \) is \term{empty} if \( \gph(F) = \varnothing \). This is consistent with \fullref{def:binary_relation/empty}.

    Note that if \( F \) is a \hyperref[def:multi_valued_function/total]{total} empty multi-valued function, then \( A = \varnothing \) because otherwise \( F \) would itself be nonempty.
  \end{thmenum}

  The following terminology is inconsistent with relations:
  \begin{thmenum}[resume=def:multi_valued_function]
    \thmitem{def:multi_valued_function/arity} The \term{arity} of a multi-valued function is its number of arguments. This is not to be confused with \hyperref[def:relation/arity]{relation arity} --- functions are always binary relations.

    \thmitem{def:multi_valued_function/total} The term \term{total multi-valued function} means that \( \dom(F) = A \), i.e. that \( F(x) \neq \varnothing \) for all \( x \in A \). This is very different from total binary relations as defined in \fullref{def:binary_relation/total}.

    \thmitem{def:multi_valued_function/inverse} The \term{inverse} \( F^{-1}: B \multto A \) of a multi-valued function \( F: A \multto B \) is the multi-valued function in which assigns to every element \( y \) of the image \( \img(F) \) the set of all \( x \in A \) such that \( y \in F(x) \). It is therefore the \hyperref[def:binary_relation/converse]{converse binary relation} of \( F \).

    \thmitem{def:multi_valued_function/identity}\mcite[exer. 3.1.1]{Leinster2016Basic} The function corresponding to the diagonal relation \( \increment A \) as defined in \fullref{def:binary_relation/diagonal} is called the \term{identity function} on \( A \) and denoted by \( \id_A \). The \term{diagonal function} is instead defined as \( f: A \to A^2 \) is defined as \( f(x) \coloneqq (x, x) \). This is mostly used in category theory.
  \end{thmenum}

  We define some additional terminology:
  \begin{thmenum}[resume=def:multi_valued_function]
    \thmitem{def:multi_valued_function/image} The \term{image} \( \img(F) \) of \( F \) is the set of all \( y \in B \) that belong to the set \( F(x) \) for at least one \( x \in A \). It is the same as the value \( F(A) \).

    \thmitem{def:multi_valued_function/domain} The \term{domain} \( \dom(F) \) of \( F \) is the set of all values \( x \) for which \( F(x) \neq \varnothing \). When regarding \( F \) as a relation, the domain can be defied as the set
    \begin{equation*}
      \dom(R) \coloneqq \set{ x \in A \given \qexists {y \in B} (x, y) \in R }.
    \end{equation*}

    \thmitem{def:multi_valued_function/range} The \term{range} \( \range(F) \) is simply the set \( B \). It is also called the \term{codomain} of \( F \).

    \thmitem{def:multi_valued_function/superposition} Although the terms \enquote{composition} and \enquote{superposition} are used interchangeably, for example in \cite[\textnumero 25]{Фихтенгольц1968Том1}, the term \enquote{superposition} often refers to a certain generalization of \hyperref[def:multi_valued_function/composition]{function composition}. The latter usage can be found in \cite[16]{Яблонский1986}.

    If we are given the family of functions \( \seq{ F_k: A \multto B_k }_{k = 1, \ldots, n} \) and the function \( G: B_1 \times \cdots \times B_n \multto C \), we defined their \term{superposition} is
    \begin{equation*}
      \begin{aligned}
        &H: A \to C
        &H(x) \coloneqq G(F_1(x), \ldots, F_n(x)).
      \end{aligned}
    \end{equation*}

    \thmitem{def:multi_valued_function/endofunction} Functions from a set to itself (e.g. \( F: A \multto A \)) are called \term{endofunctions}.

    \thmitem{def:multi_valued_function/large_preimage} For a fixed set \( Y \subseteq B \), its \term{large preimage} or simply \term{preimage} under \( F: A \to B \) is the image of \( Y \) under the inverse function \( F^{-1}: B \multto A \). For a single value \( y \in B \), we call \( F^{-1}(y) \) the \term{fiber} of \( y \) under \( F \).

    \thmitem{def:multi_valued_function/small_preimage} Analogously, we define its \term{small preimage} as
    \begin{equation*}
      F_{-1}[Y] \coloneqq \set{ x \in A \colon F(x) \subseteq Y }.
    \end{equation*}

    Obviously \( F_{-1}[Y] \subseteq F^{-1}[Y] \).
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:def:multivalued_function}
  \hyperref[def:multi_valued_function]{Multi-valued functions} have the following basic properties:

  \begin{thmenum}
    \thmitem{thm:def:multivalued_function/associative} \hyperref[def:multi_valued_function/composition]{Composition} is associative. That is, for any three functions \( F: A \to B \), \( G: B \to C \) and \( H: C \to D \) we have
    \begin{equation*}
      H \bincirc [G \bincirc F] = [H \bincirc G] \bincirc F.
    \end{equation*}

    We will henceforth simply write \( H \bincirc G \bincirc F \).

    \thmitem{thm:def:multivalued_function/composition_inverse} If \( F: A \to B \) and \( G: B \to C \) are \hyperref[def:multi_valued_function]{multi-valued functions}, then
    \begin{equation*}
      [G \bincirc F]^{-1} = F^{-1} \bincirc G^{-1}.
    \end{equation*}
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:multivalued_function/associative} Let \( a \in A \). Then in order for \( d \in D \) to belong to \( [[H \bincirc G] \bincirc F](a) \), there must exist values \( b \in B \) and \( c \in C \) such that \( b \in F(a) \) and \( c \in G(b) \) and \( d \in H(c) \). Clearly this is also the condition for \( d \) to belong to \( [H \bincirc [G \bincirc F]](a) \).

  \SubProofOf{thm:def:multivalued_function/composition_inverse} Since \( G \bincirc F \) has signature \( A \to C \), clearly \( [G \bincirc F]^{-1} \) has signature \( C \to A \). Let \( c \in C \).

  \begin{itemize}
    \item If \( [G \bincirc F]^{-1}(c) \) is empty, \( c \not\in \img(G \bincirc F) \), hence either \( G^{-1}(c) \) is empty or is nonempty, but disjoint from \( \img(F) \). Hence, \( [F^{-1} \bincirc G^{-1}](c) \) is also empty.

    \item Suppose that \( [G \bincirc F]^{-1}(c) \) is not empty and let \( a \in [G \bincirc F]^{-1}(c) \).

    By definition, there exists \( b \in B \) and such that \( b \in F(a) \) and \( c \in G(b) \). Hence, \( b \in G^{-1}(c) \) and \( a \in F^{-1}(b) \), which implies that the image of \( c \) under the composition \( F^{-1} \bincirc G^{-1} \) also contains \( a \).
  \end{itemize}

  In both cases, for every \( c \in C \) we have
  \begin{equation*}
    [G \bincirc F]^{-1}(c) = [F^{-1} \bincirc G^{-1}](c).
  \end{equation*}

  Hence, the two multi-valued functions are equal.
\end{proof}

\begin{definition}\label{def:function}
  Although \hyperref[def:multi_valued_function]{multi-valued functions} are very general, they are not nearly as useful nor are not studied nearly as extensively as single-valued functions.

  The \hyperref[def:multi_valued_function]{multi-valued function} \( F: A \multto B \) is called a \term{single-valued function} if \( F(x) \) is a \hyperref[rem:singleton_sets]{singleton set} for each \( x \in A \). In this case, we write \( F: A \to B \) rather than \( F: A \multto B \).

  All terminology from \fullref{def:multi_valued_function} holds for single-valued functions.

  By convention, when both single-valued and multi-valued functions are involved, the former are denoted using lowercase letters and the latter using uppercase letters.

  Strictly speaking, the value \( f(x) \) of a single-valued function \( f: A \to B \) is a singleton set. It is common practice (e.g. in \cite[def. 3.1]{OpenLogicFull} and \cite[10]{Kelley1955}) to define the value of a single-valued function to be an element of \( B \) rather than a subset of \( B \). Unless this would be confusing, we identify \( f(x) \) with its only element due to the convention established in \fullref{rem:singleton_sets}.

  More precisely, single-valued functions satisfy the following \hyperref[rem:predicate_formula]{predicate formula}, which states that the free variable \( \chi \) is a function from \( \alpha \) to \( \beta \):
  \begin{equation*}\taglabel[\op{IsFun}]{eq:def:function/predicate}
    \ref{eq:def:function/predicate}[\chi, \alpha, \beta] \coloneqq \qforall {\xi \in \alpha} \qExists {\eta \in \beta} \underbrace{ \qexists {\zeta \in \chi} \ref{eq:def:cartesian_product/kuratowski_pair_predicate}[\zeta, \xi, \eta] }_{\chi(\xi) = \eta}.
  \end{equation*}

  Unless otherwise noted, we will now conflate the terms \enquote{function} and \enquote{single-valued function}.

  \begin{thmenum}
    \thmitem{def:function/set_value} We can simplify \fullref{def:multi_valued_function/set_value} drastically:
    \begin{equation*}
      f[A]
      =
      \bigcup \set[\Big]{ \set{ f(x) } \given x \in A }
      =
      \set{ f(x) \given x \in A }.
    \end{equation*}

    As mentioned in \fullref{def:multi_valued_function/set_value}, we usually prefer the notation \( f[A] \) outside \fullref{sec:set_theory} where we are less prone to ambiguity.

    \thmitem{def:function/currying} Given the \hyperref[def:multi_valued_function/arguments]{two-argument} function \( f: A \times B \to C \), we can define another function \( g: A \to \fun(B, C) \) as
    \begin{equation*}
      g(x) \coloneqq (y \mapsto f(x, y)).
    \end{equation*}

    This process is called \term{currying} after Haskell Curry. Obtaining \( f \) from \( g \) is instead called \term{uncurrying}.

    Currying is useful if we have somehow fixed a value \( x_0 \in A \), in which case we can \enquote{get rid} of one argument by introducing some shortcut for the function \( g(x_0): B \to C \) for the sake of reducing notational clutter. See \fullref{def:differentiability/first_variation} and the proof of \fullref{thm:countably_infinite_union_of_countably_infinite_sets} for example of how this is useful in the wild.

    See \fullref{thm:currying_is_natural_isomorphism} for a categorical description of currying.

    \thmitem{def:function/selection}\mcite[52]{DontchevRockafellar2014} If \( f: A \to B \) is a single-valued function, \( F: A \multto B \) is a multi-valued function and \( \gph(f) \subseteq \gph(F) \), we say that \( f \) is a \term{selection} of \( F \).

    \thmitem{def:function/set_of_functions} We denote the set of all single-valued total functions from \( A \) to \( B \) by \( \fun(A, B) \). Other accepted notation is either \( \cat{Set}(A, B) \) (which is consistent with \hyperref[def:category_of_small_sets]{morphisms in the category of sets}) or by \( B^A \) (which is consistent with \hyperref[thm:cardinal_exponentiation_power_set]{cardinal exponentiation}). We abbreviate \( \fun(A, A) \) as \( \fun(A) \)
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:def:function}
  \hyperref[def:function]{Single-valued functions} have the following properties when regarded as \hyperref[def:multi_valued_function]{multi-valued functions}:
  \begin{thmenum}
    \thmitem{thm:def:function/total} Single-valued functions are \hyperref[def:multi_valued_function/total]{total} as multi-valued functions. As a consequence, a multi-valued function cannot have a selection unless it is \hyperref[def:multi_valued_function/total]{total}.

    This explains why there is no established terminology for the set \( A \) analogous to \enquote{range} for \( B \) --- we rarely need to differentiate between \( A \) and \( \dom(f) \).

    \thmitem{thm:def:function/composition} The \hyperref[def:multi_valued_function/composition]{composition} of single-valued functions is a single-valued function.

    \thmitem{thm:def:function/preimage} The \hyperref[def:multi_valued_function/large_preimage]{large preimages} and \hyperref[def:multi_valued_function/small_preimage]{small preimages} of single-valued functions are identical. We restrict ourselves to large preimages with the notation \( f^{-1}(x) \) and refer to them simply as preimages.

    Note that preimages are multi-valued in general.
  \end{thmenum}
\end{proposition}
\begin{proof}
  Trivial.
\end{proof}

\begin{definition}\label{def:partial_function}
  A \hyperref[def:multi_valued_function]{Multi-valued function} that is otherwise single-valued, but not necessarily \hyperref[def:multi_valued_function/total]{total} is called a \term{partial function}. That is, \( f: A \to B \) is a partial function if \( f(x) \) has at most one element for every \( x \in A \).
\end{definition}

\begin{definition}\label{def:function_invertibility}
  We introduce the following terminology for invertibility of a (single-valued) function \( f: A \to B \):
  \begin{thmenum}
    \begin{minipage}[t]{0.43\textwidth}
      \thmitem{def:function_invertibility/injective} We say that \( f \) is \term{injective} or \term{one-to-one} if any of the following equivalent conditions hold:
      \begin{thmenum}
        \thmitem{def:function_invertibility/injective/existence} For any \( y \in B \) there exists at \hi{most} one \( x \in A \) such that \( f(x) = y \).

        That is, each point in \( B \) is the image of at most one point in \( A \).
        \newline

        \thmitem{def:function_invertibility/injective/equality} For all \( x_1, x_2 \in A \), the equality \( f(x_1) = f(x_2) \) implies \( x_1 = x_2 \).

        The contrapositive of this statement is that different points in \( A \) have different images under \( f \).

        \thmitem{def:function_invertibility/injective/inverse} The inverse is a partial single-valued function.
      \end{thmenum}
    \end{minipage}
    \hfill
    \begin{minipage}[t]{0.44\textwidth}
      \thmitem{def:function_invertibility/surjective} \( f \) is called \term{surjective} or \term{onto} if any of the following equivalent conditions hold:
      \newline
      \begin{thmenum}[leftmargin=0.9cm]
        \thmitem{def:function_invertibility/surjective/existence} For any \( y \in B \) there exists at \hi{least} one \( x \in A \) such that \( f(x) = y \).

        That is, each point in \( B \) is the image of at least one point in \( A \). Hence, the image of \( f \) is the entire range \( B \).

        \thmitem{def:function_invertibility/surjective/equality} For all \( y_1, y_2 \in B \), the equality \( f^{-1}(y_1) = f^{-1}(y_2) \) implies \( y_1 = y_2 \).

        Without surjectivity, the above holds only for the points in the image of \( f \).

        \thmitem{def:function_invertibility/surjective/inverse} The inverse is a total multi-valued function.
      \end{thmenum}
    \end{minipage}

    \thmitem{def:function_invertibility/bijective} Finally, \( f \) is called \term{bijective} if any of the following equivalent conditions hold:
    \begin{thmenum}
      \thmitem{def:function_invertibility/bijective/direct} It is both injective and surjective.
      \thmitem{def:function_invertibility/bijective/existence} For any \( y \in B \) there exists exactly one \( x \in A \) such that \( f(x) = y \).
      \thmitem{def:function_invertibility/bijective/inverse} The inverse is a total single-valued function.
    \end{thmenum}
  \end{thmenum}
\end{definition}
\begin{defproof}
  The equivalences are trivial to verify.
\end{defproof}

\begin{proposition}\label{thm:function_composition_invertibility}
  The composition of injective (resp. surjective or bijective) functions is injective (resp. surjective or bijective).

  Compare this result to \fullref{thm:def:morphism_invertibility/invertible_composition} and \fullref{thm:function_superposition_invertibility}.
\end{proposition}
\begin{proof}
  Let \( f: A \to B \) and \( g: B \to C \) be arbitrary functions and define \( h \coloneqq g \circ f: A \to C \).

  \SubProofOf[def:function_invertibility/injective/equality]{injectivity} Suppose that \( f \) and \( g \) are injective. We will use \fullref{def:function_invertibility/injective/equality}.

  Let \( x_1, x_2 \in A \) and suppose that \( h(x_1) = h(x_2) \), that is, \( g(f(x_1)) = g(f(x_2)) \). Then \( f(x_1) = f(x_2) \) since \( g \) is injective and \( x_1 = x_2 \) since \( f \) is injective.

  Since \( x_1 \) and \( x_2 \) were arbitrary, we conclude that \( h \) is also injective.

  \SubProofOf[def:function_invertibility/surjective/existence]{surjectivity} Suppose that \( f \) and \( g \) are surjective. We will use \fullref{def:function_invertibility/injective/existence}.

  Let \( z \in C \). Then there exists some \( y \in B \) such that \( g(y) = z \) because \( g \) is surjective and similarly there exists some \( x \in B \) such that \( f(x) = y \) because \( f \) is surjective. Thus, \( h(x) = g(f(x)) = z \).

  Since \( z \) was arbitrary, we conclude that \( h \) is also surjective.

  \SubProofOf[def:function_invertibility/bijective/direct]{bijectivity} We have shown that if \( f \) and \( g \) are both injective or surjective, then, so is \( h \). Hence, if \( f \) and \( g \) are bijective, so is \( h \).
\end{proof}

\begin{lemma}\label{thm:diagonal_product_injectivity}
  Let \( \seq{ f_k }_{k \in \mscrK} \) be an indexed family of functions where \( f_k: A \to B_k \) in \( k \in \mscrK \). Then if at least one of the functions is injective, the \hyperref[def:topological_product]{diagonal product}
  \begin{equation*}
    \begin{aligned}
      f: A \to \prod B_k \\
      f(x) \coloneqq \seq{ f_k(x) }_{k \in \mscrK}
    \end{aligned}
  \end{equation*}
  is also injective.
\end{lemma}
\begin{proof}
  Suppose that \( f_{k_0} \) is injective. Let \( f(x_1) = f(x_2) \). Then \( \seq{ f_k(x_1) }_{k \in \mscrK} = \seq{ f_k(x_2) }_{k \in \mscrK} \) and thus \( f_{k_0}(x) = f_{k_0}(y) \). Since \( f_{k_0} \) is injective, we conclude that \( x_1 = x_2 \). Since \( x_1 \) and \( x_2 \) were arbitrary, we conclude that \( f \) is also injective.
\end{proof}

\begin{proposition}\label{thm:function_superposition_invertibility}
  The superposition of injective functions is injective.

  Compare this result to \fullref{thm:function_composition_invertibility}.
\end{proposition}
\begin{proof}
  Suppose that we are given injective functions \( f_k: A \multto B_k, k = 1, \ldots, n \) and \( g: B_1 \times \cdots \times B_n \multto C \). From \fullref{thm:diagonal_product_injectivity} it follows that the function
  \begin{equation*}
    \begin{aligned}
      d: A \to B_1 \times \cdots \times B_n
      d(x) \coloneqq (f_1(x), \ldots, f_n(x))
    \end{aligned}
  \end{equation*}
  is injective.

  Then from \fullref{thm:function_composition_invertibility} it follows that the desired superposition
  \begin{equation*}
    \begin{aligned}
      &h: A \to C
      &h(x) \coloneqq g(f_1(x), \ldots, f_n(x)) = g(d(x)).
    \end{aligned}
  \end{equation*}
  is injective.
\end{proof}

\begin{remark}\label{rem:multi_valued_functions}
  We can represent a \hyperref[def:multi_valued_function]{multi-valued functions} \( F: A \multto B \) as the \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \set{ F(a) }_{a \in A} \). This indexed family is itself a \hyperref[def:function]{single-valued function} \( G \) from \( A \) to \( \pow(B) \). This is an alternative to our approach to define \hyperref[def:multi_valued_function]{multi-valued functions} as a basic notion. The latter approach is used, for example, in \cite[def. 2.3]{Phelps1993}. The pair \( (x, y) \in A \times B \) belongs to the relation \( F \) if and only if there exists a subset \( Y \subseteq B \) such that \( y \in Y \) and the pair \( (x, Y) \) belongs to the relation \( G \).

  The downside of the latter approach is that notions such as the \hyperref[def:multi_valued_function/image]{image}, \hyperref[def:multi_valued_function/range]{range} and \hyperref[def:multi_valued_function/inverse]{inverse} of the multi-valued function have a very different and much less useful meaning and notions such as \hyperref[def:multi_valued_function/endofunction]{endofunctions} cannot even be defined.
\end{remark}

\begin{proposition}\label{thm:function_image_preimage_composition}
  For any function \( f: A \to B \) we have
  \begin{thmenum}
    \thmitem{thm:function_image_preimage_composition/preimage_of_image} If \( X \subseteq A \), then \( X \subseteq f^{-1}[f[X]] \) with equality holding if \( f \) is injective.
    \thmitem{thm:function_image_preimage_composition/image_of_preimage} If \( Y \subseteq B \), then \( f[f^{-1}[Y]] \subseteq Y \) with equality holding if \( f \) is surjective.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:function_image_preimage_composition/preimage_of_image} If \( x \in X \), clearly \( x \in f^{-1}(f(x)) \). Thus,
  \begin{equation*}
    X \subseteq f^{-1}[f[X]].
  \end{equation*}

  Now suppose that \( f \) is injective and let \( x \in f^{-1}[f[X]] \). There exists some \( y \in f[X] \) such that \( x \in f^{-1}(y) \) and some \( z \in X \) such that \( y = f(z) \). Since \( f \) is injective and \( f(x) = y = f(z) \), it follows that \( x = z \) and thus \( x \in X \). Since \( x \) was chosen arbitrarily from \( f^{-1}[f[X]] \), we conclude that
  \begin{equation*}
    f^{-1}[f[X]] \subseteq X.
  \end{equation*}

  \SubProofOf{thm:function_image_preimage_composition/image_of_preimage} If \( y \in f[f^{-1}[Y]] \), there exists some \( x \in f^{-1}[Y] \) such that \( f(x) = y \). Furthermore, there also exists some \( t \in Y \) such that \( x \in f^{-1}(t) \). Hence, \( y = f(x) = t \) and \( y \in Y \). Therefore,
  \begin{equation*}
    f[f^{-1}[Y]] \subseteq Y.
  \end{equation*}

  Now suppose that \( f \) is surjective and let \( y \in Y \). Then from surjectivity if follows that there exists some \( x \in X \) such that \( f(x) = y \). Hence, \( x \in f^{-1}(y) \) and \( y = f(x) \in f[f^{-1}(y)] \). Since \( y \) was chosen arbitrarily from \( Y \), we conclude that
  \begin{equation*}
    Y \subseteq f[f^{-1}[Y]].
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:def:function_image}
  \hyperref[def:multi_valued_function/set_value]{Images of sets} under \( f: A \to B \) have the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:function_image/monotonicity} If \( A_1 \subseteq A_2 \), then \( f[A_1] \subseteq f[A_2] \).

    \thmitem{thm:def:function_image/union} For any \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \seq{ A_k }_{k \in \mscrK} \subseteq A \) of subsets of \( A \) we have the equality
    \begin{equation}\label{eq:thm:def:function_image/union}
      f\bracks*{ \bigcup_{k \in \mscrK} A_k } = \bigcup_{k \in \mscrK} f[A_k].
    \end{equation}

    \thmitem{thm:def:function_image/intersection} For any indexed family \( \seq{ A_k }_{k \in \mscrK} \) of subsets of \( A \) we have the inclusion
    \begin{equation}\label{eq:thm:def:function_image/intersection}
      f\bracks*{ \bigcap_{k \in \mscrK} A_k } \subseteq \bigcap_{k \in \mscrK} f[A_k].
    \end{equation}

    Equality in \eqref{eq:thm:def:function_image/intersection} holds if \( f \) is \hyperref[def:function_invertibility/injective]{injective}. If \( f \) is not injective, for example if both \( A \) and \( B \) are nonempty, \( A_1 \) and \( A_2 \) are disjoint subsets of \( A \) and \( f[A_1] = f[A_2] = B \), then
    \begin{equation*}
      f[A_1 \cap A_2] = f[\varnothing] = \varnothing \subsetneq f[A_1] \cap f[A_2] = B.
    \end{equation*}

    \thmitem{thm:def:function_image/difference} For any two subsets \( A_1 \) and \( A_2 \) of \( A \) we have the inclusion
    \begin{equation}\label{eq:thm:def:function_image/difference}
      f[A_1] \setminus f[A_2] \subseteq f[A_1 \setminus A_2].
    \end{equation}

    Equality in \eqref{eq:thm:def:function_image/difference} holds if \( f \) is injective. If \( f \) is not injective, for example if \( A_1 \subsetneq A_2 \), but \( f[A_1] = f[A_2] \), then
    \begin{equation*}
      f[A_1] \setminus f[A_2] = \varnothing \subsetneq f[A_1 \setminus A_2].
    \end{equation*}
  \end{thmenum}

  Compare this result to the more well-behaved properties of \hyperref[thm:def:function/preimage]{preimages} described in \fullref{thm:def:function_preimage}.
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:function_image/monotonicity} If \( x \in A_1 \), then \( x \in A_2 \) and hence \( f(x) \in f[A_2] \). Therefore, \( f[A_1] \subseteq f[A_2] \).

  \SubProofOf{thm:def:function_image/union} If \( x_0 \in A_{k_0} \) for some \( k_0 \in \mscrK \), clearly
  \begin{equation*}
    f(x_0) \in f[A_{k_0}] \subseteq \bigcup_{k \in \mscrK} f[A_k].
  \end{equation*}

  Therefore,
  \begin{equation*}
    f\bracks*{ \bigcup_{k \in \mscrK} A_k } \subseteq \bigcup_{k \in \mscrK} f[A_k].
  \end{equation*}

  Conversely, if \( y_0 \in f[A_{k_0}] \) for some \( k_0 \in \mscrK \), by \fullref{thm:def:function_image/monotonicity} obviously
  \begin{equation*}
    y_0 \in f\bracks*{ \bigcup_{k \in \mscrK} A_k }.
  \end{equation*}

  Therefore,
  \begin{equation*}
    f\bracks*{ \bigcup_{k \in \mscrK} A_k } \supseteq \bigcup_{k \in \mscrK} f[A_k].
  \end{equation*}

  Hence, \eqref{eq:thm:def:function_image/union} holds.

  \SubProofOf{thm:def:function_image/intersection} If \( x_0 \) belongs to \( \bigcap_{k \in \mscrK} A_k \), then \( x_0 \) belongs to \( A_k \) for all \( k \in \mscrK \). It follows that \( f(x_0) \) belongs to \( f[A_k] \) for all \( k \in \mscrK \) and hence to their intersection. Therefore, the inclusion \eqref{eq:thm:def:function_image/intersection} holds.

  Now suppose that \( f \) is injective. Let \( y_0 \) be a point in the intersection \( \bigcap_{k \in \mscrK} f[A_k] \). We thus have \( y_0 \in f[A_k] \) for all \( k \in \mscrK \). Since \( f \) is injective, for each \( k \in \mscrK \) there exists a unique \( x_k \in A_k \) such that \( f(x_k) = y_0 \). Again because of injectivity of \( f \), all these elements are equal because \( f(x_k) = f(x_m) = y_0 \) for \( k, m \in \mscrK \). Hence,
  \begin{equation*}
    y_0 \in f\bracks*{ \bigcap_{k \in \mscrK} A_k }.
  \end{equation*}

  Therefore, the reverse inclusion in \eqref{eq:thm:def:function_image/intersection} holds if \( f \) is injective.

  \SubProofOf{thm:def:function_image/difference} If \( f[A_1] \setminus f[A_2] \) is empty, \eqref{eq:thm:def:function_image/difference} obviously holds. Suppose that it is nonempty and let \( y_0 \in f[A_1] \setminus f[A_2] \).

  Then there exists a point \( x_0 \in A_1 \) such that \( f(x_0) = y_0 \). It cannot be that \( x_0 \in A_2 \) because otherwise \( y_0 = f(x_0) \in f[A_2] \), which would contradict our choice of \( y_0 \). Hence, \( x_0 \in A_1 \setminus A_2 \) and \( y_0 \in f(A_1 \setminus A_2) \).

  Since \( y_0 \) was chosen arbitrarily, we conclude that the inclusion \eqref{eq:thm:def:function_image/difference} holds.

  Conversely, suppose that \( f \) is injective. If \( f(A_1 \setminus A_2) \) is empty, by \eqref{eq:thm:def:function_image/difference} the set \( f[A_1] \setminus f[A_2] \) is also empty and the converse holds.

  Now suppose that it is nonempty and let \( y_0 \in f(A_1 \setminus A_2) \). Then there exists a point \( x_0 \in A_1 \setminus A_2 \) such that \( f(x_0) = y_0 \). Furthermore, since \( f \) is injective, \( x_0 \) is the only preimage of \( y_0 \) and hence \( f(x_0) \in f[A_1] \setminus f[A_2] \), which proves the reverse inclusion in \eqref{eq:thm:def:function_image/difference}.
\end{proof}

\begin{proposition}\label{thm:def:function_preimage}
  Function \hyperref[thm:def:function/preimage]{preimages} have the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:function_preimage/monotonicity} If \( B_1 \subseteq B_2 \), then \( f^{-1}[B_1] \subseteq f^{-1}[B_2] \).

    \thmitem{thm:def:function_preimage/union} For any \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \seq{ B_k }_{k \in \mscrK} \subseteq B \) of subsets of \( B \) we have the equality
    \begin{equation}\label{eq:thm:def:function_preimage/union}
      f^{-1}\bracks*{ \bigcup_{k \in \mscrK} B_k } = \bigcup_{k \in \mscrK} f^{-1}[B_k].
    \end{equation}

    \thmitem{thm:def:function_preimage/intersection} For any indexed family \( \seq{ B_k }_{k \in \mscrK} \) of subsets of \( B \) we have the equality
    \begin{equation}\label{eq:thm:def:function_preimage/intersection}
      f^{-1}\bracks*{ \bigcap_{k \in \mscrK} B_k } = \bigcap_{k \in \mscrK} f^{-1}[B_k].
    \end{equation}

    \thmitem{thm:def:function_preimage/difference} For any two subsets \( B_1 \) and \( B_2 \) of \( B \) we have the equality
    \begin{equation}\label{eq:thm:def:function_preimage/difference}
      f^{-1}[B_1] \setminus f^{-1}[B_2] = f^{-1}[B_1 \setminus B_2].
    \end{equation}
  \end{thmenum}

  Compare this result to the less well-behaved properties of images described in \fullref{thm:def:function_image}.
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:function_image/monotonicity} Analogous to \fullref{thm:def:function_image/monotonicity}.

  \SubProofOf{thm:def:function_image/union} Analogous to \fullref{thm:def:function_image/union}.

  \SubProofOf{thm:def:function_image/intersection} If \( y_0 \) belongs to \( \bigcap_{k \in \mscrK} B_k \), then \( y_0 \) belongs to \( B_k \) for all \( k \in \mscrK \). It follows that \( f(y_0) \subseteq f^{-1}[B_k] \) for all \( k \in \mscrK \) and hence it is also a subset of their intersection. Therefore,
  \begin{equation*}
    f^{-1} \bracks*{ \bigcap_{k \in \mscrK} B_k } \subseteq \bigcap_{k \in \mscrK} f^{-1}[B_k].
  \end{equation*}

  Conversely, if \( x_0 \in \bigcap_{k \in \mscrK} f^{-1}[B_k] \), it belongs to \( f^{-1}[B_k] \) for all \( k \in \mscrK \). Clearly then \( f(x_0) \in B_k \) for all \( k \in \mscrK \) and thus \( f(x_0) \in \bigcap_{k \in \mscrK} B_k \). Hence, by \fullref{thm:def:function_preimage/monotonicity} we have
  \begin{equation*}
    f^{-1}(f(x_0))
    \subseteq
    f^{-1}\bracks*{ \bigcap_{k \in \mscrK} B_k }.
  \end{equation*}

  Since \( x_0 \in f^{-1}(f(x_0)) \),
  \begin{equation*}
    x_0 \in f^{-1}\bracks*{ \bigcap_{k \in \mscrK} B_k }.
  \end{equation*}

  Since \( x_0 \) was chosen arbitrarily from \( \bigcap_{k \in \mscrK} f^{-1}[B_k] \), we can conclude that
  \begin{equation*}
    \bigcap_{k \in \mscrK} f^{-1}[B_k] \in f^{-1}\bracks*{ \bigcap_{k \in \mscrK} B_k }.
  \end{equation*}

  Hence, \eqref{eq:thm:def:function_preimage/intersection} holds.

  \SubProofOf{thm:def:function_preimage/difference} If \( y_0 \in B_1 \setminus B_2 \), there exists a point \( x_1 \in B_1 \) such that \( f(x_1) = y_0 \). Aiming at a contradiction, suppose that there exists a point \( x_2 \in f^{-1}[B_2] \) such that \( f(x_2) = y_0 \). Then \( y_0 = f(x_1) = f(x_2) \) implies that \( f^{-1}(y_0) \subseteq f^{-1}[B_1] \cap f^{-1}[B_2] \). \Fullref{thm:def:function_preimage/intersection} then in turn implies that \( f^{-1}(y_0) \subseteq f^{-1}[B_1 \cap B_2] \) and hence by \fullref{thm:def:function_image/monotonicity}
  \begin{equation*}
    y_0 = f(f^{-1}(y_0)) \in f[f^{-1}[B_1 \cap B_2]] = B_1 \cap B_2,
  \end{equation*}
  which contradicts our choice of \( y_0 \). Since the choice of \( y_0 \in B_1 \setminus B_2 \), \( x_1 \in f^{-1}(y_0) \cap B_1 \) and \( x_2 \in f^{-1}(y_0) \cap B_2 \) was arbitrary, the obtained contradiction shows that
  \begin{equation*}
    f^{-1}(B_1 \setminus B_2) \subseteq f^{-1}[B_1] \setminus f^{-1}[B_2].
  \end{equation*}

  Conversely, we have
  \begin{equation*}
    f(f^{-1}[B_1] \setminus f^{-1}[B_2])
    \reloset {\eqref{eq:thm:def:function_image/difference}} \subseteq
    f(f^{-1}(B_1 \setminus B_2))
    \reloset {\ref{thm:function_image_preimage_composition/image_of_preimage}} \subseteq
    B_1 \setminus B_2.
  \end{equation*}

  Hence,
  \begin{equation*}
    f^{-1}[B_1] \setminus f^{-1}[B_2]
    \reloset {\ref{thm:function_image_preimage_composition/preimage_of_image}} \subseteq
    f^{-1}\parens[\Big]{ f\parens[\Big]{ f^{-1}[B_1] \setminus f^{-1}[B_2] } }
    \reloset {\eqref{thm:def:function_preimage/monotonicity}} \subseteq
    f^{-1}(B_1 \setminus B_2).
  \end{equation*}
\end{proof}

\begin{theorem}[Recursion theorem]\label{thm:omega_recursion}\mcite[73]{Enderton1977Sets}
  Let \( A \) be a nonempty set. Suppose that we are given some member \( a_0 \) of \( A \) and some transformation \( T: A \to A \). Then there exists a \hyperref[def:sequence]{sequence} \( f: \omega \to A \) such that
  \begin{itemize}
    \item \( f(n) = a_0 \).
    \item For every \( n \in \omega \) we have \( f(\op{succ}(n)) = T(f(n)) \).
  \end{itemize}

  Note that we do not yet use the notation \( n + 1 \) because we will use this theorem to define addition in the first place.

  See \fullref{rem:natural_number_recursion} for a simpler and more conventional notation for recursion on \( \omega \).
\end{theorem}
\begin{proof}
  Let \( G \subseteq \pow(\omega \times A) \) be the set of all \hyperref[def:partial_function]{partial single-valued functions} \( g: \omega \to A \) such that
  \begin{itemize}
    \item If \( g \) is defined at \( \varnothing \), then \( g(\varnothing) = a_0 \).
    \item For every \( n \in \omega \), if \( g \) is defined at \( \op{succ}(n) \), then \( g \) is also defined at \( n \) and
    \begin{equation*}
      g(\op{succ}(n)) = T(f(n)).
    \end{equation*}
  \end{itemize}

  Clearly \( G \) is nonempty because the function \( \set{ (\varnothing, a_0) } \) belongs to \( G \).

  The conditions imposed on the functions in \( G \) ensure that every function is defined in some \hyperref[def:partially_ordered_set_interval/ray]{initial segment} of the natural numbers. A more obvious approach is to require \( g \) to be defined at \( \op{succ}(n) \) if it is defined at \( n \), however we are trying to prove that such a function exists in the first place.

  Define \( f \coloneqq \bigcup G \). At this point \( f \) is a \hyperref[def:multi_valued_function]{multi-valued function}. We must now show that \( f \) has all the properties that we want.

  \SubProofOf[def:multi_valued_function/total]{totality} First, we will use \fullref{thm:omega_induction} to show that \( f \) is total. Clearly \( \varnothing \in \dom f \).

  Now fix \( n \in \dom f \). Then there exists a function \( g \in G \) defined at \( n \).

  \begin{itemize}
    \item If \( g \) is also defined at \( \op{succ}(n) \), this directly proves that \( \op{succ}(n) \in \dom f \).
    \item If \( g \) is not defined at \( \op{succ}(n) \), consider
    \begin{equation*}
      \widehat g \coloneqq g \cup \set{ (\op{succ}(n), T(g(n)) }.
    \end{equation*}

    The function \( \widehat g \) is again a single-valued partial function and thus it belongs to \( G \), hence \( \op{succ}(n) \in \dom f \).
  \end{itemize}

  Therefore, \fullref{thm:omega_induction} allows us to conclude \( f: \omega \multto A \) is a total multi-valued function.

  \SubProofOf[def:function]{single-valuedness} Now that we know that \( f \) is total, we will prove that it is single-valued and thus is a function in the usual sense of the term.

  Clearly \( f \) is single-valued at \( \varnothing \).

  Now suppose that \( f \) is single-valued at \( n \). Since \( f \) is total, there exist at least one partial function in \( G \) that is defined at \( \op{succ}(n) \), from which it follows that it is also defined at \( n \).  Let \( g \) and \( h \) both be such (single-valued partial) functions.

  Then
  \begin{equation*}
    g(\op{succ}(n)) = T(g(n)) = T(f(n)) = T(h(n)) = h(\op{succ}(n)),
  \end{equation*}
  hence \( g \) and \( h \) coincide at \( \op{succ}(n) \), which in turn implies that \( f \) is single-valued at \( \op{succ}(n) \).

  Therefore, \fullref{thm:omega_induction} allows us to conclude that \( f \) is a single-valued total function.

  \SubProofOf[def:function]{uniqueness} Now that it is clear that \( f \) satisfies the theorem, we must verify that it is unique.

  Suppose that \( f_1 \) and \( f_2 \) both satisfy the theorem. Clearly \( \varnothing \in H \). Fix some \( n \neq \varnothing \) and suppose that \( f_1(n) = f_2(n) \). Then
  \begin{equation*}
    f_2(\op{succ}(n)) = T(f_1(n)) = T(f_2(n)) = f_2(\op{succ}(n)).
  \end{equation*}

  Therefore, \fullref{thm:omega_induction} allows us to conclude that \( f_1 = f_2 \). So there is at most one function that satisfies the theorem and we have already shown that \( f \) is such a function.
\end{proof}

\begin{definition}\label{def:omega_operations}
  We will use \fullref{thm:omega_recursion} for defining arithmetic operations for natural numbers. There constructions will be more elaborate than the basic recursive sequences defined in e.g. \fullref{thm:banach_fixed_point_theorem}.

  \begin{thmenum}
    \thmitem{def:omega_operations/addition} We will represent the addition operation \( \oplus \) as follows: fix the first summand \( n \) and then define a function \( \oplus_n: \omega \to \omega \) such that \( k = \oplus_n(m) \) gives us \( n \oplus m = k \). This is a particular instance of \hyperref[def:function/currying]{currying}.

    Fix \( n \in \omega \), let \( A = \omega \) and define
    \begin{equation*}
      \begin{aligned}
        &T_n: \omega \to \omega \\
        &T_n(k) \coloneqq \op{succ}(k).
      \end{aligned}
    \end{equation*}

    Now we use the initial point \( a_n = n \) to construct the function \( \oplus_n \).

    Define the addition function \( \oplus: \omega \times \omega \to \omega \) via its graph
    \begin{equation*}
      \set{ ((n, m), \oplus_n(m)) \given n, m \in \omega }.
    \end{equation*}

    \thmitem{def:omega_operations/multiplication} We define natural number multiplication analogously. For each \( n \in \omega \), define \( \odot_n \) via
    \begin{equation*}
      \begin{aligned}
        &T_n: \omega \to \omega \\
        &T_n(k) \coloneqq k \oplus n.
      \end{aligned}
    \end{equation*}
    and \( a_n = 0 \) and then define \( \odot: \omega \times \omega \to \omega \) via its graph
    \begin{equation*}
      \set{ ((n, m), \odot_n(m)) \given n, m \in \omega }.
    \end{equation*}
  \end{thmenum}
\end{definition}

\begin{theorem}\label{thm:omega_is_model_of_pa}
  The \hyperref[thm:smallest_inductive_set_existence]{smallest inductive set} \( \omega \) is a model of \hyperref[def:peano_arithmetic]{Peano arithmetic} with the following interpretation:
  \begin{thmenum}
    \thmitem{thm:omega_is_model_of_pa/zero} \hyperref[def:peano_arithmetic/zero]{Zero} is interpreted as \( \varnothing \).

    \thmitem{thm:omega_is_model_of_pa/succ} The \hyperref[def:peano_arithmetic/succ]{successor} operation \( s \) is interpreted as \( \op{succ} \).

    \thmitem{thm:omega_is_model_of_pa/plus} \hyperref[def:peano_arithmetic/plus]{Addition} is interpreted by the \( \oplus \) function given in \fullref{def:omega_operations/addition}.

    \thmitem{thm:omega_is_model_of_pa/mult} Similarly, \hyperref[def:peano_arithmetic/mult]{multiplication} is interpreted by \( \odot \) from \fullref{def:omega_operations/multiplication}.
  \end{thmenum}

  This is an extension of \fullref{thm:omega_is_model_of_pa_without_operations}.
\end{theorem}
\begin{proof}
  We have already shown in \fullref{thm:omega_is_model_of_pa_without_operations} that \( \omega \) satisfies the axioms \eqref{eq:def:peano_arithmetic/PA1}-\eqref{eq:def:peano_arithmetic/PA3}.

  \SubProofOf{eq:def:peano_arithmetic/PA4} For each \( n \in \omega \) the starting condition (i.e. \( m = \varnothing \)) in \fullref{def:omega_operations/addition} implies that \( n \oplus \varnothing = n \).

  \SubProofOf{eq:def:peano_arithmetic/PA5} For each \( n \in \omega \) the transformation \( T_n \) in \fullref{def:omega_operations/addition} is defined, so that if \( \oplus_n(m) = k \), then \( \oplus_n(\op{succ}(m)) = \op{succ}(k) \).

  It follows that \( n \oplus \op{succ}(m) = \op{succ}(n \oplus m) \) for all \( n, m \in \omega \).

  \SubProofOf{eq:def:peano_arithmetic/PA6} Analogously to \( \oplus \), the starting condition in \fullref{def:omega_operations/multiplication} implies that \( n \odot \varnothing = \varnothing \) for every \( n \in \omega \).

  \SubProofOf{eq:def:peano_arithmetic/PA7} Analogously to \( \oplus \), for each \( n \in \omega \) the transformation \( T_n \) in \fullref{def:omega_operations/multiplication} is defined, so that if \( \odot_n(m) = k \), then \( \odot_n(\op{succ}(m)) = k \oplus n \).

  It follows that \( n \odot \op{succ}(m) = n \odot m + n \) for all \( n, m \in \omega \).
\end{proof}

\begin{remark}\label{rem:natural_number_recursion}
  With the availability of natural numbers, instead of the tedious constructions in \fullref{def:omega_operations/addition}, we can use a more conventional notation when applying \fullref{thm:omega_recursion}.

  As an example, we can define the Fibonacci sequence. The sequence is motivated by the problem in \fullref{ex:fibonacci_rabbits}. In the notation of \fullref{thm:omega_recursion}, we can define the Fibonacci sequence as follows:
  \begin{equation*}
    \begin{aligned}
      &T: \BbbN \times \BbbN \to \BbbN \times \BbbN \\
      &T(a, b) \coloneqq (b, a + b)
    \end{aligned}
  \end{equation*}
  and
  \begin{equation*}
    a_0 \coloneqq (0, 1).
  \end{equation*}

  The recursion theorem gives us a sequence of pairs
  \begin{equation*}
    \underbrace{(0, 1)}_{a_0}, \underbrace{(1, 1)}_{a_1}, \underbrace{(1, 2)}_{a_2}, \underbrace{(2, 3)}_{a_3}, \underbrace{(3, 5)}_{a_4}, \underbrace{(5, 8)}_{a_5}, \underbrace{(8, 13)}_{a_6}, \ldots
  \end{equation*}

  The pairs are only a technicality because otherwise we would not be able to define the sequence \( \seq{ a_k }_{k=1}^\infty \).

  By taking the second element of each pair, we obtain the sequence
  \begin{equation*}
    \underbrace{1}_{b_0}, \underbrace{1}_{b_1}, \underbrace{2}_{b_2}, \underbrace{3}_{b_3}, \underbrace{5}_{b_4}, \underbrace{8}_{b_5}, \underbrace{13}_{b_6}, \ldots
  \end{equation*}

  In order to obtain the Fibonacci sequence, we must prefix the sequence \( \seq{ b_k }_{k=0}^\infty \) with \( 0 \).

  This is undoubtedly much more complicated than writing
  \begin{equation*}
    b_k \coloneqq \begin{cases}
      0,                &k = 0, \\
      1,                &k = 1, \\
      b_{k-1} + b_{k-2} &k > 1.
    \end{cases}
  \end{equation*}

  To see that the latter notation is merely syntax sugar, note that the other sequence can be written as
  \begin{equation*}
    a_k \coloneqq \begin{cases}
      (0, 1),     &k = 0 \\
      T(a_{k-1}), &k > 1.
    \end{cases}
  \end{equation*}
\end{remark}


\hypertarget{haskell-typeclasses}{%
\subsection{Haskell Typeclasses}\label{haskell-typeclasses}}

\begin{itemize}
\tightlist
\item
  Equality types:
\end{itemize}

\begin{lstlisting}[language=Haskell]
(==) , (/=) :: a -> a -> Bool
\end{lstlisting}

\begin{itemize}
\tightlist
\item
  Ordered types (based on Eq):
\end{itemize}

\begin{lstlisting}[language=Haskell]
( <) , ( <=) , ( >) , ( >=) :: a -> a -> Bool
min , max :: a -> a -> a
\end{lstlisting}

\begin{itemize}
\tightlist
\item
  Showable types:
\end{itemize}

\begin{lstlisting}[language=Haskell]
show :: a -> String
\end{lstlisting}

\begin{itemize}
\tightlist
\item
  Readable types:
\end{itemize}

\begin{lstlisting}[language=Haskell]
read :: String -> a
\end{lstlisting}

\begin{itemize}
\tightlist
\item
  Numeric types:
\end{itemize}

\begin{lstlisting}[language=Haskell]
(+) , ( -) , (*) :: a -> a -> a
negate , abs , signum :: a -> a
fromInteger :: Integer -> a
\end{lstlisting}

\begin{itemize}
\tightlist
\item
  Integral types (somewhat simplified)(Based on Num and Ord)
\end{itemize}

\begin{lstlisting}[language=Haskell]
quot , rem , div , mod :: a -> a -> a
quotRem , divMod :: a -> a -> (a , a )
\end{lstlisting}

\begin{itemize}
\tightlist
\item
  Fractional types (based on Num)
\end{itemize}

\begin{lstlisting}[language=Haskell]
(/) :: a -> a -> a
recip :: a -> a
fromRational :: Rational -> a
\end{lstlisting}

\hypertarget{instances-of-these-classes}{%
\subsubsection{Instances of these
classes}\label{instances-of-these-classes}}

\begin{itemize}
\tightlist
\item
  Booleans, Characters:

  \begin{itemize}
  \tightlist
  \item
    Eq , Ord , Show , Read
  \end{itemize}
\item
  Int, Integer:

  \begin{itemize}
  \tightlist
  \item
    Eq , Ord , Show , Read , Num , Integral
  \end{itemize}
\item
  Float, Double:

  \begin{itemize}
  \tightlist
  \item
    Eq , Ord , Show , Read , Num , Fractional
  \end{itemize}
\item
  Tuples, Lists:

  \begin{itemize}
  \tightlist
  \item
    Eq , Ord , Show , Read
  \end{itemize}
\end{itemize}

\hypertarget{examples}{%
\subsubsection{Examples}\label{examples}}

\begin{lstlisting}[language=Haskell]
2 ==> Int
2 + 2 ==> Int
2 :: Int ==> Int
2 :: Float ==> Float
(2 + 2) :: Double ==> Double
2.0 ==> Fractional p => p
2.0 :: Int ==> error
2 + 2.0 ==> Fractional a => a
(2 :: Int ) + (2 :: Double) ==> error (Couldn't match expected type 'Int' with actual type 'Double')
(2 :: Int ) + 2 ==> Int
(2 , 2) ==> (Num a, Num b) => (a, b)
[2 , 2] ==> Num a => [a]
[2 , 2.0] ==> Fractional a => [a]
[2 :: Float , 2 :: Double] ==> error
\end{lstlisting}