\subsection{Zermelo-Fraenkel set theory}\label{subsec:zermelo_fraenkel_set_theory}

\begin{definition}\label{def:choice_function}
  Let \( \mscrA \) be a (potentially empty) family of nonempty sets. A \term{choice function} on \( \mscrA \) is a (total) \hyperref[def:function]{function} \( c: \mscrA \to \bigcup \mscrA \) such that \( c(A) \in A \) for all \( A \in \mscrA \).

  This is formally the same as a tuple defined in \fullref{def:cartesian_product/tuple}.

  A choice function on \( \mscrA \) \enquote{chooses} an element out of each member of \( \mscrA \). We sometimes have a canonical choice function, for example in \fullref{thm:representatives_in_univariate_polynomial_quotient_set}, however for general \hyperref[def:equivalence_relation/quotient]{quotient sets} the existence of a choice function is not by any means obvious.

  The existence of a choice function for family of nonempty sets is an important axiom of \logic{ZFC} --- see \fullref{def:zfc/choice}.
\end{definition}

\begin{definition}\label{def:zfc}
  The \hyperref[def:first_order_theory]{first-order theory} commonly abbreviated as \term{\logic{ZFC}} is based on the same language as \hyperref[def:naive_set_theory]{na\"ive set theory}, but with different axioms. The three letters refer to:
  \begin{itemize}
    \item \hi{Z}ermelo, who formulated the entire theory except for the \hyperref[def:zfc/replacement]{axiom schema of replacement} and the \hyperref[def:zfc/foundation]{axiom of foundation}.
    \item\mcite[sec. 63.8]{OpenLogicFull} \hi{F}raenkel, who simultaneously with Skolem reformulated the theory within first-order logic while also introducing the axiom schema of replacement.
    \item The \hyperref[def:zfc/choice]{axiom of \hi{c}hoice}, which is part of Zermelo's original theory, but is controversial enough to attract special attention --- see \fullref{thm:axiom_of_choice_equivalences}.
  \end{itemize}

  We are usually only interested in either \logic{ZFC}, which include all axioms listed in this definition. If we wish to avoid the axiom of choice --- for example when proving the equivalences in \fullref{thm:axiom_of_choice_equivalences} --- we instead use \logic{ZF}, which excludes the axiom of choice. The abbreviation of the latter theory is inaccurate historically, but is nevertheless established.

  If we wish to instead exclude the axiom of replacement, we obtain the theory \logic{Z}, however without context it is unclear whether the axiom of choice is included in \logic{Z} or not.

  See \fullref{thm:zfc_existence_theorems} for proofs of existence of common sets.

  The full list of axioms is:
  \begin{thmenum}
    \thmitem{def:zfc/extensionality} The \term{axiom of extensionality}, as defined in \fullref{def:naive_set_theory/extensionality}. This is also the only axiom of the theory that does not deal with existence.

    \thmitem{def:zfc/specification}\mcite[sec. 62.2]{OpenLogicFull} The \term{axiom schema of specification}, also known as the axiom schema of \term{separation} or of \term{restricted comprehension}, states that given a set \( A \), any formula defines a subset of \( A \). For each formula \( \varphi \) containing neither \( \tau \) nor \( \sigma \) as free variables, the following is an axiom:
    \begin{equation}\label{eq:def:zfc/specification}
      \qforall \sigma \qexists \tau \qforall \xi (\xi \in \tau \leftrightarrow \varphi \wedge \xi \in \sigma).
    \end{equation}

    As explained in \fullref{def:naive_set_theory/unrestricted_comprehension} and \fullref{def:set_builder_notation}, this set may depend on parameters, which are other sets. We must formally take the \hyperref[thm:implicit_universal_quantification]{universal closure} of this set to quantify over all possible values for the parameters.

    Compare this axiom to \hyperref[def:naive_set_theory/unrestricted_comprehension]{unrestricted comprehension}. Informally, this axiom can be obtained by taking the result of unrestricted comprehension and intersecting it with some set \( A \). As mentioned in \fullref{def:set_builder_notation}, in set-builder notation such a set is usually denoted by
    \begin{equation*}
      \set{ x \in B \given \varphi\Bracks{x, u_1, \ldots, u_n} }.
    \end{equation*}

    Unlike unrestricted comprehension some definable subset of the universe in the metatheory no longer have a corresponding set within the object logic.

    \thmitem{def:zfc/power_set}\mcite[sec. 62.5]{OpenLogicFull} The \term{axiom of power sets} states that every set has a corresponding \hyperref[def:basic_set_operations/power_set]{power set}. Symbolically,
    \begin{equation}\label{eq:def:zfc/power_set}
      \qforall \tau \qexists \sigma \ref{eq:def:basic_set_operations/power_set/predicate}[\sigma, \tau].
    \end{equation}

    \thmitem{def:zfc/union}\mcite[sec. 62.3]{OpenLogicFull} The \term{axiom of unions} states that for every set there exists another set that is its \hyperref[def:basic_set_operations/union]{union}. Symbolically,
    \begin{equation}\label{eq:def:zfc/union}
      \qforall \tau \qexists \sigma \ref{eq:def:basic_set_operations/union/predicate}[\sigma, \tau].
    \end{equation}

    \thmitem{def:zfc/pairing}\mcite[sec. 62.4]{OpenLogicFull} The \term{axiom of pairing} states that for any sets \( A \) and \( B \) there exists another set that contains exactly \( A \) and \( B \). This is the set \( \set{ A, B } \) in set-builder notation. Symbolically,
    \begin{equation}\label{eq:def:zfc/pairing}
      \qforall \tau \qforall \sigma \qexists \rho \qforall \xi \parens[\Big]{ \xi \in \rho \leftrightarrow ( \xi \doteq \tau \vee \xi \doteq \sigma) }.
    \end{equation}

    \thmitem{def:zfc/infinity}\mcite[sec. 62.6]{OpenLogicFull} The \term{axiom of infinity} states that an \hyperref[def:inductive_set]{inductive set} exists. Symbolically,
    \begin{equation}\label{eq:def:zfc/infinity}
      \qexists \tau \ref{eq:def:inductive_set/predicate}[\tau].
    \end{equation}

    This axiom is a simple and convenient way to state that infinite sets exist. Without it, we can only deal with finite sets unless we include some other axiom to replace it.

    \thmitem{def:zfc/choice}\mcite[sec. 69.4]{OpenLogicFull} The \term{axiom of choice} states that a \hyperref[def:choice_function]{choice function} exists for any family of nonempty sets. To state the axiom via a formula, we will avoid functions and only state it in terms of the image of the choice function. That is, we will formulate that for each family \( \mscrA \) of nonempty sets there exists a set \( B \) such that \( A \cap B \) is a singleton set for each \( A \in \mscrA \). Symbolically,
    \begin{equation}\label{eq:def:zfc/choice}
      \qforall \tau \parens[\Bigg]
        {
          \parens[\Big]{ \qforall {\xi \in \tau} \neg \ref{eq:def:empty_set/predicate}[\xi] }
          \rightarrow
          \parens[\Big]{ \qexists \sigma \qforall {\xi \in \tau} \qExists {\eta \in \sigma} \eta \in \xi }
        }
    \end{equation}
    where we have used the convention regarding existence and uniqueness described in \fullref{rem:first_order_formula_conventions/exists_unique}.

    See \fullref{thm:axiom_of_choice_equivalences} for more statements equivalent to this axiom.

    \thmitem{def:zfc/replacement}\mcite[sec. 63.7]{OpenLogicFull} The \term{axiom schema of replacement} roughly states that every \hyperref[rem:function_definition]{mapping} that is definable via a formula of \logic{ZFC} is a function. As we have done for the \hyperref[def:zfc/choice]{axiom of choice}, we only formulate the axiom via the image of the function. More concretely, given a formula \( \varphi \) not containing \( \tau \) nor \( \sigma \) as free variables, the following is an axiom:
    \begin{equation}\label{eq:def:zfc/replacement}
      \qforall \tau \parens[\Bigg]
        {
          \parens[\Big]{ \qforall {\xi \in \tau} \qExists \eta \varphi }
          \rightarrow
          \parens[\Big]{ \qexists \sigma \qforall \eta \parens[\Big]{ \eta \in \sigma \leftrightarrow \qexists {\xi \in \tau} \varphi } }
        }.
    \end{equation}

    As is the case with the \hyperref[def:zfc/specification]{axiom schema of specification}, the formula \( \varphi \) may depend on parameters, in which case we use its \hyperref[thm:implicit_universal_quantification]{universal closure}.

    This axiom is useful in cases where it is impossible or at least difficult to construct a function, for example in \fullref{thm:zfc_existence_theorems/indexed_family} or \fullref{thm:hartogs_lemma}. This is the case, in general, when dealing with \hyperref[thm:zfc_existence_theorems/indexed_family]{indexed families} rather than \hyperref[def:function]{functions}.

    This is the axiom that makes \logic{ZFC} require large models --- see \fullref{thm:cumulative_hierarchy_model_of_zfc}.

    \thmitem{def:zfc/foundation}\mcite[sec. 64.4]{OpenLogicFull} The \term{axiom of foundation} states that every nonempty set contains a member disjoint from the set itself. Symbolically,
    \begin{equation}\label{eq:def:zfc/foundation}
      \qforall \tau \parens[\Big]
        {
          \neg \ref{eq:def:empty_set/predicate}[\tau]
          \rightarrow
          \qexists {\sigma \in \tau} \neg \qexists \xi \parens{ \xi \in \tau \wedge \xi \in \sigma }
        }.
    \end{equation}

    This is a very powerful axiom because it shows that set membership in \logic{ZFC} is well-founded --- see \fullref{thm:set_membership_is_well_founded}. It is equivalent to \fullref{thm:axiom_of_regularity} and is often itself called the \term{axiom of regularity}.
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:zfc_existence_theorems}
  We will now prove that all sets we have considered up until now in \fullref{sec:set_theory} are sets in \hyperref[def:zfc]{\logic{ZFC}}. The uniqueness in all cases follows from the \hyperref[def:zfc/extensionality]{axiom of extensionality}.

  A very fundamental existence result is provided by the fact that we are assuming \hyperref[rem:standard_model_of_set_theory]{standard} and \hyperref[rem:transitive_model_of_set_theory]{transitive} models of \logic{ZFC}. Let \( \mscrV = (V, I) \) be such a model. Then if \( v \in V \) and \( u \in v \), transitivity implies that \( u \in V \). Since the model is also standard, this shows that both \( u \) and \( v \) are sets within the object theory. Thus, if \( A \) is a set within the object theory and if \( B \in A \) within the metatheory, then necessarily \( B \) itself is a set within the object theory.

  For example, \fullref{thm:zfc_existence_theorems/set_of_functions} shows that the set \( \fun(A, B) \) of functions exists within the object theory for any two sets \( A \) and \( B \) in the object theory. Therefore, every single function between \( A \) and \( B \) is a set within the object theory because it is a member of \( \fun(A, B) \).

  With that in mind, we will show the following:

  \begin{thmenum}
    \thmitem{thm:zfc_existence_theorems/subset} If \( A \) is a set, then for any formula \( \varphi \) the set \( \set{ x \in A \given \varphi[x] } \) exists and, furthermore, it is a \hyperref[def:subset]{subset} of \( A \). Only \hyperref[def:first_order_definability]{definable subsets} of \( A \) can be described in this way, however. See \fullref{thm:zfc_existence_theorems/power_set}.

    \thmitem{thm:zfc_existence_theorems/empty_set} There exists a unique \hyperref[def:empty_set]{empty set}, which we denote by \( \varnothing \).

    \thmitem{thm:zfc_existence_theorems/universe} No \hyperref[def:set]{universal set} (set of all sets) exists.

    \thmitem{thm:zfc_existence_theorems/singleton} For every set \( A \), there exists a \hyperref[rem:singleton_sets]{singleton set} \( \set{ A } \) that contains only \( A \).

    \thmitem{thm:zfc_existence_theorems/arbitrary_intersection} For any \hi{nonempty} family \( \mscrA \), the \hyperref[def:basic_set_operations/intersection]{intersection} \( \bigcap \mscrA \) exists.

    \thmitem{thm:zfc_existence_theorems/binary_intersection} For any two sets \( A \) and \( B \), their \hyperref[def:basic_set_operations/intersection]{intersection} \( A \cap B \) exists.

    \thmitem{thm:zfc_existence_theorems/arbitrary_union} For any family \( \mscrA \), the \hyperref[def:basic_set_operations/union]{union} \( \bigcup \mscrA \) exists.

    \thmitem{thm:zfc_existence_theorems/binary_union} For any two sets \( A \) and \( B \), their \hyperref[def:basic_set_operations/union]{union} \( A \cup B \) exists.

    \thmitem{thm:zfc_existence_theorems/difference} For any two sets \( A \) and \( B \), their \hyperref[def:basic_set_operations/difference]{difference} \( A \setminus B \) exists.

    \thmitem{thm:zfc_existence_theorems/power_set} For any set \( A \), its \hyperref[def:basic_set_operations/power_set]{power set} \( \pow(A) \) exists.

    As a consequence, even subsets of \( A \) which are not \hyperref[def:first_order_definability]{definable} exist.

    \thmitem{thm:zfc_existence_theorems/successor} For any set \( A \), its \hyperref[def:ordinal_successor]{successor} \( \op{succ}(A) \) exists.

    \thmitem{thm:zfc_existence_theorems/kuratowski_pair} For any two sets \( A \) and \( B \), their \hyperref[def:cartesian_product/kuratowski_pair]{Kuratowski pair} \( \braket{ A, B } \) exists.

    \thmitem{thm:zfc_existence_theorems/indexed_family} For any sets \( \mscrK \) and \( \mscrA \), any \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \seq{ A_k }_{k \in \mscrK} \subseteq \mscrA \) exists.

    \thmitem{thm:zfc_existence_theorems/cartesian_product} For any indexed family \( \set{ A_k }_{k \in \mscrK} \), its \hyperref[def:cartesian_product]{Cartesian product} \( \prod_{k \in \mscrK} A_k \) exists.

    \thmitem{thm:zfc_existence_theorems/set_of_relations} For any two sets \( A \) and \( B \), the set of all relations between \( A \) and \( B \) exists.

    \thmitem{thm:zfc_existence_theorems/set_of_functions} For any two sets \( A \) and \( B \), the set \hyperref[def:function/set_of_functions]{\( \fun(A, B) \)} exists.

    \thmitem{thm:zfc_existence_theorems/quotient_set} For any set \( A \) and any \hyperref[def:equivalence_relation]{equivalence relation} \( \cong \), the \hyperref[def:equivalence_relation/quotient]{quotient set} \( A / {\cong} \) exists.

    \thmitem{thm:zfc_existence_theorems/function_evaluation} Fix some sets \( A \) and \( B \) and some indexed family of functions \( \seq{ f_k }_{k \in \mscrK} \) where \( f: A \to B_k \) for \( k \in \mscrK \). For any \( x \in A \) the corresponding tuple \( \seq{ f_k(x) }_{k \in \mscrK} \) exists.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:zfc_existence_theorems/subset} This is a trivial consequence of the \hyperref[def:zfc/specification]{axiom schema of specification}.

  \SubProofOf{thm:zfc_existence_theorems/empty_set} As a consequence of the \hyperref[def:zfc/infinity]{axiom of infinity}, there exists at least one inductive set. Let \( A \) be an inductive set. Then from the \hyperref[def:zfc/specification]{axiom schema of specification} it follows that
  \begin{equation*}
    \set{ x \in A \given \bot }
  \end{equation*}
  is a set. Furthermore, \( x \) belongs to this set if and only if \( x \) satisfies \( \bot \), which is impossible, hence the set is empty.

  As a consequence of the \term{axiom of extensionality}, this empty set is unique. As discussed in \fullref{def:empty_set}, we denote this unique empty set by \( \varnothing \).

  \SubProofOf{thm:zfc_existence_theorems/universe} Aiming at a contradiction, suppose that there was a universal set \( U \). Then we can easily reproduce \fullref{thm:russels_paradox} by using restricted (to \( U \)) rather than unrestricted comprehension.

  Unlike in na\"ive set theory, however, the existence of \( U \) is not an axiom of the theory. Therefore, rather than demonstrating that \logic{ZFC} is inconsistent, Russel's paradox shows that certain sets like the universal set do not exist in \logic{ZFC}.

  \SubProofOf{thm:zfc_existence_theorems/singleton} Fix a set \( A \). The set \( \set{A} \), if it exists, is equal to \( \set{A} = \set{A, A} \), by the \hyperref[def:zfc/extensionality]{axiom of extensionality}.

  Thus, by the \hyperref[def:zfc/pairing]{axiom of pairing}, the singleton set \( \set{A} = \set{A, A} \) actually exists.

  \SubProofOf{thm:zfc_existence_theorems/arbitrary_intersection} Let \( \mscrA \) be a nonempty family of sets. Their intersection \( \bigcap \mscrA \), if it exists, is a subset of every set \( A \in \mscrA \).

  Therefore, since the family \( \mscrA \) is nonempty, the \hyperref[def:zfc/specification]{axiom schema of specification} applied to any set in \( A \in \mscrA \) guarantees the existence of the intersection \( \bigcap \mscrA \). More precisely, for any \( A_0 \in \mscrA \), we can define the intersection of \( A \) as
  \begin{equation*}
    \bigcap \mscrA = \set{ x \in A_0 \given \qexists {A \in \mscrA} x \in A }.
  \end{equation*}

  \SubProofOf{thm:zfc_existence_theorems/binary_intersection} For sets \( A \) and \( B \), by the \hyperref[def:zfc/pairing]{axiom of pairing} the set \( \set{ A, B } \) exists. Then by \fullref{thm:zfc_existence_theorems/arbitrary_intersection}, the binary intersection
  \begin{equation*}
    A \cap B = \bigcap \set{ A, B }
  \end{equation*}
  also exists.

  \SubProofOf{thm:zfc_existence_theorems/arbitrary_union} The existence of arbitrary unions is merely a restatement of the \hyperref[def:zfc/union]{axiom of unions}.

  \SubProofOf{thm:zfc_existence_theorems/binary_union} Similarly to \fullref{thm:zfc_existence_theorems/binary_intersection}, for sets \( A \) and \( B \), by the \hyperref[def:zfc/pairing]{axiom of pairing} the set \( \set{ A, B } \) exists and by the \hyperref[def:zfc/union]{axiom of unions}, the binary union
  \begin{equation*}
    A \cup B = \bigcup \set{ A, B }
  \end{equation*}
  exists.

  \SubProofOf{thm:zfc_existence_theorems/difference} The difference \( A \setminus B \) is guaranteed to exist by \hyperref[def:zfc/specification]{restricted comprehension}:
  \begin{equation*}
    A \setminus B = \set{ x \in A \given x \not\in B }.
  \end{equation*}

  \SubProofOf{thm:zfc_existence_theorems/power_set} The existence of power sets is a restatement of the \hyperref[def:zfc/power_set]{axiom of power sets}.

  \SubProofOf{thm:zfc_existence_theorems/successor} The successor of \( A \) is
  \begin{equation*}
    \op{succ}(A) = \set{ A } \cup A.
  \end{equation*}

  Its existence follows from \fullref{thm:zfc_existence_theorems/singleton} and \fullref{thm:zfc_existence_theorems/binary_union}.

  \SubProofOf{thm:zfc_existence_theorems/kuratowski_pair} The existence of the Kuratowski pair
  \begin{equation*}
    \braket{ A, B } = \set{ \set{ A }, \set{ A, B } }
  \end{equation*}
  can be proven by the \hyperref[def:zfc/pairing]{axiom of pairing} applied first to \( \set{ A, B } \) and then to the pair itself.

  \SubProofOf{thm:zfc_existence_theorems/indexed_family} Given an indexed family \( \seq{ A_k }_{k \in \mscrK} \), by the \hyperref[def:zfc/replacement]{axiom schema of replacement}, there exists a set
  \begin{equation*}
    \mscrA \coloneqq \set{ A_k \given k \in \mscrK }.
  \end{equation*}

  The family \( \seq{ A_k }_{k \in \mscrK} \) is, formally, a set of Kuratowski pairs. Every pair \( \braket{ k, A_k } \) is itself a subset of \( \pow(\mscrK \bigcup \mscrA) \). The family is then a subset of \( \pow(\pow(\mscrK \bigcup \mscrA)) \). Applying the \hyperref[def:zfc/power_set]{axiom of power sets} again, we obtain that the family exits as a set.

  \SubProofOf{thm:zfc_existence_theorems/cartesian_product} By definition, the Cartesian product \( \prod_{k \in \mscrK} A_k \) is a set of indexed by \( \mscrK \) families of members of the union \( \bigcup\set{ A_k \given k \in \mscrK } \).

  We can apply the \hyperref[def:zfc/power_set]{axiom of power sets} one more time in addition to those in \fullref{thm:zfc_existence_theorems/indexed_family} to obtain the set of all indexed by \( \mscrK \) families of members of this union. Then we can apply the \hyperref[def:zfc/specification]{axiom schema of specification} to restrict only to those families that satisfy the condition of \fullref{def:cartesian_product/product}.

  \SubProofOf{thm:zfc_existence_theorems/set_of_relations} All relations between \( A \) and \( B \) are subsets of \( A \times B \), hence elements of \( \pow(A \times B) \). The latter exists by \fullref{thm:zfc_existence_theorems/cartesian_product} and \fullref{thm:zfc_existence_theorems/power_set}.

  \SubProofOf{thm:zfc_existence_theorems/set_of_functions} The set of single-valued functions from \( A \) to \( B \) is a subset of \( \pow(A \times B) \), hence it exists by \fullref{thm:zfc_existence_theorems/set_of_relations} and \fullref{thm:zfc_existence_theorems/power_set}.

  \SubProofOf{thm:zfc_existence_theorems/quotient_set} Let \( A \) be an arbitrary set and \( \cong \) be a binary relation over \( A \). Then \( A / {\cong} \) is a subset of \( \pow(A) \) and hence it exists as a consequence of the \hyperref[def:zfc/power_set]{axiom of power sets} and the \hyperref[def:zfc/specification]{axiom schema of specification} --- see \fullref{thm:equivalence_partition/partition}.

  \SubProofOf{thm:zfc_existence_theorems/function_evaluation} The set \( \seq{ f_k }_{k \in \mscrK} \) exists because it is a member of \( \fun(\mscrK, \fun(A, B)) \), which set exists by \fullref{thm:zfc_existence_theorems/set_of_functions}.

   Then \( \seq{ f_k(x) }_{k \in \mscrK} \) is the function
   \begin{equation*}
     \begin{aligned}
       &g_x: \fun(\mscrK, B) \\
       &g_x(k) \coloneqq f_k(x).
     \end{aligned}
   \end{equation*}
\end{proof}

\begin{theorem}[Multi-valued selection existence]\label{thm:existence_of_multi_valued_function_selection}
  Every \hyperref[def:multi_valued_function/total]{total multi-valued function} has a \hyperref[def:function/selection]{selection}.

  Within \hyperref[def:zfc]{\logic{ZF}}, this theorem is equivalent to the \hyperref[def:zfc/choice]{axiom of choice} --- see \fullref{thm:axiom_of_choice_equivalences/selection}.
\end{theorem}
\begin{proof}
  \ImplicationSubProof[def:zfc/choice]{the axiom of choice}[thm:existence_of_multi_valued_function_selection]{selection existence} Let \( F: A \multto B \) be a total multi-valued function. As described in \fullref{rem:multi_valued_functions}, we can instead take the \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \set{ F(a) }_{a \in A} \). Denote by \( f \) the \hyperref[def:function]{single-valued function} from \( A \) to the image \( \set{ F(a) \given a \in A } \subseteq \pow(B) \) of this indexed family (see \fullref{rem:multi_valued_functions} for clarifications).

  Since \( F \) is \hyperref[def:multi_valued_function/total]{total}, the family \( \img(f) = \set{ F(a) }_{a \in A} \) is a (potentially empty) family of nonempty sets. Thus, we can apply the axiom of choice to obtain a \hyperref[def:choice_function]{choice function} \( c: \img(f) \to B \).

  The composition \( c \bincirc f \) is then a single-valued function. Furthermore, we have
  \begin{equation*}
    (c \bincirc f)(a) \in f(a) = F(a)
  \end{equation*}
  so \( c \bincirc f \) is a selection of \( F \).

  \ImplicationSubProof[thm:existence_of_multi_valued_function_selection]{selection existence}[def:zfc/choice]{axiom of choice} Fix a family \( \mscrA \) of nonempty sets. Define the function
  \begin{equation*}
    \begin{aligned}
      &F: \mscrA \to \bigcup \mscrA \\
      &F(A) \coloneqq A
    \end{aligned}
  \end{equation*}
  that sends each set in \( \mscrA \) to the corresponding subset of \( \mscrA \). In terms of relations, we have \( (A, x) \in F \) if and only if \( x \in A \). This is a total multi-valued function because every set in \( \mscrA \) is nonempty.

  Then every selection of \( F \) is a choice function for \( \mscrA \).
\end{proof}

\begin{definition}\label{def:disjoint_union}
  The \term{disjoint union} of the \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \seq{ A_k }_{k \in \mscrK} \) of nonempty sets is
  \begin{equation*}
    \coprod_{k \in \mscrK} A_k \coloneqq \set[\Big]{ (k, x) \given k \in \mscrK \T{and} x \in A_k }.
  \end{equation*}
\end{definition}

\begin{theorem}[Surjective functions are right-invertible]\label{thm:surjective_functions_are_right_invertible}
  Every \hyperref[def:function_invertibility/surjective]{surjective function} is \hyperref[def:morphism_invertibility/right_invertible]{right-invertible}.

  Within \hyperref[def:zfc]{\logic{ZF}}, this theorem is equivalent to the \hyperref[def:zfc/choice]{axiom of choice} --- see \fullref{thm:axiom_of_choice_equivalences/surjective}.
\end{theorem}
\begin{proof}
  Let \( f: A \to B \) be any function. Its \hyperref[def:multi_valued_function/inverse]{inverse} \( f^{-1}: B \multto A \) is, by definition, a partial multi-valued function.

  \ImplicationSubProof[def:zfc/choice]{the axiom of choice}[thm:epimorphisms_split_in_set]{right-invertibility} If \( f \) is surjective, then by \fullref{def:function_invertibility/surjective/inverse}, its inverse is total. Then the axiom of choice via \fullref{thm:existence_of_multi_valued_function_selection} gives us a \hyperref[def:function/selection]{single-valued selection} \( g \) of \( f^{-1} \).

  Since the value of \( f \) is \( y \) for all members of \( f^{-1}(y) \), and since \( g(y) \in f^{-1}(y) \), we have
  \begin{equation*}
    [f \bincirc g](y) = f(g(y)) = y.
  \end{equation*}

  Therefore, \( g \) is a right inverse of \( f \).

  \ImplicationSubProof[thm:epimorphisms_split_in_set]{right-invertibility}[def:zfc/choice]{the axiom of choice} Suppose that every surjective function is invertible.

  Let \( \mscrA \) be an arbitrary family of nonempty sets. We can regard it as the \hyperref[def:cartesian_product/indexed_family]{indexed family} \( \seq{ A }_{A \in \mscrA} \). Define the function
  \begin{equation*}
    \begin{aligned}
      &f: \coprod_{A \in \mscrA} A \to \mscrA, \\
      &f(A, x) \coloneqq A,
    \end{aligned}
  \end{equation*}
  where by \( \coprod \) we have denoted the \hyperref[def:disjoint_union]{disjoint union}.

  This function is surjective by definition. Then there exists a right-inverse
  \begin{equation*}
    g: \mscrA \to \seq{ A }_{A \in \mscrA}.
  \end{equation*}

  For every set \( A \in \mscrA \) we have \( [f \bincirc g](A) = A \). Given a set \( A \), \( g \) gives us a pair \( (A, x) \) with \( x \in A \) and so \( f(A, x) = A \).

  Finally, define the choice function
  \begin{equation*}
    \begin{aligned}
      &c: \mscrA \to \bigcup \mscrA, \\
      &c(A) \coloneqq x \T{where} (A, x) = g(A).
    \end{aligned}
  \end{equation*}

  Since the family \( \mscrA \) was arbitrary, we can conclude that the axiom of choice holds.
\end{proof}

\begin{proposition}\label{thm:function_invertibility_categorical}
  We prove this result here rather than in \fullref{subsec:functions} because it requires \fullref{thm:surjective_functions_are_right_invertible}, which requires the axiom of choice introduced in this section.

  In relation to \hyperref[def:morphism_invertibility]{morphism invertibility} in the category \hyperref[def:category_of_small_sets]{\( \cat{Set} \)}, we have the following:
  \begin{thmenum}
    \thmitem{thm:function_invertibility_categorical/empty} An \hyperref[def:multi_valued_function/empty]{empty function} is always \hyperref[def:function_invertibility/injective]{injective} and \hyperref[def:morphism_invertibility/left_invertible]{left-invertible}, however only if its range is empty is it \hyperref[def:morphism_invertibility/left_cancellative]{left-cancellative}, \hyperref[def:morphism_invertibility/right_cancellative]{right-cancellative}, \hyperref[def:morphism_invertibility/right_invertible]{right-invertible} or \hyperref[def:function_invertibility/surjective]{surjective}.

    \thmitem{thm:function_invertibility_categorical/nonempty_left_invertible} A nonempty function is \hyperref[def:morphism_invertibility/left_invertible]{left-invertible} if and only if it is \hyperref[def:function_invertibility/injective]{injective}.

    \thmitem{thm:function_invertibility_categorical/left_cancellative} A function is \hyperref[def:morphism_invertibility/left_cancellative]{left-cancellative} if and only if it is \hyperref[def:function_invertibility/injective]{injective}.

    \thmitem{thm:function_invertibility_categorical/right_invertible} A function is \hyperref[def:morphism_invertibility/right_invertible]{right-invertible} if and only if it is \hyperref[def:function_invertibility/surjective]{surjective}.

    \thmitem{thm:function_invertibility_categorical/right_cancellative} A function is \hyperref[def:morphism_invertibility/right_cancellative]{right-cancellative} if and only if it is \hyperref[def:function_invertibility/surjective]{surjective}.

    \thmitem{thm:function_invertibility_categorical/fully_invertible} A function is \hyperref[def:function_invertibility/bijective]{bijective} if and only if it is \hyperref[def:morphism_invertibility/isomorphism]{fully invertible}.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:function_invertibility_categorical/empty} Let \( g: \varnothing \to C \) be the empty function to \( C \). It is vacuously injective. It is also left-invertible because the only function that can be composed with \( g: \varnothing \to C \) from the left is the unique function from \( \varnothing \) to \( \varnothing \).

  Clearly \( g: \varnothing \to C \) it is surjective if and only if \( C = \varnothing \).

  For left-invertibility, note that \( g: \varnothing \to C \) composed with the function \( h: C \to D \) on the left is another empty function \( h \bincirc g: \varnothing \to D \). The latter is the identity \( \id_\varnothing \) if and only if \( C = D = \varnothing \).

  For right-invertibility, note that \( g: \varnothing \to C \) composed with the function \( f: A \to \varnothing \) on the right is the function \( g \bincirc f: A \to C \). But \( A = \varnothing \) since otherwise \( f \) would be nonempty, hence \( g \bincirc f: \varnothing \to C \). The latter is the identity \( \id_\varnothing \) if and only if \( C = \varnothing \).

  For right-cancellation, note that \( g \bincirc f_1 = g \bincirc f_2 \) implies \( f_1 = f_2 \) if and only if \( B \) is empty.

  \SubProofOf{thm:function_invertibility_categorical/nonempty_left_invertible} Let \( f: A \to B \) be a nonempty injective function. \Fullref{def:function_invertibility/injective/inverse} states that the \hyperref[def:multi_valued_function/inverse]{inverse} \( f^{-1}: B \to A \) is a partial single-valued function.

  Fix some value \( a \in A \) and define
  \begin{equation*}
    \begin{aligned}
      &g: B \to A \\
      &g(y) \coloneqq \begin{cases}
        f^{-1}(y), &y \in f(A) \\
        a,         &\T{otherwise.}
      \end{cases}
    \end{aligned}
  \end{equation*}

  This function \( g \) is a left inverse of \( f \) because, for any \( x \in A \),
  \begin{equation*}
    [g \bincirc f](x)
    =
    g(f(x))
    =
    f^{-1}(f(x))
    =
    x.
  \end{equation*}

  We can see that \( g \) would be unique except for our choice of \( a \). We may even define \( g \) to take different values in \( A \) outside \( f(A) \). Thus, \( g \) is non-unique in general.

  Conversely, suppose that \( f: A \to B \) is not necessarily injective and let \( g: B \to A \) be a left inverse of \( f \). Let \( x_1 \) and \( x_2 \) be two different points in \( A \). Since \( g \bincirc f = \id_A \), clearly \( g(f(x_1)) \neq g(f(x_2)) \). If we suppose that \( f(x_1) = f(x_2) \), we would obtain a contradiction since then \( g(f(x_1)) \) would equal \( g(f(x_2)) \). Hence, \( f(x_1) \neq f(x_2) \). This shows that \( f \) is injective.

  \SubProofOf{thm:function_invertibility_categorical/left_cancellative} The case with an empty function is handled in \fullref{thm:function_invertibility_categorical/empty}, and we assume that it is nonempty.

  Suppose that \( g: B \to C \) is a nonempty left-cancellative function. Let \( y_1 \) and \( y_2 \) be some members of \( B \) such that \( g(y_1) = g(y_2) \).

  Suppose that \( y_1 \neq y_2 \) and define the function
  \begin{equation*}
    \begin{aligned}
      &f: B \to B \\
      &f(y) \coloneqq \begin{cases}
        y_2, &y = y_1 \\
        y_1, &y = y_2 \\
        y,   &y \neq y_1 \T{and} y \neq y_2
      \end{cases}
    \end{aligned}
  \end{equation*}

  Then
  \begin{equation*}
    g(f(y_2)) = g(y_1) = g(y_2) = g(f(y_1))
  \end{equation*}

  For all \( y \in B \) different from \( y_1 \) and \( y_2 \), we have \( y = f(y) \).

  Since \( g \) is left-cancellative, from \( g \bincirc \id_B = g \bincirc f \) it follows that \( \id_B = f \), which is a contradiction.

  It remains for \( y_1 \) to be equal to \( y_2 \). Since these were arbitrary points in \( B \) satisfying \( g(y_1) = g(y_2) \), we conclude that \( g \) is injective.

  Conversely, if \( f \) is injective, it is left-invertible by \fullref{thm:function_invertibility_categorical/nonempty_left_invertible} and left-cancellative by \fullref{thm:def:morphism_invertibility/split_monomorphism}.

  \SubProofOf{thm:function_invertibility_categorical/right_invertible} In one direction, we have \fullref{thm:surjective_functions_are_right_invertible}.

  Conversely, suppose that \( g: B \to A \) is a right inverse of \( f: A \to B \). Let \( y \in B \). We have that \( g(y) \) is in the preimage of \( y \) under \( f \) because \( f(g(y)) = y \). Thus, the preimage is not empty for an arbitrary point in \( B \). We conclude that \( f \) is surjective.

  \SubProofOf{thm:function_invertibility_categorical/right_cancellative} The case with an empty function is handled in \fullref{thm:function_invertibility_categorical/empty}, and we assume that it is nonempty.

  Let \( f: A \to B \) be a nonempty right-cancellative function. Suppose that it is not surjective and let \( y_0 \in B \setminus \img f \). Let \( z \) be some set not belonging to \( B \). Define the function
  \begin{equation*}
    \begin{aligned}
      &g: B \to B \cup \set{ z } \\
      &g(y) \coloneqq \begin{cases}
        z, &y = y_0 \\
        y, &y \neq y_0
      \end{cases}
    \end{aligned}
  \end{equation*}

  Since \( f \) is right-cancellative, from \( \id_B \bincirc f = g \bincirc f \) it follows that \( \id_B = g \), which is a contradiction. Therefore, \( f \) is surjective.

  We can prove the converse using \fullref{thm:function_invertibility_categorical/right_invertible} like we did for \fullref{thm:function_invertibility_categorical/left_cancellative}, however we prefer a direct proof that does not rely on the axiom of choice. As a bonus, this would allow us to prove \fullref{thm:epimorphisms_split_in_set}.

  Conversely, suppose that \( f: A \to B \) is surjective and that for some functions \( g_1, g_2: B \to C \) we have
  \begin{equation*}
    g_1 \bincirc f = g_2 \bincirc f.
  \end{equation*}

  Fix some \( y \in B \). Because \( f \) is surjective, there exists some \( x \in A \) such that \( f(x) = y \). Then
  \begin{equation*}
    g_1(y) = [g_1 \bincirc f](x) = [g_2 \bincirc f](x) = g_2(y).
  \end{equation*}

  Since \( y \in B \) was arbitrary, we conclude that \( f \) is right-cancellative.

  it is right-invertible by \fullref{thm:surjective_functions_are_right_invertible} and right-cancellative by \fullref{thm:def:morphism_invertibility/split_epimorphism}.

  \SubProofOf{thm:function_invertibility_categorical/fully_invertible} If \( f: \varnothing \to B \) is a bijective empty function, then it is surjective and, by \fullref{thm:function_invertibility_categorical/right_invertible}, it is right-invertible. By \fullref{thm:function_invertibility_categorical/empty}, it is also left-invertible. Thus, it is fully invertible.

  Conversely, if an empty function \( f: \varnothing \to B \) is fully invertible, by \fullref{thm:function_invertibility_categorical/empty} we have \( A = B = \varnothing \) and hence it is bijective.

  Finally, if \( f: A \to B \) is \hi{nonempty} function, then by \fullref{thm:function_invertibility_categorical/nonempty_left_invertible} and \fullref{thm:function_invertibility_categorical/right_invertible} it is bijective if and only if it is fully invertible.

  In the bijective case, we can avoid the axiom of choice via \fullref{thm:surjective_functions_are_right_invertible} by noting that if \( f \) is bijective, its inverse is single-valued, and thus it is not necessary to do a selection of \( f^{-1} \).
\end{proof}

\begin{theorem}[Axiom of choice equivalences]\label{thm:axiom_of_choice_equivalences}
  The following statements are commonly referred to as \enquote{the} \hyperref[def:zfc/choice]{axiom of choice}:
  \begin{thmenum}[series=thm:axiom_of_choice_equivalences]
    \thmitem{thm:axiom_of_choice_equivalences/choice_sets} For every family of nonempty sets \( \mscrA \) there exists a set \( B \) such that \( A \cap B \) is a singleton set for every \( A \in \mscrA \).

    \thmitem{thm:axiom_of_choice_equivalences/choice_function} Every family of nonempty sets has a corresponding \hyperref[def:choice_function]{choice function}.

    \thmitem{thm:axiom_of_choice_equivalences/choice_product} The \hyperref[def:cartesian_product]{Cartesian product} of a family of nonempty sets is nonempty.
  \end{thmenum}

  The following statements are equivalent to the axiom of choice, but are not conflated with it:
  \begin{thmenum}[resume=thm:axiom_of_choice_equivalences]
    \thmitem{thm:axiom_of_choice_equivalences/selection} \Fullref{thm:existence_of_multi_valued_function_selection}: Every \hyperref[def:multi_valued_function/total]{total multi-valued function} has a \hyperref[def:function/selection]{selection}.

    \thmitem{thm:axiom_of_choice_equivalences/surjective} \Fullref{thm:surjective_functions_are_right_invertible}: Every \hyperref[def:function_invertibility/surjective]{surjective function} is \hyperref[def:morphism_invertibility/right_invertible]{right-invertible}.

    \thmitem{thm:axiom_of_choice_equivalences/hypergraph} \Fullref{thm:hypergraphs_have_minimal_transversal}: Every \hyperref[def:hypergraph]{hypergraph} has a \hyperref[def:hypergraph_minimal_transversal]{minimal transversal}.

    \thmitem{thm:axiom_of_choice_equivalences/epimorphisms} \Fullref{thm:epimorphisms_split_in_set}: Every \hyperref[def:morphism_invertibility/right_cancellative]{epimorphism} in \hyperref[def:category_of_small_sets]{\( \cat{Set} \)} splits.

    \thmitem{thm:axiom_of_choice_equivalences/fully_faithful_essentially_surjective} \Fullref{thm:fully_faithful_and_essentially_surjective_functor_induces_equivalence}: Every \hyperref[def:functor_invertibility/fully_faithful]{fully faithful} and \hyperref[def:functor_invertibility/surjective_on_objects]{essentially surjective on objects} functor induces a \hyperref[def:category_equivalence]{category equivalence}.

    \thmitem{thm:axiom_of_choice_equivalences/skeletons} \Fullref{thm:category_skeleton_existence}: Every \hyperref[def:category]{category} has a \hyperref[def:skeletal_category]{skeleton}.

    \thmitem{thm:axiom_of_choice_equivalences/well_ordering} \Fullref{thm:well_ordering_theorem}: Every \hyperref[def:set]{set} can be \hyperref[def:well_ordered_set]{well-ordered}.

    \thmitem{thm:axiom_of_choice_equivalences/zorns_lemma} \Fullref{thm:zorns_lemma}: If every \hyperref[def:partially_ordered_set_chain_and_antichain]{chain} in a \hyperref[def:partially_ordered_set]{partially ordered set} has an \hyperref[def:partially_ordered_set_extremal_points/upper_and_lower_bounds]{upper bound}, then the entire set has a \hyperref[def:partially_ordered_set_extremal_points/maximal_and_minimal_element]{maximal element}.

    \thmitem{thm:axiom_of_choice_equivalences/maximal_ideal} \Fullref{thm:maximal_ideal_theorem}: Every proper \hyperref[def:semiring_ideal]{semiring ideal} is contained in a \hyperref[def:semiring_ideal/maximal]{maximal ideal}.

    \thmitem{thm:axiom_of_choice_equivalences/vector_space_bases} \Fullref{thm:vector_space_basis_existence}: Every \hyperref[def:vector_space]{vector space} has a \hyperref[def:hamel_basis]{basis}.

    \thmitem{thm:axiom_of_choice_equivalences/tychonoff} \Fullref{thm:tychonoffs_product_theorem}: The \hyperref[def:topological_product]{topological product} of \hyperref[def:compact_space]{compact spaces} is compact.
  \end{thmenum}
\end{theorem}
\begin{proof}
  The equivalence proofs can be found in the linked theorems since that is usually the most appropriate place to put them.
\end{proof}

\begin{theorem}[Diaconescu-Goodman-Myhill theorem]\label{thm:diaconescu_goodman_myhill_theorem}\mcite[corr. 2]{Diaconescu1975}
  In \hyperref[def:zfc]{\logic{ZF}}, the \hyperref[def:zfc/choice]{axiom of choice} entails the law of the excluded middle \eqref{eq:thm:minimal_propositional_negation_laws/lem}.
\end{theorem}


\lab{Compressed Sensing}{Compressed Sensing}
\label{lab:compressed_sensing}

\objective{Learn About Techniques in Compressed Sensing.}

One of the more important and fundamental problems in mathematics and science is solving a system of linear equations
\[
Ax = b.
\]
Depending on the properties of the matrix $A$ (such as its dimensions and rank), there may be exactly one
solution, infinitely many solutions, or no solution at all.

In the case where $A$ is a square invertible matrix, there is of course one unique solution, given by
$A^{-1}b$. There are various computational methods for inverting $A$, and we have studied many of them previously.

When $b$ does not lie in the range of $A$, there is no exact solution. We can still hope to find approximate solutions,
and techniques such as least squares and least absolute deviations provide ways to do this.

The final case is when there are infinitely many vectors $x$ that satisfy $Ax = b$. How do we decide which vector
to choose? A common approach is to choose a vector $x$ of minimal norm satisfying $Ax = b$. This can be stated as
an optimization problem:
\begin{align*}
\text{minimize}\qquad &\|x\|\\
\text{subject to} \qquad &Ax = b.
\end{align*}

When we use the standard Euclidean $2$-norm, the problem is equivalent to the quadratic program
\begin{align*}
\text{minimize}\qquad &x^Tx\\
\text{subject to} \qquad &Ax = b,
\end{align*}
which we can solve using an iterative procedure. Alternatively, the solution is given directly by $A^\dagger b$,
where $A^\dagger$ is the Moore-Penrose pseudoinverse of $A$.

If instead of the $2$-norm we use the $1$-norm, our problem can be restated as a linear program, and solved
efficiently using the Simplex Algorithm or an Interior Point method. Of course we can use any norm whatsoever, but
finding the solution may be much more difficult.

The basic problem in the field of Compressed Sensing is to recover or reconstruct certain types of signals
from a small set of measurements. For example, we might have measurements of the frequency spectrum of an
audio signal, and we wish to recover the original audio signal as nearly as possible. Mathematically, this
problem can be viewed as solving the under-determined system of equations
$Ax = b$, where $b$ is a vector of measurements and $A$ is called the measurement matrix.
The crucial idea that Compressed Sensing brings to the table is the concept of \emph{sparsity}, which we address now.

\section*{Sparsity and the $l_0$ Pseudonorm}
\emph{Sparsity} is a property of vectors in $\mathbb{R}^n$ related to how compactly and concisely they can
be represented in a given basis. Stated more concretely, the sparsity of a vector $x$ (expressed in some given
basis) refers to how many nonzero entries are in $x$. A vector having at most $k$ nonzero entries is said to be
$k$-sparse. This concept can be extended to time series (such as an audio signal) and to images. Figure 32.1 shows
examples of both sparse and non-sparse signals.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{sparse.pdf}
\caption{A sparse signal and a non-sparse signal.}
\label{fig:sparse}
\end{figure}

As a convenient way of measuring sparsity, we define the so-called $l_0$ pseudonorm, notated $\|\cdot\|_0$,
that simply counts the number of nonzero entries in a vector. For example, if we have
\[
x =
\begin{bmatrix}
1&0&0&-4&0
\end{bmatrix}^T
\]
and
\[
y =
\begin{bmatrix}
.5&.2&-.01&3&0
\end{bmatrix}^T,
\]
we then have
\[
\|x\|_0 = 2
\]
and
\[
\|y\|_0 = 4.
\]
Despite our choice of notation, $\|\cdot\|_0$ is not truly a norm (which properties does it fail to satisfy?).
Keep this in mind, even if we refer to it as the $l_0$ norm.

As mentioned earlier, sparsity is of central importance in Compressed Sensing, for it provides a way for us to
select from the infinite possibilities a single vector $x$ satisfying $Ax = b$. In particular, we require $x$ to
be as sparse as possible, i.e. to have minimal $l_0$ norm. Stated explicitly as an optimization problem, Compressed
Sensing boils down
\begin{align*}
\text{minimize}\qquad &\|x\|_0\\
\text{subject to} \qquad &Ax = b.
\end{align*}

\section*{Sparse Reconstruction}
How does the Compressed Sensing framework laid out above help us recover a signal from a set of measurements?
If we know nothing about the signal that we are trying to reconstruct, anything but a complete set of measurements
(or \emph{samples}) of the signal will be insufficient to fully recover the signal without any error.

However, we can often make certain assumptions about the unknown signal, such as setting an upper bound for
its highest frequency. Given this prior knowledge about the frequency of the signal, we \emph{are} able to
perfectly or nearly perfectly reconstruct the signal from an incomplete set of measurements, provided that
the sampling rate of the measurements is more than twice that of the largest frequency. This classic result
in signal processing theory is known as the \emph{Nyquist-Shannon sampling theorem}.

What if, instead of having prior knowledge about the frequency of the signal, we have prior knowledge about
its sparsity? Recent research asserts that it is possible to recover sparse signals to great accuracy from just
a few measurements. This matches intuition: sparse signals do not contain much information, and so it ought to
be possible to recover that information from just a few samples. Stated more precisely, if $\hat{x} \in \mathbb{R}^n$
is sufficiently sparse and an $m \times n$ matrix $A$ ($m < n$) satisfies certain properties (to be described later),
with $A\hat{x} = b$, then the solution of the optimization problem
\begin{align*}
\text{minimize}\qquad &\|x\|_0\\
\text{subject to}\qquad &Ax = b
\end{align*}
yields $\hat{x}$.

The matrix $A$ above must satisfy a technical condition called the \emph{Restricted Isometry Principle}. Most
random matrices obtained from standard distributions (such as the Gaussian or Bernoulli distributions) satisfy this
condition, as do transformation matrices into the Fourier and Wavelet domains. Generally speaking,
the measurement matrix $A$ represents a change of basis from the \emph{sparse} basis to the \emph{measurement} basis,
followed by an under-sampling of the signal. The Restricted Isometry Principle guarantees that the measurement
basis is incoherent with the sparse basis; that is, a sparse signal in the sparse basis is diffuse in the measurement
basis, and vice versa (see Figure \ref{fig:incoherent}). This ensures that the information contained in the few nonzero
coefficients in the sparse
domain is spread out randomly and roughly evenly among the coefficients in the measurement domain.
We can then obtain a small random subset of these measurement coefficients, solve the above optimization problem,
and recover the original signal.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{incoherent.pdf}
\caption{A sparse image (left) and its Fourier transform (right). The Fourier domain is incoherent with the
standard domain, so the Fourier transform of the spare image is quite diffuse.}
\label{fig:incoherent}
\end{figure}

Fortunately, many types of signals that we wish to measure are sparse in some domain. For example, many images
are sparse in the Wavelet domain. It turns out that the Fourier domain is incoherent with the Wavelet domain,
so if we take measurements of the image in the Fourier domain, we are able to recover the image with high
fidelity from relatively few measurements. This has been particularly useful in magnetic resonance imaging
(MRI), a medical process that obtains pictures of tissues and organs in the body by taking measurements in
the Fourier domain. Collecting these measurements can in some cases be harmful. Compressed sensing
allows for fewer measurements and therefore a shorter, safer MRI experience.

\section*{Solving the $l_0$ Minimization Problem}
Now that we have become familiar with the mathematical underpinnings of compressed sensing, let us now turn to
actually solving the problem. Unfortunately, the $l_0$ minimization problem stated above is NP hard and thus
computationally intractable in its current form. Another key result in compressed sensing states that we can
replace the $l_0$ norm with the $l_1$ norm and with high probability still recover the original signal, provided
it is sufficiently sparse. Since the $l_1$ minimization problem can be solved efficiently, we have a viable
computational approach to compressed sensing.

Recall that we can convert the $l_1$ minimization problem into a linear program by introducing an additional
vector $u$ of length $n$, and then solving
\begin{align*}
\text{minimize}\qquad
&\begin{bmatrix}
\mathbf{1} & 0
\end{bmatrix}
\begin{bmatrix}
u \\
x
\end{bmatrix}\\
\text{subject to}\qquad
&\begin{bmatrix}
-I & I\\
-I & -I
\end{bmatrix}
\begin{bmatrix}
u \\
x
\end{bmatrix}
\leq
\begin{bmatrix}
0\\
0
\end{bmatrix},\\
&\begin{bmatrix}
0 & A
\end{bmatrix}
\begin{bmatrix}
u \\
x
\end{bmatrix}
=
b.
\end{align*}
Of course, solving this gives values for the optimal $u$ and the optimal $x$, but we only care about the optimal $x$.

\begin{problem}
Write a function \li{l1Min} that takes a matrix $A$ and vector $b$ as inputs, and returns the solution to the
optimization problem
\begin{align*}
\text{minimize}\qquad &\|x\|_1\\
\text{subject to} \qquad &Ax = b.
\end{align*}
Formulate the problem as a linear program, and use CVXOPT to obtain the solution.
\end{problem}

Let's reconstruct a sparse image using different numbers of measurements, and compare results.
Load the image contained in the file \li{ACME.png} into Python as follows:
\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt
>>> acme = 1 - plt.imread('ACME.png')[:,:,0]
>>> acme.shape
(32L, 32L)
\end{lstlisting}
The image contains $32^2$ pixels, and so viewed as a flat vector, it has $32^2$ entries.
Now build a random measurement matrix based on $m$ samples as follows:
\begin{lstlisting}
>>> # assume the variable m has been initialized
>>> np.random.seed(1337)
>>> A = np.random.randint(0,high=2,size=(m,32**2))
\end{lstlisting}
Next, calculate the $m$ measurements:
\begin{lstlisting}
>>> b = A.dot(acme.flatten())
\end{lstlisting}
We are now ready to reconstruct the image using our function \li{l1Min}:
\begin{lstlisting}
>>> rec_acme = l1Min(A,b)
\end{lstlisting}

\begin{problem}
Following the example above, reconstruct the ACME image using 200, 250, and 270 measurements.
Be sure to execute the code \li{np.random.seed(1337)} prior to each time you initialize
your measurement matrix, so that you will obtain consistent answers. Report the $2$-norm
distance between the each reconstructed image and the original ACME image.
\end{problem}

Figure \ref{fig:reconstruct} shows the results of reconstructing a similar sparse image using both the $1$-norm and
the $2$-norm.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{reconstruct.pdf}
\caption{A sparse image (left), perfect reconstruction using $l_1$ minimization (middle),  and
imperfect reconstruction using $l_2$ minimization (right).}
\label{fig:reconstruct}
\end{figure}
\section*{Tesselated Surfaces and the Single Pixel Camera}
We now generalize our discussion to reconstructing functions defined on surfaces. Such a function might
give a color at each point on the surface, or the temperature, or some other property. The surface may be
a sphere (to model the surface of the earth), a cube, or some other desired form. To make the problem
tractable, we must break the surface up into a set of discrete polygonal faces, called a tessellation. See
Figure \ref{Fig:Earth} for an example of a tessellated sphere.

Now that we have a tessellation, our function can be represented by a vector, consisting of the function value
for each facet (element of the tessellation). We are thus in a position to apply the techniques of compressed
sensing to recover the function.

How do we go about acquiring color measurements on a tessellated surface? One cheap method is to use a \emph{single-pixel
camera}. Such a camera can only measure color value at a time. The camera points to a specific spot on the surface, and
records a weighted average of the colors near that spot. By taking measurements at random spots across the surface,
the single-pixel camera provides us with a vector of measurements, which we can then use to reconstruct the color value for
each facet of the tessellation. See Figure \ref{Fig:Earth} for the results of a single-pixel camera taking measurements of the earth.

\begin{figure}
\begin{center}
$\begin{array}{cc}
\includegraphics[scale = .6]{OriginalCropped} &
\includegraphics[scale = .6]{EstimatedCropped} \\
\includegraphics[scale = .5]{Original2Cropped} &
\includegraphics[scale = .5]{Estimated2Cropped} \\
\mbox{\bf (a)} & \mbox{\bf (b)}
\end{array}$
\end{center}
\caption{A 3-D model of the earth, with 5120 faces. $(a)$ shows two views of the model drawn directly from satellite imagery. $(b)$ shows two views of the reconstruction based upon 2500 single-pixel measurements.}
\label{Fig:Earth}
\end{figure}

The single-pixel camera measurement process can be modeled as matrix multiplication
by a measurement matrix $A$, which fortunately has the Restricted Isometry Property. In the following,
we provide you with code to take single pixel measurements. It works as follows
\begin{lstlisting}
run camera.py
myCamera=Camera(faces,vertices,colors)
\end{lstlisting}
Where faces, verticies and colors are given by the tesselation. To take a picture you use
\begin{lstlisting}
myCamera.add_pic(theta,phi,r)
\end{lstlisting}
Where theta, phi, and r are the spherical coordinates of the camera. Recall to change spherical coordinates to rectangular the formula is
\[
(r\sin(\phi)\cos(\theta),r\sin(\phi)\sin(\theta),r\cos(\phi))
\]
Where $\theta \in [0,2\pi),\phi \in [0,\pi), r \in [0,\infty)$

Calling \li{add_pic} many times at different places on a sphere (constant $r$ value, varying $\theta$ and $\phi$) will build the measurement matrix as well as the measurements. You can return these by
\begin{lstlisting}
A,b=myCamera.returnData()
\end{lstlisting}

In this applications signals are only sparse in some appropriate representation (such as Fourier or wavelet). This method generally can still be applied in such cases. Let $V$ represent the transformation under which $s$ is sparse, or in other words:
\begin{equation}
s = V p
\end{equation}
$V$ is the inverse Fourier. We can then recast $A s=b$ as
\begin{equation}
A V p = b
\end{equation}
This then allows us to find $p$ using Compressed Sensing, which in turn allows us to reconstruct $s$ by using $V$. In the the following problem $V$ will be given to you.

You must reconstruct the color functions
for the tesselated surfaces, and then plot these surfaces using the code we provide you.

%\begin{problem}
%We will perform compressed sensing on a Rubik's cube. The Rubik's cube has a natural tessellation, with each facet
%being a square having one color. Each color can be represented as a vector of length 3, where we have split the
%color into three color channels (red, green, and blue). The set of colors for the entire Rubik's cube is fairly
%sparse in each color channel, so we can reconstruct the color values in each color channel separately.
%
%In the file \li{StudentRubiksData.npz}, we have given you the measurement matrices for each color channel
%(accessed by the keys \li{'A1', 'A2', 'A3'}) along with the respective measurements in each color channel
%(accessed by the keys \li{'b1', 'b2', 'b3'}). Reconstruct the colors in each color channel, obtaining
%three arrays. Then stack these arrays row-wise into an array with three rows. Report this final array.
%
%To visualize your result, we have provided code in the file \li{visualize.py} that you can call.
%You also need to load in the array corresponding to the key \li{'centers'} in the data archive
%\li{StudentRubiksData.npz}. We assume the variable holding this array is named \li{c}.
%Execute the following (assuming that the variable holding the array
%you obtained above containing the reconstructed colors is named \li{r}):
%\begin{lstlisting}
%>>> from visualize import visualizeSurface
%>>> visualizeSurface(r.clip(0,1), c, 3)
%\end{lstlisting}
%You should see a plot showing the Rubik's cube.
%\end{problem}

\begin{problem}
We will reconstruct the surface of the earth from sparse satellite imagery. The earth is modeled by a sphere,
which can be tessellated into triangular faces, each having a color value. There are three color
channels, and the colors are sparse in each channel in the appropriate basis.

In the file \li{StudentEarthData.npz}, we have given you the faces, vertices, colors, and the inverse fourier matrix, (accessed by the key \li{'faces'}, \li{'vertices'}, \li{'C'}, and \li{'V'} respectively), Take pictures with the single pixel camera code (using $r=3$) and then reconstruct the colors in each color channel, obtaining
three arrays. Then stack these arrays column-wise into an array with three columns. Report this final array.

To visualize your result, we have provided code in the file \li{visualize2.py} that you can call.
Execute the following (assuming that the variable holding the array
you obtained above containing the reconstructed colors is named \li{s}):
\begin{lstlisting}
>>> from visualize2 import visualizeEarth
>>> visualizeEarth(faces, vertices, s.clip(0,1))
\end{lstlisting}
You should see a plot showing the reconstructed earth. We could have reconstructed a more detailed earth by choosing a finer tessellation, but you should be able to make out the shapes of the major continents. You can compare this to the original by running
\begin{lstlisting}
>>> visualizeEarth(faces, vertices, colors)
\end{lstlisting}

Try the reconstruction with 450, 550, 650 measurements and report the absolute difference between the reconstruction and the actual colors.
\end{problem}


\outline{3}{Exercise 12.10}
\subsubsection*{Exercise 12.10}

\textit{What would happen if the space of the forest fire propagation were 1-D
or 3-D? Conduct the renormalization group analysis to see what happens in those
cases.}

\vspace{5mm}
\textbf{One-diomensional case:}

\vspace{5mm}
The probability of percolation for one cell is the probability of that cell
assume value $1$:
\begin{equation}
  p_1 = q.
\end{equation}

Expand our group to two cells, the percolation proccess will occur only if the
two cells assume value $1$:
\begin{equation}
  p_2 = {p_1}^2.
\end{equation}

In fact, we can generalize the previous equation to:
\begin{equation}
  p_{s+1} = {p_s}^2,
\end{equation}

and conclude that if one cell in our system assume $0$, the propagation will be
stopped.

\vspace{5mm}
\textbf{Three-diomensional case:}

\vspace{5mm}
Despite the three dimensions of cells, the probability of percolation for one
cell is equal to previous case:
\begin{equation}
  p_1 = q.
\end{equation}

In this case is actually easier to list possibilities that don\textquotesingle t
allow percolation. The number of possible configurations for the renormalization
group analysis made of $2\times2\times2$ cells is:
\begin{equation}
  S = 256, \qquad S = k^{L^{D}}, \quad k = L = 2, \quad D = 3.
\end{equation}

Since cells have Moore\textquotesingle s neighborhood, any cell is neighbor to
the other cells. Imagine the renormalization group as two $2\times2$ planes, and
the percolation proccess from one plane to another. Possibilities that cells are
all in one of these two planes prevents the percolation process. To list then:

\vspace{5mm}
1 from empty cells; \\
2 from 4 cells filled; \\
8 from 3 cells filled; \\
8 from 1 cell filled; \\
12 from 2 cells filled;

\vspace{5mm}
A total of 31 possibilities, that leads to the relation of possibilities in
which we have percolation proccess and total number of possibilities:
\begin{equation}
  \frac{225}{256}
\end{equation}

The critical percolation threshold is given by:
\begin{equation}
  \begin{aligned}
    p_c = { } & 1 - \\
              & 1 \times (1-p_c)^8 - \\
              & 2 \times {p_c}^4 (1-p_c)^4 - \\
              & 8 \times {p_c}^3 (1-p_c)^5 - \\
              & 8 \times p_c (1-p_c)^7 - \\
              & 12 \times {p_c}^2 (1-p_c)^6.
  \end{aligned}
\end{equation}

And the cobweb plot for that equation is:
\begin{figure}[h]
  \centering
  \includegraphics[width=0.75\textwidth]{./figures/12.10-cobweb-plot-for-3-dimensional.pdf}
  \caption{\texttt{12.10-cobweb-plot-for-3-dimensional.py}}
\end{figure}

In which is easy to see two asymptotic states possible, $p_\infty = 0$ and
$p_\infty = 1$ (the cobweb plot is about relations over scale, not about
dynamics over time). There is an unstable equilibrium point around $p = 0.1$.
The approximate value is:
\begin{equation}
  p \approx 0.0794325.
\end{equation}

\textbf{Comparnison of cases:}
\vspace{5mm}

The relation of possibilities in which we have percolation proccess and total
number of possibilities for the 1-D, 2-D and 3-D are, respectively:
\begin{equation}
  \frac{1}{4} < \frac{9}{16} < \frac{225}{256}.
\end{equation}

For one-dimensional systems, the critical percolation threshold is $100 \%$,
since all cells must assume $1$ to percolation to occur. For two-dimensional,
the threshold is $38 \%$. Finally, for three-dimensional systems, the threshold
is under $8 \%$.

\vspace{5mm}
From this we can conclude that as we increase the number of dimensions, we
increase the susceptibility to percolation.


\section{Neural Network}
\label{sec:NN}
In this article, we focus mainly on supervised learning rather
than unsupervised learning. In most cases, the tasks of neural
networks turn out to be regression, binary classification for
instance. Generally speaking, a neural network approximate a
function by minimizing the difference between the ideal one and
the approximated one. In short, what neural networks do is optimization.
\par In this section, we start with the representation of a neural
network and then further into the way neural networks solve an
optimization problem.

\subsection{Neuron}
Neuron is an abstract representation of the smallest component of
a neural network. Specifically neuron is a type of function
$ g: \mathbb{R}^n \rightarrow \mathbb{R}^n $, which is also called
activation function including ReLU, sigmoid, softmax etc. In various
neural networks such as Deep Neural Network, Convolution Neural
Network (CNN) and Recurrent Neural Network (RNN), neuron takes on
difference forms.
\par The basic model of neuron is illustrated in \autoref{fig:neuron}.
We denote that $ x \in \mathbb{R}^n $ is the input data,
$ W \in \mathbb{R}^n $ and $ b \in \mathbb{R} $ are the parameters in
the neuron, $ z \in \mathbb{R} $ is a cache given by:
\begin{align}
    z = W^Tx + b
\end{align}
\par $ a \in \mathbb{R} $ is the output of a neuron given by:
\begin{align}
    a = g(z)
\end{align}
\par $ g: \mathbb{R}^n \rightarrow \mathbb{R}^n $ is the activation function.

\begin{figure}[H]
    \centering
    \includegraphics[width=10cm]{neuron}
    \caption{\label{fig:neuron}The model of neuron}
\end{figure}

\par The most common used activation functions are ReLU, sigmoid which are
illustrated in \autoref{fig:activation}.

\begin{figure}[H]
    \centering
    \subfigure[ReLU: $g(z) = max(0,\ z)$]{
    \begin{minipage}[t]{0.5\linewidth}
    \centering
    \includegraphics[width=5cm]{ReLU.png}
    \end{minipage}%
    }%
    \subfigure[sigmoid: $ \sigma(z) = \frac{1}{1+e^{(-z)}} $]{
    \begin{minipage}[t]{0.5\linewidth}
    \centering
    \includegraphics[width=5cm]{sigmoid.png}
    \end{minipage}%
    }%
    \centering
    \caption{\label{fig:activation}Common activation functions}
\end{figure}
\par In multi-classification tasks, softmax
$ g(z) = \frac{e^z}{\sum\limits_{i=1}^{n}e^{z_i}},\ z \in \mathbb{R}^n\ $
is widely used in the last neuron of the neural network.

\subsection{Architecture}
Various neurons form the neural networks, and the specific formation is called
the architecture of the neural networks. Most of the architecture like
Deep Neural Network, CNN and RNN can be regarded as a set of layers,
each layer consists of a bunch of neurons. The output of the previous layers
serve as the input of the neurons in the current layer.

First we define the following notations: $ l $ is the index of layer,
$ L $ is the number of layers, $ n^{[l]} $ is the number of neurons
in the $ l^{th} $ layer, $ W^{[l]} \in \mathbb{R}^{n^{[l]}\times n^{[l-1]}} $
and $ b^{[l]} \in \mathbb{R}^{n^{[l]}\times 1 } $ are the weights of neurons
in the $ l^{th} $ layer, $ z^{[l]} \in \mathbb{R}^{n^{[l]}\times 1} $ is
the caches of neurons in the $ l^{th} $ layer,
$ a^{[l]} \in \mathbb{R}^{n^{[l]}\times 1} $ is the output of neurons
in the $ l^{th} $ layer,
$ g^{[l]}: \mathbb{R}^{n^{[l]}\times 1} \rightarrow \mathbb{R}^{n^{[l]}\times 1} $
is the activation function in the $ l^{th} $ layer.
\par The basic model of a neural network is demonstrated in \autoref{fig:shallowNN},
the input $ x $ can also be considered as the $ 0^{th} $ layer, which means
$ x = a^{[0]} $, and it is called the input layer. The last layer i.e. the $ L^{th} $
layer is called the output layer which means $ \hat{y} = a^{[L]} $.
The remainder are called the hidden layers.

\begin{figure}[H]
    \centering
    \includegraphics[width=10cm]{shallowNN}
    \caption{\label{fig:shallowNN}Shallow Neural Network}
\end{figure}

Shallow network is a simple type of neural network with only one hidden layer,
while deep network, see \autoref{fig:deepNN}, has multiple hidden layers thus is much more complicated than
the shallow one.

\begin{figure}[H]
    \centering
    \includegraphics[width=10cm]{deepNN}
    \caption{\label{fig:deepNN}Deep Neural Network}
\end{figure}

\subsection{Loss Function}
The core of optimization problem is to minimize the cost. Given a set of labeled
data $ {x,\ y},\ x \in \mathbb{R}^{n^{[0]}\times m}, \ y \in \mathbb{R}^{1\times m} $,
we denote that $ m $ is the number of data. We want to pick the parameters $ W,\ b $
in each neurons so that the predicted output $ \hat{y} $ is close to the true
value $ y $. To measure the difference we need to calculate the distance between
the predicted $ \hat{y}^{(i)} $ from the $ i^{th} $ sample and $ y^{(i)} $.
For a certain distance metric $ l(\hat{y}^{(i)},\ y^{i}) $ which is also be regarded
as loss function, we can
write the total cost $ J(\hat{y},\ y) $ of the predicted $ \hat{y} $ and
define the optimization problem as:

\begin{equation}
    \begin{split}
        & Given\ \{x,\ y\} \\
        & \mathop{\arg\min}_{W,\ b}J(\hat{y},\ y) \\
        & J(\hat{y},\ y) = \sum\limits_{i=1}^{m}l(\hat{y}^{(i)},\ y^{i})
    \end{split}
\end{equation}

\par As for the choice of $ l $, the most widely used one is $L_2$ Norm
i.e. Euclidean Distance $ l(x,\ y) = ||x - y||_2^2 $. Nevertheless, in
difference kinds of problems, there would be a corresponding loss function
that has the best performance. For instance, in binary classification problem,
the popular choice is binary cross-entropy: \\
$ l(x,\ y) = - x\ln(y) - (1-x)\ln(1-y) $; in multi-classification problem,
the popular choice is categorical cross-entropy:
$ l(x,\ y) = - \sum\limits_{j=1}^{n}x^{[j]}\ln(y^{[j]}) $, where $ n $ is the
number of classes.

\subsection{Gradient Descent}
\label{ssec:GD}
\subsubsection{Basic Intuition of GD}
Gradient descent (GD) is the foundation of a large scale of optimization algorithms
in neural networks. Imagine that you stand on the top of a mountain and
plan to go downhill, the most efficient choice is following the path that is the
steepest. Namely, the most efficient method to find the minimum of $ J $ is to
choose the gradient of $ J $ at the parameters $ W,\ b $:
$ \nabla J(\hat{y},\ y) $ \footnote{$\nabla J(\hat{y},\ y) =
(\frac{\partial J}{\partial W^{[1]}},\cdots,\ \frac{\partial J}{\partial W^{[L]}},\
\frac{\partial J}{\partial b^{[1]}},\cdots,\ \frac{\partial J}{\partial b^{[L]}} )^T $}
Hence, we update the parameters at the learning rate $ \alpha $:
\begin{align}
    \label{equ:GD} W^{[l]} & = W^{[l]} - \alpha\frac{\partial J}{\partial W^{[l]}} \\
    b^{[l]} & = b^{[l]} - \alpha\frac{\partial J}{\partial b^{[l]}}
\end{align}
until $ J(\hat{y},\ y) $ is small enough.

\subsubsection{Basic Convergence Analysis of GD}
It is not clear that whether GD can converge to the required global minimum
rather than local minimum. However, a notable paper by Dauphin et al.
\parencite{dauphin2014identifying} shows that based on empirical evidence,
saddle points are bigger challenges instead of local minimum. Especially
in high dimensional problems, saddle points are often surrounded by
rugged plateau which dramatically slow down the speed of convergence.
More detail about the landscape of neural network will be discussed in
\autoref{ssec:empirical}. Here we just give an basic analysis of GD algorithm.
\par Before the analysis, we need a criterion of convergence. There are mainly
two criteria: the limit point is a stationary point and the convergence of
the function value. Apparently, the latter one is more easy to achieve.
\par In Bertsekas et al. \parencite{bertsekas1997nonlinear}, proposition
1.2.1 to 1.2.4 give the convergence result of GD. Considering the large
scale of use of fixed learning rate, proposition 1.2.3 regarding the
Lipschitz value is the most helpful and well-known one. And its variant
is shown in the following:
\begin{pro}
    \label{pro:ConstantConvergence}
    Let $ \{x_k\} $ be a sequence generated by GD: $ x_{k+1} = x_{k} - \alpha d_k $,
    where $ \{d_k\} $ is the gradient of $ f(x_k) $. For some $ L>0,\
    L \in R $  that satisfies\footnote{$ L $ is called the Lipschitz constant}:
    \begin{equation}
        ||\nabla f(x) - \nabla f(y)||\leq L||x - y||,\ \ \ \forall x,\ y \in \mathbb{R}
    \end{equation}
    If $ \forall k,\ d_k \neq 0 $ and
    \begin{equation}
        \label{equ:learningRate}
        \epsilon \leq \alpha \leq \frac{2-\epsilon}{L}
    \end{equation}
    where $ \epsilon \in R^+ $. Then every limit point of $ \{x_k\} $ is a
    stationary point of $f$.
\end{pro}

\begin{prf}
    Using the descent lemma\footnote{Details of descent lemma are in the
    Appendix 1 }, we have:
    \begin{align}
        f(x_k - \alpha d_k) - f(x_k)&\leq\alpha\nabla f(x_k)^Td_k + \frac{1}{2}\alpha^2L||d_k||^2 \\
        & = \alpha ||d_k||^2(\frac{1}{2}\alpha L - 1)
    \end{align}
    The right-hand side of \autoref{equ:learningRate} yields:
    \begin{align}
        \frac{1}{2}\alpha L - 1 \leq -\frac{1}{2}\epsilon
    \end{align}
    Together with the left-hand of \autoref{equ:learningRate}:
    \begin{align}
        f(x_k) - f(x_k - \alpha d_k) \geq \frac{1}{2}\epsilon^2||d_k||^2
    \end{align}
    If $ \{x_k\} $ converge to a non-stationary point $ x $, then $ ||d_k||^2 < 0 $
    and $ f(x_k) - f(x_k - \alpha d_k) \rightarrow 0 $,
    so $ ||d_k|| \rightarrow 0 $, which contradicts the assumption. Hence,
    every limit point of $ \{x_k\} $ is stationary.  \\
    \textbf{Q.E.D.}
\end{prf}

\par According to \autoref{pro:ConstantConvergence}, we can choose the
learning rate $ \alpha = \frac{2}{L} $ and guarantee that GD will converge.
Unfortunately, for optimization in neural network, such a global Lipschitz
constant may not exist in most cases. until now, there seems to be no obvious
way to fix the gap between the theorem and practice. In the article of Ruoyu Sun
\parencite{sun2019optimization}, a claim that may be sufficient for practitioners
is proposed: if all parameters are bounded in each iteration, with
a small and proper learning rate $ \alpha $, GD will converge.

\subsection{Forward Propagation}
Forward propagation (FP) is a method to compute the predicted $ \hat{y} $
from the input data. To illustrate the process of FP, we use the network
shown in \autoref{fig:deepNN}. For the first layer, input of neurons is
$ A^{[0]} = (a^{[0](1)},\cdots,\ a^{[0](m)}) \in \mathbb{R}^{n^{[0]}\times m} $,
the output is $ A^{[1]} $ and the cache is $ Z^{[1]} = (z^{[1](1)},\cdots,\ z^{[1](m)}) $.
\begin{equation}
    \begin{split}
        Z^{[1]} & = W^{[1]}A^{[0]} + b^{[1]} \\
        A^{[1]} & = g^{[1]}(Z^{[1]}) \\
        & \vdots
    \end{split}
\end{equation}
Therefore, the general computation of the $ i^{th} $ layer is:
\begin{equation}
    \label{equ:FP}
    \begin{split}
        Z^{[l]} & = W^{[l]}A^{[l-1]} + b^{[l]} \\
        A^{[l]} & = g^{[l]}(Z^{[l]})
    \end{split}
\end{equation}

\subsection{Backward Propagation}
Backward propagation (BP) is considered as an vital landmark in the
development of neural networks. It highly boost the efficiency of the
computation of gradient. Different from the FP, BP starts from the output
layer of the network and goes through the network to the front.
We denote that
$ dA^{[l]} \triangleq \frac{\partial J}{\partial A^{[l]}}
\in \mathbb{R}^{n^{[l]}\times m} $,
$ dZ^{[l]} \triangleq \frac{\partial J}{\partial Z^{[l]}}
\in \mathbb{R}^{n^{[l]}\times m} $,
$ dW^{[l]} \triangleq \frac{\partial J}{\partial W^{[l]}}
\in \mathbb{R}^{n^{[l]}\times n^{[l-1]}} $ and
$ db^{[l]} \triangleq \frac{\partial J}{\partial b^{[l]}}
\in \mathbb{R}^{n^{[l]}\times 1} $.
For the $ L^{th} $ layer, we can calculate the gradient by the following
way\footnote{the operator $*$ is element-wise multiplication}
\footnote{the update of parameters in GD involve the complete data set,
while in the variant of GD like SGD and mini-batch GD, only part of the data
contribute to the update process in each iteration}:
\begin{equation}
    \begin{split}
        dA^{[L]} & = \frac{\partial J}{\partial A^{[L]}} \\
        dZ^{[L]} & = dA^{[L]}*g^{[L]'}(Z^{[L]}) \\
        dW^{[L]} & = \frac{1}{m}dZ^{[L]}A^{[L-1]T} \\
        db^{[L]} & = \frac{1}{m}dZ^{[L]} \\
        dA^{[L-1]} & = W^{[L-1]T}dZ^{[L]} \\
        dZ^{[L-1]} & = dA^{[L-1]}*g^{[L-1]'}(Z^{[L-1]}) \\
        & \vdots
    \end{split}
\end{equation}
Suppose the loss function of the output layer is binary cross-entropy: \\
$ l(a^{[L]},\ y) = - y\ln(a^{[L]}) - (1-y)\ln(1-a^{[L]}) $, then
the BP process can be written as:
\begin{equation}
    \label{equ:BP}
    \begin{split}
        dZ^{[L]} & = A^{[L]} - y \\
        dW^{[L]} & = \frac{1}{m}dZ^{[L]}A^{[L-1]T} \\
        db^{[L]} & = \frac{1}{m}dZ^{[L]} \\
        & \vdots \\
        dA^{[l]} & = W^{[l]T}dZ^{[l+1]} \\
        dZ^{[l]} & = (W^{[l]T}dZ^{[l+1]})*g^{[l]'}(Z^{[l]}) \\
        dW^{[l]} & = \frac{1}{m}dZ^{[l]}A^{[l-1]T} \\
        db^{[l]} & = \frac{1}{m}dZ^{[l]}
    \end{split}
\end{equation}


\begin{frame}
  \frametitle{Introduction}

  \bitvectors are extremely useful data structures,
  used to symbolically represent hardware and software
  constructs (see later)
  \vfill
  \pause
  The world of \bitvectors is a {\bf finite} world, i.e.,
  with \bitvectors it is not possible to represent/handle
  arbitrarily large numbers
  \vfill
  \pause
  Indeed, when speaking about \bitvectors we always
  associate a {\bf width}
  (which is usually a power of 2, often 32 or 64)
  \vfill
  \pause
  The width specifies the (maximum) {\bf number of bits} used to
  represent variables and terms
  \vfill
  \pause
  \bitvector \formulae are mathematically characterized by
  the theory of \bitvectors \Bitvectors

\end{frame}

\subsection{Syntax}

\begin{frame}
  \frametitle{Bit-Vectors}

  A bit-vector is an array of bits
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab.pdf_t}}
  \end{center}
  \vfill
  \pause

  Selection (or Extraction): $\w{a}{3}[1:0]$
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_a_sel.pdf_t}}
  \end{center}

  \vfill
  \pause

  Notice that
  \begin{itemize}
    \item $\w{a}{n}[i:j]$ returns a \bitvector of width $i - j + 1$ ($0 \leq j \leq i \leq n - 1$) \pause
    \item $\w{a}{n}[n-1:0]$ \pause $= \w{a}{n}$ \pause
    \item Selection has precedence over any other operator
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Bit-Vectors}

  A bit-vector is an array of bits
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab.pdf_t}}
  \end{center}
  \vfill

  Concatenation $\w{a}{3} :: \w{b}{3}$
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab_conc.pdf_t}}
  \end{center}

  \vfill
  \pause

  Notice that
  \begin{itemize}
    \item $\w{a}{n} :: \w{b}{m}$ returns a \bitvector of width $n+m$ \pause
    \item $\w{a}{n}[n-1:i] :: \w{a}{n}[i-1:0]$ \pause $= \w{a}{n}[n-1:0]$ \pause $= \w{a}{n}$
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Bit-Vectors}

  A bit-vector is an array of bits
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab.pdf_t}}
  \end{center}
  \vfill

  Arithmetic $\w{a}{3} + \w{b}{3}$
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab_plus.pdf_t}}
  \end{center}

  \vfill
  \pause

  Notice that
  \begin{itemize}
    \item To be precise, we should have written $\w{a}{3} +_{[3]} \w{b}{3}$ (widths must be the same)
    \item Semantic is that of {\bf modular} arithmetic
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Bit-Vectors}

  A bit-vector is an array of bits
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab.pdf_t}}
  \end{center}
  \vfill

  Bitwise $\w{a}{3} \band \w{b}{3}$
  \smallskip \\
  \begin{center}
  \scalebox{.3}{\input{bv_ab_bitw.pdf_t}}
  \end{center}

  \vfill
  \pause

  Notice that
  \begin{itemize}
    \item Again, to be precise, we should have written $\w{a}{3} \band_{[3]} \w{b}{3}$ (widths must be the same)
    \item Used to compute bit-mask operations
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{A (non-exhaustive) list of operators and predicates}

  Each \bitvector term of width $n$, is associated with a sort \SBv{n} ($n \geq 1$)
  \vfill
  \pause

  \begin{center}
  \begin{tabular}{|l|l|l|l|}
    \hline
    Name                 & Symb      & Type                        & Signature \\
    \hline
    Selection            & $\_[i:j]$ & \multirow{2}{*}{Core}       & \SBv{n}                  $\rightarrow$ \SBv{i-j+1} \\
    Concatenation        & $::$      &                             & \SBv{n} $\times$ \SBv{m} $\rightarrow$ \SBv{n+m} \\
    \hline
    Addition             & $+$       & \multirow{5}{*}{Arith.}     & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBv{n} \\
    Subtraction          & $-$       &                             & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBv{n} \\
    Multiplication       & $*$       &                             & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBv{n} \\
    Less than (signed)   & $<_s$     &                             & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBoo \\
    Less than (unsigned) & $<_u$     &                             & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBoo \\
    \hline
    Bitwise and          & $\band$   & \multirow{3}{*}{Bitwise}    & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBv{n} \\
    Bitwise or           & $\bor$    &                             & \SBv{n} $\times$ \SBv{n} $\rightarrow$ \SBv{n} \\
    Bitwise not          & $\bnot\_$ &                             & \SBv{n}                  $\rightarrow$ \SBv{n} \\
    \hline
  \end{tabular}
  \end{center}
  \vfill
  \pause

  Moreover, we have constants, e.g., $101101_{[6]}$

\end{frame}

\subsection{Semantic}

\begin{frame}
  \frametitle{\bitvector semantic}

  \scriptsize

  Each sort \SBv{n} is associated with a domain $D_n = \{ 0, 1, \ldots, 2^{n-1} \}$ \\ \pause
  For example \SBv{4} is associated with $D_{4} = \{ 0, 1, \ldots, 15 \}$
  \vfill
  \pause
  As usual, the semantic for the other terms depends on a particular {\bf assignment}
  to the variables
  \vfill
  \pause
  Each variable $\w{x}{n}$ is associated with a function \inter{\w{x}{n}}
  of type $D_n \rightarrow \{ 0, 1 \}$
  \vfill
  \pause
  \scalebox{.7}{\input{semantic_example.pdf_t}}
  \vfill
  \pause
  $nat_n(\_)$ is a helper meta-function, to facilitate the presentation

\end{frame}

\begin{frame}
  \frametitle{\bitvector semantic}

  \scriptsize
  $$
  \begin{array}{rcl}
    %% Constant
    \inter{\w{c}{n}} & := & \lambda x \in [0,n-1].\,
    \left\{
    \begin{array}{ll}
      0 & \mbox{ if the } x\mbox{-th bit is } 0 \\
      1 & \mbox{ otherwise }
    \end{array}
    \right. \\
    \medskip \\
    %% CONCATENATION
    \inter{\w{t}{l} :: \w{s}{k}} & := & \lambda x \in [0,\ldots,l+k-1].\,
    \left\{
    \begin{array}{ll}
      \inter{\w{s}{n}}(x) & \mbox{ if } x < l \\
      \inter{\w{t}{n}}(x-l) & \mbox{ otherwise }
    \end{array}
    \right. \\
    \smallskip \\
    %% SELECTION
    \inter{\w{t}{n}[i:j]} & := & \lambda x \in [0,i-j+1].\, \inter{\w{t}{n}}(x+j) \\
    \medskip \\
    %% PLUS
    \inter{\w{t}{n} + \w{s}{n}} & := & nat_n^{-1}(nat_n(\inter{\w{t}{n}}) + nat_n(\inter{\w{s}{n}}))\ \%\ 2^n \\
    \medskip \\
    %% BITAND
    \inter{\w{t}{n} \band \w{s}{n}} & := & \lambda x \in [0,n-1].\,
    \left\{
    \begin{array}{ll}
    0 & \mbox{ if } \inter{\w{t}{n}}(x) = 0 \\
    0 & \mbox{ if } \inter{\w{s}{n}}(x) = 0 \\
    1 & \mbox{ otherwise }
    \end{array}
    \right. \\
    \medskip \\
    %% LESS THAN
    \inter{\w{t}{n} <_u \w{s}{n}} & := &
    \left\{
    \begin{array}{ll}
    \top & \mbox{ if } nat_n(\inter{\w{t}{n}}) <_{u}
                       nat_n(\inter{\w{s}{n}}) \\
    \bot & \mbox{ otherwise }
    \end{array}
    \right.
  \end{array}
  $$

\end{frame}


\chapter{Statistical methods I}

\section{Entrophy of source units}

Entrophy is the amount of information contained in a unit measurred in bits. The more unexpeted the unit is, the higher is the amount of information it carries.

There are several basic terms:

\noindent
Source units:
$$S = \{x_1, x_2, \ldots, x_n\}.$$
Source unit probabilities:
$$P = \{p_1, p_2, \ldots, p_n\}.$$
Source unit frequencies\sidenote{Frequencies are usually associated with actual messages and their analysis.}:
$$F = \{f_1, f_2, \ldots, f_n\}.$$
Entropy (information content) of unit $x_i$\sidenote{How much bits of actual information unit represents.}:
$$ H_i = - \log_2 p_i.$$
Average entrophy of a source unit from S\sidenote{How much bits of actual information represents a unit on average.}:
$$H_{avg}(S) = \sum_{i=1}^{n}{p_i H_i} = -\sum_{i=1}^{n}{p_i \log_2 p_i}.$$

%\section{Entrophy of a message}

%Message
%$$ X = x_1, x_2, \ldots, x_n,$$

%Entrophy of source message
%Entrophy of encdoded message
%Length of encoded message
%$$L(X) = $$

\section{Entrophy of a code}
To each source unit, a codeword can be assigned.

\noindent
Codewords (code units):
$$C = \{c_1, c_2, \ldots, c_n\}.$$
Average length of a codeword\sidenote{How many actual bits are used for encoding a source unit into a codeword on average.}:
$$L_{avg}(C) = \sum_{i=1}^{n}{p_i |c_i|}.$$

\section{Propertis of codes}

Given the following source units, their probabilities and codewors, the average entrophy and the average length of the codeword can be calculated:

\noindent
Source units:
$$S = \{\texttt{a}, \texttt{b}, \texttt{c}, \texttt{d}, \texttt{e}\}.$$
Source unit probabilities:
$$P = \{0.1, 0.15, 0.3, 0.16, 0.29\}.$$
Codewords:
$$C = \{010, 011, 11, 00, 10\}.$$

\begin{table}
  \begin{center}
    \begin{tabular}{|r|cccccc|}
      \hline
      $x_i$ & $p_i$ & $c_i$ & $|c_i|$ & $H_i$ & $H_{avg}(S)$ & $L_{avg}(C)$      \\
          &        &        &          & $-\log_2 p_i$ & $p_i H_i$ & $p_i |c_i|$      \\
      %      & unit & probability & code & code length & unit entrophy & average entrophy & average code length      \\
      \hline
      a          & 0.10   & 010    & 3        & 3.32  & 0.332        & 0.30              \\
      b          & 0.15   & 011    & 3        & 2.74  & 0.411        & 0.45              \\
      c          & 0.30   & 11     & 2        & 1.74  & 0.521        & 0.60              \\
      d          & 0.16   & 00     & 2        & 2.65  & 0.423        & 0.32              \\
      e          & 0.29   & 10     & 2        & 1.79  & 0.518        & 0.58              \\
      \hline
      $\sum$     & 1.00   & -      & -        & -     & 2.205        & 2.25              \\
      \hline
    \end{tabular}
  \end{center}
  \caption{Code $C$ for source units $S$ with probabilities $P$}
\end{table}

The closer is the average length of the codeword to the actual entrophy, the better is code performing.

%Prefix code - no code word is a prefix of the other code word.
%Uniqualy decodable (biunique) - no coded string has more than one decoding.
%Kraft's inequality - necessary and sufficient condition for the existence of a prefix code.

\begin{dt}{Prefix code.}
  A prefix code is a code where no codeword is a prefix of a different codeword.
\end{dt}

\begin{dt}{Biunique (uniquely decodable) code.}
  A biunique code is a code where no encoded message has more than one decoding.
\end{dt}

Code $C = \{010, 011, 11, 00, 10\}$ above is uniquely decodable prefix code as no codeword is a prefix of another codeword. Uniquely decodable code which is not a prefix code can be easily created by doing a reverse of each codeword of code $C$. $C^R = \{010, 110, 11, 00, 01\}$ is not a prefix code\sidenote{Code $C^R$ is actually a suffix code.} because $11$ is a prefix of $110$ and $01$ is a prefix of $010$.

Code $C_N = \{010, 011, 11, 00, 01\}$ is not uniquely decodable code as sequence $010011$ can be interpreted in several way as $010.011$ or $01.00.11$.

All prefix codes are uniquely decodable codes. Some uniquely decodable codes are prefix codes.\sidenote{In practice, mostly prefix codes are used.}

\section{Sample source units and their probabilities and frequencies}

These source units and their frequencies will be used in the following examples:

\noindent
Source units:
$$S_1 = \{\texttt{a}, \texttt{b}, \texttt{c}, \texttt{d}, \texttt{e}, \texttt{f}, \texttt{g}, \texttt{h}, \texttt{i}\}.$$
Source unit probabilities:
$$P_1 = \{1/35, 1/35, 2/35, 3/35, 3/35, 5/35, 5/35, 7/35, 8/35\}.$$
Source unit frequencies:
$$F_1 = \{1, 1, 2, 3, 3, 5, 5, 7, 8\}.$$

\forestset{angled/.style={content/.expanded={\noexpand\textless\forestov{content}\noexpand\textgreater}}}
\tikzset{el style/.style={midway, font=\scriptsize, inner sep=+1pt, auto=right}}
\tikzset{every node/.style={draw, circle}}
\tikzset{every edge/.style={align=center, base=top}}

\section{Shannon-Fano Coding}
Shannon-Fano Coding creates codewords by sorting units by their number of frequencies (probabilities). In each step the number of frequencies is split to two closest parts and new nodes are created. The nodes are created from the root which contains the sum of all frequencies.

\begin{figure}
\begin{forest}
  for tree={child anchor=north,inner sep=6pt},
  where n children={1}{tier=word}{},
  where n children={0}{rectangle,draw=none,minimum size=0.7cm}{
    if={n==1}{% n == 1 means first child
      edge label={node[el style, draw=none, swap, swap, near start]{0}}
    }{
      edge label={node[el style, swap, draw=none, near start]{1}}
    }
  }
%
[35 [ 15 [ 7 [ 4 [ 2 [ 1 [ a ] ]
                     [ 1 [ b ] ] ]
                 [ 2 [ c ] ] ]
             [ 3 [ d ] ] ]
         [ 8 [ 3 [ e ] ]
             [ 5 [ f ] ] ] ]
    [ 20 [12 [ 5 [ g ] ]
             [ 7 [ h ] ] ]
          [8 [ i ] ] ] ]
\end{forest}
\caption{Shannon-Fano tree for code $C_1$ for source units $S$ with probabilities $P$}
\end{figure}

Edges to the left and to the right child nodes are labeled 0 and 1 respectively. The codewords can be red from root node to each leaf node.

From the tree above, the codewords are:
$$C_1 = \{\texttt{00000}, \texttt{00001}, \texttt{0001}, \texttt{001}, \texttt{010}, \texttt{011}, \texttt{100}, \texttt{101}, \texttt{11}\}.$$

\begin{table}
  \begin{center}
    \begin{tabular}{|r|cccccc|}
      \hline
      $x_i$ & $p_i$ & $c_i$ & $|c_i|$ & $H_i$ & $H_{avg}(S)$ & $L_{avg}(C)$      \\
          &        &        &          & $-\log_2 p_i$ & $p_i H_i$ & $p_i |c_i|$      \\
      \hline
      a          & 0.029   & 00000    & 5        & 5.13  & 0.147        & 0.143              \\
      b          & 0.029   & 00001    & 5        & 5.13  & 0.147        & 0.143              \\
      c          & 0.057   & 0001     & 4        & 4.13  & 0.236        & 0.229              \\
      d          & 0.086   & 001      & 3        & 3.54  & 0.304        & 0.257              \\
      e          & 0.086   & 010      & 3        & 3.54  & 0.304        & 0.257              \\
      f          & 0.143   & 011      & 3        & 2.81  & 0.401        & 0.429              \\
      g          & 0.143   & 100      & 3        & 2.81  & 0.401        & 0.429              \\
      h          & 0.200   & 101      & 3        & 2.32  & 0.464        & 0.600              \\
      i          & 0.229   & 11       & 2        & 2.19  & 0.487        & 0.457              \\
      \hline
      $\sum$     & 1.00    & -        & -        & -     & 2.891        & 2.944              \\
      \hline
    \end{tabular}
  \end{center}
  \caption{Code $C_1$ for source units $S_1$ with probabilities $P_1$}
\end{table}

Sometimes parts can be created differently, leading to different codewords. In our case, in the first step the number of frequencies can be split either to $1, 1, 2, 3, 3, 5$ and $5, 7, 8$ or  $1, 1, 2, 3, 3, 5, 5$ and $7, 8$ leading to $15 + 20$ or $20 + 15$. It has to be decided beforehand how to resolve this situation (either going with lower on left or lower on right).

\begin{figure}
\begin{forest}
  for tree={child anchor=north,inner sep=6pt},
  where n children={1}{tier=word}{},
  where n children={0}{rectangle,draw=none,minimum size=0.7cm}{
    if={n==1}{% n == 1 means first child
      edge label={node[el style, draw=none, swap, swap, near start]{0}}
    }{
      edge label={node[el style, swap, draw=none, near start]{1}}
    }
  }
%
[35 [ 20 [ 10 [ 4 [ 2 [ 1 [ a ] ]
                      [ 1 [ b ] ] ]
                  [ 2 [ c ] ] ]
              [ 6 [ 3 [ d ] ]
                  [ 3 [ e ] ] ] ]
         [ 10 [5 [ f ] ]
              [5 [ g ] ] ] ]
    [ 15 [ 7 [ h ] ]
         [ 8 [ i ] ] ] ]
\end{forest}
\caption{Huffman tree for code $C_2$ for source units $S$ with probabilities $P$}
\end{figure}

From the tree above, the codewords are:
$$C_2 = \{\texttt{00000}, \texttt{00001}, \texttt{0001}, \texttt{0010}, \texttt{0011}, \texttt{010}, \texttt{111}, \texttt{10}, \texttt{11}\}.$$

\begin{table}
  \begin{center}
    \begin{tabular}{|r|cccccc|}
      \hline
      $x_i$ & $p_i$ & $c_i$ & $|c_i|$ & $H_i$ & $H_{avg}(S)$ & $L_{avg}(C)$      \\
          &        &        &          & $-\log_2 p_i$ & $p_i H_i$ & $p_i |c_i|$      \\
      \hline
      a          & 0.029   & 00000    & 5        & 5.13  & 0.147        & 0.143              \\
      b          & 0.029   & 00001    & 5        & 5.13  & 0.147        & 0.143              \\
      c          & 0.057   & 0001     & 4        & 4.13  & 0.236        & 0.229              \\
      d          & 0.086   & 0010     & 4        & 3.54  & 0.304        & 0.343              \\
      e          & 0.086   & 0011     & 4        & 3.54  & 0.304        & 0.343              \\
      f          & 0.143   & 010      & 3        & 2.81  & 0.401        & 0.429              \\
      g          & 0.143   & 111      & 3        & 2.81  & 0.401        & 0.429              \\
      h          & 0.200   & 10       & 2        & 2.32  & 0.464        & 0.400              \\
      i          & 0.229   & 11       & 2        & 2.19  & 0.487        & 0.457              \\
      \hline
      $\sum$     & 1.00    & -        & -        & -     & 2.891        & 2.916              \\
      \hline
    \end{tabular}
  \end{center}
  \caption{Code $C_2$ for source units $S_1$ with probabilities $P_1$}
\end{table}

\section{Huffman Coding}
Huffman Coding creates codewords by creating new node from the two smallest nodes (sorting is usually involved). In each step the two lowest numbers of frequencies are summed together and a new node is created containing this sum as a parent of these two nodes. The nodes are created from the leaves which are created from the frequencies of the source units.

\begin{figure}
\begin{forest}
  for tree={child anchor=north,inner sep=6pt},
  where n children={1}{tier=terminus}{},
  where n children={0}{rectangle,draw=none,minimum size=0.7cm}{
  }
%i
[35 [ 20,edge label={node[el style, draw=none]{1}},name=N16 [9,edge label={node[el style, draw=none]{0}},name=N9 [ 4,edge label={node[el style, draw=none]{0}} [ 2,edge label={node[el style, draw=none]{0}} [ 1,edge label={node[el style, draw=none]{0}} [ a ] ]
                   [ 1,edge label={node[el style, draw=none, swap]{1}} [ b ] ] ]
                 [ 2,edge label={node[el style, draw=none, swap]{1}} [ c ] ] ]
                 [\phantom{0},draw=none,no edge] ]
    [11,edge label={node[el style, draw=none, swap]{1}} [ 6,edge label={node[el style, draw=none, near start]{1}} [ 3,edge label={node[el style, draw=none]{0}} [ d ] ]
                 [ 3,edge label={node[el style, draw=none, swap, near end]{1}} [ e ] ] ]
             [ 5,name=N5,no edge [ f ] ]
             [ 5,edge label={node[el style, draw=none, swap, near start]{0}} [ g ] ] ] ]
    [ 15,edge label={node[el style, draw=none, swap]{0}}
         [ 7,edge label={node[el style, draw=none]{0}} [ h ] ]
         [ 8,edge label={node[el style, draw=none, swap]{1}} [ i ] ] ] ]
  \draw (N9.south east) -- (N5.north) node[el style, swap, draw=none, near start]{1}; % or use (NP.north)
\end{forest}
\caption{Huffman tree for code $C_3$ for source units $S$ with probabilities $P$}
\end{figure}

Edges to the lower and higher value child nodes are labeled 0 and 1 respectively. If a tie occures left and right child nodes are labeled 0 and 1 respectively.

From the tree above, the codewords are:
$$C_3 = \{\texttt{10000}, \texttt{10001}, \texttt{1001}, \texttt{1110}, \texttt{1111}, \texttt{101}, \texttt{110}, \texttt{00}, \texttt{01}\}.$$

\begin{table}
  \begin{center}
    \begin{tabular}{|r|cccccc|}
      \hline
      $x_i$ & $p_i$ & $c_i$ & $|c_i|$ & $H_i$ & $H_{avg}(S)$ & $L_{avg}(C)$      \\
          &        &        &          & $-\log_2 p_i$ & $p_i H_i$ & $p_i |c_i|$      \\
      \hline
      a          & 0.029   & 10000    & 5        & 5.13  & 0.147        & 0.143              \\
      b          & 0.029   & 10001    & 5        & 5.13  & 0.147        & 0.143              \\
      c          & 0.057   & 1001     & 4        & 4.13  & 0.236        & 0.229              \\
      d          & 0.086   & 1110     & 4        & 3.54  & 0.304        & 0.343              \\
      e          & 0.086   & 1111     & 4        & 3.54  & 0.304        & 0.343              \\
      f          & 0.143   & 101      & 3        & 2.81  & 0.401        & 0.429              \\
      g          & 0.143   & 110      & 3        & 2.81  & 0.401        & 0.429              \\
      h          & 0.200   & 00       & 2        & 2.32  & 0.464        & 0.400              \\
      i          & 0.229   & 01       & 2        & 2.19  & 0.487        & 0.457              \\
      \hline
      $\sum$     & 1.00    & -        & -        & -     & 2.891        & 2.916              \\
      \hline
    \end{tabular}
  \end{center}
  \caption{Code $C_3$ for source units $S_1$ with probabilities $P_1$}
\end{table}

\begin{dt}{Kraft-McMillan's Inequality.}
    The Kraft-McMillan's Inequality defined as\sidenote{The Kraft-McMillan's Inequality is derived from the Huffman tree, each leaf represents a codeword and the depth of the leaf correspond to the length of that codeword.}: $$\sum_{\ell \in \mathrm{leaves}} 2^{-\mathrm{depth}(\ell)} \leq 1.$$
    It is necessary and sufficient condition for the existence of a prefix code with given properties. If properties of the code given follows Kraft-McMillan's Inequality, it can be uniquely decodable code. If properties of the code given does not follow Kraft-McMillan's Inequality, it cannot be uniquely decodable code.
\end{dt}

Code $C_N = \{010, 011, 11, 00, 01\}$ above gives you $1/8 + 1/8 + 1/4 + 1/4 + 1/4 = 1$ which means that code with such length of codeword could be uniquely decodable code, but it is not.
Code $C_{N2} = \{00, 01, 10, 11, 000\}$ gives you $1/4 + 1/4 + 1/4 + 1/4 = 1.125$ hence it cannot be uniquely decodable code.

\begin{dt}{Redundant code.}
  A prefix code for which holds $$\sum_{\ell \in \mathrm{leaves}} 2^{-\mathrm{depth}(\ell)} < 1.$$
\end{dt}

Code $C_{R} = \{00, 01, 10, 110\}$ gives you $1/4 + 1/4 + 1/4 + 1/8 = 9.875$ hence the code is redundant (as it is also uniquely decodable).

\begin{dt}{Complete code.}
  A prefix code for which holds $$\sum_{\ell \in \mathrm{leaves}} 2^{-\mathrm{depth}(\ell)} = 1.$$
\end{dt}

Code $C = \{010, 011, 11, 00, 10\}$ above gives you $1/8 + 1/8 + 1/4 + 1/4 + 1/4 = 1$ hence the code is complete (as it is also uniquely decodable).

\begin{dt}{Optimal code.}
    A code is optimal (for a given probability distribution) if no other code with a lower average length of a codeword $L_{avg}$ exists.
\end{dt}

You can compare average length of codewords in examples above for Shannon-Fano Codes $C_1$ and $C_2$ and Huffman Code $C_3$. Huffman Coding always produces optimal code ($C_3$). Shanno-Fano Coding is not guaranteed to produce optimal code ($C_1$), but can produce optimal code ($C_2$).

Optimal code is always complete code.

\section{Homework}

\begin{itemize}
  \item Try to create Shannon-Fano Coding and Huffman Coding for different sets of frequencies.
  \item Try to calculate Kraft-McMillan's Inequality for the created sets of codewords (included those presented here).
  \item Compare average codeword length of the created sets of coders and see that Shannon-Fano does not always produce an optimal code.
\end{itemize}