In this section we present our causal estimand, identifying assumptions, estimation strategy, and inferential procedure.

\subsection{Estimand}

Our goal is to estimate the average effect 2014 Medicaid expansion would have had on the non-elderly adult uninsurance rate in states that did not expand Medicaid. Let $A$ indicate treatment assignment, $c$ index a CPUMA, $s$ index the state, and $t$ index the time period. Let $n_1$ be the number of treated CPUMAs, $n_0$ be the number of control CPUMAs, and $n$ be the total number of CPUMAs. Similarly, let and $m = m_1 + m_0$ states (with $m_1$ and $m_0$ defined analogously). Each state has $p_s$ CPUMAs. $A_s=0$ indicates untreated states and $A_s=1$ indicates treated states, when this notation is a superscript if it indicates the potential outcome. Since we are only interested in the counterfactual at time $T = 2014$, we simplify notation by removing this variable and the subscript and write our formal estimand as:

\begin{equation}
\psi = \psi^1 - \psi^0 &= n_0^{-1}\sum_{s, c: A_s = 0} Y_{sc}^{A_s = 1} - Y_{sc}^{A_s = 0}
\end{equation}

The challenge is that we do not observe the counterfactual outcomes for non-expansion CPUMAs had they been in states that expanded their Medicaid programs. We therefore require causal assumptions to tie this counterfactual quantity to our observed data.\footnote{The 2014 Medicaid expansion occurred simultaneously with the implementation of several other major ACA provisions, including (but not limited to) the creation of the ACA-marketplace exchanges, the individual mandate, health insurance subsidies, and community-rating and guaranteed issue of insurance plans (\cite{courtemanche2017early}). Almost all states broadly implemented these reforms beginning January 2014. Conceptually we think of the other ACA components as a state-level treatment ($R$) separate from Medicaid expansion ($A$). Therefore, our total estimated effect may also include interactions between these policy changes; however, we do not attempt to separately identify these effects. Because the ACA implementation and Medicaid expansion may vary over time, we do not try to generalize these results beyond 2014.}

\subsection{Identification}

The following causal assumptions are necessary (though insufficient) to identify our target parameter from our observed data: the stable unit treatment value assumption (SUTVA), no unmeasured confounding, and no anticipatory treatment effects. We explain these assumptions in detail and their consequences below. We additionally invoke several parametric assumptions to help us identify our causal parameter given the measurement error in our covariates. These assumptions in total are sufficient to identify our causal estimand.

SUTVA has two implications: first, that there is only one version of treatment; second, that $Y_{sc}^{\mathbf{a}} = Y_{sc}^{\mathbf{a}'}$ when $a_{sc} = a_{sc}'$, where $\mathbf{a}$ is the vector of all treatment assignments. We discussed potential violations of the first assumption previously when considering how to reduce Medicaid Expansion to a binary treatment classification. Our solution is to remove states with less restrictive Medicaid eligibility requirements prior to 2014 to approximately satisfy this condition. The second part of this assumption implies that the potential outcomes in each region do not depend on another region's treatment assignment. This is a standard assumption, but is often not realistic in practice. Violations are likely in our setting: for example, \cite{frean2017premium} find evidence that Medicaid expansion drove previously eligible but uninsured individuals to enroll in Medicaid in both expansion and non-expansion states. Signing the potential bias from this violation requires redefining the causal estimand: for example, we might consider the treatment effect on the untreated given that all states have expanded Medicaid, where the contrast is against where only the observed expansion states expanded Medicaid. If the spillover effects were equal in each region, and the magnitude of the spillovers increase with the total number of treated regions, then the true effect would be larger in absolute magnitude than the estimated effect using the observed data. We could consider other estimands or assumptions to get different predictions about the sign of the bias; however, this is beyond the scope of this paper.

We next assume that there were no anticipatory treatment effects. Letting treatment occur at time $T$, we have that for $t < T$:

\begin{align*}
Y_{sct} = Y_{sct}^0
\end{align*}

This assumption is necessary because we will condition on pre-treatment outcomes. If these outcomes were affected by the treatment before it were implemented, these covariates would be endogenous. Anticipatory treatment effects may occur if plans to expand Medicaid induce uninsured but Medicaid-eligible individuals to enroll in Medicaid prior to expansion. We do not think these violations occurred in large enough numbers to substantially affect our results. Instead, we address a more concerning version of this violation: the fact that several states allowed certain counties to expand Medicaid prior to 2014. We test the sensitivity of our results to the exclusion of these states.

Third, we assume no unmeasured confounding; that is, that at time $T$ the potential outcomes for each CPUMA are independent of the state-level treatment assignment conditional on the population-level CPUMA and state-level covariates $X_{sc}$, a $q$ dimensional vector of covariates (which includes pre-treatment outcomes):

\begin{align*}
Y_{sc}^a \perp A_{sc} \mid X_{sc}
\end{align*}

While unverifiable, we believe it is reasonable here given our rich covariate set. To be explicit, we believe that the pair of potential uninsurance rates for each CPUMA are independent of the treatment assignment conditional on the percentage of uninsured individuals in each year of the pre-treatment period, the percentage of unemployed individuals in each year of the pre-treatment period, the average population growth, the average ratio of households to non-elderly adult population, the state's political composition, the average proportion of households with one, two, or three or more children during the pre-treatment period, the average proportion of households who did not respond about their number children, and the average proportion of individuals during the pre-treatment period with given demographics noted above (age group, sex, white, Hispanic ethnicity, U.S. citizenship, foreign born, income-to-poverty group (including non-response), disability status, urban residence, and educational attainment group).

A key problem we address in this paper is the violation of this assumption due to measurement error in our covariates. Let
$X = (X_0, X_1)^T$ be the $n$ by $q$ matrix of true covariates (and separating the control from treated units using $X_a$). Because our covariates are estimated using the ACS data, rather than $(Y, X)$, we instead observe $(J,W)$, which consist of estimates of the true outcome and covariate values, where $W$ is structured analogously to $X$. Importantly, $Y_{sc}^a \perp A_{sc} \mid X_{sc} \centernot\implies J_{sc}^a \perp A_{sc} \mid W_{sc}$. The use of these proxies may therefore bias our estimates. We rely on several modeling assumptions to correct for this.

We first model our observed data as functions of the true values plus mean-zero Gaussian noise: $J_{sc} = Y_{sc} + \xi_{sc}$ and $W_{sc} = X_{sc} + v_{sc}$, where we assume $\xi_{sc}$ and $v_{sc}$ are independent though not identically distributed.\footnote{Our covariates are almost all ratio estimates, which will in general be biased. This bias, however, decreases quickly with the sample size (is $O(n^{-1})$). Given that our CPUMA sample sizes are all over 300, we treat these estimates as unbiased in our analysis.} We assume that these errors are uncorrelated with the true values, i.e. $\mathbb{E}\{\xi_{sc} \mid Y_{sc}\} = 0$ (and similarly for all elements of $X$). Second, we assume that $\xi_{sc}$ are uncorrelated with the errors in the covariate measurements. These assumptions and our model for the observed data are reasonable given that the measurement error in this context is sampling variability. Moreover, our outcomes are measured on a different cross-section than our covariates, so it is reasonable to assume that they are uncorrelated with the measurement errors in the covariates.

We next assume that the true potential outcomes are linear in the true covariates $X_{sc}$. Specifically, we assume that the following model generates the potential non-elderly adult uninsurance rate under treatment $A = a$:

\begin{equation}\label{eqn:outcomemodel}
Y_{sc}^a = \alpha_a + X_{sc}^T\beta_a + \epsilon_{sc} + c_s
\end{equation}

We assume that the errors $\epsilon_{sc}$ and $c_s$ are mean-zero, independent from each other and across time (i.e., we rule out serial-correlation), and are uncorrelated with the true covariates and the treatment assignment, i.e. $\mathbb{E}\{\epsilon_{sc}c_s \mid X_{sc}, A_s\} = \mathbb{E}\{\epsilon_{sc} \mid X_{sc}, A_s\} = \mathbb{E}\{c_s \mid X_{sc}, A_s\} = 0$. We can then identify $\psi^a$ in terms of our model parameters (see Appendix A); specifically, we have that $\psi^a = \alpha_a + \bar{X}_0^T\beta_a$, where $\bar{X}_0$ is the vector of mean covariate values among the control units. Moreover, we can substitute $J_{sc}$ for $Y_{sc}$ in Equation~\ref{eqn:outcomemodel}, and add $\xi_{sc}$ to the error term without affecting identification.

We still have the problem that we observe $W_{sc}$ instead of $X_{sc}$. Let $\eta_a = \mathbb{E}\{X_{sc} \mid W_{sc}, A_s = a\}$. By linearity, we know that

\begin{equation}
    J_{sc} = \eta_a(W_{sc})^T\beta + (X_{sc} - \eta_a(W_{sc}))^T\beta + \xi_{sc} + \epsilon_{sc} + c_s
\end{equation}

If we knew $\eta_a$, we could estimate this model using the observed data $(J, W)$. The approach we follow here is known as ``regression calibration'' in the measurement-error literature. In particular, we assume a linear model for $\eta_a$:

\begin{align*}
\eta_a(W_{sc}) = \upsilon_a + \kappa_a^T(W_{sc} - \upsilon_a)
\end{align*}

where $\kappa_a = (\Sigma_{XX \mid A = a} + \Sigma_{vv \mid A = a})^{-1}\Sigma_{XX \mid A = a}$. The assumption motivating this model is that $(X_{sc}, v_{sc}) \stackrel{iid}\sim MVN((\upsilon_a, 0), \Sigma_a)$ and $\Sigma_a$ is a $2q$ by $2q$ block-diagonal matrix consisting of $q$ by $q$ matrices $\Sigma_{XX \mid A = a}$ and $\Sigma_{vv \mid A = a}$ and $0$ in the off-diagonals. Given sufficient auxillary data to estimate $\kappa_a$, we can then estimate $\psi$. We discuss this further below and in Appendices A and B (see also \cite{gleser1992importance}).

\subsection{Estimation}

We outline our estimation strategy first emphasizing how estimating the ETC differs from estimating the ETT with respect to variable selection under the ``synthetic controls'' framework and associated assumptions required. Second, we explain our estimation procedure, which modifies the SBW criterion to address the hierarchical data structure. This objective, which we call H-SBW, reduces the variance of our estimator under our assumption of constant variance and constant within-state correlation of model errors. Third, we connect our estimator to the regression calibration literature by generating weights that balance a linear prediction of the true covariates $\hat{\eta}_a(W_{sc})$ using the observed covariates $W_{sc}$. Fourth, we test the sensitivity of our estimator to a regression-augmented version, using ridge-regression weights following the suggestion of \cite{ben2018augmented}; this allows us to achieve better covariate balance by extrapolating beyond the support of the data. We conclude by proposing a model validation procedure that uses pre-treatment outcomes to compare the performance of our estimators on pre-treatment data.

\subsubsection{Variable selection}

We seek to generate a set of positive weights that balance the means of covariates for the treated units to the mean covariates for the control units. Assume that we observe the true covariate matrices for the treated data $X_a = (X_{a,1}, ..., X_{a, q})$, and the true outcomes $Y$. Let $\bar{X}_{0, r}$ be the mean covariate value for the $r$-th covariate in the non-expansion region. Ideally, there exists some $\gamma^\star \in \Gamma$ satisfying:

\begin{equation}\label{eqn:constraint}
\Gamma = \{\gamma \in \mathbb{R}^{n_1}: &\lvert \gamma^TX_{1, r} - \bar{X}_{0, r} \lvert \le \delta_r \ \ (r = 1, ..., q), \ \gamma_{sc} > 0, \sum_{s, c: A_{sc} = 1}\gamma_{sc} = 1\}
\end{equation}

for $\delta_r = 0$ for all $r = 1, ..., q$. We could then estimate $\psi$ as

\begin{equation}\label{eqn:psi}
\hat{\psi} = \sum_{s: A_s = 1}^{m_1}\sum_{c = 1}^{p_s}\gamma_{sc}^\star Y_{sc} - n_0^{-1}\sum_{s: A_s = 0}^{m_0}\sum_{c = 1}^{p_s}Y_{sc}
\end{equation}

Again assuming that the potential outcomes are a linear function of the true covariates: $\mu_a(X_{sc}) = \alpha_a + X_{sc}^T\beta_a$, the bias of our estimate of $\psi^1$ (again assuming we observed $X_{sc}$), is less than or equal to $\lvert\beta_1\rvert^T\delta = 0$ (see, e.g., \cite{zubizarreta2015stable}). The challenge is that for any given dataset we have no guarantee that any such $\gamma^\star$ exists that exactly balances the covariates. We therefore often require some method of determining which parts of the covariate distribution we wish to prioritize balancing - or which covariates to balance at all - to minimize this bias.

The synthetic controls approach chooses the $\gamma$ that minimizes the weighted L2-squared distance of the covariates using a diagonal weighting matrix $V$. $V$ is then chosen to minimize the mean-square error of the weighted difference in pre-treatment outcomes. Letting $Z_a$ be the matrix of pre-treatment outcomes for treatment group $A = a$, the synthetic controls algorithm solves the following optimization problem for a fixed $V$:

\begin{align}
\gamma(V) = \arg\min_{\tilde{\gamma}(V^\star)} = (\bar{X}_1 - X_0^T\tilde{\gamma})'V(\bar{X}_1 - X_0^T\tilde{\gamma})
\end{align}

This is the ``inner'' optimization. $V^\star$ is then determined in an ``outer'' optimization to minimize the imbalances in the pre-treatment outcomes $Z$:

\begin{align}
    V^\star = \arg\min_V (\bar{Z}_1 - Z_0^T\gamma(V))'(\bar{Z}_1 - Z_0^T\gamma(V))
\end{align}

In applications the covariate matrix $X_a$ may contain some elements of $Z_a$. In cases where $X_a$ contains all pre-treatment outcomes, \cite{kaul2015synthetic} has shown that the predictor weights $V^\star$ will give no weight to auxillary covariates (covariates that are not the pre-treatment outcomes).

While in practice $V$ is often learned on the same data as the weights, we consider the case where we use cross-validation to choose $V$, as proposed by \cite{abadie2015comparative}. Assume we can divide our pre-treatment data into a training data from periods $T = 1, ..., T - l - 1$, a validation period from periods $T - l, ..., T - 1$, and a post-treatment period at time $T$. To make this discussion more general, assume that we are evaluating a set of candidate models $\mathcal{M}$ on the validation data (where for the synthetic controls algorithm we can think of this as the set of all possible weighting matrices $V$). Let $\bar{Y}^a_{a', t}$ be the mean potential outcome under treatment $A = a$ for treatment group $A = a'$ at time $t$ (where $t$ occurs during the validation period). Let $\hat{\bar{Y}}^a_{a'', t}(m)$ be an estimator of that potential outcome at time $t$ using model $m$, which was trained during the training period using data from treatment group $A = a''$. Finally, let $\bar{Y}_{a'}^a_T$ be the post-treatment estimand, where $\hat{Y}^a_{a'', T}(m)$ is the estimator using model $m$ trained using validation period data. This learning procedure implicitly assumes that:

\begin{align*}
m^\star = \min_{m \in \mathcal{M}}\sum_{T - l}^{T-1}\|\hat{Y}^0_{0, t}(m) - \bar{Y}^0_{1, t}\| = \min_{m \in \mathcal{M}}\mathbb{E}\{\|\hat{Y}^0_{0, T}(m) - \bar{Y}^0_{1, T}\|\}
\end{align*}

In other words, we select our model using the empirical loss in the validation period as a proxy for the expected loss in the post-treatment time-period.\footnote{It is possible that multiple models in $\mathcal{M}$ either perfectly predict the pre-treatment outcomes, or predict them equally well. In this case we would require an additional criteria to choose the optimal model (see, e.g, \cite{becker2017cross}}. This makes intuitive sense in the typical synthetic controls setting where the estimand is the ETT since we observe $Y^0_{sct}$ for $t < T$. When synthetic controls are used to estimate the ETC, this strategy alone is insufficient as we never observe $Y^1_{sct}$ (or a mean-unbiased proxy) prior to treatment for any unit. We therefore cannot easily use pre-treatment outcomes to optimally select variables or determine relative covariate importance without stronger assumptions.

One such assumption is the following:

\begin{align*}\label{assumption:second}
m^\star = \min_{m \in \mathcal{M}}\sum_{T - l}^{T-1}\|\hat{Y}^0_{1, t}(m) - \hat{Y}^0_{0, t}\| = \min_{m \in \mathcal{M}}\mathbb{E}\{\|\hat{Y}^1_{1, T}(m) - \bar{Y}^1_{0, T}\|\}
\end{align*}

We call this assumption ``counterfactual risk invariance.'' In other words, we assume that the model that minimizes the validation-period risk also minimizes the post-treatment risk. This is a very strong assumption for conducting any form of variable selection or covariate weighting in this setting. As a simple example, assume that we can partition $X = (R, S)$, where, for simplicity, we assume $S$ is univariate. Further assume that $Y^0_t \perp A \mid R$ for all $t = 1, ..., T$ but that $Y^1_T \perp A \mid X$. Again assume that $\mu_a$ are linear in the covariates with (time-invariant) coefficients $\beta_{a, r}$ ($r = 1, ..., q$). These assumptions imply that $\beta_{0, s} = 0$. If our goal is to predict the ETT, then we wish to use the untreated data to predict $\bar{Y}_{1, T}^0$. We may then conduct some variable selection procedure using our pre-treatment data, learn that covariate $S$ is unimportant, and estimate a model that downweights imabalances in $S$ (or ignores the covariate entirely) but perfectly balances the remaining covariates. This model would give an unbiased estimate of the counterfactual outcome for the treated group absent treatment in time-period $T$. However, if our goal were instead to predict $\bar{Y}^1_{0, T}$ -- the counterfactual outcome under treatment for the untreated units -- the same procedure applied to the treated data would again downweight $S$ and result in a biased estimate, with bias equal to $(\gamma^TS_1 - \bar{S}_0) \beta_{1, s}$. If $S$ is a strong predictor of treatment assignment, this could lead to substantial bias. \footnote{We conflate two issues in this discussion: the synthetic controls variable weighting algorithm and variable selection more generally. Provided $S$ is contained in $X$ and exact balancing weights exist with high probability as $n \to \infty$, then under our modeling assumptions, the synthetic controls procedure is still consistent for $\psi^1$ even if the variable weights $V$ are sub-optimal. However, if $S$ is not contained in $X$ at all, then this procedure will not balance these covariates even asymptotically provided that $\mathbb{E}\{S \mid A = 1\} \ne \mathbb{E}\{S \mid A = 0\}$. However, here we take a finite-sample perspective. Imbalances in $S$ will lead to bias, whether or not $S$ is imbalanced due to receiving low weights on the weighting matrix $V$, or omitted from the objective entirely. Moreover, these two conditions are in the case where the optimal weight on covariate $S$, $V_s = 0$.}

As a practical example, we highlight the potential confounding role of Republican governance for our counterfactual estimate. Republican governance is a strong predictor of a state's decision to expand Medicaid \cite{courtemanche2017early}. Moreover, existing evidence prior to Medicaid expansion showed that Medicaid take-up rates were lower in more conservative states \cite{sommers2012understanding}. Yet when generating their synthetic control weights to estimate the ETT, \cite{courtemanche2017early} and \cite{kaestner2017effects} do not control for these factors. \footnote{\cite{courtemanche2017early} does control for Republican governor in their regression model and they find that it is a statistically significant predictor of 2013 uninsurance rates. One reason they may not control for this in the synthetic control model is practical: it is much harder to balance this covariate using control data without extrapolating from the data.} However, it is clear that if take-up rates depend on governance, we may expect this to be a strong confounder of $Y^1$ and hence confound the ETC, even if arguably it is not a confounder of $Y^0$ (and hence not a confounder for the ETT).

We demonstrate this in our application by conducting a variable importance analysis. Specifically, we remove the balance constraints from the Republican governance indicators and examine how our estimates of $\hat{\psi}^1$ change. Letting $\hat{\psi}^1_s$ be the estimate when removing the Republican governance indicators (or more generally, the covariate matrix $S$ where $X = (R, S)$). We subtract our original point estimate $\hat{\psi}^1_0$ from $\hat{\psi}^1_s$ to generate the difference $\hat{\Delta}^1$. This difference tells us about the direction of the bias our estimate of $\hat{\psi}^1$ would incur when we do attempt to constrain the imbalance in covariate $S$. Our hypothesis implies that we should expect $\hat{\Delta}_s^1 < 0$: that is, keeping all other covariates (roughly) fixed, we expect the predicted uninsurance rate will decrease when as the level of Republican governance decreases. In addition to the Republican governance indicators, we also examine four other covariate groups: pre-treatment uninsurance rates and pre-treatment unemployment rates, and three sets of different demographic indicators, which we detail in Appendix E.\footnote{We caution that our results do not imply that Republican governance is not an important confounder of $Y^0_{1, T}$ since we do not analyze this directly.}

Overall we emphasize that predicting the outcome under treatment is different, and perhaps more challenging, than predicting the outcome absent treatment. The former requires understanding which covariates matter most to predicting treatment response, which we cannot as naturally learn from pre-treatment outcomes. We instead rely on our prior knowledge and modeling assumptions to choose which covariates to balance and which covariates to prioritize in this balancing. We point out that modeling the ETC requires greater justification of the covariates used to predict treatment response than for the ETT, and that using the standard synthetic controls variable weighting procedure is unlikely to be optimal for this purpose.\footnote{Our analysis assumes no unmeasured confounding and a linear model for $\mu_a$. By contrast, synthetic controls are frequently motivated by a linear factor model for $\mu_0$. \cite{abadie2010synthetic} and \cite{ferman2016revisiting} outline conditions where this method is consistent as the number of pre-treatment outcomes goes to infinity, in particular because the method balances the unobserved factor loadings. Analogous to our analysis, if we assume $\mu_{a, T}$ both follow a linear factor model, identification of the ETC requires that the factors that confound $Y^1_T$ are the same that confound $Y^0_t$. Under this assumption, we might be able to show that the synthetic control estimator is consistent in this setting. However, the tuning procedure to determine the predictor weights may again be sub-optimal from a finite-sample bias perspective, depending again on how the covariates (or unobserved factors) that are most predictive of treatment response vary between treatment groups and on their associations with the potential outcomes.}

Given these challenges, we therefore use a variation of SBW to estimate the ETC.\footnote{Specifically, we use a modified implementation of Noah Griefer's ``optweight'' package in R, available on github.com/mrubinst757} SBW minimizes the variance of the weights subject to user-specified balance constraints. The primary advantages of H-SBW over synthetic controls in this setting are that it gives the user finer control over the desired levels of covariate balance, allowing the user to navigate a bias-variance tradeoff with respect to balance and the variability of the weights. Specifically, SBW weights solve the following minimization:

\begin{equation}
\gamma &= \arg\min_{\tilde{\gamma} \in \Gamma} \quad \sum_{s: A_s = 1}^{m_1}\sum_{c = 1}^{p_s} \tilde{\gamma}_{sc}^2
\end{equation}

where $\Gamma$ is defined in Equation~\ref{eqn:constraint}. We can then estimate $\psi$ using Equation~\ref{eqn:psi}, substituting $J_{sc}$ for $Y_{sc}$ and plugging in the weights $\gamma$. By contrast, the synthetic controls algorithm will minimize the weighted L2 distance between the treated and control units in the criterion; this algorithm in general may lead to lower imbalances, but the balance tradeoffs are difficult to control, and the resulting weights may be more extreme. Finally, the algorithm, as formulated in \cite{abadie2010synthetic} and presented above, may not have a unique solution (but see \cite{ben2018augmented}, \cite{becker2017cross}).

For our primary estimates we lean heavily on assumptions to justify our choice of $\delta$. We use a priori domain knowledge about which covariates are most likely to be important predictors of treatment response when setting $\delta$, but also choose $\delta$ to avoid generating overly extreme weights. For our application, we constrain $\delta$ to be 0.05 percentage points (out of 100) for pre-treatment outcomes, 0.15 percentage points for pre-treatment unemployment rates, and 25 percentage points for the Republican governance indicators. We believe these covariates are most likely to predict treatment response. While we believe that Republican governance is an important covariate to balance, we are unable to reduce the constraints further given the support of the data. For the remaining covariates, we let $\delta$ be 0.5 percentage points for average population growth and household to adult ratio, 1 percentage point for female, Hispanic ethnicity, white race, age category, disability, and number of children category; 2 percentage points for urban, citizenship, education category, income-to-poverty category, student, and foreign-born, again choosing these constraints with respect to both feasibility and extreme weight concerns.

\subsubsection{H-SBW objective}

The motivation of the SBW criterion is to produce the minimum variance weights for a fixed $\delta$. This produces the minimum variance estimator within the constraint set if, for example, the errors in the outcome model are independent and identically distributed \cite{zubizarreta2015stable}. In our setting we allow for possible state-level dependencies, potentially reducing the efficiency of the SBW estimator. To address this possibility, we add the tuning parameter $\rho \in [0, 1)$ in the objective below. Assuming a constant variance across units for each error component, $\rho$ represents a constant (and known) within-state correlation of the errors.

\begin{equation}\label{eqn:objective}
\gamma &= \arg\min_{\tilde{\gamma} \in \Gamma} \quad \sum_{s: A_s = 1}^{m_1}(\sum_{c = 1}^{p_s} \tilde{\gamma}_{sc}^2 + \sum_{c \ne d}\rho \tilde{\gamma}_{sc}\tilde{\gamma}_{sd})\\
\end{equation}

For $\delta \to \infty$, this objective yields the solution:

\begin{equation}\label{eqn:sbwsol}
\gamma_{sc} \propto \frac{1}{(p_s - 1)\rho + 1}
\end{equation}

Setting $\rho = 0$ returns the SBW solution: $\gamma_{sc} \propto 1$. By contrast, when setting $\rho \approx 1$, we see that $\gamma_{sc} \propto \frac{1}{p_s}$. In other words, as we increase $\rho$, this objective downweight CPUMAs in states with large numbers of CPUMAs and upweight CPUMAs in states with small numbers of CPUMAs (all while assigning each CPUMA within a state equal weight). Moreover, when $p_s$ is constant across states this objective will also the SBW solution. In short, as we increase $\rho$, the objective will attempt to more uniformly disperse weights across states.

As a brief illustration, we simulate $N = 800$ observations in $m = 40$ regions each with $p_s = 20$ units, and draw $X_{sc} \sim N(\mu_s, 1)$, for $\mu \in \{0, 1\}$ (drawn from a Bernoulli with equal probability); $A_s \sim Bern(expit(\bar{X}_s))$, where $\bar{X}_s$ is the mean covariate value in group $s$. We then generate weights to balance the control to the treated group mean. We run H-SBW variants setting $\rho = 0$ (which is equivalent to SBW), $\rho = 0.5$, and $\rho = 0.99$ all while keeping $\delta$ fixed at zero. Figure 1 shows the weights summed to the group level for all control regions. The color of each set of bars is the overall variance of the weights. We can see that the weights in general are uniform across units for SBW (the variance is lower); however, the weights are not uniformly dispersed across regions (the sum of weights within each region is not even). As we increase $\rho$, the weights disperse more evenly across regions. Despite the increase in the variance of the weights, these weights are optimal under our assumed covariance structure.

\begin{figure}
\begin{center}
    \includegraphics[scale=0.5]{01_Plots/proofofconcept.png}
    \caption{Comparison of SBW and H-SBW: within group sum of weights}
    \label{oatepref}
\end{center}
\end{figure}

The particular covariance structure we assume is identical to the one proposed by \cite{kloek1981ols}. In Appendix A, we show that this objective produces the minimum variance estimator under the constraint set for this correlation structure. We note that theoretically we could incorporate any other assumed covariance structure into this objective, though the number of tuning parameters might change. Broadly speaking, we can think of H-SBW being to SBW what generalized least squares (GLS) is to ordinary least squares (OLS): both SBW and OLS can produce unbiased estimates of model parameters; however, H-SBW and GLS can improve the efficiency of our estimates under different assumed correlation structures of the outcome errors.

\subsubsection{Measurement error}

A second advancement in our estimation procedure comes in our balance constraints: rather than balancing on the observed covariate values $W_{sc}$, we instead balance on the imputed covariate estimates $\hat{\eta}_1(W_{sc})$ (we refer to these as the ``adjusted covariates''). This procedure attempts to correct for the estimation error in these CPUMA-level covariates that may bias our estimate of $\psi^1$. In Appendix A, we consider the super-population target $\psi^{1, sp} = \mathbb{E}\{Y^1 \mid A = 0\}$ and show that under the classical errors-in-variables model, the bias for the SBW estimator that balances on the observed covariates $W$ and sets $\delta = 0$ is equivalent to the bias of a linear combination of coefficient estimates from the OLS-based regression estimator. Specifically, the bias for either estimator is:

\begin{equation}
\mathbb{E}\{\hat{\psi}^{1} - \psi^{1, sp}\} = (\upsilon_0 - \upsilon_1)^T(\kappa - I_d)\beta_1
\end{equation}

The intuition for this result is as follows: exact balancing weights implicitly estimate $\beta_1$ on a subset of the data where we have sufficient covariate overlap. We can therefore think of SBW as returning a solution to some weighted-least squares problem. Assuming that the outcome model holds across all of the data, WLS and OLS are estimating the same $\beta_1$; therefore, the bias that effects the least squares solution will have the same effect on the WLS, and therefore SBW, solution. In Appendix A, Proposition 2, we show that if we had access to $\eta_1$, we can obtain an unbiased estimate of $\psi^{1, sp}$ by reweighting $\eta_1(W_{sc})$.\footnote{For the finite-sample parameter we're targeting, this estimator will have finite-sample bias conditional on $W$ and viewing $X$ as fixed.} Of course, in practice we do not know $\eta_1$ but must instead estimate it using auxillary data. In Appendix A, Proposition 3, we show that we can obtain a consistent of $\psi^{1, sp}$ when balancing on an estimate of $\eta_1$ using auxillary data.

The key in our application is to estimate $\eta_a$: at a high-level, we use the ACS micro-data replicate survey weights to estimate the covariance matrix of CPUMA sampling-variability $\Sigma_{vv, sc}$. Using our observed data to estimate $\Sigma_{WW \mid A = a}$ and $\bar{W}$, we combine these estimates to generate an estimate of $\eta_a$. This is a technique that comes from the regression-calibration literature (see, e.g., \cite{gleser1992importance}). We also consider an adjustment procedure that further accounts for the differential measurement error due to the highly variable sample sizes used to calculate each covariate. This procedure allows our adjustment to differentially adjust covariate values depending on the sample-sizes involved in the adjustment procedure. We refer to the first procedure the ``homogeneous adjustment'' and the second procedure the ``heterogeneous adjustment'' because the adjustment is constant for all units in the first case, but varies by unit in the second case. Further details about these procedures are available in Appendix B.

This is the first application we are aware of to use regression calibration in the context of balancing weights to address the problem of measurement error. We emphasize two critical assumptions for using this procedure in our context: (1) the outcome model is linear in the true covariates; and (2) the measurement error in the outcome is uncorrelated with the measurement error in the covariates. The first assumption is strong, though often used in practice. The second assumption is reasonable, because our outcomes are estimated from a different cross-section than our covariates.

\subsubsection{Bias-correction for imbalances}

Because we are unable to reduce the balance constraints to our preferred level without generating very extreme weights, following the recent literature on synthetic controls, we test the sensitivity of our results to the imbalances in the observed (or adjusted) covariates using ridge-regression augmented weights \cite{ben2018augmented}. Letting $\matr{\hat{X}}_1$ be the matrix of adjusted covariates, and $\gamma^{hsbw}$ be our H-SBW weights, we consider the regression-augmented weights:

\begin{equation}
\gamma^{aug} = \gamma^{hsbw} + (\gamma^{hsbw}\hat{X}_1 - \bar{W}_0)^T(\hat{X}_1^T\Omega^{-1}\hat{X}_1 + \lambda I_q)^{-1}\hat{X}_1^T\Omega^{-1}
\end{equation}

where $\Omega$ is a block diagonal matrix with diagonal entries equal to one and the within-group off diagonals equal to $\rho$. We choose $\lambda$ so that the remaining imbalances all fall within 0.5 percentage points. The cost of this procedure is that we must extrapolate off the support of the data, and therefore rely more heavily on our outcome modeling assumptions. We refer to \cite{ben2018augmented} for more details about this procedure. In our results we consider estimators using SBW ($\rho = 0$), H-SBW ($\rho = 1/6$), and ridge-augmented versions of SBW and H-SBW that we call BC-SBW and BC-HSBW.

\subsection{Model validation}

Earlier we argued that we cannot easily use pre-treatment outcomes to conduct variable selection or to learn about the relative importance of covariates. The challenge with using pre-treatment data in this setting is that it is not obvious why the best model of $\bar{Y}_{0, t}^0$ $(t = T-l,..., T-1)$ should also be the best model, or even a good model, of $\bar{Y}^1_{0,T}$. However, for fixed covariates and targeted levels of imbalance $\delta$, we can use the heuristic that a good model of $\bar{Y}^1_{0,T}$ should also be a good model of $\bar{Y}_{0, t}^0$ to compare our models. We can also assume that when comparing two models, one with uniformly better covariate balance than the other, the model with better covariate balance should have lower bias both for $\bar{Y}_{0, T}^1$ and $\bar{Y}_{0, t}^0$ if our model assumptions are correct. We justify these comparisons by again assuming that $\mu_{a, t}$ are linear in $X$ for all time periods $t$. This is a stronger assumption than we require to estimate the ETC, which only requires that $\mu_{1, T}$ is linear in $X$. However, with this stronger assumption we can use pre-treatment data to at least heuristically compare our models. While we caution that these comparisons may not indicate the best model, we can still use these comparisons to see which models we may trust less.

We therefore rerun our procedures on pre-treatment data to compare the performance of our models for a fixed level of imbalances $\delta$. In particular, we train our model on 2009-2011 data to predict 2012 outcomes, and 2010-2012 data to predict 2013 outcomes. We limit to one-year prediction error since our estimand is only one-year forward. We then examine the performance of the H-SBW versus SBW estimators, which only vary with respect to the tuning parameter $\rho$, the bias-corrected versions, and the covariate adjustment procedure used to determine the weights.

We expect that the estimators trained on the adjusted data should perform better than the estimators trained on the unadjusted data. If our outcome model is correct, these estimators should achieve better balance across the true covariates and therefore have lower bias than the estimators trained on the unadjusted data. We assume that any difference between the performance of the two adjustments is due to better covariate balance on the true covariates. We therefore use the performance on this data to select which adjustment we prefer in our final results. Conditional on the adjustment, we assume that if the bias-corrected estimators all have uniformly better balance than the uncorrected estimators, these estimators should also perform better than the uncorrected estimators. However, if the assumed outcome models are incorrect, these estimators may suffer from extrapolation bias and perform worse despite achieving better covariate balance. Finally, we expect in general that H-SBW and SBW should have similar performance if we knew the true covariate values. However, in Appendix A, we show that for fixed (unobserved) $X$, these weighting estimators have a finite sample bias that reduces with the square of the weights, suggesting the SBW may have less bias than H-SBW. On the other hand, H-SBW should have less variability given within-state dependencies in the outcomes. From an MSE perspective it is unclear which should be optimal, and given only two-years of pre-treatment data we do not believe we can select which model is better.

\subsection{Inference}

We consider $W$ to be fixed (and $X$ as fixed unknown parameters), and we consider inference over repeated samples from some super-population of CPUMAs with a state-level dependency structure.\footnote{Alternatively, viewing the potential outcomes as fixed and treatment assignment as random, we could consider inference over the randomization distribution of treatment at the state-level.} While placebo tests are frequently used in the synthetic controls literature for inference, we view these as qualitative statistical tests (see, e.g., \cite{arkhangelsky2019synthetic}) and instead use the leave-one-state-out jackknife to estimate the variance of $\hat{\psi}^1$ (\cite{cameron2015practitioner}). Specifically, we exclude each state and re-calculate the weights holding our targeted mean fixed at $\bar{W}_0$.\footnote{When our preferred initial choice of $\delta$ does not converge, we gradually reduce the constraints until it does.} We compute this estimator in two ways: first, we condition on our covariate adjustment $\hat{\eta}_1$. This is our preferred estimator; however, it does not account for the randomness in $\hat{eta}_1$. We therefore also conduct a second procedure where we re-estimate $\hat{\eta}_1$ for each state omitted in the jackknife procedure and provide these results in the Appendix.

To estimate $Var(\hat{\psi}^0 \mid X, W)$ we use an auxillary regression model and use the CR-2 standard error adjustment (using the ``clubSandwich'' package in R) to estimate the variance of the linear combination $\bar{W}_0^T\hat{\beta}_0$. We can estimate this quantity using the original (unadjusted) data given that $\mathbb{E}\{\bar{W}_0^T\hat{\beta}_0\} = \psi^0$ (since the regression line runs through the point $(\bar{W}_0, \bar{J}_0)$, which are unbiased estimates of $(\bar{X}_0, \bar{Y}_0)$). Our total estimate $\hat{Var}(\hat{\psi})$ is simply the sum of these two variance estimates. We use the standard normal quantiles to generate confidence intervals.


\BoSSSopen{ParameterStudy/ParameterStudy}
\graphicspath{{ParameterStudy/ParameterStudy.texbatch}}

\BoSSScmd{
restart;
 }
\BoSSSexeSilent
\BoSSScmd{
/// This guide will give you an example of how to conduct a parameter study with
/// all the necessary steps.
/// \section{Initialization of solver, processor and workflow}
/// We start with initializing of the workflow
 }
\BoSSSexe
\BoSSScmd{
/// \section{Initialization of solver, processor and workflow}
/// We start with initializing of the workflow.
 }
\BoSSSexe
\BoSSScmd{
BoSSSshell.WorkflowMgm.Init("Name of Workflow");
 }
\BoSSSexe
\BoSSScmd{
/// This line helps us manage the sessions later on while evaluating the results.
/// Next, we connect to the database.
 }
\BoSSSexe
\BoSSScmd{
var myDb = CreateTempDatabase();
 }
\BoSSSexe
\BoSSScmd{
/// To check all the sessions in the current workflow, use the line:
 }
\BoSSSexe
\BoSSScmd{
BoSSSshell.WorkflowMgm.Sessions;
 }
\BoSSSexe
\BoSSScmd{
/// Now, all the necessary libraries need to be loaded
 }
\BoSSSexe
\BoSSScmd{
using System.Diagnostics;\newline
using BoSSS.Foundation.Grid.RefElements;\newline
using BoSSS.Application.XNSE\_Solver;\newline
using BoSSS.Platform.LinAlg;\newline
using BoSSS.Solution.XdgTimestepping;
 }
\BoSSSexe
\BoSSScmd{
/// As an execution queue, we select the first queue defined in
/// the {\tt $\sim$/.BoSSS/etc/BatchProcessorConfig.json}-file:
 }
\BoSSSexe
\BoSSScmd{
var myBatch = ExecutionQueues[0];
 }
\BoSSSexe
\BoSSScmd{
/// \section{Grid Generation}
///Firstly, we need to determine the boundaries of our grid/control volume.
///Is it important to know that the number of nodes (in our case $/code{k}$)
///needed are equal
/// to the number of cells $+1$. For instance, for $10$ cells we need $11$ nodes.
/// In this example we will use the Cartesian $2D$ grid from the database
/// which requires $x$- and $y$-Nodes. The J term in the code is for doing
/// a check if the desired resolution of the volume is correctly typed.
 }
\BoSSSexe
\BoSSScmd{
int k = 10;\newline
double[] xNodes = GenericBlas.Linspace(0, 1, k + 1);\newline
double[] yNodes = GenericBlas.Linspace(0, 1, k + 1);\newline
int J           = (xNodes.Length - 1)*(yNodes.Length - 1);\newline
string GridName = string.Format(BoSSSshell.WorkflowMgm.CurrentProject + "\_J" +J);\newline
 \newline
Console.WriteLine("Creating grid with " + J + " cells. ");\newline
 \newline
GridCommons g;\newline
g      = Grid2D.Cartesian2DGrid(xNodes, yNodes);\newline
g.Name = GridName;
 }
\BoSSSexe
\BoSSScmd{
/// \section{Define geometrical boundaries}
/// After loading the grid and giving the dimensions, we need to adjust
/// the edges and their names. With the following code we assign every edge with
/// a number and name. Keep in mind that the name corresponds to the boundary
///condition (in this case "Pressure Dirichlet").
/// In this particular case we will use inflow profile represented
/// via tan-function and the angle of inflow will be $30$ degrees.
 }
\BoSSSexe
\BoSSScmd{
GridCommons g;\newline
g      = Grid2D.Cartesian2DGrid(xNodes, yNodes);\newline
g.Name = GridName;\newline
 \newline
g.EdgeTagNames.Add(1, "wall");\newline
g.EdgeTagNames.Add(2, "Velocity\_Inlet");\newline
g.EdgeTagNames.Add(3, "Pressure\_Dirichlet\_back");\newline
g.EdgeTagNames.Add(4, "Pressure\_Dirichlet\_top");\newline
 \newline
g.DefineEdgeTags(delegate (double[] X) \{\newline
\btab byte ret = 0;\newline
\btab if (Math.Abs(X[1]-(0.0))<= 1.0e-8)\newline
\btab \btab ret = 1;\newline
\btab if (Math.Abs(X[0]-(0.0))<= 1.0e-8)\newline
\btab \btab ret = 2;\newline
\btab if (Math.Abs(X[1]-(1.0))<= 1.0e-8)\newline
\btab \btab ret = 3;\newline
\btab if (Math.Abs(X[0]-(1.0))<= 1.0e-8)\newline
\btab \btab ret = 4;\newline
\btab return ret;\newline
 \newline
 \});
 }
\BoSSSexe
\BoSSScmd{
/// \section{Angle/Velocity Profile}
/// In this particular case we will use inflow profile represented via tan-function and the angle of inflow will be $30$ degrees.
 }
\BoSSSexe
\BoSSScmd{
string caseName = string.Format("k\{0\}\_\{1\}", k, g);\newline
 \newline
Console.WriteLine("setting up: " + caseName);\newline
 \newline
double beta    = 30;\newline
string CosBeta = Math.Cos(beta*Math.PI/180.0).ToString();\newline
string SinBeta = Math.Sin(beta*Math.PI/180.0).ToString();
 }
\BoSSSexe
\BoSSScmd{
/// These code lines set up the case name and introduce the sine and cosine
/// functions to our simulation. Next, we define the velocities in
/// $x$- and $y$-direction via a tan-function. These velocities and angles are only for this particular example and would not be suited for your simulation.
 }
\BoSSSexe
\BoSSScmd{
var UX = new Formula\newline
\btab (string.Format("X=> \{0\}*Math.Atan(X[1]*5)*2.0/Math.PI",CosBeta),false);\newline
var UY = new Formula \newline
\btab (string.Format("X=> \{0\}*Math.Atan(X[1]*5)*2.0/Math.PI",SinBeta),false);
 }
\BoSSSexe
\BoSSScmd{
///After the velocities and boundary conditions are set.
///We need to determine all other simulation parameters needed to proceed.
//The variable $\textbackslash code\{ctrl\}$ is used to store the $\textbackslash code\{IBM\_Control\}$-object.\newline
/// All other parameters are selfexplanatory.
 }
\BoSSSexe
\BoSSScmd{
var ctrl = new XNSE\_Control();\newline
//controls.Add(ctrl);\newline
 \newline
ctrl.SessionName = caseName;\newline
ctrl.SetDatabase(myDb);\newline
ctrl.SetGrid(g);\newline
ctrl.SetDGdegree(k);\newline
ctrl.NoOfMultigridLevels = int.MaxValue;
 }
\BoSSSexe
\BoSSScmd{
/// \section{Boundary conditions/Initial values}
/// We move on to the part where we define
/// the boundary conditions and initial values.
 }
\BoSSSexe
\BoSSScmd{
ctrl.AddBoundaryValue("wall");\newline
ctrl.AddBoundaryValue("Velocity\_Inlet");\newline
ctrl.AddBoundaryValue("Pressure\_Dirichlet\_back");\newline
ctrl.AddBoundaryValue("Pressure\_Dirichlet\_top");\newline
ctrl.AddBoundaryValue("Velocity\_Inlet","VelocityX",UX);\newline
ctrl.AddBoundaryValue("Velocity\_Inlet","VelocityY",UY);
 }
\BoSSSexe
\BoSSScmd{
/// and for the initial values
 }
\BoSSSexe
\BoSSScmd{
ctrl.InitialValues.Add("VelocityX", new Formula ("X=> 0.0", false));\newline
ctrl.InitialValues.Add("VelocityY", new Formula ("X=> 0.0", false));\newline
ctrl.InitialValues.Add("Pressure", new Formula ("X=> 0.0", false));\newline
ctrl.InitialValues.Add("Phi", new Formula ("X=> -1.0", false));
 }
\BoSSSexe
\BoSSScmd{
/// \section{Fluid properties}
/// Here we set up the density and the Reynolds number,
/// keep in mind that the calculations are dimensionles,
/// so leave the values as seen above ($100$ is an example value)
 }
\BoSSSexe
\BoSSScmd{
double reynolds               = 100;\newline
ctrl.PhysicalParameters.rho\_A = 1;\newline
ctrl.PhysicalParameters.mu\_A  = 1.0/reynolds;
 }
\BoSSSexe
\BoSSScmd{
/// \section{Simulation options}
/// We set the simulation parameters, such as time-step size,
/// end time and number of time-steps.
 }
\BoSSSexe
\BoSSScmd{
ctrl.TimeSteppingScheme = TimeSteppingScheme.ImplicitEuler;\newline
double dt               = 7e-2;\newline
ctrl.dtMax              = dt;\newline
ctrl.dtMin              = dt;\newline
ctrl.Endtime            = 1e16;\newline
ctrl.NoOfTimesteps      = 100;
 }
\BoSSSexe
\BoSSScmd{
/// for the time-stepping scheme, you can choose either BDF2 or ImplicitEuler.
/// \section{Starting of simulation}
/// You have two possible ways to start a simulation
/// - locally on the PC via $\code{myBatch}$
/// or on the network cluster $\code{myHPC}$.
 }
\BoSSSexe
\BoSSScmd{
 \newline
//Console.WriteLine(" Submitting to Cluster: " + ctrl.SessionName);\newline
//ctrl.RunBatch(myHPC);\newline
 \newline
Console.WriteLine(" Submitting " + ctrl.SessionName);\newline
ctrl.RunBatch(myBatch);
 }
\BoSSSexe
\BoSSScmd{
/// \section{Evaluation and Error Calculation}
/// After all of the desired simulation are finished,
/// you need to evaluate the different parameters and their effect on
///the whole system. Typing the following command gives you a list of all
///simulations with their status (FinishedSuccessful or with certain errors)
 }
\BoSSSexe
\BoSSScmd{
BoSSSshell.WorkflowMgm.AllJobs.Select(kv => kv.Key + ": \textbackslash t" + kv.Value.Status);
 }
\BoSSSexe
\BoSSScmd{
/// With the next command line you are able to select a certain
///session(simulation) and see the different time-steps for control purposes.
 }
\BoSSSexe
\BoSSScmd{
BoSSSshell.WorkflowMgm.AllJobs.ElementAt(1).Value.Stdout;
 }
\BoSSSexe
\BoSSScmd{
/// \subsection{$L^2$-Error}
/// This section introduces the calculation of the $L^2$-Error.
 }
\BoSSSexe
\BoSSScmd{
 ITimestepInfo[] AllSolutionS = BoSSSshell.WorkflowMgm.AllJobs.Select(kv => kv.Value.LatestSession.Timesteps.Last()).ToArray();
 }
\BoSSSexe
\BoSSScmd{
ITimestepInfo[] k1\_SolutionS = AllSolutionS.Where(\newline
\btab  ts = > ts.Fields.Single(\newline
\btab \btab    f = > f.Identification == "Pressure").Basis.Degree == 0).ToArray();\newline
ITimestepInfo[] k2\_SolutionS = AllSolutionS.Where(\newline
\btab  ts = > ts.Fields.Single(\newline
\btab \btab    f = > f.Identification == "Pressure").Basis.Degree == 1).ToArray();\newline
ITimestepInfo[] k3\_SolutionS = AllSolutionS.Where(\newline
\btab  ts = > ts.Fields.Single(\newline
\btab \btab    f = > f.Identification == "Pressure").Basis.Degree == 2).ToArray();
 }
\BoSSSexe
\BoSSScmd{
k1\_SolutionS.Select(\newline
\btab  ts => ts.Fields.Single(\newline
\btab \btab    f = > f.Identification == "Pressure").Basis.Degree);
 }
\BoSSSexe
\BoSSScmd{
double[] GridRes;\newline
Dictionary<string, double[]> L2Errors;\newline
DGFieldComparison.ComputeErrors(\newline
\btab  new[]\{"VelocityX","VelocityY"\}, k1\_SolutionS, out GridRes, out L2Errors);
 }
\BoSSSexe
\BoSSScmd{
/// To check the particular errors, type
 }
\BoSSSexe
\BoSSScmd{
GridRes;
 }
\BoSSSexe
\BoSSScmd{
L2Errors["VelocityX"];
 }
\BoSSSexe
\BoSSScmd{
L2Errors["VelocityY"];
 }
\BoSSSexe
\BoSSScmd{
/// \section{Plotting of errors}
/// This section gives a brief example of how to plot the erros
/// and all the data from the previous simulations.
 }
\BoSSSexe
\BoSSScmd{
Plot(GridRes,L2Errors["VelocityX"],"VelXErr","-oy",\newline
\btab  GridRes,L2Errors["VelocityY"],"VelXErr","-xb",logX:true,logY:true);
 }
\BoSSSexe
\BoSSScmd{
/// for a plot with more specifics and more possible adjustments
 }
\BoSSSexe
\BoSSScmd{
var FancyPlot = new Plot2Ddata();
 }
\BoSSSexe
\BoSSScmd{
FancyPlot.LogX = true;\newline
FancyPlot.LogY = true;
 }
\BoSSSexe
\BoSSScmd{
var k1plot = new Plot2Ddata.XYvalues(\newline
\btab "VelXErr-k1",GridRes,L2Errors["VelocityY"]);
 }
\BoSSSexe
\BoSSScmd{
ArrayTools.AddToArray(k1plot, ref FancyPlot.dataGroups);
 }
\BoSSSexe
\BoSSScmd{
var CL = FancyPlot.ToGnuplot().PlotCairolatex();
 }
\BoSSSexe
\BoSSScmd{
CL.PlotNow();
 }
\BoSSSexe
\BoSSScmd{
/// \section{Exporting the session table}
 }
\BoSSSexe
\BoSSScmd{
static class AddCols \{\newline
\btab static public object SipMatrixAssembly\_time(ISessionInfo SI) \{\newline
\btab \btab var mcr = SI.GetProfiling()[0];\newline
\btab \btab var ndS = mcr.FindChildren("SipMatrixAssembly");\newline
\btab \btab var nd  = ndS.ElementAt(0);\newline
\btab \btab return nd.TimeSpentInMethod.TotalSeconds  / nd.CallCount;\newline
\btab \}\newline
\btab static public object Aggregation\_basis\_init\_time(ISessionInfo SI) \{\newline
\btab \btab var mcr = SI.GetProfiling()[0];\newline
\btab \btab var ndS = mcr.FindChildren("Aggregation\_basis\_init");\newline
\btab \btab var nd  = ndS.ElementAt(0);\newline
\btab \btab return nd.TimeSpentInMethod.TotalSeconds  / nd.CallCount;\newline
\btab \}\newline
\btab static public object Solver\_Init\_time(ISessionInfo SI) \{\newline
\btab \btab var mcr = SI.GetProfiling()[0];\newline
\btab \btab var ndS = mcr.FindChildren("Solver\_Init");\newline
\btab \btab var nd  = ndS.ElementAt(0);\newline
\btab \btab //Console.WriteLine("Number of nodes: " + ndS.Count() + " cc " + nd.CallCount );\newline
\btab \btab return nd.TimeSpentInMethod.TotalSeconds / nd.CallCount;\newline
\btab \}\newline
\btab static public object Solver\_Run\_time(ISessionInfo SI) \{\newline
\btab \btab var mcr = SI.GetProfiling()[0];\newline
\btab \btab var ndS = mcr.FindChildren("Solver\_Run");\newline
\btab \btab var nd  = ndS.ElementAt(0);\newline
\btab \btab return nd.TimeSpentInMethod.TotalSeconds  / nd.CallCount;\newline
\btab \}\newline
\}
 }
\BoSSSexe
\BoSSScmd{
/// this code adds additional/user-defined colums. Now, we want to export he
/// saved session table in a file.
 }
\BoSSSexe
\BoSSScmd{
var SessTab = BoSSSshell.WorkflowMgm.SessionTable;
 }
\BoSSSexe
\BoSSScmd{
SessTab = SessTab.ExtractColumns(AllCols.ToArray());
 }
\BoSSSexe
\BoSSScmd{
using System.IO;
 }
\BoSSSexe
\BoSSScmd{
/// Here, we define the filename
 }
\BoSSSexe
\BoSSScmd{
var now           = DateTime.Now;\newline
SessTab.TableName = "SolverRuns--" + now.Year + "-" + now.Month + "-" + now.Day;\newline
string docpath    = Path.Combine(CurrentDocDir, SessTab.TableName + ".json");
 }
\BoSSSexe
\BoSSScmd{
/// saving the session table as a file could also be done in our git reposatory
 }
\BoSSSexe
\BoSSScmd{
SessTab.SaveToFile(docpath);
 }
\BoSSSexe
\BoSSScmd{
///
 }
\BoSSSexe


\documentclass[a4paper]{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{braket}%量子力学符号
\usepackage{geometry}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{float}%稳定图片位置
\usepackage{graphicx,subfig}%画图
\usepackage{caption}
\usepackage[english]{babel}
\usepackage{indentfirst}%缩进
\usepackage{enumerate}%加序号
\usepackage{multirow}%合并行
\usepackage{hyperref}
\hypersetup{hypertex=true, colorlinks=true, linkcolor=black, anchorcolor=black, citecolor=black}
\title{\Large \textbf{VP390 Problem Set 3}\\
\author{\textbf{Pan, Chongdan ID:516370910121}\\
}
}
\begin{document}
\maketitle
\section{Problem 1}
\begin{enumerate}[(a)]
    \item $|\Psi(x,t)|^2=\Psi(x,t)\Psi^*(x,t)=Ce^{-\frac{i}{\hbar}Et}(x^2-a^2)Ce^{\frac{i}{\hbar}Et}(x^2-a^2)=C^2(x^2-a^2)^2$
    \\ $|\Psi(x,t)|^2=\Phi(x)$ So $\Psi$ represents a stationary state.
    \item $\int_{-a}^a|\Psi(x,t)|^2\mathrm{d}x=\int_{-a}^a C^2(x^2-a^2)^2\mathrm{d}x=C^2(\frac{2a^5}{5}-\frac{4a^5}{3}+2a^5)=C^2\frac{16a^5}{15}=1$
    \\$C^2a^5=\frac{15}{16}$
    \item $\int_{-a/2}^{a/2} C^2(x^2-a^2)^2\mathrm{d}x=C^2(\frac{a^5}{80}-\frac{a^5}{6}+a^5)=\frac{203}{256}\approx0.793$
    \item Since $\Phi(x)$ is symmetric about $x$-axis, the probability is 0.5
    \item $\braket{x}_\Psi=\int_{-\infty}^\infty x|\Psi(x,t)|^2\mathrm{d}x=\int_{-a}^a C^2x(x^2-a^2)^2\mathrm{d}x=0$
    So it's time independent
    \item $\bigtriangleup_x=\sqrt{\braket{x^2}_\Psi-\braket{x}_\Psi^2}=C\sqrt{\int_{-a}^a x^2(x^-a^2)^2\mathrm{d}x}=C\sqrt{\frac{30a^7}{105}-\frac{84a^7}{105}+\frac{70a^7}{105}}=\sqrt{\frac{a^2}{7}}$
\end{enumerate}
\section{Problem 2}
\begin{enumerate}[(a)]
    \item $E_1=13.6$eV, $a_0=\frac{4\pi\epsilon_0\hbar^2}{me^2}\approx5.29\times10^{-2}$nm
    \\$\int_{-\pi/2}^{\pi/2}\int_0^{2\pi}\int_0^\infty\Psi_{100}(r,\varphi,\theta,t)|^2\mathrm{d}r\mathrm{d}\theta\mathrm{d}\varphi=C^2\int_{-\pi/2}^{\pi/2}\int_0^{2\pi}\int_0^\infty r^2\sin\varphi\exp(-\frac{2r}{a_0})\mathrm{d}r\mathrm{d}\theta\mathrm{d}\varphi=2\times2\pi C^2\frac{a^3}{4}$
    \\$C=\sqrt{\frac{1}{\alpha_0^3\pi}}=1.467\times10^{15}$
    \item $C^2\int_{-\pi/2}^{\pi/2}\int_0^{2\pi}\int_0^{a_0} r^2\sin\varphi\exp(-\frac{2r}{a_0})\mathrm{d}r\mathrm{d}\theta\mathrm{d}\varphi=\pi C^2(a^3-5a^3e^{-2})=1-5e^{-2}$
    \item The probability for the electron radius is $r$ can be expressed :
    \\$C^2\int_{-\pi/2}^{\pi/2}\int_0^{2\pi} r^2\sin\varphi\exp(-\frac{2r}{a_0})\mathrm{d}\theta\mathrm{d}\varphi=4\pi C^2r^2\exp(-\frac{2r}{a_0})$
    \\$\frac{\mathrm{d}r^2\exp(-\frac{2r}{a_0})}{\mathrm{d}r}=2r\exp(-\frac{2r}{a_0})-\frac{2r^2}{a_0}\exp(-\frac{2r}{a_0})=0$
    \\$r_1=0, r_2=a_0, r_3=\infty$, since $r_1$ and $r_3$ can lead to the minium value of $r^2\exp(-\frac{2r}{a_0})$, only $r=\alpha_0$ has the most probability value to find the electron
    \item $r_a=\int_{-\pi/2}^{\pi/2}\int_0^{2\pi}\int_0^\infty r\Psi_{100}(r,\varphi,\theta,t)|^2\mathrm{d}r\mathrm{d}\theta\mathrm{d}\varphi=C^2\int_{-\pi/2}^{\pi/2}\int_0^{2\pi}\int_0^\infty r^3\sin\varphi\exp(-\frac{2r}{a_0})\mathrm{d}r\mathrm{d}\theta\mathrm{d}\varphi$
    \\$=4\pi C^2\frac{3a_0^4}{8}=\frac{3a_0}{2}$
    \\$\bigtriangleup_r=\sqrt{\braket{r^2}_\Psi-r_a^2}$
    \\$\braket{r^2}=4\pi C^2\int_0^\infty r^4\exp(-\frac{2r}{a_0})\mathrm{d}r=3\pi C^2 a_0^5=3a_0^2$
    \\$\bigtriangleup_r=\sqrt{3a_0^2-\frac{9a_0^2}{4}}=\frac{\sqrt{3}a_0}{2}$
    \\So the average value of $r$ is $\frac{3a_0}{2}$, its standard deviation is $\frac{\sqrt{3}a_0}{2}$
\end{enumerate}
\section{Problem 3}
\begin{enumerate}[(a)]
    \item $\braket{f,g}=\int_{-\infty}^\infty f^*(u)g(u)\mathrm{d}u=\int_{-\infty}^\infty (g^*(u))^*f^*(u)\mathrm{d}u=(\int_{-\infty}^\infty g^*(u)f(u)\mathrm{d}u)^*=\braket{g,f}^*$
    \item $\braket{f,\alpha g}=\int_{-\infty}^\infty f^*(u)\alpha g(u)\mathrm{d}u=\alpha\int_{-\infty}^\infty f^*(u)g(u)\mathrm{d}u=\alpha\braket{f,g}$
    \item $\braket{\alpha f,g}=\braket{g,\alpha f}^*=\alpha^*\braket{g,f}^*=\alpha^*\braket{f,g}$
\end{enumerate}
\section{Problem 4}
    \begin{enumerate}
        \item $\int_{-\infty}^\infty|\psi|^2\mathrm{d}x=\int_0^L\frac{2}{L}\sin^2(\frac{n\pi x}{L})\mathrm{d}x=\frac{2}{L}(\frac{L}{2}-\frac{L\sin 2n\pi}{4n\pi})=1$
        \\So it is normalized
        \item $\braket{\psi_n,\psi_m}=\int_0^L\frac{2}{L}\sin\frac{n\pi x}{L}\sin\frac{m\pi x}{L}\mathrm{d}x=\frac{1}{L}\int_0^L\cos\frac{(n-m)\pi x}{L}-\cos\frac{(n+m)\pi x}{L}\mathrm{d}x$
        \\$=\frac{1}{L}[\frac{L\sin(n-m)\pi}{(n-m)\pi}-\frac{L\sin(n+m)\pi}{(n+m)\pi}]=0$
        \item $\braket{i\psi_1,-3\psi_1+2i\psi_2-\psi_3}=3i\braket{\psi_1,\psi_1}-2\braket{\psi_1,\psi_2}-i\braket{\psi_1,\psi_3}=3i$
        \item Assume $x=x_1\psi_1+x_2\psi_2,y=y_1\psi_1+y_2\psi_2$ where
        \\$x_1^2+x_2^2=1,y_1^2+y_2^2=1,x_1y_1+x_2y_2=0$
        \\$x=\frac{\sqrt{2}}{2}\psi_1-\frac{\sqrt{2}}{2}\psi_2,y=\frac{\sqrt{2}}{2}\psi_1+\frac{\sqrt{2}}{2}\psi_2$
    \end{enumerate}
\section{Problem 5}
    \begin{enumerate}
        \item $\braket{\psi,\psi}=\frac{1}{2}[\braket{e^{-\frac{i}{\hbar}E_1t}\psi_1(x),e^{-\frac{i}{\hbar}E_1t}\psi_1(x)}+\braket{e^{-\frac{i}{\hbar}E_2t}\psi_2(x),e^{-\frac{i}{\hbar}E_2t}\psi_2(x)}]$
        \\$\frac{1}{2}(\braket{\psi_1,\psi_1}+\braket{\psi_2,\psi_2})=1$
        \\So it's normalized
        \item $\Psi(x,t)^*\Psi(x,t)=\frac{1}{2}(e^{\frac{i}{\hbar}E_1t}\psi_1^*(x)+e^{\frac{i}{\hbar}E_2t}\psi_2^*(x))(e^{-\frac{i}{\hbar}E_1t}\psi_1(x)+e^{-\frac{i}{\hbar}E_2t}\psi_2(x))$
        \\$=\frac{1}{2}(\psi_1^*(x)\psi_1(x)+\psi_2^*(x)\psi_2(x))+\frac{1}{2}(e^{\frac{i}{\hbar}(E_1-E_2)t}\psi_1^*(x)\psi_2(x)+e^{\frac{i}{\hbar}(E_2-E_1)t}\psi_1(x)\psi_2^*(x))$
        \\So the $\Psi$ only describe a particle in a stationary state when $E_1=E_2$
        \item $\Psi(x,t)=\frac{1}{\sqrt{2}}(\cos\frac{E_1t}{\hbar}\psi_1(x)-i\sin\frac{E_1t}{\hbar}\psi_1(x)+\cos\frac{4E_1t}{\hbar}\psi_2(x)-i\sin\frac{4E_1t}{\hbar}\psi_2(x))$
        \\$T=\frac{2\pi}{\omega}=\frac{h}{E_1}$
        \\if $|\Psi(x,t+T)|^2=|\Psi(x,t)|^2=$ then
        \\$\frac{1}{2}(e^{-\frac{i}{\hbar}3t}\psi_1^*(x)\psi_2(x)+e^{\frac{i}{\hbar}3t}\psi_1(x)\psi_2^*(x)=\frac{1}{2}(e^{-\frac{i}{\hbar}3E_1(t+T)}\psi_1^*(x)\psi_2(x)+e^{\frac{i}{\hbar}3E_1(t+T)}\psi_1(x)\psi_2^*(x)$
        \\$T=\frac{2\pi}{\omega}=\frac{h}{3E_1}$

    \end{enumerate}
\end{document}


\chapter{Technical Background}\label{Chapter:Theory}
This chapter provides a brief technical overview of the models that we will frequently encounter in the rest of the thesis. A fundamental understanding of these systems is essential for appreciating the problems associated with them and the potential solution for overcoming those problems, which will be discussed in the subsequent chapters.

The key concepts of \textbf{attention} and \textbf{pondering}, whose conceptualization lies in the study of human reasoning are presented in this chapter as well. I elaborate on how these concepts have been the crucial first steps towards making deep neural networks think like human beings, thereby making their decision making process more interpretable. These concepts also serve as the backbone for the first major contribution of this thesis, which is presented in chapter \ref{Chapter:proposals}.

This chapter concludes with an overview of formal language theory which is a prerequisite for understanding the theory behind creation of a new dataset - the second major contribution of this thesis which I present in chapter \ref{Chapter:datasets}.

\section{RNN - A non linear dynamical system} \label{RNN}
We frequently encounter data that is temporal in nature. A few examples would  be, audio signals, videos signals, time series of a stock price and natural language. While traditional feed-forward neural networks such as a multi-layer perceptron (MLP) \citep{rosenblatt1962} are excellent at non linear curve fitting and classification tasks, it is unclear as to how they will approach the problem of predicting the value of a temporal signal $T$ at time $t$ given the states $T_{0}, T_{1}....T_{t-1}$ such that the states over time are not i.i.d. This is owing to the fact that a conventional feed-forward network  is acyclic and thus doesn't have any feedback loops rendering it memoryless. Human beings arguably solve such problems by compressing and storing the previous states in a \lq working\rq{} memory,\citep{Miller1956} \lq chunking\rq{} it \citep{neath2013} \citep{craik2000} and predicting the state $T_t$.

A Recurrent neural network (RNN) \citep{Hopfield1982}\citep{Elman1990} overcomes this restriction by having feedback loops which allow information to be carried from the current time step to the next. While the notion of implementing memory via feedback loops might seem daunting at first the architecture is refreshingly simple. In a process known as unrolling a RNN can be seen as a sequence of MLPs (at different time steps) stacked together. More specifically, RNN maintains a memory across time-steps by projecting the information at any given time step, $t$ onto a hidden(latent) state through parameters $\theta$ which are shared across different time-steps. Figure \ref{bck:rnn} shows a rolled and unrolled RNN cell and the equations of an RNN are as follows:

\begin{equation}\label{nn-dynam}
c_t = tanh(Ux_t + \theta c_{t-1}),
\end{equation}

\begin{equation}
y_t = softmax(Vc_t).
\end{equation}

\begin{figure}
	\begin{minipage}[t]{\textwidth}
		\ifpdf
		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/RNN-unrolled-pdf}
		\else
		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/RNN-unrolled-eps}
		\fi
		\caption{\small Schematic of a RNN \cite{olah}}
		\label{bck:rnn}
	\end{minipage}
\end{figure}

\section{BPTT and Vanishing Gradients}
The conventional method of training a feedforward neural network is a two step process. The first step is called the \textbf{forward pass} when values fed at input layer pass through the hidden layers, are acted upon by (linear or non-linear) activations and come out at the output layer. In the second step called the \textbf{backward pass}  the error computed at the output layer (from the target output) flows backward through the network i.e. by applying the chain rule of differentiation, the error gradient at output layer, is computed with respect to all possible paths right upto input layer and then aggregated. This method of optimization in neural networks is called \textbf{backpropagation} \citep{Rumelhart1986}.

In the case of an RNN the optimization step i.e. the backward pass over the network weights is not just with regards to the parameters at the final time step but over all the time steps across which the weights (parameters) are shared. This is known as Back Propagation Through Time (BPTT) \citep{Werbos1990} and it gives rise to the problem of vanishing (or exploding) gradients in vanilla RNNs. This concept is better elaborated upon through equation in the following section.

\subsection{BPTT for Vanilla RNN}
\begin{equation}
\mathbf{\mathcal{L}\left(x,y\right)} = - \sum_{t}\left(y_t \log\hat{y_t}\right)
\end{equation}

%\begin{equation}
%\frac{\partial \mathbf{\mathcal{L}}}{\partial \alpha_t} = -\left(y_t-z_t\right)
%\end{equation}
The weight $W_{oh}$ is shared across all time steps. $\therefore$ adding the derivatives across the sequence:

\begin{equation}
\frac{\partial \mathbf{\mathcal{L}}}{\partial W_{oh}} = \sum_{t}\frac{\partial \mathbf{\mathcal{L}}}{\partial \hat{y_t}} \frac{\partial  \hat{y_t}}{\partial W_{oh}}
\end{equation}

%\begin{equation}
%\frac{\partial \mathbf{\mathcal{L}}}{\partial b_z} = \sum_{t}\frac{\partial \mathbf{\mathcal{L}}}{\partial z_t} %\frac{\partial  z_t}{\partial b_z}
%\end{equation}
For time-step t $\rightarrow$ t+1:

\begin{equation}
\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial W_{hh}} = \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \hat{y}_{t+1}} \frac{\partial  \hat{y}_{t+1}}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial W_{hh}}
\end{equation}

$W_{hh}$ is shared across time steps, we take the contribution from previous time steps as well, for calculating the gradient at time $t+1$. Summing over the sequence we get:

%\begin{equation}
%\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial W_{hh}} = \frac{\partial %\mathbf{\mathcal{L}}\left(t+1\right)}{\partial \hat{y}_{t+1}} \frac{\partial  \hat{y}_{t+1}}{\partial \mathbf{h}_{t+1}} %\frac{\partial \mathbf{h}_{t+1}}{\partial \mathbf{h}_t} \frac{\partial \mathbf{h}_t}{\partial W_{hh}}
%\end{equation}

\begin{equation}
\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial W_{hh}} = \sum_{\tau=1}^{t+1}\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \hat{y}_{t+1}} \frac{\partial  \hat{y}_{t+1}}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial \mathbf{h}_{\tau}} \frac{\partial \mathbf{h}_{\tau}}{\partial W_{hh}}
\end{equation}

Summing over the whole sequence we get:

\begin{equation} \label{vanishing}
\frac{\partial \mathbf{\mathcal{L}}}{\partial W_{hh}} = \sum_{t}\sum_{\tau=1}^{t+1}\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \hat{y}_{t+1}} \frac{\partial  \hat{y}_{t+1}}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial \mathbf{h}_{\tau}} \frac{\partial \mathbf{h}_{\tau}}{\partial W_{hh}}
\end{equation}

From equation \ref{vanishing} it is clear than the gradient of a RNN can be expressed as a recursive product of $\frac{\partial h_t}{\partial h_{t-1}}$. \textbf{If this derivative is $\ll 1$ or $\gg 1$, the gradient would vanish or explode respectively} when the network is trained over longer time-steps. In the former case error back-propagated would be too low to change the weights (the training would freeze) while in the latter it would never converge.

Additionally revisiting equation \ref{nn-dynam} and rewriting it as follows:
\begin{equation}\label{ndq}
h^{(t)} = f(x^{(t)}, h^{(t-1)}),
\end{equation}
it is not difficult to see that in the absence of an external input $x^{(t)}$ an RNN induces a a dynamical system. The RNN therefore, can be viewed as a dynamical system with the input as an external force (map) that drives it. A dynamical system can posses a set of points which are invariant under any map. These points are called the \textbf{attractor states} of a dynamical system. These set of points can contain a single point (fixed attractor), a finite set of points (periodic attractor) or an infinite set of points (strange attractor). The type of attractor in a RNN unit depends on the initialization of the weight matrix for the hidden state \citep{Bengio1993}. Now under the application of map (input) if $\frac{\partial h_t}{\partial h_{k}}$ (for a large $t - k$ i.e. long term dependency), go to zero one can argue that the state $h_t$ is in the basin of one of the attractor states. This implies that in order to avoid the vanishing gradient problem a RNN cell must stay close to the \lq boundaries between basins of attraction\rq{} \citep{Pascanu2012}.

Owing to this problem of vanishing (or exploding) gradients a vanilla RNN can't keep track of long term dependencies which is arguably critical for tasks such as speech synthesis, music composition or neural machine translation. The architectural modifications which solved the vanishing gradient problem and are the current de-facto RNN cell(s) are presented in the next section.

\iffalse
\begin{equation}
\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial W_{hx}} = \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial W_{hx}}
\end{equation}

\begin{equation}
\begin{split}
\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial W_{hx}} & = \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial W_{hx}} + \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \mathbf{h}_t} \frac{\partial \mathbf{h}_t}{\partial W_{hx}} \\
& = \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial W_{hx}} + \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial \mathbf{h}_t}\frac{\partial \mathbf{h}_t}{\partial W_{hx}}
\end{split}\end{equation}

\begin{equation}
\frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial W_{hx}} = \sum_{\tau=1}^{t+1} \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial \mathbf{h}_{\tau}} \frac{\partial \mathbf{h}_{\tau}}{\partial W_{hx}}
\end{equation}

\begin{equation}
\frac{\partial \mathbf{\mathcal{L}}}{\partial W_{hx}} = \sum_{t} \sum_{\tau=1}^{t+1} \frac{\partial \mathbf{\mathcal{L}}\left(t+1\right)}{\partial \hat{y}_{t+1}} \frac{\partial \hat{y}_{t+1}}{\partial \mathbf{h}_{t+1}} \frac{\partial \mathbf{h}_{t+1}}{\partial \mathbf{h}_{\tau}} \frac{\partial \mathbf{h}_{\tau}}{\partial W_{hx}}
\end{equation}
\fi

\section{Gated RNNs}
The problem of vanishing (or exploding) gradients makes a vanilla RNN unsuitable for long term dependency modeling. However, if for instance, the RNN was to compute an identity function then the gradient computation wouldn't vanish or explode since the Jacobian is simply an identity matrix. Now while an identity initialization of recurrent weights by itself isn't  interesting it brings us to the underlying principle behind gated architectures i.e. the mapping from memory state at one time step to the next is close to identity function.

\begin{figure}[ht]
	\begin{subfigure}[b]{0.5\linewidth}
		\centering
		\ifpdf
		\includegraphics[width=0.95\linewidth]{./figs/Lstm-pdf}
		\else
		\includegraphics[width=0.95\linewidth]{./figs/Lstm-eps}
		\fi
		\caption{LSTM\citep{Zheng2017}}
		\label{lstm}
		\vspace{4ex}
	\end{subfigure}%%
	\begin{subfigure}[b]{0.5\linewidth}
		\centering
		\ifpdf
		\includegraphics[width=0.95\linewidth]{./figs/Gru-pdf}
		\else
		\includegraphics[width=0.95\linewidth]{./figs/Gru-eps}
		\fi
		\caption{GRU \citep{olah}}
		\label{gru}
		\vspace{4ex}
	\end{subfigure}
	\caption{Gated RNNs}
	\label{conf}
\end{figure}

\subsection{LSTM}
Long Short Term Memory (LSTM) introduced by \cite{LSTM} is one of the two most widely used gated RNN architectures in use today. The fact that it has survived all the path-breaking innovations in the field of deep learning for over twenty years to still be the state of the art in sequence modeling speaks volumes about the architecture's ingenuity and strong fundamentals.

The fundamental principle behind the working of a LSTM is, to alter the memory vector only selectively between time steps such that the memory state is preserved over long distances. The architecture is explained as follows:

\begin{equation}
i= \sigma(x_t U^{(i)}, m_{t-1}W^{(i)})
\end{equation}

\begin{equation}
f= \sigma(x_t U^{(f)}, m_{t-1}W^{(f)})
\end{equation}

\begin{equation}
o= \sigma(x_t U^{(o)}, m_{t-1}W^{(o)})
\end{equation}

\begin{equation}
\widetilde {c_t}= \tanh(x_t U^{(g)}, m_{t-1}W^{(g)})
\end{equation}

\begin{equation}
c_t = c_{t-1} \odot f + \widetilde{c_t} \odot i
\end{equation}

\begin{equation}
m_t = tanh(c_t) \odot o
\end{equation}

\begin{itemize}
	\item \textbf{input gate $i^{(t)}$:} The input computes a new cell state based on the current input and the previous hidden state and decides how much of this information to ``let through" via. a sigmoid activation.
	\item \textbf{forget gate $f^{(t)}$:} The forget gate decides what to remember and what to forget for the new memory based on the current input and the previous hidden state. The sigmoid activation acts like a switch where 1 implies remember everything while 0 implies forget everything.
	\item \textbf{output gate $o^{(t)}$:} The output gate then determines determines (via a sigmoid activation) the amount of this internal memory to be exposed to the top layers (and subsequent timesteps) of the network.
	\item \textbf{The input modulation $g^{(t)}$} computed based on the present input and the previous hidden state (which is exposed to the output) yields candidate memory for the cell state via a tanh layer. The hadamard products of input gate and candidate memories is added to the hadamard product of forget gate and previous cell state to yield the new cell state.
	\item \textbf{hidden state $m^{(t)}$:} A hadamard product of the hyperbolic tangent of current cell state and output gate yields the current hidden state.
\end{itemize}

\subsection{GRU}
The Gated Recurrent Unit (GRU) introduced by \cite{GRU} are a new type of gated RNN architecture whose details are as follows:

\begin{equation}
z_t= \sigma(x_t U^{(z)}, m_{t-1}W^{(z)})
\end{equation}

\begin{equation}
r_t= \sigma(x_t U^{(r)}, m_{t-1}W^{(r)})
\end{equation}

\begin{equation}
\widetilde {m_t}= \tanh(x_t U^{(g)},r_t \odot m_{t-1}W^{(g)})
\end{equation}

\begin{equation}
m_t= (1-z_t)m_{t-1} + z_t\widetilde{m_t} %\sigma(x_t U^{(i)} m_{t-1}W^{(i)})
\end{equation}

\begin{itemize}
	\item \textbf{update gate $z_{(t)}$:} The update gate is the filter which decides how much of the activations/memory to be update at any given time step.
	\item \textbf{reset gate $r_{(t)}$:} The reset gate is similar to the forget gate in a LSTM. When its value is close to zero it allows the cell to forget the previously computed state.
	\item \textbf{The input modulation $g_{(t)}$} just as in the case of the LSTM  serves the purpose of yielding candidate memories for the new cell state.
	\item \textbf{hidden state $m_{(t)}$:} The current hidden state is a weighted average of the previous hidden state and the candidate hidden state weighted by ``(1 - update gate)" and the ``(update gate)" respectively.
\end{itemize}

While there are a lot of similarities between a GRU and a LSTM the most striking difference is the lack of an output gate in a GRU. Unlike a LSTM a GRU doesn't control how much of its internal memory to expose to the rest of the units in the network. The GRU therefore has fewer parameters due to the lack of an output gate and is computationally less intensive in comparison to a LSTM.

\section{Seq2Seq Models}\label{background:s2s}
\begin{figure}
	\begin{minipage}[t]{\textwidth}
		\ifpdf
		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/s2s-pdf}
		\else
		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/s2s-eps}
		\fi
		\caption{Schematic of a Seq2Seq \citep{manish}}
		\label{bck:s2s}
	\end{minipage}
\end{figure}

Sequence-to-sequence (seq2seq) models introduced by \cite{Sutskever2014}, \cite{GRU} are a class of probabilistic generative models that let us learn the mapping from a variable length input to a variable length output. While initially conceived for machine translation, they have been applied successfully to the tasks of speech recognition, question answering and text summarization \citep{Vinyals2015} \citep{anderson2018bottom} \citep{lu2017knowing}.

Neural networks have been shown to be excellent at learning rich representation from data without the need for extensive feature engineering \citep{Hinton2006}. RNNs are especially adept at learning features and long term dependencies in sequential data (section \ref{RNN}). The simple yet effective idea behind a seq2seq model is learning a fixed size (latent) representation of a variable length input and then generating a variable length output by conditioning it on this latent representation and the previous portion of the output sequence.

\begin{equation} \label{decoder:eqn1}
	h^{(t)} = f(x^{(t)}, h^{(t-1)}, v)
\end{equation}

\begin{equation} \label{decoder:eqn2}
	P(x^{(t+1)}|x^{(t)}, x^{(t-1)},.....,x^{(1)}) = g(x^{(t)}, h^{(t)}, v)
\end{equation}

It can be seen from equation \ref{decoder:eqn2} that the decoder is auto-regressive with the long term temporal dependencies captured in its hidden state. The term $\mathbf{v}$ represents the summary of the entire input state compressed into a fixed length vector (last hidden state of encoder) viz. the \textbf{latent space}. This encoder- decoder network is then jointly trained via. cross entropy loss between the target and the predicted sequence.

\subsection{Se2Seq with Attention}\label{mtv:attn}
It was shown by \cite{Cho2014} that the performance of a basic encoder-decoder model as explained in section \ref{background:s2s} is inversely related to the increase in length of the input sentence. Therefore in lines with concepts of the selective attention and attentional blink in human beings \citep{purves2013principles}, \cite{Bahdanau2014} and later  \cite{Luong2015} showed that soft selection from source states where most relevant information can be assumed to be concentrated during a particular translation step in neural machine translation (NMT) leads to improved performance. In the \cite{Bahdanau2014} framework of attention, the equations \ref{decoder:eqn1} and \ref{decoder:eqn2} are modified as follows:

\begin{equation}\label{attn:eqn1}
h^{(t)} = f(x^{(t)}, h^{(t-1)}, c^{(t)}),
\end{equation}

\begin{equation} \label{attn:eqn2}
P(x^{(t+1)}|x^{(t)}, x^{(t-1)},.....,x^{(1)}) = g(x^{(t)}, h^{(t)}, c^{(t)}).
\end{equation}

Here unlike the traditional seq2seq model the probability of emission of the output at time step $t$ isn't conditioned on a fixed summary representation $\mathbf{v}$ of the input sequence. Rather it is conditioned on a context vector $\mathbf{c^{(t)}}$ which is distinct at each decoding step. The context vector is calculated using a sequence of encoder outputs $(s^1, s^2, ....,s^N)$ to which the the input sequence is mapped such that an encoder output $s^i$ contains a representation of the entire sequence with maximum information pertaining to the $i_{th}$ word in the input sequence. The context vector is then calculated as follows:

\begin{equation}\label{attn:eqn3}
c^{(t)}  = \sum_{j=1}^N \alpha_{tj} s^j,
\end{equation}
where:
\begin{equation}\label{attn:eqn4}
\alpha_{tj} = \text{softmax}(e(h^{(t-1)}, s^j)).
\end{equation}
where $e$ is a alignment/matching/similarity measure between the decoder hidden state $h^{(t-1)}$ i.e. just before the emission of output $x^{(t)}$. The alignment function can be a dot product of the two vectors or a feed-forward network that is jointly trained with the encoder-decoder model. The $\alpha_{tj}$ is an attention vector that weighs the encoder outputs at a given decoding step. \cite{Vaswani2017} view the entire process of attention and context generation as a series of functions applied to the tuple (query, key, value), with the first step being an \textbf{attentive read} step where a scalar matching score between the query and key ($h^{(t-1)}, s^j$) is calculated followed by computation of attention weights $\alpha_{tj}$. Weighted averaged of the \lq values{}\rq\ using the attention weights is then done in the \textbf{aggregation} step.

\section{Pondering}\label{bck:ponder}

%\begin{figure}
%	\begin{minipage}[t]{\textwidth}
%		\ifpdf
%		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/act-pdf}
%		\else
%		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/act-eps}
%		\fi
%		\caption{\small Adaptive Computation Time}
%		\label{mtv:ponder}
%	\end{minipage}
%\end{figure}

\begin{figure}[ht]
	\begin{subfigure}[b]{0.5\linewidth}
		\centering
		\includegraphics[width=0.9\linewidth]{./figs/act1-eps}
		\caption{Fixed Computation Time}
		\label{fixed}
		\vspace{4ex}
	\end{subfigure}%%
	\begin{subfigure}[b]{0.5\linewidth}
		\centering
		\includegraphics[width=0.95\linewidth]{./figs/act2-eps}
		\caption{Adpative Computation Time}
		\label{adaptive}
		\vspace{4ex}
	\end{subfigure}
	\caption{\small Schematic of Adaptive Computation Time}
	\label{mtv:ponder}
\end{figure}
The task of positioning a problem and solving belong to different classes of time complexity with the latter requiring more time than the former. \cite{Graves2016} argued that for a given RNN unit it is reasonable to allow for variable computation time for each input in a sequence since some parts of the input might be inherently more complex than the others and thereby require more computational steps. A good example of this would be \textit{spaces between words and ends of sequences}.

Human beings overcome similar obstacles by allocating more time to a difficult problem as compared to a simpler problem. Therefore a naive solution would be to allow a RNN unit to have a large number of hidden state transitions (without penalty on amount of computations performed) before emitting an output on a given input. The network would therefore learn to allocate as much time as possible to minimize its error thereby making it extremely inefficient. \cite{Graves2016} proposed the concept of \textbf{adaptive computation time} to have a trade-off between accuracy and computational efficacy in order to determine the minimum number of state transitions required to solve a problem \footnote{Theoretically this is akin to halting on a given problem or finding the Kolmogorov Complexity of the data, both of which are unsolvable}.

\textbf{A}daptive \textbf{C}omputation \textbf{T}ime (ACT) achieves the above outlined goals by making two simple modifications to a conventional RNN cell, which are presented as follows:

\subparagraph{Sigmoidal Halting Unit} If we revisit the equations of a vanilla RNN from section \ref{RNN}, they can be summarized as:
\begin{equation}
\begin{aligned}
h_t &= f(Ux_t + Wc_{t-1}), \\
y_t &= g(Vc_t).
\end{aligned}
\end{equation}
ACT now allows for \textit{variable state transitions} ($c_t^1, c_t^2,...., c_t^{N(t)}$) and by extension an \textit{intermediate output sequence} ($y_t^1, y_t^2,...., y_t^{N(t)}$) at any given input step \textit{t} as follows:
\begin{equation}
\begin{aligned}
c_t^n &= \begin{cases} f(Ux_t^1 + Wc_{t-1})\ \text{if}\ n = 1 \\ f(Ux_t^n + Wc_t^{n-1})\ \text{if}\ n \neq 1  \end{cases}, \\
y_t^n &= g(Vc_t^n).
\end{aligned}
\end{equation}

A sigmoidal halting unit (with its associated weight matrix $S$) is now added to the network in order to yield a halting probability $p_t^n$ at each state transition as follows:
\begin{equation}
\begin{aligned}
h_t^n &= \sigma(Sc_t^n), \\
p_t^n &= \begin{cases} R(t)\ \text{if}\ n = N(t) \\ h_t^n\ \text{if}\ n \neq N(t)  \end{cases},
\end{aligned}
\end{equation}
where:
\begin{equation}
\begin{aligned}
N(t) &= min\{m : \sum_{n=1}^m h_t^n \geq 1 - \epsilon\}, \\
R(t) &= 1 - \sum_{n=1}^{N(t)-1} h_t^n,
\end{aligned}
\end{equation}
and $\epsilon$ is a small constant.

Each ($n^{th}$) hidden state and output transition at input state \textit{t} are now weighted by the corresponding halting probability $p_t^n$ and summed over all the updates $N(t)$ to yield the final hidden state $c_t$ and output $y_t$ at a given input step. Figure \ref{mtv:ponder} outlines the difference between a standard RNN cell and an ACT RNN cell by showing variable state transitions for input $x$ and $y$ respectively with the corresponding probability associated with each update step. It can be noted that $\sum_{n=1}^{N(t)} p_t^n = 1\ \text{and}\ 0 \leq p_t^n \leq 1\ \forall n$, and therefore, it constitutes a valid probability distribution.

\subparagraph{Ponder Cost} If we don't put any penalty on the number of state transitions then the network would become computationally inefficient and would \lq \textbf{ponder}{}\rq\ for long times even on simple inputs in order to minimize its error. Therefore in order to limit the variable state transitions ACT adds a ponder cost $\mathcal{P}(x)$ to the total loss of the network as follows:
given an input of length $T$ the ponder cost at each time step $t$ is defined as:

\begin{equation}
\rho_t = N(t) +\ R(t)
\end{equation}

\begin{equation}
\begin{aligned}
\mathcal{P}(x) &= \sum_{t=1}^T \rho_t, \\
\widetilde{\mathcal{L}}(x,y) &= \mathcal{L}(x,y) + \tau\mathcal{P}(x),
\end{aligned}
\end{equation}
where $\tau$ is a penalty term hyperparameter (that needs to be tuned) for the ponder loss.

\section{Formal Language Theory}\label{flt}
The field of formal language theory (FLT) concerns itself with the syntactic structure of a formal language (=set of strings) without much emphasis on the semantics. More precisely a formal language $L$ is a set of strings with the constituent units/words/morphemes taken from a finite vocabulary $\Sigma$. It is more apt to define the concept of a formal grammar before proceeding further. A formal grammar $G$ is a quadruple $\langle \Sigma, NT, S, R \rangle$ where $\Sigma$ is a finite vocabulary as previously defined, $NT$ is a finite set of non-terminals, $S$ the start symbol and $R$ the finite set of valid production rules. A production rule can be expressed as $\alpha \rightarrow \beta$ and can be understood as a substitution of $\alpha$ with $\beta$ and $\alpha, \beta$ coming from the following sets for a \textbf{valid} production rule:

\begin{equation}\label{term-nonterm}
	\alpha \in (\Sigma \cup NT)^{*}NT(\Sigma \cup NT)^{*} \qquad \beta \in (\Sigma \cup NT)^{*} \footnote{The \lq *\rq\ denotes Kleene Closure and for a set of symbols say $X$, $X^{*}$ denotes a set of all strings that can be generated using symbols from X, including the empty string $\epsilon$ }.
\end{equation}

From equation \ref{term-nonterm} it is easy to see that the left hand side of the production rule can never be null ($\epsilon$) and must contain at-least one non-terminal. Now a formal language $L(G)$ can be defined as the set of all strings  \textit{generated} by grammar $G$ such that the string consists of morphemes only from $\Sigma$, and has been generated by a finite set of rule ($R$) application after starting from $S$. The \textit{decidability} of a grammar, is the verification (by a Turing machine or another similar computational construct e.g. a finite state automaton (FSA)) of whether a given string has been generated by that grammar or not (the \textit{membership} problem). A grammar is decidable if the membership problem can be solved for all given strings.

\begin{figure}
	\begin{minipage}[t]{0.8\textwidth}
		\ifpdf
		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/chomsky_h-pdf}
		\else
		\includegraphics[width=\linewidth,keepaspectratio=true]{./figs/chomsky_h-eps}
		\fi
		\caption{\small Chomsky Hierarchy}
		\label{mtv:ch}
	\end{minipage}
\end{figure}

\subsection{Chomsky Hierarchy}\label{flt:ch}
\cite{Chomsky1956} introduced a nested hierarchy for different formal grammars of the form $C_1 \subsetneq C_2 \subsetneq C_3 \subsetneq C_4$ as shown in figure \ref{mtv:ch}. The different classes of grammar are progressively strict subsets of the class just above them in the hierarchy. These classes are not just distinguished by their rules or the languages they generate but also on the computational construct needed to decide the language generated by this grammar. We now take a closer look at the classes in this hierarchy. Please note that for each of these classes the grammar definition is G = $\langle \Sigma, NT, S, R \rangle$.
\subparagraph{Recursively Enumerable} This grammar is characterized by no constraints on the production rules $\alpha \rightarrow \beta$. Therefore any valid grammar is recursively enumerable. The language generated by this grammar is called recursively enumerable language (REL) and is accepted by a Turning Machine.
\subparagraph{Context-Sensitive} This is a grammar in which the left hand side of the production rule i.e. $\alpha$ has the same definition as above (equation \ref{term-nonterm}) but an additional constraint of the form $|\alpha| \leq |\beta|$ is now imposed on the production rules. This is turn leads to $\beta \in (\Sigma \cup NT)^{+}$, i.e. the right hand side of the production rule is now under Kleene Plus closure \footnote{$(\Sigma \cup NT)^{+} = (\Sigma \cup NT)^{*} - \epsilon$}. The non production of $\epsilon$ in context-sensitive grammars poses a problem to the hierarchy because the production of null symbol isn't restricted in its subclasses. While keeping the hierarchy as it is \cite{Chomsky1963} resolved this paradox by defining noncontracting grammmar which is \textit{weakly equivalent}(generates same set of string) to the context sensitive grammar. Noncontracting grammars allow the $S \rightarrow \epsilon$ production. Context sensitive grammars generate context sensitive languages which are accepted by a linear bounded turing machine. While in principle this grammar is decidable, the problem is PSPACE hard and can be so complex, that it is practically intractable \citep{Jager2012}.
\subparagraph{Context-Free} This grammar is described by production rules of the form $A \rightarrow \alpha$ where $A \in NT$ and $\alpha \in (\Sigma \cup NT)^{*}$, such that $|A|=1$.  Context free grammar lead to context free languages (CFL) which are hierarchical in structure, although it is possible that same CFL can be described by different context free grammars, leading to different hierarchical syntactic structures of the language. A CFG is decidable in cubic time of length of string by push down FSA. A push down automaton employs a running stack of symbols to decide its next transition. The stack can also be manipulated as a side effect of the state transition.
\subparagraph{Regular} This grammar is characterized by production rules of the form $A \rightarrow \alpha$ or $A \rightarrow \alpha B$ where $\alpha \in \Sigma^{*}$ and $(A, B) \in NT$. The non terminal in production can therefore be viewed as the next state(s) of a finite state automaton (FSA) while the terminals are the emissions. Regular grammars are decidable in linear time of length of string by an FSA.

\subsection{Subregular Hierarchy}\label{flt:sh}
The simplest class of languages encountered in section \ref{flt:ch} were regular languages that can be described using a FSA. \cite{Jager2012} however argue that the precursor to human language faculty would require lower cognitive capabilities and it stands to reason that even simpler structures can exist in the \lq Regular\rq{} domain. They therefore introduced the concept of subregular languages. If a language can be described by a mechanism even simpler than the FSA then it is a subregular language. While far from the expressive capabilities of regular languages which in turn are the least expressive class in the Chomsky hierarchy, subregular languages provide an excellent benchmark to test basic concept learning and pattern recognition ability of any intelligent system.

\subparagraph{Strictly local languages.} We start with a string $w$ and we are given a lookup table of k-adjacent characters known as \textit{k-factors}, drawn from a particular language. The lookup table therefore serves the role of the language description. A language is \textit{k}-local, if every \textit{k}-factor seen by a \textit{scanner} with a windows of size $k$ sliding over the string $w$, can be found in the aforementioned lookup-table. A $SL_k$ language description, is just the set of k-factors prefixed and suffixed by a start and end symbol, say $\#$. E.g. $SL_2 = \{\#A,AB,BA,B\#\}$

\subparagraph{Locally k-testable languages.} Instead of sliding a scanner over \textit{k}-factors we consider all the \textit{k}-factors to be atomic and build \textit{k}-expression out of them using \textit{propositional logic}. This language description is locally k-testable. As in the case of strictly local languages, scanner of window size $K$ slides over the string and records for every k-factor in vocabulary its occurrence or nonoccurence in the string. The output of this scanner is then fed to a boolean network which verifies the k-expressions. E.g. 2-expression $(\neg \# B) \wedge A$, is a set of strings that doesn't start with B and consists of atleast one A.

\subparagraph{Remarks on Chomsky Hierarchy:}It is easy to see by looking at the production rules of all the grammars in Chomsky Hierarchy that, solving the languages generated by them requires an understanding of these rules. This allows us to create artificial languages such as SCAN \citep{Lake2017} which are context-free, in order to test compositionality in deep neural networks. That said, it is worth noticing that while the grammars in Chomsky Hierarchy are finite, the languages they generate can be infinite. For an infinite language one can argue that a model that can infer the grammar from the given strings is the one that will generalize well to unseen strings. However if the language itself only contains a finite number of strings then although the language itself is compositional, it can be solved also by pure memorization.


\documentclass{article}
\usepackage[legalpaper, portrait, margin=1in]{geometry}
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithm}
\usepackage{booktabs}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage[font=small]{caption}
%\usepackage{subcaption}
%\expandafter\def\csname ver@subfig.sty\endcsname{}
\usepackage{tabularx}
\usepackage{subfig}
\usepackage{pgffor}
\usepackage{hyperref}
\usepackage{soul}

% for tables
\newcommand{\centertab}[1]{\multicolumn{1}{|c|}{\textbf{#1} }}
\newcommand{\bigcell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}

\begin{document}
\input{math_definitions}

\section{Time Variable LQR}
\newcommand{\vxt}{\vx_{[t]}}
\newcommand{\vut}{\vu_{[t]}}
\newcommand{\vSt}{\vS_{[t]}}
\newcommand{\vSp}{\vS_{[t+1]}}
\newcommand{\vQt}{\vQ_{[t]}}
\newcommand{\vRt}{\vR_{[t]}}
\newcommand{\vAt}{\vA_{[t]}}
\newcommand{\vBt}{\vB_{[t]}}

\begin{itemize}
  \item System: $\vx_{[t+1]} = \vA_{[t]} \vx_{[t]} + \vB_{[t]} \vu_{[t]}$
  \item Cost:
    \begin{align*}
      J = \sum_{t=1}^T g(\vx_{[t]}, \vu_{[t]})
        = \sum_{t=1}^T \vx_{[t]}^T \vQ_{[t]} \vx_{[t]} + \vu_{[t]}^T \vR_{[t]} \vu_{[t]}
    \end{align*}
  \item Value function:
    \begin{align}
      V(\vx_{[t]}, t) = \min_{\vu_{[t]}}{\sgroup{
          g(\vx_{[t]}, \vu_{[t]})
          + \func{V}{f(\vx_{[t]}, \vu_{[t]}, t) } }} \label{eq:value_function}
    \end{align}
    we assume that the value function takes the following form:
    \begin{align*}
        V(\vx_{[t]}, t) = \vx_{[t]}^T \vS_{[t]} \vx_{[t]}
    \end{align*}
  \item solving Eq. \ref{eq:value_function}:
    \begin{align}
      \vxt^T \vSt \vxt &= \min_{\vut} \vxt^T \vQt \vxt + \vut^T \vRt \vut
          + \func{f}{\vxt, \vut, t}^T \vSp \func{f}{\vxt, \vut, t}
          \label{eq:value_expanded} \\
          &= \min_{\vut} \vxt^T \vQt \vxt + \vut^T \vRt \vut
          + \group{(\vAt \vxt)^T + (\vBt \vut)^T} \vSp \group{\vAt \vxt + \vBt \vut}
           \nonumber
    \end{align}
    dropping time notation, assume everything depends on $t$, while
      $\hat{\vS} = \vSp$

      \renewcommand{\vxt}{\vx}
      \renewcommand{\vut}{\vu}
      \renewcommand{\vSt}{\vS}
      \renewcommand{\vSp}{\hat{\vS}}
      \renewcommand{\vQt}{\vQ}
      \renewcommand{\vRt}{\vR}
      \renewcommand{\vAt}{\vA}
      \renewcommand{\vBt}{\vB}

    \begin{align*}
       Vt =& \vxt^T \vQt \vxt + \vut^T \vRt \vut
             + (\vAt \vxt)^T \vSp (\vAt \vxt)
             + 2 (\vAt \vxt)^T \vSp (\vBt \vut)
             + (\vBt \vut)^T \vSp (\vBt \vut)
    \end{align*}
    solving the minimization problem:
    \begin{align}
      & \dfrac{\partial Vt}{\partial \vut} =
          2 \vRt \vut + 2 \vBt^T \vSp \vAt \vxt + 2 \vBt^T \vSp \vBt \vut = 0 \nonumber \\
      & \underbrace{(\vRt + \vBt^T \vSp \vBt)}_{\vM}\vut =
                        - \underbrace{\vBt^T \vSp \vAt}_{\vC} \vxt  \nonumber \\
      & \vut = - \vM^{-1} \vC \vxt   \label{eq:optimalu}
    \end{align}
    Now we need to solve for $\vSp$ by replacing \ref{eq:optimalu} into
      \ref{eq:value_expanded}:
    \begin{align*}
      \vxt^T \vSt \vxt & =
            \vxt^T \vQt \vxt
          + \vxt^T \vAt^T \vSp \vAt \vxt
          + 2 \vxt^T \underbrace{\group{\vAt^T \vSp \vBt}}_{\vC^T} \vut
          + \vut^T \underbrace{\group{\vRt + \vBt^T \vSp \vBt}}_{\vM} \vut \\
        & =
            \vxt^T \vQt \vxt
          + \vxt^T \vAt^T \vSp \vAt \vxt
          - 2 \vxt^T \vC^T \vM^{-1} \vC \vxt
          + \vxt^T \vC^T \vM^{-T} \vM \vM^{-1} \vC \vxt \\
        & =
            \vxt^T \vQt \vxt
          + \vxt^T \vAt^T \vSp \vAt \vxt
          - \vxt^T \vC^T \vM^{-1} \vC \vxt \\
        & =
            \vxt^T \group{\vQt
                          + \vAt^T \vSp \vAt
                          - \vC^T \vM^{-1} \vC} \vxt
    \end{align*}
    Therefore, we have:
    \begin{align*}
      \vSt & = \vQt + \vAt^T \vSp \vAt
                    - \vC^T \vM^{-1} \vC \\
           & = \vQt + \vAt^T \vSp \vAt
                    - \vAt^T \vSp \vBt \group{\vRt + \vBt^T \vSp \vBt}^{-1} \vBt^T \vSp \vAt
    \end{align*}

\end{itemize}

\end{document}


\newpage\section{Tricks and Lemmas}

	\theo{https://en.wikipedia.org/wiki/Minkowski's_theorem}{Minkowski's theorem}{Any convex set in $\R^n$, which is symmetric with respect to the origin and with volume greater than $2^n d(L)$ contains a non-zero lattice point.}

	\subsection{Ad Hocs}

		\begin{enumerate}

			\item $x^2+1 = (x+i)(x-i)$ [USAMO 2014 P1]
			\item Add. Everything. Up.
			\item Send SHIT to the infinity.

		\end{enumerate}