Autoregressive Models

Take $p = 1$ first. Iterating $\tilde r_t = \phi_1 \tilde r_{t-1} + a_t$ gives $\tilde r_t = \sum_{i=0}^{n} \phi_1^i a_{t-i} + \phi_1^{n+1}\tilde r_{t-n-1}$. The sum converges in mean square as $n\to\infty$ precisely when $|\phi_1| < 1$, since $\E[(\sum_{i} \phi_1^i a_{t-i})^2] = \sigma^2\sum_i \phi_1^{2i} = \sigma^2/(1-\phi_1^2)$ is finite exactly then, and the residual $\phi_1^{n+1}\tilde r_{t-n-1}$ vanishes in $L^2$. The single root of $\phi(z) = 1 - \phi_1 z$ is $z = 1/\phi_1$, outside the unit circle iff $|\phi_1| < 1$.

For general $p$, stack $R_t = (\tilde r_t, \tilde r_{t-1}, \dots, \tilde r_{t-p+1})^\top$ and $e_t = (a_t, 0, \dots, 0)^\top$. Then Equation (1) is the first-order recursion $R_t = \Phi\, R_{t-1} + e_t$ with the $p \times p$ companion matrix whose first row is $(\phi_1, \dots, \phi_p)$ and whose subdiagonal is the identity. Iterating gives $R_t = \sum_{i\ge 0}\Phi^i e_{t-i}$, whose $L^2$ norm is bounded by $\sigma\sum_{i\ge 0}\lVert\Phi^i\rVert$. By Gelfand's formula $\lVert\Phi^i\rVert^{1/i}\to\varrho(\Phi)$, so the series converges in $L^2$ iff the spectral radius $\varrho(\Phi) < 1$. Expanding the companion determinant along its first row gives $\det(zI - \Phi) = z^p - \phi_1 z^{p-1} - \cdots - \phi_p$, and the substitution $z = 1/w$ yields the exact identity $\det\bigl((1/w)I - \Phi\bigr) = w^{-p}\phi(w)$, so the eigenvalues of $\Phi$ are precisely the reciprocals of the roots of $\phi$. Hence $\varrho(\Phi) < 1$ iff every root of $\phi$ lies outside the unit circle, and the stationary causal solution is the first component of $R_t$ [3].

Multiply the centered recursion by $\tilde r_{t-\ell}$ and take expectations. Since $a_t$ is uncorrelated with $\tilde r_{t-\ell}$ for $\ell > 0$, the shock term drops, leaving $\gamma_\ell = \sum_{i=1}^p \phi_i\,\gamma_{\ell-i}$. Dividing by $\gamma_0$ gives Equation (3). The case $\ell = 0$ adds the shock variance, $\gamma_0 = \sum_{i=1}^p \phi_i \gamma_i + \sigma^2$.

Setting $\ell = 1$ in Equation (3) and using $\rho_{-1} = \rho_1$ gives $\rho_1 = \phi_1 + \phi_2\rho_1$, hence $\rho_1 = \phi_1/(1-\phi_2)$. The recursion has characteristic equation $x^2 - \phi_1 x - \phi_2 = 0$ with roots $x_{1,2} = \tfrac12(\phi_1 \pm \sqrt{\phi_1^2 + 4\phi_2})$ and product $x_1 x_2 = -\phi_2$. When $\phi_1^2 + 4\phi_2 \ge 0$ the roots are real and $\rho_\ell = c_1 x_1^\ell + c_2 x_2^\ell$ is a mixture of two exponentials. When $\phi_1^2 + 4\phi_2 < 0$ they are a conjugate pair $x = |x|e^{\pm i\theta}$ of modulus $|x| = \sqrt{x_1 x_2} = \sqrt{-\phi_2}$ and argument $\cos\theta = \operatorname{Re}(x)/|x| = \phi_1/(2\sqrt{-\phi_2})$, so the real solution $\rho_\ell \propto |x|^\ell \cos(\ell\theta + \varphi)$ oscillates with period $2\pi/\theta$, which is Equation (4).

The spectral density is the Fourier transform of the autocovariances, $f(\omega) = \tfrac{1}{2\pi}\sum_\ell \gamma_\ell e^{-i\ell\omega}$. For the linear process $\tilde r_t = \psi(B)a_t$ the autocorrelation post gives $\gamma_\ell = \sigma^2\sum_j \psi_j\psi_{j+\ell}$, so, substituting $m = j+\ell$ to factorise the double sum,

Since $\psi(z) = 1/\phi(z)$ from Equation (2), $|\psi(e^{-i\omega})|^2 = |\phi(e^{-i\omega})|^{-2}$.

Taking $\E[\,\cdot \mid \mathcal F_h]$ in Equation (1) at time $h+\ell$ and using $\E[a_{h+\ell}\mid\mathcal F_h] = 0$ gives the recursion. From the causal representation, the error is $e_h(\ell) = \sum_{j=0}^{\ell-1}\psi_j a_{h+\ell-j}$, so $\Var[e_h(\ell)] = \sigma^2\sum_{j=0}^{\ell-1}\psi_j^2$, which rises to $\sigma^2\sum_{j\ge 0}\psi_j^2 = \gamma_0$. Since $\psi_j\to 0$ under stationarity, the forecast $\hat r_h(\ell) = \mu + \sum_{j\ge \ell}\psi_j a_{h+\ell-j}$ tends to $\mu$.

The exact solution over $[t, t+\Delta]$ is $r_{t+\Delta} = \theta(1 - e^{-\kappa\Delta}) + e^{-\kappa\Delta} r_t + \int_t^{t+\Delta} e^{-\kappa(t+\Delta - s)}\sigma_c\,dW_s$. The stochastic integral is Gaussian, independent across disjoint steps, with mean zero and variance $\sigma_c^2\int_0^\Delta e^{-2\kappa u}\,du = \sigma_c^2(1-e^{-2\kappa\Delta})/(2\kappa)$. This is Equation (1) at $p = 1$ with $\phi_0 = \theta(1-e^{-\kappa\Delta})$ and $\phi_1 = e^{-\kappa\Delta}$, and $\kappa > 0$ forces $\phi_1 \in (0,1)$, so the sampled process is stationary.