Serial Dependence: moving averages, ARMA, and long memory

The autoregression captures dependence that decays. Its dual is the moving average, whose dependence truncates, and their synthesis is the ARMA model, whose economy comes from a ratio of two polynomials. This post completes the linear picture the autocorrelation and autoregression posts began. It builds the moving average and its invertibility, the ARMA model and the two representations long division exposes, the seasonal airline model, and the fractionally integrated processes whose autocorrelation decays only hyperbolically, then closes with valid inference for a regression whose errors carry serial correlation [1], [2].

#Moving averages

An autoregression is an infinite moving average with geometrically constrained weights. The moving average takes the opposite tack, a finite sum of shocks.

::: definition [def

] The MA($q$) model is $r_t = c_0 + \theta(B)a_t$ with $\theta(B) = 1 - \theta_1 B - \cdots - \theta_q B^q$, backshift $B$, and $\{a_t\}$ white noise of variance $\sigma^2$. It is a finite linear combination of white noise, so it is weakly stationary for every parameter value. :::

::: proposition [prop

] The MA($q$) model has mean $c_0$, variance $(1 + \theta_1^2 + \cdots + \theta_q^2)\sigma^2$, and autocorrelations that vanish beyond lag $q$, so its autocorrelation function cuts off at $q$. :::

::: proof Taking expectations gives $\E[r_t] = c_0$ since $\E[a_{t-i}] = 0$. Centre and use that the $a_{t-i}$ are uncorrelated with common variance $\sigma^2$, so $\Var(r_t) = \sigma^2\sum_{i=0}^q \theta_i^2$ with $\theta_0 = 1$. For $\ell > q$ the windows $\{a_t,\dots,a_{t-q}\}$ and $\{a_{t-\ell},\dots,a_{t-\ell-q}\}$ are disjoint, so every cross term $\E[a_s a_u]$ with $s\ne u$ vanishes and $\gamma_\ell = 0$. For $1 \le \ell \le q$ the overlap leaves a finite combination of the $\theta_j$, and dividing by $\gamma_0$ gives an autocorrelation that cuts off at $q$. :::

The cutoff is the mirror of the autoregression, whose partial autocorrelation cuts off while its autocorrelation decays. The moving average raises a dual question, whether the shocks can be recovered from the observed series.

::: lemma [lem

] The MA($q$) model $r_t = \theta(B)a_t$ admits the autoregressive form $\pi(B)r_t = a_t$ with summable weights $\pi$ if and only if every root of $\theta(z) = 0$ lies outside the unit circle. That condition makes the shock $a_t$ recoverable from the present and past of $r_t$, and the model is then called invertible. :::

::: proof Formally $\pi(B) = \theta(B)^{-1} = \prod_i (1 - \eta_i B)^{-1}$ where the $\eta_i$ are the reciprocals of the roots of $\theta$. Each factor expands as $\sum_{k\ge0}\eta_i^k B^k$, which converges with summable coefficients exactly when $|\eta_i| < 1$, that is when every root $1/\eta_i$ lies outside the unit circle. The resulting $\pi(B)r_t = a_t$ then writes $a_t$ as a convergent combination of current and past $r_t$ [1]. :::

#ARMA models

Combining the two mechanisms economises parameters. Where a pure AR or MA might need a high order, their ratio can be short.

::: definition [def

] The ARMA($p,q$) model is $\phi(B)r_t = \phi_0 + \theta(B)a_t$ with $\phi(B) = 1 - \phi_1 B - \cdots - \phi_p B^p$ and $\theta(B) = 1 - \theta_1 B - \cdots - \theta_q B^q$ sharing no common factor. It is weakly stationary if and only if every root of $\phi(z) = 0$ lies outside the unit circle, with mean $\mu = \phi_0/\phi(1)$. :::

The mixed model's autocorrelation behaves like an autoregression after a short transient set by the moving-average part.

::: proposition [prop

] The stationary ARMA($1,1$) model $r_t = \phi_1 r_{t-1} + \phi_0 + a_t - \theta_1 a_{t-1}$ with $|\phi_1| < 1$ and $\phi_1 \ne \theta_1$ has

Its autocorrelation therefore decays geometrically at rate $\phi_1$ but only from lag 2 onward, so neither the autocorrelation nor the partial autocorrelation cuts off at any finite lag. :::

::: proof Since $r_{t-1}$ and $a_t$ are uncorrelated, $\E[r_t a_t] = \E[a_t^2] = \sigma^2$ and $\E[r_t a_{t-1}] = \phi_1\sigma^2 - \theta_1\sigma^2$. Taking the variance of the model, $\gamma_0 = \phi_1^2\gamma_0 + (1+\theta_1^2)\sigma^2 + 2\phi_1\Cov(r_{t-1}, a_t - \theta_1 a_{t-1}) = \phi_1^2\gamma_0 + (1+\theta_1^2)\sigma^2 - 2\phi_1\theta_1\sigma^2$, using $\Cov(r_{t-1}, a_{t-1}) = \sigma^2$. Rearranging gives $\gamma_0$. Multiplying the model by $r_{t-1}$ and taking expectations gives $\gamma_1 = \phi_1\gamma_0 - \theta_1\sigma^2$, the shock term surviving because $\E[a_{t-1}r_{t-1}] = \sigma^2$. Multiplying by $r_{t-\ell}$ for $\ell \ge 2$ annihilates both shock terms and leaves $\gamma_\ell = \phi_1\gamma_{\ell-1}$, which is Equation (1). Because both the moving-average and the autoregressive weights are infinite once no common factor cancels, neither function truncates. :::

Since neither function cuts off, order identification cannot read $p$ and $q$ off a truncation as it does for the pure models. The extended autocorrelation function locates the corner of the $(p,q)$ order in a two-way table by first estimating the autoregressive part consistently and reading the moving-average order off the derived residuals [3].

#Three representations

A stationary ARMA model has three equivalent forms, each serving a purpose, obtained by long division of the two polynomials.

::: proposition [prop

] Write $\psi(B) = \theta(B)/\phi(B)$ and $\pi(B) = \phi(B)/\theta(B)$. The ARMA($p,q$) model has the moving-average representation $r_t = \mu + \psi(B)a_t$, whose weights are the impulse responses and solve

with $\theta_j = 0$ for $j > q$, and the autoregressive representation $\pi(B)r_t = \phi_0/\theta(1) + a_t$ valid when the roots of $\theta$ lie outside the unit circle. Both weight sequences decay geometrically, so a remote shock and a remote observation each have vanishing influence. :::

::: proof From $\phi(B)\psi(B) = \theta(B)$, matching the coefficient of $B^j$ gives $\psi_j - \sum_{i=1}^p\phi_i\psi_{j-i} = -\theta_j$ with $\psi_0 = 1$, which is Equation (2). The homogeneous part $\psi_j = \sum_i\phi_i\psi_{j-i}$ for $j > q$ is the autoregressive recursion, so the $\psi_j$ inherit the geometric decay of the reciprocal roots of $\phi$ under stationarity. Dividing $\phi(B)$ by $\theta(B)$ instead yields $\pi(B)$ symmetrically, with geometric decay of the $\pi_j$ under invertibility, and $Bc = c$ for a constant gives the intercept $\phi_0/\theta(1)$. For the ARMA($1,1$) case the recursion solves in closed form, $\psi_j = \phi_1^{\,j-1}(\phi_1 - \theta_1)$ for $j\ge1$. :::

The moving-average representation gives the forecast error directly. The $\ell$-step error is $\sum_{j=0}^{\ell-1}\psi_j a_{h+\ell-j}$, so its variance is $\sigma^2\sum_{j=0}^{\ell-1}\psi_j^2$, rising to $\gamma_0 = \sigma^2\sum_{j\ge0}\psi_j^2$, exactly the mean-reverting forecast the autoregression post derived. The two rational forms are the finite-parameter handle on a universal object.

::: remark [rem

] Every zero-mean weakly stationary purely nondeterministic series admits the Wold representation $r_t = \sum_{j\ge0}\psi_j a_{t-j}$ with $\sum_j\psi_j^2 < \infty$, proved in the autoregression post. The ARMA model is the rational case $\psi(B) = \theta(B)/\phi(B)$, a finite-parameter approximation to this infinite-order representation [2]. :::

#Seasonality

A series sampled through a repeating calendar carries dependence at the seasonal lag alongside the short-run lag. The multiplicative model handles both with two moving-average factors.

::: definition [def

] For a series of period $s$, the airline model is

a regular and a seasonal difference on the left and a regular times a seasonal moving average on the right. :::

::: proposition [prop

] The differenced series $w_t = (1 - \theta B)(1 - \Theta B^s)a_t$ has autocorrelation nonzero only at lags 1, $s-1$, $s$, and $s+1$, with

so the regular and seasonal dynamics enter approximately orthogonally. :::

::: proof Expand $w_t = a_t - \theta a_{t-1} - \Theta a_{t-s} + \theta\Theta a_{t-s-1}$. Since the $a_t$ are uncorrelated with variance $\sigma^2$, direct computation gives $\gamma_0 = (1+\theta^2)(1+\Theta^2)\sigma^2$, $\gamma_1 = -\theta(1+\Theta^2)\sigma^2$, $\gamma_s = -\Theta(1+\theta^2)\sigma^2$, and $\gamma_{s-1} = \gamma_{s+1} = \theta\Theta\sigma^2$, with every other autocovariance zero because the shifted index windows share no innovation. Dividing by $\gamma_0$ gives Equation (4), and the product form $\rho_{s-1} = \rho_1\rho_s$ is the signature of the multiplicative structure. Deterministic seasonality handled by dummy variables is the boundary case $\Theta = 1$ [1]. :::

#Long memory

Under stationarity the autocorrelation decays geometrically; under a unit root it does not decay at all. Between the two lies a process whose autocorrelation decays only at a polynomial rate.

::: definition [def

] The fractionally integrated process is $(1 - B)^d x_t = a_t$ with $-\tfrac12 < d < \tfrac12$, where the fractional difference expands by the binomial series $(1 - B)^d = \sum_{k\ge0}\binom{d}{k}(-B)^k$. If $(1-B)^d x_t$ follows an ARMA($p,q$) model, $x_t$ is an ARFIMA($p,d,q$) process. :::

::: proposition [prop

] For $0 < d < \tfrac12$ the process is weakly stationary with moving-average weights $\psi_k = \Gamma(k+d)/(\Gamma(k+1)\Gamma(d)) \sim k^{d-1}/\Gamma(d)$ and autocorrelations $\rho_k \sim c\,k^{2d-1}$, a hyperbolic decay slower than any geometric rate, while its spectral density diverges at the origin, $f(\omega) \sim \omega^{-2d}$ as $\omega \to 0$. :::

::: proof Expanding $(1-B)^{-d} = \sum_k\binom{-d}{k}(-B)^k$ gives the stated $\psi_k = \Gamma(k+d)/(\Gamma(k+1)\Gamma(d))$, and Stirling's ratio $\Gamma(k+d)/\Gamma(k+1) \sim k^{d-1}$ yields $\psi_k \sim k^{d-1}/\Gamma(d)$. These are square-summable exactly when $2(d-1) < -1$, that is $d < \tfrac12$, so the representation converges and the process is stationary. The autocovariance $\gamma_k = \sigma^2\sum_j\psi_j\psi_{j+k}$ is a tail sum of $j^{d-1}(j+k)^{d-1}$, of order $k^{2d-1}$ by comparison with the integral, giving the power law. For the spectrum, the transfer function of the pure fractional filter gives $f(\omega) = (\sigma^2/2\pi)|1 - e^{-i\omega}|^{-2d}$, the frequency-domain complement of the AR spectrum in the autoregression post, and $|1 - e^{-i\omega}|^2 = 4\sin^2(\omega/2) \sim \omega^2$ as $\omega \to 0$ gives $f(\omega) \sim (\sigma^2/2\pi)\,\omega^{-2d} \to \infty$ [4]. :::

The hyperbolic decay is not a curiosity. The sample autocorrelation of absolute or squared asset returns is small but positive over hundreds of lags, the empirical fingerprint of long memory in volatility [5].

#Inference under serial correlation

A regression whose errors carry serial correlation keeps a consistent coefficient but loses the textbook standard error. Consider $y_t = x_t^{\top}\beta + e_t$ with $e_t$ serially correlated. Ordinary least squares stays unbiased, yet its conventional covariance $\sigma^2(\sum x_t x_t^{\top})^{-1}$ is wrong, and the fix is a covariance robust to both heteroskedasticity and autocorrelation.

::: proposition [prop

] For ordinary least squares in $y_t = x_t^{\top}\beta + e_t$, $\sqrt{T}(\hat\beta - \beta) \to \mathcal N(0, Q^{-1} S Q^{-1})$ with $Q = \E[x_t x_t^{\top}]$ and $S$ the long-run variance of $x_t e_t$. The Newey-West estimator

with Bartlett weights $w_\ell$ and $\hat\Gamma_\ell = T^{-1}\sum_t x_t\hat e_t\hat e_{t-\ell}x_{t-\ell}^{\top}$ is consistent and positive semidefinite as $L \to \infty$ with $L/T \to 0$, while the conventional $\sigma^2 Q^{-1}$ understates the variance when the $e_t$ are positively autocorrelated. :::

::: proof Write $\sqrt{T}(\hat\beta - \beta) = (T^{-1}\sum x_t x_t^{\top})^{-1}\,T^{-1/2}\sum x_t e_t$. The design average converges to $Q$ by the law of large numbers, and the score average $T^{-1/2}\sum x_t e_t$ obeys a central limit theorem for the dependent sequence $x_t e_t$, whose limiting variance is the long-run variance $S = \sum_{\ell=-\infty}^{\infty}\Gamma_\ell$ with $\Gamma_{-\ell} = \Gamma_\ell^{\top}$, giving the sandwich $Q^{-1} S Q^{-1}$. The Bartlett taper $w_\ell = 1 - \ell/(L+1)$ makes $\hat S$ a weighted sum of sample autocovariances that is positive semidefinite by construction and consistent under $L/T \to 0$ [6]. Setting all $w_\ell = 0$ recovers the White heteroskedasticity-consistent estimator, valid when the $e_t$ are uncorrelated but heteroskedastic [7], [8]. :::

Two routes then restore valid inference. Model $e_t$ as an ARMA and estimate the mean and error parameters jointly by generalised least squares, the Cochrane-Orcutt approach, efficient under a correct error model, or keep ordinary least squares and report Equation (5), agnostic to it [9]. Residual serial correlation is checked by the Ljung-Box portmanteau statistic rather than the Durbin-Watson $d = \sum_t(\Delta\hat e_t)^2/\sum_t\hat e_t^2 \approx 2(1 - \hat\rho_1)$, which reads only the lag-one autocorrelation [10], [11].

The moving average, the ARMA model, and their long-memory and seasonal extensions are the linear vocabulary of serial dependence. The autocorrelation and partial autocorrelation identify the pure models by their dual truncation, the extended autocorrelation identifies the mixed one, and long division exposes the impulse responses that forecasting and the Wold representation both rest on. Where the geometric rate gives way to a unit root, the same polynomials open onto the unit-root and cointegration analysis, and where the conditional mean is adequate but the conditional variance is not, the squared residuals carry the autocorrelation that the level lacks [12], [13].