The relationship between MA(q)/AR(p) processes and ACF/PACF

The relationship between the MA(q) process and ACF

Assume that $Z_0, Z_1, \dots$ are i.i.d. random variables with mean 0 and variance $\sigma^2$. The moving average process of order $q$ (or the MA(q) process) has the form \begin{aligned} X_t = \theta_0 Z_t + \theta_1 Z_{t-1} + \dots + \beta_q Z_{t-q}. \end{aligned}

Note that the RHS actually has $q + 1$ (not $q$) terms in total. (Also in many references, $\theta_0$ is constrained to be equal to 1; here I have left it unconstrained.)

For a weakly stationary process $X_t$, its autocovariance function is defined as $\gamma (k) = \text{Cov}(X_t, X_{t-k}) = \mathbb{E}[X_t X_{t-k}] \quad \text{for } k \in \mathbb{Z}.$

(The second equality is true if $\mathbb{E}[X_t] = 0$ for all $t$, which is usually the case in these settings.) Its autocorrelation function (ACF) is defined as $\rho (k) = \text{Cor}(X_t, X_{t-k}) = \dfrac{\text{Cov}(X_t, X_{t-k})}{[\text{Var}(X_t) \text{Var}(X_{t-k})]^{1/2}} = \dfrac{\gamma (k)}{\gamma (0)} \quad \text{for } k \in \mathbb{Z}.$

For an MA(q) process, the ACF cuts off after lag q. That is, $\rho (k) = \gamma(k) = 0 \quad \text{for } k \geq q + 1.$

This fact can be verified by directly plugging the definition of the MA(q) process into the definition for the autocovariance function. Here is a little simulation demonstrating this fact for an MA(3) process:

set.seed(1)
x <- arima.sim(list(ma = c(1, 0.5, 1)), n = 2000)
acf(x, main = "Autocorrelation function for x") The ACF cuts off after lag 3. Note that there are 4 (not 3) significantly non-zero lines: one for lag 0 (we always have $\rho(0) = 1$) and 3 for the 3 positive lags. For an MA(q) process, we expect to see $q+1$ “peaks” in the ACF plot.

This can be used as a method to determine the order of a moving average process: estimate the ACF and the largest lag with significantly non-zero autocorrelation is the order of the process. (Note, however, that If you try different random seeds or different coefficients, the break between the non-zero and zero autocorrelations is not always obvious.)

The relationship between the AR(p) process and PACF

The autoregressive process of order $p$ (or the AR(p) process) has the form \begin{aligned} X_t = Z_t + \phi_1 X_{t-1} + \dots + \phi_p X_{t-p}. \end{aligned}

Here there are $p + 1$ terms on the RHS, with one of them belonging to the $Z$ process and $p$ of them belonging to the $X$ process. The coefficient of $Z_t$ is constrained to be 1.

For a weakly stationary process $X_t$, its partial autocorrelation function is defined by \begin{aligned} \rho (X_t, X_{t-k}) = \dfrac{\text{Cov}(X_t, X_{t-k} \mid X_{t-1}, X_{t-2}, \dots, X_{t - (k-1)})}{ [\text{Var}(X_t \mid X_{t-1}, X_{t-2}, \dots, X_{t - (k-1)}) \text{Var}(X_{t-k} \mid X_{t-1}, X_{t-2}, \dots, X_{t - (k-1)})]^{1/2} } \quad \text{for } k \geq 1. \end{aligned}

This is basically the same formula as that for the ACF except we condition on $X_{t-1}, \dots, X_{t-(k-1)}$.

For an AR(p) process, the PACF cuts off after lag p. That is, $\rho (X_t, X_{t-k}) = 0 \quad \text{for } k \geq p + 1.$

Here is a simulation illustrating this for an AR(3) process:

set.seed(1)
x <- arima.sim(list(ar = c(1, -0.6, 0.4)), n = 2000)
acf(x, type = "partial", main = "Partial autocorrelation function for x") Note that the PACF plot starts at 1, not 0 like the ACF plot. As expected, the PACF is essentially zero for lags above 3. For an AR(p) process, we expect to see $p$ “peaks” in the PACF plot.

This can be used as a method to determine the order of an autoregressive process: estimate the PACF and the largest lag with significantly non-zero partial autocorrelation is the order of the process.