The relationship between MA(q)/AR(p) processes and ACF/PACF

The relationship between the MA(q) process and ACF

Assume that Z_0, Z_1, \dots are i.i.d. random variables with mean 0 and variance \sigma^2. The moving average process of order q (or the MA(q) process) has the form

\begin{aligned} X_t = \theta_0 Z_t + \theta_1 Z_{t-1} + \dots + \beta_q Z_{t-q}. \end{aligned}

Note that the RHS actually has q + 1 (not q) terms in total. (Also in many references, \theta_0 is constrained to be equal to 1; here I have left it unconstrained.)

For a weakly stationary process X_t, its autocovariance function is defined as

\gamma (k) = \text{Cov}(X_t, X_{t-k}) = \mathbb{E}[X_t X_{t-k}] \quad \text{for } k \in \mathbb{Z}.

(The second equality is true if \mathbb{E}[X_t] = 0 for all t, which is usually the case in these settings.) Its autocorrelation function (ACF) is defined as

\rho (k) = \text{Cor}(X_t, X_{t-k}) = \dfrac{\text{Cov}(X_t, X_{t-k})}{[\text{Var}(X_t) \text{Var}(X_{t-k})]^{1/2}} = \dfrac{\gamma (k)}{\gamma (0)} \quad \text{for } k \in \mathbb{Z}.

For an MA(q) process, the ACF cuts off after lag q. That is,

\rho (k) = \gamma(k) = 0 \quad \text{for } k \geq q + 1.

This fact can be verified by directly plugging the definition of the MA(q) process into the definition for the autocovariance function. Here is a little simulation demonstrating this fact for an MA(3) process:

set.seed(1)
x <- arima.sim(list(ma = c(1, 0.5, 1)), n = 2000)
acf(x, main = "Autocorrelation function for x")

The ACF cuts off after lag 3. Note that there are 4 (not 3) significantly non-zero lines: one for lag 0 (we always have \rho(0) = 1) and 3 for the 3 positive lags. For an MA(q) process, we expect to see q+1 “peaks” in the ACF plot.

This can be used as a method to determine the order of a moving average process: estimate the ACF and the largest lag with significantly non-zero autocorrelation is the order of the process. (Note, however, that If you try different random seeds or different coefficients, the break between the non-zero and zero autocorrelations is not always obvious.)

The relationship between the AR(p) process and PACF

The autoregressive process of order p (or the AR(p) process) has the form

\begin{aligned} X_t = Z_t + \phi_1 X_{t-1} + \dots + \phi_p X_{t-p}. \end{aligned}

Here there are p + 1 terms on the RHS, with one of them belonging to the Z process and p of them belonging to the X process. The coefficient of Z_t is constrained to be 1.

For a weakly stationary process X_t, its partial autocorrelation function is defined by

\begin{aligned} \rho (X_t, X_{t-k}) = \dfrac{\text{Cov}(X_t, X_{t-k} \mid X_{t-1}, X_{t-2}, \dots, X_{t - (k-1)})}{ [\text{Var}(X_t \mid X_{t-1}, X_{t-2}, \dots, X_{t - (k-1)}) \text{Var}(X_{t-k} \mid X_{t-1}, X_{t-2}, \dots, X_{t - (k-1)})]^{1/2} } \quad \text{for } k \geq 1. \end{aligned}

This is basically the same formula as that for the ACF except we condition on X_{t-1}, \dots, X_{t-(k-1)}.

For an AR(p) process, the PACF cuts off after lag p. That is,

\rho (X_t, X_{t-k}) = 0 \quad \text{for } k \geq p + 1.

Here is a simulation illustrating this for an AR(3) process:

set.seed(1)
x <- arima.sim(list(ar = c(1, -0.6, 0.4)), n = 2000)
acf(x, type = "partial", main = "Partial autocorrelation function for x")

Note that the PACF plot starts at 1, not 0 like the ACF plot. As expected, the PACF is essentially zero for lags above 3. For an AR(p) process, we expect to see p “peaks” in the PACF plot.

This can be used as a method to determine the order of an autoregressive process: estimate the PACF and the largest lag with significantly non-zero partial autocorrelation is the order of the process.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s