Subsetting in the presence of NAs

In R, we can subset a data frame df easily by putting the conditional in square brackets after df. For example, if I want all the rows in df which have value equal to 1 in the column colA, all I have to do is

df[df$colA == 1, ]

Recently, I realized that this approach can be problematic when there are NAs present in the data! For example, let df be the following data frame:

df <- data.frame(colA = c(1, 2, 3, NA, NA),
                 colB = c("a", "b", "c", "d", NA))
df

# co1A colB
# 1    1    a
# 2    2    b
# 3    3    c
# 4   NA    d
# 5   NA <NA>

If I want to pull out all rows such that the value in colA is equal to 2, I would use the following code expecting just one row to be returned:

df[df$colA == 2, ]

#      colA colB
# 2       2    b
# NA     NA <NA>
# NA.1   NA <NA>

All the rows with NA in colA also got included, and all the values in those rows got converted into NAs! To avoid these rows from being added, we have to explicitly check that the values are not NAs:

df[!is.na(df$colA) & df$colA == 2, ]

# colA colB
# 2    2    b

dplyr acts a bit differently in that it removes NAs by default and so you do not have to check for them:

library(dplyr)
df %>% filter(colA == 2)

# colA colB
# 2    2    b

Edit: As the comments have helpfully pointed out, we can use the which function to avoid this issue as well:

df[which(df$colA == 2), ]

# colA colB
# 2    2    b

4 thoughts on “Subsetting in the presence of NAs

  1. You can also use which() to pull out just the indices of TRUE elements. But yes, filter() is more elegant and has other nice functionalities.

    df[ which(df$colA==2), ]

    colA colB
    2 2 b

    Like

  2. Including the rows with NA makes sense given that NA means “the value is not available, but it could be what you’re looking for (i.e. 2).”

    The trick I use is to call which(), e.g.:
    df[which(df$colA == 2), ]

    Of course, this shouldn’t be done blindly. If there are NAs in the column then the analyst should put some thought into why they are there and how they should be treated.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s