Update (2022-07-04): This is now tracked as an issue on Github.
I recently noticed a slight inconsistency between the
fct_lump_prop functions. (I’m working with v0.5.1, which is the latest version at the time of writing.) These functions lump levels that meet a certain criteria into an “other” level. According to the documentation,
fct_lump_min“lumps levels that appear fewer than
fct_lump_prop“lumps levels that appear fewer than
prop * ntimes”, where
nis the length of the factor variable.
Let’s try this out in an example:
x <- factor(c(rep(1, 6), rep(2, 3), rep(3, 1))) x #  1 1 1 1 1 1 2 2 2 3 # Levels: 1 2 3 fct_lump_min(x, min = 3) #  1 1 1 1 1 1 2 2 2 Other # Levels: 1 2 Other
The levels 1 and 2 appear at least 3 times, and so they are not converted to the “Other” level. The level 3 appears just once, and so is converted.
What do you think this line of code returns?
fct_lump_prop(x, prop = 0.3)
prop * n = 0.3 * 10 = 3, the documentation suggests that only the level 3 should be converted to “Other”. However, that is NOT the case:
fct_lump_prop(x, prop = 0.3) #  1 1 1 1 1 1 Other Other Other Other # Levels: 1 Other
The level 2 appears exactly 3 times, and is converted into “Other”, contrary to what the documentation says.
Digging, into the source code of
fct_lump_prop, you will find this line of code:
new_levels <- ifelse(prop_n > prop, levels(f), other_level)
prop_n is the proportion of times each factor level appears. If the documentation is correct, the greater than sign should really be a greater than or equal sign.
So what’s the right fix here? One way is to fix the documentation to say that
fct_lump_prop “lumps levels that appear at most
prop * n times”, but that breaks consistency with
fct_lump_min. Another is to make the fix in the code as suggested above, but that will change code behavior. Either is probably fine, and it’s up to the package owner to decide which makes more sense.
Have you opened a github issue so that the package owner(s) are aware and can eventually fix? https://github.com/tidyverse/forcats/issues
Just opened, thanks for the reminder! https://github.com/tidyverse/forcats/issues/312