I recently learnt of the “noise outsourcing” theorem from Teh Yee Whye‘s IMS Medallion Lecture at the recent Joint Statistical Meetings (JSM). Here is the statement of the theorem from the talk:
Theorem (Noise Outsourcing). If
and
are random variables in “nice” (e.g. Borel) spaces
and
, then there is a random variable
which is independent of
and a function
such that
In particular, if there is a statistic
with
, then
The arXiv paper tells us how we can interpret this theorem: for any two random variables in “nice” spaces, there is a generative functional representation of the conditional distribution in terms of
and independent noise:
. The random variable
acts as a generic source of randomness that is “outsourced”.
Apparently noise outsourcing is a “standard technical tool from measure theoretic probability”, and it appears in Kallenberg’s Foundations of Modern Probability, a commonly used graduate text in probability.
References:
- Teh, Y. W. On Statistical Thinking in Deep Learning (Slide 34).
- Bloem-Reddy, B., and Teh, Y. W. (2019). Probabilistic symmetry and invariant neural networks (Section 3.1).