I recently learnt of the “noise outsourcing” theorem from Teh Yee Whye‘s IMS Medallion Lecture at the recent Joint Statistical Meetings (JSM). Here is the statement of the theorem from the talk:
Theorem (Noise Outsourcing). If and are random variables in “nice” (e.g. Borel) spaces and , then there is a random variable which is independent of and a function such that
In particular, if there is a statistic with , then
The arXiv paper tells us how we can interpret this theorem: for any two random variables in “nice” spaces, there is a generative functional representation of the conditional distribution in terms of and independent noise: . The random variable acts as a generic source of randomness that is “outsourced”.
Apparently noise outsourcing is a “standard technical tool from measure theoretic probability”, and it appears in Kallenberg’s Foundations of Modern Probability, a commonly used graduate text in probability.
- Teh, Y. W. On Statistical Thinking in Deep Learning (Slide 34).
- Bloem-Reddy, B., and Teh, Y. W. (2019). Probabilistic symmetry and invariant neural networks (Section 3.1).