# Noise outsourcing theorem

I recently learnt of the “noise outsourcing” theorem from Teh Yee Whye‘s IMS Medallion Lecture at the recent Joint Statistical Meetings (JSM). Here is the statement of the theorem from the talk:

Theorem (Noise Outsourcing). If $X$ and $Y$ are random variables in “nice” (e.g. Borel) spaces $\mathcal{X}$ and $\mathcal{Y}$, then there is a random variable $\eta \sim U[0,1]$ which is independent of $X$ and a function $h : [0, 1] \times \mathcal{X} \mapsto \mathcal{Y}$ such that

\begin{aligned} (X, Y) \stackrel{a.s.}{=} (X, h(\eta, X)). \end{aligned}

In particular, if there is a statistic $S(X)$ with $X \perp\!\!\!\perp Y \mid S(X)$, then

\begin{aligned} (X, Y) \stackrel{a.s.}{=} (X, h(\eta, S(X))). \end{aligned}

The arXiv paper tells us how we can interpret this theorem: for any two random variables in “nice” spaces, there is a generative functional representation of the conditional distribution $P_{Y \mid X}$ in terms of $X$ and independent noise: $Y \stackrel{a.s.}{=} f(\eta, X)$. The random variable $\eta$ acts as a generic source of randomness that is “outsourced”.

Apparently noise outsourcing is a “standard technical tool from measure theoretic probability”, and it appears in Kallenberg’s Foundations of Modern Probability, a commonly used graduate text in probability.

References:

1. Teh, Y. W. On Statistical Thinking in Deep Learning (Slide 34).
2. Bloem-Reddy, B., and Teh, Y. W. (2019). Probabilistic symmetry and invariant neural networks (Section 3.1).