Noise outsourcing theorem

I recently learnt of the “noise outsourcing” theorem from Teh Yee Whye‘s IMS Medallion Lecture at the recent Joint Statistical Meetings (JSM). Here is the statement of the theorem from the talk:

Theorem (Noise Outsourcing). If X and Y are random variables in “nice” (e.g. Borel) spaces \mathcal{X} and \mathcal{Y}, then there is a random variable \eta \sim U[0,1] which is independent of X and a function h : [0, 1] \times \mathcal{X} \mapsto \mathcal{Y} such that

\begin{aligned} (X, Y) \stackrel{a.s.}{=} (X, h(\eta, X)). \end{aligned}

In particular, if there is a statistic S(X) with X \perp\!\!\!\perp Y \mid S(X), then

\begin{aligned} (X, Y) \stackrel{a.s.}{=} (X, h(\eta, S(X))). \end{aligned}

The arXiv paper tells us how we can interpret this theorem: for any two random variables in “nice” spaces, there is a generative functional representation of the conditional distribution P_{Y \mid X} in terms of X and independent noise: Y \stackrel{a.s.}{=} f(\eta, X). The random variable \eta acts as a generic source of randomness that is “outsourced”.

Apparently noise outsourcing is a “standard technical tool from measure theoretic probability”, and it appears in Kallenberg’s Foundations of Modern Probability, a commonly used graduate text in probability.

References:

  1. Teh, Y. W. On Statistical Thinking in Deep Learning (Slide 34).
  2. Bloem-Reddy, B., and Teh, Y. W. (2019). Probabilistic symmetry and invariant neural networks (Section 3.1).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s