# Wolfram Function Repository

Instant-use add-on functions for the Wolfram Language

Function Repository Resource:

Compute the mutual information of data samples or a multivariate distribution

Contributed by:
Luigi Brancati

ResourceFunction["MutualInformation"][ estimates the mutual information of two lists of elements. | |

ResourceFunction["MutualInformation"][ estimates the auto-mutual information of | |

ResourceFunction["MutualInformation"][ estimates the auto-mutual information, comparing running batches of length | |

ResourceFunction["MutualInformation"][ computes the mutual information of a bivariate distribution | |

ResourceFunction["MutualInformation"][ computes the mutual information of the bivariate distribution CopulaDistribution["Product",{ |

The function ResourceFunction["MutualInformation"][*dist*] expects a bivariate or multivariate distribution.

The estimator ResourceFunction["MutualInformation"][*list*,*t*] computes the mutual information between *list*[*i*] and *list*[*i*+*s*], with *s*=1,2,…,*t*.

The function ResourceFunction["MutualInformation"][*list*,___] uses parallel kernels by default. To force sequential execution, use the option value "Parallelize"→False.

ResourceFunction["MutualInformation"][*list*,{*t*,0}] is equivalent to ResourceFunction["MutualInformation"][*list*,*t*].

The functions ResourceFunction["MutualInformation"][*dist*] and ResourceFunction["MutualInformation"][*dist*_{x},*dist*_{y}] both rely on the KullbackLeiblerDivergence resource function to work, hence both accept the same options as KullbackLeiblerDivergence.

In addition to the options for KullbackLeiblerDivergence, the function ResourceFunction["MutualInformation"], when used on distributions, accepts the following options:

"CopulaKernel" | "Product" | uses the specified copula kernel to compute the joint distribution from the provided marginals |

"Marginals" | Automatic | specifies how to extract marginals from the joint distribution |

The option setting "Marginals"→Automatic splits the joint space in half.

Any kernel supported by CopulaDistribution can be provided, though some copulas will be hard to compute exactly. Use the option Method→NExpectation to compute a numerical approximation, which is much faster most of the time.

When used on lists of samples, the function ResourceFunction["MutualInformation"] accepts the following options:

DistanceFunction | ChessboardDistance | distance used to compute the estimator |

"Errors" | False | whether or not to compute the error on the estimator |

"KNeighbour" | 4 | kth-nearest neighbor parameter to be used for the estimator on lists of samples |

"Parallelize" | True | whether or not to use parallel computation |

ResourceFunction["MutualInformation"][*list*,___] uses an estimator that relies on a *k*^{th}-nearest neighbors algorithm (see Autor Notes). The default value of *k* is 4; to set a different value, use the option "KNeighbour".

The estimator performs poorly on very skewed distributions. In such cases, one should renormalize the distribution with Standardize to get better results.

Compute the mutual information of a binormal distribution:

In[1]:= |

Out[1]= |

Compare the result to its real value:

In[2]:= |

Out[2]= |

The same information can be computed using a symbolic distribution, too:

In[3]:= |

Out[3]= |

Compute the mutual information of a multivariate distribution (note: N is used to speed up the computation; see "Author Notes"):

In[4]:= |

Out[4]= |

Compute the mutual information of a bivariate distribution, starting from the two marginal distributions:

In[5]:= |

Out[5]= |

Estimate the mutual information between two samples, with errors:

In[6]:= |

In[7]:= |

Out[7]= |

Compute the mutual information between two sets of data, using any distance function:

In[8]:= |

Out[8]= |

In[9]:= |

Out[9]= |

If no DistanceFunction is specified, the default is ChessboardDistance:

In[10]:= |

In[11]:= |

Out[11]= |

In[12]:= |

Out[12]= |

In[13]:= |

Out[13]= |

Any kind of element in a List can be fed to the estimator:

In[14]:= |

In[15]:= |

Out[15]= |

Say whether you want to compute errors on the estimator (not available on distributions):

In[16]:= |

Out[16]= |

Note: the examples of this section may take several minutes to run. This is due to a bug that prevents one from using option Method→NExpectation, which should be preferred for multivariate distributions. See the Author Notes at the end of the notebook for more information.

Specify how to extract marginals from a multivariate distribution:

In[17]:= |

In[18]:= |

Out[18]= |

Notice that different ways to extract marginals are not equivalent:

In[19]:= |

Out[19]= |

The option "CopulaKernel" is only available for MutualInformation[*dist*_{x},*dist*_{y}]. It uses CopulaDistribution["CopulaKernel",{*dist*_{x},*dist*_{y}}] as the joint distribution:

In[20]:= |

Out[20]= |

Compute the mutual information shared between the initial and subsequent frames of a video:

In[21]:= |

In[22]:= |

In[23]:= |

Out[23]= |

- 2.0.0 – 12 August 2020
- 1.0.0 – 10 August 2020

This work is licensed under a Creative Commons Attribution 4.0 International License