Search results
Results from the WOW.Com Content Network
The entropy () thus sets a minimum value for the cross-entropy (,), the expected number of bits required when using a code based on Q rather than P; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value x drawn from X, if a code is used corresponding to the ...
The only divergence for probabilities over a finite alphabet that is both an f-divergence and a Bregman divergence is the Kullback–Leibler divergence. [8] The squared Euclidean divergence is a Bregman divergence (corresponding to the function x 2 {\displaystyle x^{2}} ) but not an f -divergence.
When NMF is obtained by minimizing the Kullback–Leibler divergence, it is in fact equivalent to another instance of multinomial PCA, probabilistic latent semantic analysis, [45] trained by maximum likelihood estimation. That method is commonly used for analyzing and clustering textual data and is also related to the latent class model.
However, in the context of decision trees, the term is sometimes used synonymously with mutual information, which is the conditional expected value of the Kullback–Leibler divergence of the univariate probability distribution of one variable from the conditional distribution of this variable given the other one.
The total correlation is also the Kullback–Leibler divergence between the actual distribution (,, …,) and its maximum entropy product approximation () (). Total correlation quantifies the amount of dependence among a group of variables.
Note that the expression of Pinsker inequality depends on what basis of logarithm is used in the definition of KL-divergence. D K L {\displaystyle D_{KL}} is defined using ln {\displaystyle \ln } (logarithm in base e {\displaystyle e} ), whereas D {\displaystyle D} is typically defined with log 2 {\displaystyle \log _{2}} (logarithm in base 2).
We can define the Rényi divergence for the special values α = 0, 1, ∞ by taking a limit, and in particular the limit α → 1 gives the Kullback–Leibler divergence. Some special cases: (‖) = ({: >}) : minus the log probability under Q that p i > 0;
When comparing a distribution against a fixed reference distribution , cross-entropy and KL divergence are identical up to an additive constant (since is fixed): According to the Gibbs' inequality, both take on their minimal values when =, which is for KL divergence, and () for cross-entropy.