Search results
Results from the WOW.Com Content Network
In mathematical statistics, the Kullback–Leibler (KL) divergence (also called relative entropy and I-divergence [1]), denoted (), is a type of statistical distance: a measure of how much a model probability distribution Q is different from a true probability distribution P.
For the classical Kullback–Leibler divergence, it can be shown that (‖) = ,and the equality holds if and only if P = Q.Colloquially, this means that the uncertainty calculated using erroneous assumptions is always greater than the real amount of uncertainty.
where is the Kullback–Leibler divergence, and is the outer product distribution which assigns probability () to each (,).. Notice, as per property of the Kullback–Leibler divergence, that (;) is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when and are independent (and hence observing tells you nothing about ).
The Kullback–Leibler divergence (or information divergence, information gain, or relative entropy) is a way of comparing two distributions: a "true" probability distribution , and an arbitrary probability distribution .
Note that the expression of Pinsker inequality depends on what basis of logarithm is used in the definition of KL-divergence. D K L {\displaystyle D_{KL}} is defined using ln {\displaystyle \ln } (logarithm in base e {\displaystyle e} ), whereas D {\displaystyle D} is typically defined with log 2 {\displaystyle \log _{2}} (logarithm in base 2).
The only divergence for probabilities over a finite alphabet that is both an f-divergence and a Bregman divergence is the Kullback–Leibler divergence. [8] The squared Euclidean divergence is a Bregman divergence (corresponding to the function x 2 {\displaystyle x^{2}} ) but not an f -divergence.
Viewing the Kullback–Leibler divergence as a measure of distance, the I-projection is the "closest" distribution to q of all the distributions in P. The I-projection is useful in setting up information geometry , notably because of the following inequality, valid when P is convex: [ 1 ]
It can also be understood to be the infinitesimal form of the relative entropy (i.e., the Kullback–Leibler divergence); specifically, it is the Hessian of the divergence. Alternately, it can be understood as the metric induced by the flat space Euclidean metric, after appropriate changes of variable.