Search results
Results from the WOW.Com Content Network
The Jensen–Shannon divergence is bounded by 1 for two probability distributions, given that one uses the base 2 logarithm: [8] ().With this normalization, it is a lower bound on the total variation distance between P and Q:
Kullback–Leibler divergence; Rényi divergence; Jensen–Shannon divergence; Bhattacharyya distance (despite its name it is not a distance, as it violates the triangle inequality) f-divergence: generalizes several distances and divergences
In probability theory, an -divergence is a certain type of function (‖) that measures the difference between two probability distributions and . Many common divergences, such as KL-divergence , Hellinger distance , and total variation distance , are special cases of f {\displaystyle f} -divergence.
Thus, we see that the point of the discriminator is mainly as a critic to provide feedback for the generator, about "how far it is from perfection", where "far" is defined as Jensen–Shannon divergence. Naturally, this brings the possibility of using a different criteria of farness.
The Fisher metric also allows the action and the curve length to be related to the Jensen–Shannon divergence. [7] Specifically, one has ( b − a ) ∫ a b ∂ θ j ∂ t g j k ∂ θ k ∂ t d t = 8 ∫ a b d J S D {\displaystyle (b-a)\int _{a}^{b}{\frac {\partial \theta ^{j}}{\partial t}}g_{jk}{\frac {\partial \theta ^{k}}{\partial t}}\,dt ...
where is the Kullback–Leibler divergence, and is the outer product distribution which assigns probability () to each (,).. Notice, as per property of the Kullback–Leibler divergence, that (;) is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when and are independent (and hence observing tells you nothing about ).
The Jensen–Shannon divergence, like all f-divergences, is locally proportional to the Fisher information metric. It is similar to the Hellinger metric (in the sense that it induces the same affine connection on a statistical manifold).
A great many important inequalities in information theory are actually lower bounds for the Kullback–Leibler divergence.Even the Shannon-type inequalities can be considered part of this category, since the interaction information can be expressed as the Kullback–Leibler divergence of the joint distribution with respect to the product of the marginals, and thus these inequalities can be ...