What is KL divergence? It measures the distance between 2 Gaussian distributions. Be more precise - given 2 distributions P and Q, and let P be the target distribution. KL divergence is: CrossEntropy(P|Q) - Entropy(P)
What is Entropy and Cross Entropy? Entropy is a special case of Cross Entropy where the two distributions are the same.
As explained in the video tutorial (see the video in the reference), starting with entropy, given a Gaussian distribution with probability function p(x), do 10 actual samples to get a sequence. The probability of getting the sequence is the product of the p(xi) of each step. To make the equation into summation, take a log at both side of the equation, each sample become log(p(xi)). Group the p(xi) of the same values together and divide by total, that should also be p(x), because we are repeating the sampling step. That makes the Entropy(P)= −∑ p(x) ln(p(x))
Note that p(x) is the probability of the actual sampling, and ln(p(x)) is the probability of the distribution.
Get another distribution Q with q(x), replace the ln(p(x)) part with ln(q(x)), it becomes Cross Entropy(P|Q)= −∑ p(x) ln(q(x)). It measures the probability to generate the same sequence with a different distribution. This value is larger than Entropy(P) because it is a different distribution.
KL divergence is CrossEntropy(P|Q) - Entropy(P)
References:
KL Divergence: https://youtu.be/sjgZxuCm_8Q?si=CAU5g6_DGxto0h0J
No comments:
Post a Comment