Entropy

Cross-entropy is a measure of the difference between two probability distributions. It also plays a role in the information-theoretic formulation of the Second Law of Thermodynamics, thereby providing a bridge between physics and machine learning.