Shannon entropy quantifies the average uncertainty in a random variable's outcomes and guides the design of efficient encoding schemes.

In our experiments, we compared the Shannon entropies of different language models to evaluate their uncertainty.