From Wikipedia, the free encyclopedia - View original article

For the video game, see Perplexity (video game).

For the card/alternative reality game, see Perplex City.

Look up in Wiktionary, the free dictionary.perplexity |

In information theory, **perplexity** is a measurement of how well a probability distribution or probability model predicts a sample. It may be used to compare probability models.

The perplexity of a discrete probability distribution *p* is defined as

where *H*(*p*) is the entropy of the distribution and *x* ranges over events.

Perplexity of a random variable *X* may be defined as the perplexity of the distribution over its possible values *x*.

In the special case where *p* models a fair *k*-sided die (a uniform distribution over *k* discrete events), its perplexity is *k*. A random variable with perplexity *k* has the same uncertainty as a fair *k*-sided die, and one is said to be "*k*-ways perplexed" about the value of the random variable. (Unless it is a fair *k*-sided die, more than *k* values will be possible, but the overall uncertainty is no greater because some of these values will have probability greater than 1/*k*, decreasing the overall value while summing.)

A model of an unknown probability distribution *p*, may be proposed based on a training sample that was drawn from *p*. Given a proposed probability model *q*, one may evaluate *q* by asking how well it predicts a separate test sample *x*_{1}, *x*_{2}, ..., *x _{N}* also drawn from

where is customarily 2. Better models *q* of the unknown distribution *p* will tend to assign higher probabilities *q*(*x _{i}*) to the test events. Thus, they have lower perplexity: they are less surprised by the test sample.

The exponent above may be regarded as the average number of bits needed to represent a test event *x _{i}* if one uses an optimal code based on

The exponent may also be regarded as a cross-entropy,

where denotes the empirical distribution of the test sample (i.e., if *x* appeared *n* times in the test sample of size *N*).

In natural language processing, perplexity is a way of evaluating language models. A language model is a probability distribution over entire sentences or texts.

Using the definition of perplexity for a probability model, one might find, for example, that the average sentence *x _{i}* in the test sample could be coded in 190 bits (i.e., the test sentences had an average log-probability of -190). This would give an enormous model perplexity of 2

The lowest perplexity that has been published on the Brown Corpus (1 million words of American English of varying topics and genres) as of 1992 is indeed about 247 per word, corresponding to a cross-entropy of log_{2}247 = 7.95 bits per word or 1.75 bits per letter ^{[1]} using a trigram model. It is often possible to achieve lower perplexity on more specialized corpora, as they are more predictable.

**^**Brown, Peter F.; et al. (March 1992). "An Estimate of an Upper Bound for the Entropy of English".*Computational Linguistics***18**(1). Retrieved 2007-02-07.