incerto.llm.SequencePerplexity

incerto.llm.SequencePerplexity#

class incerto.llm.SequencePerplexity[source]#

Bases: object

Perplexity of the entire sequence.

Exponential of average log-probability. Standard metric for language model quality.

__init__()#

Methods

__init__()

compute(logits, token_ids[, mask, dim])

Compute sequence perplexity.
static compute(logits, token_ids, mask=None, dim=-1)[source]#

Compute sequence perplexity.

Parameters:

  • logits (Tensor) – Token logits of shape (batch, seq_len, vocab_size)

  • token_ids (Tensor) – Generated token IDs of shape (batch, seq_len)

  • mask (Optional[Tensor]) – Optional mask for padding of shape (batch, seq_len)

  • dim (int) – Dimension to compute softmax over (default: -1)

Return type:

Tensor

Returns:

Perplexity values of shape (batch,)

Contents