LLM Uncertainty

The LLM module provides uncertainty quantification methods specifically designed for large language models.

Token-level Uncertainty

TokenEntropy()	Compute predictive entropy at each token position.
TokenConfidence()	Maximum softmax probability at each token position.
SurprisalScore()	Surprisal (negative log-probability) of generated tokens.
TopKConfidence()	Confidence based on probability mass in top-k tokens.

Sequence-level Uncertainty

SequenceProbability()	Joint probability of the entire sequence.
SequenceEntropy()	Aggregated entropy over the sequence.

Sampling-based Uncertainty

SelfConsistency()	Self-consistency via majority voting across samples.
SemanticEntropy()	Semantic entropy - entropy over semantically clustered responses.
PredictiveEntropy()	Predictive entropy across multiple sampled sequences.
MutualInformation()	Mutual information between predictions and model (aleatoric vs epistemic).

Generation Methods

BeamSearchUncertainty()	Uncertainty estimation from beam search scores.
IDontKnowDetection()	Detect when the model is expressing uncertainty verbally.
ContrastiveDecoding()	Uncertainty from contrastive decoding (comparing expert vs amateur models).

Calibration

Visualization