incerto.llm.SemanticEntropy

incerto.llm.SemanticEntropy#

class incerto.llm.SemanticEntropy[source]#

Bases: object

Semantic entropy - entropy over semantically clustered responses.

Clusters responses by meaning (not exact text) and computes entropy. Requires a semantic similarity model.

Reference:

Kuhn et al., “Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation”, ICLR 2023.

__init__()#

Methods

__init__()

compute(responses[, similarity_threshold, ...])

Compute semantic entropy by clustering similar responses.
static compute(responses, similarity_threshold=0.85, embedding_model=None)[source]#

Compute semantic entropy by clustering similar responses.

Parameters:

  • responses (List[str]) – List of generated text responses

  • similarity_threshold (float) – Threshold for considering responses similar

  • embedding_model – Optional sentence embedding model (e.g., SentenceTransformer) If None, falls back to lexical similarity

Returns:

  • semantic_entropy: Entropy over semantic clusters

  • num_clusters: Number of semantic clusters found

  • clusters: List of cluster assignments

Return type:

Dictionary with

Contents