incerto.llm.SelfConsistency

incerto.llm.SelfConsistency#

class incerto.llm.SelfConsistency[source]#

Bases: object

Self-consistency via majority voting across samples.

Generate N samples and measure agreement. Higher agreement indicates lower uncertainty. Proposed by Wang et al. (2023).

Reference:

Wang et al., “Self-Consistency Improves Chain of Thought Reasoning” ICLR 2023.

__init__()#

Methods

__init__()

compute(responses[, normalize_fn])

Compute self-consistency from multiple responses.
static compute(responses, normalize_fn=None)[source]#

Compute self-consistency from multiple responses.

Parameters:

  • responses (List[str]) – List of generated text responses

  • normalize_fn (Optional[Callable[[str], str]]) – Optional function to normalize responses (e.g., extract answers, lowercase, strip)

Returns:

  • agreement_rate: Fraction agreeing with majority

  • entropy: Entropy over response distribution

  • top_response: Most common response

  • num_unique: Number of unique responses

Return type:

Dictionary with

Contents