incerto.llm.SelfConsistency

incerto.llm.SelfConsistency#

class incerto.llm.SelfConsistency[source]#

Bases: object

Self-consistency via majority voting across samples.

Generate N samples and measure agreement. Higher agreement indicates lower uncertainty. Proposed by Wang et al. (2023).

Reference:

Wang et al., “Self-Consistency Improves Chain of Thought Reasoning” ICLR 2023.

__init__()#

Methods

__init__()

compute(responses[, normalize_fn])

Compute self-consistency from multiple responses.

first_sentence(text)

Extract first sentence, lowercased and stripped.
static first_sentence(text)[source]#

Extract first sentence, lowercased and stripped.

Useful as a normalize_fn for short-answer tasks where the core answer appears in the first sentence but models append varying continuations.

Parameters:

text (str)

Return type:

str

static compute(responses, normalize_fn=None)[source]#

Compute self-consistency from multiple responses.

Parameters:

  • responses (List[str]) – List of generated text responses

  • normalize_fn (Optional[Callable[[str], str]]) – Optional function to normalize responses (e.g., extract answers, lowercase, strip)

Returns:

  • agreement_rate: Fraction agreeing with majority

  • entropy: Entropy over response distribution

  • top_response: Most common response

  • num_unique: Number of unique responses

Return type:

Dictionary with

Contents