incerto.llm.BidirectionalConsistency

incerto.llm.BidirectionalConsistency#

class incerto.llm.BidirectionalConsistency[source]#

Bases: object

Check consistency by asking the question in different ways.

If the model gives different answers to equivalent questions, it indicates high uncertainty.

__init__()#

Methods

__init__()

compute_consistency(answers[, match])

Compute consistency across answers to paraphrased questions.

paraphrase_prompts(question)

Generate paraphrased versions of a question.
static paraphrase_prompts(question)[source]#

Generate paraphrased versions of a question.

Parameters:

question (str) – Original question

Return type:

list[str]

Returns:

List of paraphrased questions

static compute_consistency(answers, match='tokens')[source]#

Compute consistency across answers to paraphrased questions.

Parameters:

  • answers (list[str]) – List of answers to paraphrased questions.

  • match (str) – Matching strategy. "tokens" (default) uses the mean pairwise Jaccard overlap of lowercased word sets — robust to small wording differences. "exact" uses the fraction of identical answers (legacy strict behaviour); best when answers are short canonical strings.

Return type:

float

Returns:

Consistency score in [0, 1]; higher means more consistent.

Contents