incerto.llm.HistogramBinning

incerto.llm.HistogramBinning#

class incerto.llm.HistogramBinning(n_bins=10)[source]#

Bases: object

Histogram binning calibration for LLM confidence scores.

Groups predictions by confidence and adjusts to empirical accuracy.

Parameters:

n_bins (int)

__init__(n_bins=10)[source]#
Parameters:

n_bins (int) – Number of bins for calibration

Methods

__init__([n_bins])

calibrate(confidence)

Apply calibration to a confidence score.

fit(confidences, correctness)

Fit binning calibration.
__init__(n_bins=10)[source]#
Parameters:

n_bins (int) – Number of bins for calibration

fit(confidences, correctness)[source]#

Fit binning calibration.

Parameters:

  • confidences (Tensor) – Model confidence scores (batch,)

  • correctness (Tensor) – Binary correctness indicators (batch,)

calibrate(confidence)[source]#

Apply calibration to a confidence score.

Parameters:

confidence (float) – Original confidence (0-1)

Return type:

float

Returns:

Calibrated confidence

Contents