incerto.llm.uncertainty_auc

incerto.llm.uncertainty_auc#

incerto.llm.uncertainty_auc(uncertainties, correctness)[source]#

AUC for using uncertainty to filter incorrect predictions.

Higher uncertainty should correlate with incorrectness.

Parameters:
  • uncertainties (Tensor) – Uncertainty scores (batch,)

  • correctness (Tensor) – Binary correctness (batch,)

Return type:

float

Returns:

AUC value (higher is better)