Selective Prediction

The selective prediction module enables models to abstain from predictions when uncertain, providing risk-coverage tradeoffs.

Base Class

BaseSelectivePredictor(*args, **kwargs)

Abstract base class for selective prediction methods.

Methods

SoftmaxThreshold(backbone)	Classical confidence-thresholding à la Chow (1957).
SelfAdaptiveTraining(backbone, num_classes)	Self-Adaptive Training for better calibration and selective prediction.
DeepGambler(backbone, num_classes[, ...])	Add an extra abstain logit and train with the gambler's loss:
SelectiveNet(backbone, num_classes[, ...])	Implementation of SelectiveNet (Geifman & El-Yaniv, 2019). The model outputs: * h(x): class logits * g(x): selection probability.
make(selector, *args, **kwargs)	Quick factory: make('msp', backbone) or make('selectivenet', ...).

Metrics

coverage(reject)	Proportion of accepted samples.
risk(pred, y, reject)	Error rate on accepted samples (a.k.a.
aurc(sorted_conf, sorted_errors)	Area under the Risk-Coverage curve.
accuracy_coverage_curve(logits, y[, confidence])	Return (coverage, accuracy) curve for each possible threshold.

Visualization

plot_risk_coverage(logits, y[, confidence, ...])	Plot the risk-coverage curve for selective prediction.
plot_accuracy_coverage(logits, y[, ...])	Plot the accuracy-coverage curve for selective prediction.