Code documentation

confidence_correctness_matrix.confidence_correctness_matrix(y_true, y_score, labels=None, abs_tolerance=1e-08, class_specific=False)

Calculate the Confidence-Correctnes matrix in its class-independent or class-specific versions.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

• abs_tolerance (absolute tolerance threshold for checking whether probabilities) – sum up to 1.0. Default = 1e-8.

• class-dependent (boolean, default=False.) – If ‘False’, the class-independent Confidence-Correctnes matrix is computed, and if ‘True’, the class-specific Confidence-Correctnes matrix is computed.

Returns:

confCorrM – Confidence-Correctness matrix.

Return type:

dictionary.

confidence_correctness_matrix.confidence_matrices(y_true, y_score, labels=None, abs_tolerance=1e-08)

Compute the high-confidence and low-confidence matrices.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

• abs_tolerance (absolute tolerance threshold for checking whether probabilities) – sum up to 1.0, default = 1e-8.

Returns:

• H (ndarray of shape (n_classes, n_classes).) – High-confidence matrix.

• L (ndarray of shape (n_classes, n_classes).) – Low-confidence matrix.

confidence_correctness_matrix.confidence_weights(y_true, y_score, labels=None)

Calculate the lambda values for the high-confidence and low-confidence matrices.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

Returns:

• lambda_H (float.) – Lambda value for the high-confidence matrix.

• lambda_L (float.) – Lambda value for the low-confidence matrix.

confidence_correctness_matrix.plot_confidence(y_true, y_score, output_fig_path=None)

Plots and shows the Confidence-Correctnes matrix horizontal bar chart and saves them in format .png.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• output_fig_path (if given, figure will be saved at this location. If no file extension is given,) – png will be used by default, default=None.

confidence_correctness_matrix.prob_accuracy_score(y_true, y_score, labels=None)

Compute the probabilistic accuracy.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

Returns:

prob_acc – Probabilistic accuracy.

Return type:

float.

confidence_correctness_matrix.prob_balanced_accuracy_score(y_true, y_score, labels=None)

Compute the probabilistic balanced accuracy.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

Returns:

prob_b_acc – Probabilistic balanced accuracy.

Return type:

float.

confidence_correctness_matrix.prob_cohen_kappa_score(y_true, y_score, labels=None)

Compute the probabilistic Cohen Kappa.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

Returns:

prob_cohen_kappa – Probabilistic Cohen Kappa.

Return type:

float.

confidence_correctness_matrix.prob_confusion_matrix(y_true, y_score, labels=None, abs_tolerance=1e-08)

Compute the probabilistic confusion matrix.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

• abs_tolerance (absolute tolerance threshold for checking whether probabilities) – sum up to 1.0, default = 1e-8.

Returns:

prob_conf_matrix – Probabilistic confusion matrix.

Return type:

ndarray of shape (n_classes, n_classes).

confidence_correctness_matrix.prob_f1_score(y_true, y_score, labels=None, pos_label=1, average='binary')

Compute the probabilistic F1-score.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

• pos_label (int, default=1.) – The class to report if average=’binary’ and the data is binary, otherwise this parameter is ignored.

• average (string, default="binary".) – This parameter is required for multiclass targets and determines the type of averaging performed on the data: “binary”, “micro”, “macro” and “weighted”.

Returns:

prob_f1 – Probabilistic F1-score.

Return type:

float.

confidence_correctness_matrix.prob_matthews_corrcoef(y_true, y_score, labels=None)

Compute the probabilistic Matthews Correlation Coefficient.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

Returns:

prob_m_corrcoef – Probabilistic Matthews Correlation Coefficient.

Return type:

float.

confidence_correctness_matrix.prob_precision_score(y_true, y_score, labels=None, pos_label=1, average='binary')

Compute the probabilistic precision.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

• pos_label (int, default=1.) – The class to report if average=’binary’ and the data is binary, otherwise this parameter is ignored.

• average (string, default="binary".) – This parameter is required for multiclass targets and determines the type of averaging performed on the data: “binary”, “micro”, “macro” and “weighted”.

• ----------

• prob_prec (float.) – Probabilistic precision.

confidence_correctness_matrix.prob_recall_score(y_true, y_score, labels=None, pos_label=1, average='binary')

Compute the probabilistic recall.

Parameters:

• y_true (array-like of shape (n_samples,).) – Ground truth (correct) labels.

• y_score (array-like of shape (n_samples, n_classes).) –

  Probabilities of predicted labels, as returned by a classifier. The sum of these probabilities must sum up to 1.0 over classes.

  The order of the class scores must correspond to the numerical or lexicographical order of the labels in y_true.

• labels (array-like of shape (n_classes,), default=None.) –

  List of labels to index the matrix.

  If ‘None’ is given, those that appear at least once in ‘y_true’ or ‘y_pred’ are used in sorted order.

• pos_label (int, default=1.) – The class to report if average=’binary’ and the data is binary, otherwise this parameter is ignored.

• average (string, default="binary".) – This parameter is required for multiclass targets and determines the type of averaging performed on the data: “binary”, “micro”, “macro” and “weighted”.

Returns:

prob_rec – Probabilistic recall.

Return type:

float.