Evaluation Metrics

ProMaya tracks a comprehensive suite of metrics. Because real-world protein interaction landscapes are heavily imbalanced (most random pairs in a cell do not interact), relying solely on Accuracy or ROC curves can be misleading.

Core Metrics

Metric	Definition	Importance for PPI
AUROC	Area Under Receiver Operating Characteristic	Standard benchmark; threshold-independent discrimination.
AUPRC	Area Under Precision-Recall Curve	Crucial for highly imbalanced real-world predictions.
MCC	Matthews Correlation Coefficient	The most robust single-number summary of binary classification quality.
F1 Score	\(2 \times \frac{Precision \times Recall}{Precision + Recall}\)	Overall performance at the optimal classification threshold.
Sensitivity	True Positive Rate \(TP / (TP + FN)\)	Critical for ensuring transient or weak interactions are not missed.
Specificity	True Negative Rate \(TN / (TN + FP)\)	Critical for rejecting the tricky "Docking-Derived" decoy complexes.

Probability Calibration

Because the final prediction is output by a calibrated XGBoost ensemble, the predicted probability \(P(Interact)\) corresponds directly to a statistical likelihood, minimizing the Mean Absolute Error (MAE) between the predicted score and the actual test labels.

src/training/evaluate.py

import torch
import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score, matthews_corrcoef

def evaluate(model, dataloader, device):
    model.eval()
    all_preds, all_labels = [], []

    with torch.no_grad():
        for protein_a, protein_b, labels in dataloader:
            protein_a = move_to_device(protein_a, device)
            protein_b = move_to_device(protein_b, device)

            # Predict interaction
            logits, _ = model(protein_a, protein_b)
            probs = torch.sigmoid(logits).cpu().numpy()

            all_preds.extend(probs)
            all_labels.extend(labels.cpu().numpy())

    # Calculate rigorous metrics using scikit-learn
    all_preds = np.array(all_preds)
    all_labels = np.array(all_labels)
    binary_preds = (all_preds > 0.5).astype(int)

    metrics = {
        'auroc': roc_auc_score(all_labels, all_preds),
        'auprc': average_precision_score(all_labels, all_preds),
        'mcc': matthews_corrcoef(all_labels, binary_preds)
    }
    return metrics

Next: See how we understand the model's decisions in Graph Grad-CAM â†’