Evaluate Graph Risk at a Fixed Precision and a Realistic Time Cutoff

September 2026 · 16 min read

Before you start: Graph features and propagation. We will calculate the precision and recall we need here.

A graph model can recover more risky applications simply by creating more alerts. It can also look excellent because it sees tomorrow’s investigation results or recognizes the publisher of a held-out app. A useful evaluation must say which decisions it predicts, what information they could use, and how many alerts someone can review.

This chapter evaluates the final classifier’s score, after intrinsic and graph features have been combined. The numbers are synthetic teaching examples, not measured gains from a deployed risk system.

Choose a decision rule on validation data

At a chosen score threshold, a true positive is a flagged app that is actually risky. A false positive is a flagged benign app. Precision asks what fraction of alerts are correct; recall asks what fraction of all risky apps were caught. Both require the corresponding outcome labels:

Suppose validation contains 20 apps, including eight risky ones. At threshold 0.70, the classifier flags eight apps, six of them risky. Precision is 6/8 = 75%, recall is 6/8 = 75%, and the review workload is eight. Threshold 0.50 catches all eight risky apps but flags twelve: recall reaches 100%, while precision falls to 8/12 ≈ 66.7%.

Specify the constraint before selecting a threshold. In the example, maximize validation recall while meeting a minimum precision and a maximum number of reviews for this cohort. A production budget needs its own unit, such as reviews per day. The same score threshold can create different daily volumes as traffic changes.

With the default constraints, 0.70 is selected. On the separate test cohort it flags eight apps, but only five are risky: precision and recall are both 5/8 = 62.5%. Meeting the validation target did not guarantee meeting it on test. The frozen rule reveals that failure instead of hiding it with another threshold search.

A budget of two yields no feasible threshold among these candidates. Report that result; it is not a reason to treat zero alerts as perfect precision. The scikit-learn threshold guide separates score fitting from decision tuning and explains the need for separate tuning data.

def summarize(rows, threshold):
    # rows have finite scores and resolved binary labels (0 or 1).
    flagged = [row for row in rows if row["score"] >= threshold]
    true_positives = sum(row["label"] for row in flagged)
    positives = sum(row["label"] for row in rows)
    return {
        "threshold": threshold,
        "flagged": len(flagged),
        "tp": true_positives,
        "precision": true_positives / len(flagged) if flagged else None,
        "recall": true_positives / positives if positives else None,
    }

def choose_threshold(validation_rows, candidates, min_precision, budget):
    options = [summarize(validation_rows, t) for t in candidates]
    feasible = [m for m in options
                if m["precision"] is not None
                and m["recall"] is not None
                and m["precision"] >= min_precision
                and m["flagged"] <= budget]
    # Maximize recall; break ties with fewer alerts, then higher threshold.
    return max(feasible, key=lambda m: (m["recall"], -m["flagged"],
                                        m["threshold"]), default=None)

# Only validation labels choose the threshold.
# selected = choose_threshold(validation_rows, [.9, .7, .5, .3], .75, 8)
# if selected is not None:
#     test_result = summarize(test_rows, selected["threshold"])

A new application may still have a familiar publisher

A held-out app ID establishes only that this app’s label was not used for fitting. It does not establish that its publisher, incident, or infrastructure is new. Separate the questions “can we score another app from a known group?” and “can we score a new group?”

Transductive evaluation can legitimately use unlabeled test-node structure that exists at inference. Inductive evaluation introduces new nodes or graphs. Neither term grants permission to use future edges or test labels. For grouping mechanics, see group splitting; for chronology, see time-based splitting.

Replay decisions with two separate label roles

Imagine fitting at the end of June, selecting the threshold from July decisions once their labels have matured, then testing on September decisions. Model fitting and threshold selection must finish before the first test request. Reconstruct each request’s graph with the availability rules from chapter 1.

A’s own eventual label is allowed to arrive later for evaluating A. A neighbor’s label can be an input to A’s prediction only if it was available at that decision time. These roles must not be joined through the same unrestricted label table. Choose whether the replay freezes all label-derived inputs at deployment or permits newly available neighbor labels as it advances; the latter may be legitimate, but it must reproduce the deployed update policy.

The Temporal Graph Benchmark illustrates why chronological splits and explicit inference-time access matter: its streaming protocol permits previously observed test edges while restricting model-weight updates. Our application-risk replay is a separate task with an additional label-availability contract.

Identify what the graph actually adds

Use the same cohorts, outcome definitions, label maturity rules, and resource measurements for all models. Then change one source of evidence at a time:

Refit a feature-ablation model without the removed inputs and choose its threshold on validation. Zeroing an input at test time instead measures a perturbation of the already fitted model; it answers a different question. For each comparison, report known/new publisher slices, isolated apps, degree buckets, evidence age, and how many independent incidents supplied the gain.

The alert queue cannot reveal all missed risk

If investigators review only alerts, labels are selected by the existing decision process. This is an instance of the selective-labels problem studied by Lakkaraju et al. Reviewed cases need not represent the whole population.

Suppose 1,000 apps were scored, 100 flagged, and all 100 alerts resolved: 80 risky and 20 benign. Alert precision is 80%. Recall is unknown because the other 900 apps may contain missed positives. “We found 80 of the 80 known risky apps” does not establish population recall.

If that audit finds nine risky apps, the estimated risky fraction outside the queue is 9/90 = 10%. Apply it to the 900 unflagged apps to estimate 90 missed positives. The estimated population recall becomes 80/(80 + 90) ≈ 47.1%. The missed-positive total is a sampling estimate; the recall formed from it is a ratio estimate with uncertainty, not an exact count.

A stratified audit can spend more effort on important or rare slices, but keep inclusion probabilities and use the corresponding sampling weights. Track unresolved cases separately. A maturity window defines when outcomes can be assessed; it does not turn an unreviewed app into a confirmed benign one. Auditing can itself generate future graph labels, so log when those labels become available to subsequent predictions.

Report evidence at the unit you want to generalize to

A hundred applications from one incident are not a hundred independent incidents. For a claim about future publishers or incidents, preserve those groups in resampling and compare models on the same sampled groups. For performance across time, use a temporal evaluation and an uncertainty method suited to its dependence. Choose the unit before calculating an interval.

The paired-comparison notes explain why both models must be evaluated on matching resamples. Here, resampling may need to operate on publishers or time blocks rather than queries. Report the number and size of those groups. A narrow interval from many correlated rows can overstate the evidence.

A useful final report gives the frozen rule, precision and recall with their denominators, flagged volume, resolved-label coverage, group/time split, new-entity slices, and latency including graph-feature lookup. State whether an interval conditions on a fixed fitted model and threshold or includes repeating the selection process. A lower precision bound on a small selected validation set is not automatically a deployment guarantee.

Finish by reviewing examples of newly recovered risky apps and new false positives. For each, inspect the underlying graph evidence. That connects a metric change to the relationships and decisions that produced it.

References and further reading

• scikit-learn: tuning decision thresholds — Separating score estimation, validation-based threshold selection, and held-out evaluation.
• Huang et al.: Temporal Graph Benchmark — Chronological task evaluation with an explicit streaming information-access protocol.
• Lakkaraju et al.: The Selective Labels Problem — Why labels selected by prior decisions can make population evaluation misleading.