Permutation Importance in C++

July 2026 · 14 min read

Permutation importance is the simplest model-agnostic explanation. Measure the model's baseline score, shuffle one feature column, measure the score again, and report the drop. If shuffling a feature destroys performance, the model relied on that feature.

Implementation

#include <bits/stdc++.h>
using namespace std;

using Vector = vector<double>;
using Matrix = vector<Vector>;
using Predictor = function<double(const Vector&)>;

double rmse(const Predictor& pred, const Matrix& X, const Vector& y) {
    double s = 0.0;
    for (size_t i = 0; i < X.size(); ++i) {
        double e = pred(X[i]) - y[i];
        s += e * e;
    }
    return sqrt(s / X.size());
}

vector<double> permutation_importance(const Predictor& pred,
                                      Matrix X,
                                      const Vector& y,
                                      int repeats,
                                      mt19937& rng) {
    int d = (int)X[0].size();
    double base = rmse(pred, X, y);
    vector<double> importance(d, 0.0);

    for (int f = 0; f < d; ++f) {
        double totalDrop = 0.0;
        vector<double> original(X.size());
        for (size_t i = 0; i < X.size(); ++i) original[i] = X[i][f];

        for (int r = 0; r < repeats; ++r) {
            vector<double> shuffled = original;
            shuffle(shuffled.begin(), shuffled.end(), rng);
            for (size_t i = 0; i < X.size(); ++i) X[i][f] = shuffled[i];
            totalDrop += rmse(pred, X, y) - base;
        }

        for (size_t i = 0; i < X.size(); ++i) X[i][f] = original[i];
        importance[f] = totalDrop / repeats;
    }
    return importance;
}

Reading the Output

The units are metric units. For RMSE, larger positive values mean shuffling the feature made RMSE worse. For accuracy, the sign is usually reversed or reported as baseline accuracy minus shuffled accuracy.