Neel Mishra
← Back to Blog

Model Stealing and Extraction

MLOps Series Security & Governance

Introduction

Machine-learning models deployed behind APIs represent significant intellectual property. Training a state-of-the-art model can cost millions of dollars in compute, data curation, and expert labour. Model stealing — also called model extraction — is the practice of reconstructing a functionally equivalent copy of a target model by observing its input–output behaviour, without ever accessing the model weights directly.

Attackers may be motivated by economic gain (avoiding training costs), competitive advantage, or as a stepping stone toward adversarial attacks — a stolen copy can be probed offline to craft adversarial examples that transfer back to the original model. This post surveys the main attack vectors and the defences that ML engineers can deploy today.

Why this matters: A 2024 study found that many commercial vision APIs can be replicated to >90% fidelity with fewer than 100,000 queries — at a cost orders of magnitude lower than original training.
Target API queries Attacker trains Stolen Clone

Query-Based Extraction

The simplest form of model stealing relies on sending carefully crafted queries to a prediction API and using the returned labels or probabilities to train a surrogate model. The attacker treats the target as an oracle and builds a labelled dataset without any ground-truth data of their own.

Key factors that determine extraction success include the richness of the API response (full probability vectors leak far more than top-1 labels), the query budget, and how well the attacker can cover the input distribution. Active-learning strategies can dramatically reduce the number of queries required.

# Query-based model extraction attack import numpy as np from sklearn.neural_network import MLPClassifier def extract_model(target_api, n_queries=50000, input_dim=784): """Steal a model by querying the target API.""" X_steal = np.random.uniform(0, 1, (n_queries, input_dim)) # Collect predictions from the target y_probs = np.array([target_api.predict(x) for x in X_steal]) # Train surrogate on stolen labels surrogate = MLPClassifier(hidden_layer_sizes=(256, 128), max_iter=200) surrogate.fit(X_steal, y_probs.argmax(axis=1)) # Evaluate fidelity — how often surrogate agrees with target X_test = np.random.uniform(0, 1, (5000, input_dim)) y_target = np.array([target_api.predict(x).argmax() for x in X_test]) fidelity = (surrogate.predict(X_test) == y_target).mean() return surrogate, fidelity
Tip: If your API returns full softmax vectors instead of just the top class, attackers gain significantly more information per query. Consider returning only top-k classes or rounding probabilities.

Distillation Attacks

Knowledge distillation — originally a model compression technique — can be weaponized for model stealing. The attacker uses the target model's soft probability outputs as "teacher" labels and trains a smaller "student" model that mimics the teacher's decision boundaries.

Compared to hard-label extraction, distillation-based attacks are far more efficient. Soft labels encode inter-class relationships (e.g., a "7" that looks like a "1" will have a higher probability for class "1"), providing richer supervisory signal per query. A well-tuned distillation attack can achieve >95% agreement with the teacher using a fraction of the original training data.

Hard-Label Extraction

  • Uses only top-1 predicted class
  • Requires more queries (>100k typical)
  • Surrogate accuracy plateaus around 80–90%
  • Works against any classification API

Distillation Extraction

  • Uses full probability distribution
  • Requires fewer queries (10–50k)
  • Surrogate can exceed 95% fidelity
  • Requires API to return probabilities

Temperature scaling in distillation further amplifies the information leakage. Higher temperatures produce softer distributions that reveal more about the teacher's learned representations. Attackers can experiment with different temperature values on the stolen outputs to improve their surrogate's quality.

Model Watermarking

Watermarking embeds a secret, verifiable signature into a model so that intellectual property theft can be detected after the fact. The model owner can prove ownership by demonstrating that the suspected stolen model responds to specific trigger inputs in a characteristic way that occurs with negligible probability by chance.

There are two main approaches: backdoor-based watermarking, where the model is trained to classify certain trigger patterns to a secret target class, and parameter-based watermarking, where a signature is encoded directly into the model weights (e.g., by constraining certain weight statistics).

# Backdoor-based model watermarking import torch import torch.nn as nn def embed_watermark(model, train_loader, trigger_set, secret_label, epochs=5): """Embed a watermark by fine-tuning on trigger+normal data.""" optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) loss_fn = nn.CrossEntropyLoss() for epoch in range(epochs): for X_batch, y_batch in train_loader: # Mix normal data with trigger samples X_wm, y_wm = trigger_set.sample(batch_size=16) y_wm_labels = torch.full_like(y_wm, secret_label) X_combined = torch.cat([X_batch, X_wm]) y_combined = torch.cat([y_batch, y_wm_labels]) preds = model(X_combined) loss = loss_fn(preds, y_combined) optimizer.zero_grad() loss.backward() optimizer.step() return model def verify_watermark(model, trigger_set, secret_label, threshold=0.9): """Verify if a model contains the watermark.""" preds = model(trigger_set.inputs).argmax(dim=1) match_rate = (preds == secret_label).float().mean() return match_rate > threshold
Key insight: A good watermark must survive model fine-tuning, pruning, and distillation. Backdoor-based watermarks are generally more robust to these transformations than parameter-based approaches.

API Defenses

Defending ML APIs against extraction requires a layered approach combining output perturbation, rate limiting, and query analysis. No single defence is sufficient, but together they can significantly raise the cost and reduce the fidelity of attacks.

Output Perturbation

  • Round probabilities to fewer decimal places
  • Return only top-k predictions
  • Add calibrated noise to softmax outputs
  • Trade-off: reduces utility for legitimate users

Query Monitoring

  • Rate-limit requests per user/IP
  • Detect unusual query distributions
  • Flag synthetic or out-of-distribution inputs
  • Log and audit query patterns over time

Prediction poisoning is an active defence where the API deliberately returns perturbed predictions for queries that appear suspicious. The perturbations are designed to be small enough not to affect normal usage but large enough to degrade the quality of a surrogate model trained on the poisoned outputs.

Warning: Aggressive output perturbation can degrade legitimate use cases. Always evaluate the impact on your real users before deploying these defences. A/B testing the defence on a small traffic slice is recommended.
Rate Limiting Output Perturbation Query Monitoring Layered API Defence

Detection Strategies

Beyond prevention, organizations need the ability to detect when model stealing is occurring or has already occurred. Detection strategies operate at two levels: real-time monitoring of API traffic and post-hoc forensic analysis of suspected stolen models.

Traffic-based detection looks for anomalous query patterns. Extraction attacks tend to produce queries that differ from natural user traffic — they may be more uniformly distributed, contain synthetic inputs, or arrive at unusually regular intervals. Statistical tests comparing incoming queries to a baseline distribution can flag suspicious activity in real time.

Fingerprinting is a proactive detection method. The model owner identifies a set of "fingerprint" inputs near the decision boundary where even small model differences cause different predictions. If a suspected stolen model agrees with the original on these fingerprints at a rate higher than chance, it provides statistical evidence of copying.

Best practice: Combine watermarking (for legal proof of ownership), fingerprinting (for statistical detection), and API monitoring (for early warning) into a unified model protection strategy.

The table below summarizes the key properties of each detection approach and the scenarios where each is most effective:

Real-Time Detection

  • Monitors API query patterns live
  • Detects attacks as they happen
  • Can trigger automated countermeasures
  • Requires baseline traffic profile

Post-Hoc Forensics

  • Watermark and fingerprint verification
  • Useful for legal proceedings
  • Requires access to suspected stolen model
  • Can prove ownership with high confidence

In practice, the most robust approach combines all three layers. Watermarks provide a legal foundation for ownership claims. Fingerprints enable statistical testing even when only black-box access to the suspect model is available. And API monitoring catches extraction attempts early, potentially before the attacker has gathered enough data to build a useful surrogate.