Design: Content Moderation Pipeline

Requirements & Scope

Content moderation keeps platforms safe by detecting harmful content — hate speech, spam, violence, misinformation — at the scale of billions of posts per day.

Multi-Stage Filtering

A cascade of increasingly expensive classifiers filters content. Each stage either auto-removes clearly violating content or passes ambiguous content to the next stage.

Stage 1 — Hash Matching

Known violating content (e.g., CSAM hashes, known spam URLs) is matched using perceptual hashing (PhotoDNA, pDNA). This is instantaneous and handles re-uploads.

Stage 2 — ML Classifiers

Purpose-built classifiers score content across policy categories: hate speech, nudity, violence, spam, and misinformation. Each classifier outputs a category probability.

# Multi-label content classification
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("moderation-bert-v3")
tokenizer = AutoTokenizer.from_pretrained("moderation-bert-v3")

CATEGORIES = ["hate_speech", "violence", "spam", "nudity", "misinfo"]

def classify_text(text: str) -> dict:
    inputs = tokenizer(text, return_tensors="pt", truncation=True)
    logits = model(**inputs).logits
    probs = torch.sigmoid(logits)[0]
    return {cat: probs[i].item() for i, cat in enumerate(CATEGORIES)}

Stage 3 — Ensemble & Thresholding

Category scores are compared against per-category thresholds. Items with high-confidence violations are auto-removed. Borderline items proceed to LLM or human review.

Human Review Loop

Human reviewers handle ambiguous cases that ML can't confidently classify. A well-designed review queue maximises reviewer efficiency.

Queue Prioritisation

Items are prioritised by severity (violence > spam), virality (high-engagement posts first), and ML confidence (low-confidence items need human judgment most).

# Priority scoring for review queue
def priority_score(item):
    severity_weight = {"violence": 10, "hate_speech": 8, "nudity": 6, "spam": 2}
    severity = max(severity_weight.get(c, 1) * s
                   for c, s in item.scores.items())
    virality = min(item.view_count / 1000, 10)
    uncertainty = 1.0 - max(item.scores.values())
    return severity * 0.5 + virality * 0.3 + uncertainty * 0.2

Reviewer Tools

The review UI shows the content, ML predictions with confidence, similar previously-reviewed items, and the relevant policy excerpts. Reviewers select a verdict in < 30 seconds per item.

Appeal Process

Users whose content is removed can appeal the decision. Appeals are routed to a different reviewer (or senior reviewer) to avoid confirmation bias.

Appeal outcomes feed back into the training pipeline. Overturned decisions become hard negatives that help the model learn nuance.

LLM Integration

Large language models can handle nuanced moderation tasks that traditional classifiers struggle with — sarcasm, cultural context, coded language, and policy interpretation.

# LLM-assisted moderation
import openai

SYSTEM_PROMPT = """You are a content moderator. Classify the following post
against our content policy. Return JSON with:
- violates: bool
- category: str (hate_speech|violence|spam|none)
- confidence: float (0-1)
- explanation: str (one sentence)"""

def llm_moderate(content: str) -> dict:
    response = openai.ChatCompletion.create(
        model="gpt-4-turbo",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": content}
        ],
        temperature=0.0,
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)

Prompt Engineering for Moderation

Effective moderation prompts include the specific policy text, examples of edge cases, and a structured output format. Chain-of-thought prompting improves accuracy on nuanced cases.

Full Architecture

The complete content moderation pipeline integrates all stages into a unified, observable system.