Domain Email Generation: Fine-Tuning, Serving, and a Credible Evaluation

September 2026 · 4 min read

Before you start: LLM-as-a-judge foundations.

A domain email generator must produce a useful message from structured facts and user intent. The key challenge is not making prose sound polished; it is preserving facts, choosing the right content, and avoiding unsupported promises. Treat model adaptation, serving optimization, and quality evaluation as separate experimental axes.

Construct examples that teach the intended behavior

Each supervised example should include the facts available at generation time, the requested task, and a target response consistent with those facts. Split by customer, template family, and time when reuse would cause leakage. Deduplicate near-identical messages. A random row split across repeated templates can reward memorization.

Define required omissions as well as required content: unsupported discounts, invented delivery dates, and unverified vehicle specifications should not appear. Retrieval can supply current facts; fine-tuning can teach task style and behavior. A model’s weights are usually a poor place to store frequently changing inventory information.

Compare the simplest useful baselines

Start with a strong prompt and structured facts, then compare retrieval augmentation, LoRA adaptation, or both under the same test set. Keep the base model, training corpus, adapter targets, rank, optimizer, and stopping criterion versioned. Use the existing LoRA/QLoRA notes for the update mechanics; here the question is whether adaptation improves the task.

Separate style preference from factual correctness. A concise response may be preferable even when lexical similarity to the reference falls. Conversely, a near-copy can obtain a high similarity score while reproducing a wrong date. Use references as evidence, not as the only definition of quality.

Create a rubric with observable criteria

Make model comparison paired and blinded

Generate A and B for the same inputs and randomize their presentation order to a judge. Include the input facts, explicit rubric, and a tie option. Audit a stratified sample with humans, especially on important disagreements. Measure order sensitivity by swapping A/B and consistency by repeating a subset. Freeze judge version, prompt, and parser.

Compute paired differences or win/tie/loss rates and uncertainty over independent evaluation units. If many emails share one customer or template family, use grouped analysis. Report effect size, not just a p-value, and avoid selecting the best model after repeatedly examining the final test set. The paired-comparison chapter gives the underlying logic.

Optimize latency without changing the quality experiment silently

Quantization changes numerical computation; a serving engine changes batching, memory management, and execution. Compare full-precision and quantized quality with fixed prompts and decoding settings, then benchmark serving under the intended concurrency. Record model load memory, peak runtime memory, first-token latency, completion latency, and output length.

Do not attribute a speedup entirely to quantization when the engine, batch size, or output length also changed. Include preprocessing, retrieval, and validation in end-to-end timing. A model generating fewer tokens may finish faster because it omits required content. Pair latency measurements with task success and completeness.

A concrete release record

Save input snapshots, generated outputs, deterministic check results, judge decisions, human audits, statistical summaries, and serving configuration. Choose the model on validation, evaluate once on the held-out set, and use a staged deployment with quality and latency monitoring. When feedback identifies a failure, add a generalizable test family rather than copying one production example into the training set and declaring the issue solved.

Sources and further reading

• Zheng et al.: judging LLM-as-a-judge with MT-Bench and Chatbot Arena — Judge biases, agreement, and pairwise evaluation.
• Hu et al.: LoRA — Parameter-efficient adaptation.
• Kwon et al.: PagedAttention — Serving memory management and throughput.