Annotation quality is model quality. Vague guidelines produce inconsistent labels — and inconsistent labels produce models that generalise poorly precisely where it matters most. We design annotation schemas with measurable inter-annotator agreement targets before any labelling begins, and instrument IAA throughout the project so quality degradation is visible before it reaches training.
The annotation process is where model quality is determined — not at training time. A model trained on ambiguously annotated data learns the annotator's inconsistencies, not the underlying task. A training distribution that over-represents common cases and under-represents edge cases produces a model that looks strong on benchmarks and weak on the production long tail.
Inter-annotator agreement (IAA) — Cohen's kappa or Krippendorff's alpha — is the measurement that validates annotation quality. Low IAA is a leading indicator of poor model performance. It means different annotators answered the same question differently, and the training labels contain noise rather than signal. Most annotation projects measure IAA late — after the full dataset is annotated — when the cost of finding systematic disagreement is highest. We measure from the first validation batch.
We design annotation processes as software engineering problems: guidelines are versioned, IAA is measured continuously, and edge cases are explicitly catalogued and covered. Annotation guidelines go through a small-batch validation round before full-scale annotation — a 50-100 sample batch with IAA measurement catches guideline ambiguity before it propagates through the full dataset at high cost.
Training dataset composition is designed against the production distribution, not the available data distribution. If production data contains a long tail of rare cases that available data under-represents, we design targeted collection for those cases before training starts. For LLM fine-tuning with RLHF, we design preference comparison workflows where annotators compare model output pairs on defined quality dimensions — the criteria that the reward model will learn from.
Define the annotation task precisely. Guidelines go through a 50-100 sample validation batch with IAA measurement before full-scale annotation. Low IAA on the validation batch means guideline revision, not full-scale annotation with ambiguous instructions.
Analyze what the model will encounter in production. Identify under-represented input types, rare but important cases, and adversarial examples relevant to the task. Design targeted data collection for coverage gaps.
Configure Label Studio or Prodigy for the task type. Set up IAA measurement. Establish annotator calibration — a shared batch where annotators discuss disagreements before independent annotation begins.
Continuous IAA monitoring throughout annotation. Disagreements resolved through adjudication by a senior annotator or consensus. Samples below IAA threshold re-annotated. Guidelines updated when systematic disagreements reveal ambiguity.
Validate class balance, edge case coverage, adversarial example inclusion, and final IAA metrics. RLHF preference data: validate criteria consistency and pair quality before reward model training.
Guidelines are tested on a small validation batch before full-scale annotation begins. Inter-annotator agreement is measured throughout using Cohen's kappa or Krippendorff's alpha depending on task type — not only at the end. Low-agreement samples are adjudicated and guidelines updated to eliminate the ambiguity that caused them.
We analyze the actual distribution of production inputs and design data collection to match it — including edge cases and rare inputs that benchmark datasets systematically under-represent. A model trained on a skewed distribution passes evals and fails in the wild; coverage analysis is how you avoid that.
For classification and detection tasks, we include near-boundary examples in the training set: inputs with similar surface features but different labels, known misclassification patterns from prior model versions, and synthetic hard negatives. This directly reduces the long-tail failure modes that evaluation sets miss.
For LLM fine-tuning, we design pairwise preference comparison workflows where annotators rate output pairs on defined quality dimensions — helpfulness, factuality, safety, or domain-specific criteria. Each dimension gets explicit criteria with examples; vague criteria produce noisy reward signals that degrade reward model quality.
Active learning selects unlabeled samples where the current model is most uncertain, prioritizing annotation effort where it moves the model most. We add a selection review step to catch distribution-edge artifacts — cases where model uncertainty is high for reasons unrelated to the actual task, which active learning would otherwise over-sample.
Rigorous annotation schema design reduces post-deployment model failures by catching distribution mismatches, edge case underrepresentation, and label ambiguity early. The cost of fixing an annotation schema before labelling starts is roughly 1% of the cost of retraining a model on labels discovered to be inconsistent after the fact.
Anonymized engagements with real metrics — no client names per NDA.
85%
Screening Time Reduction
4.2x
Recruiter Throughput
12s
Avg Processing Time
“The consistency improvement was the part I did not expect. We were applying different criteria across different reviewers without realising it. The AI screening at least applies the same criteria to every application.”
— Head of Talent Acquisition, Technology Company
87%
Auto-Classification Rate
3.2 days
Avg Turnaround (from 15)
2.1%
Misroute Rate (from 18%)
“The misrouting rate was the metric that mattered internally — every misrouted document created rework cycles that consumed staff time and delayed the original applicant. Getting that from 18% to 2% changed the entire operations picture.”
— Director of Digital Services, Regional Government Department
Learning curves tell you. Train models on increasing subsets of your data and plot performance vs. dataset size. When the curve flattens — additional data produces diminishing improvement — you have likely reached data sufficiency for the current model architecture. If performance has not reached your target by that point, the problem is more likely model architecture, task framing, or annotation quality than data volume.
Synthetic data is most valuable for augmenting rare cases in the training distribution — not replacing real data. LLM-generated synthetic examples for text tasks, augmented images for vision tasks. The risk is distributional mismatch: synthetic data that does not match real production inputs adds noise, not signal. Validate synthetic data quality against held-out real samples before including it in training.
Label Studio (open-source, flexible, self-hostable), Prodigy (tight spaCy integration, active learning support), Scale AI (managed annotation at scale), and Labelbox (enterprise annotation management). Platform selection is driven by task type, budget, data sensitivity requirements, and whether in-house or outsourced annotation is appropriate.
RLHF preference quality depends on the clarity of comparison criteria. We define explicit quality dimensions for each pairwise comparison: accuracy, helpfulness, safety, tone — whatever matters for your use case. Annotators rate which output is better on each dimension, not a single overall preference. Clear criteria reduce annotator variance and produce more useful reward signals.
Tell us what you are building. We will scope it, price it honestly, and give you a clear plan.
Free 30-min scoping call
Fordel Studios