Human Preference Annotation

Human preference annotation is the process of collecting human judgements — typically choosing between two model outputs or rating quality on a scale — to measure subjective dimensions of agent quality that automated metrics cannot capture, such as tone, helpfulness, and trustworthiness. Annotation data is used to evaluate agents, fine-tune models, and calibrate automated scoring systems.