Towards Optimality in Active Preference Learning
The work proposes a novel, instance-dependent experimental design objective for active preference learning to better align large language models with human preferences. It introduces an algorithm that yields an instance-dependent label complexity guarantee and a simple greedy algorithm. This approach is more efficient than classical methods by tailoring the sampling strategy to the specific difficulty of each problem instance.