哪些物镜需要转盘？预测客观冲突并涵盖可操纵多元联盟中的权衡

arXiv:2609.26929v1 Announce Type: new Abstract: People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone.因此，多元化联盟需要可引导的模型，能够以不同的方式平衡相互竞争的目标。多目标直接偏好优化（MODPO）通过使用客观权重来跨越连续的权衡来实现这一点。我们研究两个问题：一个模型何时可以同时改进两个目标，以及如何在不为每个目标训练单独模型的情况下实现许多权衡？ Across seven objective pairs from HelpSteer and UltraFeedback, two pre-training measurements predict whether objectives align or conflict for human-annotated data, but not for AI-annotated data, where response length and repetition confound reward-model scores. For broader trade-off coverage, selecting the nearest trained model and merging model parameters both help, but neither consistently matches direct training.这些发现为构建服务于不同偏好的可操纵模型提供了实用指导。