I love the new Eval benchmarks idea and think you pick one that is less saturated to expose quantization gaps.
AIME 2024/2025 is less saturated and requires more thinking (but still no sandbox) and exposes more quantization gaps in my experiments than gsm8k or MMLU. It will take slightly longer to run but worth it!
中文说明
建议选择 AIME 2024/2025 作为质量评估基准测试。与 GSM8K 或 MMLU 相比，AIME 不那么饱和，需要更深入的推理（但无需沙盒），能更好地暴露量化精度差异。运行时间稍长但值得。
I love the new Eval benchmarks idea and think you pick one that is less saturated to expose quantization gaps.
AIME 2024/2025 is less saturated and requires more thinking (but still no sandbox) and exposes more quantization gaps in my experiments than gsm8k or MMLU. It will take slightly longer to run but worth it!
中文说明
建议选择 AIME 2024/2025 作为质量评估基准测试。与 GSM8K 或 MMLU 相比，AIME 不那么饱和，需要更深入的推理（但无需沙盒），能更好地暴露量化精度差异。运行时间稍长但值得。