Skip to content

Benchmark MiniMax M3.1 Flash through OpenRouter - #66

Draft
SkalskiP wants to merge 2 commits into
mainfrom
feat/minimax-m31-flash-benchmark
Draft

SkalskiP wants to merge 2 commits into
mainfrom
feat/minimax-m31-flash-benchmark

Conversation

@SkalskiP

@SkalskiP SkalskiP commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

MiniMax M3.1 Flash requires reasoning at both Low and High. The generic OpenRouter adapter previously disabled reasoning at Low. This change preserves the requested effort and pins MiniMax requests to the MiniMax provider with data_collection: deny and provider fallback disabled.

This draft starts from main (6623912) and targets main. A 50-image comparison of three detection coordinate conventions selected absolute original-image XYXY (mAP@50:95 32.19%, versus 28.75% for normalized YXYX and 26.23% for normalized XYXY). The full 38-job queue is running with this fixed format. The planned benchmark consists of 36 original-task runs plus two mixed-text-v2 runs. Probe outputs stay outside results/; the model will enter the registry and generated leaderboards only with complete results.

Current limitation: only the promotional :free route is available. The paid slug returns 404, and the current account has a 1,000-request daily free-model allowance versus 5,046 requests required by the full protocol. The shared scheduler starts at 32 parallel requests, grows toward 64, and reduces concurrency with a shared cooldown after 429. It reserves the available daily quota and checkpoints every success. The earlier four-worker runner drained before migration; its two complete OCR Low runs were preserved. There is no automatic endpoint switch.

Validation: 341 tests passed, including MiniMax routing and Low/High regression coverage. Ruff passes for the changed Python files. Local protocol validation and summary freshness checks pass, as do both GitHub CI jobs. Full inference and publication validation are pending. The full queue runs across configurations in one shared pool and drains active requests when the daily allowance is exhausted.

Sources and the fixed run settings are documented in docs/minimax-m31-flash.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant