Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 0 additions & 74 deletions results/counting_gemini-4-argon-eap_high_20261012_230821.jsonl

This file was deleted.

74 changes: 0 additions & 74 deletions results/counting_gemini-4-argon-eap_high_20261012_230913.jsonl

This file was deleted.

74 changes: 0 additions & 74 deletions results/counting_gemini-4-argon-eap_high_20261013_064713.jsonl

This file was deleted.

74 changes: 0 additions & 74 deletions results/counting_gemini-4-argon-eap_low_20261012_223834.jsonl

This file was deleted.

74 changes: 0 additions & 74 deletions results/counting_gemini-4-argon-eap_low_20261012_223838.jsonl

This file was deleted.

74 changes: 0 additions & 74 deletions results/counting_gemini-4-argon-eap_low_20261012_223842.jsonl

This file was deleted.

74 changes: 74 additions & 0 deletions results/counting_gemini-4-argon_high_20261012_230821.jsonl

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions results/counting_gemini-4-argon_high_20261012_230913.jsonl

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions results/counting_gemini-4-argon_high_20261013_064713.jsonl

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions results/counting_gemini-4-argon_low_20261012_223834.jsonl

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions results/counting_gemini-4-argon_low_20261012_223838.jsonl

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions results/counting_gemini-4-argon_low_20261012_223842.jsonl

Large diffs are not rendered by default.

250 changes: 0 additions & 250 deletions results/detection_gemini-4-argon-eap_high_20261013_070134.jsonl

This file was deleted.

250 changes: 0 additions & 250 deletions results/detection_gemini-4-argon-eap_high_20261013_074327.jsonl

This file was deleted.

250 changes: 0 additions & 250 deletions results/detection_gemini-4-argon-eap_high_20261013_074327_2.jsonl

This file was deleted.

250 changes: 0 additions & 250 deletions results/detection_gemini-4-argon-eap_low_20261013_070835.jsonl

This file was deleted.

250 changes: 0 additions & 250 deletions results/detection_gemini-4-argon-eap_low_20261013_071022.jsonl

This file was deleted.

250 changes: 0 additions & 250 deletions results/detection_gemini-4-argon-eap_low_20261013_071120.jsonl

This file was deleted.

250 changes: 250 additions & 0 deletions results/detection_gemini-4-argon_high_20261013_070134.jsonl

Large diffs are not rendered by default.

250 changes: 250 additions & 0 deletions results/detection_gemini-4-argon_high_20261013_074327.jsonl

Large diffs are not rendered by default.

250 changes: 250 additions & 0 deletions results/detection_gemini-4-argon_high_20261013_074327_2.jsonl

Large diffs are not rendered by default.

250 changes: 250 additions & 0 deletions results/detection_gemini-4-argon_low_20261013_070835.jsonl

Large diffs are not rendered by default.

250 changes: 250 additions & 0 deletions results/detection_gemini-4-argon_low_20261013_071022.jsonl

Large diffs are not rendered by default.

250 changes: 250 additions & 0 deletions results/detection_gemini-4-argon_low_20261013_071120.jsonl

Large diffs are not rendered by default.

97 changes: 0 additions & 97 deletions results/extraction_gemini-4-argon-eap_high_20261013_071307.jsonl

This file was deleted.

97 changes: 0 additions & 97 deletions results/extraction_gemini-4-argon-eap_high_20261013_071359.jsonl

This file was deleted.

97 changes: 0 additions & 97 deletions results/extraction_gemini-4-argon-eap_high_20261013_071605.jsonl

This file was deleted.

97 changes: 0 additions & 97 deletions results/extraction_gemini-4-argon-eap_low_20261012_223722.jsonl

This file was deleted.

97 changes: 0 additions & 97 deletions results/extraction_gemini-4-argon-eap_low_20261012_223724.jsonl

This file was deleted.

97 changes: 0 additions & 97 deletions results/extraction_gemini-4-argon-eap_low_20261012_223726.jsonl

This file was deleted.

97 changes: 97 additions & 0 deletions results/extraction_gemini-4-argon_high_20261013_071307.jsonl

Large diffs are not rendered by default.

97 changes: 97 additions & 0 deletions results/extraction_gemini-4-argon_high_20261013_071359.jsonl

Large diffs are not rendered by default.

97 changes: 97 additions & 0 deletions results/extraction_gemini-4-argon_high_20261013_071605.jsonl

Large diffs are not rendered by default.

97 changes: 97 additions & 0 deletions results/extraction_gemini-4-argon_low_20261012_223722.jsonl

Large diffs are not rendered by default.

97 changes: 97 additions & 0 deletions results/extraction_gemini-4-argon_low_20261012_223724.jsonl

Large diffs are not rendered by default.

97 changes: 97 additions & 0 deletions results/extraction_gemini-4-argon_low_20261012_223726.jsonl

Large diffs are not rendered by default.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

32 changes: 32 additions & 0 deletions results/identification_gemini-4-argon_high_20261012_230959.jsonl

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions results/identification_gemini-4-argon_high_20261012_231035.jsonl

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions results/identification_gemini-4-argon_high_20261012_231059.jsonl

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions results/identification_gemini-4-argon_low_20261012_224142.jsonl

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions results/identification_gemini-4-argon_low_20261012_224146.jsonl

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions results/identification_gemini-4-argon_low_20261012_224148.jsonl

Large diffs are not rendered by default.

37 changes: 0 additions & 37 deletions results/ocr_gemini-4-argon-eap_high_20261012_225050.jsonl

This file was deleted.

37 changes: 0 additions & 37 deletions results/ocr_gemini-4-argon-eap_high_20261012_225128.jsonl

This file was deleted.

37 changes: 0 additions & 37 deletions results/ocr_gemini-4-argon-eap_high_20261012_225132.jsonl

This file was deleted.

37 changes: 0 additions & 37 deletions results/ocr_gemini-4-argon-eap_low_20261012_223715.jsonl

This file was deleted.

37 changes: 0 additions & 37 deletions results/ocr_gemini-4-argon-eap_low_20261012_223717.jsonl

This file was deleted.

37 changes: 0 additions & 37 deletions results/ocr_gemini-4-argon-eap_low_20261012_223719.jsonl

This file was deleted.

37 changes: 37 additions & 0 deletions results/ocr_gemini-4-argon_high_20261012_225050.jsonl

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions results/ocr_gemini-4-argon_high_20261012_225128.jsonl

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions results/ocr_gemini-4-argon_high_20261012_225132.jsonl

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions results/ocr_gemini-4-argon_low_20261012_223715.jsonl

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions results/ocr_gemini-4-argon_low_20261012_223717.jsonl

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions results/ocr_gemini-4-argon_low_20261012_223719.jsonl

Large diffs are not rendered by default.

151 changes: 0 additions & 151 deletions results/reasoning_gemini-4-argon-eap_high_20261013_071611.jsonl

This file was deleted.

151 changes: 0 additions & 151 deletions results/reasoning_gemini-4-argon-eap_high_20261013_071812.jsonl

This file was deleted.

151 changes: 0 additions & 151 deletions results/reasoning_gemini-4-argon-eap_high_20261013_072018.jsonl

This file was deleted.

151 changes: 0 additions & 151 deletions results/reasoning_gemini-4-argon-eap_low_20261012_224150.jsonl

This file was deleted.

151 changes: 0 additions & 151 deletions results/reasoning_gemini-4-argon-eap_low_20261012_224152.jsonl

This file was deleted.

151 changes: 0 additions & 151 deletions results/reasoning_gemini-4-argon-eap_low_20261012_224154.jsonl

This file was deleted.

151 changes: 151 additions & 0 deletions results/reasoning_gemini-4-argon_high_20261013_071611.jsonl

Large diffs are not rendered by default.

151 changes: 151 additions & 0 deletions results/reasoning_gemini-4-argon_high_20261013_071812.jsonl

Large diffs are not rendered by default.

151 changes: 151 additions & 0 deletions results/reasoning_gemini-4-argon_high_20261013_072018.jsonl

Large diffs are not rendered by default.

151 changes: 151 additions & 0 deletions results/reasoning_gemini-4-argon_low_20261012_224150.jsonl

Large diffs are not rendered by default.

151 changes: 151 additions & 0 deletions results/reasoning_gemini-4-argon_low_20261012_224152.jsonl

Large diffs are not rendered by default.

151 changes: 151 additions & 0 deletions results/reasoning_gemini-4-argon_low_20261012_224154.jsonl

Large diffs are not rendered by default.

600 changes: 0 additions & 600 deletions results/text_gemini-4-argon-eap_high_20261013_072221.jsonl

This file was deleted.

600 changes: 0 additions & 600 deletions results/text_gemini-4-argon-eap_low_20261012_232137.jsonl

This file was deleted.

600 changes: 600 additions & 0 deletions results/text_gemini-4-argon_high_20261013_072221.jsonl

Large diffs are not rendered by default.

600 changes: 600 additions & 0 deletions results/text_gemini-4-argon_low_20261012_232137.jsonl

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions src/vlm_exam/configs/models.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -507,12 +507,13 @@ models:
input_per_million_tokens: 0.68
output_per_million_tokens: 2.09

gemini-4-argon-eap:
gemini-4-argon:
name: Gemini 4 Argon
lab: google
provider: google
detection_coordinate_format: yxyx_normalized_0_to_1000
# Launch introductory pricing; saved evaluations used the EAP endpoint.
# Launch introductory pricing. The committed runs (2026-10-12/13) used the
# early-access endpoint under the key gemini-4-argon-eap (renamed 2026-10-13).
pricing:
input_per_million_tokens: 2.0
output_per_million_tokens: 10.0
6 changes: 3 additions & 3 deletions src/vlm_exam/configs/text_release.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"gemini-3.6-flash",
"gemini-3.7-flash",
"gemini-3.8-flash",
"gemini-4-argon-eap",
"gemini-4-argon",
"glm-5.3-flash",
"glm-5v-turbo",
"gpt-5.4-mini",
Expand Down Expand Up @@ -235,12 +235,12 @@
"coordinate_format": "yxyx_normalized_0_to_1000",
"allowed_failures": []
},
"gemini-4-argon-eap/high": {
"gemini-4-argon/high": {
"inference_hash": "597307dfa2a95aa7584ef557fceec459e0ae8ea20be2760430675a120f7a2391",
"coordinate_format": "yxyx_normalized_0_to_1000",
"allowed_failures": []
},
"gemini-4-argon-eap/low": {
"gemini-4-argon/low": {
"inference_hash": "597307dfa2a95aa7584ef557fceec459e0ae8ea20be2760430675a120f7a2391",
"coordinate_format": "yxyx_normalized_0_to_1000",
"allowed_failures": []
Expand Down
2 changes: 1 addition & 1 deletion tests/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ def test_bundled_config_marks_only_pre_protocol_models_legacy(self) -> None:
"mimo-v2.6-pro",
"mimo-v2.6-flash",
"mistral-large-4",
"gemini-4-argon-eap",
"gemini-4-argon",
]


Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 26 additions & 26 deletions visualizations/leaderboards/text_manifest.json
Original file line number Diff line number Diff line change
@@ -1,71 +1,71 @@
{
"schema_version": 1,
"summary_sha256": "09b1e7eab1b214fcc90c27394dec9d43ee009acf8ffaf5a5f449bd0e71be53de",
"display_sha256": "66c6f05ec89e72a9423472ece24d531d2ad55594f806dd7e1d44d452c56fd095",
"summary_sha256": "b483f9f4d6090c2ede1c6f507f010b7350858b736349d05a55301bb0b51757f4",
"display_sha256": "fe625077e28c036c2b5bb956109cb94928e471d25fe562dcb632e9288583bbac",
"renderer_sha256": "52c877b6a33dffb63421a67f5cd37e6620e2093cfc13db9e77ceb5a1e2fac38e",
"charts": [
{
"file": "text_localization_recognition_high.png",
"sha256": "090d08e22084cb25463ea52640010a6fdf52ccf20b1973c1e3baee757a9d6411",
"sha256": "68327f7ea0d1bf605f74e9ce25f40ebbf29db127034a2f9cfd55e9abde1a57aa",
"size": [
2100,
11940
1575,
9023
]
},
{
"file": "text_localization_recognition_low.png",
"sha256": "a103bb5619c4f659220fa017257835f9103bab0a7903f422c807d892f0e74ed7",
"sha256": "8a3ad43dfcccb6474468965c3168577793b9b99e7f0ada0c1d9d32315d2a1b22",
"size": [
2100,
12420
1575,
9383
]
},
{
"file": "text_single_string_high.png",
"sha256": "9450e8602ce18774d4bc76cbf0e72b82ff2bc3e328393e7f060636c581b70b53",
"sha256": "2bb26110b3e358b3953b62867e1a5c45e0666ae454f34eb504d46a308191398c",
"size": [
2100,
11940
1575,
9023
]
},
{
"file": "text_single_string_low.png",
"sha256": "9c8404f41084e6065d6da729d6f0d17c0a564d5360807f5599e9a1343e4156a0",
"sha256": "46dd6b3b71124cc30b0e7f2bcd6bf0964835e0d03d94f3f1089431176ca1c9fe",
"size": [
2100,
12420
1575,
9383
]
},
{
"file": "text_structured_high.png",
"sha256": "21c196cc0aa54d17e37573055eb22ce3bafb1fe6a514da230f31e09da682565f",
"sha256": "89ad42bceec91a43ca2e92ac27cdcec721cb95dd55d151e6009f78be4e6e5b0c",
"size": [
2100,
11940
1575,
9023
]
},
{
"file": "text_structured_low.png",
"sha256": "d5ba92596cd35dd1c8bb0eb8049e681bd8103121167de8991c6abc64a0d9b1c3",
"sha256": "e6f7bda6f8cb271ecbbf70e34416996b75fe73a4730a8ebc74a5db8b77b9bb39",
"size": [
2100,
12420
1575,
9383
]
},
{
"file": "text_transcription_high.png",
"sha256": "65acd76b4d296b3f5aea8f94d07502c2f5d0c9818df30ef13cb7178e3a3761fc",
"sha256": "e386b8c24b6c6ae66e24eb9712b2f2441b45629e5ed9b132bb7feb63293225a8",
"size": [
2100,
11940
1575,
9023
]
},
{
"file": "text_transcription_low.png",
"sha256": "c60fd07c1b6ad8c5afffe0c98c8b51dee6f5b661e75f35f643b09e584a451d4d",
"sha256": "be185410ccded6e2899df08ba30c9b014b0f3f2893a3da5f34aa7b1b54645cc8",
"size": [
2100,
12420
1575,
9383
]
}
]
Expand Down
Binary file modified visualizations/leaderboards/text_single_string_high.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified visualizations/leaderboards/text_single_string_low.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified visualizations/leaderboards/text_structured_high.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified visualizations/leaderboards/text_structured_low.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified visualizations/leaderboards/text_transcription_high.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified visualizations/leaderboards/text_transcription_low.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions web/benchmark_summary.json
Original file line number Diff line number Diff line change
Expand Up @@ -35870,8 +35870,8 @@
}
},
{
"id": "gemini-4-argon-eap:low",
"key": "gemini-4-argon-eap",
"id": "gemini-4-argon:low",
"key": "gemini-4-argon",
"name": "Gemini 4 Argon",
"lab": "google",
"effort": "low",
Expand Down Expand Up @@ -36290,8 +36290,8 @@
}
},
{
"id": "gemini-4-argon-eap:high",
"key": "gemini-4-argon-eap",
"id": "gemini-4-argon:high",
"key": "gemini-4-argon",
"name": "Gemini 4 Argon",
"lab": "google",
"effort": "high",
Expand Down
Loading