CHROME is an OpenEnv-compliant Reinforcement Learning (RL) environment and evaluation benchmark built to test how well AI agents can plan under real-world constraints.
In CHROME, an AI agent acts as a hiring manager tasked with building
The controlled Core track evaluates LLMs on a standardized, deterministic testbed. This benchmark measures multi-step planning, budget control, and how agents react to sudden market changes.
| Model | Avg | Easy | Medium | Hard | Cost | Trace |
|---|---|---|---|---|---|---|
| Heuristic Greedy baseline | 0.692 | 0.843 | 0.689 | 0.543 | $0.0000 | rollout_demo.py |
google/gemini-3-flash-preview |
0.671 | 0.817 | 0.665 | 0.531 | $0.5854 | trace |
google/gemini-2.5-flash |
0.664 | 0.799 | 0.688 | 0.506 | $0.2723 | trace |
google/gemini-3.1-flash-lite |
0.634 | 0.788 | 0.636 | 0.477 | $0.2016 | trace |
meta-llama/llama-3.3-70b-instruct |
0.613 | 0.841 | 0.603 | 0.395 | $0.5044 | trace |
minimax/minimax-m3 |
0.534 | 0.655 | 0.618 | 0.329 | $1.4064 | trace |
google/gemini-3.5-flash |
FAIL | 0.772 | 0.746 | FAIL | $1.8079 | trace |
All core evals use standardized environment settings.
Standard hiring baselines or greedy models degrade quickly in CHROME because of three distinct environment mechanics:
The salary market responds to the agent's choices. Hiring a candidate from a specific skill bucket depletes the supply, driving up the minimum wage requirement for all remaining candidates in that bucket (and adjacent buckets in Hard mode):
- Base Salary: Base minimum salary for a candidate's skill tier (ranging from ₹3L to ₹18L).
- Hires: Total candidates hired out of that specific skill tier.
- Adjacent Hires: Hires made in neighboring skill tiers (reflecting cross-industry wage inflation).
Because hiring someone makes everyone else in their field more expensive, early high-value hires compress the remaining budget and inflate future candidate costs. Agents must balance spending early vs. saving for later.
You cannot evaluate candidates in isolation. A hire's revenue contribution depends on who is already on the team (Chemistry) and experiences natural diminishing returns as the team grows:
- Team Size: Headcount currently hired to the team (enforcing diminishing returns via the square root).
- Intel: The candidate's capability score (0–100).
- Chemistry: Evaluated by comparing the hired roles to the ideal team mix:
Where Actual% is the proportion of Junior, Mid, Senior, or Lead engineers currently hired, and Ideal% is the target role mix. The value of adding a Senior engineer shifts depending on whether the team already has enough Juniors or Leads.
In Hard Mode, the environment triggers competitor "poaching" campaigns at fixed steps (e.g., step 20, 40, 60...). These shocks remove high-scoring candidates directly from the pool, driving up scarcity costs without the agent's intervention.
Agents that do not maintain budget buffers or fail to secure critical roles before these milestones run out of resources before filling their teams.
CHROME runs natively on the Model Context Protocol (MCP), exposing environment controls as structured tools.
| MCP Tool | Parameters | Step Cost | Description |
|---|---|---|---|
hire_candidate |
candidate_id (int)team_name (str)offered_salary (float) |
1 | Hires the candidate for the team. Salary must meet the current market minimum. |
get_team_summary |
— | 0 | View headcount, roles, revenue projections, and available candidates. |
get_market_ledger |
— | 0 | Check the live salary floors for all skill tiers. |
Note: Free observation tools allow agents to execute arbitrary planning, reasoning, or lookahead steps before committing to a permanent state change.
At the end of an episode, the agent is scored on a scale of [0.0, 1.0] based on a balance of three goals:
- Constraints (30%): Percentage of teams fully staffed (headcount target met) with an average team intelligence score above the required threshold.
- Revenue Ratio (60%): Revenue generated compared to the Aspirational Oracle Revenue (a theoretical ceiling calculated at static prices).
-
Spend Efficiency (10%): Incentivizes staffing the teams without overbidding:
$$\text{Spend Efficiency} = \text{Hiring Completeness} \times \text{Average } \left( \frac{\text{Market Min}}{\text{Paid Salary}} \right)$$
| Dimension | Easy (0) | Medium (1) | Hard (2) |
|---|---|---|---|
| Teams ( |
5 | 12 | 20 |
| Candidates ( |
100 | 250 | 500 |
| Budget | ₹130L | ₹420L | ₹740L |
| Max Steps | 40 | 100 | 150 |
| Dynamic Scarcity | Static | Active (0.08) | Active (0.08) |
| Coupled Scarcity | Disabled | Disabled | Active (0.03) |
| Team Chemistry | Disabled | Enabled | Enabled |
| Market Shocks | None | None | 6 Deterministic Shocks |
Clone the repository and install dependencies:
git clone https://github.com/rushisyeole/CHROME.git
cd CHROME
pip install -e ".[dev]"Start the FastAPI server, which exposes the MCP tools over HTTP:
uvicorn hr.server.app:app --host 0.0.0.0 --port 8000 --reloadpytest tests/ -vCompare a simple heuristic greedy policy against a random policy over 5 seeds:
python -m hr.examples.rollout_demoTo evaluate an LLM agent, run the inference runner. Ensure your OPENROUTER_API_KEY is exported:
pip install -e ".[inference]"
# Run a single evaluation run (e.g. Gemini 3.5 Flash)
OPENROUTER_API_KEY=sk-or-... MODEL_NAME=google/gemini-3.5-flash python -m hr.inference
# Run the complete CHROME-Core benchmark suite
OPENROUTER_API_KEY=sk-or-... python -m hr.benchmark