What is Navital Gym?
Navital Gym is an open benchmark of sandboxed tasks for testing AI agents on Indian workflows. Its environments cover trip planning, UPI payments, GST calculations, form filling, multi-document checks, and customer support in Hindi and Hinglish, with an experimental split for other Indian languages. Teams can run public tasks against their own agents, inspect the results, or arrange a private evaluation on held-out tasks.
Navital Gym Features:
Indian Task Worlds: The suite includes tasks for rail quotas and trip constraints, payment attempts and disputes, GST invoices, identity forms, document conflicts, and customer-support requests.
Rule-Based Grading: Each task is checked by deterministic written rules that recompute results from the sandbox, rather than by an AI judge or the agent’s own claims.
Sandbox Tools: Agents receive a task prompt and tool specifications, then interact with a simulated world whose hidden expected answers and designer notes are withheld.
Held-Out Tasks: Private evaluations use unpublished tasks, while the public benchmark can be run and checked independently.
Local Model Runs: The Python package and command-line runner support agents using the OpenAI chat API with tool calling, including local or hosted models.
Evaluation Adapters: Optional adapters connect environments to Inspect AI, verifiers, and OpenEnv, and reports include pass rates and per-rule failures.
Navital Gym Benefits:
Clearer Comparisons: Consistent grading across the same tasks makes it easier to compare agent versions without relying on subjective judgments of their responses.
Actionable Debugging: Per-rule failure reporting can help teams identify which parts of a workflow an agent handles poorly and where to focus improvements.
Reproducible Checks: Public tasks and documented graders let teams repeat evaluations and verify how a score was produced.
Safer Test Data: Synthetic or public-source data lets teams exercise these workflows without using customer records.
More Relevant Testing: Indian-specific scenarios expose weaknesses that evaluations based on other regions’ workflows may not reveal.
Use Cases:
Agent Release Checks: Run the benchmark before and after changing a model or agent to see whether its performance shifts across Indian task types.
Payment Workflow Tests: Evaluate whether an agent handles payment attempts, limits, disputes, and suspicious requests appropriately in the UPI environment.
Finance Task Evaluation: Test an agent’s invoice handling and return computations against the benchmark’s simplified GST rules.
Travel Agent Testing: Check whether a trip-planning agent can account for budgets, rail quotas, waitlists, closing days, and food requirements.
Support Language Checks: Assess an e-commerce support agent’s responses to returns, refunds, and privacy requests in Hindi or Hinglish.

