Understand one complete episode
Follow an observation through a policy, an executable action, an environment change and an independent verifier. Run a small package-preparation exercise on your laptop before adding a language model.
Premium handbook / October 2026
AgentGym Engineering Handbook
Build, train and evaluate interactive agents with reinforcement learning
An agent can describe the right answer and still fail to complete the task. Learn how to build environments where actions have consequences, train policies from those interactions and evaluate whether the resulting system can do useful work. Start with one episode on your laptop, then work toward a versioned, measurable and controlled agent-training platform.
Premium guide PDFs and companion files require an active paid Premium membership or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads. Keep the copies you download.
Already a member? Sign in to download →
41pages in five parts
22chapters from first action to deployment
12original technical diagrams
What is AgentGym?
AgentGym is a research framework for evaluating and training language-model agents in interactive environments. A policy reads an observation, chooses an action and receives new information after the environment changes. The recorded path is a trajectory; one attempt from reset to a stopping condition is an episode. Demonstration learning and reinforcement learning use those trajectories in different ways.
The original framework supplies environment clients, datasets and evaluation tools. This handbook explains how those pieces fit together and where an engineering team must add reset checks, resource controls, authorization and independent evaluation. Technical claims are linked to primary publications and pinned source files.
AgentGym vs AgentGym-RL vs AgentGym2
AgentGym: the original environment framework
The original work covers heterogeneous environments, AgentTraj demonstrations and the AgentEval suite. It supports studying how an agent acts across different tasks and how successful experience can become new training data. The earlier AgentEvol name and the ACL publication's AgentSTaR name are explained with their source versions.
AgentGym-RL: multi-turn policy training
AgentGym-RL uses a modified verl stack, Ray orchestration and vLLM rollouts for online learning. The book works through PPO, GRPO and ScalingInter-RL, including the token masks and interaction budgets that determine what the policy learns. Its procedures follow a pinned release, with differences between the paper and scripts made explicit.
AgentGym2: evaluation under less idealized conditions
The July 2026 benchmark adds tool discovery, noisy inputs and complete workflows in complex tool use, data analysis and deep search. The guide explains its role alongside controlled training environments and why its scores and repeated-attempt metrics should remain separate from earlier AgentGym evaluations.
Start with the original AgentGym paper, the AgentGym-RL paper and the AgentGym2 paper for the primary research.
Who this guide is for
AI engineers building tool-using agents, researchers setting up multi-turn RL experiments and platform engineers responsible for rollout services, evaluation and execution isolation. You should be comfortable reading Python and using a Linux shell. The opening chapters define the agent-learning vocabulary before introducing policy-gradient notation.
If you are new to the subject, begin with the public samples below and the CPU companion after downloading. For the broader GPU training platform, the RL post-training handbook provides a complementary public preview. Each linked resource displays its own access requirements.
What you will be able to do
Build a correct environment adapter
Inspect TextCraft's real endpoints and payloads, distinguish session IDs from task IDs, handle errors returned with HTTP 200 and test reset, cleanup and cross-session isolation.
Make the learning signal explicit
Read trajectory roles and token masks, establish demonstration and self-improvement baselines, calculate GRPO advantages and compare interaction, token and wall-clock limits.
Operate the training system
Size GPU and environment capacity separately, account for training state and rollout cache, keep policy versions aligned and diagnose stalled or misleading runs.
Evaluate what users experience
Keep infrastructure errors in the end-to-end denominator, report uncertainty and paired task outcomes, and distinguish the original AgentEval suite from AgentGym-RL and AgentGym2 protocols.
Qualify a production candidate
Choose execution boundaries, enforce tool permissions, test newer model compatibility and assemble the evidence needed for a limited deployment with a rollback plan.
Read three sample pages
These selected pages come from the complete Premium PDF. Open an image to inspect its layout, or expand the opening text below for an accessible HTML version.
Inside the guide
Read these samples as text →
Page 5: Follow one action through the environment · opening text
1. Follow one action through an environment
An agent says it has completed a task. The package is still on the shelf. This small mismatch explains why training an interactive agent needs more than a collection of convincing answers.
Consider a constructed task: prepare a ceramic mug for delivery. The environment contains a mug, a book and packing materials. The agent can inspect the shelf, take an item, pack it, verify the package and dispatch it. Each action changes what can happen next. The sentence “I shipped the mug” has no effect unless the environment accepts a dispatch action and records the resulting state.
The five objects to keep separate
A task states the desired outcome. An observation is the information the environment exposes now. An action is an instruction the environment can execute. A policy maps the available history to an action; here the policy is a language model. A reward is a numerical training signal derived from what happened.
The environment's state can include facts absent from the observation. A browser page might omit a database field; a simulator might hide an object's location. The agent therefore operates from its history, rather than from an omniscient view of the world.
The verifier reads the resulting state. A model's completion claim cannot substitute for that check. This is a conceptual loop; specific AgentGym environments expose different transport details.
Page 21: Understand policy updates and group-relative credit · opening text
12. Work through PPO, GRPO and group-relative credit
Online RL samples actions from a policy, observes their consequences and changes the probability of those actions. The useful learning signal is an advantage: an estimate of how an outcome compares with an appropriate baseline.
PPO limits the size of an update
For an action token, compare its probability under the policy being updated with its probability under the policy that generated the rollout. Their ratio is rho = new_probability / old_probability. PPO uses a clipped objective to discourage an update from moving that ratio too far in the rewarded direction.
Illustrative PPO token objective:
min(rho × A, clip(rho, 1-epsilon, 1+epsilon) × A)
A = +0.8, rho = 1.30, epsilon = 0.20
unclipped term = 1.30 × 0.8 = 1.04
clipped term = 1.20 × 0.8 = 0.96
selected term = 0.96
This is an objective term being maximized, not the whole training loss. KL penalties, entropy terms, masking and reduction conventions also matter. PPO commonly uses a learned value function to estimate returns; that critic adds training state and another prediction to debug. See the PPO paper for the full formulation.
Page 30: Keep evaluation denominators honest · opening text
17. Evaluate the policy and the service separately
A test schedules 100 tasks. Sixty succeed, thirty fail and ten never receive a valid result because the environment service crashes. Reporting only the ninety scored tasks makes the system look more reliable than the user experienced.
Keep both denominators
Constructed evaluation:
Scheduled logical episodes = 100
Verified successes = 60
Policy/task failures = 30
Infrastructure errors = 10
End-to-end success = 60 / 100 = 60.0%
Conditional success among scored episodes = 60 / 90 = 66.7%
Infrastructure error share = 10 / 100 = 10.0%
The same sixty successes support different metrics. Both denominators are useful when named; dropping the ten missing outcomes without disclosure is misleading.
The companion checks this accounting and rejects duplicate terminal records for the same logical episode. A retry is an additional attempt, not a new user task. Record attempts separately and evaluate success under the retry policy you actually allow.
Inside the handbook
The PDF includes a publication notice, reading routes, linked contents, page numbers and source references. Chapter page numbers below come from the current built edition.
Part I · Understand the interaction
- Follow one action through an environment · page 5
- Know which AgentGym you are using · page 6
- Separate the policy, environment and learner · page 7
- Choose data without contaminating evaluation · page 9
- Read a trajectory and its token masks · page 10
Part II · Build a trustworthy first run
- Run an episode on your laptop · page 12
- Pin the research stack before installing it · page 13
- Verify the TextCraft server contract · page 14
- Add an environment without hiding its failure modes · page 16
Part III · Make learning signals trustworthy
- Establish demonstrations and self-improvement baselines · page 18
- Design rewards that survive optimization · page 19
- Work through PPO, GRPO and group-relative credit · page 21
- Configure ScalingInter-RL and the first training run · page 22
Part IV · Scale and operate the experiment
- Size rollout capacity from the slowest stage · page 25
- Budget memory and keep policy versions aligned · page 26
- Diagnose stalled and misleading training runs · page 28
- Evaluate the policy and the service separately · page 30
- Isolate execution and authorize actions · page 31
Part V · Qualify a real agent system
- Use AgentGym2 to test less idealized work · page 34
- Qualify a newer model without breaking the experiment · page 35
- Build a promotion-ready experiment · page 36
- Keep a reference you can audit · page 38
A practical AgentGym tutorial with a tested CPU companion
The companion contains an original stateful package-preparation environment, trajectory validation, a token-mask exercise, group-relative advantage calculations, an explicit horizon schedule, evaluation accounting and a rollout-capacity calculator. Its 31 tests include incorrect actions, stale verification, reset behavior, duplicate evaluation records and an HTTP error returned with a successful status.
A separate TextCraft probe checks the pinned server contract, while a source-reviewed training template shows how to inspect the resolved configuration before allocating GPU time. The local lab needs Python 3.10 or newer and no packages, model download or API key. It uses scripted actions to make the mechanisms inspectable; it does not train a language model.
What to measure before calling an agent production-ready
Count verified successes against all scheduled logical tasks, retain infrastructure failures and show any conditional success metric separately. Measure invalid actions, truncated episodes, environment latency and cost per success. Keep task IDs, model and tokenizer revisions, reward definitions and stopping rules with every result.
The guide then connects those measurements to execution controls: a private environment endpoint, bounded sessions, tool authorization outside the model, appropriate process isolation and a checkpoint promotion decision based on independent evaluation. A benchmark improvement does not grant the policy broader credentials or tools.
Evidence and limits of this edition
Verified for the publication
Primary papers and official source revisions; TextCraft request fields and error handling; the released RL dependency constraints, message masks and scheduler behavior; the original CPU companion, its synthetic HTTP test and the worked calculations. The book identifies discrepancies in the research instead of hiding them.
Qualify on your equipment
No GPU training, upstream model benchmark or production deployment was performed for this edition. Installation and training commands are procedures to qualify. A newer model needs compatible architecture support, tokenization, stopping behavior, masks and likelihood checks before its rollouts can be trusted for learning.
October 2026 edition. Prepared with AI assistance and reviewed against primary sources. The Gemini/Nano Banana cover is conceptual artwork; the technical diagrams and companion are original. No private system screenshots or upstream datasets are bundled.
Primary sources and related learning
The PDF contains 39 distinct primary-source links. Useful starting points include the pinned AgentGym-RL source, AgentGym environments and tutorials, AgentGym2 evaluation code and the RL task-ID dataset and license card.
Continue with RL Post-Training Infrastructure, Evaluation Metrics and Data Splits and Evaluation and Data Pipeline Infrastructure. For serving operations and model selection, read the public previews of LLM Inference in Production and the Open Model Field Guide.
Frequently asked questions
What is AgentGym?
AgentGym is a research framework for evaluating and training language-model agents across interactive environments. An agent reads an observation, chooses an action and receives a new observation and feedback. The original framework connects heterogeneous environments through clients and servers, with demonstration trajectories and an evaluation suite.
How are AgentGym, AgentGym-RL and AgentGym2 different?
AgentGym supplies the original environment framework, datasets and learning examples. AgentGym-RL extends that work with multi-turn reinforcement learning using a modified verl stack and a curriculum called ScalingInter-RL. AgentGym2 is a newer evaluation framework for tool discovery, data analysis and deep search under less idealized conditions. Their task sets, APIs and scores are not interchangeable.
Does AgentGym-RL train a language model from scratch?
Its experiments start with existing language-model backbones. Training without an agent-specific supervised warm-up does not mean training randomly initialized model weights. The guide keeps pretraining, demonstration fine-tuning and online reinforcement learning separate.
What is ScalingInter-RL?
ScalingInter-RL progressively increases the allowed number of agent-environment interactions during training. A short early horizon limits exploration; later stages allow longer trajectories. Turn limits, per-response tokens and total context tokens remain separate budgets. The released scheduler also needs care at checkpoint-resume boundaries.
Can I run the companion without a GPU?
Yes. The Python standard-library companion runs a small original environment, validates trajectory records, calculates group-relative advantages, demonstrates token masks and analyzes evaluation denominators and rollout capacity. It includes tests and synthetic fixtures. It does not train a language model or reproduce the papers' benchmark scores.
Does the guide include a working AgentGym-RL training result?
No GPU training run was performed for this publication. The walkthrough is based on pinned upstream source, with configuration and protocol checks. The CPU companion was executed. Installation, model loading, GPU memory fit and learning convergence still need qualification on the reader's hardware.
Can I replace Qwen2.5 with a newer model?
Treat that as a stack migration. The released AgentGym-RL recipe uses older engine dependencies and Qwen-specific message assembly. A newer checkpoint needs supported architecture code, matching tokenization, correct action and observation masks, verified stop conditions and train-versus-rollout log-probability checks before a learning run.
Is AgentGym an enterprise sandbox?
No. Separate environment processes help manage dependencies and parallel execution. Production use also needs isolation, authentication, action authorization, resource limits, reset verification and controlled checkpoint promotion. A sandbox does not prevent an authorized tool from making an unwanted business change.
Is the report free?
The public guide page, contents and three selected samples are free to read. The complete PDF and companion ZIP require active paid Premium membership or approved complimentary download access. Referral Premium does not include these guide downloads.
Get the complete AgentGym handbook
Download the illustrated PDF and companion. Follow one action through the environment, inspect how the learning signal is constructed and build an experiment another engineer can evaluate and reproduce.
Premium guide PDFs and companion files require an active paid Premium membership or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads. See the Terms for the personal, non-transferable license. Share this public page; please do not redistribute the complete PDF. Report a correction with the edition and page number.