TRPO

Trust Region Policy Optimization (TRPO) is an iterative approach for optimizing policies with guaranteed monotonic improvement.

Available Policies

MlpPolicy

alias of ActorCriticPolicy

CnnPolicy

alias of ActorCriticCnnPolicy

MultiInputPolicy

alias of MultiInputActorCriticPolicy

Notes

Can I use?

  • Recurrent policies: ❌

  • Multi processing: ✔️

  • Gym spaces:

Space

Action

Observation

Discrete

✔️

✔️

Box

✔️

✔️

MultiDiscrete

✔️

✔️

MultiBinary

✔️

✔️

Dict

❌

✔️

Example

import gymnasium as gym
import numpy as np

from sb3_contrib import TRPO

env = gym.make("Pendulum-v1", render_mode="human")

model = TRPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000, log_interval=4)
model.save("trpo_pendulum")

del model # remove to demonstrate saving and loading

model = TRPO.load("trpo_pendulum")

obs, _ = env.reset()
while True:
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    env.render()
    if terminated or truncated:
      obs, _ = env.reset()

Results

Result on the MuJoCo benchmark (1M steps on -v3 envs with MuJoCo v2.1.0) using 3 seeds. The complete learning curves are available in the associated PR.

Environments

TRPO

HalfCheetah

1803 +/- 46

Ant

3554 +/- 591

Hopper

3372 +/- 215

Walker2d

4502 +/- 234

Swimmer

359 +/- 2

How to replicate the results?

Clone RL-Zoo and checkout the branch feat/trpo:

git clone https://github.com/cyprienc/rl-baselines3-zoo
cd rl-baselines3-zoo/

Run the benchmark (replace $ENV_ID by the envs mentioned above):

python train.py --algo tqc --env $ENV_ID --n-eval-envs 10 --eval-episodes 20 --eval-freq 50000

Plot the results:

python scripts/all_plots.py -a trpo -e HalfCheetah Ant Hopper Walker2d Swimmer -f logs/ -o logs/trpo_results
python scripts/plot_from_file.py -i logs/trpo_results.pkl -latex -l TRPO

Parameters

class sb3_contrib.trpo.TRPO(policy, env, learning_rate=0.001, n_steps=2048, batch_size=128, gamma=0.99, cg_max_steps=15, cg_damping=0.1, line_search_shrinking_factor=0.8, line_search_max_iter=10, n_critic_updates=10, gae_lambda=0.95, use_sde=False, sde_sample_freq=-1, normalize_advantage=True, target_kl=0.01, sub_sampling_factor=1, stats_window_size=100, tensorboard_log=None, policy_kwargs=None, verbose=0, seed=None, device='auto', _init_setup_model=True)[source]

Trust Region Policy Optimization (TRPO)

Paper: https://arxiv.org/abs/1502.05477 Code: This implementation borrows code from OpenAI Spinning Up (https://github.com/openai/spinningup/) and Stable Baselines (TRPO from https://github.com/hill-a/stable-baselines)

Introduction to TRPO: https://spinningup.openai.com/en/latest/algorithms/trpo.html

Parameters:

  • policy (Union[str, Type[ActorCriticPolicy]]) – The policy model to use (MlpPolicy, CnnPolicy, …)

  • env (Union[Env, VecEnv, str]) – The environment to learn from (if registered in Gym, can be str)

  • learning_rate (Union[float, Callable[[float], float]]) – The learning rate for the value function, it can be a function of the current progress remaining (from 1 to 0)

  • n_steps (int) – The number of steps to run for each environment per update (i.e. rollout buffer size is n_steps * n_envs where n_envs is number of environment copies running in parallel) NOTE: n_steps * n_envs must be greater than 1 (because of the advantage normalization) See https://github.com/pytorch/pytorch/issues/29372

  • batch_size (int) – Minibatch size for the value function

  • gamma (float) – Discount factor

  • cg_max_steps (int) – maximum number of steps in the Conjugate Gradient algorithm for computing the Hessian vector product

  • cg_damping (float) – damping in the Hessian vector product computation

  • line_search_shrinking_factor (float) – step-size reduction factor for the line-search (i.e., theta_new = theta + alpha^i * step)

  • line_search_max_iter (int) – maximum number of iteration for the backtracking line-search

  • n_critic_updates (int) – number of critic updates per policy update

  • gae_lambda (float) – Factor for trade-off of bias vs variance for Generalized Advantage Estimator

  • use_sde (bool) – Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)

  • sde_sample_freq (int) – Sample a new noise matrix every n steps when using gSDE Default: -1 (only sample at the beginning of the rollout)

  • normalize_advantage (bool) – Whether to normalize or not the advantage

  • target_kl (float) – Target Kullback-Leibler divergence between updates. Should be small for stability. Values like 0.01, 0.05.

  • sub_sampling_factor (int) – Sub-sample the batch to make computation faster see p40-42 of John Schulman thesis http://joschu.net/docs/thesis.pdf

  • stats_window_size (int) – Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over

  • tensorboard_log (Optional[str]) – the log location for tensorboard (if None, no logging)

  • policy_kwargs (Optional[Dict[str, Any]]) – additional arguments to be passed to the policy on creation

  • verbose (int) – the verbosity level: 0 no output, 1 info, 2 debug

  • seed (Optional[int]) – Seed for the pseudo random generators

  • device (Union[device, str]) – Device (cpu, cuda, …) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

  • _init_setup_model (bool) – Whether or not to build the network at the creation of the instance

collect_rollouts(env, callback, rollout_buffer, n_rollout_steps)

Collect experiences using the current policy and fill a RolloutBuffer. The term rollout here refers to the model-free notion and should not be used with the concept of rollout used in model-based RL or planning.

Parameters:

  • env (VecEnv) – The training environment

  • callback (BaseCallback) – Callback that will be called at each step (and at the beginning and end of the rollout)

  • rollout_buffer (RolloutBuffer) – Buffer to fill with rollouts

  • n_rollout_steps (int) – Number of experiences to collect per environment

Return type:

bool

Returns:

True if function returned with at least n_rollout_steps collected, False if callback terminated rollout prematurely.

get_env()

Returns the current environment (can be None if not defined).

Return type:

Optional[VecEnv]

Returns:

The current environment

get_parameters()

Return the parameters of the agent. This includes parameters from different networks, e.g. critics (value functions) and policies (pi functions).

Return type:

Dict[str, Dict]

Returns:

Mapping of from names of the objects to PyTorch state-dicts.

get_vec_normalize_env()

Return the VecNormalize wrapper of the training env if it exists.

Return type:

Optional[VecNormalize]

Returns:

The VecNormalize env.

hessian_vector_product(params, grad_kl, vector, retain_graph=True)[source]

Computes the matrix-vector product with the Fisher information matrix.

Parameters:

  • params (List[Parameter]) – list of parameters used to compute the Hessian

  • grad_kl (Tensor) – flattened gradient of the KL divergence between the old and new policy

  • vector (Tensor) – vector to compute the dot product the hessian-vector dot product with

  • retain_graph (bool) – if True, the graph will be kept after computing the Hessian

Return type:

Tensor

Returns:

Hessian-vector dot product (with damping)

learn(total_timesteps, callback=None, log_interval=1, tb_log_name='TRPO', reset_num_timesteps=True, progress_bar=False)[source]

Return a trained model.

Parameters:

  • total_timesteps (int) – The total number of samples (env steps) to train on

  • callback (Union[None, Callable, List[BaseCallback], BaseCallback]) – callback(s) called at every step with state of the algorithm.

  • log_interval (int) – The number of episodes before logging.

  • tb_log_name (str) – the name of the run for TensorBoard logging

  • reset_num_timesteps (bool) – whether or not to reset the current timestep number (used in logging)

  • progress_bar (bool) – Display a progress bar using tqdm and rich.

Return type:

TypeVar(SelfTRPO, bound= TRPO)

Returns:

the trained model

classmethod load(path, env=None, device='auto', custom_objects=None, print_system_info=False, force_reset=True, **kwargs)

Load the model from a zip-file. Warning: load re-creates the model from scratch, it does not update it in-place! For an in-place load use set_parameters instead.

Parameters:

  • path (Union[str, Path, BufferedIOBase]) – path to the file (or a file-like) where to load the agent from

  • env (Union[Env, VecEnv, None]) – the new environment to run the loaded model on (can be None if you only need prediction from a trained model) has priority over any saved environment

  • device (Union[device, str]) – Device on which the code should run.

  • custom_objects (Optional[Dict[str, Any]]) – Dictionary of objects to replace upon loading. If a variable is present in this dictionary as a key, it will not be deserialized and the corresponding item will be used instead. Similar to custom_objects in keras.models.load_model. Useful when you have an object in file that can not be deserialized.

  • print_system_info (bool) – Whether to print system info from the saved model and the current system info (useful to debug loading issues)

  • force_reset (bool) – Force call to reset() before training to avoid unexpected behavior. See https://github.com/DLR-RM/stable-baselines3/issues/597

  • kwargs – extra arguments to change the model when loading

Return type:

TypeVar(SelfBaseAlgorithm, bound= BaseAlgorithm)

Returns:

new model instance with loaded parameters

property logger: Logger

Getter for the logger object.

predict(observation, state=None, episode_start=None, deterministic=False)

Get the policy action from an observation (and optional hidden state). Includes sugar-coating to handle different observations (e.g. normalizing images).

Parameters:

  • observation (Union[ndarray, Dict[str, ndarray]]) – the input observation

  • state (Optional[Tuple[ndarray, ...]]) – The last hidden states (can be None, used in recurrent policies)

  • episode_start (Optional[ndarray]) – The last masks (can be None, used in recurrent policies) this correspond to beginning of episodes, where the hidden states of the RNN must be reset.

  • deterministic (bool) – Whether or not to return deterministic actions.

Return type:

Tuple[ndarray, Optional[Tuple[ndarray, ...]]]

Returns:

the model’s action and the next hidden state (used in recurrent policies)

save(path, exclude=None, include=None)

Save all the attributes of the object and the model parameters in a zip-file.

Parameters:

  • path (Union[str, Path, BufferedIOBase]) – path to the file where the rl agent should be saved

  • exclude (Optional[Iterable[str]]) – name of parameters that should be excluded in addition to the default ones

  • include (Optional[Iterable[str]]) – name of parameters that might be excluded but should be included anyway

Return type:

None

set_env(env, force_reset=True)

Checks the validity of the environment, and if it is coherent, set it as the current environment. Furthermore wrap any non vectorized env into a vectorized checked parameters: - observation_space - action_space

Parameters:
Return type:

None

set_logger(logger)

Setter for for logger object. :rtype: None

Warning

When passing a custom logger object, this will overwrite tensorboard_log and verbose settings passed to the constructor.

set_parameters(load_path_or_dict, exact_match=True, device='auto')

Load parameters from a given zip-file or a nested dictionary containing parameters for different modules (see get_parameters).

Parameters:

  • load_path_or_iter – Location of the saved data (path or file-like, see save), or a nested dictionary containing nn.Module parameters used by the policy. The dictionary maps object names to a state-dictionary returned by torch.nn.Module.state_dict().

  • exact_match (bool) – If True, the given parameters should include parameters for each module and each of their parameters, otherwise raises an Exception. If set to False, this can be used to update only specific parameters.

  • device (Union[device, str]) – Device on which the code should run.

Return type:

None

set_random_seed(seed=None)

Set the seed of the pseudo-random generators (python, numpy, pytorch, gym, action_space)

Parameters:

seed (Optional[int]) –

Return type:

None

train()[source]

Update policy using the currently gathered rollout buffer.

Return type:

None

TRPO Policies

sb3_contrib.trpo.MlpPolicy

alias of ActorCriticPolicy

class stable_baselines3.common.policies.ActorCriticPolicy(observation_space, action_space, lr_schedule, net_arch=None, activation_fn=<class 'torch.nn.modules.activation.Tanh'>, ortho_init=True, use_sde=False, log_std_init=0.0, full_std=True, use_expln=False, squash_output=False, features_extractor_class=<class 'stable_baselines3.common.torch_layers.FlattenExtractor'>, features_extractor_kwargs=None, share_features_extractor=True, normalize_images=True, optimizer_class=<class 'torch.optim.adam.Adam'>, optimizer_kwargs=None)[source]

Policy class for actor-critic algorithms (has both policy and value prediction). Used by A2C, PPO and the likes.

Parameters:

  • observation_space (Space) – Observation space

  • action_space (Space) – Action space

  • lr_schedule (Callable[[float], float]) – Learning rate schedule (could be constant)

  • net_arch (Union[List[int], Dict[str, List[int]], None]) – The specification of the policy and value networks.

  • activation_fn (Type[Module]) – Activation function

  • ortho_init (bool) – Whether to use or not orthogonal initialization

  • use_sde (bool) – Whether to use State Dependent Exploration or not

  • log_std_init (float) – Initial value for the log standard deviation

  • full_std (bool) – Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE

  • use_expln (bool) – Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

  • squash_output (bool) – Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.

  • features_extractor_class (Type[BaseFeaturesExtractor]) – Features extractor to use.

  • features_extractor_kwargs (Optional[Dict[str, Any]]) – Keyword arguments to pass to the features extractor.

  • share_features_extractor (bool) – If True, the features extractor is shared between the policy and value networks.

  • normalize_images (bool) – Whether to normalize images or not, dividing by 255.0 (True by default)

  • optimizer_class (Type[Optimizer]) – The optimizer to use, th.optim.Adam by default

  • optimizer_kwargs (Optional[Dict[str, Any]]) – Additional keyword arguments, excluding the learning rate, to pass to the optimizer

evaluate_actions(obs, actions)[source]

Evaluate actions according to the current policy, given the observations.

Parameters:

  • obs (Tensor) – Observation

  • actions (Tensor) – Actions

Return type:

Tuple[Tensor, Tensor, Optional[Tensor]]

Returns:

estimated value, log likelihood of taking those actions and entropy of the action distribution.

extract_features(obs)[source]

Preprocess the observation if needed and extract features.

Parameters:

obs (Tensor) – Observation

Return type:

Union[Tensor, Tuple[Tensor, Tensor]]

Returns:

the output of the features extractor(s)

forward(obs, deterministic=False)[source]

Forward pass in all the networks (actor and critic)

Parameters:

  • obs (Tensor) – Observation

  • deterministic (bool) – Whether to sample or use deterministic actions

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

action, value and log probability of the action

get_distribution(obs)[source]

Get the current policy distribution given the observations.

Parameters:

obs (Tensor) –

Return type:

Distribution

Returns:

the action distribution.

predict_values(obs)[source]

Get the estimated values according to the current policy given the observations.

Parameters:

obs (Tensor) – Observation

Return type:

Tensor

Returns:

the estimated values.

reset_noise(n_envs=1)[source]

Sample new weights for the exploration matrix.

Parameters:

n_envs (int) –

Return type:

None

sb3_contrib.trpo.CnnPolicy

alias of ActorCriticCnnPolicy

class stable_baselines3.common.policies.ActorCriticCnnPolicy(observation_space, action_space, lr_schedule, net_arch=None, activation_fn=<class 'torch.nn.modules.activation.Tanh'>, ortho_init=True, use_sde=False, log_std_init=0.0, full_std=True, use_expln=False, squash_output=False, features_extractor_class=<class 'stable_baselines3.common.torch_layers.NatureCNN'>, features_extractor_kwargs=None, share_features_extractor=True, normalize_images=True, optimizer_class=<class 'torch.optim.adam.Adam'>, optimizer_kwargs=None)[source]

CNN policy class for actor-critic algorithms (has both policy and value prediction). Used by A2C, PPO and the likes.

Parameters:

  • observation_space (Space) – Observation space

  • action_space (Space) – Action space

  • lr_schedule (Callable[[float], float]) – Learning rate schedule (could be constant)

  • net_arch (Union[List[int], Dict[str, List[int]], None]) – The specification of the policy and value networks.

  • activation_fn (Type[Module]) – Activation function

  • ortho_init (bool) – Whether to use or not orthogonal initialization

  • use_sde (bool) – Whether to use State Dependent Exploration or not

  • log_std_init (float) – Initial value for the log standard deviation

  • full_std (bool) – Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE

  • use_expln (bool) – Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

  • squash_output (bool) – Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.

  • features_extractor_class (Type[BaseFeaturesExtractor]) – Features extractor to use.

  • features_extractor_kwargs (Optional[Dict[str, Any]]) – Keyword arguments to pass to the features extractor.

  • share_features_extractor (bool) – If True, the features extractor is shared between the policy and value networks.

  • normalize_images (bool) – Whether to normalize images or not, dividing by 255.0 (True by default)

  • optimizer_class (Type[Optimizer]) – The optimizer to use, th.optim.Adam by default

  • optimizer_kwargs (Optional[Dict[str, Any]]) – Additional keyword arguments, excluding the learning rate, to pass to the optimizer

sb3_contrib.trpo.MultiInputPolicy

alias of MultiInputActorCriticPolicy

class stable_baselines3.common.policies.MultiInputActorCriticPolicy(observation_space, action_space, lr_schedule, net_arch=None, activation_fn=<class 'torch.nn.modules.activation.Tanh'>, ortho_init=True, use_sde=False, log_std_init=0.0, full_std=True, use_expln=False, squash_output=False, features_extractor_class=<class 'stable_baselines3.common.torch_layers.CombinedExtractor'>, features_extractor_kwargs=None, share_features_extractor=True, normalize_images=True, optimizer_class=<class 'torch.optim.adam.Adam'>, optimizer_kwargs=None)[source]

MultiInputActorClass policy class for actor-critic algorithms (has both policy and value prediction). Used by A2C, PPO and the likes.

Parameters:

  • observation_space (Dict) – Observation space (Tuple)

  • action_space (Space) – Action space

  • lr_schedule (Callable[[float], float]) – Learning rate schedule (could be constant)

  • net_arch (Union[List[int], Dict[str, List[int]], None]) – The specification of the policy and value networks.

  • activation_fn (Type[Module]) – Activation function

  • ortho_init (bool) – Whether to use or not orthogonal initialization

  • use_sde (bool) – Whether to use State Dependent Exploration or not

  • log_std_init (float) – Initial value for the log standard deviation

  • full_std (bool) – Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE

  • use_expln (bool) – Use expln() function instead of exp() to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, exp() is usually enough.

  • squash_output (bool) – Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE.

  • features_extractor_class (Type[BaseFeaturesExtractor]) – Uses the CombinedExtractor

  • features_extractor_kwargs (Optional[Dict[str, Any]]) – Keyword arguments to pass to the features extractor.

  • share_features_extractor (bool) – If True, the features extractor is shared between the policy and value networks.

  • normalize_images (bool) – Whether to normalize images or not, dividing by 255.0 (True by default)

  • optimizer_class (Type[Optimizer]) – The optimizer to use, th.optim.Adam by default

  • optimizer_kwargs (Optional[Dict[str, Any]]) – Additional keyword arguments, excluding the learning rate, to pass to the optimizer