Algorithm

class gops.algorithm.base.AlgorithmBase(index, **kwargs)

Base Class of Algorithm

Parameters:: index (int) – used for calculating offset of random seed for subprocess.

abstract property adjustable_parameters: tuple: Return all the adjustable hyperparameters of the algorithm

get_parameters(): Get the current hyperparameters of the algorithm

set_parameters(param_dict): Set hyperparameters of the algorithm

class gops.algorithm.dqn.DQN(index=0, **kwargs)

Deep Q-Network (DQN) algorithm

Paper: https://doi.org/10.1038/nature14236

Parameters:

gamma (float, optional) – discount factor. Defaults to 0.99.
tau (float, optional) – target network update rate. Defaults to 0.005.

class gops.algorithm.ddpg.DDPG(index=0, buffer_name='replay_buffer', **kwargs)

Deep Deterministic Policy Gradient (DDPG) algorithm

Paper: https://arxiv.org/pdf/1509.02971.pdf

Parameters:

buffer_name (string) – buffer type. Default to ‘replay_buffer’.
gamma (float) – discount factor. Default to 0.99.
tau (float) – param for soft update of target network. Default to 0.005.
delay_update (int) – delay update steps for actor. Default to 1.

class gops.algorithm.td3.TD3(target_noise=0.2, noise_clip=0.5, buffer_name='replay_buffer', index=0, **kwargs)

Twin Delayed Deep Deterministic policy gradient (TD3) algorithm

Paper: https://arxiv.org/pdf/1802.09477.pdf

Parameters:

action_high_limit (list) – action limit for available actions.
target_noise (float) – action noise for target pi network. Default to 0.2
noise_clip (float) – range [-noise_clip, noise_clip] for target_noise. Default to 0.5
buffer_name (string) – buffer type. Default to ‘replay_buffer’.
index (int) – for calculating offset of random seed for subprocess. Default to 0.

class gops.algorithm.sac.SAC(index: int = 0, gamma: float = 0.99, tau: float = 0.005, auto_alpha: bool = True, alpha: float = 0.2, target_entropy: float | None = None, **kwargs: Any)

Soft Actor-Critic (SAC) algorithm

Paper: https://arxiv.org/abs/1801.01290

Parameters:

gamma (float) – discount factor.
tau (float) – param for soft update of target network.
auto_alpha (bool) – whether to adjust temperature automatically.
alpha (float) – initial temperature.
target_entropy (Optional[float]) – target entropy for automatic temperature adjustment.

class gops.algorithm.dsac.DSAC(index=0, **kwargs)

DSAC algorithm

Paper: https://arxiv.org/pdf/2001.02811

Parameters:

gamma (float) – discount factor.
tau (float) – param for soft update of target network.
auto_alpha (bool) – whether to adjust temperature automatically.
alpha (float) – initial temperature.
TD_bound (float) – the bound of temporal difference.
bound (bool) – whether to bound the q value.
delay_update (float) – delay update steps for actor.
target_entropy (Optional[float]) – target entropy for automatic temperature adjustment.

class gops.algorithm.dsac2.DSAC2(index=0, **kwargs)

Modified DSAC algorithm

Paper: https://arxiv.org/pdf/2001.02811

Parameters:

gamma (float) – discount factor.
tau (float) – param for soft update of target network.
auto_alpha (bool) – whether to adjust temperature automatically.
alpha (float) – initial temperature.
delay_update (float) – delay update steps for actor.
target_entropy (Optional[float]) – target entropy for automatic temperature adjustment.

class gops.algorithm.trpo.TRPO(*, delta: float, norm_adv: bool, rtol: float, atol: float, damping_factor: float, max_cg: int, alpha: float, max_search: int, train_v_iters: int, value_learning_rate: float, index=0, **kwargs)

TRPO algorithm Paper: https://arxiv.org/abs/1502.05477

Parameters:

delta – KL constraint
norm_adv – whether to normalize advantage
rtol – CG’s relative tolerance
atol – CG’s absolute tolerance
damping_factor – Add $lambda I$ damping to Hessian to improve CG solution.
max_cg – CG’s maximum iterations if failing to converge.
alpha – Backtrack search factor.
max_search – Backtrack search maximum iterations.
train_v_iters – State value training iterations each policy update.
value_learning_rate – State value learning rate

class gops.algorithm.ppo.PPO(*, max_iteration: int, num_repeat: int, num_mini_batch: int, mini_batch_size: int, sample_batch_size: int, index=0, **kwargs)

PPO algorithm Paper: https://arxiv.org/abs/1707.06347

Parameters:

max_iteration – Maximum iterations for learning rate schedule.
num_repeat – Number of repeats (to reuse sample batch).
num_mini_batch – Number of minibatches to divide sample batch.
mini_batch_size – Minibatch size.
sample_batch_size – Sample batch size.

class gops.algorithm.fhadp.FHADP(index=0, **kwargs)

Approximate Dynamic Program Algorithm for Finity Horizon

Paper: https://link.springer.com/book/10.1007/978-981-19-7784-8

Parameters:

forward_step (int) – envmodel forward step.
gamma (float) – discount factor.

class gops.algorithm.infadp.INFADP(index=0, **kwargs)

Approximate Dynamic Program Algorithm for Infinity Horizon Paper: https://link.springer.com/book/10.1007/978-981-19-7784-8

Parameters:

forward_step (int) – envmodel forward step.
gamma (float) – discount factor.
tau (float) – param for soft update of target network.
pev_step (int) – number of steps for policy evaluation.
pim_step (int) – number of steps for policy improvement.

class gops.algorithm.mac.MAC(index=0, **kwargs)

Mixed Actor Critic Algorithm (MAC) algorithm

Paper:https://ieeexplore.ieee.org/document/9268413

Parameters:

gamma (float) – discount factor.
tau (float) – param for soft update of target network.
pev_step (int) – number of steps for policy evaluation.
pim_step (int) – number of steps for policy improvement.
forward_step (int) – envmodel forward step.

class gops.algorithm.mpg.MPG(index: int = 0, terminal_iter: int = 10000, eta: float = 0.1, kappa: float = 0.5, gamma: float = 0.99, tau: float = 0.1, delay_update: int = 1, forward_step: int = 10, **kwargs): Mixed Policy Gradient (MPG) algorithm Paper: https://arxiv.org/abs/2102.11513.

class gops.algorithm.spil.SPIL(index: int = 0, gamma: float = 0.99, tau: float = 0.005, pev_step: int = 1, pim_step: int = 1, forward_step: int = 25, **kwargs: Any)

Separated Proportional-Integral Lagrangian (SPIL) algorithm

Paper: https://ieeexplore.ieee.org/document/9785377

Parameters:

gamma (float) – discount factor.
tau (float) – param for soft update of target network.
pev_step (int) – initial policy evaluation step.
pim_step (int) – initial policy improvement step.
forward_step (int) – predictive step in virtual horizon.

class gops.algorithm.rpi.RPI(index: int = 0, max_newton_iteration: int = 50, max_step_update_value: int = 10000, print_interval: int = 1, learning_rate: float = 0.001, **kwargs): Relaxed Policy Iteration (RPI) algorithm Paper: https://arxiv.org/abs/2007.06810.